Payload delivery: inline or link

A watch run can deliver its records two ways. By default they ride inside the notification. Set config.payload_delivery_type to "link" and the notification carries a pre-signed S3 URL instead, pointing at one file that holds every record of that run.

{
  "config": {
    "trigger": { "type": "interval", "every_hours": 24 },
    "payload_delivery_type": "link"
  }
}

This is available on every watch: person, company, and job discovery watches, and person and company entity watches. It applies on create and on PATCH.

Why you would switch

Most receivers cap how large a request body they accept, and a body over the cap is refused whole. An AWS Lambda proxy integration stops at 6 MB, an ALB target at 1 MB. A run that delivers 1,000 enriched people can pass either of those. Inline delivery splits a large run across several requests to stay under a receiver’s cap. Link delivery sidesteps the cap instead. The body carries no records, so its size does not grow with the run. Pick link if your receiver has a small body cap, or if you would rather load a run as one file than reassemble it from several requests.

The two modes

	`inline` (default)	`link`
Where the records are	In the notification body, under `results`	In one file, behind a pre-signed S3 URL
Requests per run	One per chunk, so several on a large run	Always one
Body size	Grows with the run	Fixed, a few hundred bytes
File format	JSON, in the body	NDJSON, one record per line
URL lifetime	n/a	5 days, signed

Set it on a new watch

curl --request POST \
  --url https://api.crustdata.com/watch/person/search \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "filters": {
      "op": "and",
      "conditions": [
        { "field": "experience.employment_details.current.title", "type": "(.)", "value": "Machine Learning Engineer" }
      ]
    },
    "config": {
      "trigger": { "type": "interval", "every_hours": 24 },
      "max_results_per_run": 1000,
      "payload_delivery_type": "link"
    },
    "notifications": [ { "type": "webhook", "url": "https://your-app.com/webhooks/crustdata" } ]
  }'

Switch an existing watch

payload_delivery_type is one of the mutable config keys, alongside the schedule and the result cap. Change it and the watch’s filters, tracked fields, and sort order stay exactly as they were. A config PATCH is validated as a whole block, so repeat the watch’s current trigger alongside the key you are changing. Leave it out and the request comes back 400 with config.trigger.type must be one of ['interval']. The keys you do send are merged into the stored config rather than replacing it, so anything you omit keeps its current value.

curl --request PATCH \
  --url https://api.crustdata.com/watch/person/search/46849 \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "config": {
      "trigger": { "type": "interval", "every_hours": 24 },
      "payload_delivery_type": "link"
    }
  }'

curl --request PATCH \
  --url https://api.crustdata.com/watch/person/46936 \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "config": {
      "trigger": { "type": "interval", "every_hours": 24 },
      "payload_delivery_type": "link"
    }
  }'

The change takes effect on the next run. Runs already delivered keep whatever they were delivered as.

The watch’s config in a create, GET, or PATCH response does not echo payload_delivery_type today. To confirm which mode a run used, read metadata.payload_delivery.type on the notification, or payload_delivery.type on the run summary.

What the notification looks like

Both modes carry the same metadata, including summary.delivered and summary.truncated, so your record counts do not move. Only the records themselves change places.

{
  "metadata": {
    "watch_id": 46849,
    "kind": "discovery",
    "dataset": "person",
    "api_version": "2025-11-01",
    "run_id": 64200,
    "notification_id": "ntf_64200",
    "delivered_at": "2026-08-27T03:20:00.000000Z",
    "summary": { "delivered": 842, "total_count": 1204, "max_results_per_run": 1000, "truncated": true },
    "payload_delivery": {
      "type": "link",
      "url": "https://crustdata-batch-api-data.s3.amazonaws.com/watcher/watch=46849/run=64200/records.ndjson?...&X-Amz-Expires=432000",
      "format": "ndjson",
      "bytes": 4823901,
      "expires_at": "2026-09-01T03:20:00.000000+00:00"
    }
  }
}

{
  "metadata": {
    "watch_id": 46849,
    "kind": "discovery",
    "dataset": "person",
    "api_version": "2025-11-01",
    "run_id": 64200,
    "notification_id": "ntf_64200",
    "delivered_at": "2026-08-27T03:20:00.000000Z",
    "summary": { "delivered": 842, "total_count": 1204, "max_results_per_run": 1000, "truncated": true },
    "payload_delivery": { "type": "inline" }
  },
  "results": { "added": [ { "basic_profile": { "…": "…" }, "crustdata_person_id": 6324687 } ] }
}

On a link delivery the body has no results key at all. Read metadata.payload_delivery.type and branch on it rather than testing for the presence of results.

Field	Type	Meaning
`type`	string	`"link"` or `"inline"`. Where the records of this delivery actually travelled.
`url`	string	Pre-signed S3 URL for the run’s file. Present only when `type` is `"link"`.
`format`	string	`"ndjson"`.
`bytes`	integer	Size of the file.
`expires_at`	string	When the link stops working, 5 days after it was issued.

metadata.payload_delivery is present on every delivery, inline and link alike. An inline delivery carries { "type": "inline" } with no other keys, so read type before you reach for url.

The file

One file per run, in NDJSON: one complete JSON object per line, newline terminated, with no wrapping array. You can stream it line by line without holding the whole run in memory. Each line is the same record the inline body would have carried, with no container around it.

{"basic_profile": {"name": "…"}, "crustdata_person_id": 6324687}
{"basic_profile": {"name": "…"}, "crustdata_person_id": 6324688}

{"changes": [{"field": "headcount.total", "type": "=>", "value": 1000, "from": 900, "to": 1500}], "record": {"basic_info": {"name": "…"}, "crustdata_company_id": 12345}}
{"changes": [{"field": "headcount.total", "type": "=>", "value": 1000, "from": 970, "to": 1042}], "record": {"basic_info": {"name": "…"}, "crustdata_company_id": 12346}}

A discovery watch’s lines are the raw dataset records, without the { "added": [ … ] } wrapper the inline body uses, because a file of lines needs no wrapper. An entity watch’s lines are the same { changes, record } objects the inline body carries. Fetch the file with a plain GET. The URL carries its own authentication in the query string, so send no authorization header:

curl --silent 'https://crustdata-batch-api-data.s3.amazonaws.com/watcher/watch=46849/run=64200/records.ndjson?...&X-Amz-Expires=432000' \
  | jq -c '.crustdata_person_id'

The URL is signed for 5 days. Download the file when the notification arrives rather than storing the URL for later. If you need the run again after that, the run summary issues a fresh link over the same records.

Every channel gets the same link

A run writes its file once, before it fans out. A webhook, a Slack message, a Google Chat message, and an email from the same run all carry the same link to the same bytes. Slack, Google Chat, and email messages already show a bounded preview of a run rather than every record. On a link watch those messages keep their preview and gain the link, so a reader who wants the records the preview left out has the full file.

Reading a past run

The run-summary endpoint follows the watch’s mode too.

GET https://api.crustdata.com/watch/{dataset}/{watch_id}/runs/{run_id}/summary

On a link watch the response carries a top-level payload_delivery block, and each entry in notifications keeps its delivery outcome but drops the records:

{
  "id": 64200,
  "status": "SUCCESS",
  "new_records_count": 842,
  "payload_delivery": {
    "type": "link",
    "url": "https://crustdata-batch-api-data.s3.amazonaws.com/watcher/watch=46849/run=64200/records.ndjson?...&X-Amz-Expires=432000",
    "format": "ndjson",
    "bytes": 4823901,
    "expires_at": "2026-09-01T03:20:00.000000+00:00"
  },
  "notifications": [
    { "sent_at": "2026-08-27T03:20:00Z", "http_status": 200 }
  ]
}

On an inline watch, payload_delivery is { "type": "inline" } and notifications[].payload carries the records as before. Reading a run hands you the same file the run delivered, so the bytes match what your channel received. A watch created with no channels never pushed anything, so the first read writes the file and links it. A run that delivered nothing offers no link. A SKIPPED or FAILED run reports { "type": "inline" } with an empty notifications list.

Inline is never overridden

A watch left on inline posts its records at any size. Delivery does not switch a large run to a link on its own, because the mode is your choice and a silent switch would change the body shape your receiver parses. A large inline run is split instead: records are grouped into requests of at most ~1 MB or 100 records, and each request carries its own notification_id (ntf_{run_id}_{index}) and a metadata.chunk.index. A run that fits in one request keeps the plain ntf_{run_id} id and no chunk key, so a small run looks exactly as it always has. If your receiver rejects an oversize body, that is the signal to move the watch to link.

If the file cannot be written

Writing the file can fail. When it does, the run falls back to delivering the records inline and says so: metadata.payload_delivery reports { "type": "inline" } and the body carries results as usual. The delivery still reaches you, without the link. This is the reason to branch on metadata.payload_delivery.type rather than on the watch’s configured mode. A link watch can still hand you an inline body.

Test sends

A test send on a link watch to a webhook mirrors a real delivery: the body carries metadata.payload_delivery with a link and no results, alongside the usual metadata.test: true. The file holds the sample records. Each test preview writes its own file rather than reusing the run’s, so a preview never overwrites the records of a real run.

Errors

A value other than "inline" or "link" is rejected on create and on PATCH:

400 Bad Request

{
  "non_field_errors": [
    "config.payload_delivery_type must be one of ['inline', 'link']."
  ]
}

The check is case sensitive, so "LINK" is rejected. Omitting the key, or sending it as null, means inline.

Pricing

Neither mode changes what a watch costs. You pay per record delivered, at the same rate either way, and reading the file costs no credits. See Pricing for the per-watch rates.

Person discovery watcher

Turn a person search filter into a recurring feed.

Person entity watcher

Watch a known list of people for profile changes.

Company discovery watcher

Turn a company search filter into a recurring feed.

Job watcher

Follow new job postings matching your filters.

​Why you would switch

​The two modes

​Set it on a new watch

​Switch an existing watch

​What the notification looks like

​The file

​Every channel gets the same link

​Reading a past run

​Inline is never overridden

​If the file cannot be written

​Test sends

​Errors

​Pricing

​Related

Person discovery watcher

Person entity watcher

Company discovery watcher

Job watcher

Why you would switch

The two modes

Set it on a new watch

Switch an existing watch

What the notification looks like

The file

Every channel gets the same link

Reading a past run

Inline is never overridden

If the file cannot be written

Test sends

Errors

Pricing

Related