Web fetch - Crustdata Docs

POST /screener/web-fetch

POST /web/enrich/live

This page lists every behavioral and contract change between the legacy web-fetch endpoint and the current one. Use it as a side-by-side reference when porting an integration; each section gives the old shape, the new shape, and the smallest edit that closes the gap.

Topic	Legacy	Current
Path	`POST /screener/web-fetch`	`POST /web/enrich/live`
Method	`POST` with a JSON body	`POST` with a JSON body
Auth	`Authorization: Token <key>`	`Authorization: Bearer <key>`
Version	(not required)	`x-api-version: 2025-11-01` (required)
Base URL	`https://api.crustdata.com`	`https://api.crustdata.com`

Two header changes are required. The authorization scheme moved from Token <key> to Bearer <key>, and the x-api-version: 2025-11-01 header is now required on every call. Calls missing either header are rejected.

The companion search endpoint has also moved — POST /screener/web-search is replaced by Web Search at POST /web/search/live. The two endpoints are now strictly separated; the legacy ?fetch_content=true mode on /screener/web-search is removed, and /web/enrich/live is the only way to retrieve page content.

1. Request body — top-level keys

The request shape is preserved almost exactly. One field was renamed semantically (solveCloudflare → human_mode); everything else carries across as-is.

Legacy key	Current key	Notes
`urls`	`urls`	Required. Array of fully-qualified URLs (each must include `http://` or `https://`). Maximum 10 URLs per request (unchanged).
`solveCloudflare`	`human_mode`	Renamed. Same boolean semantics — attempt a browser-like retrieval path when standard fetch is blocked. Now neutrally named.

One value rename to apply across your codebase: solveCloudflare → human_mode. Same true/false semantics. No other request keys need to change.

2. Removed features

Free-form `error` strings

Legacy 400 and 500 responses returned a single error string. The current endpoint returns a structured error envelope — see Error responses.

`solveCloudflare` key name

Renamed to human_mode. The legacy key is not accepted by the new endpoint.

3. Field-name mapping

The request and response shapes are nearly identical — one request key was renamed and one response field was renamed.

Request fields

Legacy field	Current field
`urls`	`urls`
`solveCloudflare`	`human_mode`

Response per-result fields

The response remains a top-level array, one entry per URL submitted. Each entry’s shape changed in one place: pageTitle → title.

Legacy field	Current field	Notes
`success`	`success`	Unchanged. Boolean.
`url`	`url`	Unchanged. The URL that was fetched. `null` when the fetch fails.
`timestamp`	`timestamp`	Unchanged. Unix timestamp in seconds (not milliseconds). `null` when the fetch fails.
`pageTitle`	`title`	Renamed. Same value — the page title parsed from the HTML `<title>` tag. `null` when the fetch fails.
`content`	`content`	Unchanged. Full HTML content. `null` when the fetch fails.

Partial failures stay in the response. If you submit 5 URLs and 2 fail, you still receive 5 entries — the failed entries have success: false and url, timestamp, title, content all null. Don’t reject the whole response on a partial failure.

4. Behavior and limits

Topic	Legacy	Current
URLs per call	1–10	1–10 (unchanged)
Max URL length	2000 characters	2000 characters (unchanged)
Response envelope	Top-level array of fetch results	Top-level array of fetch results (unchanged shape)
Partial-failure handling	Returns one entry per URL with null fields on failure	Returns one entry per URL with null fields on failure (unchanged)
Cloudflare bypass parameter	`solveCloudflare`	`human_mode`
Empty `urls` array	`400` with `"This list may not be empty"`	`400` with `{ error: { type: "invalid_request", message: "urls: This list may not be empty.", metadata: [] } }`
Missing `urls` field	`400` with `"This field is required"`	`400` with `{ error: { type: "invalid_request", message: "urls: This field is required.", metadata: [] } }`

5. Error responses

The error envelope changed shape. Status codes are unchanged (400, 401, 402, 500).

{
    "error": "urls: This field is required."
}

401 continues to use a flat { "message": "Invalid API key in request" } shape — parse based on HTTP status. Common error cases:

Status	Cause
`400`	`urls` missing or empty array.
`400`	More than 10 URLs submitted.
`400`	A URL longer than 2000 characters.
`401`	Missing or invalid `Authorization` header.
`402`	Insufficient credits.
`500`	Internal error while fetching content.

Per-URL failures are not 4xx/5xx responses. When the endpoint succeeds at the request level but individual URLs cannot be retrieved, you get 200 with the failing entries marked success: false. Only request-level problems (missing urls, invalid auth, etc.) return non-2xx.

6. End-to-end example

Fetching content for a handful of URLs — written against both endpoints.

curl --request POST \
  --url 'https://api.crustdata.com/screener/web-fetch' \
  --header 'authorization: Token YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --data '{
    "urls": [
      "https://example.com",
      "https://www.crustdata.com"
    ],
    "solveCloudflare": false
  }'

Response shape (both endpoints return a top-level array):

[
    {
        "success": true,
        "url": "https://www.example.com",
        "timestamp": 1765188176,
        "pageTitle": "Example Domain",
        "content": "<html lang=\"en\"><head><title>Example Domain</title>..."
    }
]

Migration checklist

Switch from POST /screener/web-fetch to POST /web/enrich/live.
Change the Authorization scheme from Token to Bearer.
Add x-api-version: 2025-11-01 header to every request.
Rename the request key solveCloudflare → human_mode.
Rename response reads pageTitle → title.
Update error handlers for the new error.type / error.message envelope.
If you previously composed /screener/web-search?fetch_content=true into a single call, split it into a Web Search call followed by a /web/enrich/live call with the result URLs.
Continue handling partial failures: a successful request can still contain entries with success: false and null fields.

Documentation Index

​1. Request body — top-level keys

​2. Removed features

​Free-form error strings

​solveCloudflare key name

​3. Field-name mapping

​Request fields

​Response per-result fields

​4. Behavior and limits

​5. Error responses

​6. End-to-end example

​Migration checklist

​See also

1. Request body — top-level keys

2. Removed features

Free-form `error` strings

`solveCloudflare` key name

3. Field-name mapping

Request fields

Response per-result fields

4. Behavior and limits

5. Error responses

6. End-to-end example

Migration checklist

See also