You are viewing the documentation of the new API versions. We would love to hear from you. You can write to use at support@crustdata.co for feedback and clarifications.
Reference for Web Fetch: request parameters, response body details, error handling, common gotchas, and API summary.
Reference material for Web Fetch: request parameters,
response body details, error handling, common gotchas, and the API summary.For walk-through examples, see Web Fetch and
Recipes.
The response is a top-level array. Each item corresponds to one URL in your
request.
Field
Type
Description
success
boolean
Whether this URL was fetched successfully.
url
string?
The URL that was fetched. null if the fetch failed.
timestamp
integer?
Unix timestamp (seconds) when fetched. null on failure.
title
string?
The <title> tag content. null on failure.
content
string?
Full HTML content of the page. null on failure.
Timestamps: Fetch timestamps are in seconds. Search timestamps are
in milliseconds. Account for this when comparing timestamps across
endpoints.
Two kinds of failure, two places to check:
Request-level errors (400, 401) — the entire request failed. You get an error object, not an array. Caused by missing fields, empty arrays, or bad auth.
Per-URL failures within a 200 — individual entries with success: false and null fields. Caused by unreachable URLs, timeouts, or bot protection.
Always check the HTTP status first, then check success for each entry
in the array.