Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.crustdata.com/llms.txt

Use this file to discover all available pages before exploring further.

POST /screener/web-search
POST /web/search/live
This page lists every behavioral and contract change between the legacy web-search endpoint and the current one. Use it as a side-by-side reference when porting an integration; each section gives the old shape, the new shape, and the smallest edit that closes the gap.
TopicLegacyCurrent
PathPOST /screener/web-searchPOST /web/search/live
MethodPOST with a JSON bodyPOST with a JSON body
AuthAuthorization: Token <key>Authorization: Bearer <key>
Version(not required)x-api-version: 2025-11-01 (required)
Base URLhttps://api.crustdata.comhttps://api.crustdata.com
Two header changes are required. The authorization scheme moved from Token <key> to Bearer <key>, and the x-api-version: 2025-11-01 header is now required on every call. Calls missing either header are rejected.
The companion content-fetch endpoint has also moved — the legacy ?fetch_content=true query parameter and the standalone POST /screener/web-fetch route are replaced by Web Fetch at POST /web/enrich/live. Both endpoints are now strictly separated.

​
1. Request body — top-level keys

The request shape is mostly preserved. Three parameters were renamed (camelCase to snake_case), one was renamed semantically (solveCloudflare → human_mode), and the ?fetch_content=true query parameter was removed in favor of calling the dedicated fetch endpoint.
Legacy keyCurrent keyNotes
queryqueryRequired. Same semantics. Max length raised from 1000 to 5000 characters.
geolocationlocationRenamed. Same ISO 3166-1 alpha-2 country codes (US, GB, JP, …).
sourcessourcesUnchanged enum: news, web, scholar-articles, scholar-articles-enriched, scholar-author, ai, social.
sitesiteUnchanged.
startDatestart_dateRenamed to snake_case. Same Unix-seconds timestamp.
endDateend_dateRenamed to snake_case. Same Unix-seconds timestamp.
solveCloudflarehuman_modeRenamed. Same boolean semantics — attempt a browser-like retrieval path when standard access is blocked. Now neutrally named.
numPagespageRenamed. Now represents the page number to fetch (1-indexed), not “number of pages to fetch in one call”. Call repeatedly to paginate.
?fetch_content=true (query string)(removed)Augmented fetch is no longer offered on this endpoint. Call Web Fetch at POST /web/enrich/live separately.
numPages → page is a semantic change, not just a rename. The legacy numPages meant “fetch N pages of results in a single call”. The current page means “return page number N”. To fetch multiple pages, call the endpoint once per page.

​
2. Removed features

​
?fetch_content=true query parameter

The legacy endpoint accepted a ?fetch_content=true query parameter, which augmented the response with fetched page contents under a contents array. This parameter has been removed. To retrieve page content for search-result URLs, call Web Fetch at POST /web/enrich/live with the URLs returned by /web/search/live.

​
Companion POST /screener/web-fetch

The legacy standalone fetch route has moved to POST /web/enrich/live. Same input shape ({ urls: [...] }), same per-call cap of 10 URLs.

​
numPages multi-page fetch in one call

The legacy numPages parameter retrieved up to N result pages in a single response. The current endpoint serves one page per call via the page parameter. Pre-check your credit budget before paginating aggressively.

​
Free-form error strings

Legacy 400 and 500 responses returned a single error string. The current endpoint returns a structured error envelope — see Error responses.

​
3. Field-name mapping

The response envelope is the same overall, but two fields under metadata were renamed and failed_pages / empty_pages were added for paginated diagnostics.

​
Request fields

Legacy fieldCurrent field
geolocationlocation
startDatestart_date
endDateend_date
solveCloudflarehuman_mode
numPagespage

​
Response fields

Legacy fieldCurrent fieldNotes
successsuccessUnchanged.
queryqueryUnchanged.
timestamptimestampUnchanged. Unix milliseconds.
results[]results[]Same array. Each result now carries a source field.
metadata.totalResultsmetadata.total_resultsRenamed to snake_case.
(new)metadata.failed_pages[]Pages that failed to return results.
(new)metadata.empty_pages[]Pages that returned no results.
contents[] (when ?fetch_content=true)(removed) — call Web Fetch separatelyDrop this field; fetch separately.

​
Per-result fields

The legacy response returned the same per-result shape for all sources. The current response carries a top-level source field on each entry and adds source-specific fields. Common fields (title, url, snippet, position) are unchanged.
Legacy fieldCurrent fieldNotes
titletitleUnchanged.
urlurlUnchanged.
snippetsnippetUnchanged. Present for web, news, scholar-articles results.
positionpositionUnchanged. 1-indexed.
(new)sourceThe source type that returned this result (web, news, scholar-articles, …).
Source-specific additions on the current endpoint:
SourceAdditional response fields
scholar-articlesauthors[], citations (integer), pdf_url, metadata (citation string).
scholar-authorname, affiliation, website, interests[], articles[], citations, h_index, i10_index.
aicontent (generated overview), references[], images[] (no snippet for this source).

​
4. Sources

Source values are unchanged: news, web, scholar-articles, scholar-articles-enriched, scholar-author, ai, social. For the per-source response shape and recipe-style usage, see Web Search sources.

​
5. Error responses

The error envelope changed shape. Status codes are unchanged (400, 401, 402, 500). 
{
    "error": "query: This field is required."
}
401 continues to use a flat { "message": "Invalid API key in request" } shape — parse based on HTTP status. Common error cases:
StatusCause
400query missing, empty, or whitespace-only.
400sources[] contains a value outside the supported enum.
400start_date is greater than or equal to end_date.
400Insufficient credits for the requested number of pages.
401Missing or invalid Authorization header.
500Internal error while performing the search.

​
6. End-to-end example

A multi-source search with site restriction and date filtering — written against both endpoints. 
curl --request POST \
  --url 'https://api.crustdata.com/screener/web-search' \
  --header 'authorization: Token YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --data '{
    "query": "distributed systems",
    "geolocation": "US",
    "sources": ["web", "news"],
    "site": "example.com",
    "startDate": 1728259200,
    "endDate": 1730937600,
    "numPages": 1
  }'
Fetching page contents — split into two calls now: 
curl --request POST \
  --url 'https://api.crustdata.com/screener/web-search?fetch_content=true' \
  --header 'authorization: Token YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --data '{ "query": "crustdata", "geolocation": "US" }'

​
Migration checklist

  • Switch from POST /screener/web-search to POST /web/search/live.
  • Change the Authorization scheme from Token to Bearer.
  • Add x-api-version: 2025-11-01 header to every request.
  • Rename request keys: geolocation → location, startDate → start_date, endDate → end_date, solveCloudflare → human_mode, numPages → page.
  • Update pagination logic — page returns one page, not N pages. Loop over page values to paginate.
  • Drop the ?fetch_content=true query parameter. Call Web Fetch at POST /web/enrich/live with the result URLs instead.
  • If you used the legacy POST /screener/web-fetch route, switch to POST /web/enrich/live (same body shape, same 10-URL-per-call cap).
  • Rename metadata.totalResults reads to metadata.total_results.
  • Read the new source field on each result if you display source labels client-side.
  • Adopt per-source response fields when you query scholar-articles, scholar-author, or ai (see Field-name mapping).
  • Update error handlers for the new error.type / error.message envelope.
  • If you used the legacy auto-augmented contents[] array on the response, replace it with a follow-up call to Web Fetch.

​
See also