Web Search reference - Crustdata Docs

Reference material for Web Search: request parameters, response body, error handling, common gotchas, and the API summary. For walk-through examples, see Web Search and Examples. For result shapes and field presence by source, see Sources.

Request parameter reference

Parameter	Type	Required	Default	Description
`query`	string	Yes	—	Search query text. Max 5,000 characters. Supports search operators like `site:` and `filetype:`.
`location`	string	No	—	ISO 3166-1 alpha-2 country code for region-specific results (e.g., `"US"`, `"GB"`, `"JP"`).
`sources`	string[]	No	—	Sources to query: `web`, `news`, `scholar-articles`, `scholar-articles-enriched`, `scholar-author`, `ai`, `social`. Omitting this field searches all sources.
`site`	string	No	—	Restrict results to a domain (e.g., `"linkedin.com/company"`, `"github.com"`). Max 500 characters.
`start_date`	integer	No	—	Unix timestamp (seconds). Only results after this date.
`end_date`	integer	No	—	Unix timestamp (seconds). Only results before this date. Must be > `start_date`.
`human_mode`	boolean	No	`false`	Attempt a browser-like retrieval path when standard search access is blocked by bot protection.
`page`	integer	No	`1`	Number of result pages to aggregate into the response. Minimum: `1`.

Response fields reference

Field	Type	Description
`success`	boolean	Whether the search executed successfully.
`query`	string	The query as interpreted by the API (includes `site:` prefix if `site` was set).
`timestamp`	integer	Unix timestamp in milliseconds when the search was performed.
`results`	array	Search results. Shape varies by `source` — see Sources.
`metadata.total_results`	integer	Total number of results available across all pages (may exceed the number in the `results` array if you requested fewer pages).
`metadata.failed_pages`	array	Page numbers that failed to return results.
`metadata.empty_pages`	array	Page numbers that returned no results.

Timestamps: Search timestamp is in milliseconds. Fetch timestamp is in seconds. Divide Search timestamps by 1000 when comparing across endpoints.

Error handling

Search returns 400 for invalid requests and 401 for auth failures.

{
    "error": {
        "type": "invalid_request",
        "message": "query: This field is required.",
        "metadata": []
    }
}

{
    "error": {
        "type": "invalid_request",
        "message": "sources: {0: [ErrorDetail(string='\"invalid_source\" is not a valid choice.', code='invalid_choice')]}",
        "metadata": []
    }
}

{
    "message": "Invalid API key in request"
}

Common gotchas

Mistake	Fix
Omitting `sources` and expecting uniform results	Different sources return different fields. Specify `sources` explicitly for predictable parsing.
Using `site` with `scholar-author` or `ai` sources	`site` only applies to `web` and `news` sources. It has no effect on academic or deep research searches.
Expecting `snippet` in deep research mode results	Deep research mode returns `content` and `references` instead of `snippet` and `position`.
Expecting `position` in scholar-author results	Academic author results don’t have `position` — they have `name`, `affiliation`, `citations`, etc.
Using `start_date` >= `end_date`	`start_date` must be strictly less than `end_date`.

API reference summary

Detail	Value
Endpoint	`POST /web/search/live`
Auth	Bearer token + `x-api-version: 2025-11-01`
Pricing	`1 credit per query`
Request	`query` (required). Optional: `location`, `sources`, `site`, `start_date`, `end_date`, `human_mode`, `page`.
Response	Object: `{ success, query, timestamp, results[], metadata }`
Errors	`400` (bad request), `401` (bad auth)

See the full API reference for the complete OpenAPI schema.

Sources

Web Search supports seven source types. Each returns a different result shape — always specify sources explicitly when you need predictable parsing.

Source	Best use case	Fetchable `url`?	`site` effective?	Date filters effective?
`web`	General web search	Yes	Yes	Yes
`news`	News articles	Yes	Yes	Yes
`scholar-articles`	Academic papers	Yes	No	Yes
`scholar-articles-enriched`	Papers + author profiles	Yes	No	Yes
`scholar-author`	Researcher profiles	No	No	No
`ai`	AI-generated summaries	No	No	No
`social`	Social media mentions	Yes	No	No

Result shapes by source

The results[] array shape depends on the source field of each result. Use this reference when parsing multi-source responses.

web / news
scholar-articles
scholar-author
ai
social

Standard web and news results share the same shape.

Field	Type	Description
`source`	string	`"web"` or `"news"`.
`title`	string	Page title.
`url`	string	Page URL.
`snippet`	string	Text excerpt.
`position`	integer	Result position (1-based).

{
    "source": "web",
    "title": "Crustdata: Real-Time B2B Data Broker via API or Data Feed",
    "url": "https://crustdata.com/",
    "snippet": "Crustdata is a B2B data provider offering real-time company & people datasets.",
    "position": 1
}

Academic article results include citation data, author information, and optional PDF links.

Field	Type	Description
`source`	string	`"scholar-articles"` or `"scholar-articles-enriched"`.
`title`	string	Article title.
`url`	string	Link to the article.
`snippet`	string	Abstract excerpt.
`metadata`	string	Citation string: `"Author - Year - Publisher"`.
`pdf_url`	string?	Direct PDF link, if available.
`position`	integer	Result position (1-based).
`authors`	array	`[{ name, profile_url, profile_id }]`.
`citations`	integer	Total citation count.

{
    "source": "scholar-articles",
    "title": "Understanding deep learning",
    "url": "https://books.google.com/books?hl=en&lr=lang_en&id=rvyxEAAAQBAJ",
    "snippet": "...to this field understand the principles behind deep learning.",
    "metadata": "SJD Prince - 2023 - books.google.com",
    "pdf_url": null,
    "position": 1,
    "authors": [
        {
            "name": "SJD Prince",
            "profile_url": "https://scholar.google.com/citations?user=fjm67xYAAAAJ&hl=en&oi=sra",
            "profile_id": "fjm67xYAAAAJ"
        }
    ],
    "citations": 618
}

Use scholar-articles-enriched instead of scholar-articles to get richer author profile data. The result shape is the same, with more author fields populated.

Author profile results have a completely different shape — no snippet, position, or title. Instead, you get a full researcher profile.

Field	Type	Description
`source`	string	`"scholar-author"`.
`url`	string	Academic profile URL.
`name`	string	Author full name.
`affiliation`	string	Institutional affiliation.
`website`	string?	Personal or institutional website.
`interests`	array	`[{ title, link }]` — research interests.
`thumbnail`	string?	Profile photo URL.
`citations`	object	`{ all, since_2020 }` — total and recent counts.
`h_index`	object	`{ all, since_2020 }`.
`i10_index`	object	`{ all, since_2020 }`.
`articles`	array	Top publications: `[{ title, url, year, citations, authors, publication }]`.

{
    "source": "scholar-author",
    "url": "https://scholar.google.com/citations?user=NMS69lQAAAAJ&hl=en&oi=ao",
    "name": "Jeff Dean",
    "affiliation": "Google Chief Scientist, Google Research and Google DeepMind",
    "website": "http://research.google.com/people/jeff",
    "interests": [
        { "title": "Distributed systems", "link": "https://scholar.google.com/..." }
    ],
    "citations": { "all": 401624, "since_2020": 231008 },
    "h_index": { "all": 114, "since_2020": 78 },
    "i10_index": { "all": 319, "since_2020": 203 },
    "articles": [
        {
            "title": "MapReduce: simplified data processing on large clusters",
            "url": "https://scholar.google.com/...",
            "year": "2008",
            "citations": "37255",
            "authors": "J Dean, S Ghemawat",
            "publication": "Communications of the ACM 51 (1), 107-113, 2008"
        }
    ]
}

Deep research mode returns a single AI-generated overview with source references. No snippet, position, or standard search fields.

Field	Type	Description
`source`	string	`"ai"`.
`title`	string	Always `"AI Overview"`.
`content`	string	AI-generated overview text.
`references`	array	Source articles: `[{ title, url, snippet }]`.
`images`	array	Embedded images: `[{ url, alt, width, height }]`.

{
    "source": "ai",
    "title": "AI Overview",
    "content": "The primary difference between uv and pip is speed and scope...",
    "references": [
        {
            "title": "uv vs pip: Managing Python Packages and Dependencies",
            "url": "https://realpython.com/uv-vs-pip/",
            "snippet": "When it comes to Python package managers..."
        }
    ],
    "images": []
}

Social media results use the same shape as web/news results.

Field	Type	Description
`source`	string	`"social"`.
`title`	string	Post or page title.
`url`	string	Post URL.
`snippet`	string	Post excerpt.
`position`	integer	Result position (1-based).

Social search results may return empty for some queries depending on availability. Always check results.length before processing.

Result ordering and ranking

When querying a single source, position reflects the source’s natural ranking order. When querying multiple sources, results from different sources are interleaved and position may reflect a per-source rank rather than a global rank. metadata.total_results is the total count across all requested sources and pages.

Parsing multi-source responses

When you query multiple sources at once (or omit sources), the results[] array can contain items with different shapes. Always check the source field of each result to determine which fields are available:

for (const result of response.results) {
    switch (result.source) {
        case "web":
        case "news":
        case "social":
            // Standard: title, url, snippet, position
            console.log(result.title, result.url);
            break;
        case "scholar-articles":
        case "scholar-articles-enriched":
            // Academic: standard fields + authors, citations, pdf_url, metadata
            console.log(result.title, result.citations, result.authors);
            break;
        case "scholar-author":
            // Author profile: name, affiliation, h_index, articles[]
            console.log(result.name, result.affiliation, result.h_index);
            break;
        case "ai":
            // AI overview: content, references[]
            console.log(result.content, result.references);
            break;
    }
}

Field presence by source

Use this reference to determine which fields are present for each source type.

Naming note: The API uses metadata in two different contexts. The response-level metadata is an object with total_results, failed_pages, and empty_pages. The per-result metadata field (scholar-articles only) is a citation string like "Author - Year - Publisher". Always use the full path (response.metadata vs result.metadata) to avoid confusion.

Standard fields — present in web, news, social, and scholar-articles / scholar-articles-enriched:

Field	Sources with this field	Notes
`source`	All sources	Always present
`title`	`web`, `news`, `social`, `scholar-articles*`, `ai`	AI: always `"AI Overview"`
`url`	`web`, `news`, `social`, `scholar-articles*`, `scholar-author`	Academic author: profile link
`snippet`	`web`, `news`, `social`, `scholar-articles*`	Absent in `ai`, `scholar-author`
`position`	`web`, `news`, `social`, `scholar-articles*`	Absent in `ai`, `scholar-author`

Academic article fields — scholar-articles and scholar-articles-enriched only:

Field	Type	Notes
`metadata`	string	Citation string: `"Author - Year - Publisher"`
`pdf_url`	string?	Direct PDF download link — handle outside Web Fetch
`authors`	array	`[{ name, profile_url, profile_id }]`
`citations`	integer	Total citation count

Academic author fields — scholar-author only:

Field	Type	Notes
`name`	string	Author full name
`affiliation`	string	Institutional affiliation
`website`	string?	Personal or institutional website
`interests`	array	`[{ title, link }]`
`thumbnail`	string?	Profile photo URL
`citations`	object	`{ all, since_2020 }` — different type than scholar-articles
`h_index`	object	`{ all, since_2020 }`
`i10_index`	object	`{ all, since_2020 }`
`articles`	array	`[{ title, url, year, citations, authors, publication }]`

Deep research mode fields — ai only:

Field	Type	Notes
`content`	string	AI-generated overview text
`references`	array	`[{ title, url, snippet }]` — fetch these URLs
`images`	array	`[{ url, alt, width, height }]`

​Request parameter reference

​Response fields reference

​Error handling

​Common gotchas

​API reference summary

​Sources

​Result shapes by source

​Result ordering and ranking

​Parsing multi-source responses

​Field presence by source

Request parameter reference

Response fields reference

Error handling

Common gotchas

API reference summary

Sources

Result shapes by source

Result ordering and ranking

Parsing multi-source responses

Field presence by source