Skip to main content
The Web APIs give you programmatic access to web search and webpage content fetching. Search across web, news, Google Scholar, Google AI mode, and social media sources — or fetch the raw HTML of any public URL.

Web Search

Search the web across 7 source types: web, news, Scholar articles, Scholar authors, AI mode, social, and enriched Scholar.

Web Fetch

Fetch the HTML content of up to 10 public URLs in one request for content extraction and analysis.

​
At a glance

SearchFetch
EndpointPOST /web/search/livePOST /web/enrich/live
Required fieldsqueryurls
Optional fieldsgeolocation, sources, site, startDate, endDate, numPages, solveCloudflaresolveCloudflare
Response shapeObject with success, query, timestamp, results[], metadataArray of { success, url, timestamp, pageTitle, content }
PaginationnumPages (request multiple pages)—
Max items per request~10 results per page (platform behavior)10 URLs
Timestamp unitMillisecondsSeconds
Error codes400, 401400, 401

​
Before you start

You need:
  • A Crustdata API key
  • A terminal with curl (or any HTTP client)
  • The required header: x-api-version: 2025-11-01
Convention used in these docs: Information labeled “OpenAPI contract” reflects the formal API specification. Information labeled “Current platform behavior” (such as rate limits and credit costs) describes observed behavior that may change. See the API reference for the formal OpenAPI spec.

​
Quickstart: search the web

The fastest way to get started is a simple web search. This single request returns search results with titles, URLs, snippets, and positions. 
curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "crustdata",
    "sources": ["web"],
    "geolocation": "US"
  }'
Response trimmed for clarity.
The response contains:
  • success — whether the search executed successfully.
  • results[] — an array of search results, each with source, title, url, snippet, and position.
  • metadata.totalResults — the total number of results available (may exceed the displayed count if you didn’t request all pages).

​
End-to-end: Search → Company Enrich

The most common workflow chains a web search with a downstream Crustdata API call. Here is a complete 3-step example:
1

Search for a company's website

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{"query": "ADAMSBROWN, LLC website", "sources": ["web"]}'
Extract: results[0].url → "https://www.adamsbrowncpa.com/"
2

Normalize the URL to a domain

const domain = new URL("https://www.adamsbrowncpa.com/")
  .hostname.replace("www.", ""); // "adamsbrowncpa.com"
3

Enrich the company via Company API

curl --request POST \
  --url https://api.crustdata.com/company/enrich \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{"domains": ["adamsbrowncpa.com"]}'
This returns the full company profile: name, headcount, funding, industry, and more.

​
Which API should you start with?

If you want to…Start with
Find companies, news, academic papers, or social mentionsSearch
Get the HTML content of specific URLs for processingFetch
Do both — search then scrape the top resultsSearch first, then Fetch the URLs

​
Common workflows

1

Competitive intelligence

Search for a competitor’s name across news and web sources, then Fetch the top result URLs to extract full article content.
2

Find company domain → Enrich

Search with the company name + “website” to discover the domain (first result URL). Then pass it to the Company Enrich API for the full company profile.
3

Find LinkedIn → Identify company

Search with the company name and site: "linkedin.com/company" to get the LinkedIn URL. Then pass it to the Company Identify API.
4

Find person → Enrich

Search with a person’s name and site: "linkedin.com/in" to find their LinkedIn profile URL. Then pass it to the Person Enrich API.
5

Academic research

Search with sources: ["scholar-articles"] to find papers with citation data, or sources: ["scholar-author"] to get full author profiles with h-index metrics.
6

AI-powered answers

Search with sources: ["ai"] to get a Google AI-generated overview with source references.
7

Content monitoring

Fetch the same set of URLs on a schedule and diff the content field to detect changes.

​
Choosing a search source

The Web Search API supports seven source types. Each returns a different result shape — always specify sources explicitly for predictable parsing.
SourceWhat it returnsSafe to pass url to Fetch?Typical downstream action
webStandard web resultsYesFetch page content, discover domains/profiles
newsNews articlesYesFetch full article, monitor press coverage
scholar-articlesAcademic articlesYesDownload PDF via pdf_url, analyze citations
scholar-articles-enrichedArticles with richer author dataYesSame as above, plus follow author profiles
scholar-authorResearcher profilesNoRead citation metrics and publications directly from the result
aiAI-generated overviewNoUse content directly; fetch references[].url for source articles
socialSocial media postsYesMonitor social mentions
For Fetch workflows: Only pass URLs from sources marked “Yes” in the fetchable url column directly to Web Fetch. For AI results, fetch the references[].url values instead. For scholar-author results, the url is a profile page, not a content page.
OpenAPI contract: The site, startDate, and endDate parameters are defined in the spec. Current platform behavior: These parameters primarily affect web and news results. scholar-author and ai searches may not filter by these parameters.

​
Cross-API workflow map

The Web APIs are often the first step in a larger pipeline. Here’s how they connect to other Crustdata APIs:
Starting pointWeb Search query patternExtract from resultPass to
Company name → company profile"ACME Inc website", sources: ["web"]results[0].url → domainCompany Enrich (domains)
Company name → identify company"ACME Inc", site: "linkedin.com/company"results[0].url → LinkedInCompany Identify (professional_network_profile_urls)
Person name → person profile"Jane Smith Google", site: "linkedin.com/in"results[0].url → LinkedInPerson Enrich (professional_network_profile_urls)
Any search → full article contentAny search queryresults[].urlWeb Fetch (urls)
AI overview → source articles"topic", sources: ["ai"]results[0].references[].urlWeb Fetch (urls)

​
Error handling

​
Next steps

  • Web Search — search the web, news, scholars, AI, and social media.
  • Web Fetch — fetch the HTML content of public URLs.
  • Web API Examples — ready-to-copy patterns for common use cases.