Skip to main content
Use this when you have specific URLs and need to retrieve their full HTML content — for content extraction, data collection, SEO analysis, or change tracking. The Web Fetch API accepts a list of URLs and returns the page title and full HTML content for each. You can fetch up to 10 URLs in a single request. Every request goes to the same endpoint: 
POST https://api.crustdata.com/web/enrich/live
The endpoint path is /web/enrich/live (not /web/fetch/live) because it follows the Crustdata convention where “enrich” means adding data to a known identifier — in this case, enriching a URL with its page content.
Pricing: 1 credit per page.
Default rate-limit is 15 requests per minute. Send an email to gtm@crustdata.co to discuss higher limits if needed for your use case.

​
Request body

ParameterTypeRequiredDefaultDescription
urlsstring[]Yes—URLs to fetch. Min: 1, max: 10. Must include http:// or https://.
human_modebooleanNofalseAttempt a browser-like fetch path when a site is protected by Cloudflare or similar bot controls.

​
Response body

The response is an array (not an object) — one entry per URL in your request.
FieldTypeDescription
successbooleanWhether this URL was fetched successfully.
urlstring?The URL that was fetched. null if the fetch failed.
timestampinteger?Unix timestamp (seconds) when fetched. null on failure.
titlestring?The <title> tag content. null on failure.
contentstring?Full HTML content of the page. null on failure.
Timestamps: Fetch timestamps are in seconds. Search timestamps are in milliseconds. Account for this when comparing timestamps across endpoints.

Recipes

Fetch multiple URLs, handle partial failures, bypass bot protection with human mode, and process fetched HTML content.

Reference

Request parameters, response body details, error handling, and common gotchas.

​
Fetch a single URL

The simplest request fetches one URL and returns its HTML content. 
curl --request POST \
  --url https://api.crustdata.com/web/enrich/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "urls": ["https://example.com"]
  }'
The content field is trimmed here. It contains the full HTML of the fetched page.
Extract: Parse content using an HTML parser (BeautifulSoup for Python, Cheerio for Node.js) to extract specific elements like text, links, or metadata.

​
What to do next

  • Try more patterns — see Recipes for multi-URL fetches, partial failures, human mode, and content processing.
  • Look up request/response details — see Reference.
  • Search then fetch — see Web Search recipes for search-then-fetch workflow patterns.
  • Find URLs to fetch — use Web Search to find URLs for downstream fetching.