> ## Documentation Index
> Fetch the complete documentation index at: https://docs.crustdata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Fetch reference

> Reference for Web Fetch: request parameters, response body details, error handling, common gotchas, and API summary.

Reference material for [Web Fetch](/web-docs/fetch/introduction): request parameters,
response body details, error handling, common gotchas, and the API summary.

For walk-through examples, see [Web Fetch](/web-docs/fetch/introduction) and
[Examples](/web-docs/fetch/examples).

<Snippet file="web-auth-headers.mdx" />

***

## Request parameter reference

| Parameter    | Type      | Required | Default | Description                                                                                       |
| ------------ | --------- | -------- | ------- | ------------------------------------------------------------------------------------------------- |
| `urls`       | string\[] | Yes      | —       | URLs to fetch. Min: 1, max: 10. Must include `http://` or `https://`.                             |
| `human_mode` | boolean   | No       | `false` | Attempt a browser-like fetch path when a site is protected by Cloudflare or similar bot controls. |

## Response fields reference

The response is a top-level array. Each item corresponds to one URL in your
request.

| Field       | Type     | Description                                                                                                     |
| ----------- | -------- | --------------------------------------------------------------------------------------------------------------- |
| `success`   | boolean? | Whether this URL was fetched successfully. May be `null` when the result entry contains only error information. |
| `url`       | string?  | The URL that was fetched. `null` if the fetch failed.                                                           |
| `timestamp` | integer? | Unix timestamp (**seconds**) when fetched. `null` on failure.                                                   |
| `title`     | string?  | The `<title>` tag content. `null` on failure.                                                                   |
| `content`   | string?  | Full HTML content of the page. `null` on failure.                                                               |

<Note>
  **Timestamps:** Fetch timestamps are in **seconds**. Search timestamps are
  in **milliseconds**. Account for this when comparing timestamps across
  endpoints.
</Note>

<Note>
  **Two kinds of failure, two places to check:**

  * **Request-level errors** (`400`, `401`) — the entire request failed. You get an error object, not an array. Caused by missing fields, empty arrays, or bad auth.
  * **Per-URL failures** within a `200` — individual entries with `success: false` and `null` fields. Caused by unreachable URLs, timeouts, or bot protection.

  Always check the HTTP status first, then check `success` for each entry
  in the array.
</Note>

***

## Error handling

Fetch returns request-level errors for invalid input or auth failures.
These are separate from per-URL `success: false` entries within a `200`
response.

<CodeGroup>
  ```json 400 — missing urls theme={"theme":"vitesse-black"}
  {
      "error": {
          "type": "invalid_request",
          "message": "urls: This field is required.",
          "metadata": []
      }
  }
  ```

  ```json 400 — empty urls array theme={"theme":"vitesse-black"}
  {
      "error": {
          "type": "invalid_request",
          "message": "urls: This list may not be empty.",
          "metadata": []
      }
  }
  ```

  ```json 401 — bad API key theme={"theme":"vitesse-black"}
  {
      "message": "Invalid API key in request"
  }
  ```
</CodeGroup>

<Snippet file="web-error-responses.mdx" />

***

## Common gotchas

| Mistake                                  | Fix                                                                                                             |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Omitting `http://` or `https://` in URLs | All URLs must include the protocol prefix.                                                                      |
| Sending more than 10 URLs                | The API accepts a maximum of 10 URLs per request. Batch larger lists.                                           |
| Assuming response order matches request  | Match results by the `url` field, not by array index.                                                           |
| Treating a `200` as all-success          | A `200` can contain failed entries. Check `success` for each item.                                              |
| Sending an empty `urls` array            | Returns `400`: `"urls: This list may not be empty."`.                                                           |
| Expecting JavaScript-rendered content    | **Current platform behavior:** The API fetches server-side HTML. JavaScript-heavy SPAs may return minimal HTML. |
| Comparing Search and Fetch timestamps    | Search uses milliseconds, Fetch uses seconds. Divide Search by 1000 to compare.                                 |

***

## API reference summary

| Detail       | Value                                                                                              |
| ------------ | -------------------------------------------------------------------------------------------------- |
| **Endpoint** | `POST /web/enrich/live`                                                                            |
| **Auth**     | Bearer token + `x-api-version: 2025-11-01`                                                         |
| **Pricing**  | `1 credit per page`                                                                                |
| **Request**  | `urls` (1–10 URLs, required). Optional: `human_mode`.                                              |
| **Response** | Top-level array: `[{ success, url, timestamp, title, content }]`                                   |
| **Errors**   | `400` (bad request), `401` (bad auth), `500` (server error). Per-URL failures appear in the array. |

See the [full API reference](/openapi-specs/2025-11-01/introduction) for
the complete OpenAPI schema.