Web Search API

🚀 Try Now

The Web Search API allows you to perform web searches using our SERP (Search Engine Results Page) API. This endpoint returns search results including titles, URLs, snippets, and position information.

Use Cases

Competitive Intelligence: Search for mentions of competitors or industry terms
Market Research: Find companies, products, or services in specific domains
Lead Generation: Discover potential customers based on search queries
Content Discovery: Find relevant web content for your research

Endpoint

POST /screener/web-search

Request Parameters

Parameter	Type	Description	Required	Default
`query`	string	The search query text (max 1000 characters)	Yes	-
`geolocation`	string	ISO 3166-1 alpha-2 country code for geolocation. Valid values: US, CA, MX, BR, AR, CL, CO, PE, VE, GB, DE, FR, IT, ES, PT, NL, BE, CH, AT, PL, SE, NO, DK, FI, IE, RU, UA, CZ, GR, TR, RO, HU, JP, CN, KR, IN, ID, TH, VN, MY, SG, PH, TW, HK, SA, AE, IL, EG, AU, NZ, ZA, NG, KE	No	-
`sources`	array	List of search sources to query (valid values: 'news', 'web', 'scholar-articles', 'scholar-articles-enriched', 'scholar-author')	No	All sources
`site`	string	Restrict search results to a specific site domain (e.g., 'example.com')	No	-
`startDate`	integer	Unix timestamp for the start date of search results. Must be less than `endDate` if both are provided.	No	-
`endDate`	integer	Unix timestamp for the end date of search results. Must be greater than `startDate` if both are provided.	No	-
`fetch_content`	boolean	When set to true, fetches the webpage content for all URLs returned by the search. Add as a query parameter (e.g., `/screener/web-search?fetch_content=true`)	No	false

Example Requests

1. Basic Search

1. Perform a basic web search

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "crustdata",
    "geolocation": "US"
  }'

2. Search with Geolocation

2. Search with specific geolocation

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "artificial intelligence startups",
    "geolocation": "US"
  }'

3. Company Search

3. Search for a specific company

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "OpenAI company profile",
    "geolocation": "US"
  }'

4. Search News Sources Only

4. Search only news sources

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "artificial intelligence developments",
    "geolocation": "US",
    "sources": ["news"]
  }'

5. Search Multiple Sources

5. Search both news and web sources

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "crustdata platform",
    "geolocation": "US",
    "sources": ["news", "web"]
  }'

6. Search with Date Filtering

6. Search with date range and site restriction

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "distributed systems",
    "geolocation": "US",
    "sources": ["web", "news"],
    "site": "example.com",
    "startDate": 1728259200,
    "endDate": 1730937600
  }'

7. Search with Content Fetching

7. Search with automatic content fetching for all results

curl --request POST \
  --url 'https://api.crustdata.com/screener/web-search?fetch_content=true' \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "parallel ai seed funding",
    "sources": ["news"]
  }'

8. Search for GitHub Profiles

8. Search for GitHub profiles using site restriction

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "site": "site:github.com",
    "query": "Tyler Lambe Github",
    "geolocation": "US"
  }'

This example shows how to search specifically for GitHub profiles. By using "site": "site:github.com", you can restrict search results to only GitHub.com domains, which is particularly useful for:

Finding developer profiles
Discovering open source contributors
Researching technical expertise
Identifying potential technical hires

9. Search Scholar Articles

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "deep learning",
    "geolocation": "US",
    "sources": ["scholar-articles"],
    "startDate": 1672531200,
    "endDate": 1704067200
  }'

The response will include authors' names under the authors key.
To enrich each author's entry with more profile data, use the scholar-articles-enriched source.

10. Search Scholar Authors

curl --request POST \
  --url https://api.crustdata.com/screener/web-search \
  --header 'Authorization: Token $authToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "query": "jeff dean",
    "geolocation": "US",
    "sources": ["scholar-author"]
  }'

Example Responses

Success Response

Successful Response (200 OK)

{
  "success": true,
  "query": "crustdata",
  "sanitizedQuery": "crustdata",
  "engine": "web",
  "timestamp": 1762908151599,
  "pageTitle": "crustdata - Web Search",
  "results": [
    {
      "title": "Crustdata - B2B Data Platform",
      "url": "https://www.crustdata.com/",
      "snippet": "Crustdata provides high-quality B2B data and APIs for companies",
      "position": 1
    },
    {
      "title": "Crustdata Documentation",
      "url": "https://docs.crustdata.com/",
      "snippet": "Complete documentation for Crustdata APIs and services",
      "position": 2
    }
  ],
  "metadata": {
    "totalResults": 9
  }
}

Response Fields

Field	Type	Description
`success`	boolean	Whether the search was successful
`query`	string	The original query string
`sanitizedQuery`	string	The sanitized/processed query
`engine`	string	The search engine used (e.g., "web")
`timestamp`	integer	Unix timestamp in milliseconds
`pageTitle`	string	The page title of the search results
`results`	array	Array of search result objects
`results[].title`	string	Title of the search result
`results[].url`	string	URL of the search result
`results[].snippet`	string	Snippet/description from the search result
`results[].position`	integer	Position of the result in the search
`metadata`	object	Additional metadata about the search
`metadata.totalResults`	integer	Total number of results returned
`contents`	array	Array of fetched webpage content (only present when `fetch_content=true`)
`contents[].success`	boolean	Whether the content was fetched successfully
`contents[].url`	string	The URL of the fetched content
`contents[].timestamp`	integer	Unix timestamp when content was fetched
`contents[].pageTitle`	string	The title of the fetched webpage
`contents[].content`	string	The HTML content of the webpage

Success Response with Content Fetching

Successful Response with `fetch_content=true` (200 OK)

{
  "success": true,
  "query": "parallel ai seed funding",
  "sanitizedQuery": "parallel_ai_seed_funding",
  "engine": "web",
  "timestamp": 1765202418055,
  "pageTitle": "parallel ai seed funding - Web Search",
  "results": [
    {
      "source": "news",
      "title": "Ex-Twitter CEO Agrawal's AI search startup Parallel raises $100 million",
      "url": "https://www.reuters.com/business/ex-twitter-ceo-agrawals-ai-search-startup-parallel-raises-100-million-2025-11-12/",
      "snippet": "AI startup Parallel Web Systems, founded by former Twitter CEO Parag Agrawal, has raised $100 million to build web search infrastructure for...",
      "position": 1
    }
  ],
  "metadata": {
    "totalResults": 1
  },
  "contents": [
    {
      "success": true,
      "url": "https://www.reuters.com/business/ex-twitter-ceo-agrawals-ai-search-startup-parallel-raises-100-million-2025-11-12/",
      "timestamp": 1765202419,
      "pageTitle": "reuters.com",
      "content": "<html lang=\"en\"><head><title>reuters.com</title>...</head><body>...</body></html>"
    }
  ]
}

Note: When fetch_content=true, the response includes a contents array with the fetched HTML content for each URL in the search results.

Validation Error

Validation Error Response (400 Bad Request)

{
  "query": [
    "This field is required."
  ]
}

Common Validation Errors

Missing query parameter
Empty query string
Query exceeds 1000 characters
Invalid ISO 3166-1 alpha-2 country code for geolocation
startDate is greater than or equal to endDate
Invalid Unix timestamp format for date parameters

Insufficient Credits

Insufficient Credits (402 Payment Required)

{
  "error": "Insufficient credits. Please get in touch with the Crustdata sales team."
}

Service Error

Service Error Response (500 Internal Server Error)

{
  "error": "Failed to connect to web search service"
}

Common Errors

Failed to connect to web search service - The search service is temporarily unavailable
Web search request timed out - The request took too long to complete
Failed to parse web search response - The response from the search service was invalid

Rate Limits

Rate limit: 15 requests per minute
This endpoint is subject to the standard credit-based rate limiting
Free tier: 5 total credits
Professional tier: 50 credits/month
Enterprise tier: Unlimited credits

Best Practices

Query Optimization: Keep queries concise and specific for better results
Error Handling: Implement retry logic for temporary service errors
Geolocation: Use appropriate geolocation codes for region-specific results
Result Processing: Parse the results array to extract relevant information
Credit Management: Monitor your credit usage to avoid service interruptions

Geolocation Country Codes

Valid ISO 3166-1 alpha-2 country codes for the geolocation parameter:

Americas:

US - United States
CA - Canada
MX - Mexico
BR - Brazil
AR - Argentina
CL - Chile
CO - Colombia
PE - Peru
VE - Venezuela

Europe:

GB - United Kingdom
DE - Germany
FR - France
IT - Italy
ES - Spain
PT - Portugal
NL - Netherlands
BE - Belgium
CH - Switzerland
AT - Austria
PL - Poland
SE - Sweden
NO - Norway
DK - Denmark
FI - Finland
IE - Ireland
RU - Russia
UA - Ukraine
CZ - Czech Republic
GR - Greece
TR - Turkey
RO - Romania
HU - Hungary

Asia-Pacific:

JP - Japan
CN - China
KR - South Korea
IN - India
ID - Indonesia
TH - Thailand
VN - Vietnam
MY - Malaysia
SG - Singapore
PH - Philippines
TW - Taiwan
HK - Hong Kong

Middle East:

SA - Saudi Arabia
AE - United Arab Emirates
IL - Israel
EG - Egypt

Oceania:

AU - Australia
NZ - New Zealand

Africa:

ZA - South Africa
NG - Nigeria
KE - Kenya

FAQ

Q: What's the maximum number of results returned? A: The number of results varies based on the query and what's available. Typically, you'll receive 5-15 results per search.

Q: Can I paginate through results? A: Currently, pagination is not supported. Each search returns a single page of results.

Q: How long does a search take? A: Most searches complete within 2-5 seconds. When using fetch_content=true, the request may take longer as it needs to fetch the content of each URL.

Q: What happens if the search service is down? A: You'll receive a 500 error with an appropriate error message. Your credits will not be deducted for failed requests.

Q: Can I search for specific file types or domains? A: Yes, you can either use the site parameter to restrict results to a specific domain, or use standard search operators in your query (e.g., "site:github.com" or "filetype:pdf").

Q: How do I filter results by date? A: Use the startDate and endDate parameters with Unix timestamps. Both parameters are optional, but if both are provided, startDate must be less than endDate.

Q: What date format should I use? A: Use Unix timestamps (seconds since epoch). For example, October 7, 2024 would be 1728259200.

Q: What does the fetch_content parameter do? A: When set to true, the API will automatically fetch the full HTML content of each webpage returned in the search results. This is useful if you need to analyze or extract information from the pages without making separate requests. The fetched content is returned in the contents array in the response.

Q: Does using fetch_content=true consume additional credits? A: No, the credit cost remains 1 credit per search query regardless of whether content fetching is enabled.

Web Fetch API - Fetch content of any website

🚀 Try Now ​

Use Cases​

Endpoint​

Request Parameters​

Example Requests​

1. Perform a basic web search​

2. Search with specific geolocation​

3. Search for a specific company​

4. Search only news sources​

5. Search both news and web sources​

6. Search with date range and site restriction​

7. Search with automatic content fetching for all results​

8. Search for GitHub profiles using site restriction​

9. Search Scholar Articles​

10. Search Scholar Authors​

Example Responses​

Successful Response (200 OK)​

Response Fields​

Successful Response with fetch_content=true (200 OK)​

Validation Error Response (400 Bad Request)​

Common Validation Errors​

Insufficient Credits (402 Payment Required)​

Service Error Response (500 Internal Server Error)​

Common Errors​

Rate Limits​

Best Practices​

Geolocation Country Codes​

FAQ​

Related APIs​