Firecrawl Crawl
Crawl multiple related pages starting at a website URL, returning the crawl status and collected content. Parameters • Required url: starting URL string. All other top-level fields are optional. • Scope: includePaths/excludePaths (arrays of URL path patterns), limit (page limit), maxDiscoveryDepth (link-discovery depth), crawlEntireDomain (whole domain rather than starting path), allowSubdomains and allowExternalLinks. • Discovery and pacing: sitemap = skip/include/only; ignoreQueryParameters, deduplicateSimilarURLs, delay between requests, and maxConcurrency. prompt accepts natural-language crawl instructions. • scrapeOptions applies to each page. formats accepts markdown, html, rawHtml, screenshot, links, summary, changeTracking, branding, json, query and audio. Content controls include onlyMainContent, includeTags/excludeTags, removeBase64Images and redactPII. jsonOptions accepts prompt/schema; queryOptions requires prompt (up to 10,000 characters) and mode (directQuote/freeform; the schema declares freeform as its default). parsers supports pdf; pdfOptions.maxPages is an integer from 1 to 10,000. screenshotOptions accepts quality, fullPage and viewport; viewport requires width and height. Other options include waitFor, mobile, location.country/languages, proxy (basic/stealth/enhanced/auto), maxAge, storeInCache, zeroDataRetention, skipTlsVerification and profile (name is required when profile is provided; saveChanges is optional). lockdown is an exposed boolean whose meaning is not defined by the selected source. Do not assume unspecified defaults, time units or numeric ranges. Undeclared top-level or scrapeOptions fields are not accepted. Async behavior and results Submitting starts a background task and returns a task ID. After the task completes, use ae_get_result to retrieve the saved result; this does not start another crawl. The crawl result may contain id, url, status, data, next, total, completed, expiresAt, success, error and warning. data contains collected documents, with no fixed document structure promised here. next may provide a URL for another page of results, so do not assume all results fit in one response. A terminal result is not necessarily a successful crawl. Crawls can be slow and responses can exceed token limits. High page/depth limits can produce oversized output; low limits can miss pages. A /* starting-URL wildcard is not recommended. For one page use scraping; for tighter control consider map plus scrape. Safe mode is read-only: webhooks and interactive actions are unavailable.
Pricing
The charge per call is (max(1.2 × processing units, 0)) AE credits.
Parameters
Optional parameters use the platform defaults when omitted.
| Parameter | Type | Required | Description | Default |
|---|---|---|---|---|
url | string | Yes | No parameter description. | — |
delay | number | No | Delay between crawl requests. | — |
limit | number | No | No parameter description. | — |
prompt | string | No | Natural-language instructions for the crawl. | — |
sitemap | enum (skip | include | only) | No | No parameter description. | — |
excludePaths | array | No | URL path patterns to exclude from the crawl. | — |
includePaths | array | No | URL path patterns to include in the crawl. | — |
scrapeOptions | object | No | Options applied when scraping each crawled page. | — |
scrapeOptions.proxy | enum (basic | stealth | enhanced | auto) | No | No parameter description. | — |
scrapeOptions.maxAge | number | No | Maximum acceptable age of cached results. | — |
scrapeOptions.mobile | boolean | No | Whether to use a mobile browser viewport. | — |
scrapeOptions.formats | array | No | Output formats to generate for each scraped page. | — |
scrapeOptions.parsers | array | No | Parsers to apply when processing scraped content. | — |
scrapeOptions.profile | object | No | Browser profile settings for the scrape operation. | — |
scrapeOptions.profile.name | string | Yes | No parameter description. | — |
scrapeOptions.profile.saveChanges | boolean | No | Whether to save changes made to the browser profile. | — |
scrapeOptions.waitFor | number | No | Time to wait before scraping a rendered page. | — |
scrapeOptions.location | object | No | Geographic settings for page retrieval. | — |
scrapeOptions.location.country | string | No | No parameter description. | — |
scrapeOptions.location.languages | array | No | Languages to use for page retrieval. | — |
scrapeOptions.lockdown | boolean | No | No parameter description. | — |
scrapeOptions.redactPII | boolean | No | Whether to redact personally identifiable information from scraped content. | — |
scrapeOptions.pdfOptions | object | No | Options for PDF processing. | — |
scrapeOptions.pdfOptions.maxPages | integer | No | Maximum number of PDF pages to process. | — |
scrapeOptions.excludeTags | array | No | HTML tags to exclude from extracted content. | — |
scrapeOptions.includeTags | array | No | HTML tags to include in extracted content. | — |
scrapeOptions.jsonOptions | object | No | Options for structured JSON extraction. | — |
scrapeOptions.jsonOptions.prompt | string | No | Instructions for extracting structured JSON data. | — |
scrapeOptions.jsonOptions.schema | object | No | JSON schema defining the structured data to extract. | — |
scrapeOptions.queryOptions | object | No | Options for query-based extraction. | — |
scrapeOptions.queryOptions.mode | enum (directQuote | freeform) | Yes | No parameter description. | freeform |
scrapeOptions.queryOptions.prompt | string | Yes | Instructions for the query-based extraction. | — |
scrapeOptions.storeInCache | boolean | No | Whether to store scraped results in the cache. | — |
scrapeOptions.onlyMainContent | boolean | No | Whether to extract only the main page content. | — |
scrapeOptions.screenshotOptions | object | No | Options for page screenshots. | — |
scrapeOptions.screenshotOptions.quality | number | No | Image quality for the screenshot. | — |
scrapeOptions.screenshotOptions.fullPage | boolean | No | Whether to capture the full page in the screenshot. | — |
scrapeOptions.screenshotOptions.viewport | object | No | Viewport settings used to render screenshots. | — |
scrapeOptions.screenshotOptions.viewport.width | number | Yes | No parameter description. | — |
scrapeOptions.screenshotOptions.viewport.height | number | Yes | No parameter description. | — |
scrapeOptions.zeroDataRetention | boolean | No | Whether to disable retention of scraped data. | — |
scrapeOptions.removeBase64Images | boolean | No | Whether to remove base64-encoded images from extracted content. | — |
scrapeOptions.skipTlsVerification | boolean | No | Whether to skip TLS certificate verification. | — |
maxConcurrency | number | No | Maximum number of concurrent crawl requests. | — |
allowSubdomains | boolean | No | Whether to follow links to subdomains. | — |
crawlEntireDomain | boolean | No | Whether to crawl the entire domain instead of limiting the crawl to the starting path. | — |
maxDiscoveryDepth | number | No | Maximum link-discovery depth from the starting URL. | — |
allowExternalLinks | boolean | No | Whether to follow links to external domains. | — |
ignoreQueryParameters | boolean | No | Whether to ignore URL query parameters when identifying URLs. | — |
deduplicateSimilarURLs | boolean | No | Whether to deduplicate similar URLs during the crawl. | — |
Invocation example
Copy the request and explore its response example.
/agent-api/v1/tool/executeX-Api-KeySubmit first, then query with the returned task_id until the task completes.
curl --request POST \
--url 'https://agentearth.ai/agent-api/v1/tool/execute?tool_name=ae_firecrawl_crawl' \
--header "X-Api-Key: $AGENTEARTH_API_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
"params": {
"includePaths": [
"/fiction/89455/chapter/.*"
],
"limit": 400,
"maxConcurrency": 10,
"maxDiscoveryDepth": 3,
"scrapeOptions": {
"formats": [
"markdown"
],
"onlyMainContent": true,
"proxy": "stealth"
},
"sitemap": "include",
"url": "https://qyuedu.net/chapter-directory/89455"
}
}'{
"description": "task accepted; poll the result via query_tool with this task_id",
"query_tool": "ae_get_result",
"status": "created",
"task_id": "60d321b8-d20c-4573-9793-339547b80597"
}This example shows part of the response. Copied content is valid JSON.

