Firecrawl Crawl

Usage-based pricing
ae_firecrawl_crawl

Crawl multiple related pages starting at a website URL, returning the crawl status and collected content. Parameters • Required url: starting URL string. All other top-level fields are optional. • Scope: includePaths/excludePaths (arrays of URL path patterns), limit (page limit), maxDiscoveryDepth (link-discovery depth), crawlEntireDomain (whole domain rather than starting path), allowSubdomains and allowExternalLinks. • Discovery and pacing: sitemap = skip/include/only; ignoreQueryParameters, deduplicateSimilarURLs, delay between requests, and maxConcurrency. prompt accepts natural-language crawl instructions. • scrapeOptions applies to each page. formats accepts markdown, html, rawHtml, screenshot, links, summary, changeTracking, branding, json, query and audio. Content controls include onlyMainContent, includeTags/excludeTags, removeBase64Images and redactPII. jsonOptions accepts prompt/schema; queryOptions requires prompt (up to 10,000 characters) and mode (directQuote/freeform; the schema declares freeform as its default). parsers supports pdf; pdfOptions.maxPages is an integer from 1 to 10,000. screenshotOptions accepts quality, fullPage and viewport; viewport requires width and height. Other options include waitFor, mobile, location.country/languages, proxy (basic/stealth/enhanced/auto), maxAge, storeInCache, zeroDataRetention, skipTlsVerification and profile (name is required when profile is provided; saveChanges is optional). lockdown is an exposed boolean whose meaning is not defined by the selected source. Do not assume unspecified defaults, time units or numeric ranges. Undeclared top-level or scrapeOptions fields are not accepted. Async behavior and results Submitting starts a background task and returns a task ID. After the task completes, use ae_get_result to retrieve the saved result; this does not start another crawl. The crawl result may contain id, url, status, data, next, total, completed, expiresAt, success, error and warning. data contains collected documents, with no fixed document structure promised here. next may provide a URL for another page of results, so do not assume all results fit in one response. A terminal result is not necessarily a successful crawl. Crawls can be slow and responses can exceed token limits. High page/depth limits can produce oversized output; low limits can miss pages. A /* starting-URL wildcard is not recommended. For one page use scraping; for tighter control consider map plus scrape. Safe mode is read-only: webhooks and interactive actions are unavailable.

Productivity
Usage-based pricing

Pricing

The charge per call is (max(1.2 × processing units, 0)) AE credits.

Parameters

Optional parameters use the platform defaults when omitted.

ParameterTypeRequiredDescriptionDefault
urlstringYesNo parameter description.—
delaynumberNoDelay between crawl requests.—
limitnumberNoNo parameter description.—
promptstringNoNatural-language instructions for the crawl.—
sitemapenum (skip | include | only)NoNo parameter description.—
excludePathsarrayNoURL path patterns to exclude from the crawl.—
includePathsarrayNoURL path patterns to include in the crawl.—
scrapeOptionsobjectNoOptions applied when scraping each crawled page.—
scrapeOptions.proxyenum (basic | stealth | enhanced | auto)NoNo parameter description.—
scrapeOptions.maxAgenumberNoMaximum acceptable age of cached results.—
scrapeOptions.mobilebooleanNoWhether to use a mobile browser viewport.—
scrapeOptions.formatsarrayNoOutput formats to generate for each scraped page.—
scrapeOptions.parsersarrayNoParsers to apply when processing scraped content.—
scrapeOptions.profileobjectNoBrowser profile settings for the scrape operation.—
scrapeOptions.profile.namestringYesNo parameter description.—
scrapeOptions.profile.saveChangesbooleanNoWhether to save changes made to the browser profile.—
scrapeOptions.waitFornumberNoTime to wait before scraping a rendered page.—
scrapeOptions.locationobjectNoGeographic settings for page retrieval.—
scrapeOptions.location.countrystringNoNo parameter description.—
scrapeOptions.location.languagesarrayNoLanguages to use for page retrieval.—
scrapeOptions.lockdownbooleanNoNo parameter description.—
scrapeOptions.redactPIIbooleanNoWhether to redact personally identifiable information from scraped content.—
scrapeOptions.pdfOptionsobjectNoOptions for PDF processing.—
scrapeOptions.pdfOptions.maxPagesintegerNoMaximum number of PDF pages to process.—
scrapeOptions.excludeTagsarrayNoHTML tags to exclude from extracted content.—
scrapeOptions.includeTagsarrayNoHTML tags to include in extracted content.—
scrapeOptions.jsonOptionsobjectNoOptions for structured JSON extraction.—
scrapeOptions.jsonOptions.promptstringNoInstructions for extracting structured JSON data.—
scrapeOptions.jsonOptions.schemaobjectNoJSON schema defining the structured data to extract.—
scrapeOptions.queryOptionsobjectNoOptions for query-based extraction.—
scrapeOptions.queryOptions.modeenum (directQuote | freeform)YesNo parameter description.freeform
scrapeOptions.queryOptions.promptstringYesInstructions for the query-based extraction.—
scrapeOptions.storeInCachebooleanNoWhether to store scraped results in the cache.—
scrapeOptions.onlyMainContentbooleanNoWhether to extract only the main page content.—
scrapeOptions.screenshotOptionsobjectNoOptions for page screenshots.—
scrapeOptions.screenshotOptions.qualitynumberNoImage quality for the screenshot.—
scrapeOptions.screenshotOptions.fullPagebooleanNoWhether to capture the full page in the screenshot.—
scrapeOptions.screenshotOptions.viewportobjectNoViewport settings used to render screenshots.—
scrapeOptions.screenshotOptions.viewport.widthnumberYesNo parameter description.—
scrapeOptions.screenshotOptions.viewport.heightnumberYesNo parameter description.—
scrapeOptions.zeroDataRetentionbooleanNoWhether to disable retention of scraped data.—
scrapeOptions.removeBase64ImagesbooleanNoWhether to remove base64-encoded images from extracted content.—
scrapeOptions.skipTlsVerificationbooleanNoWhether to skip TLS certificate verification.—
maxConcurrencynumberNoMaximum number of concurrent crawl requests.—
allowSubdomainsbooleanNoWhether to follow links to subdomains.—
crawlEntireDomainbooleanNoWhether to crawl the entire domain instead of limiting the crawl to the starting path.—
maxDiscoveryDepthnumberNoMaximum link-discovery depth from the starting URL.—
allowExternalLinksbooleanNoWhether to follow links to external domains.—
ignoreQueryParametersbooleanNoWhether to ignore URL query parameters when identifying URLs.—
deduplicateSimilarURLsbooleanNoWhether to deduplicate similar URLs during the crawl.—

Invocation example

Copy the request and explore its response example.

POST/agent-api/v1/tool/executeX-Api-Key

Submit first, then query with the returned task_id until the task completes.

Request coderequest.sh
curl --request POST \
  --url 'https://agentearth.ai/agent-api/v1/tool/execute?tool_name=ae_firecrawl_crawl' \
  --header "X-Api-Key: $AGENTEARTH_API_KEY" \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "params": {
    "includePaths": [
      "/fiction/89455/chapter/.*"
    ],
    "limit": 400,
    "maxConcurrency": 10,
    "maxDiscoveryDepth": 3,
    "scrapeOptions": {
      "formats": [
        "markdown"
      ],
      "onlyMainContent": true,
      "proxy": "stealth"
    },
    "sitemap": "include",
    "url": "https://qyuedu.net/chapter-directory/89455"
  }
}'
Response exampleresponse.json
JSONPartial content
{
  "description": "task accepted; poll the result via query_tool with this task_id",
  "query_tool": "ae_get_result",
  "status": "created",
  "task_id": "60d321b8-d20c-4573-9793-339547b80597"
}

This example shows part of the response. Copied content is valid JSON.