Bright Data Scrape Batch

Usage-based pricing
ae_brightdata_scrape_batch

Fetch 1–5 web pages per batch with Bright Data. Provide the page URLs in the required urls array and optionally choose one format for the batch: markdown (default), text for plain text without links or headings, html for raw HTML, or ld_json for available schema.org data such as price, rating, or author. Results are returned as a JSON array in the same order as the input URLs. Each successful item contains url, title, format, and content; each failed item contains url and error. If a page has no data in the chosen format, its result falls back to Markdown with a note. The tool does not render JavaScript, and pages taking longer than 35 seconds time out.

Productivity
Usage-based pricing

Pricing

The charge per call is (max(1.8 × distinct successfully returned webpages, 0)) AE credits. All applicable conditions are combined to calculate the total.

Charges are based on distinct webpages returned successfully with non-empty content; duplicate URLs are counted once.

Parameters

Optional parameters use the platform defaults when omitted.

ParameterTypeRequiredDescriptionDefault
urlsarrayYes要批量抓取的网页 URL 数组,包含 1–5 个 URL;结果按输入 URL 的顺序返回。—
formatenum (markdown | text | html | ld_json)No整批网页内容的返回格式。可选 markdown(默认,Markdown)、text(不含链接和标题格式的纯文本)、html(原始 HTML)或 ld_json(schema.org 结构化数据,如价格、评分、作者)。若某个页面没有所选格式的数据,则该页面回退返回 Markdown,并附带说明。markdown

Invocation example

Copy the request and explore its response example.

POST/agent-api/v1/tool/executeX-Api-Key
Request coderequest.sh
curl --request POST \
  --url 'https://agentearth.ai/agent-api/v1/tool/execute?tool_name=ae_brightdata_scrape_batch' \
  --header "X-Api-Key: $AGENTEARTH_API_KEY" \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "params": {
    "format": "text",
    "urls": [
      "https://zhuanlan.zhihu.com/p/27673095",
      "https://zhuanlan.zhihu.com/p/47326038",
      "https://www.sohu.com/a/923925326_120332207"
    ]
  }
}'
Response exampleresponse.json
JSONPartial content
"[{\"error\":\"Page timed out: https://zhuanlan.zhihu.com/p/27673095 did not load within 35 s. The page is likely blocked or very slow. Details: Crawl aborted on job cancel\",\"url\":\"https://zhuanlan.zhihu.com/p/27673095\"}]"

This example shows part of the response. Copied content is valid JSON.

Guides using this tool

See how this tool fits into real workflows