Bright Data Run Scraper

From 1.8 credits/call
ae_brightdata_run_scraper

Run a Bright Data scraper to collect records for URLs or queries. Provide dataset_id from search_scrapers, a method copied exactly from the scraper's collection_methods, and a nonempty input array with one object per URL or query. Check get_scraper_details for that dataset_id and method first, and build each input object from sample_input or input_fields. collect_by_url returns one record per input; discover_by_* methods return up to limit_per_input records per input. The optional limit_per_input applies only to discovery methods, accepts integers from 1 to 1000, and defaults to 10. Every returned record is billed. The call waits up to 45 seconds and returns collected records if complete, or snapshot_id with status "running" if processing continues. Use get_scraper_progress to check the run, then get_scraper_results when ready. Records with an error field explain failed inputs; other record fields depend on the selected scraper and method.

Productivity
From 1.8 credits/call

Pricing

From 1.8 credits/call
ConditionCharge
Each call1.8 AE credits per record, based on actual returned records.

Starting price applies to successful calls with at least 1 returned records. Returned records that fail because of incorrect input are still billed. collect_by_url returns one record per input; for discover_by_*, the discovery limit per input affects the number of records returned, and charges are based on the actual returned records.

Parameters

Optional parameters use the platform defaults when omitted.

ParameterTypeRequiredDescriptionDefault
inputarrayYesList of input objects shaped like sample_input from get_scraper_details, with one object per URL or query to process.—
methodstringYesOne of the scraper collection_methods, exactly as listed (some names are misspelled on purpose)—
dataset_idstringYesdataset_id from search_scrapers—
limit_per_inputintegerNodiscover_by_* only10

Invocation example

Copy the request and explore its response example.

POST/agent-api/v1/tool/executeX-Api-Key

Submit first, then query with the returned task_id until the task completes.

Request coderequest.sh
curl --request POST \
  --url 'https://agentearth.ai/agent-api/v1/tool/execute?tool_name=ae_brightdata_run_scraper' \
  --header "X-Api-Key: $AGENTEARTH_API_KEY" \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "params": {
    "dataset_id": "gd_le8e811kzy4ggddlq",
    "input": [
      {
        "max_reviews": 25,
        "reviews_to_not_include": [],
        "url": "https://www.amazon.com/dp/B0B2P9YMFG",
        "variation_specific": true
      }
    ],
    "method": "collect_by_url"
  }
}'
Response exampleresponse.json
JSONPartial content
{
  "description": "task accepted; poll the result via query_tool with this task_id",
  "query_tool": "ae_get_result",
  "status": "created",
  "task_id": "0a4e941f-df47-4e50-8447-4533e13b8bd3"
}

This example shows part of the response. Copied content is valid JSON.

Guides using this tool

See how this tool fits into real workflows