Scrape a list of URLs

When you already know which pages you want, POST /job queues all of them at once. It returns an id immediately; you poll for progress and fetch results when it finishes. Each URL costs the same as a single task.

Queue the job

curl https://scraping-api.datafuel.ai/api/v1/job \  --request POST \  --header "Content-Type: application/json" \  --header "X-API-Key: df_key_your_key_here" \  --header "Idempotency-Key: catalog-import-2026-09-18" \  --data '{    "type": "unlocker",    "multithreaded": true,    "proxy_type": "Basic",    "attributes": {      "urls": [        "https://example.com/p/1",        "https://example.com/p/2",        "https://example.com/p/3"      ],      "result_format": "markdown",      "main_content_only": true    }  }'

{ "id": "job_…" }

The attributes are the same as for a single task, with urls (plural) in place of url. For llm_scraping jobs use prompts instead. multithreaded: true runs the URLs in parallel up to your account's concurrency limit; leave it off to fetch them one by one.

Poll until it finishes

GET /job/{id}

The response carries status (pending, processing, completed, failed) and counters for finished and failed tasks. Poll every two seconds while the status is pending or processing. There are no webhooks yet.

Collect the results

GET /job/{id}/results

Returns every task of the job in one response. Each entry has the same shape as a single task result: the page content under result.data and the metadata envelope next to it. A failed URL shows up with status: failed, credits_used: 0 and an error string; it does not fail the whole job.

Billing

Each URL is charged when it is queued and refunded if it fails. Blocked pages count as failed. GET /users/@me/balance shows the net effect.

Job or crawl?

A job fetches exactly the URLs you give it. A crawl discovers URLs by following links. If you can list the URLs up front, a job is cheaper and its cost is known before you start. If you only have a start page, see Crawl a site, or run Map the URLs of a site first and feed its output into a job.