Crawl a site

Follow links from a start URL, scrape every page, and keep the cost under control with limits and path filters.

Written By Carmine Cella

Last updated About 3 hours ago

POST /crawl fetches a start URL, follows the links it finds breadth-first, and scrapes every page with the unlocker options you pass. It returns a job id at once and grows its own set of pages while it runs.

Start a crawl

curl https://scraping-api.datafuel.ai/api/v1/crawl \  --request POST \  --header "Content-Type: application/json" \  --header "X-API-Key: df_key_your_key_here" \  --header "Idempotency-Key: docs-crawl-2026-09-18" \  --data '{    "proxy_type": "Basic",    "attributes": {      "url": "https://example.com/docs",      "max_pages": 100,      "max_depth": 3,      "include_paths": ["^/docs/"],      "exclude_paths": ["\\.pdf$", "\\?page="],      "result_format": "markdown",      "main_content_only": true    }  }'

{ "job_id": "job_…" }

Limits and filters

Attribute	Default	Notes
`max_pages`	100	Hard stop, cap 10 000. Pages are billed as they are queued, so this bounds the cost.
`max_depth`	3	Start URL is depth 0. Cap 10.
`include_paths`	none	RE2 patterns on `path?query`. When set, only matching URLs are followed. Up to 20.
`exclude_paths`	none	RE2 patterns on `path?query`. Matching URLs are never followed. Exclude wins. Up to 20.
`include_subdomains`	false	Also follow links on subdomains.
`allow_backward_links`	false	When false, only URLs under the start URL's path are followed, so a crawl of `/docs` never wanders into `/blog`.
`concurrency`	5	Pages of this crawl in flight at once, also bounded by your account limit.

All unlocker options apply per page: result_format, extract_selector, main_content_only, headers. With js_rendering: true every page is rendered in a browser and links are taken from the rendered DOM, so client-side rendered sites crawl correctly. result_use_ai is rejected on crawl. Sitemaps, robots.txt and webhooks are not used yet.

Poll and read results

GET /crawl/{job_id}                      status, page counters, stop_reasonGET /crawl/{job_id}/results?limit=100    pages + next_cursor

Poll every two seconds while status is pending or processing. Results can be read while the crawl runs; pages not yet fetched appear as status stubs. Pass next_cursor back as cursor until the response has no next_cursor.

stop_reason says why the crawl ended: max_pages, max_depth_exhausted, insufficient_credits or cancelled. The same id also works with GET /job/{id}.

Cost

Each page is a regular unlocker task charged when it is queued: 1 credit on Basic, 10 on Premium, 5 and 20 with js_rendering. Failed pages, blocked ones included, are refunded, and the links on a blocked page are never followed, so a blocked start URL is a free zero-page crawl.

total_cost on the crawl is the sum charged at queue time and does not subtract refunds. Sum credits_used over the results for the net figure.

Keep it cheap

Run Map the URLs of a site first. One credit tells you how many pages exist.
Set include_paths to the section you need and max_pages to what you can afford.
Leave js_rendering off unless the map came back empty.
If you can list the URLs, use a job instead. See Scrape a list of URLs.