Which endpoint do I use?

DataFuel has four write endpoints. They share one request envelope (type, proxy settings, attributes) and one result format, so switching between them is cheap. Pick by what you have and what you need.

Synchronous endpoints hold the connection and return the result in the response. Asynchronous ones return an id at once; you poll for status and then fetch results.

Rules of thumb

• Start with a task on the request engine (js_rendering off). It is the cheapest and fastest option and keeps the exact URL you asked for.
• Turn js_rendering on only when you have to. Signs you need it: the result is empty, redirected is true and you wanted the original filtered URL, or /map came back with no_links_on_page. Browser pricing applies.
• Discover before you crawl. /map costs one credit and tells you how big a section is. Then /crawl with include_paths and a max_pages you can afford.
• Use a job when you already know the URLs. It is cheaper and more predictable than a crawl.
• Never loop /task over a list you got from /map. That is a job.
• Send an Idempotency-Key on every write. A retry with the same key attaches to the running task instead of charging twice. See Retry safely with idempotency keys.

The request envelope

Every write looks the same:

{  "type": "unlocker",  "proxy_type": "Basic",  "proxy_country": "US",  "attributes": { "url": "https://example.com/p/1", "result_format": "markdown" }}

• type picks the engine: unlocker for pages, llm_scraping for AI engines. Map and crawl have their own routes and do not take a type.
• proxy_type is Basic or Premium; proxy_country is an ISO country code. Premium routes through residential IPs and costs ten times more.
• attributes is engine-specific: url and result options for unlocker, prompt and engine for LLM scraping.

Each of the four endpoints has its own guide: Scrape a single page, Scrape a list of URLs, Map the URLs of a site, Crawl a site.

What can be switched off

Task types and LLM engines can be disabled at runtime by DataFuel operators, for example when an upstream AI engine is down. GET /api/v1/capabilities (no key needed) lists what is on and why anything is off. A call to a disabled module answers 503 with code MODULE_UNAVAILABLE or ENGINE_UNAVAILABLE and charges nothing. Check capabilities once before a large batch or an llm_scraping call.