Map the URLs of a site

Get a deduplicated list of a site's URLs for one credit, before you decide what to scrape.

Written By Carmine Cella

Last updated About 3 hours ago

POST /map reads a site's sitemaps and the links on the start page and returns one deduplicated list of same-site URLs. Nothing is scraped and the price is flat, so it is the cheapest way to see how big a section is before you spend credits on a crawl or a job.

Request

curl https://scraping-api.datafuel.ai/api/v1/map \  --request POST \  --header "Content-Type: application/json" \  --header "X-API-Key: df_key_your_key_here" \  --data '{    "attributes": {      "url": "https://example.com/docs",      "search": "guide",      "limit": 500    }  }'

The call is synchronous and returns the list in the response.

Attribute	Effect
`url`	Start page. Sitemaps are looked up from the site root.
`search`	Keep only URLs whose path contains this text.
`limit`	Maximum number of URLs returned.
`sitemap`	Sitemap URL to use instead of the discovered ones.
`sitemap_only`	Skip the start page, use sitemaps only.
`ignore_sitemap`	Skip sitemaps, use links on the start page only.
`include_subdomains`	Also return URLs on subdomains of the start host.

proxy_type and proxy_country work as on any other request.

Price

One credit per call on Basic, ten on Premium, regardless of how many links come back.

When the list is empty

A map that returns no_links_on_page usually hit a page that renders its navigation client-side. Retry the same request with js_rendering: true, which fetches the start page in a browser at browser pricing, or point sitemap at a known sitemap URL.

What to do with the list

Feed it into POST /job to scrape exactly those pages. See Scrape a list of URLs.
Use it to choose include_paths and a realistic max_pages for a crawl. See Crawl a site.