Crawl at scale
Async by design, delivered your way
Long crawls run in the background. Start a job, get an ID straight away, and pick up results as pages finish.
Check progress any time with the job ID
Get a webhook the second each page finishes
Cancel a running crawl mid-flight


