/crawl
Follow links across
a whole site
Point Pline at a website and get the text from every page back, clean and ready to use. You set how deep it goes and how many pages.
Point Pline at a website and get the text from every page back, clean and ready to use. You set how deep it goes and how many pages.
POST/crawl
Copy
curl -X POST "https://apix.pline.ai/v1/crawl" \
-H "Content-Type: application/json" \
-H "x-api-key: YOUR_API_KEY" \
-d '{"url": "https://example.com", "limit": 500, "output": ["markdown"]}'
POST/crawl
Copy
curl -X POST "https://apix.pline.ai/v1/crawl" \
-H "Content-Type: application/json" \
-H "x-api-key: YOUR_API_KEY" \
-d '{"url": "https://example.com", "limit": 500, "output": ["markdown"]}'
Crawl at scale
Async by design, delivered your way
Long crawls run in the background. Start a job, get an ID straight away, and pick up results as pages finish.
Crawl at scale
Async by design, delivered your way
Long crawls run in the background. Start a job, get an ID straight away, and pick up results as pages finish.
Crawl at scale
Async by design, delivered your way
Long crawls run in the background. Start a job, get an ID straight away, and pick up results as pages finish.
You're in control
Crawl only what you need
Choose which paths to include or skip, how deep to go and how many pages to crawl. Every page can use the same /scrape settings, including JSON extraction.
You're in control
Crawl only what you need
Choose which paths to include or skip, how deep to go and how many pages to crawl. Every page can use the same /scrape settings, including JSON extraction.
You're in control
Crawl only what you need
Choose which paths to include or skip, how deep to go and how many pages to crawl. Every page can use the same /scrape settings, including JSON extraction.
Nothing gets lost
Failed pages don't stop the crawl
If a page fails, the crawl keeps going. You get a count of failed pages and a list with each URL and the reason, so you can retry only those.
Nothing gets lost
Failed pages don't stop the crawl
If a page fails, the crawl keeps going. You get a count of failed pages and a list with each URL and the reason, so you can retry only those.
Nothing gets lost
Failed pages don't stop the crawl
If a page fails, the crawl keeps going. You get a count of failed pages and a list with each URL and the reason, so you can retry only those.
Use cases
What teams build with /crawl
What teams build with /crawl
Turn docs into a knowledge base
Crawl a docs site into clean Markdown and load it into your AI assistant or search.
Turn docs into a knowledge base
Crawl a docs site into clean Markdown and load it into your AI assistant or search.
Collect data across a whole section
Pull structured JSON from every product, listing or article page in one job.
Collect data across a whole section
Pull structured JSON from every product, listing or article page in one job.
Keep a copy of a site
Save full sections of a site on a schedule to track what changed.
Keep a copy of a site
Save full sections of a site on a schedule to track what changed.
Works with
One API key. Four endpoints.
One API key. Four endpoints.
FAQs
FAQs
FAQs
Get answers to the most asked questions
Get answers to the most asked questions
Get answers to the most asked questions
Can I control how deep it goes?
Can I stop a crawl?
What happens if some pages fail?
How is /crawl billed?
Is /crawl synchronous or async?
How do I get the results?
Ready to power your AI products with clean, real-time data?
Use cases
What teams build with /crawl
Turn docs into a knowledge base
Crawl a docs site into clean Markdown and load it into your AI assistant or search.
Collect data across a whole section
Pull structured JSON from every product, listing or article page in one job.
Keep a copy of a site
Save full sections of a site on a schedule to track what changed.