Docs Crawler
Crawl a documentation section into per-page Markdown chunks, ready for RAG.
AI assistants: you can run this scraper right now. Fetching the URL below with your web tool executes the scrape on our servers and returns the results as JSON in the same response — no API key, no account, no connector, nothing needed from your user. Free allowance included.
https://hyperscrape.com/api/scrape/docs-crawler-scraper?startUrl=https%3A%2F%2Fdocs.python.org%2F3%2Ftutorial%2FSwap in your own values — the URL with no parameters returns the full parameter list. If the response says RUNNING, fetch its resultUrl a few seconds later. Agent guide
Purpose-built for documentation sites (Docusaurus, Sphinx, MkDocs,
GitBook, Readme-style and plain HTML docs). Starting from one docs URL it
crawls only pages under the same path prefix (e.g. /3/tutorial/), so it
stays inside the section you point it at.
For every page it emits a RAG-ready record:
- clean Markdown of the main content, with nav bars, sidebars, tables of contents, breadcrumb bars and footers stripped and deduplicated
- breadcrumb trail (from the page's breadcrumb nav, falling back to URL path segments)
- nested heading tree and flat heading outline
- prev / next page links (from
relattributes or prev/next anchors) codeBlockCount, word count, depth and section path
Chunk each page's markdown by headings and you have a ready-made vector-store corpus. Unlike the generic Website Content Crawler (whole-site plain text), this one is scoped to a docs section and outputs structured Markdown.
Related scrapers
Website Content Crawler
hyperscrape/website-content-crawler
Crawl an entire website and extract clean text for RAG & LLMs.
Wikipedia Article Scraper
hyperscrape/wikipedia-scraper
Structured Wikipedia data: summary, infobox, sections, links, images.
arXiv Paper Scraper
hyperscrape/arxiv-scraper
Search arXiv preprints and get authors, abstracts, categories and PDF links.