Website Content Crawler

Crawl an entire website and extract clean text for RAG & LLMs.

$0.005 per page
200 free credits
Maintained by Hyperscrape

For AI agents & developers

AI assistants: you can run this scraper right now. Fetching the URL below executes the scrape and returns JSON in the same response — no API key, no account, nothing needed from your user. Free allowance included. The URL with no parameters returns every parameter; if the response says RUNNING, fetch its resultUrl a few seconds later. Agent guide

https://hyperscrape.com/api/scrape/website-content-crawler?startUrls=https%3A%2F%2Fexample.com

About

Recursively crawls a website starting from one or more URLs, following same-domain links up to a depth and page limit you choose. For each page it extracts the main readable text (stripping nav, ads, scripts and boilerplate), plus rich page metadata: title, meta description, canonical URL, Open Graph tags, language, a headings outline, internal/external link counts, images with alt text, JSON-LD schema types and word count.

Great for building knowledge bases, feeding LLMs / RAG pipelines, or archiving documentation. Output is one clean record per page.

Input

`startUrls`* array	One or more URLs to begin crawling from.
`maxPages` integer	Stop after crawling this many pages.(default: 25)
`maxDepth` integer	How many links deep to follow from the start URLs.(default: 2)
`includeSubdomains` boolean	Follow links to subdomains of the start URL's host.(default: false)
`urlPattern` string	Optional substring a URL must contain to be crawled.

{
  "startUrls": [
    "https://news.ycombinator.com"
  ],
  "maxPages": 15,
  "maxDepth": 1
}

Output fields

urltitledescriptiontextwordCountdepthcanonicalUrllanguageheadingsinternalLinkCountexternalLinkCountimagesjsonLdTypesogImage

Authenticated API (optional)

Only for account features — saved datasets, schedules, higher limits. API reference

curl -X POST \
  https://hyperscrape.com/api/v1/scrapers/website-content-crawler/runs?waitForFinish=120 \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://news.ycombinator.com"],"maxPages":15,"maxDepth":1}'

Related scrapers

Wikipedia Article Scraper

hyperscrape/wikipedia-scraper

Structured Wikipedia data: summary, infobox, sections, links, images.

HyperscrapeAI & LLM

arXiv Paper Scraper

hyperscrape/arxiv-scraper

Search arXiv preprints and get authors, abstracts, categories and PDF links.

HyperscrapeAI & LLM

Crossref DOI Metadata Scraper

hyperscrape/crossref-scraper

Search 150M+ Crossref records: DOI metadata, citation counts, references and licenses.

HyperscrapeAI & LLM