llms.txt Generator

Crawl a site's key pages and generate a ready-to-publish llms.txt file.

$0.005 per page
200 free credits
Maintained by Hyperscrape

For AI agents & developers

AI assistants: you can run this scraper right now. Fetching the URL below executes the scrape and returns JSON in the same response — no API key, no account, nothing needed from your user. Free allowance included. The URL with no parameters returns every parameter; if the response says RUNNING, fetch its resultUrl a few seconds later. Agent guide

https://hyperscrape.com/api/scrape/llms-txt-generator?siteUrl=https%3A%2F%2Fexample.com

About

Generates an llms.txt file — the emerging standard that tells LLMs what a website is about and which pages matter — for any site you point it at.

How it works: it fetches the homepage, discovers key pages from the site's own navigation plus a sample of sitemap.xml, prioritizes the classics (docs, pricing, about, blog, guides…), then visits each one to capture its title and a one-line summary from its meta description or first paragraph.

Output:

one type: "llms-txt" item containing the complete generated file: # Site Title, a > blockquote description, and ## Docs / ## Pricing / ## About / ## Blog / ## Optional sections of - [Page title](url): summary lines per the llmstxt.org convention
one type: "page" item per visited page with the url, title, summary and the section it was filed under

Drop the generated text at /llms.txt on your site, or use it as a compact, LLM-friendly map of any third-party site.

Input

`siteUrl`* string	The website to generate an llms.txt for (its homepage or root).
`maxPages` integer	How many key pages to visit for titles and summaries (besides the homepage).(default: 10)
`siteDescription` string	Optional one-liner used as the '>' description instead of the homepage meta description.

{
  "siteUrl": "https://hono.dev",
  "maxPages": 8
}

Output fields

typellmsTxtsiteTitlesiteDescriptionpageCountsectionsurltitlesummarysection

Authenticated API (optional)

Only for account features — saved datasets, schedules, higher limits. API reference

curl -X POST \
  https://hyperscrape.com/api/v1/scrapers/llms-txt-generator/runs?waitForFinish=120 \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"siteUrl":"https://hono.dev","maxPages":8}'

Related scrapers

Website Content Crawler

hyperscrape/website-content-crawler

Crawl an entire website and extract clean text for RAG & LLMs.

HyperscrapeAI & LLM

Wikipedia Article Scraper

hyperscrape/wikipedia-scraper

Structured Wikipedia data: summary, infobox, sections, links, images.

HyperscrapeAI & LLM

arXiv Paper Scraper

hyperscrape/arxiv-scraper

Search arXiv preprints and get authors, abstracts, categories and PDF links.

HyperscrapeAI & LLM