# Hyperscrape — full agent reference

> Web scraping platform, AI-first. Everything below is callable with **no account and no API key** (free daily allowance per IP). When the allowance runs out the response includes a sign-up link — a free account has 200 free credits and issues an API key to continue.

- Concise overview: https://hyperscrape.com/llms.txt
- OpenAPI spec: https://hyperscrape.com/api/openapi.json
- Human docs: https://hyperscrape.com/docs
- Sign up: https://hyperscrape.com/sign-up

## 1. Keyless GET tools (simplest — works from any URL-fetch tool)

One GET request, JSON response, no auth. Directory with examples: https://hyperscrape.com/api/tools

With an API key, append `&token=<api key>` to any tool URL for account-billed calls with no daily cap. When the keyless daily allowance runs out, the 429 response includes `claimUrl` (send the user there to create a free account — 200 free credits) and `claimStatusUrl` (poll with GET; returns `{status:"ready", apiKey}` exactly once when they finish) — then continue in the same conversation with `&token=`.

### Google Maps Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/google-maps-scraper?q=<value>`
- Example: `https://hyperscrape.com/api/tools/google-maps?q=coffee+shops+in+Austin+TX`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Scrape Google Maps for free. Enter a search like “dentists in Austin” and get business names, ratings, addresses, phones and websites in seconds. No signup, no code.

### Email Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/email-scraper?url=<value>`
- Example: `https://hyperscrape.com/api/tools/email?url=https://example.com`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Extract email addresses, phone numbers and social links from any website for free. Paste a URL, get contacts. No signup or code required.

### Google Search Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/google-search-scraper?q=<value>`
- Example: `https://hyperscrape.com/api/tools/google-search?q=best+crm+for+small+business`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Scrape Google search results for free. Enter a keyword and get every organic result — position, title, URL and snippet — ready to export. No code needed.

### Amazon Product Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/amazon-scraper?asin=<value>`
- Example: `https://hyperscrape.com/api/tools/amazon?asin=B09B8V1LZ3`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Scrape any Amazon product for free: price, rating, review count, images, brand and availability. Paste a product URL or ASIN — no code, no signup.

### Amazon Reviews Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/amazon-reviews?asin=<value>`
- Example: `https://hyperscrape.com/api/tools/amazon-reviews?asin=B09B8V1LZ3`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Scrape Amazon customer reviews for free: review text, star ratings, titles, dates and verified-purchase flags. Paste a product URL or ASIN — no code, no signup.

### Instagram Profile Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/instagram-scraper?username=<value>`
- Example: `https://hyperscrape.com/api/tools/instagram?username=nasa`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Scrape any public Instagram profile for free: followers, following, posts count, bio and profile picture. Just enter a username — no login, no code.

### YouTube Video Scraper

- Endpoint: `GET https://hyperscrape.com/api/tools/youtube-scraper?url=<value>`
- Example: `https://hyperscrape.com/api/tools/youtube?url=https://www.youtube.com/watch?v=aqz-KE-bpKQ`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Extract data from any YouTube video for free: exact views, likes, channel, duration, publish date, tags and description. Paste a link — no code.

### WHOIS Lookup

- Endpoint: `GET https://hyperscrape.com/api/tools/whois-lookup?domain=<value>`
- Example: `https://hyperscrape.com/api/tools/whois?domain=example.com`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Free WHOIS/RDAP lookup for any domain: registrar, creation and expiry dates, nameservers and status. Bulk-friendly, structured, no signup.

### URL to Markdown Converter

- Endpoint: `GET https://hyperscrape.com/api/tools/url-to-markdown?url=<value>`
- Example: `https://hyperscrape.com/api/tools/markdown?url=https://en.wikipedia.org/wiki/Web_scraping`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Convert any webpage to clean Markdown for free. Perfect for LLM prompts, RAG pipelines and note-taking. Paste a URL, get Markdown instantly.

### Meta Tag Checker

- Endpoint: `GET https://hyperscrape.com/api/tools/meta-tag-checker?url=<value>`
- Example: `https://hyperscrape.com/api/tools/meta-tags?url=https://example.com`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Check any page's meta tags for free: title, description, canonical, robots, Open Graph and Twitter cards — with validation flags. No signup.

### Sitemap URL Extractor

- Endpoint: `GET https://hyperscrape.com/api/tools/sitemap-extractor?url=<value>`
- Example: `https://hyperscrape.com/api/tools/sitemap?url=https://example.com/sitemap.xml`
- Returns: JSON `{ tool, input, itemCount, items: [...] }`
- Extract all URLs from any sitemap.xml for free — with lastmod dates and section breakdowns. Paste a sitemap link, download the URL list as CSV.

## 2. Get anything — every scraper (and any page) via keyless GET

```
GET https://hyperscrape.com/api/scrape?url=https://example.com
GET https://hyperscrape.com/api/scrape?url=https://example.com&fields=title: h1|price: .price
GET https://hyperscrape.com/api/scrape/<slug>?<param>=<value>
```

The first form returns ANY page as clean markdown — no scraper needed. The second runs a custom extraction you define with CSS selectors (`name: selector`, separate fields with `|`), so you can scrape sites we have no named scraper for. The third runs any scraper listed below; array params repeat or separate with `|`, or pass `?input=<url-encoded JSON>`. `GET https://hyperscrape.com/api/scrape` (no params) returns the machine-readable directory with each scraper's parameters. Keyless calls use a free trial per agent/IP — reuse the returned `agentId`; on exhaustion the response carries `claimUrl`/`claimStatusUrl` (see section 1) to continue with `&token=`.

## 3. Agent API — every scraper, keyless POST

```
POST https://hyperscrape.com/api/agent/scrape
Content-Type: application/json

{ "scraper": "<slug>", "input": { ... }, "agentId": "<reuse across calls>" }
```

GET the same URL to list scrapers programmatically. Responses include `agentId` — reuse it. When the free trial is spent, the response carries `claimUrl` (give it to the user to create an account) and `claimStatusUrl` (poll it; once the user finishes it returns an API key exactly once, then continue against the authenticated API).

## 4. MCP server

```
https://hyperscrape.com/api/mcp
```

Streamable-HTTP MCP endpoint; every scraper is exposed as an MCP tool. Keyless calls use the free trial; authenticate with `Authorization: Bearer <api key>` for full access. Send `X-Hyperscrape-Agent: <agentId>` to keep one identity across keyless calls.

## 5. Authenticated REST API (after sign-up)

```
POST https://hyperscrape.com/api/v1/scrapers/<slug>/runs?waitForFinish=120
Authorization: Bearer hs_live_...
```

Full reference: https://hyperscrape.com/docs/api-reference

## All scrapers

### Website Content Crawler (`website-content-crawler`)

Crawl an entire website and extract clean text for RAG & LLMs.

Input fields:
  - `startUrls` (array, required): One or more URLs to begin crawling from.
  - `maxPages` (integer): Stop after crawling this many pages.
  - `maxDepth` (integer): How many links deep to follow from the start URLs.
  - `includeSubdomains` (boolean): Follow links to subdomains of the start URL's host.
  - `urlPattern` (string): Optional substring a URL must contain to be crawled.

### Universal Web Scraper (`web-scraper`)

Extract structured data from any page using CSS selectors.

Input fields:
  - `urls` (array, required): Pages to scrape.
  - `fields` (array, required): One per line as `name: selector`. Use `@attr` for attributes and `[]name:` for lists.
  - `includeHtml` (boolean): Add the page's full HTML to each record.
  - `includePageInfo` (boolean): Add rich page metadata (title, meta/OG tags, headings, link/image stats, JSON-LD types, word count) to each record.

### News & Article Extractor (`article-extractor`)

Extract clean article text, author, date, tags and images from any URL.

Input fields:
  - `urls` (array, required): News articles or blog posts to extract.
  - `includeHtml` (boolean): Also return the article body's inner HTML (for re-rendering).

### Universal Product Scraper (`ecommerce-product-scraper`)

Extract full product data from most online stores via structured data.

Input fields:
  - `urls` (array, required): Product pages to scrape.
  - `useUnlocker` (boolean): Route through the Web Unlocker for protected stores (metered).

### Product Price Monitor (`price-monitor`)

Track prices, stock and full product data across product pages — ideal on a schedule.

Input fields:
  - `productUrls` (array, required): Product pages to monitor.
  - `useUnlocker` (boolean): Route requests through the Web Unlocker for sites that block bots (metered).

### Amazon Product Scraper (`amazon-product-scraper`)

Extract Amazon product title, price, rating, images, variants, BSR and specs.

Input fields:
  - `products` (array, required): Amazon product URLs or bare 10-character ASINs.
  - `domain` (select): Which Amazon marketplace to use for bare ASINs.

### Amazon Search Scraper (`amazon-search-scraper`)

Scrape Amazon search results — ASIN, title, price, rating, sponsored flag and rank.

Input fields:
  - `keywords` (array, required): Search terms to run on Amazon.
  - `domain` (select): Marketplace.
  - `maxPages` (integer): Search result pages to scrape per keyword.

### Amazon Reviews Scraper (`amazon-reviews-scraper`)

Extract customer reviews, ratings and verified-purchase flags from Amazon.

Input fields:
  - `products` (array, required): Amazon product URLs or 10-character ASINs.
  - `domain` (select): Marketplace to use.
  - `maxReviews` (integer): How many reviews to collect per product.

### Google Search Results (SERP) (`google-search-scraper`)

Scrape organic Google results — title, URL, snippet, sitelinks, PAA and related searches.

Input fields:
  - `queries` (array, required): One search query per line.
  - `resultsPerQuery` (integer): How many organic results to collect per query (max 100).
  - `countryCode` (string): Two-letter country code for localized results (gl parameter).
  - `language` (string): Interface language (hl parameter).
  - `includeExtras` (boolean): Also emit "People also ask" questions and related searches as items (type: peopleAlsoAsk / relatedSearch).

### Google Maps Places Scraper (`google-maps-scraper`)

Scrape local businesses from Google Maps — name, rating, address, phone, hours, coordinates.

Input fields:
  - `queries` (array, required): Place searches, e.g. "dentists in Chicago".
  - `maxPerQuery` (integer): How many places to collect per query.

### Google News Scraper (`google-news-scraper`)

Track news for any query, topic section or location via Google News.

Input fields:
  - `queries` (array): Search terms to track (supports Google News operators like when:7d, site:).
  - `topics` (array): Headline sections to pull: WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SCIENCE, SPORTS, HEALTH.
  - `geos` (array): Cities/regions for local headline feeds (e.g. New York).
  - `language` (string): Two-letter language code (hl).
  - `country` (string): Two-letter country code (gl).
  - `maxPerFeed` (integer): How many articles to collect per query/topic/location.

### Google Play App Scraper (`google-play-app-scraper`)

Extract app rating, installs, developer and reviews from Google Play.

Input fields:
  - `apps` (array, required): Google Play package ids or full store URLs.
  - `language` (string): Two-letter language code (hl).
  - `useUnlocker` (boolean): Route through the Web Unlocker if Play blocks requests.

### Apple App Store Scraper (`app-store-scraper`)

Get iOS app details — rating, price, developer, category, size.

Input fields:
  - `mode` (select): Look up specific apps by ID, or search by keyword.
  - `queries` (array, required): Numeric app IDs (lookup mode) or search terms (search mode).
  - `country` (string): Two-letter storefront code.
  - `limitPerSearch` (integer): How many results to return per keyword (search mode).

### Instagram Profile Scraper (`instagram-profile-scraper`)

Get follower counts, bio, profile details and recent posts for Instagram accounts.

Input fields:
  - `usernames` (array, required): Instagram usernames or profile URLs.
  - `includeRecentPosts` (boolean): Include up to 12 recent posts (id, caption, likes, comments, timestamp) when available.

### TikTok Profile Scraper (`tiktok-profile-scraper`)

Get TikTok follower counts, likes, bio, verification and recent video stats.

Input fields:
  - `usernames` (array, required): TikTok usernames (without @) or profile URLs.
  - `includeRecentVideos` (boolean): Include recent videos with stats when the profile page embeds them.

### Twitter / X Profile Scraper (`twitter-profile-scraper`)

Extract X (Twitter) profile bio, location, website, join date, followers and tweet counts.

Input fields:
  - `handles` (array, required): X/Twitter handles (without @) or profile URLs.

### LinkedIn Company Scraper (`linkedin-company-scraper`)

Extract LinkedIn company details — industry, size, HQ, founded, specialties, followers.

Input fields:
  - `companies` (array, required): LinkedIn company slugs or full /company/ URLs.

### Indeed Jobs Scraper (`indeed-jobs-scraper`)

Scrape Indeed job listings — title, company, rating, salary, remote flag and more.

Input fields:
  - `queries` (array, required): Job searches (title or keywords).
  - `location` (string): City, state, or ZIP to search within.
  - `domain` (string): Indeed country domain.
  - `maxPerQuery` (integer): How many jobs to collect per query.

### Glassdoor Company Scraper (`glassdoor-company-scraper`)

Extract Glassdoor company profile — rating, reviews, size, industry, CEO approval.

Input fields:
  - `companyUrls` (array, required): Full Glassdoor /Overview/ or /Reviews/ company URLs.

### Yelp Business Scraper (`yelp-business-scraper`)

Extract Yelp business details — rating, reviews, categories, hours, photos.

Input fields:
  - `businessUrls` (array, required): Full Yelp biz URLs (e.g. https://www.yelp.com/biz/...).

### Trustpilot Reviews Scraper (`trustpilot-reviews-scraper`)

Scrape company reviews, ratings and reviewer details from Trustpilot.

Input fields:
  - `companies` (array, required): Company domains (e.g. apify.com) or full Trustpilot review URLs.
  - `maxReviews` (integer): How many reviews to collect per company.

### Zillow Property Scraper (`zillow-property-scraper`)

Extract Zillow listing details — price, Zestimate, beds/baths, history, photos.

Input fields:
  - `propertyUrls` (array, required): Full Zillow /homedetails/ URLs.

### Booking.com Hotel Scraper (`booking-hotel-scraper`)

Extract hotel details — rating breakdown, amenities, coordinates — from Booking.com pages.

Input fields:
  - `hotelUrls` (array, required): Full Booking.com /hotel/ URLs.
  - `useUnlocker` (boolean): Route through the Web Unlocker if Booking blocks requests.

### On-Page SEO Analyzer (`seo-page-analyzer`)

Audit any page's title, meta, headings, links and structured data.

Input fields:
  - `urls` (array, required): Pages to audit.

### Sitemap URL Extractor (`sitemap-extractor`)

Pull every URL from a site's XML sitemap (including nested indexes).

Input fields:
  - `url` (string, required): A sitemap.xml URL, or a domain to auto-discover it from robots.txt.
  - `maxUrls` (integer): Stop after collecting this many URLs.

### Broken Link Checker (`broken-link-checker`)

Crawl a site, check every link and report status, redirects and dead ends.

Input fields:
  - `startUrl` (string, required): The page to start crawling from.
  - `maxPages` (integer): How many internal pages to scan for links.
  - `checkExternal` (boolean): Also verify links pointing to other domains.
  - `onlyBroken` (boolean): Skip healthy links in the output (summary is always included).

### Contact Details Scraper (`contact-info-extractor`)

Find emails, phones, addresses, socials and contact forms on any website.

Input fields:
  - `websites` (array, required): Domains or URLs to extract contact details from.
  - `crawlContactPages` (boolean): Also visit /contact, /about, /impressum, /team and similar pages.
  - `maxPagesPerSite` (integer): How many pages to check per website.

### Website Tech Stack Detector (`tech-stack-detector`)

Identify 150+ frameworks, CMS, analytics, CDNs and tools behind any site.

Input fields:
  - `urls` (array, required): Websites to fingerprint.

### RSS & Atom Feed Reader (`rss-feed-reader`)

Turn RSS 2.0, Atom and RDF feeds into rich, structured JSON items.

Input fields:
  - `feedUrls` (array, required): RSS, Atom or RDF feed URLs.
  - `maxItemsPerFeed` (integer): Limit entries returned from each feed.

### Hacker News Scraper (`hacker-news-scraper`)

Collect top, new, best, Ask & Show HN stories with full metadata.

Input fields:
  - `feed` (select): Which Hacker News feed to pull.
  - `limit` (integer): How many stories to fetch.
  - `includeComments` (boolean): Fetch a small tree of top comments for each story (slower; adds one request per comment).
  - `maxCommentsPerStory` (integer): Cap on comments fetched per story (top-level + replies).

### Reddit Subreddit Scraper (`reddit-scraper`)

Scrape posts from any subreddit — title, score, comments and more.

Input fields:
  - `subreddits` (array, required): Subreddit names (without r/).
  - `sort` (select): How to sort posts.
  - `timeframe` (select): Only applies when sort is Top.
  - `limit` (integer): How many posts to collect from each subreddit.

### Stack Overflow Questions Scraper (`stackoverflow-scraper`)

Collect Stack Overflow questions with full bodies, answers and scores.

Input fields:
  - `tags` (array, required): Stack Overflow tags to filter by (e.g. python, javascript).
  - `sort` (select): Question ordering.
  - `limit` (integer): How many questions to collect.
  - `includeAnswers` (boolean): Fetch top answers for each question (one extra batched request per page).
  - `answersPerQuestion` (integer): Max answers to attach to each question (top-voted; accepted always kept).

### Wikipedia Article Scraper (`wikipedia-scraper`)

Structured Wikipedia data: summary, infobox, sections, links, images.

Input fields:
  - `titles` (array, required): Wikipedia article titles to fetch.
  - `language` (string): Two-letter Wikipedia language code.
  - `fullExtract` (boolean): Fetch the entire article text, not just the intro.
  - `includeDetails` (boolean): Fetch the article HTML to extract the infobox and section outline (one extra request per article).

### GitHub Repository Scraper (`github-repo-scraper`)

Stars, forks, languages, topics, releases and contributors for any repo.

Input fields:
  - `repos` (array, required): Repositories as owner/name (full GitHub URLs also accepted).
  - `includeLanguages` (boolean): Fetch per-language byte counts (one extra request each).
  - `includeContributors` (boolean): Count contributors, exact up to 100 (one extra request each).

### YouTube Video Metadata (`youtube-video-metadata`)

Get title, channel, views, likes and description for YouTube videos.

Input fields:
  - `videos` (array, required): YouTube video URLs or 11-character video IDs.

### arXiv Paper Scraper (`arxiv-scraper`)

Search arXiv preprints and get authors, abstracts, categories and PDF links.

Input fields:
  - `query` (string, required): Free-text search. Supports arXiv field prefixes like ti:"..." (title) or au:... (author).
  - `category` (string): Optional arXiv category to restrict to, e.g. cs.AI, cs.CL, stat.ML, math.CO.
  - `sortBy` (select): How to order results.
  - `maxItems` (integer): How many papers to fetch.

### Bitcoin Address Scraper (`bitcoin-address-scraper`)

On-chain stats for any Bitcoin address — balance, totals, USD value, recent txs.

Input fields:
  - `addresses` (array, required): Bitcoin addresses to look up (one per line).
  - `includeTransactions` (boolean): Fetch the most recent transactions for each address (one extra request each).
  - `maxTransactions` (integer): How many recent transactions to include per address (API returns up to 50).

### CoinGecko Coins Scraper (`coingecko-coins-scraper`)

Live crypto market data — price, market cap, volume, supply and ATH for any coin.

Input fields:
  - `coinIds` (array): Specific CoinGecko coin IDs to fetch (one per line, e.g. bitcoin, ethereum). Leave empty to get the top coins by market cap.
  - `vsCurrency` (string): Currency to quote prices in (usd, eur, gbp, btc, eth…).
  - `maxItems` (integer): Maximum number of coins to return (ranked by market cap when no IDs given).

### CoinPaprika Scraper (`coinpaprika-scraper`)

Crypto tickers from CoinPaprika — price, mcap, % change across 9 timeframes, ATH.

Input fields:
  - `coinIds` (array): Specific CoinPaprika coin IDs (one per line, e.g. btc-bitcoin, eth-ethereum). Leave empty for the top coins by rank.
  - `quoteCurrency` (select): Currency to quote prices in.
  - `includeDetails` (boolean): Fetch per-coin details (description, links, tags…). One extra request per coin — keep the coin count small.
  - `maxItems` (integer): Maximum number of coins to return (ranked when no IDs given).

### Crates.io Package Scraper (`crates-io-scraper`)

Downloads, versions, license and links for any Rust crate from the crates.io API.

Input fields:
  - `crates` (array, required): Crate names as they appear on crates.io, one per line.
  - `includeDailyDownloads` (boolean): Fetch per-day downloads for the last ~90 days (one extra request per crate).
  - `maxRecentVersions` (integer): How many of the most recent versions to include per crate.

### Crossref DOI Metadata Scraper (`crossref-scraper`)

Search 150M+ Crossref records: DOI metadata, citation counts, references and licenses.

Input fields:
  - `query` (string, required): Free-text search across bibliographic metadata.
  - `filterType` (select): Optionally restrict to a Crossref work type.
  - `fromYear` (integer): Only works published in or after this year.
  - `untilYear` (integer): Only works published in or before this year.
  - `sort` (select): How to order results.
  - `maxItems` (integer): How many works to fetch.

### Crypto Price History Scraper (`crypto-price-history-scraper`)

Historical price, market cap and volume time series for any cryptocurrency.

Input fields:
  - `coinIds` (array, required): CoinGecko coin IDs to fetch history for (one per line, e.g. bitcoin, ethereum).
  - `vsCurrency` (string): Currency to quote prices in (usd, eur, gbp, btc, eth…).
  - `days` (integer): How many days back from now (1 = ~5-min points, 2–90 = hourly, more = daily).
  - `maxItems` (integer): Maximum total data points to return across all coins.

### Deezer Artist (`deezer-artist-scraper`)

Get a music artist's Deezer profile, fan count, top tracks and full album list.

Input fields:
  - `artists` (array, required): Artist names, numeric Deezer IDs, or deezer.com artist URLs — one per line.
  - `maxTopTracks` (integer): How many of the artist's most popular tracks to include.
  - `maxAlbums` (integer): How many albums (newest first) to include per artist.

### DefiLlama TVL Scraper (`defillama-scraper`)

DeFi protocols ranked by TVL — per-chain TVL breakdown, changes, mcap and links.

Input fields:
  - `protocols` (array): Specific protocols to fetch, by name or slug (one per line, e.g. aave, uniswap). Leave empty for the top protocols by TVL.
  - `category` (string): Only include protocols in this category (e.g. Lending, Dexs, Liquid Staking).
  - `chain` (string): Only include protocols deployed on this chain (e.g. Ethereum, Solana, Base).
  - `maxItems` (integer): Maximum number of protocols to return (ranked by TVL).

### Docker Hub Repository Scraper (`dockerhub-scraper`)

Pulls, stars, description and recent tags for any Docker Hub image.

Input fields:
  - `repositories` (array, required): Docker images as name, namespace/name, or Docker Hub URL. Official images resolve to the 'library' namespace.
  - `tagsPerRepo` (integer): How many of the most recently pushed tags to include (0 to skip).
  - `includeFullDescription` (boolean): Include the repository's full markdown description (can be long).

### Exchange Rates Scraper (`exchange-rates-scraper`)

Live & historical FX rates for 30+ currencies — with trend stats per pair.

Input fields:
  - `baseCurrency` (string): ISO 4217 code of the base currency (rates are quoted as 1 base = X target).
  - `targetCurrencies` (array): Currency codes to quote against, one per line. Leave empty for all supported currencies.
  - `historyDays` (integer): How many calendar days of daily history to include per pair (0 = latest rate only).
  - `maxItems` (integer): Maximum number of currency pairs to return.

### GitHub Issues Scraper (`github-issues-scraper`)

Extract issues from any public repo with labels, assignees, reactions and comment counts.

Input fields:
  - `repos` (array, required): Repositories as owner/name (or full GitHub URLs), one per line.
  - `state` (select): Which issues to fetch.
  - `labels` (string): Comma-separated list of labels the issues must have (optional).
  - `sort` (select): Ordering of returned issues (newest/most first).
  - `includePullRequests` (boolean): The GitHub API returns PRs in the issues list; include them too.
  - `maxItemsPerRepo` (integer): Maximum issues to collect per repository.

### GitHub Releases Scraper (`github-releases-scraper`)

Fetch release history for any repo with assets, sizes and download counts.

Input fields:
  - `repos` (array, required): Repositories as owner/name (or full GitHub URLs), one per line.
  - `maxReleasesPerRepo` (integer): Maximum releases to collect per repository (newest first).
  - `includePrereleases` (boolean): Also return releases marked as pre-release.
  - `includeDrafts` (boolean): Also return draft releases (only visible with a token that can see them).

### GitHub Repository Search Scraper (`github-repo-search-scraper`)

Search GitHub repos by keyword, language and stars — full metadata for every match.

Input fields:
  - `query` (string, required): Keywords plus optional GitHub search qualifiers (stars:, topic:, org:, in:name...).
  - `language` (string): Restrict to a primary language (adds language:<value> to the query).
  - `minStars` (integer): Only repos with at least this many stars (adds stars:>=N). 0 = no filter.
  - `sort` (select): Result ordering.
  - `maxItems` (integer): Maximum repositories to return.

### GitHub Trending Scraper (`github-trending-scraper`)

Scrape GitHub's trending repositories with stars gained today, language and contributors.

Input fields:
  - `language` (string): Filter by language slug (e.g. typescript, python, rust). Empty = all languages.
  - `since` (select): Time window for the trending calculation.
  - `spokenLanguage` (string): Optional spoken language filter (e.g. en, zh, ja).
  - `limit` (integer): Maximum repositories to return (the page lists up to 25).

### GitHub User Scraper (`github-user-scraper`)

Profile data plus a recent-activity summary for any GitHub user or organization.

Input fields:
  - `usernames` (array, required): GitHub logins or profile URLs, one per line.
  - `includeActivity` (boolean): Summarize the user's latest public events (one extra request per user).

### Hacker News Comments Scraper (`hn-comments-scraper`)

Extract the full nested comment tree of any Hacker News story in one shot.

Input fields:
  - `stories` (array, required): One story per line — a news.ycombinator.com item URL or a raw numeric id.
  - `maxComments` (integer): Stop after collecting this many comments for each story.
  - `maxDepth` (integer): How deep into nested replies to go (0 = top-level comments only).

### App Store Reviews (iTunes) (`itunes-reviews-scraper`)

Deep-dive one iOS app's reviews and rating stats across multiple country stores.

Input fields:
  - `appIds` (array, required): Numeric App Store app IDs (from the app's store URL), one per line.
  - `maxReviewsPerApp` (integer): How many reviews to return per app per country store (Apple embeds ~10-20 recent ones).
  - `countries` (array): Two-letter App Store country codes to loop through (us, gb, de, ...).

### Lobsters Scraper (`lobsters-scraper`)

Pull hottest, newest or tag-filtered stories from Lobste.rs with full metadata.

Input fields:
  - `feed` (select): Which Lobste.rs feed to pull.
  - `tag` (string): Tag to filter by when feed is "By tag" (e.g. rust, security, ai).
  - `limit` (integer): How many stories to collect.

### MusicBrainz Artist (`musicbrainz-artist-scraper`)

Look up any music artist on MusicBrainz: profile, links, tags and full discography.

Input fields:
  - `artists` (array, required): One artist or band name per line.
  - `maxReleasesPerArtist` (integer): How many release groups (albums, EPs, singles…) to include per artist.

### npm Package Scraper (`npm-package-scraper`)

Full npm registry metadata plus weekly & monthly download counts for any package.

Input fields:
  - `packages` (array, required): npm package names, one per line. Scoped packages like @babel/core work too.
  - `includeDownloads` (boolean): Fetch weekly and monthly download counts (two extra requests per package).

### OpenAlex Works Scraper (`openalex-scraper`)

Search 250M+ scholarly works on OpenAlex: citations, concepts, open-access status.

Input fields:
  - `query` (string, required): Full-text search across work titles, abstracts and fulltext.
  - `filter` (string): Optional raw filter string, e.g. publication_year:2024,is_oa:true (comma = AND). See docs.openalex.org.
  - `sort` (select): How to order results.
  - `maxItems` (integer): How many works to fetch.

### Packagist Package Scraper (`packagist-package-scraper`)

Downloads, GitHub stats, versions and dependencies for any Composer/PHP package.

Input fields:
  - `packages` (array, required): Composer package names as vendor/package, one per line.

### Podcast Episodes (`podcast-episodes-scraper`)

Extract every episode of a podcast from its RSS feed — audio URL, duration, show notes and more.

Input fields:
  - `feedUrls` (array, required): One podcast RSS feed URL per line (get them from the Podcast Search scraper).
  - `maxEpisodesPerFeed` (integer): Limit how many episodes (newest first, as ordered in the feed) to return per podcast.

### Podcast Search (Apple Podcasts) (`podcast-search-scraper`)

Search the Apple Podcasts directory and get show metadata, RSS feed URLs and artwork.

Input fields:
  - `queries` (array, required): One search term per line (show name, topic, or publisher).
  - `maxResultsPerQuery` (integer): How many podcasts to return for each query (API max 200).
  - `country` (string): Two-letter country code of the Apple Podcasts store to search.

### PubMed Article Scraper (`pubmed-scraper`)

Search PubMed biomedical literature and get rich article metadata as JSON.

Input fields:
  - `query` (string, required): PubMed search term. Supports full PubMed syntax (field tags like [Title], [au], [pdat]).
  - `sort` (select): Result ordering.
  - `maxItems` (integer): How many articles to fetch.

### PyPI Package Scraper (`pypi-package-scraper`)

Rich PyPI metadata plus daily/weekly/monthly download stats for any Python package.

Input fields:
  - `packages` (array, required): PyPI package names, one per line (case/underscore insensitive).
  - `includeDownloads` (boolean): Fetch day/week/month download counts from pypistats.org (one extra request per package).

### Reddit Comments Scraper (`reddit-comments-scraper`)

Extract the full nested comment tree of any Reddit post, with scores and depth.

Input fields:
  - `postUrls` (array, required): One Reddit post per line — full URL, redd.it short link, or raw post id.
  - `sort` (select): Which comment ordering Reddit should return.
  - `maxComments` (integer): Stop after collecting this many comments for each post.
  - `maxDepth` (integer): How deep into nested replies to go (0 = top-level comments only).

### Reddit Search Scraper (`reddit-search-scraper`)

Search all of Reddit (or one subreddit) and extract full post metadata.

Input fields:
  - `queries` (array, required): One search query per line.
  - `subreddit` (string): Limit the search to a single subreddit (without r/). Leave empty to search all of Reddit.
  - `sort` (select): How to sort search results.
  - `timeframe` (select): Only include posts from this window (applies to relevance/top/comments).
  - `limit` (integer): How many posts to collect for each query.

### Reddit User Scraper (`reddit-user-scraper`)

Profile any Reddit user: karma breakdown, account age, plus their recent posts.

Input fields:
  - `usernames` (array, required): One Reddit username per line (with or without the u/ prefix).
  - `includePosts` (boolean): Also fetch the user's most recent submissions as separate items.
  - `maxPosts` (integer): How many recent submissions to collect for each user.

### RubyGems Package Scraper (`rubygems-package-scraper`)

Downloads, versions, dependencies and links for any Ruby gem from the official API.

Input fields:
  - `gems` (array, required): RubyGems gem names, one per line.
  - `includeVersionHistory` (boolean): Fetch the full version list for release dates and version counts (one extra request per gem).

### Semantic Scholar Paper Scraper (`semantic-scholar-scraper`)

Search Semantic Scholar and get citations, open-access PDFs and rich paper metadata.

Input fields:
  - `query` (string, required): Free-text search across paper titles and abstracts.
  - `yearRange` (string): Optional publication year or range: 2024, 2020-2024, or 2015- (open-ended).
  - `fieldsOfStudy` (string): Optional comma-separated fields, e.g. Computer Science, Medicine, Biology, Physics.
  - `openAccessOnly` (boolean): Only return papers that have a free public PDF.
  - `maxItems` (integer): How many papers to fetch.

### Stock Price History (OHLC) (`stock-price-history-scraper`)

Daily/weekly/monthly OHLC candles with volume for any Yahoo Finance ticker.

Input fields:
  - `symbols` (array, required): Yahoo Finance symbols, one per line (AAPL, MSFT, ^GSPC, BTC-USD, EURUSD=X…).
  - `range` (select): How far back to fetch candles.
  - `interval` (select): Granularity of each candle.
  - `maxItemsPerSymbol` (integer): Cap on candles returned per symbol (most recent kept).

### YouTube Playlist Scraper (`youtube-playlist-scraper`)

Extract playlist metadata plus every entry with index, duration and channel.

Input fields:
  - `playlists` (array, required): YouTube playlist URLs or playlist IDs (one per line).
  - `maxVideos` (integer): Cap on entries returned per playlist (first page holds up to 100).

### YouTube Search Scraper (`youtube-search-scraper`)

Scrape YouTube search results: videos with views, channel, duration and more.

Input fields:
  - `queries` (array, required): One search query per line. Each query costs one page fetch.
  - `maxResults` (integer): Cap on videos returned per query (a page holds ~20).
  - `sortBy` (select): Result ordering.

### Bing Search Scraper (`bing-search-scraper`)

Scrape Bing organic results: position, title, URL, snippet, deep links and pagination.

Input fields:
  - `queries` (array, required): One search query per line.
  - `pagesPerQuery` (integer): How many result pages (of 10 results) to fetch per query.
  - `market` (string): Optional Bing market code to localize results (e.g. en-US, de-DE, fr-FR).
  - `maxResults` (integer): Hard cap on total results across all queries.

### DuckDuckGo Search Scraper (`duckduckgo-search-scraper`)

Scrape DuckDuckGo organic results with decoded URLs, snippets and positions.

Input fields:
  - `queries` (array, required): One search query per line.
  - `region` (string): Optional DuckDuckGo region code (kl), e.g. us-en, uk-en, de-de.
  - `timeRange` (select): Restrict results by recency.
  - `maxResults` (integer): Hard cap on total results across all queries.

### Google Autocomplete Scraper (`google-autocomplete-scraper`)

Harvest Google search suggestions per keyword, locale and optional a–z seed expansion.

Input fields:
  - `keywords` (array, required): One seed keyword per line.
  - `languages` (array): Interface language codes to query, one per line (e.g. en, de, fr, es).
  - `country` (string): Optional two-letter country code to localize suggestions (e.g. us, de, in).
  - `expandAlphabet` (boolean): Also query each keyword suffixed with a…z for long-tail suggestions (capped globally).
  - `maxResults` (integer): Hard cap on total suggestions collected.

### Wikipedia Search Scraper (`wikipedia-search-scraper`)

Full-text Wikipedia search with intro extracts, images, descriptions and page metadata.

Input fields:
  - `queries` (array, required): One full-text search query per line.
  - `language` (string): Wikipedia language edition subdomain (en, de, fr, es, ja, …).
  - `resultsPerQuery` (integer): How many pages to return for each query.

### YouTube Suggest Scraper (`youtube-suggest-scraper`)

Collect YouTube search autocomplete suggestions with rank for every seed keyword.

Input fields:
  - `keywords` (array, required): One seed keyword per line.
  - `language` (string): Interface language code (en, de, es, ja, …).
  - `country` (string): Optional two-letter country code (us, gb, in, …).
  - `maxPerKeyword` (integer): Cap on suggestions kept for each seed keyword.

### eBay Search Results (`ebay-search-scraper`)

Scrape eBay search listings — price, condition, shipping, seller, sold count and more.

Input fields:
  - `queries` (array, required): One eBay search query per line.
  - `maxItemsPerQuery` (integer): How many results to collect per query (max 240, paginates automatically).
  - `sortOrder` (select): How eBay should sort the results.
  - `buyItNowOnly` (boolean): Exclude auction listings.
  - `useUnlocker` (boolean): Route through the Web Unlocker if eBay blocks direct requests (metered).

### eBay Product Details (`ebay-product-scraper`)

Full eBay item data — price, condition, seller feedback, shipping, returns, specifics, images.

Input fields:
  - `urls` (array, required): eBay item page URLs (ebay.com/itm/...) or bare numeric item IDs, one per line.
  - `useUnlocker` (boolean): Route through the Web Unlocker if eBay blocks direct requests (metered).

### Etsy Listing Scraper (`etsy-product-scraper`)

Extract Etsy listing data — price, rating, reviews, shop, variations, images and tags.

Input fields:
  - `urls` (array, required): Etsy listing page URLs, one per line.
  - `useUnlocker` (boolean): Route through the Web Unlocker if Etsy blocks direct requests (metered).

### AliExpress Product Details (`aliexpress-product-scraper`)

Scrape AliExpress products — tiered prices, rating, orders, store info, SKU variants, images.

Input fields:
  - `urls` (array, required): AliExpress product page URLs (aliexpress.com/item/<id>.html), one per line.

### Shopify Store Products (`shopify-products-scraper`)

Dump any Shopify store's full catalog — products, variants, prices, stock flags, images, tags.

Input fields:
  - `storeUrls` (array, required): Shopify store URLs (one per line). Any page of the store works.
  - `maxProducts` (integer): Cap on products collected per store (paginates 250 at a time).
  - `includeCollections` (boolean): Also emit the store's collections (from /collections.json) as items with type "collection".

### Remote OK Jobs Scraper (`remoteok-jobs-scraper`)

Pull the latest remote job listings from Remote OK with salary, tags, company and apply links.

Input fields:
  - `tag` (string): Only keep jobs carrying this tag (case-insensitive; e.g. python, react, marketing). Leave empty for all.
  - `search` (string): Free-text filter matched against position, company and tags (case-insensitive). Leave empty for all.
  - `limit` (integer): Maximum number of job items to return.

### We Work Remotely Jobs Scraper (`weworkremotely-jobs-scraper`)

Collect remote job listings from We Work Remotely category feeds with region and full description.

Input fields:
  - `categoryFeed` (select): Which We Work Remotely category feed to pull.
  - `limit` (integer): Maximum number of job items to return.
  - `fetchJobPages` (boolean): Also fetch each job's page to extract the apply URL and company profile link (slower; one extra request per job).

### HN Who Is Hiring Scraper (`hn-hiring-scraper`)

Parse the latest "Ask HN: Who is hiring?" thread into structured job listings.

Input fields:
  - `storyId` (string): HN story ID of a specific 'Who is hiring?' thread. Leave empty to auto-detect the latest month's thread.
  - `query` (string): Only keep listings whose text contains this term (case-insensitive), e.g. 'rust', 'remote', 'san francisco'.
  - `limit` (integer): Maximum number of job listings to return.

### Greenhouse Jobs Scraper (`greenhouse-jobs-scraper`)

Extract every open role from any company's Greenhouse job board, with full descriptions.

Input fields:
  - `boards` (array, required): Greenhouse board tokens (company identifiers), one per line — e.g. stripe, gitlab, datadog.
  - `titleSearch` (string): Only keep jobs whose title contains this text (case-insensitive).
  - `limit` (integer): Maximum total number of job items across all boards.

### Lever Jobs Scraper (`lever-jobs-scraper`)

Extract all open roles from any company's Lever job board, with teams, salary and requirements.

Input fields:
  - `companies` (array, required): Lever company slugs, one per line — the part after jobs.lever.co/ (e.g. palantir, zoox).
  - `titleSearch` (string): Only keep jobs whose title contains this text (case-insensitive).
  - `limit` (integer): Maximum total number of job items across all companies.

### Certificate Expiry Monitor (`certificate-expiry-monitor`)

Track SSL/TLS certificate expiry per domain via Certificate Transparency logs.

Input fields:
  - `domains` (array, required): Domains to check (bare domains or URLs), one per line.

### Keyword Monitor (`keyword-monitor`)

Watch pages and feeds for keywords and get context snippets whenever they appear.

Input fields:
  - `sources` (array, required): Web pages or RSS/Atom feed URLs, one per line.
  - `keywords` (array, required): Keywords or phrases to look for, one per line.
  - `caseSensitive` (boolean): Match keywords exactly as typed instead of ignoring case.
  - `maxSnippetsPerKeyword` (integer): How many context snippets to keep for each keyword per source.

### Meta Tags Scraper (`meta-tags-scraper`)

Extract every meta tag, favicon and head directive from any page, with validation flags.

Input fields:
  - `urls` (array, required): Pages to inventory.

### Page Change Monitor (`page-change-monitor`)

Fingerprint pages with content hashes so scheduled runs reveal exactly what changed.

Input fields:
  - `urls` (array, required): One URL per line. Each page becomes one output item.
  - `selector` (string): Watch only this region of each page (e.g. "main", "#pricing", ".changelog"). The full-page hash is still included.

### Uptime Checker (`uptime-checker`)

Ping your URLs and record status, response time, redirects and content checks.

Input fields:
  - `urls` (array, required): One URL per line. Each becomes one output item.
  - `containsText` (string): If set, the check only passes when the response body contains this text (case-insensitive).
  - `timeoutSeconds` (integer): Fail the check if no response within this many seconds.

### WHOIS RDAP Scraper (`whois-rdap-scraper`)

Structured WHOIS for any domain via RDAP: registrar, dates, nameservers, status, DNSSEC, abuse contact.

Input fields:
  - `domains` (array, required): Domains to look up, one per line (no scheme needed).

### DNS Records Scraper (`dns-records-scraper`)

Full DNS record lookup per domain — A/AAAA/MX/TXT/NS/CNAME/SOA/CAA plus SPF, DMARC and provider detection.

Input fields:
  - `domains` (array, required): Bare domains to resolve (protocol/path stripped automatically).
  - `recordTypes` (array): Which record types to query (default: all of A, AAAA, MX, TXT, NS, CNAME, SOA, CAA). Each type costs one request per domain.
  - `checkDmarc` (boolean): Perform the extra _dmarc.<domain> TXT lookup (one more request per domain).

### SSL Certificate Scraper (`ssl-cert-scraper`)

Certificate-transparency history per domain from crt.sh — issuers, validity windows, SANs and wildcards.

Input fields:
  - `domains` (array, required): Domains to look up in certificate-transparency logs.
  - `maxCertsPerDomain` (integer): Cap on certificate items returned per domain (newest first).
  - `excludeExpired` (boolean): Only fetch certificates that have not yet expired (faster on busy domains).
  - `includeSubdomains` (boolean): Query %.domain to include certificates issued for subdomains.

### Redirect Tracker (`redirect-tracker-scraper`)

Where does a URL actually land? Start → final URL with protocol, www, meta-refresh and canonical checks.

Input fields:
  - `urls` (array, required): URLs to check (bare domains get https:// prefixed; use http:// explicitly to test upgrades).

### OpenGraph & Twitter Card Scraper (`opengraph-scraper`)

Complete og:/twitter: tag inventory with parsed images, oEmbed discovery and per-platform link previews.

Input fields:
  - `urls` (array, required): Pages whose social metadata to extract.

### Schema.org Structured Data Scraper (`schema-org-scraper`)

Every JSON-LD block and microdata scope on a page, normalized, counted and linted for common types.

Input fields:
  - `urls` (array, required): Pages to extract structured data from.

### Google Trends — Trending Searches (`google-trends-scraper`)

Real-time trending searches from Google Trends by country, with news coverage per trend.

Input fields:
  - `geos` (array): Two-letter country codes to pull trending searches for (e.g. US, GB, IN).
  - `maxPerGeo` (integer): How many trending searches to collect per country.

### Bing News Scraper (`bing-news-scraper`)

Bing News results by keyword or category — headlines, sources, snippets, dates, images.

Input fields:
  - `queries` (array): Keyword searches to run against Bing News (RSS path, most reliable).
  - `categories` (array): Section pages to pull: world, us, business, politics, technology, science, entertainment, sports, health. (HTML path — may be blocked on some IPs.)
  - `maxPerFeed` (integer): Cap on articles per query or category.

### GDELT Global News Scraper (`gdelt-news-scraper`)

Search 65-language global news coverage via the GDELT 2.0 DOC API — free, no key.

Input fields:
  - `queries` (array, required): Keyword/phrase queries (GDELT syntax: "exact phrase", OR, domain:cnn.com, …).
  - `timespan` (string): How far back to search: e.g. 1h, 1d, 1w, 2m (max ~3 months).
  - `tone` (string): Optional tone threshold, e.g. "<-5" (negative), ">5" (positive), "<-2".
  - `sourceCountry` (string): Restrict to publishers from one country (name, e.g. "france").
  - `sourceLang` (string): Restrict to one article language (name, e.g. "spanish").
  - `sort` (select): How to order results.
  - `maxPerQuery` (integer): How many articles to collect per query (API max 250).

### Front-Page Headlines Scraper (`frontpage-headlines-scraper`)

Top-stories headlines from BBC, Guardian, NPR, NYT, Al Jazeera & more in one run.

Input fields:
  - `outlets` (array): Curated outlets to pull: bbc, guardian, npr, nyt, aljazeera, cbs, abc, sky, fox, politico, france24, dw. Empty = all.
  - `customFeeds` (array): Extra RSS/Atom feed URLs to include alongside the curated outlets.
  - `maxPerOutlet` (integer): How many headlines to collect per outlet/feed.

### RSS Feed Discovery (`rss-discovery-scraper`)

Find, validate and preview the RSS/Atom feeds of any website.

Input fields:
  - `urls` (array, required): Sites to discover feeds for (homepages or blog roots work best).
  - `probeCommonPaths` (boolean): Also try well-known feed locations like /feed and /rss.xml.
  - `maxFeedsPerSite` (integer): Stop after this many validated feeds per site.

### Yellow Pages Search Scraper (`yellowpages-scraper`)

Extract businesses from yellowpages.com search — phone, address, rating, categories, website.

Input fields:
  - `searchTerms` (array, required): What kind of business to search for — one term per line.
  - `location` (string, required): City/state or ZIP, e.g. "New York, NY" or "90210".
  - `maxResults` (integer): How many listings to collect per search term (about 30 per page).

### BBB Business Profile Scraper (`bbb-business-scraper`)

Scrape BBB.org business profiles — rating, accreditation, contacts, complaint summary.

Input fields:
  - `profileUrls` (array, required): Full bbb.org business profile URLs, one per line.

### Clutch Agency Directory Scraper (`clutch-agencies-scraper`)

Scrape Clutch.co directory listings — ratings, reviews, rates, team size, location, services.

Input fields:
  - `directoryUrls` (array, required): Clutch.co category/directory pages, one per line.
  - `maxAgencies` (integer): How many agencies to collect per directory URL.

### Email Finder & Pattern Detector (`email-finder-scraper`)

Find emails on a company site, detect the address pattern, and guess emails for given names.

Input fields:
  - `domain` (string, required): Domain or website URL of the company, e.g. mozilla.org.
  - `names` (array): Full names to generate candidate addresses for, one per line.
  - `maxPages` (integer): How many pages of the site to scan for addresses.

### Company Enrichment Scraper (`company-enrichment-scraper`)

Turn a domain into a rich company profile — name, logo, socials, contacts, tech, size hints.

Input fields:
  - `domains` (array, required): Company domains or website URLs, one per line.
  - `maxPagesPerDomain` (integer): Homepage + how many about/contact pages to check (total).

### URL to Markdown (`url-to-markdown-scraper`)

Convert any web page into clean, faithful CommonMark Markdown with YAML front-matter.

Input fields:
  - `urls` (array, required): Web pages to convert to Markdown.
  - `mainContentOnly` (boolean): Convert only the <main>/<article> region instead of the full page.
  - `includeFrontMatter` (boolean): Prepend a YAML front-matter block (title, sourceUrl, author, published, fetchedAt) to the markdown.

### Docs Crawler (`docs-crawler-scraper`)

Crawl a documentation section into per-page Markdown chunks, ready for RAG.

Input fields:
  - `startUrl` (string, required): A page inside the documentation section to crawl.
  - `maxPages` (integer): Stop after this many documentation pages.
  - `pathPrefix` (string): Only crawl URLs whose path starts with this. Defaults to the start URL's directory.
  - `includeMarkdown` (boolean): Include the converted Markdown body in each item (turn off for outline-only crawls).

### FAQ Extractor (`faq-extractor-scraper`)

Pull every question & answer pair from any page — schema.org FAQPage plus smart heuristics.

Input fields:
  - `urls` (array, required): Pages to mine for FAQ question/answer pairs.
  - `includeSummary` (boolean): Push one summary item per page with counts and sources.
  - `heuristics` (boolean): Also mine dt/dd lists, details/summary accordions and ?-headings (not just JSON-LD).

### llms.txt Generator (`llms-txt-generator`)

Crawl a site's key pages and generate a ready-to-publish llms.txt file.

Input fields:
  - `siteUrl` (string, required): The website to generate an llms.txt for (its homepage or root).
  - `maxPages` (integer): How many key pages to visit for titles and summaries (besides the homepage).
  - `siteDescription` (string): Optional one-liner used as the '>' description instead of the homepage meta description.

### Readability Batch (`readability-batch-scraper`)

Batch-convert URL lists into minimal, uniform, LLM-ready text records.

Input fields:
  - `urls` (array, required): Any list of web page URLs (non-HTML entries are skipped gracefully).
  - `maxTextChars` (integer): Cap the extracted text at this many characters.

### Docker Hub Search Scraper (`dockerhub-search-scraper`)

Search Docker Hub and collect images with pulls, stars, badges and update dates.

Input fields:
  - `queries` (array, required): Keywords to search Docker Hub for, one per line.
  - `maxPerQuery` (integer): Maximum images to collect for each query.
  - `includeDetails` (boolean): Fetch each repository's detail record for last-updated date, categories and storage size (one extra request per result).

### VS Code Marketplace Scraper (`vscode-marketplace-scraper`)

Search VS Code extensions with installs, ratings, versions, categories and links.

Input fields:
  - `queries` (array, required): Search terms, one per line (extension names, keywords, or publisher names).
  - `maxPerQuery` (integer): Maximum extensions to collect for each query.
  - `sortBy` (select): Result ordering.

### Chrome Web Store Scraper (`chrome-webstore-scraper`)

Extension details from the Chrome Web Store: users, rating, version, developer.

Input fields:
  - `extensions` (array, required): Chrome Web Store detail URLs or 32-character extension IDs, one per line.

### GitLab Projects Scraper (`gitlab-projects-scraper`)

Search public GitLab.com projects or look up single repos — stars, forks, topics.

Input fields:
  - `queries` (array, required): Search keywords (e.g. 'terraform aws') or exact project paths like 'gitlab-org/gitlab' — one per line. Entries containing '/' are treated as paths.
  - `maxPerQuery` (integer): Maximum projects to collect per search keyword (paths always return one).
  - `orderBy` (select): Ordering for keyword searches.

### Stack Exchange Answers Scraper (`stackexchange-answers-scraper`)

Every answer for given Stack Exchange questions — markdown bodies, scores, owners.

Input fields:
  - `questions` (array, required): Question URLs (any Stack Exchange site) or bare numeric question IDs, one per line.
  - `site` (string): Stack Exchange site API name used for bare numeric IDs (e.g. stackoverflow, superuser, askubuntu, math).

### IMDb Title Scraper (`imdb-title-scraper`)

Rich movie & series details from IMDb title pages: ratings, cast, crew, plot and more.

Input fields:
  - `titles` (array, required): IMDb title URLs or ids (tt0111161), one per line.
  - `maxItems` (integer): Maximum number of titles to scrape.

### IMDb Search Scraper (`imdb-search-scraper`)

Instant IMDb search results — titles and people with year, type, top actors and poster.

Input fields:
  - `queries` (array, required): One search term per line (movie titles, series, actor names…).
  - `titlesOnly` (boolean): Keep only movies/series (drop people results).
  - `maxResultsPerQuery` (integer): The API returns up to ~8 suggestions per query.

### Rotten Tomatoes Scraper (`rottentomatoes-scraper`)

Tomatometer & audience scores, critics consensus, cast and details from RT movie pages.

Input fields:
  - `movies` (array, required): Rotten Tomatoes movie URLs or slugs (e.g. m/the_dark_knight), one per line.
  - `maxItems` (integer): Maximum number of movie pages to scrape.

### Goodreads Book Scraper (`goodreads-book-scraper`)

Book details from Goodreads: ratings, review counts, genres, series, ISBNs and more.

Input fields:
  - `books` (array, required): Goodreads book URLs or numeric ids (e.g. 5107), one per line.
  - `maxItems` (integer): Maximum number of book pages to scrape.

### Open Library Books Scraper (`openlibrary-books-scraper`)

Search Open Library's catalog: authors, editions, subjects, ISBNs, covers and ratings.

Input fields:
  - `queries` (array, required): One search query per line — titles, authors or general terms.
  - `resultsPerQuery` (integer): How many books to return for each query.
  - `includeWorkDetails` (boolean): Also fetch each work's /works API record for the full description and subject list (one extra request per book).

### SEC Filings Full-Text Search Scraper (`sec-filings-scraper`)

Full-text search SEC EDGAR filings and get filing metadata plus direct document URLs.

Input fields:
  - `query` (string, required): Full-text search phrase. Use double quotes for exact phrases, e.g. "climate risk".
  - `forms` (string): Optional comma-separated form types to filter by, e.g. 10-K,10-Q,8-K.
  - `startDate` (string): Only filings filed on or after this date.
  - `endDate` (string): Only filings filed on or before this date.
  - `maxItems` (integer): How many filing documents to return.

### SEC Company Facts (XBRL) Scraper (`sec-company-facts-scraper`)

Pull XBRL financial time series (revenue, net income, assets…) straight from SEC EDGAR.

Input fields:
  - `identifiers` (array, required): Company tickers (AAPL, MSFT) or CIK numbers (320193), one per line.
  - `concepts` (array): us-gaap concept tags to extract (leave empty for a default set of key financials).
  - `pointsPerConcept` (integer): Most recent reported periods to keep for each concept.
  - `maxItems` (integer): Overall cap on emitted data points.

### Federal Register Document Scraper (`federal-register-scraper`)

Search US Federal Register rules, proposed rules, and notices with full metadata.

Input fields:
  - `query` (string): Full-text search term. Leave empty to list the latest documents.
  - `documentType` (select): Restrict to a single document type.
  - `agency` (string): Optional agency slug, e.g. environmental-protection-agency or securities-and-exchange-commission.
  - `publishedAfter` (string): Only documents published on or after this date.
  - `publishedBefore` (string): Only documents published on or before this date.
  - `maxItems` (integer): How many documents to fetch.

### Yahoo Finance Quote Scraper (`yahoo-finance-quote-scraper`)

Live quotes for stocks, ETFs, indices, FX, and crypto from Yahoo Finance.

Input fields:
  - `tickers` (array, required): Ticker symbols, one per line (max 50).

### Finviz Stock Screener Scraper (`finviz-screener-scraper`)

Run any Finviz screener filter and export the matching stocks as structured data.

Input fields:
  - `filters` (string): Comma-separated Finviz filter codes, e.g. cap_mega or exch_nasd,sec_technology. Leave empty for all stocks.
  - `order` (string): Optional sort column, e.g. -marketcap, price, -change, ticker.
  - `signal` (string): Optional Finviz signal, e.g. ta_topgainers, ta_newhigh, ta_unusualvolume.
  - `maxItems` (integer): How many stocks to return (20 per page).

### Telegram Channel Scraper (`telegram-channel-scraper`)

Scrape public Telegram channels — posts with views, dates, media and channel stats. No login.

Input fields:
  - `channels` (array, required): Public channel usernames or t.me links, one per line (without @).
  - `postsPerChannel` (integer): How many recent posts to collect from each channel.
  - `includeChannelInfo` (boolean): Also emit one item per channel with title, subscribers and description (type: channel).

### Mastodon Profile Scraper (`mastodon-profile-scraper`)

Scrape any Mastodon profile on any instance — stats, bio fields and recent posts with counts.

Input fields:
  - `accounts` (array, required): Handles as user@instance (any instance) or profile URLs, one per line.
  - `statusesPerAccount` (integer): How many recent statuses to collect for each account.
  - `excludeReplies` (boolean): Skip statuses that are replies to other users.
  - `excludeBoosts` (boolean): Skip boosted (reblogged) statuses.

### Bluesky Profile Scraper (`bluesky-profile-scraper`)

Scrape Bluesky profiles and recent posts with like, repost and reply counts — no auth needed.

Input fields:
  - `actors` (array, required): Bluesky handles, profile URLs or DIDs, one per line.
  - `postsPerActor` (integer): How many recent posts to collect for each actor.
  - `feedFilter` (select): Which author-feed entries to include.

### Pinterest Search Scraper (`pinterest-search-scraper`)

Scrape Pinterest search results — pins with title, description, image, board and save counts.

Input fields:
  - `queries` (array, required): One Pinterest search query per line.
  - `pinsPerQuery` (integer): Maximum pins to collect per query (limited to what the first results page embeds).

### Twitch Channel Scraper (`twitch-channel-scraper`)

Scrape Twitch channel pages — display name, bio, live status and stream details when embedded.

Input fields:
  - `channels` (array, required): Twitch channel names or URLs, one per line.

### App Store Reviews Scraper (`app-store-reviews-scraper`)

Batch-collect Apple App Store ratings and recent customer reviews for many apps at once.

Input fields:
  - `apps` (array, required): Numeric app IDs (e.g. 544007664) or App Store URLs.
  - `country` (string): Two-letter storefront country code (us, gb, de, ...).
  - `maxReviewsPerApp` (integer): Cap per app. Apple only embeds ~10-20 recent reviews per storefront.

### Capterra Reviews Scraper (`capterra-reviews-scraper`)

Scrape software ratings, per-category scores and pros/cons reviews from Capterra.

Input fields:
  - `products` (array, required): Capterra product URLs (capterra.com/p/<id>/<name>/) or '<id>/<name>' pairs.
  - `maxReviewsPerProduct` (integer): How many reviews to collect per product.

### G2 Reviews Scraper (`g2-reviews-scraper`)

Scrape product ratings and detailed user reviews (likes/dislikes) from G2.

Input fields:
  - `products` (array, required): G2 product slugs (e.g. notion) or g2.com product URLs.
  - `maxReviewsPerProduct` (integer): How many reviews to collect per product (~25 per page).

### Google Play Reviews Scraper (`google-play-reviews-scraper`)

Scrape Google Play app reviews with ratings, dates, thumbs-up counts and developer replies.

Input fields:
  - `apps` (array, required): Package IDs (e.g. com.whatsapp) or Play Store URLs.
  - `maxReviewsPerApp` (integer): How many reviews to collect per app.
  - `sort` (select): Which reviews to fetch first.
  - `language` (string): Interface/review language code (hl), e.g. en, de, fr.
  - `country` (string): Storefront country code (gl), e.g. US, GB, DE.

### Sitejabber Reviews Scraper (`sitejabber-reviews-scraper`)

Scrape business ratings and customer reviews from Sitejabber review pages.

Input fields:
  - `businesses` (array, required): Business domains (e.g. amazon.com) or Sitejabber review URLs.
  - `maxReviewsPerBusiness` (integer): How many reviews to collect per business (20 per page).

### Instagram Hashtag Scraper (`instagram-hashtag-scraper`)

Scrape Instagram hashtag pages: total post count plus top and recent posts with stats.

Input fields:
  - `hashtags` (array, required): Hashtags (with or without #) or explore/tags URLs.
  - `maxPosts` (integer): Maximum posts (top + recent combined) to return per hashtag.

### Instagram Posts Scraper (`instagram-posts-scraper`)

Scrape recent Instagram posts per profile with likes, comments, captions and media details.

Input fields:
  - `usernames` (array, required): Instagram usernames or profile URLs to pull recent posts from.
  - `maxPostsPerProfile` (integer): Maximum recent posts to return per profile (the page embeds up to 12).

### TikTok Hashtag Scraper (`tiktok-hashtag-scraper`)

Scrape TikTok hashtag (challenge) pages: total views, video count and top videos with stats.

Input fields:
  - `hashtags` (array, required): TikTok hashtags (with or without #) or /tag/ URLs.
  - `maxVideos` (integer): Maximum embedded videos to return per hashtag.

### TikTok Video Scraper (`tiktok-video-scraper`)

Get full stats for TikTok videos: plays, likes, comments, shares, author, music and hashtags.

Input fields:
  - `videoUrls` (array, required): TikTok video URLs (https://www.tiktok.com/@user/video/123…) or bare video IDs.

### Twitter (X) Search Scraper (`twitter-search-scraper`)

Scrape X/Twitter search results: tweet text, author, timestamps and engagement counts.

Input fields:
  - `queries` (array, required): Keywords, hashtags or X search operators (e.g. `from:nasa`, `#ai`).
  - `maxTweets` (integer): Maximum tweets to return per query.
  - `latest` (boolean): Use the 'Latest' (live) tab instead of 'Top' results.

### Walmart Product Scraper (`walmart-product-scraper`)

Extract Walmart product title, brand, price, rating, seller, images, variants and specs.

Input fields:
  - `urls` (array, required): Walmart product page URLs (walmart.com/ip/...).

### Walmart Search Scraper (`walmart-search-scraper`)

Scrape Walmart search results — title, price, rating, reviews, seller and sponsored flag.

Input fields:
  - `queries` (array, required): One Walmart search query per line.
  - `maxItemsPerQuery` (integer): How many results to collect per query (one page, up to 40).

### Target Product Scraper (`target-product-scraper`)

Extract Target.com product title, price, rating, TCIN, brand, images and availability.

Input fields:
  - `urls` (array, required): Target product page URLs (target.com/p/...).

### Best Buy Product Scraper (`bestbuy-product-scraper`)

Extract Best Buy product SKU, title, price, was-price, rating, model, specs and images.

Input fields:
  - `urls` (array, required): Best Buy product page URLs (bestbuy.com/site/... ending in <sku>.p).

### Home Depot Product Scraper (`homedepot-product-scraper`)

Extract Home Depot product title, brand, price, rating, model, SKU, images and specs.

Input fields:
  - `urls` (array, required): Home Depot product page URLs (homedepot.com/p/...).

### Realtor.com Listings Scraper (`realtor-listings-scraper`)

Extract Realtor.com search results — price, beds/baths, status, agent, photos, coordinates.

Input fields:
  - `searchUrls` (array, required): Full realtor.com search URLs, e.g. https://www.realtor.com/realestateandhomes-search/Austin_TX (filters/sorting in the URL are kept).
  - `maxItems` (integer): Maximum number of listings to return across all search URLs.

### Redfin Property Scraper (`redfin-property-scraper`)

Extract Redfin home details — price, estimate, beds/baths, history, photos, coordinates.

Input fields:
  - `propertyUrls` (array, required): Full Redfin property-detail URLs (the ones containing /home/<id>).

### Apartments.com Listings Scraper (`apartments-listings-scraper`)

Extract Apartments.com results — name, address, rent & beds range, amenities, phone, photos.

Input fields:
  - `urls` (array, required): Search URLs (e.g. https://www.apartments.com/austin-tx/) or individual listing URLs.
  - `maxItems` (integer): Maximum number of properties to return across all URLs.

### Craigslist Search Scraper (`craigslist-search-scraper`)

Scrape craigslist search results — title, price, location, URL, images and coordinates.

Input fields:
  - `city` (string, required): The craigslist city subdomain, e.g. austin, newyork, sfbay, chicago (the part before .craigslist.org).
  - `category` (string): Craigslist category code: sss = all for sale, apa = apartments, cta = cars & trucks, zip = free, jjj = jobs.
  - `query` (string): What to search for. Leave empty to list everything in the category.
  - `maxItems` (integer): Maximum number of postings to return.

### Airbnb Listings Scraper (`airbnb-listings-scraper`)

Extract Airbnb search results — nightly & total price, rating, reviews, coordinates, images.

Input fields:
  - `searchUrls` (array, required): Full airbnb.com search URLs, e.g. https://www.airbnb.com/s/Austin--TX--United-States/homes (dates/filters in the URL are kept).
  - `maxItems` (integer): Maximum number of listings to return across all search URLs.