Datasets & Storage

Datasets

Every run writes its results to a dataset — an append-only, ordered list of JSON items. Each scraper documents the fields its items contain on its store page.

Exporting

From the console, open any dataset and export it as JSON or CSV. CSV export flattens nested objects into dotted columns (e.g. socials.twitter) and joins arrays with ; .

Over the API:

# JSON (default)
curl ".../api/v1/datasets/ID/items?format=json&limit=1000&offset=0" \
  -H "Authorization: Bearer $KEY"

# CSV
curl ".../api/v1/datasets/ID/items?format=csv" \
  -H "Authorization: Bearer $KEY" -o results.csv

Key-value store

Alongside the dataset, each run gets a key-value store for non-tabular output — think of it as a place for summaries, screenshots-metadata, or auxiliary results (for example, the Google scraper stores "People also ask" questions there).

Retention

Data is retained for 30 days, then cleaned up automatically. Export anything you need to keep long-term.