Quarry: frequently asked questions

Quarry is a data acquisition platform. It reaches the external sources your operations depend on, such as websites, portals and public registries, extracts clean, validated facts, and keeps them current on a schedule. It runs inside your own environment, so you see the outside world without building and maintaining custom scrapers.

Structured facts from public-origin, licensed and government sources: prices and availability, tenders and procurement notices, registry and tax records, published lists, supplier and partner portals, and travel fares and hotel rates. It handles plain web pages, JavaScript-heavy sites, APIs, PDFs and scanned documents.

Teams that act on what happens outside their walls: pricing and revenue teams tracking competitors, procurement and compliance teams watching tenders and registries, risk teams monitoring suppliers and partners, and travel businesses tracking fares and availability.

A home-grown scraper bundles everything into one script: fetching, retries, proxies, blocks, parsing and delivery. In Quarry, each source is a small declaration of what to fetch and which fields to extract, and the platform handles how: the fetch method, escalation when blocked, proxies, retries, scheduling, validation, delivery and monitoring. A new source is configuration and a parser, not a new crawler.

No. Quarry does not sell or resell data. It is software that runs in your environment and acquires the sources you choose, for your own use, with each source's terms of access assessed per engagement.

Cheapest method first. Quarry starts with a lightweight HTTP request and steps up through heavier methods, such as a JavaScript challenge sandbox, then a light browser, then a hardened browser, then a browser with CAPTCHA solving, only when a source demands it. Each failure is classified (timeout, blocked or rate-limited) and retried with its own back-off; a job that still fails is held in a review queue rather than lost.

Browsers and CAPTCHA solving are slow and expensive. Quarry uses them only for the sources that need them, so routine collection stays fast and inexpensive, and on-demand requests stay within the methods you approve for that source.

Most sources need no code: extraction rules use JSON paths, CSS selectors, XPath or regular expressions, with fallbacks and a chain of transforms such as trimming, date and currency normalisation and type conversion. Multi-step or unusual sources add a small Python parser. Scanned PDFs and images are read with built-in OCR.

Every request is checked against the source's input contract before any work is done, and results are checked against the source's output schema field by field before they are delivered. Field-level changes are tracked run to run, so consumers see what changed, and an alert is raised when a source's page structure changes and extraction starts to fail.

As often as each source needs: on demand, in batches, or on cron schedules with time zones. Long lists can be paced over a window, for example spreading thousands of lookups across a day, and rolling date ranges are resolved at run time so schedules never go stale.

Quarry escalates to a stronger fetch method automatically, a per-source circuit breaker stops it hammering a site that is failing, and broken extraction raises an alert. Failed jobs land in a review queue where they can be retried once the source is fixed. Source maintenance can be included as a service.

Yes, by configuration. Per-source rate limits and concurrency limits keep collection polite, robots.txt is honoured in crawl mode, and each source's terms of access are assessed per engagement, with the platform configured to respect them.

For many public and government sources, no. For commercial sites protected against bots, residential proxies are usually needed. Quarry routes requests through the proxy providers of your choice, or through providers RUSH procures and manages for you, and handles geography, sticky sessions, health scoring and bandwidth budgets across them.

A local OCR model solves simple image CAPTCHAs inside your environment at no cost. For harder challenges, Quarry uses the solving provider of your choice behind it. CAPTCHA handling is only used for sources that need it.

Each source can carry a daily cost budget and each proxy provider a bandwidth budget, enforced before a job runs. Quarry records what every job did and cost, so you know what every source costs to keep current.

Through the REST API, synchronously for on-demand lookups or asynchronously for jobs and batches; by signed webhooks with retry; as JSONL files; into S3-compatible object storage or PostgreSQL tables; as events; and as a download link posted to Google Chat or Microsoft Teams for scheduled results. A per-entity results store keeps the latest result for every entity you track.

Quarry acquires external data, Smelter harmonizes it with your own data into one governed record, and Assembler acts on it. They are sibling platforms, deployed independently: Quarry's completion events trigger ingestion in Smelter, where external facts are validated and joined to your own data, and Quarry can also deliver directly to Assembler's integration layer.

Yes. Quarry is a standalone platform and delivers into whatever you already run: your data warehouse, object storage, databases, applications or chat channels.

Inside your environment: private cloud or on premises, on containers. Outbound traffic goes only to the sources you target, through the proxies you choose; no acquired data leaves your environment. RUSH can operate Quarry for you inside your walls as a managed service.

Not for acquisition itself: Quarry has to reach the external sources it collects from. Everything else, including storage, results, the dashboard and monitoring, stays inside your environment, and Quarry is typically deployed in a controlled network zone with outbound access limited to its targets.

Users sign in to the Quarry dashboard and APIs with accounts or API keys, with viewer, operator and administrator roles. Each team sees only its own sources and results. Quarry refuses to start without strong credentials and refuses requests aimed at private networks, cloud metadata and internal services.

Every service exposes metrics, and Quarry ships ready-made dashboards for fetching, proxies, browsers and extraction, alert rules and a trace for every job. The admin dashboard shows jobs, schedules, sources, proxy health, the review queue and service-level objectives.

Go, Rust, Python and TypeScript services connected by gRPC contracts; a queue and cache on Valkey; MongoDB for results; Chromium and Firefox browser engines; Tesseract OCR; and Prometheus, OpenTelemetry and Grafana for observability. There is no proprietary runtime.

Quarry is licensed for a term of 3, 5 or 10 years in Starter, Growth and Enterprise editions. Each edition includes a defined number of active sources, monthly jobs, proxy bandwidth and CAPTCHA solves, with support coverage that increases by edition; usage above the allowance and extra sources are available as add-ons. Annual support and updates accompany the licence, and source builds, source maintenance and managed operations are optional services.

Yes. Source builds are offered as fixed-scope services, from open sites and APIs through JavaScript-heavy sites to bot-protected sites, and source maintenance keeps collectors working as sites change.