Web scraping services vs licensed data feeds: what an auditor will ask

Most companies run on outside data: tariffs, prices, tenders, sanctions lists, competitor catalogues. It reaches them in two main ways. Someone collects it from websites, or a provider sells it under a licence.

The difference, side by side

The three checks an auditor will ask about

Whatever the source, an auditor or regulator will want the same three answers. Try them on a source of your own.

A word on personal data

If the data includes people, such as names, phone numbers or emails, India's DPDP Act applies even when the data was public. Collect only what you need, and keep a record of why. See our DPDP compliance checklist.

How to choose a web scraping service

1. Does it record the source and time for every record?
2. Does it check each site's terms before collecting?
3. Does it validate data before delivering it, or send you raw pages?
4. Does it alert you when a source changes or goes stale?
5. Can it run inside your environment, so the data never passes through a third party?

When to use which

Common official sources in India

• CBIC: customs tariff and notifications that change duty rates.
• DGFT: foreign trade policy, notifications and trade notices.
• Central Public Procurement Portal: government tenders.
• RBI: reference rates, circulars and master directions.

Official sources are the safest starting point because the origin is clear. They also change format without warning, which is why monitoring matters.

How a good collection pipeline works

1. Schedule: each source is checked as often as it changes, not more.
2. Fetch: politely, at a reasonable rate, as the site allows.
3. Extract: pages are turned into fields, such as tariff line, rate and date.
4. Validate: each record is checked. Is the rate a number? Is the date real?
5. Deliver: clean records go to your systems, with their source attached.
6. Monitor: if a site changes or stops updating, someone is told.

Staying on the right side of the line

• Read and respect each site's terms of use and robots.txt.
• Never get around a login, paywall or technical block.
• Keep request rates low enough not to burden the site.
• Collect personal data only with a clear purpose, and treat it under DPDP.
• Keep a record of what you collected, from where, and when.

When in doubt, ask the publisher for permission or a licence. It is often easier than people think.

Build it yourself, or use a service?

Writing a script that reads one page is easy. Keeping fifty sources running for years is not: sites change, data drifts, and the person who wrote the script moves on. A service is worth it when the data matters to decisions, there are many sources, or you need proof of origin for audits.

Quality checks that catch bad data

• Completeness: are all expected fields present?
• Format: is a date a real date, and a rate a number?
• Range: is a duty rate between 0% and 100%?
• Freshness: when did this source last change?
• Duplicates: is the same record arriving twice?
• Sudden jumps: did half the values change overnight? Often that means the site changed, not the data.

Records that fail should be held for a person, not passed on quietly.

What to keep as proof

• The source address and the time each record was collected.
• A copy of the terms of use, or the licence, at the time.
• Which checks each record passed.
• Who approved any record that failed a check.

The cost of getting it wrong

Bad outside data rarely fails loudly. A stale tariff rate means wrong quotes for weeks. A missed sanctions update means a deal you should not have signed. A changed website layout means a field quietly fills with the wrong value. Checks and monitoring cost far less than these mistakes.

Key terms

• API: a structured way for one system to request data from another, usually more reliable than reading web pages.
• Rate limit: how many requests a site allows in a given time.
• Provenance: the record of where data came from and how it changed.

Where Samvrik fits

Quarry brings outside data in from official sources, licensed feeds and public sites, and checks each record for origin, licence and freshness before it reaches you. Every record keeps its source. It runs inside your own environment. See how Quarry works.

Common questions