A scraper that fails silently is worse than one that crashes. The one that crashes tells you. The one that fails silently hands you a CSV with half the rows, the pipeline runs green, the dashboard refreshes, and somebody makes a decision on incomplete data without ever finding out.
AI agents made this worse, for a structural reason. An agent doesn't flag its own uncertainty: it resolves it silently. If the instruction was ambiguous, it picks one interpretation with total confidence and moves on. If the selector doesn't match, it may return an empty list instead of failing. If the field doesn't exist, it may write
nullThat's why scraped data validation isn't a final tidying-up step: it's the part of the work that decides whether you have a dataset or a file shaped like one. Here are the seven checks worth demanding before you call a scraper good, with the engineering reasoning behind each.
The rule that organizes everything: nobody grades their own work
Before the list, the principle. In Agentic Engineering, Addy Osmani frames it as a non-negotiable constraint: the agent that did the work never decides that the work is good. One subagent writes, a different one verifies.
Ken Kousen, in Claude Code: Up and Running, arrives at the same place from another angle, with a line that applies to any data delivery:
Reports are testimony, not evidence.
When an agent tells you "the scraper works correctly and extracted the requested data," that isn't a fact: it's a claim you still have to verify. And the way to verify it is against the data, not against the summary. This is the same standard Data Factory applies to every delivery — the evidence travels with the dataset.
There's a case in Kousen's book that makes the point. A product's price badge and recommendations were broken for six weeks without anyone noticing, because a 400 error stayed hidden behind the frontend. No alert, no test covering it, and the system kept running. The checks below exist so that kind of failure lasts hours, not weeks.
1. Record count against the historical average
The simplest check, and the one that catches the most. "Did it return rows?" isn't enough. The question is did it return the number of rows it normally returns?
What to set up:
- Store the count from every run.
- Compare today's run against the average of the last 7 days.
- Define an explicit deviation threshold that raises an alert (say, ±20%) and that blocks delivery when crossed.
A scraper that went from 12,000 to 11,400 records is probably fine. One that went from 12,000 to 400 changed site, not catalog. And one that returned exactly 12,000 five days running might be serving you a cache.
2. Null rate per required field
A correct count can hide empty rows. The second check looks inside.
Split fields into required and optional, and set a null ceiling for each required one. If SKU, price, currency, and URL are required, 1% nulls is already a signal; 15% is a broken scraper that hasn't crashed yet.
What matters is that the ceiling is a written number, not a judgment call. "Few nulls" can't be checked. "Under 1% nulls on SKU, price, currency, and URL" can.
3. Strict schema validation
The schema isn't documentation: it's a gate. Every record validates against a type definition before it enters the dataset, with tools like Pydantic in Python or the equivalent in your stack.
This covers three distinct failures:
- Types: a price that arrived as the string
instead of a number."1,299.00" - Formats: a date that arrived as
when the contract says ISO.DD/MM/YYYY - Unexpected fields: a new field that appeared, or one that vanished.
And there's a security reason on top of the quality one: if you use a language model to extract fields, a strict output schema stops an instruction hidden in the page from ending up as a field full of garbage. We work through it in the piece on prompt injection.
4. Ranges and business rules
Correct types don't guarantee sensible values. A price of
0-509,999,999Rules worth writing down:
- Prices inside a reasonable range for the category.
- Dates not in the future (unless the case justifies it, as with flights or events).
- Stock as a non-negative integer.
- Currencies from a closed list.
- Cross-field coherence: if there's a discounted price, it should be lower than the list price.
This is the check that catches the most expensive error in pricing: not the missing value, but the value that's there and wrong.
5. Deduplication with an explicit key
Decide in advance what makes a record unique, and check it. It's usually a combination: site + SKU + date, or URL + timestamp.
Two failures it catches:
- Pagination duplicates: the scraper walked page 3 twice because the site reordered the listing mid-extraction.
- Collapsed variants: two sizes of the same product stored as a single record, or the reverse.
That second one is a product decision, not a technical one, and it's exactly the kind of ambiguity an agent will resolve on its own if you don't define it. Is each variant a row, or is it one row with a variants field? If it isn't written down, every run can answer differently.
6. Parsing tests against stored fixtures
This is the check that saves the most time over the long run, and it comes straight from Kousen's book. The acid test he proposes is a question: can you verify this without opening a browser?
If the answer is no, the logic is trapped in the interface. The fix is to split the scraper into three pieces:
- Fetch: get the HTML.
- Parsing: turn HTML into records.
- Normalization: clean and type.
With that separation, you save a sample of real HTML as a file (a fixture) and test the parser against that file — no network, in milliseconds. The example in the book is an app that got to 15 tests running in 129 milliseconds.
The concrete benefit for a data team: when a site changes, you add the new HTML as a fixture and you have a test that fails on the exact line. No hitting the site, no waiting for the nightly run, no burning proxies.
The flip side, which the same book warns about: don't mock what isn't yours. If the site changes, your fixtures keep passing and you never find out. That's why fixtures need a companion: a periodic check against the live site that compares the current structure against the stored one.
7. A sample reviewed by a person
The seventh check is deliberately not automated. A sample of 20 to 50 records, set beside the pages they came from, looked at by somebody.
This catches the class of error no test covers because nobody thought to measure it: the "price" field that is actually pulling the 12-installment financing price, the description that arrives with the cookie notice glued to it, the product name that includes the seller's name.
Kousen has a closing question that works very well here: what did we miss? Green tests don't detect what was never asked for. In his example, with five browsers green, the global review found that left-arrow keyboard support was missing. Nobody had asked for it and nobody had tested it.
How to harden the checklist: stop the agent from loosening the verifier
There's an implementation detail without which the seven checks are worth nothing.
A red test can be turned green two ways: by fixing the code, or by weakening the test. An agent whose objective is "make the tests pass" can lower a threshold, comment out an assert, or add a retry, and declare victory. Kousen's book calls this "don't move the goalposts" and proposes a concrete defense: a hook that blocks edits to tests, to threshold configuration, and to continuous integration workflows, including command-line attempts.
The rule comes out like this: the agent can edit the extractor; it cannot edit what judges the extractor.
And when reviewing, look at the tests that changed first. The worst thing in a maintenance pull request isn't a badly written parser: it's a weakened check that makes everything pass.
The checklist, in one table
| # | Check | Failure it detects | Threshold you have to write |
|---|---|---|---|
| 1 | Count vs. 7-day average | Partial extraction, pagination cut short | Maximum acceptable deviation (e.g. ±20%) |
| 2 | Nulls per required field | Broken selector on one field | Null ceiling per field (e.g. <1%) |
| 3 | Schema validation | Changed types and formats | Versioned schema |
| 4 | Ranges and business rules | Values present but absurd | Range per field and category |
| 5 | Deduplication | Pagination duplicates, variants | Explicit unique key |
| 6 | Tests against fixtures | Parsing regression | Tests that gate delivery |
| 7 | Hand-reviewed sample | The error nobody thought to measure | Sample size and frequency |
The seven checks together are what Osmani calls the evidence package: diff, test output, record sample, before-and-after counts, and the fields the agent couldn't resolve. None of this "the agent says it works." That, in the book's vocabulary, is verification theater: the existence of a check doesn't make you safe.
Where this fits in a data service
If your team is building this system from scratch, the order Osmani recommends is the opposite of the intuitive one: tests and checks first, then owners, then the written specification, and only then agent autonomy.
At Data Factory these checks live inside the delivery: validation isn't something the client has to assemble after the file lands. If what you need is senior people building this system inside your own workflow, with the code staying on your side, that's Data Squad.
Frequently asked questions
**Which of the seven checks should I implement first if I'm short on time?**Count against the historical average (check 1) and null rate per required field (check 2). They're the two cheapest to implement and they catch the largest share of silent failures. With those two you move from "I don't know whether my dataset is good" to "I would find out if it broke."
**Why isn't it enough that the scraper throws no errors?**Because the absence of an error isn't a sign of success. A
try/except**What is a fixture, and why keep them?**It's a sample of real HTML saved as a file. It lets you test the parser offline, in milliseconds, and it gives you a record of what the site looked like at each point in time. When the site changes, the new fixture plus a test points at exactly what broke.
**How often should a sample be reviewed by hand?**It depends on how critical the delivery is. For a daily pricing flow, a weekly sample plus a mandatory one every time the site changes structure. For data feeding financial decisions, on every run with changes. The hand-reviewed sample is the only check that catches errors nobody anticipated.
**Can an AI agent run this validation?**Yes, as long as it isn't the same agent that wrote the extractor and the thresholds are written numbers rather than judgment calls. Better still: checks 1 through 5 are better as deterministic scripts than as an agent. A script can't talk itself into believing the result is fine.
