Asking an AI agent to write you a scraper takes minutes now. Anyone can do it: paste the URL, list the fields you want, and one round later you have code that runs, throws no exceptions, and returns clean JSON.

The problem is everything that comes after. Because that clean JSON might have half the prices parsed wrong, it might have stopped at page one of a 400-page listing, or it might be returning zero rows behind a

try/except

that swallows errors in silence. None of that is visible by reading the code.

Addy Osmani sums it up in a line in Agentic Engineering: generating code is already cheap, and what matters now is who is accountable for what happens next. Applied to web data, the translation is uncomfortable: AI-generated scrapers didn't make reliable data cheaper, they made code that looks like it works cheaper.

The bottleneck moved somewhere else

Osmani uses a distinction that organizes the whole problem: there is an inner loop and an outer loop.

Inner loop: write the code, run it, adjust it.
Outer loop: decide what to build, review it, integrate it, deploy it, watch that it keeps working, and answer to whoever receives the data.

Agents ate the inner loop whole. What they left untouched is the outer one. And that is where 100% of the real work on a data project lives: validating that the extraction is complete, detecting when the site changed, explaining to a client why yesterday's dataset had 12,000 records and today's has 400.

Dex Horthy, quoted in the book, puts it without anesthesia: the problem isn't that you have too many pull requests, it's that you have too many bad pull requests. Review is the bottleneck. In data work, the same sentence reads: the problem isn't that you don't have scrapers, it's that you don't know which of the ones you have are lying to you.

That is also the line between a pipeline you can defend and one you merely hope about, and it is why Data Factory treats verification as part of delivery rather than as something the client's team bolts on afterwards.

Why green tests won't save you

There is an example in the book that applies to anyone working with data. A verifier agent discovered that a "performance is fine" verdict had been measured on desktop only, when the thing that mattered was mobile. The finding wasn't false. It was irrelevant, measured along the wrong dimension, and nobody had noticed.

The scraping version of that mistake is so common it's a classic: "it worked on the 20 pages I tested." The parser handles the 20 pages the agent iterated on and breaks on the remaining 380, where the site changes the price format, drops in an "out of stock" badge that shifts the selector by one div, or paginates differently past page 50.

The other case Osmani tells is even more direct. An agent fixed a real bug, the tests went green, coverage on the modified files was 100%, and the automated review came back clean. Production wouldn't boot. There was a circular dependency that only surfaced with one specific module enabled. The author's conclusion is the sentence worth pinning to the wall: the agent was blind exactly where the tests were blind, and confident in the same proportion.

100% coverage on what changed says absolutely nothing about what was never tested.

The most expensive mistake: thinking you're in control

Osmani proposes five zones for placing how disciplined your process actually is, from pure vibe coding — throwaway scripts, you look at nothing — to a fleet of agents reviewing each other. And he flags which zone is the dangerous one:

The most dangerous mistake is being in zone 3 while believing you're in zone 4.

Zone 3 is where most professional teams live: you have a plan in your head, you review reasonably, but the tests don't go far enough. Zone 4 is the one with a spec before any code, every meaningful diff genuinely reviewed, tests that gate the merge, and a clear owner.

There is no visible difference between the two until something breaks. They feel identical. A data team in zone 3 ships datasets with exactly the same confidence as one in zone 4 — right up to the day a client makes a pricing decision based on competitor prices that were normalized wrong.

Dan Shapiro, also quoted in the book, estimates that around 90% of teams who call themselves "AI-native" are stuck at level 2 of his five-step ladder. It isn't a tooling problem. It's a problem of not knowing where you're standing.

More volume, less quality: what the numbers say

This is where enthusiasm collides with measurement. The figures the book brings are worth reviewing — all of them from public sources that should be checked before you quote them in a proposal, because the book is an early release:

Faros AI measured telemetry from around 22,000 developers over two years. The aggregate result: more throughput, less quality, and the gap widening. More incidents, and review times rising faster than the extra output.
The 2025 METR study: 16 expert developers expected to be 24% faster with AI. They were 19% slower. And the part that matters most: they believed they had been 20% faster. Perception does not work as a metric.
The 2026 repeat of the experiment, with 57 developers and more than 800 tasks, landed close to zero.
Osmani, on his own experience, reports up to 5x on greenfield projects with good specifications and a wash when debugging legacy code.

That last data point is the one that matters most if you work with web data. A new scraper for a tidy site is "greenfield with a good spec" — that's where AI genuinely pays. Fixing 40 scrapers that break when sites change is "debugging legacy code" — there the benefit shrinks hard, and what's left is the cost of verification.

And Horthy names the structural problem: nothing in model training penalizes eroding maintainability. Benchmarks look at whether tests pass, never at architecture. With that incentive, agent-built codebases calcify and turn legacy in three to six months.

What does work: autonomy follows who gets hurt

The actionable part of the book is a criterion, not a tool. Osmani doesn't grade agent autonomy by technical difficulty but by consequence: autonomy follows who gets hurt.

Translated to a portfolio of web data work, it looks like this:

Type of work	Autonomy	Why
Proof of concept to assess whether a site is extractable	High	If it goes wrong, you throw it away. Blast radius near zero
New scraper for a public site, with the output schema fixed	Medium-high	Failures show up fast and are cheap to correct
Maintaining a production flow with daily delivery	Low	A silent error reaches the client before you see it
Pipelines carrying prices, stock, or billing	Minimal	A misparsed price that arrives quietly costs far more than the scraper

Notice the table says nothing about how hard the task is. It talks about what happens if it fails. A two-line selector change in a pricing flow is riskier than a brand-new 300-line scraper for a POC.

There's an operational corollary too: that constraint cannot live in your memory. Osmani is explicit that his own memory of which risk mode he's in is the least reliable component of the system. That's why the rules belong in repository configuration and automated checks, not in the good intentions of whoever is at the keyboard.

The four pillars, and the order to build them in

The book organizes the work into four pillars: plan, steer, test, and own.

Plan. The specification comes before the prompt. One hour of alignment turns a six-hour review into a twenty-minute one.
Steer. Scoped tasks and diffs reviewed according to risk, not size.
Test. This is the wall. And the rule that isn't negotiable: the agent that did the work never decides that the work is good.
Own. Someone accountable. An agent has no reputation and nobody to call at three in the morning when the delivery didn't go out.

The most counterintuitive recommendation in the chapter is about sequence. If you're starting from zero, build the pillars backwards: tests and automated checks first, then who owns what, then written intent, and only then let agents run with autonomy.

Most teams do exactly the opposite: they start by releasing agents and promise to add verification "once things calm down." Things never calm down.

What this means when the product is data, not code

Here is the point the book doesn't have to resolve and we do. When what you deliver is a dataset, verification isn't optional and it isn't an engineering best practice: it is the product.

A client doesn't buy a scraper. They buy the confidence that the number landing in their inbox at 7 a.m. is the number that's on the site. And that confidence isn't produced by the model that wrote the code — it's produced by the system that checks it: schema validation, record counts compared against the trailing average, null rate per field, prices inside expected ranges, deduplication, and a sample a human actually looks at.

That's what Data Factory does: continuous extraction with verification included, not as an extra the client's team has to assemble. And if what your team needs is senior capacity inside its own workflow, with code ownership staying on the client side, that's Data Squad.

The underlying decision — build it in-house or delegate it — is what we work through in our piece on managed web scraping versus doing it in-house with agents.

Frequently asked questions

**Can AI write scrapers on its own yet?**It can write the code, and do it well, especially for new sites with a clear specification. What it still can't do alone is decide whether the data it extracted is complete and correct, because that validation requires knowing what you expected to find. If the agent that wrote the parser is also the one declaring that it works, you don't have verification: you have an opinion.

**So should I avoid AI agents for scraping?**Not at all — use them where they pay off: building the first version of an extractor, exploring a new site, generating fixtures, writing tests. Osmani's data point is clear: up to 5x on greenfield work with a good specification, a wash when it comes to debugging legacy code. The mistake is assuming that 5x applies to maintenance.

**How do I know whether my scraped data is reliable?**With checks that don't depend on whoever produced the data: record counts against the historical average, null rate per required field, type and range validation, deduplication, and comparison against a hand-verified sample. We work through it field by field in the validation checklist.

**Why aren't green tests enough?**Because tests measure what someone thought to measure. A scraper with 100% coverage on its parser can be returning zero rows if nobody wrote the test that compares the count against what was expected. In the book, the extreme case is a change with green tests, 100% coverage, and a clean automated review that still took production down.

**How much time do agents really save?**Less than it feels like. The METR study found expert developers were 19% slower while believing they had been 20% faster. The practical conclusion isn't to stop using agents, it's to stop measuring the benefit by feel and start measuring it per completed task.