Data Project Specification: The Seven Parts That Matter

There's an experiment in Agentic Engineering that explains why data projects go wrong better than any argument could.

A friend of Addy Osmani's, skeptical about AI agents, handed one a single paragraph describing a reporting endpoint. Twelve minutes later he had a pull request that compiled, ran, and passed his tests. He got excited, and then did the obvious thing: he ran the same paragraph through five agents in parallel and went to make coffee.

All five came back with code that compiled and worked, and that looked like it had been written by a competent engineer. But none of them solved the same problem. One used a different authentication model. One silently changed the response shape. One wrote to a different database. One was correct but with a useless query interface. The last invented half the specification by copying a similar feature from the repository, and got lucky.

Osmani's conclusion is the line to take away:

He didn't run one spec five times. He ran five different specs. He had written one; the agents completed the other four.

If you have ever asked for "pull the data from this site" and received something that technically complied and was useless, you just read the explanation.

Vagueness branches (and it almost never looks vague)

The central principle of the chapter: every ambiguity you leave in a specification gets resolved without you, once for every agent or person who reads it.

The trouble is that vagueness doesn't look vague. "The catalog data" sounds like a requirement. "Up-to-date prices" sounds like a requirement. "Make it clean" sounds like a requirement. None of them decides anything:

Each of those phrases is, in Osmani's words, a fork with no signpost. The agent — or the vendor, or the internal team — picks a path with total confidence and without telling you the fork existed.

What follows is the counterintuitive part: the faster agents execute, the more the specification matters. Ambiguity used to be bounded by typing speed. Now you can leave a hundred decisions implicit and have all of them resolved, plausibly and wrongly, before the coffee cools.

Dex Horthy has an arithmetic worth memorizing:

• a bad line of code is a bad line of code;
• a bad line of plan can generate hundreds of bad lines;
• a bad line of research — misunderstanding where something lives — can generate thousands.

Attention pays off better upstream than in the two-thousand-line diff at the end.

The seven parts of a data project specification

Osmani proposes seven categories. Small specs use two or three; large ones use all seven. Here they are translated to a web data project.

1. Objective

One paragraph, in plain language: what, and what for. Not "extract prices from three sites," but "daily prices and stock from three competitors so the commercial team can adjust its weekly pricing."

The test: if you can't say it in three sentences, you don't understand the objective yet. And it has a second, very concrete function: when there are two reasonable ways to implement something, the objective is the tiebreaker.

2. Acceptance criteria

Conditions you can check by looking at the result. "Good quality data" can't be checked. This can:

• at least 98% of the products in the target catalog;
• required fields (SKU, price, currency, stock, URL, timestamp) with under 1% nulls;
• delivery available before 7:00 local time.

Osmani is blunt that cutting corners here is the most common cause of work that meets the specification and that humans reject anyway.

3. Edge cases and invariants

What has to stay true no matter how it's implemented:

• the output schema doesn't change without a new version;
• the agreed request-rate limits per domain are never exceeded;
• no personal data is collected;
• no real accounts are used to log in.

The book's observation is a good one: these cases used to be caught in review by a senior engineer who had already been burned. Your agent hasn't been burned yet.

4. Integration boundaries

What's in and what's explicitly out. "Changes only in this site's module; don't touch the shared proxy module."

And the part almost nobody writes, which is the most valuable: the escape hatch. What the agent does when it hits a wall. The right answer is usually "stop and say so," not "work it out however you can." Without that, one agent rewrites the authentication layer while another adds a field, and both pass the tests.

5. Definition of done

The state in which you accept the delivery, process included. Here is the chapter's most useful distinction: acceptance says it meets the requirements; done is what makes it deliverable.

For a data flow: tests against fixtures green, schema validation clean, a 50-row sample attached, a full run in the staging environment with before-and-after counts.

6. Notes and background

What someone experienced would tell you over coffee: "this site loads prices through an async call, the API is at this endpoint"; "we tried one browser automation tool and kept getting blocked, use this other one in low-profile mode"; "this site changes its identifiers on every deploy, use text-based selectors."

Osmani says the difference between converging in one run or in three is almost always in this section. It's the cheapest to write and the most often skipped.

7. Open questions, each with a way to narrow it

Do product variants go as separate rows? First look at how the site presents them and propose; by default, yes.

The reason this section exists: the agent doesn't flag its own uncertainty, it resolves it silently. Unnamed unknowns are the exact place where runs go off the rails.

The discipline of length: under 800 words

Here comes the constraint that surprises people. A longer specification isn't a better one: the longer it gets, the worse the adherence to all of its rules. Models lose consistency somewhere around 150 to 200 instructions in a prompt.

The book's reference points:

And the rule of thumb for not over-engineering: if the specification is longer than the expected deliverable, it's too long. But objective, criteria, and boundaries are never the parts you drop.

There's one more thing that explicitly does not belong in a spec: the mandated stack, the pre-designed API, the file-by-file plan. In the book's words: explicit about the what, quiet about the how. Say what success looks like and what can't break, and let whoever implements propose the plan. Every implementation detail you prescribe takes judgment away from the person who knows that site better than you do.

The specs you can cheat

This is the section worth reading twice if you're about to contract data extraction out. The principle:

Any criterion that can be satisfied without solving the problem will be satisfied that way.

Examples, in data terms:

try/except

null

The other antipatterns the book lists have very recognizable names: the vague spec dressed up as a technical one (sophisticated vocabulary that still decides nothing: "idempotent retry mechanism" without naming the backoff, the attempt ceiling, or which operations); the one that hides assumptions ("standard error handling," "our conventions" — which ones, where?), which Osmani calls the most seductive because the author feels they communicated something and communicated nothing; the one that is actually a complaint ("the data we have is a mess, make it better"); and the expired one, which was correct when it was written and is still being read as truth.

On that last one there's a rule worth adopting: never set a spec running that you no longer believe in.

The step almost nobody takes: ask for the plan before execution

Kousen brings a classic idea into this territory: Design by Contract, from Bertrand Meyer. Applied, the contract has postconditions (what has to be true when it finishes), preconditions, invariants, and negative criteria (what must not happen).

And he makes the observation that justifies this whole article:

An agent that fails a postcondition doesn't throw an exception. It simply tells you it's done.

Which is why the highest-yield habit, according to both books, is to ask for the written plan before a line of code is written, and review it. A good plan restates the criteria, lists the steps in order, makes assumptions explicit, and flags the open questions.

That review separates two failures that look alike and are fixed differently:

• if the plan misreads a clear spec, the problem is the reading;
• if it reads it correctly and leads somewhere you didn't want, you asked for the wrong thing, and no amount of execution was going to save you.

Better still, and it costs five minutes: ask whoever is going to implement to critique the spec before starting. What ambiguities do you see? Which edge cases are missing? Which invariants aren't stated? Osmani reckons thirty minutes with the specification usually saves a full round of work.

One more detail, and it's a management one: the contract freezes when it's approved. Changing the definition of done mid-run is a new negotiation, not a quiet tweak. A contract that moves isn't a contract.

The commercial bonus: the same spec helps you sell

Here's something the book doesn't say, and that for a data project is nearly free.

A well-written spec, with a little polish, is the technical annex of the commercial proposal:

• the acceptance criteria are the service level agreement;
• the invariants are the guarantees;
• the out-of-scope list is what isn't included in the price;
• the definition of done is the sign-off criterion for the delivery.

One piece of work serves you for selling, for building, and for verifying. It's probably the best-invested half hour of the whole project.

At Data Factory this definition is the first step of the process: fields, frequency, format, and criteria are agreed before any code is written. If what you need is senior people writing these specifications inside your own team, that's Data Squad.

Frequently asked questions

**How long should a data project specification be?**Between 300 and 500 words for a single-site flow; up to 800 for a project with several sources. Longer isn't better: rule adherence drops as length grows. What can never be missing is the objective, checkable criteria, and boundaries.

**What if I don't know exactly what data I need yet?**That goes in the open questions section, with a default value for each one. It's very different from leaving them unwritten: an open question with a default is a decision you can revisit; an unwritten question is a decision somebody will make for you without telling you.

**Why shouldn't I specify the tools and the implementation?**Because every detail you prescribe takes judgment away from whoever knows the site better than you, and if the recipe is wrong, the result is worse. The book's rule is explicit about the what, quiet about the how: define the outcome and the constraints, and let the implementer propose the plan.

**How do I avoid acceptance criteria that can be gamed?**By putting in numbers and examples. "No errors" is satisfied by swallowing exceptions; "at least 11,500 records with under 1% nulls on the required fields" isn't. The test is to ask yourself: is there any way to satisfy this literally without solving my problem? If there is, rewrite it.

**Does this spec help if I hire a vendor instead of using agents?**Even more so. A vendor, just like an agent, fills in the gaps you leave — they just take longer to do it. And you end up with it converted into the contract's technical annex: acceptance criteria as the service level, out-of-scope as the limit of what was contracted.

Further reading

• Scraped data validation: 7 checks before you trust it
• AI-generated scrapers: cheap to write, hard to trust
• Managed scraping versus in-house with agents
• The real cost of scraping with AI agents
• Data extraction steps
• Types of data extraction
• How to choose the right scraping provider
• Web scraping use cases that drive business