AI scraping vs traditional scraping: are selectors finished?

No, selectors aren't finished. AI scraping solves two things very well: getting started fast on sites you've never seen, and surviving redesigns. Traditional scraping still wins on what matters at scale: cost per page, speed and predictable results. In production, what works is combining them: AI writes and repairs the selectors, and the selectors do the heavy lifting.

The problem: the selector that breaks on a Tuesday

A traditional scraper knows the price lives in

span.price-final

. It works for months. One Tuesday the site gets a redesign, the class is renamed

product-price__value

and the scraper returns nothing. Nobody finds out until the report comes out full of zeros.

The promise of AI scraping is that this stops happening: instead of telling the scraper where the price is, you tell it what a price is and let it find it. The promise is real. So is the cost.

What traditional scraping is (in one sentence)

You download the HTML and find the data with an exact rule: a CSS or XPath selector pointing at a specific tag. It's deterministic: given the same HTML it always returns the same thing, in milliseconds, with nothing to pay per page. Its weakness is that the rule depends on the site's design.

What AI scraping is (it's three different things)

"AI scraping" gets used for three techniques that have nothing to do with each other. It's worth separating them, because they cost and perform very differently.

1. Direct extraction: you hand over the page, it hands back JSON

You convert the HTML to text or Markdown, send it to a language model with an instruction ("extract the title, price and availability of each product, as JSON matching this schema") and the model returns the data. Tools like Firecrawl, ScrapeGraphAI and Crawl4AI package that flow up for you.

Upside: it works on any site without writing a single selector. Downside: every page goes through a model, and that gets paid for in time (seconds, not milliseconds) and in money (fractions of a cent to several cents per page, depending on the model and the size of the HTML). And the result can vary between runs.

2. Selector generation: the AI writes the scraper

You show the HTML to an assistant like Claude Code, tell it what you want, and it writes the traditional scraper for you, selectors and all. The model is used once, at build time. After that the scraper runs on its own, free and fast.

This is the one that changed our day-to-day work the most. A scraper that used to take an hour of inspector work and trial and error now comes out in minutes. But the result is a traditional scraper, with exactly the same fragility in the face of a redesign.

3. Automatic repair: the AI fixes the broken selector

The traditional scraper runs. When a selector returns nothing, a process sends the new HTML to a model, asks for the corrected selector, tests it and, if it works, swaps it in. The model only steps in when something breaks, which is once every several months per site.

This is what people picture when they say "a scraper that fixes itself". It combines the zero cost of the traditional approach with the resilience of AI.

The same site, two methods

Goal: the title and price of the 20 books on the homepage of books.toscrape.com.

Traditional (the full script is in the first-scraper tutorial):

for book in soup.select("article.product_pod"):
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").text

Two selectors, zero cost per page, around 300 milliseconds for the whole page including the download.

With AI (direct extraction). The flow, without tying it to a specific tool:

1. Download the page HTML.
2. Convert it to Markdown and strip menus, scripts and the footer.
3. Send to the model:

   "From this content, extract every book.
    Return JSON in this shape, with no extra text:
    [{"title": string, "price": number, "currency": string}]"

4. Validate that the response is valid JSON and matches the schema.
5. Save.

Result: the same JSON, without having looked at the HTML even once. Time: 3 to 10 seconds per page depending on the model. Cost: depends on the model and the size of the HTML. Commercial tools publish prices ranging from thousandths of a dollar per page up to several cents, and it's worth checking them at the time because they change often.

For one page, the difference is anecdotal. For a million, it isn't.

AI scraping vs traditional: comparison table

The math nobody shows you

Let's assume an AI extraction cost of 1 US cent per page, which sits inside the range published by commercial tools in 2026.

Traditional isn't free: somebody has to write and maintain the selectors. But that cost is per site, not per page. Past a few thousand pages a day across the same sites, traditional scraping with automatic repair wins by a mile. With a hundred different sites and a handful of pages from each, direct extraction wins by a mile.

When to use each one

• Few sites, many pages, every day (price monitoring, catalogs): traditional, built with AI and with automatic repair. That's our production standard.
• Many different sites, few pages from each (market research, lead enrichment, prospecting): direct extraction. Writing a hundred scrapers makes no sense.
• Unstructured text (descriptions, reviews, terms and conditions, PDF spec sheets): direct extraction. Selectors understand position, not content.
• A prototype to validate whether the data is useful at all: direct extraction today, and if the project continues, move it to traditional.
• Sites with heavy JavaScript: either one, but neither gets you out of using a browser. AI doesn't replace Playwright.

Common mistakes

• Sending the model the entire HTML. A store's HTML weighs 200 or 500 KB, packed with scripts and menus. Convert it to Markdown and clean it up first: you pay ten times less and the model gets it right more often.
• Not validating the JSON. The model can return a price as text, invent a field or cut the list short. Always validate against a schema and reject anything that doesn't match.
• Trusting that "AI doesn't break". It breaks differently: the model changes, the tool version changes, the price changes. It's still maintenance, just of something else.
• Using AI to evade blocks. It doesn't work for that. The site blocks the request before the model sees anything. Blocks are solved with pacing and good practices.
• Forgetting the legal side because "the AI did it". The rules are the same: robots.txt, terms of use, no personal data. The method doesn't change the legal framework.

Resource: the stack cheat sheet

The one-page PDF with the stack we use in 2026, including the AI scraping tools (which ones, what for and what they charge). Comment STACK on the reel or carousel for this post and we'll send it over by DM.

What about at scale?

The honest answer to "are selectors finished?" is that writing them by hand is finished. At AUTOScraping, AI builds the scrapers and repairs them; the selectors do the everyday work, because they're the only thing that scales to millions of pages without the invoice scaling with them.

That requires both working together, monitoring to know when something broke, and people deciding which method fits when. With DataFactory you don't pick a method: you define what data you need and how often, and you get it clean. Whatever combination of AI and selectors we run behind the scenes is our problem, and it's a problem we enjoy.

Frequently asked questions

**Does AI scraping replace BeautifulSoup or Scrapy?**No. It replaces the work of writing selectors by hand, or avoids it on low-volume sites. To crawl large sites, the crawling is still done by Scrapy or something like it.

**Does AI scraping work on sites with JavaScript?**Same as traditional: somebody has to render the page first. Several AI tools bundle a browser behind the scenes, but it's the same cost.

**Is AI scraping more accurate?**On unstructured text, yes. On tabular data from a stable site, traditional is more accurate because it's deterministic: it doesn't invent, doesn't round and doesn't change its mind between runs.

**How much does AI scraping cost?**It depends on the model, the page size and the tool. In 2026, commercial options range from thousandths of a dollar per page to several cents. Check current pricing before deciding: it changes every few months.

**What about the legality of AI scraping?**Nothing changes. What's public and permitted can be extracted with any method. What isn't, isn't with AI either.