Anyone who has maintained a scraper for more than a few months knows the pattern: it works fine for weeks, then a site ships a redesign and half your selectors silently start returning nothing or, worse, the wrong thing. The fix used to be manual — open dev tools, find the new class names, patch the script, repeat forever. AI extraction is the first real alternative to that cycle, and it's worth understanding what it actually changes and what it doesn't.
The Old Way: Selectors as a Bet on Markup
A CSS or XPath selector encodes an assumption: "the price will always be the third span inside a div with class product-card." That assumption is only true until it isn't. Selectors are fast, cheap, and completely deterministic when they're right — and silently wrong the moment a site's front-end team ships anything, since nobody on that team owes your scraper backward compatibility.
The New Way: Describe What You Want, Not Where It Is
AI-based extraction flips the instruction. Instead of "read the third span in this div," you give a model a schema — "extract product name, price, and availability" — and the model reads the rendered page (or its text/DOM) and returns values matching that schema, regardless of which specific element they happen to live in this week. The model is doing what a human skimming the page would do: recognizing "this is the price" from context, not from a memorized DOM path.
This is the same underlying idea as Zyte's Automatic Extraction and similar products — give the system a target shape, not a map of the page.
Where It Genuinely Wins
- Redesign resilience. A layout change that would break every selector on the page often doesn't touch AI extraction at all, since it's reading meaning rather than position.
- Cross-site generalization. One extraction approach can work across dozens of differently-built sites in the same category (e.g. job listings, product pages) without a bespoke selector map for each one.
- Handling messy or inconsistent markup. Sites that mix structured and unstructured content, or that render the same field differently depending on the page state, are much harder to hit reliably with rigid selectors.
Where It Doesn't Win
- Cost and latency. A selector match is free and instant. A model call costs money and takes longer. For a page you're re-scraping thousands of times a day, that difference adds up fast — which is why production systems don't send every page to the model (more on that below).
- Determinism. A selector either matches or it doesn't. A model can, in principle, misread a field — which is why confidence scoring and schema-pinning (constraining the output to a fixed structure and set of allowed values) matter more here than in traditional scraping. You want the model producing "extraction failed, low confidence" rather than a plausible-looking wrong number.
- Stable, simple, high-volume sources. If a page's structure genuinely never changes and you're pulling millions of rows, a hand-tuned selector is still cheaper and faster. AI extraction earns its cost on messy or fast-changing sources, not on the easy ones.
The Practical Answer: Both, Applied Selectively
The systems that actually work well in production don't choose one approach — they use the cheap, deterministic method wherever it's reliable, and fall back to AI extraction only where it earns its cost. A common shape: check for structured data the page already publishes (many sites embed JSON-LD or a hidden internal API with clean data, no extraction needed at all), fall back to previously-learned selectors for a site you've crawled before, and only call an LLM when neither of those applies or confidence is low. Done well, that keeps the expensive step to a minority of pages instead of every single one — Foxpull's pipeline is built around exactly that idea, learning a site's structure on the first crawl so repeat runs lean on it instead of re-asking a model every time.
What Schema-Pinning Means in Practice
Rather than asking a model to "extract the interesting data" freeform, you give it an exact schema — field names, types, which are required — and constrain the output to match. This turns an open-ended task into a fill-in-the-form task, which is both more reliable and easier to validate automatically before the data ever reaches you.
What This Means If You're Building Your Own
If you're building extraction yourself: don't reach for an LLM as the first tool. Check for JSON-LD and internal APIs first — they're free and exact when they exist. Use selectors for sources you control or that are genuinely stable. Reserve AI extraction for the sources that are messy, inconsistent, or that redesign often enough that selector maintenance is eating your time — that's specifically where the tradeoff (cost and latency, for redesign-resilience and less maintenance) pays off.
See Schema-Based Extraction in Action
Foxpull reads a page and proposes the exact columns with real example values — try it free on any page.
Try Foxpull