The Real Cost of Web Scraping: Where Your Money Actually Goes

"How much should scraping this site cost?" is a harder question than it looks, because the honest answer is: it depends entirely on how much the target site resists being scraped. Two sites with the same number of pages can cost 10x apart to crawl reliably. Here's what actually drives the number, so you can tell a fair quote from a guess.

The Four Cost Layers

1. The Fetch Itself

A plain HTTP request from your own server is essentially free. The moment a site blocks datacenter IPs, you need a proxy — and proxy pricing spans a wide range depending on tier. Datacenter proxies are cheapest (fractions of a cent per request) but easiest for sites to detect and block. Residential proxies (IPs that look like real home connections) cost meaningfully more, typically priced per GB of traffic. Mobile proxies, the hardest tier to block, cost more again. A page that a datacenter proxy handles fine costs a fraction of one that needs to escalate all the way to mobile.

2. Browser Rendering

If the target site needs JavaScript execution to show its content, you need a headless browser instance, not just a fetch. That's meaningfully more compute per page — memory, CPU, and wall-clock time all go up, often 10–50x versus a plain request. Sites that render everything server-side skip this cost entirely; sites built as JS-heavy single-page apps don't have that option.

3. Extraction

Turning the fetched page into structured rows costs nothing extra if the site exposes clean structured data (JSON-LD, a public API) or if you're reusing a selector map you've already built for that site. It costs real money if the page needs AI-based extraction — an LLM call per page, priced per token, that reads the content and returns the schema you asked for. A well-built pipeline reserves that cost for pages that actually need it rather than paying it on every single page.

4. Human Verification

For anything shipping to a client or feeding a business decision, someone checking the output before delivery isn't optional overhead — it's the difference between "the data is probably right" and "the data is right." That's genuine labor cost, and it's the layer most flat-rate scraping tools skip entirely, leaving you to discover extraction errors after the fact.

Why Flat Per-Page Pricing Doesn't Reflect Reality

A lot of scraping services quote a flat rate per page or per 1,000 records, regardless of source. That works fine as an average across a large, mixed portfolio of easy sites — but it means an easy site is subsidizing a hard one inside the same rate card, and if you're only scraping the hard site, you're either overpaying against what it should cost, or the provider is quietly eating the loss and looking for ways to cut corners on your job.

The alternative is pricing measured from the actual crawl: which proxy tier the page needed, whether a browser had to run, how much extraction cost. That's a fairer number in both directions — you don't overpay for an easy site, and a provider doesn't lose money quietly on a hard one and let quality slip as a result. It's the model Foxpull uses: the free live demo runs on the same fetch path a paid job would, so the price it quotes is based on what your specific page actually took to crawl, not an industry-average guess.

Questions Worth Asking Any Scraping Vendor

• Does the price change based on how hard my specific target is to crawl? If every job is the same flat rate regardless of source, ask how that's possible given the cost spread above.
• Is there a human check before delivery, or is it fully automated? Neither answer is wrong, but you should know which one you're paying for.
• What happens if a page changes and extraction starts failing silently? Ongoing/recurring jobs need an answer here, not just a one-time delivery guarantee.
• Can I see the actual fetched page and screenshot, not just the extracted rows? Being able to audit a raw source against the output builds trust fast; if a vendor can't show you that, be skeptical of the numbers.