Web Scraping 101: From Raw HTML to Structured Data

We write a lot about pulling structured data out of documents. Web pages have the same problem: the information you actually want — a price, a spec sheet, a list of job postings, a table of exchange rates — is buried inside markup, scripts, ads, and ten years of accumulated div soup. Web scraping is just document data extraction where the document happens to be a live web page instead of a PDF.

This is a practical walkthrough of how scraping actually works today, why it breaks in ways beginners don't expect, and where the line is between "write your own script" and "just use a service."

What "Scraping" Actually Involves

Every scraping job is really four separate problems stacked on top of each other. Most tutorials only cover the first one, which is why so many scripts work in a demo and fail in production.

1. Fetching the Page

Getting the bytes. For a huge number of sites this is a single HTTP GET request. For a growing number of sites it isn't — see the JavaScript section below.

2. Parsing the Markup

Turning raw HTML into something you can query: a DOM tree you can walk with CSS selectors or XPath. Libraries like BeautifulSoup, lxml, and Cheerio all do this part well and it's rarely the hard part.

3. Locating the Data

Deciding which elements on the page correspond to the fields you want. This is the part that breaks every time a site redesigns, because a selector like div.product-card > span:nth-child(3) is a bet on markup that isn't guaranteed to stay stable.

4. Getting There Without Getting Blocked

Fetching at any real volume runs into rate limits, IP blocks, and bot-detection challenges. This is the part beginners hit last and underestimate most — a scraper that works perfectly for 50 requests can be completely dead by request 500.

Why Simple Scripts Break: JavaScript-Rendered Pages

A plain HTTP request gets you the HTML the server sent — nothing more. On a growing share of the web, that HTML is a nearly empty shell with a <div id="root"></div> and a bundle of JavaScript that fetches the real data and paints it into the page after load. View source on a modern React, Vue, or Next.js site and you'll often find none of the content you can see in the browser.

Fixing this means running an actual browser engine (Playwright, Puppeteer, or Selenium) headlessly, waiting for the page to finish rendering, and then reading the DOM. That's a heavier operation — more memory, more time per page, and its own detection surface, since headless browsers have fingerprints that differ from a real one out of the box.

A useful shortcut before reaching for a browser: check if the page has a hidden JSON API. Open your browser's network tab, reload the page, and look for an XHR/fetch request returning JSON. Sites frequently load their own content from a documented-or-not internal API — if you can call that directly, you skip rendering entirely and get cleaner data than scraping HTML ever would.

Structuring What You Extract

"Structured data" means every row has the same shape: the same columns, consistently typed, with missing fields represented consistently (empty string vs. null vs. omitted — pick one and stick to it). A few habits make the difference between a scraper that produces a usable spreadsheet and one that produces a mess someone has to clean up by hand:

• Define the schema before you write the scraper. Decide the exact columns you need and their types up front. Extracting "whatever's there" and reconciling it afterward takes longer than doing it right the first time.
• Normalize as you go. Strip currency symbols and thousands separators from prices, parse dates into ISO format, trim whitespace. Do it at extraction time, not in a separate cleanup pass.
• Validate before you store. A price field that comes back empty or a date that fails to parse is a sign the page layout changed under you — catch it immediately instead of finding out when someone opens the spreadsheet.
• Keep a raw copy. Store the fetched HTML (or at least the JSON payload) alongside the parsed row for a while. When a selector silently starts extracting the wrong thing, the raw copy is the only way to tell whether the site changed or your logic is broken.

A Minimal Working Example

For a simple, server-rendered page, a scraper can be genuinely small. This fetches a page and pulls out a list of items with a name and price:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/products", timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")

rows = []
for card in soup.select(".product-card"):
    name = card.select_one(".product-name")
    price = card.select_one(".product-price")
    if name and price:
        rows.append({
            "name": name.get_text(strip=True),
            "price": price.get_text(strip=True).replace("$", ""),
        })

print(rows)

That's step 1–3 in about fifteen lines. It's also the version that breaks the moment the site adds a login wall, switches to client-side rendering, or notices you're making a thousand requests a minute from one IP — which is exactly where step 4 starts to matter.

Respect the Rules While You're At It

Before scraping any site: check robots.txt for paths that are explicitly off-limits, read the terms of service if the data is commercially sensitive, keep your request rate reasonable (a real user doesn't load 50 pages a second), and don't scrape content that sits behind a login. None of this is optional if you want the data source to still be there next month.

When to Stop Writing Scrapers Yourself

Rolling your own scraper makes sense when it's a handful of pages, the site is server-rendered, and you're only running it occasionally. It stops making sense once you're maintaining selectors across dozens of sites, babysitting a proxy pool, or re-writing extraction logic every time a target site redesigns — at that point the maintenance cost is the whole project.

That's the gap tools like Foxpull are built for: point it at a page, it proposes the columns it can extract (with real example values from your page), runs the crawl, and hands back a spreadsheet — without you maintaining a selector map. Worth trying on a page you were about to write a scraper for anyway; the live demo is free and doesn't require signup.