Getting blocked isn't usually about being "caught" doing something sneaky — most of the time it's a script behaving nothing like a human visitor, and a site's defenses noticing. The fixes are mostly about being a good citizen and being predictable in the right ways, not about outsmarting anyone. Here's what actually moves the needle, roughly in the order you should try them.
1. Read robots.txt First
It's one request: https://example.com/robots.txt. It tells you which paths the site doesn't want automated traffic touching, and often lists a preferred crawl delay. Respecting it isn't just polite — it's the fastest way to avoid the paths most likely to have aggressive bot defenses in the first place, since those are usually the same paths.
2. Slow Down
The single biggest cause of blocks is request rate, not headers or IPs. A human clicking through a site can't generate ten requests a second. Add a delay between requests (even a randomized 1–3 seconds), and most rate-limit-based blocking never triggers at all. If a site's docs or robots.txt state a crawl delay, use it.
3. Send Realistic Headers
A request with no User-Agent, or the default one your HTTP library sends (python-requests/2.31.0), is an immediate tell. Set a real browser User-Agent and include the headers a browser actually sends — Accept, Accept-Language, Referer where appropriate. This isn't about deception so much as not standing out for no reason; an empty or clearly-a-script header set is the laziest signal a bot filter checks first.
4. Reuse Sessions and Cookies
Opening a fresh connection for every single request looks nothing like a browser, which reuses a session across a whole visit. Use a persistent session (requests.Session() or equivalent) so cookies and connection state carry over naturally between requests to the same site.
5. Cache Aggressively
The cheapest way to avoid a block is to not make the request at all. If you're re-checking the same page repeatedly, cache the response and respect any Cache-Control or ETag headers the site sends. This also happens to be the best way to keep your own costs down — proxy and browser time is the most expensive part of any scraping pipeline, so re-fetching pages you already have is pure waste.
6. Rotate IPs Only When You Actually Need To
If you're making a handful of requests, your own IP is fine. Once you're crawling at real volume, a single IP will get rate-limited or blocked regardless of how polite your headers are, and that's where a proxy pool comes in — rotating requests across multiple IPs so no single one accumulates enough traffic to look automated. Datacenter proxies are cheap but easier for sites to flag; residential and mobile IPs cost more but blend in with normal traffic. Match the tier to the site: don't reach for expensive residential proxies against a site that's never blocked a datacenter IP in its life.
7. Only Reach for a Headless Browser When You Need To
Running a full browser is 10–50x heavier than a plain HTTP request and has its own fingerprint to manage (WebGL, canvas, navigator properties all differ from a real browser by default in naive setups). Reserve it for pages that genuinely require JavaScript rendering. Checking whether a plain fetch already contains the data you need — before assuming you need a browser — saves both money and detection risk.
8. Handle Failure Gracefully, Don't Hammer
When a request comes back blocked or rate-limited, the worst response is to immediately retry. Back off exponentially (wait longer after each consecutive failure), and if a site keeps refusing after several attempts, stop and revisit your approach rather than turning a soft rate limit into a full IP ban.
9. Know When to Stop Fighting It Yourself
Some sites run genuinely sophisticated bot defenses — JS challenges, device fingerprinting, CAPTCHAs that escalate based on behavior. At that point you're not writing a scraper anymore, you're maintaining an arms-race pipeline: proxy tiers, browser fingerprint management, CAPTCHA solving, and constant tuning as the defenses change. That's a full-time job on its own for a handful of hard sites.
The Honest Tradeoff
Every tip above reduces how often you get blocked. None of them make hard sites easy forever — anti-bot systems update too. Services like Foxpull exist specifically to absorb that maintenance: pages are fetched on an escalation ladder that only spends more (a better proxy tier, then a real browser) when the cheaper option actually fails, so you're not paying browser-grade cost for a page a plain request would've handled fine.
The Line You Shouldn't Cross
None of the above is about evading detection to access something you're not meant to have. Public pages, reasonable rates, honoring robots.txt, and no login-walled content — that combination keeps scraping in the "normal automated traffic" category almost every site tolerates. Trying to defeat a site's explicit access controls is a different activity with different legal exposure, and no blog post tip belongs in that category.
Skip the Maintenance
Foxpull runs the escalation ladder for you and only pays for what a page actually needs. Try the free live demo on any page.
Try Foxpull