Most scraping projects start the same way. The script works on your laptop, you point it at a few thousand pages, and somewhere around page 300 the website starts sending back error pages, puzzles to solve, or nothing at all.
That isn’t bad luck. The site noticed that one address was asking for far more pages than a person ever would. This guide covers why that happens, how proxies help (and where they don’t), and how to plan a job so you know roughly what it will cost before you start.
Why scrapers get blocked
A website can’t see who you are. It sees an IP address, the details your software sends with each request, and the pattern of pages you ask for. Blocks usually come from one of these:
- Too many requests from one address. A person reads a handful of pages a minute. A script can ask for hundreds.
- The address belongs to a hosting company. Ordinary shoppers don’t browse from cloud servers, so many sites treat that traffic with suspicion from the first request.
- The software looks like software. Missing browser details, an outdated browser name, or the exact same settings on every request.
- The behaviour isn’t human. Visiting every product in alphabetical order, at the same speed, around the clock.
Proxies solve the first two. They spread your requests across many addresses and, with residential or mobile IPs, make each request come from an ordinary home or phone connection. They don’t fix the last two. A proxy changes where your requests come from, not how they look. If your scraper announces itself as a script, a new IP only buys you a few more pages.
Which proxy type to use
There’s no single best proxy for scraping. The right choice depends on how hard the site works to keep automated visitors out, so it pays to test before you commit to a large job.
- Datacenter proxies ($0.89/GB) work well on public directories, government and open data sites, news and blogs, smaller shops and your own websites. They’re fast and cheap, but strict sites spot them easily.
- Residential proxies ($1.69/GB) are the default for large retailers, marketplaces, travel and booking sites, real estate listings and search engines. Each request comes from a real household connection.
- Mobile proxies ($3.50/GB) are for the hardest targets: social networks, mobile-only pages and sites that block even residential traffic. Sites rarely block carrier IPs, because many real phone users share them.
A simple way to decide: run a few hundred pages through datacenter proxies first. If almost every page comes back with real content, stay there and save the money. If you see a lot of block pages, switch the same job to residential. All three types draw from one balance, so switching doesn’t mean buying a new plan.
Rotating or sticky sessions
Rotating sessions give you a different IP on every request. Use them when each page stands on its own: product pages, listings, articles, profile pages. Spreading the work across many addresses is what keeps any single one from standing out.
Sticky sessions keep the same IP for a while, up to two hours on our network. Use them when the site expects one visitor to move through several steps, for example:
- running a search, then opening the results
- paging through results where the site remembers where you are
- choosing a delivery location before prices appear
- anything behind a sign-in you’re allowed to use
Switching IP halfway through one of these flows looks odd to the site, a bit like a shopper teleporting to another city between pages. Keep it sticky until the flow is done, then move on to a fresh IP.
Pick the right location
Many sites change what they show based on where the visitor is. Prices, currency, language, stock and even which products exist can all be different. Some sites send foreign visitors to another version of the site, or don’t serve them at all.
Match the proxy location to the data you want. If you’re collecting German prices, use German IPs. If the results depend on a region or city, such as local listings or delivery availability, use residential proxies with state or city targeting. Collecting from the wrong country is one of the most common reasons scraped data “looks wrong” even when nothing was blocked.
Go slower than you think
Having thousands of IPs available isn’t permission to hit a site as hard as possible. It’s bad for the site, and it’s bad for your results, because a site under pressure tightens its defences for everyone.
- Add short, slightly random pauses between requests instead of a fixed rhythm.
- Limit how many pages you load from the same site at the same time.
- When a site starts sending “too many requests” or error pages, slow down. Adding more IPs at that point usually makes things worse.
- Retry a failed page a couple of times with a fresh IP, then give up and log it.
- Run big jobs at the site’s quiet hours for its local time zone.
A scraper that runs slowly and finishes is worth more than a fast one that gets shut out halfway.
How much bandwidth you need
Proxies are billed by the gigabyte, so the size of each page matters more than the number of IPs. A typical product or listing page is somewhere between 100 KB and 500 KB if you only download the page itself and skip images and video. Loading the same page in a full browser, with every picture, font and tracking script, can easily be ten times that.
Here’s a quick estimate. Say you want 50,000 product pages at about 250 KB each. That’s roughly 12 GB, or about $20.28 on residential proxies and about $10.68 on datacenter proxies. Add 10 to 20% for retries.
A few ways to use less:
- If you use a browser, turn off images, video and fonts unless you need them.
- Many sites load their data separately from the page layout. Collecting that data directly is often a fraction of the size of the full page.
- Don’t re-download pages that haven’t changed since your last run.
Unused bandwidth on our plans never expires, so it’s fine to buy a little more than your estimate and keep the rest for next time.
Stay on the right side of the rules
Collecting publicly available information is a normal part of research, pricing and product work. A few rules keep it that way:
- Stick to pages anyone can see without signing in, unless you have permission.
- Be careful with personal data. Names, emails and profile details are covered by privacy laws such as GDPR, even when they’re public.
- Read the site’s robots.txt file and terms. They tell you what the owner is and isn’t happy with.
- Don’t republish content you don’t have the rights to.
- Never load a site so heavily that it slows down for real visitors.
Our Acceptable Use Policy sets out what our network can and can’t be used for. This isn’t legal advice. If your project touches personal data or a grey area, it’s worth a quick conversation with a lawyer.
Checklist before your first big run
- Test 100 to 500 pages on the cheapest proxy type that might work.
- Check a sample of the saved pages by eye. Block pages often look like success to a script.
- Set the proxy country to match the data you need.
- Use rotating IPs for single pages and sticky IPs for multi-step flows.
- Set a sensible pace and a retry limit.
- Estimate bandwidth from your test pages and top up with a little to spare.
Questions and answers
- Is web scraping legal?
- In many countries, collecting publicly available information is lawful, but it depends on what you collect and how. Personal data, copyrighted content and content behind a sign-in all come with extra rules. Check the site's terms, avoid personal data unless you have a lawful reason to process it, and get legal advice for anything sensitive.
- How many proxies do I need for scraping?
- With rotating proxies you don't buy IPs one by one. You pay for the bandwidth you use, and each request can come from a different IP in the pool. What you need to plan is how many gigabytes the job will use, not how many addresses.
- Why am I still blocked when using proxies?
- A proxy changes where your requests come from, not how they look. If your scraper sends unusual browser details, makes requests at a perfectly even pace, or skips the steps a real visitor would take, a site can still recognise it. Slow down, use realistic browser settings, and try residential or mobile IPs for strict sites.
- Should I use datacenter or residential proxies for scraping?
- Try datacenter proxies first, because they are the cheapest. If most of your test pages come back with real content, stick with them. If you see many block pages or puzzles, move the same job to residential proxies, which come from real household connections and are much harder to block.