A scraping job that pulled twelve thousand product pages an hour on a single static address dropped to under four thousand the week a team switched to a rotating pool. Nothing in the code changed. The parsing was identical, the target list was identical, the server running the job was the same box in the same rack. The only new variable was the rotation layer, and it had quietly eaten two-thirds of the throughput without throwing a single error.
The gap between the requests you expected and the ones you actually completed
Most teams plan capacity from a clean arithmetic: requests per second times seconds equals coverage. Then the real numbers come in lower and stay lower. The gap is rarely a bug. It is the accumulated cost of every small delay that rotation introduces on the path between your worker and the site you are collecting from. Each individual cost looks trivial. Multiplied across millions of requests, they become the difference between finishing an overnight run and watching it spill into business hours.
Where each rotated request loses precious milliseconds
When an address changes, the shortcuts that made your last request fast usually reset. DNS lookups may be repeated. Routing has to be resolved through the proxy. The socket you would have reused is gone. A request that could have completed in 120 milliseconds on a warm path now spends 300 or 400 milliseconds standing everything back up. That extra couple hundred milliseconds is invisible on one request and catastrophic across a queue of half a million.
The handshake tax of standing up fresh connections over and over
The most expensive part of a cold request is the negotiation before any data moves. A new TCP connection and a fresh TLS handshake together involve several round trips before the first byte of your actual request leaves. Over a long-distance path, each round trip might cost 80 to 150 milliseconds. Per-request rotation, in its most naive form, pays this tax on every single call. You are not slow because the target is slow. You are slow because you keep introducing yourself.
Geographic distance between your pool and your targets
Rotation pools scatter your exits across regions, and physics does not negotiate. A request routed through an address on another continent adds real transit time in both directions, then adds it again for every handshake round trip. If your targets are concentrated in one region and your pool is not, you are paying a distance penalty on most of your traffic. Pinning rotation to exits near your targets, where the use case allows it, recovers time you cannot get back any other way.
Retries and backoffs that multiply under the hood
Rotation raises the odds that any given request lands on a slow, overloaded, or already-flagged exit. Those requests time out or return errors, and your retry logic dutifully sends them again, often with an exponential backoff that grows the wait each time. A 2 percent failure rate on a pool feels harmless until you notice each failure costs the full timeout plus the retry plus the pause between attempts. The hidden throughput loss is not the failures themselves but the seconds of dead air they insert into an otherwise full pipeline.
Connection pooling and keep-alive as the first speed recovery
The single highest-return fix is to stop paying the handshake tax on every request. Keep connections alive and reuse them for multiple requests through the same exit before rotating. Sticky sessions that hold an address for a short window let you amortize the expensive setup across several calls instead of one. Teams building on a stack like Scrapy Python can pair its built-in connection reuse with a rotation policy that switches on a sensible interval rather than blindly per request, and often reclaim most of the lost throughput without touching anything else. A Chicago analytics team doing exactly this doubled effective request rate in an afternoon.
Tuning concurrency against your Scrapy Python workers so the pipeline stays full
Once cold-start cost drops, concurrency becomes the lever. The goal is enough parallel requests in flight that slow ones never leave workers idle, but not so many that you overwhelm the pool or trip target defenses. Raise concurrency gradually and watch where completed-request rate stops climbing. That plateau is your ceiling. Set download timeouts tight enough that a stalled request gets abandoned quickly instead of hoarding a worker slot for thirty seconds.
Measuring real throughput so the fixes hold over time
Every one of these adjustments needs a number attached, because pools drift and targets change their behavior. Track completed requests per minute, not attempted. Track median and tail latency separately, since it is the slow tail that quietly drains your rate. Log retry counts and the share of time workers sit idle waiting on connections. When those metrics live on a dashboard you actually look at, a creeping slowdown surfaces as a trend instead of ambushing you as a missed deadline weeks later.