Per-network limits
Scrapers often spread their requests over many addresses in the same network, so that each address stays under the per-address limits. Per-network limits count every address in a /24 (IPv4) or /48 (IPv6) network together. They're set per cluster: on Clusters, click Edit, and find Per-network limits under Traffic limits.
The two limits
| Limit | What it counts | Over the limit | Default |
|---|---|---|---|
| Website protection limit | Requests a second per site from one network, on domains with Website protection | Visitors get a Captcha Protection check, or a "too many requests" page when Captcha Protection isn't available. A visitor who passes the check isn't held back by this limit. | 150 a second, burst 1,500 |
| Network protection limit | New connections a second to the sentry from one network, for every domain | New connections are dropped. A network is never temporarily banned, so other visitors on it can still connect once the rate drops. | 100 a second, burst 500 |
Each limit is Off, Log mode or On. Log mode counts networks that go over the limit and logs them, without limiting anyone. The Network protection limit is generous by default, because connections can't be checked.
Clusters that existed before release 1.1 start with the Website protection limit in log mode and the Network protection limit off. New clusters start with both on.
Never counted
- The Website protection limit never counts verified Identified traffic, your never-check addresses, your own sentries, origin servers and cluster VIP, your SSH allow-list and certificate checks.
- The Network protection limit never counts your own sentries, origin servers and cluster VIP. Verified search engines keep their own crawler limit.
In your logs
A dropped connection shows on Logs as Network limit. In log mode, a network over the Website protection limit shows as Network over the limit (log mode) (net-limit). Visitors who got the check show as Browser check: too many requests. See Log reasons.
What you can do: if a busy office, school or mobile network is being held back, raise the limit or burst, or switch the limit to log mode.
To slow down scrapers on particular addresses, such as search or filter links, use URL rules, which can also count per network.
Per-network limits need sentry release 1.1 or later: older sentries ignore them.