Skip to main content

URL rules

Scrapers often hammer the same kinds of address: search results, sorting and filter links, and add-to-cart links. URL rules slow them down, check them or block them on just those addresses, so the rest of the site works as usual. They apply on domains with Website protection.

Where to set them​

  • Settings › URL rules, on the All clusters tab: rules for every domain with Website protection in your account, now and in future. Up to 20 rules.
  • Settings › URL rules, on a cluster's tab: rules for domains on that cluster only, on top of the rules for all clusters. Up to 20 rules.
  • A domain's URL rules tab: rules for that domain only, up to 10. The tab also lists every rule that applies to the domain under Rules that apply, with where each one was set.
  • The owner area, if you let domain owners add URL rules (see Domain owners). Up to 10 per domain.

What a rule matches​

FieldWhat it does
AddressA path starting with /, such as /search. End it with * to match every address that starts with it, such as /shop/*. Leave it empty for any address.
RequestsPage views (GET and HEAD) or Every request, including form posts
Query stringUp to 5 conditions, and every one must match: a parameter is present, equals a value, starts with a value, is a number, a parameter name starts with some text, or any query string

A rule needs an address or at least one query string condition.

Actions​

ActionWhat happens
Rate limitCounts matching requests per visitor address or per network (/24 for IPv4, /48 for IPv6), at 6, 30 or 60 a minute, or 2 or 5 a second. Over the limit, visitors get a Captcha Protection check, or a "too many requests" page when Captcha Protection isn't available.
Captcha Protection checkAsks the visitor to complete a quick browser check before the page opens.
BlockAnswers matching requests with the block page.
Temporary banBlocks matching requests. An address that makes 5 of them within a minute is also temporarily banned from every site on that sentry, for the cluster's Temp ban (minutes).

When a request matches more than one rule, the strongest action applies: temporary ban, then block, then Captcha Protection check, then rate limit.

Log mode first​

Every new rule starts in log mode: matching requests are counted and logged, but not limited, checked or blocked. Click Enforce when you're happy with what's being logged, or Back to log mode to go back. While a domain's Website protection rules are in log mode, its URL rules only log too.

Ready-made rules​

Under Add a ready-made rule you can add rules for WordPress search, WordPress REST API listings, WordPress comment reply links, WooCommerce sorting and filter links, WooCommerce add-to-cart links, the WooCommerce Store API, Magento search and PrestaShop search. They start in log mode, and you can edit them after adding.

Never matched​

Your never-check addresses, your own sentries, origin servers and cluster VIP, your SSH allow-list, certificate checks and payment gateway notifications are never matched. Verified Identified traffic, such as search engines, is also never matched unless you untick Never apply to verified Identified traffic, such as search engines on the rule.

Domain owners​

On Owner access, Add URL rules on their domain (domains with Website protection only) lets domain owners add their own rules on the owner area's URL rules tab. It's on by default. Your rules show there under Set by your host, and owners can't change them.

Temporary bans are yours to control. Owners can only choose Temporary ban when you tick Use temporary bans in their URL rules, which is off by default, because a ban blocks the address from every site on the sentry. Without it, they see Your host doesn't allow temporary bans. Choose another action.

If you take a permission away, the owner's rules (or just their ban rules) are paused and kept, and come back if you allow it again. On the domain's URL rules tab, owner rules are marked Domain owner, and paused ones say Paused: owners can't change URL rules or Paused: owners can't use temporary bans.

In your logs​

Matching requests show on Logs as URL rule (url-rule) or URL rule (counts towards a temporary ban) (url-ban) with the rule's number, or as Browser check: URL rule and Browser check: too many requests. Blocked visitors see the block page with Reason: this kind of request is not allowed on this site. Bans show on the Temporary bans tab with the rule as their reason, and can be lifted early there. See Log reasons.

URL rules need sentry release 1.1 or later: older sentries ignore them.