Blunt Anti-Bot Rate Limiting and Blocking Punishes Legitimate Humans and Verified Visitors, Not Just Crawlers

Summary

Sites fighting scrapers with crude rate limits and IP throttles are collateral-damaging real people and even locking out logged-in users. Documented case: GitLab's 60-requests-per-hour unauthenticated cap counts every document, XHR, fetch, manifest.json and GraphQL call, so a single merge-request page burns ~30 of the 60 — a normal visitor is IP-banned after two pages for an hour, and once rate-limited they cannot even reach the login page to authenticate. Site owners face an impossible trade-off: leave the door open to abusive AI crawlers and eat the bandwidth, or clamp down and destroy the browsing experience for humans and legitimate search engines. Existing bot controls are binary (allow/block) and IP- or user-agent-based, with no way to say 'humans free, verified search bots free, unverified automated agents pay a price.' The absence of a granular, per-request monetization/verification layer means every mitigation is a blunt instrument that bleeds SEO and human traffic. This is a severe, unresolved pain for developer-facing and content sites alike.