AI crawlers DDoS self-hosted sites, exhausting server resources with no simple defense

Summary

Self-hosters report that AI crawlers flood their servers so aggressively that nginx alone consumes half of an 8-core Xeon just handling HTTP requests, even when the actual application (Forgejo) is stopped. This is a recurring, real-time 'DDoS by crawler' situation where operators have no easy, reliable way to stop the flood without risking collateral damage to legitimate traffic. Existing bot controls (robots.txt, user-agent blocking) are trivially ignored by AI scrapers, WAF/rate-limit rules are fragile and manual, and blocking by IP or UA risks cutting off verified search engines and damaging SEO. Meanwhile, operators are being forced to absorb bandwidth, CPU and infrastructure costs while crawlers take their content for free. The problem is unsolved and highly painful: site owners are 'fucking tired' of fighting an endless stream of bots, and there is no simple, deployable layer that distinguishes verified search bots from predatory AI crawlers, throttles or charges the latter, and leaves humans and SEO untouched. A turnkey, stack-native pay-per-crawl / bot-auth layer that sheds crawler load while monetizing it would be immediately valuable to indie publishers, self-hosters and backend teams.