It’s almost necessary at this point. At least some form of AI scraper prevention.
I had to take my public repos down a couple days ago, individuals and belligerents using botnets make blocking scrapers via normal means (user-agent/CIDR block) ineffective. So things like CloudFlare or Anubis are becoming necessary.
Ironic that they have their repos hiding behind Cloudflare, then.
well it’s anti-LLM crawlers
It’s almost necessary at this point. At least some form of AI scraper prevention.
I had to take my public repos down a couple days ago, individuals and belligerents using botnets make blocking scrapers via normal means (user-agent/CIDR block) ineffective. So things like CloudFlare or Anubis are becoming necessary.