Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
If they have 10k URLs and in parallel find out all links via different IPs, they get new links.
When they do that again with the new links, there is no connection between visits, because the IPs differ. It looks like someone else is requesting what is behind the link.