Sorta like how people complain about bots scraping Lemmy, even though federation already exists as a standardized protocol for distributing data. Like any scraper who wanted to efficiently scrape Lemmy would just spin up their own instance and let federation do the scraping for them. It would even have the added benefit that they could set their server to ignore delete requests, so deleted posts/comments wouldn’t get automatically removed from their server. And then they could scrape as much as they wanted without impacting anyone else.
But they don’t want to do that, because it would require the smallest modicum of forethought. They don’t care that scrapers are trashing the Internet and causing massive bandwidth issues for hosters. They just want the data, and they want it now. All of those “bots are flooding my server and eating all my bandwidth, so legitimate users can’t actually access the site” complaints are for other people.
I bet that if someone went to The Internet Archive, they could pay them to get timestamped snapshots of professionally-spidered stuff at zero load to the websites. I’m sure that it’d cost something for all the hard drives and probably something for labor, but so does spidering the whole Internet yourself. The people running the bots clearly have the funds available to run them at massive scale.
Sorta like how people complain about bots scraping Lemmy, even though federation already exists as a standardized protocol for distributing data. Like any scraper who wanted to efficiently scrape Lemmy would just spin up their own instance and let federation do the scraping for them. It would even have the added benefit that they could set their server to ignore delete requests, so deleted posts/comments wouldn’t get automatically removed from their server. And then they could scrape as much as they wanted without impacting anyone else.
But they don’t want to do that, because it would require the smallest modicum of forethought. They don’t care that scrapers are trashing the Internet and causing massive bandwidth issues for hosters. They just want the data, and they want it now. All of those “bots are flooding my server and eating all my bandwidth, so legitimate users can’t actually access the site” complaints are for other people.
I bet that if someone went to The Internet Archive, they could pay them to get timestamped snapshots of professionally-spidered stuff at zero load to the websites. I’m sure that it’d cost something for all the hard drives and probably something for labor, but so does spidering the whole Internet yourself. The people running the bots clearly have the funds available to run them at massive scale.