New York is set to become the first state to impose guardrails on “stealth crawlers,” or unauthorized software that trawls news sources to scrape content.
For AI purposes the really useful part of a news site is the actual news - you know, the stuff that changes practically every minute - not the “structure” of the site.
These news sites aren’t being scraped for training data anymore but to provide near-realtime up to date information to the models.
Meaning e.g. Gemini can scan your news article, extract the useful information for the user, and deliver it to the user, without them ever going to your news site and providing the interaction that at the end of the day is converted to money - money your site needs to run.
What are you on about?
For AI purposes the really useful part of a news site is the actual news - you know, the stuff that changes practically every minute - not the “structure” of the site.
These news sites aren’t being scraped for training data anymore but to provide near-realtime up to date information to the models.
Meaning e.g. Gemini can scan your news article, extract the useful information for the user, and deliver it to the user, without them ever going to your news site and providing the interaction that at the end of the day is converted to money - money your site needs to run.