Reminder that this is made by Ben Zhao, the University of Chicago professor who illegally stole open source code for his last data poisoning scheme.
This is the same article I have read before, and it covers technology that doesn’t work in reality.
The inventors need to read up on a new fringe technology called “anti aliasing”, which quickly and easily removes the protection.
That isn’t neccesarily true, though for now there’s no way to tell since they’ve yet to release their code. If the timeline is anything like their last paper it will be out around a month after publication, which will be Nov 20th.
There have been similar papers for confusing image classification models, not sure how successful they’ve been IRL.
MIT Technology Review got an exclusive preview of the research
The article was published 3 days after the arxiv release. How is this an “exclusive preview”?
Successfully tricking existing models by a few crafted samples doesn’t seem like a significant achievement. Can someone highlight what exactly is interesting here? Anything that can’t be resolved by routine adjustments to loss/evaluation functions?
I don’t believe for a second that this works, and if it did, it would be trivial to get around.
It claims to “change the pixel values imperceptibly”. That just isn’t how these generative models work. These models are just looking at the colors, the same way a human would. If it’s imperceptible to a human, it won’t affect these models. They could subtly influence it, perhaps, but it would be nothing near the scale they claim.
My first thought was that they’re trying to cash in, but from what I can tell it seems to be free (for now, at least?). Is it for academic “cred”? Or do they somehow actually think this works?
It just seems to be such a direct appeal to non-tech-savvy people that I can’t help but question their motivations.
Luddites trying to smash machine looms
Whoever invents sich a thing simply underestimates the target groups’ ability to analyzes this and in a not-so-far future will filter such things out.
This is the best summary I could come up with:
A new tool lets artists add invisible changes to the pixels in their art before they upload it online so that if it’s scraped into an AI training set, it can cause the resulting model to break in chaotic and unpredictable ways.
The tool, called Nightshade, is intended as a way to fight back against AI companies that use artists’ work to train their models without the creator’s permission.
Using it to “poison” this training data could damage future iterations of image-generating AI models, such as DALL-E, Midjourney, and Stable Diffusion, by rendering some of their outputs useless—dogs become cats, cars become cows, and so forth.
Nightshade exploits a security vulnerability in generative AI models, one arising from the fact that they are trained on vast amounts of data—in this case, images that have been hoovered from the internet.
Gautam Kamath, an assistant professor at the University of Waterloo who researches data privacy and robustness in AI models and wasn’t involved in the study, says the work is “fantastic.”
Junfeng Yang, a computer science professor at Columbia University, who has studied the security of deep-learning systems and wasn’t involved in the work, says Nightshade could have a big impact if it makes AI companies respect artists’ rights more—for example, by being more willing to pay out royalties.
The original article contains 1,108 words, the summary contains 217 words. Saved 80%. I’m a bot and I’m open source!
thank you AI bot!
now poison it
Making AI companies pay royalties would cause them to charge for any use of their AI image generators, putting such technology beyond the reach of people who could not justify paying. The rest of us will miss out on the interesting images they might have created.
