What these articles never say is how many hallucinated bugs the LLM found that either weren’t real or were actually exploitable. The LLM didn’t find these with any confidence it highlighted areas of interest that actual security researchers then needed to investigate and confirm or rule out.
What these articles never say is how many hallucinated bugs the LLM found that either weren’t real or were actually exploitable.
It literally wouldn’t matter if it did.
The fact that it found exploitable bugs means that these bugs need to be addressed. To be clear, I care much more about the security flaws and fixing them than how they were discovered.
In the article it says the ffmpeg vulns were found by an “autonomous” agent and that it produced a proof-of-concept for each. So what do you base your claims on? They seem quite contrary to that.
Even if there was still a lot of human work involved, it seems that the LLM-Agents can help a lot with security research, as the number of (real) zero-days that are beeing found recently (with the help of AI) seems to spike (telling from what I read, e.g. here on Lemmy, or the number of security updates for my distro).
It’s states they were produced which I’m taking to mean found and it’s ambigously worded so it’s unclear if the article is actually claiming it generated PoC for all of them. It doesn’t say how many if any hallucinated results were produced or how much effort was needed to fully confirm, basically down played the human involvement.
It’s great that so many bugs are being found and squashed but it’s implied LLMs are doing all the work when what they are actually doing is assisting as a tool.
I agree that the wording is a bit ambiguous, I interpreted it the way it seems more natural to me. In the post by the researcher(s) themselves, it says in the tldr paragraph that the “agent produces concrete, reproducible PoC inputs to confirm its findings” but also that they (probably humans) “explored the exploitability of the issues and developed a PoC demonstrating a RCE exploit primitive”. Apparently it finds the vulnerabilities very concretely but humans were involved for the full-blown exploit. It also doesn’t say much about the number of false-positives.
I’m not in the business, so I can’t tell how much of the work such agents are actually saving. Since the articles don’t say much about the amount of human involvement, the imagination conveyed by them probably depends strongly on the (knowledge of the) reader. But in my opinion it is a bit of stretch to say this is downplaying it. It should be noted though, that the article probably sources its information from a post by the company selling that AI.
With that information, the “without any confidence” and “area of interest” parts of your previous post still seem misleading.
In my experience, when given real world data to run on, they don’t hallucinate that often. Its when you ask it to regurgitate stored info that its off by a wild amount sometimes. Fixing or comparing code is like AI 101, unlike code generation where it may be incorrect.
I strongly disagree. Every response longer than one line of code or longer than 1-2 sentences on a non-trivial task, has at least inaccuracies or mistakes. When working with something AI has created, I had to go in and fix it manually in 100% of those cases. That’s why I limit the use of AI to only type-ahead suggestions, as I can easily verify them and don’t waste more time than creating the result manually.
I’m frankly quite annoyed by the amount and extend of anti-AI hate in this community. It almost seems like this is a pure anti-AI rage community. The capabilities and the utility of LLMs are basically always denied or downplayed as much as possible without running into obvious contradictions.
It would be so nice to have some differentiated and insightful discussions here, about what can or cannot be done with AI, positive and negative impacts it has, possible new use cases, how AI should or should not be used, how the overall benefits of AI can be maximized and the overall negative effects minimized, what the world with AI could be like in the future, …
I was trying to have some insightful discussion on the actual capability of LLM which is difficult when the involvement of the human element is played down amd the role of the LLM is played up to feed the hype machine. It’s hard to acknowledge the real capabilities and weaknesses when the capabilities are always over reported and the weaknesses down played or denied.
It’s great that so many bugs are getting discovered but as I say there is no reporting on what effort was needed to sift and review the LLM output or how functional or understandable any PoC were… The article doesn’t directly even state the PoC were directly produced by the LLM and reads very ambigously.
I think some of that is because the reporting is focused on the new stuff, that was previously not possible. That human work is involved and some of the weaknesses are not really new. But also because the information in this case comes from a company that wants to sell their AI. I agree that the reporting is probably biased and not really sharp and therefore limited in usefulness.
Also, my (second) comment was not specifically about your comment but generally about the “vibe” of this community
what the world with AI could be like im the future
Imagine this trend line increasing a bit more rapidly https://en.wikipedia.org/wiki/Carbon_dioxide_in_the_atmosphere_of_Earth
CO2 emissions are a huge problem, of course. But it is not specific to AI. Data centers are starting to become a significant factor of energy consumption but I think it will stay very manageable compared to other consumers and given the utility it provides. And since data centers luckily natively require electricity, it is much easier, compared to e.g. transportation, to switch them to renewables. And renewables are very often already the cheapest source of energy anyways. So I think AI is just another thing that humans do that requires energy, and it comes with the same tradeoffs (the utility vs. the cost of sourcing that energy). So in my opinion we should mainly focus on accelerating the transition to green energy.
Here’s a good overview about AI carbon emissions I just found: https://www.carbonbrief.org/ai-five-charts-that-put-data-centre-energy-use-and-emissions-into-context/
What the fuck is a zero day in the context of ffmpeg?
Its not like its a system service that you can get ingress through…
“AI found 21 bugs in massive video project” sounds like junior developer shit hungry to get some shit on their resume.
Even if it wasn’t AI slop, this wouldn’t be impressive.
if you upload any image on lemmy, it goes through ffmpeg to be converted into a webp file.
now, you can feed arbitrary input to that ffmpeg process
I never thought of doing that with ffmpeg. Why ffmpeg, instead of imagmagick?
ffmpeg is on any system, has a consistent user interface for many different conversions
It just so happens that many video codecs are based on image formats, so ffmpeg already has a lot of the complex machinery to do so available to also implement these image formats - internally it can just handle it as a single frame of video with specialized formats for that.
Imagemagick (and other tools) also work, but why use multiple pieces of software if what you already have is adequate? ImageMagick is also software, and can also have vurnabilities.
My understanding is that ffmpeg is the bedrock that all video streaming services use. I’m suspicious it’s a bigger deal than you think
Don’t forget various conversion services, which includes photo cloud backups.
FFMPEG in the command line generally has permission to access the entire non-sudo filesystem and delete files.
Yes but why are we allowing user input to be fed to an executable in that environment?
This is the environment that almost all user software is executed.
Its not like its a system service that you can get ingress through…
With a competently crafted payload, you could perhaps get in via someone’s transcoding pipeline.
Does nobody isolate ffmpeg and friends from their application?
I can’t imagine you’d have much fun breaking into a container that terminates the moment the original ffmpeg stops, or over-runs its max execution time…
Container escapes do exist, and they have shared kernel with the host
If you’re running rootless containers, it’s less of a concern. I’m trying to move all of my public containers to podman for this reason
Sure, you’d need a second exploit to escalate from there.
ffmpeg is expected to run for extended periods of time, given its use in transcoding.
From the article
Most are heap or stack overflows in parsers and demuxers, spanning components from the TS demuxer to the VP9 decoder. depthfirst says some already carry CVE identifiers; its writeup lists nine, CVE-2026-39210 through CVE-2026-39218, and notes the rest are fixed but not yet numbered. It also published a PoC.
I imagine there are many web services around the world which use ffmpeg to handle user submitted content.
Tell me you know nothing about the intricacies of media playback (especially hardware accelerated), without telling me…
Discovering bugs is not “AI slop”.
That term refers to something the AI made. This is just product testing, where a real human then fixes it as intended.
Many moderate to large open source projects are effectively being ddosed by vibe coders submitting hallucinated and non-issue bug reports solely because claude or copilot said so. Those reports are absolutely slop, no different from anything else generative ai platforms shit out.
But sure, go off on how this particular kind of output from an LLM is somehow different from rest.
Edit: fixed a typo
