In my experience, the LLM usually proposes a solution, but it’s a crapshoot whether it works well. And plenty of bugfixes go well beyond a few lines, especially in older code.
I use this shit on a daily basis at work. They work well, as long as you’re choosing the right models, making the right engineering decisions, and have a good set of baseline rules.
Do you have a guide on which baseline rules are good? I use Claude Opus 5.5 at work and while it’s definitely more helpful than not having it, it also does some incredibly baffling things that a human would realize immediately are unproductive.
“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.
“Cleaning up” seems a little generous. It’s finding them, but someone still has to patch them.
Most vulnerability patches are a few lines of code. Finding it is the hard part.
Even then, the AI session that found it probably already has a solution to fix it.
In my experience, the LLM usually proposes a solution, but it’s a crapshoot whether it works well. And plenty of bugfixes go well beyond a few lines, especially in older code.
I use this shit on a daily basis at work. They work well, as long as you’re choosing the right models, making the right engineering decisions, and have a good set of baseline rules.
Do you have a guide on which baseline rules are good? I use Claude Opus 5.5 at work and while it’s definitely more helpful than not having it, it also does some incredibly baffling things that a human would realize immediately are unproductive.
“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.