A new Coddy Developer Survey found that four in five developers, 80%, say their use of AI has felt more like a dependence than an advantage.
A new Coddy Developer Survey found that four in five developers, 80%, say their use of AI has felt more like a dependence than an advantage.
I use Claude all the time at work. It is good. But it makes massive mistakes, it misses tests, it confidently says something it screwed up will be fixed by something that certainly isn’t the right way to fix the problem.
I recently explained to a colleague: if you can use 1 AIU (arbitrary quantity of ai usage) and get 10% productivity bump, that doesn’t mean 5 AIU gets you 50% and 10 doubles your speed. The AI will do and say promising things, make you believe it’s on the verge of solving the problems, but it never quite arrives. There’s always one more problem and if you’re very lucky the AI will find it itself, but most likely it will be found when you pass it on to another person and it’s completely useless.
Let me put it this way: in addition to development, I use Claude to help with production support issues. It wrote some scripts I didn’t have time to and it pulls logs and data from multiple systems — honestly it works great and has saved me so much time. But I’m constantly in meetings and so I set Claude to investigate an incident so I can focus on my meeting and return when I have time, and it gets RCA wrong well over 50% of the time.
If it is so bad at RCA, how do you imagine it is fixing the bugs in the code it finds? Badly. It misunderstands the cause of problems, and so it fixes the wrong things until it has cobbled together the creakiest of code that passes the test. In fact I think AI is far worse at fixing code than it is at writing it in the first place.
I’m not anti AI. I’m trying to find ways to make it effective. And my teams are seeing 20-30% productivity gains - I think because they are skeptical about AI rather than trusting. But it has to be used appropriately, and everywhere I look, even within my own company, people are trying to do too much with it and creating huge problems I have to sort through.
It is increasing activity, not necessarily productivity. I see the issue that LLMs do not recognize and cannot communicate when they are missing necessary information to push the inference in the direction that is needed.
It’s a bit like when you go into a diner and order a chicken sandwich, and the cook has read every book containing a chicken sandwich recipe in the world, no common sense, no budget / time limits on getting ingredients and an attention deficit disorder. You will (most of the time) get something that resembles a chicken sandwich, but it will probably never be to your taste. A normal cook has a lot of real world context to make sure that the sandwich is tasty - he knows what area he works in and what a chicken sandwich in this area normally contains, he knows how different ingredients interact with each other, and he is limited to what is available in his kitchen. If he knows you personally, he will be able to fine-tune the sandwich to your taste.
LLM output does not have ANY of that context. Even things we do not actively think about - because it’s just the way how things are - are simply not available to an LLM. Only way to make sure our hypothetical ADD-ridden cook with eidetic memory makes a chicken sandwich worth a shit is to provide him with as much clear, non-ambiguous information as possible; and since the connection to the real world is missing, there can never be enough information - or rather, we are in the dark about what pieces of information would be needed to shift the probability towards the thing we actually want.
I’m on your side of the argument here dude, but you really need to fucking chill with the ableism
Please explain how this is ableism in anyway. This nonexistant “person” does not resemble any human alive on this planet at all. Stop getting your panties in a bunch. I am very careful not using language that is used to insult or denigrate people. ADD isnt in the class of “r-word” language at all, and i actually fullfill 9/9 of the medical criteria for diagnosing ADD, so it was the first thing i grabbed at because it is so relateable to my own life. (and my chicken sandwiches are quite tasty too.)
Great reply. I agree with a lot what you said, but I want to make two points:
First, based on a human breakdown and pointing of stories as if AI weren’t being used, using the methodology we have used for years, we are seeing real delivery velocity increases. Now measurement is tricky; story points are arbitrary for a team. Some of those gains are in stuff like “wrote AI tool” or “created AI documentation” which means we’re being more productive at things we never used to need to do. Still, when you factor all of that in as best you can, we are seeing a measurable 20% increase. (The analysis does point at 30%, but I see holes in how those numbers are built.)
Second:
I don’t think we are completely in the dark. I think we are working on figuring out how to improve the context. The problem I see with the current approach is everyone is doing it independently, their methodology is poor because we can’t afford the tokens for exhaustive tests that are invalidated with the next model, and the results are entirely subjective (and frequently written by AI to sound like absolute success).
So I come up with something that is amazing for my teams, and you try it, but because you’re in a different domain my technique isn’t quite right, and because our lives have provided us with different context, you don’t implement it quite the way I would have, and so your results differ and is it the domain, or the implementation, or did I just get some lucky rolls?
One of the things I do in my spare time because I’ve been fascinated by AI for years, is I use AI to write stories. I come up with a premise, maybe do some world building, write an opening, and just let AI go and see what happens, steering it when necessary.
Now there are two ways to steer: you can “reroll” or you can just edit the text directly. And what I’ve seen over years of doing this is the more human text you inject in, the better the results are. Even if you reroll to steer in the direction you want, the AI will eventually start producing utter gibberish. The quality starts to go down and then falls off a cliff.
Human input slows that or even stops it altogether. Human input is an essential element in getting good results out of AI. Because bullshit built upon bullshit is multiplicative. An AI that is 99% good falls to 50% good very quickly.
My point is that humans need to interact with the AI to provide that context you correctly point out the AI can never have, and it has to be frequent because you get to a point of saturation where a human is overwhelmed by the volume of text.
“Asking the AI to write stories for you” does not equal “using AI to write stories”
I use it lots of ways. My posts are long enough without getting into asides like that, don’t you think?
This is a user issue. If it missed stuff, you didnt describe what you want clearly enough. This right here is exactly why im not worried about it taking over
It’s probably a legacy code quality issue too. If your code is a mess, the only fix it can find is a kludgy one. The first thing I had it do, once it was good enough to do real work, was a legacy refactor to get architecture up to best practice standards. Heck, even OpenAI’s Harness Engineering specs say that this is essential to do first. I suspect that many of the people saying it comes up with shitty solutions are the same ones that always complain about over-engineering.
You’ll also get subpar results if you just have it start coding without fully architecting its solution first. Just like a real developer.
It can’t “take over”, it’s has no sentience or actual intelligence
I’m not worried about it taking over. Describe exactly what you want, clearly is exactly the job of a programmer and why we aren’t going to be replaced by middle management wielding an army of AI.
The difference is code largely can’t be ambiguous where human language is subject to interpretation. So the question is what it takes to describe what is required in a way that the AI cannot misinterpret ambiguity, and so that agentic AI doesn’t have ambiguity decided confidently wrong by the orchestration layer.
This is not strictly a user problem. AI will confidently make bad assumptions about ambiguity and propagate those to sub agents and you wind up with garbage. I’m good at this — would never claim to be the best and maybe you are better, but I’m good at it — and it still causes me grief. One criticism I have of the whole endeavor is that proficiency seems stubbornly difficult to transfer.