Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools.

The online retail giant said there had been a “trend of incidents” in recent months, characterized by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT.

Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established.”

  • Simulation6@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    1
    ·
    41 minutes ago

    I always saw a code review like a dissertation defense. Why did you choose to implement the requirement in this way? Answers like ‘I found a post on Stackoverflow’ or ‘the AI told me to’ would only move the question back one step; why did you choose to accept this answer?
    I was a very unpopular reviewer.

  • BrianTheeBiscuiteer@lemmy.world
    link
    fedilink
    English
    arrow-up
    88
    arrow-down
    1
    ·
    8 hours ago

    Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added.

    So instead of getting a human to write it and AI peer reviewing it you want the most expensive per hour developers to look at stuff a human didn’t write and the other engineers can’t explain? Yeah, this is where the efficiency gains disappear.

    I read stuff from one of my Jr’s all the time and most of it is made with AI. I don’t understand most of it and neither does the Dev. He keeps saying how much he’s learned from AI but peer programming with him is the pits. I try to say stuff like, “Oops! Looks like we forgot the packages.” And then 10 secs of silence later, “So you can go to line 24 and type…”

    • RandallFlagg@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      3 hours ago

      Lol I would be your Jr, except instead of 10 seconds of silence it would be 10 seconds of me frantically clacking on the keyboard “add a block to this for these packages with proper syntax, I forgot to include it” to claude. Then I’d of course be all discombobulated and shit so I wouldn’t even bother to open code, I’d just ctrl-c about 100 lines somewhere around the general area of where I think the new code block should go, then ctrl-v the whole thing into the chat box because why not the company is paying out the dick for these tokens so might as well use them.

      And two weeks later half our website crashes which results in you having to go to a meeting where management tells you to keep a closer eye on me. Which is basically what you had been already doing before AI but now you get to babysit me and claude!

    • Joe@discuss.tchncs.de
      link
      fedilink
      English
      arrow-up
      3
      ·
      4 hours ago

      It’ll be temporary, a gut reaction to add more experienced engineers in the loop. These folks will try to codify and then push better checks/guardrails into CI/CD and tooling to save themselves time. Given how new this all is, it’s almost the blind leading the blind though.

      Amazon might also have some poor system boundaries, leading to non-critical systems/code impacting critical systems. Or they just let junior devs with their AI tools run wild on critical components without adequate guardrails… also likely. :-P

    • mrgoosmoos@lemmy.ca
      link
      fedilink
      English
      arrow-up
      33
      ·
      8 hours ago

      I am not a developer, but:

      I told the owner of the company recently that, and I quote, “I will fucking kill myself if my job becomes reviewing AI output”

    • TheTechnician27@lemmy.world
      link
      fedilink
      English
      arrow-up
      71
      ·
      edit-2
      11 hours ago

      Exactly. If you’re too stupid or lazy to adequately vet what your LLM puts out yourself, it shouldn’t be somebody else’s job to wade through the sewage you’re producing. You either shouldn’t be using one or, if you can’t do your job without it, you shouldn’t have that job.

      —Someone who doesn’t use genAI but has spent way too much time digging through LLM slop

      • Lost_My_Mind@lemmy.world
        link
        fedilink
        English
        arrow-up
        46
        ·
        10 hours ago

        You know what my favorite pizza topping is? Bleach.

        Dominoes REFUSES to put bleach on my pizza, so I gotta do it myself. I found out about it from AI. Now my pizza tastes great! The downside is having to go to the hospital to get a stomach pump everytime.

      • inclementimmigrant@lemmy.world
        link
        fedilink
        English
        arrow-up
        11
        ·
        edit-2
        9 hours ago

        I mean honestly yeah, I’m not going to waste my time with some junior developer who can’t explain how the code works and how it interacts with whatever framework I’m working on. I ain’t got time for that nonsense, especially when the code I deal with involves safety critical sections of code.

        Honestly if my work ever decided to allow unfettered AI code generation into my code base, I would immediately look for a new job at that point.

    • Hegar@fedia.io
      link
      fedilink
      arrow-up
      14
      ·
      10 hours ago

      It’s going to make snr devs get fired, surely?

      They either refuse to sign off when boss wants them to and get fired or sign off and get fired when ai code they signed off on causes issues.

      • frank@sopuli.xyz
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 hour ago

        Or quit/find new jobs. I suspect that’s by design by Business Idiots.

        *Get rid of the most expensive engineers and the cheaper ones can just use AI to make up the difference in output. And we can make the lower engineers the fall guy when convenient and replace them at our leisure *

        The disdain bosses have for average people is astonishing.

      • Windex007@lemmy.world
        link
        fedilink
        English
        arrow-up
        11
        ·
        9 hours ago

        Bingo.

        Maybe not outright fired, but absolutely open them up to career limits based on what you described.

        All of Amazon’s code undergoes code reviews already. Accepting a PR is already spiritually a sign off.

        This is just explicitly a threat, explicitly trying to find someone to hold accountable because you can’t hold ai accountable. What are they gonna do, fire the ai? Sign here to be the fall guy. Fuck off.

    • mojofrododojo@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      9 hours ago

      it’s pretty fucking stark right? these are the devs that stayed after management mandated they USE the shit in the first place, now they want the same devs to become responsible for what the shit does to their codebases.

  • Otter@lemmy.ca
    link
    fedilink
    English
    arrow-up
    62
    ·
    10 hours ago

    Do the senior engineers NOT sign off on changes to systems that can take down the production servers? Even if we take out the LLM created code, this sounds like a bigger problem

    • pageflight@piefed.social
      link
      fedilink
      English
      arrow-up
      35
      ·
      10 hours ago

      We may start to see people realize that “have the AI generate slop, humans will catch the mistakes” actually is different from “have humans generate robust code.”

      • 🌞 Alexander Daychilde 🌞@lemmy.world
        link
        fedilink
        English
        arrow-up
        17
        ·
        8 hours ago

        Not only that, but writing code is so much easier than understanding code you didn’t write. Seems like either you need to be able to trust the AI code, or you’re probably better of writing it yourself. Maybe there’s some simple yet tedious stuff, but it has to be simple enough to understand and verify faster than you could write it. Or maybe run code through AI to check for bugs and check out any bugs it finds…

        I definitely have trusted AI to write miniature pointless little projects - like a little PHP page that loaded music for the current directory and showed a simple JS player in a webpage so I could share Christmas music with my family and friends. No database, no file uploading or anything. It worked decently, although not perfectly, and that’s all it needed to do.

        • Mirror Giraffe@piefed.social
          link
          fedilink
          English
          arrow-up
          2
          ·
          3 hours ago

          I’ve been writing a slightly larger project with frontend, bff and backend and I need to take it in small batches so that I can catch when it misunderstands or outright does a piss job of implementing something. I’ve been focusing a lot on getting all the unit tests I need in place which makes me feel a bunch better.

          The bigger and more complex the projects get, the harder it is for the LLM to keep stuff in context which means I’ll have to improve my chunking out smaller scoped implementations or start writing code myself I think.

          All in all I feel pretty safe with my project and pleased with the agents work but I need to increase testing further before bringing anything live.

        • slaacaa@lemmy.world
          link
          fedilink
          English
          arrow-up
          3
          ·
          4 hours ago

          This is true not just with code, but with many types of complex outputs. Going through and fixing somebody’s horrible excel model is much worse than building a good one yourself. And if the quality is really bad, it’s also just easier to do it yourself.

    • PattyMcB@lemmy.world
      link
      fedilink
      English
      arrow-up
      25
      arrow-down
      1
      ·
      10 hours ago

      I guarantee there’s so much pressure on those engineers to deliver code that they rubber stamp a ton of it with the intention of “fixing it later”

      Source: I’ve worked in software for 20+ years and know a lot of folks working for and who have worked for Amazon

      • PabloSexcrowbar@piefed.social
        link
        fedilink
        English
        arrow-up
        5
        ·
        9 hours ago

        That’s basically the story at all the big tech companies, from what I’ve heard. In my time at Facebook, I felt like the only person who actually read the merge requests that people sent me before hitting it with “LGTM”

        • tal@lemmy.today
          link
          fedilink
          English
          arrow-up
          4
          ·
          edit-2
          8 hours ago

          If companies are going to place increasing reliance on review due to having lower-quality submissions, then they should probably evaluate employees weighting review quality (say, oh, rate of bugs subsequently discovered in reviewed commits or something like that).

          • ragas@lemmy.ml
            link
            fedilink
            English
            arrow-up
            1
            ·
            2 hours ago

            Sure. i’ll review your code favourably if you do the same with mine.

            That is also a way to get no bugs at all.

      • criss_cross@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        9 hours ago

        When I worked there 20% of the work we had to do had to go through a senior engineer. And getting his time was like pulling teeth.

        More of the time he would just nitpick grammar in docs and then finally rubber stamp work. It was awful.

    • mrgoosmoos@lemmy.ca
      link
      fedilink
      English
      arrow-up
      7
      ·
      8 hours ago

      the way private companies work is that they require their employees to produce more than is reasonable given the work quality that is expected.

      when this discrepancy is pointed out, it’s handwaved away. when the discrepancy results in problems, as it most obviously will, somebody is found to place the blame on.

      it’s not the developer’s faults. it’s a management decision.

      source: I’m talking out of my ass I’m just a salty employee who is seeing this happen at their own workplace when it didn’t used to, at least not to this level

  • Airfried@piefed.social
    link
    fedilink
    English
    arrow-up
    9
    ·
    9 hours ago

    The way AI is being pushed onto workers on a global scale has to be the dumbest thing to ever happen in the work space. Executives are getting hysterical over something they don’t even try to understand and even governments shower companies in subsidies if they do anything with AI. Of course the only result so far are mass layoffs and exploding costs for energy and hardware. All the while economies are crumbling everywhere because of course they do when mass unemployment sweeps around the globe. And again, governments everywhere are subsiding this crap with tax payer money. What’s even worse than all of that is the insane environmental damage all of this causes. But I’ll have to cut myself short here because I’m just getting increasingly upset here.

    I guess what I’m trying to say is: We’re funding our own decline in rapid speed. Human stupidity has found a new peak in 2026 and it’s not even close. I knew the way AI was advertised was completely overblown years ago but I never anticipated it would get this bad this quickly.

  • IchNichtenLichten@lemmy.wtf
    link
    fedilink
    English
    arrow-up
    5
    ·
    9 hours ago

    They want to move fast and break things but they still want a few meat bags around to blame when things inevitable blow up in their faces.

    • PrincessLeiasCat@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      3
      ·
      7 hours ago

      They don’t want to break the things that hurt their bottom line.

      Break society and make everyone hate each other? Hey buddy - small price to pay. Gotta think of the shareholders, after all. (╯°□°)╯︵ ┻━┻

  • hperrin@lemmy.ca
    link
    fedilink
    English
    arrow-up
    4
    ·
    9 hours ago

    xD

    Guess that all-in-on-AI attitude was not such a bold and brilliant idea after all.