• Flames5123@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    0
    ·
    7 days ago

    It’s because of how token streaming works. As the reply comes, it generates token after token. Then it uses those in the next token generation. It sees the previous one and associates it with “wrong” then it generates the “correction” tokens. I see this all the time at my work when it goes down the wrong path before correcting itself.