• iamthetot@piefed.ca
    link
    fedilink
    English
    arrow-up
    11
    ·
    edit-2
    6 小时前

    Worth noting that a token is not necessarily a word, though can be. One word could also take multiple tokens. It can also vary from LLM to LLM and their tokenization methods.