Sources and leaks from Amazon, Adobe, Atlassian, Citi, and more show what is really happening with AI right now: companies are trying to rein in AI use as costs spiral out of control.
Not entirely wrong, but tokens are not just “fake” in the way, for example, an in-game currency is. They’re the fundamental “units” of data, both input and output, processed by the model. For most models, tokens are just a certain number of characters or words. So they’re not completely untethered from the model. If we’re both using Clankerbot v5.1: Sloppy Logic Edition™️, your tokens are defined in the same way mine are.
This is near the edge of my limited understanding, but AFAIK, yeah they can mess with token costs and billing schemes all they want. They could theoretically charge us 2 different costs per token, or do surge pricing or some shit.
if they wanted to change the actual size/definition of what a token is though, that would require a whole new model (or at least a major revision).
You aren’t totally wrong. Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.
But what they use for calculating your bill is something different today.
Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.
I think you might have it mixed up with parameters, rather than tokens. Parameters are how big the model is, and are an indirect measure of how capable it is. Bigger models tend to be more capable.
But what they use for calculating your bill is something different today.
The tokenizer varies a little, but I don’t think it’s changed measurably from tokens. You pay an amount for a million tokens worth of processing. The tokeniser difference just alters how text is converted to tokens, but the tokens themselves don’t change all that much.
If anything, I’d honestly put the issue more with reasoning chains in models, where they basically babble to themselves inside of a <think> tag, that most interfaces hide/collapse. It makes them work better, but vastly increases the amount of tokens per operation.
They have been getting longer and more sophisticated with newer models. So you might have a model now that basically repeats the output multiple times whilst refining and drafting the non-reasoning output.
If you’re making it generate a lot, that’ll balloon the usage, and thus price.
That doesn’t make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?
Not entirely wrong, but tokens are not just “fake” in the way, for example, an in-game currency is. They’re the fundamental “units” of data, both input and output, processed by the model. For most models, tokens are just a certain number of characters or words. So they’re not completely untethered from the model. If we’re both using Clankerbot v5.1: Sloppy Logic Edition™️, your tokens are defined in the same way mine are.
This is near the edge of my limited understanding, but AFAIK, yeah they can mess with token costs and billing schemes all they want. They could theoretically charge us 2 different costs per token, or do surge pricing or some shit.
if they wanted to change the actual size/definition of what a token is though, that would require a whole new model (or at least a major revision).
You aren’t totally wrong. Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.
But what they use for calculating your bill is something different today.
I think you might have it mixed up with parameters, rather than tokens. Parameters are how big the model is, and are an indirect measure of how capable it is. Bigger models tend to be more capable.
The tokenizer varies a little, but I don’t think it’s changed measurably from tokens. You pay an amount for a million tokens worth of processing. The tokeniser difference just alters how text is converted to tokens, but the tokens themselves don’t change all that much.
If anything, I’d honestly put the issue more with reasoning chains in models, where they basically babble to themselves inside of a <think> tag, that most interfaces hide/collapse. It makes them work better, but vastly increases the amount of tokens per operation.
They have been getting longer and more sophisticated with newer models. So you might have a model now that basically repeats the output multiple times whilst refining and drafting the non-reasoning output.
If you’re making it generate a lot, that’ll balloon the usage, and thus price.
That doesn’t make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?