Yeah. Not a surprise.
everyone has bet on AI getting good enough to fully replace humans fast. CEOs mandated use of AI. The results weren’t as great as expected. CEOs started limiting AI use to counter the token cost explosion after employees found out how to waste tokens fast.
At the same time, China’s AI development is driven by the party instead of companies. They aren’t so much interested in money as they are in the strategic solution to the demographic problem caused by the one-child policy (which worked a bit too well too fast). The companies there plan on making money by providing the compute (they literally have the power and a two digit number of nucular GW under construction right now).
So their models are almost as good as US ones and freely downloadable to run on whatever hardware you want. That naturally limits the longterm-achievable AI token prices to little more than the cost of just providing the raw compute.US AI companies are also in a cut-throat competition for customers right from the start. That obviously doesn’t help to keep prices high. Currently, they all burn money so fast that it’s hard for a human mind to comprehend.
None of the US AI companies will survive the next decade if they don’t actually are the only one making AI actually able to fully replace human workers. They will all go bankrupt and might take the whole US economy with them.
Nvidia will probably be the real winner if they don’t fuck this up somehow (not sure if that is even possible) because they are the ones selling the shovels in this gold rush.In two decades, it will be normal to have the capabilities of current frontier models running locally on your Chinese phone.
Oh yeah “intelligence” is cheap AF…
As the price of AI decreases, it will become more widespread and pervasive.
AI is already free. These are the subscriptions that companies and consumers who already want to use it pay for.
The cost decline for a given level of performance does tend to slow over time
Even in their tests, there are big drops in 2026 models, and recent ones.
Opus 4.8 (may 2026), by far best model at the time, gets equaled by deepseek 4 pro (july 31) and 4.1 flash (sept 10th) at less than 1/100th the cost. MiMo 2.6 pro (sept 21) is even cheaper and beats opus 4.8 scores by a wide margin. sonnet 4.6 max to gpt luna high is also a 99% drop in a short time for lower performance models.
The other day I saw a video talking about this new innovation on LLM inference side of things where they keep some more used weights in RAM and others less used on disk. I always suspected from the sample code I stumbled upon on the IA world that should be extreme opportunities for optimizations. But I cannot stress it enough how dumb the LLM world is where the basics of implementing an LRU cache is pass of as some big innovation. Like any half competent comp-sci or comp-eng professional know about the basics of mitigating this basics bottlenecks like “the data does not fit on available RAM”, “The disk is slow”, etc.
So is not surprising that now that it seems that the “powerfulness” of this LLMs is starting to plateau that we would start to see some improvement in performance/resource utilization and hence running costs.
Keep in mind many of the optimizations you’re talking about (LRU caching of experts, for example) are only really relevant at the single user local inference scale. As in, an individual wants to run a big model on their machine, but they don’t have enough VRAM to fit the model and KV cache. Accordingly, you’re basically only describing hobbyist and research projects, which aren’t really representative of the AI inference industry as a whole.
Commercial inference keeps everything resident in VRAM, so expert caching isn’t necessary. So these things won’t help costs. A lot of other low hanging fruit (like hierarchical KV cache) has also existed for a long time for production-ready inference engines.
I am not sure it would not help commercial solutions, if all experts are used all the time, sure, but if for example the usage is biased for some experts it would enable one machine to serve more users in parallel or save on VRAM or DRAM without compromising response time, hence cutting costs.
That works for “Mixture of Experts” models. These are basically models with distinct sets of weights and only a subset of them will be used on any particular query. The rest can sit on a disk.
It doesn’t work for dense models, where every weight is used all the time. There’s nothing inactive so a cache has nothing to exploit.
Moore’s Law hasn’t been applicable for years.
Yeah gimme some more of that 14nm++++++
That’s cause you burnt money equivalent of gdp of Spain in that time frame, and I am being generous here.
I forsee a future for LLMs which is where the only people who will be using it is where they HAVE to use it. When it’s the only reasonable tool for the job. And it will be expensive and unprofitable to run.
It has some use cases, but they are few. The market can’t support what they’re people want LLMs to be
Thanks in part to China who release a lot of research free.
All that R&D, and they need to squeeze efficiency out more at this stage. Faster and more specialized is better than big chingus do everything, and means we can hopefully stop spending all that money on making more datacenters.
Personally, I don’t use the big chingus models at all… I much prefer local gen if I use it for things. Much better privacy that way, and I’m not throwing money at these jokers.
Are there some local models that one can use that are not giving our data away and on par with chat GBT?
Ollama, Open WebUI, and Qwen, or one of the edits of it. The latest Qwen 26B is in a very good delta of smart enough to know how to help with stuff (I use it as a second opinion/initial proofread code checker) and small enough to run on a macbook or a gaming pc, and Open WebUI is the self host ChatGPT-like interface that works with Ollama, the turn key ai model runner backend. Couple that with your own searxng instance self hosted to give it the ability to search the web for answers if you want to give it that. Run it all in a virtual machine, of course.
For media content, wan2gp (wan2 for the gpu poor) has an all in one stop shop for self hosted audio, image, and video. Just beware people don’t like generated stuff in prod, but its useful for quick concepts, or to do certain types of edits of existing stuff.
There’s also tts and speech recognition via the whisper models, which sounds a lot better than the speak and spell voices from the past (unless you enjoy the degeneracy of spamming JOHN MADDEN JOHN MADDEN in the Steven Hawking voice) and can transcribe words a lot faster (realtime) and more accurately than in the past. The best ones run via Vulkan and thus work on any card that supports Vulkan.
If you run Nextcloud, you can also connect Ollama to it to get the Nextcloud AI working.
Great input, thank you I appreciate it. Next off for me is learning how to do a virtual machine! I do have ollama currently on a computer connected to a speech to text app. And I find that very useful to have
Virt-manager for a gui, qemu-kvm and libvirtd for the hypervisor. You will need to pass thru a graphics card, which makes it unavailable for the host machine entirely. I use debian stable for the vm, no gui, just ssh to manage it.
I wonder how they plan to match “prices are in free fall” to “the AI industry will have to make trillions a year in order not to go bust”.
On the other hand, “prices in free fall” might be the answer they got from AI…
At this point the cloud model firms are basically banking on making a superinteligence before anyone else and taking over the planet, otherwise they go bankrupt. I wish I was kidding.
Nvidia wins either way though, local models, cloud models, shovels always sell. So they have that overvalued but still realistic bedrock to build houses of cards on.
Nvidia wins unless some other company starts selling cheaper, faster, more efficient matrix multiplication machines.
I’ve read some articles about radically different inference architectures that may tilt the scales, but I know this is wishful thinking because I really would like Nvidia to fail badly.
Linus_nvidia.gif
Plenty have tried, all have fallen over flat on their face when it comes to actually providing usable drivers or production at scale. AMD’s still completely half assing it even today and Intel’s OneAPI and Vino is a bloated joke.
But yes I would love a future where AMD finally hires an actual software team.
Yeah, OpenAI released Astra just a couple of weeks ago, then around last week they released Sol 6, which is around half the price of Astra. So you have around the same level of “reasoning” at half the price. They will never turn a profit, at least for the next several years or so.
So revenue is falling. The only way the bubble grows is forcing this shit into even more places?
the price of thought
Oh fuck off.
No actual thought here in this comment
These fuckers would sell us air if they could.
I saw it in a documentary called Total Recall
That’s cool and all, but when can I buy RAM again?
Never if you are in the US. In a few years if you are allowed to buy Chinese.
In 4 years or never. The latter probably being the most likely, since they are not just keeping RAM from you for AI purposes. They don’t want you to own you own hardware anymore, so they just simply stop manufacturing consumergrade hardware.
They don’t want you to own [your] own hardware anymore
Daily, it seems, I realize I’m one of the non-telepathic people: I can’t immediately know chip-maker company CEO motives and goals, and I just don’t see the conspiracy you do. What am I thinking right now?
Probably not never - CXMT are trying to aggressively expand to fill the market now the major players have left, but it will still be a few years before prices really come down as a result.
