In my regular research (behind a paywall), I have been saying for a while that I think the future of AI is not large language models (LLM), but small language models (SLM) run on local desktop computers or even mobile phones.
I don’t doubt that locally hostable models will be important for agentic tasks but I suspect they are still going to be bigger than most can comfortably host for time being. I still think there will be a place for the super large models for more complex reasoning although how much will be due to the intrinsic knowledge in the weights and how much due to the plumbing around them remains too be seen.
And if course LLM’s are not going to be the end point of the search for AGI. Whatever their architecture they will still need copious amounts of compute.
I’m really interested to see what comes of the LLM ASIC idea - it worked wonders for hardware video codecs. I could see the smartphone industry picking up again on annual releases worthy of upgrades if they included dedicated AI chips not beholden to a cloud subscription. I think we’re reaching the point that the frontier models are becoming “good enough” to use for a while before uploading to the next.
That’s kind of what I’m expecting going forward too. Local models will get good enough in a year or two for most tasks, and then you just have a specialized chip like the GPU to run them.
A specialised chip wouldn’t be nessesary though. GPU’s can run smaller models just fine the main issue is enough memory to store larger ones without resorting to CPU memory. Power consumption probably is the main gain to be had with NPU’s. You’ll still be limited by slow cpu memory once you run out of faster memory.
Oh didn’t see that. This is really cool! I suppose it does work similarly to hardware codecs, with the same very big trade-off being you get locked into a specific model. But considering this is an emerging technology maybe they could be made small enough to have multiple models on a single chip! (similar to codecs) Or just have one really big model that would be more future proof, and the price-performance would be so much greater than running the model on a GPU.
LLM ASICs are different to NPUs. The idea is to encode a specific model’s weights (all billions of them) as a gigantic resistor network on the chip itself. Analog, where ohms law becomes the processor, and no RAM is necessary - the chip IS the model. It would pave the way for cost effective, fast, low-power local inference. Given the analog circuitry, there would be issues like interference, temperature sensitivity and so on, but we’re seeing models get good results even when heavily quantised, so the idea has legs. The big disadvantage is you’re stuck with that one model forever, so we’re likely to see these first appear in non-consumer applications like automotive, defence etc. A pretty exciting concept that solves some of the environmental issues around AI.
but I suspect they are still going to be bigger than most can comfortably host for time being
ARM and CXMT both have the potential to change this situation rather dramatically.
ARM: The RAM/VRAM divide is kind of a legacy thing at this point, doing more harm than good in the x86 (desktop OS) space, but ARM doesn’t have the same baggage.
CXMT: We know the DRAM cartel have previously engaged in price-fixing, and the current shortage looks suspiciously similar to their old behavior. When confronted by a new challenger, they might be forced to actually compete.
I wouldn’t say it’s totally legacy. (v)ram bandwidth does matter and while the Apple M-series chips do well with their unified memory architecture don’t forget it’s fixed because it’s part of the CPU chip.
Hence “x86 (desktop OS) space”. It’s not intrinsically part of x86, but it has settled in as a conventional piece of x86 desktop OSes. x86 consoles and ARM desktops don’t assume the same principle, and they can get a lot more mileage out of SoCs as a result.
Yes. The issue is not that ARM systems don’t/can’t communicate with a dGPU over PCIe. It’s that x86 systems (edit: running conventional x86 desktop OSes) have to basically pretend they’re using PCIe to communicate with an iGPU. (Not literally, but a lot of the primitives they use were inherited from AGP/PCI and don’t make a lot of sense for an SoC.)
I don’t doubt that locally hostable models will be important for agentic tasks but I suspect they are still going to be bigger than most can comfortably host for time being. I still think there will be a place for the super large models for more complex reasoning although how much will be due to the intrinsic knowledge in the weights and how much due to the plumbing around them remains too be seen.
And if course LLM’s are not going to be the end point of the search for AGI. Whatever their architecture they will still need copious amounts of compute.
Alibaba just announced a chip specifically for running local models. We’ll see what it ends up going for. https://wccftech.com/alibabas-tsmc-built-5nm-risc-v-chip-xuantie-c950-now-runs-qwen-3-8-27b-model-natively-unlocking-massive-vertical-integration-tailwinds/
I’m really interested to see what comes of the LLM ASIC idea - it worked wonders for hardware video codecs. I could see the smartphone industry picking up again on annual releases worthy of upgrades if they included dedicated AI chips not beholden to a cloud subscription. I think we’re reaching the point that the frontier models are becoming “good enough” to use for a while before uploading to the next.
That’s kind of what I’m expecting going forward too. Local models will get good enough in a year or two for most tasks, and then you just have a specialized chip like the GPU to run them.
A specialised chip wouldn’t be nessesary though. GPU’s can run smaller models just fine the main issue is enough memory to store larger ones without resorting to CPU memory. Power consumption probably is the main gain to be had with NPU’s. You’ll still be limited by slow cpu memory once you run out of faster memory.
Read my post above to a different commenter, there’s a new type of ASIC being talked about that’s very different to an NPU or GPU, no RAM necessary.
Oh didn’t see that. This is really cool! I suppose it does work similarly to hardware codecs, with the same very big trade-off being you get locked into a specific model. But considering this is an emerging technology maybe they could be made small enough to have multiple models on a single chip! (similar to codecs) Or just have one really big model that would be more future proof, and the price-performance would be so much greater than running the model on a GPU.
Also I can wait for Nvidia’s stock price to completely die as soon as this becomes mainstream.
NPU’s have been in phone SoCs for years now.
LLM ASICs are different to NPUs. The idea is to encode a specific model’s weights (all billions of them) as a gigantic resistor network on the chip itself. Analog, where ohms law becomes the processor, and no RAM is necessary - the chip IS the model. It would pave the way for cost effective, fast, low-power local inference. Given the analog circuitry, there would be issues like interference, temperature sensitivity and so on, but we’re seeing models get good results even when heavily quantised, so the idea has legs. The big disadvantage is you’re stuck with that one model forever, so we’re likely to see these first appear in non-consumer applications like automotive, defence etc. A pretty exciting concept that solves some of the environmental issues around AI.
More info for the curious: https://taalas.com/the-path-to-ubiquitous-ai/
ARM and CXMT both have the potential to change this situation rather dramatically.
ARM: The RAM/VRAM divide is kind of a legacy thing at this point, doing more harm than good in the x86 (desktop OS) space, but ARM doesn’t have the same baggage.
CXMT: We know the DRAM cartel have previously engaged in price-fixing, and the current shortage looks suspiciously similar to their old behavior. When confronted by a new challenger, they might be forced to actually compete.
That has nothing to do with x86 though…
I wouldn’t say it’s totally legacy. (v)ram bandwidth does matter and while the Apple M-series chips do well with their unified memory architecture don’t forget it’s fixed because it’s part of the CPU chip.
Hence “x86 (desktop OS) space”. It’s not intrinsically part of x86, but it has settled in as a conventional piece of x86 desktop OSes. x86 consoles and ARM desktops don’t assume the same principle, and they can get a lot more mileage out of SoCs as a result.
Aren’t there arm workstations that still take pcie cards and have a similar architecture to x86?
Yes. The issue is not that ARM systems don’t/can’t communicate with a dGPU over PCIe. It’s that x86 systems (edit: running conventional x86 desktop OSes) have to basically pretend they’re using PCIe to communicate with an iGPU. (Not literally, but a lot of the primitives they use were inherited from AGP/PCI and don’t make a lot of sense for an SoC.)
@jumping_redditor @kibiz0r
You really need to read this series:
https://marcin.juszkiewicz.com.pl/2025/07/07/arm-desktop-2025-attempt-part-one/