In my regular research (behind a paywall), I have been saying for a while that I think the future of AI is not large language models (LLM), but small language models (SLM) run on local desktop computers or even mobile phones.
That’s kind of what I’m expecting going forward too. Local models will get good enough in a year or two for most tasks, and then you just have a specialized chip like the GPU to run them.
A specialised chip wouldn’t be nessesary though. GPU’s can run smaller models just fine the main issue is enough memory to store larger ones without resorting to CPU memory. Power consumption probably is the main gain to be had with NPU’s. You’ll still be limited by slow cpu memory once you run out of faster memory.
Oh didn’t see that. This is really cool! I suppose it does work similarly to hardware codecs, with the same very big trade-off being you get locked into a specific model. But considering this is an emerging technology maybe they could be made small enough to have multiple models on a single chip! (similar to codecs) Or just have one really big model that would be more future proof, and the price-performance would be so much greater than running the model on a GPU.
That’s kind of what I’m expecting going forward too. Local models will get good enough in a year or two for most tasks, and then you just have a specialized chip like the GPU to run them.
A specialised chip wouldn’t be nessesary though. GPU’s can run smaller models just fine the main issue is enough memory to store larger ones without resorting to CPU memory. Power consumption probably is the main gain to be had with NPU’s. You’ll still be limited by slow cpu memory once you run out of faster memory.
Read my post above to a different commenter, there’s a new type of ASIC being talked about that’s very different to an NPU or GPU, no RAM necessary.
Oh didn’t see that. This is really cool! I suppose it does work similarly to hardware codecs, with the same very big trade-off being you get locked into a specific model. But considering this is an emerging technology maybe they could be made small enough to have multiple models on a single chip! (similar to codecs) Or just have one really big model that would be more future proof, and the price-performance would be so much greater than running the model on a GPU.
Also I can wait for Nvidia’s stock price to completely die as soon as this becomes mainstream.