Apple has recently overhauled its entire M-Series chip plans, scrapping the launch of the M6 Pro and M6 Max processors and jumping straight to the M7 series. While the base M6 SoC is expected to launch, Apple is moving into the M7 series without the M6 Pro/Max variants, with plans to offer some...
Local LLMs are cool but also pretty slow compared to cloud. If you have to wait half an hour for your Feature while coding you might still opt for the cloud agent.
Yeah. But they’re slow because most of us are GPU peasants. If someone were willing to drop $3-5K on a rig, they could probably run decent, dense models at greater than cloud speeds. Hell, with enough black magic, they could do it with less, but they’d have to go deep into the weeds.
OTOH, $3-5K buys you a shit ton on Open Router, Claude, Chat, Lumo etc.
The game is entirely rigged for “you will own nothing and be happy about it”.
Yes, they are slower. However, I think that the pricing we’re going to see from the cloud providers might be enough to deter quite a lot of people. At least I hope so:
The fact that we’re already used to blazing speed generation kinda sucks. Local models are a much more sustainable way of unlocking the benefits of LLMs than giant ecosystem- and community-destroying data centers.
I also hope that don’t get me wrong, but as I said: Waiting for the LLM agent to finish coding is currently a bottleneck in software development, they don’t pay high salaries for watching the AI code, they will prefer faster agents even if they are expensive, because they are not only paying the AI Company but also the software engineer overseeing them.
I think that is only going to last as long as the AI providers are willing to operate at a loss. The issue is even with the newer higher price points rolled out this year, they’re still losing money. The slower AI machines may be the answer once the REAL profit earning price for the use tokens hits the market. I forsee lots of alternative work going on while the small LLM’s are cooking the data. We will have to see once these machines start to roll out, what the use for LLMs will be and how it’s applied. I am hopeful.
Have you tried running a local model on a M series Mac?
Yes ofc I ran Gemma 4 for example, but compare that to the speed of Gemini in the cloud the difference is massive.
How much RAM do you have and which version of the model did you run?
Local LLMs can be just as fast as long your device clears the requirements. If you noticed a huge difference, there’s a really good chance that you tried to use a model that requires more RAM than you have
I ran Gemma 4 31 B quantized so it fits in my RAM. The decoding speed was decent, but if you look at the newest models for example Gemini flash 3.5 they have a decoding speed of 280 token per second, they generate an entire page before my Mac locally generates a sentence.
That is a bit too much for your hardware, even the Q4_0. You needed a smaller version (26B likely would suit you better. It would be faster and is a MoE)
Actually they can be much faster given sufficient VRAM and not a lot of concurrent users.