I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
It’s kind of insane that integrated wikis with citations/documentation aren’t the centerpiece of (Reddit) communities.
Instead, we got… Discord?
It’s a bit misleading.
Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27V also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
Also, AI or not is really irrelevant for this case.
They said as much:
Saflor uses AI tools and does not believe that AI itself is the problem. Goldman pointed out that his arguments against Memes Apps would be largely the same even without the AI aspect. But Saflor considers Memes Apps’ platforms to be examples of irresponsible AI products, where the operators problematically advertise that you can “fire your ad agency” and replace all creative work with a meme generator.
Which sounds to me like effectively abolishing the copyright system for anyone who can’t afford paying lawyers to hunt down every little infringement, only leaving it up for wealthy corporations.
And yeah, that would be a really terrible precedent…
Well, the current state is that there are a couple of “best” models, but literally hundreds of independent providers serving them. As an example, one can get GLM 5.1 from its trainer, or one can get it from DigitalOcean, or Baidu, or SiliconFlow ASICs, or get it at very high speed from Cerebras ASICs, or AMD providers, or finetune it from a number of services, or rent the self hosting…
The companies aren’t keeping the models to themselves, and that blows the marketplace open to a boatload of competitors.
They don’t have to host it themselves. They could use a number of providers for the same model, and basically keep doing whatever they were doing with OpenAI/Anthropic via the exact same APIs.
Bonsai? Or whatever it’s called? It’s a con, so far; it’s not better than smaller models quantized to 3-4 bits.
I love, love the idea of bitnet, but it only seems to work with models trained from scratch, which no one has done at scale yet.
You want this one:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main
Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3
For faster inference at the cost of a little more VRAM usage:
https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3
And you run those in:
https://github.com/theroyallab/tabbyAPI
And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
I also forgot to emphasize this, but Xiaomi’s plan, in my opinion, is undiscovered fruit.
GLM had a similar coding plan, but once it got in the news and popular, it got WAY more expensive and limited. I’m grandfathered into 6 more months of a GLM plan you literally cannot buy now.
And I think Xiaomi is in the same situation GLM was 6+ months ago. It’s a fantastic model series, but unlike Kimi/GLM no one knows about it yet, which is how it’s still $60 for a year.
+1 for Hermes.
If you have a newer Nvidia GPU, you can run Qwen 27B via exllamav3 and get good quality/speed in 16GB. And it’s worth the trouble, as 27B is an amazing model.
If it’s AMD, yeah, a3B is a good bet, depending on how much spare CPU RAM you have.
I was thinking of Xiaomi’s Lite coding plan:
https://platform.xiaomimimo.com/token-plan
My mistake, it’s $60. But that’s still dirt cheap.
You can read their terms of service; coding APIs are generally less invasive than something like the ChatGPT app, but don’t bet on any privacy either.
For my personal setup, I actually run most queries locally, via MiMo 2.5, but send “bulk” privacy-insensetive prompts to the same model on the API, since its so much faster. I kinda use them in parallel.
Mistral is still around, but for local LLMs… they’re kinda irrelevant, sadly. Their models have regressed. I think they’re being choked by ambiguous EU law.
If you’re looking for “western” weights, I’d look at the Laguna series first:
https://huggingface.co/poolside
And ik_llama.cpp quantizations like this: https://huggingface.co/sigargv/Laguna-M.1-GGUF/tree/main
There’s a couple of other interesting startups, but TBH its hard to keep track of where they’re from.
Oh that’s perfect!
You can’t realistically run Kimi (as it’s a 1T+ model), but you’re set. There’s a glut of excellent 120B-300B models for you to choose from.
I’m quantizing MiMo 2.5 specifically to cram into 24GB/128GB as I type this. It’s tight, and will use up most of your memory, but it’s fantastically smart and plenty fast. The quantization won’t finish cooking until tomorrow, but I’ll upload it to huggingface then.
But there are others existing quants would fit, like Deepseek Flash IQ3_S: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF/tree/main/UD-IQ3_S
Or… well, any of these! https://huggingface.co/models?num_parameters=min%3A128B%2Cmax%3A256B&library=gguf&sort=modified
The key for you is to run the ik_llama.cpp fork: https://github.com/ikawrakow/ik_llama.cpp/
It’s specifically optimized for hybrid (CPU + GPU) inference on Nvidia desktops; you’ll get MUCH faster speeds than mainline llama.cpp or anything based on it. It also supports some more exotic quantization type; as an example, I’m quantizing MiMo 2.5 as a hybrid quant, with the dense layers at IQ6K/Q8_0 and the sparse experts as an IQ3_KT “trellis” quant type. This should yield a higher fidelity quantization than a typical Q3 GGUF while taking less RAM, at the cost of taking forever to quantize and a slight speed hit.
Some other quantization types (like the KS or R4 types) are specifically configured to be fast on CPU.
There are specialized “quant cookers” that make GGUFs specifically for ik_llama.cpp, like:
https://huggingface.co/ubergarm
https://huggingface.co/AesSedai
https://huggingface.co/sigargv/Laguna-M.1-GGUF
https://huggingface.co/models?other=ik_llama.cpp&sort=modified
Mine will be here once I upload it:
https://huggingface.co/Downtown-Case
For anyone else reading this: none of this is applicable to you if you have an AMD/Intel GPU, or an older Nvidia GPU, or less than a certain amount of RAM, or a non AVX2 CPU or… well, there’s all sorts of caveats.
The optimal runtime is different for everyone. As an example, exllamav3 is WAY better than llama.cpp on modern Nvidia GPUs until you get above a certain amount of CPU RAM; then using all that RAM for hybrid inference makes more sense. On AMD, different backends work better on different GPUs, and… well, you get the point. Basically all LLM running advice is irrelevant without specifics of your hardware, even this post will be obsolete in a month.
Nothing on the internet is real.
Don’t feed the trolls.
I feel like if there was a dedicated class in schools worldwide, that just did nothing but teach these two things over and over again, the world would be a better place.
…Because the fundamental issue is that people are believing this guy. And feeding him. He is a troll, making cash being a troll; he cannot help himself. But the public maybe possibly can.
I just meant that you have to be cognizant of what went into the quantization.
As an example, a “Q4_K_M” could be too much quantization to be usable on one model, and an inefficient waste of space on the other. Two Q4_K_Ms of the exact same model could be completely different, one totally borked. Or one particular Q4_K_M could excel in one task, but be totally useless for another, even with the exact same settings, when a slightly different sized or type of quantization would excel.
It’s a deep rabbit hole. It’s not random either; there are distinct technical reasons behind every case mentioned above.
And that’s not even at the cutting edge quantization anymore, though what’s “cutting edge” completely depends on your particular hardware and use case.
I’m trying to make this sound daunting on purpose.
Many people have really horrible experience with a default “ollama run” for this exact reason, because the defaults are terrible and the customization is critical to getting coherent, performant output.
Unquantized LLMs, on the other hand, are basically always run the same way: vllm docker image on a big server, official weights. There’s less to “go wrong” trying to squeeze it on hardware with unofficial runtimes and compressors.
https://sleepingrobots.com/dreams/stop-using-ollama/
And this is just the tip of the iceberg for ollama. They’re the same kind of scammy tech bros as OpenAI.
The best setup depends on your hardware. There is no “easy button” unfortunately, quantized LLMs are just too intense and finicky to run without making some informed choices.
It also depends on what you want to do with the LLM. For example, some are too slow or bad at long context for agenic use, some quantizations are great at scripts but terrible outside that, or vice versa.
But LM Studio and Qwen 3.5 35B Q4 is probably the “easiest” flat recommendation I can make.
Or… honestly, just pay $40 for basically unlimited usage for a year from an API, then roll your own frontend.
If you’re running stuff overnight on a homelab, you can get away with using very large models as agents in CPU RAM.
Another option is a hybrid approach: farm “dumb” privacy-insensitive calls to an API you can subscribe to for like $40 a year, and let your local LLM do the orchestrating. This is what I do: I run MiMo locally, but also farm calls out to the exact same LLM in the cloud when appropriate… And I don’t mind one bit. I’m happy to tip them for open sourcing it in the first place.
And Llama and Mistral are ancient history at this point.
The cutting edge of local is lightyears better now. It’s basically where ChatGPT/Anthropic were not that long ago, with a bit less world knowledge because of the size.
The localllama crowd has know this for years.
It happened faster than I expected, though; OpenAI/Anthropic hardly even got the chance to tighten the screws.
No.
Why would they? “Moderate” Republicans would get zero engagement on anything, and if they’re way down the MAGA rabbit hole they are on different platforms to indulge in that.
People here label “a particular liberal/leftist that’s not in my political tribe” as Republican all the time, though. It’s honestly mind boggling to me. Lemmy is in like the 99th percentile of the leftism scale in the US.