A 3060?
Exllama/TabbyAPI is still worth looking at if you are trying to run a model purely in GPU RAM. It’s easily the most VRAM efficient backend, it just doesn’t support CPU offloading (which is useful for MoEs if you have considerable spare CPU RAM) and more optimized for 4xxx and up Nvidia cards.
And TabbyAPI has a docker container you can use. Look for “exl3” models on huggingface.
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
Mostly, yeah.
Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded, but you know will be fine in regular use without too many programs open.
In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap CPU RAM, and don’t OOM on the GPU either.
Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.
Mind you, I’m running Mimo, not the big Mimo Pro.
But yeah. I really like the model, even for one of its size. And it hardly feels quantized as a trellis quant.
CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.
This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.
I completely disagree.
Frankly, I find the description “VC funding a FOSS” offensive. They aren’t funding the engine. I’ve been messing with LLM inference engines since 2022, and Ollama is the worst I’ve seen in the community.
They misname models for SEO. They leech off llama.cpp while deliberately hiding attribution yet redirecting GH support requests there. They sometimes make their own GGUFs+forked releases which are broken and incompatibile with upstream llama.cpp, just so they can get a release out a day ahead for hype, even though it doesn’t really work and they’ll never upstream one line. They set a default context size thats basically unusable, they screw up chat templates and deep internal code with no obvious indicators, they release suboptimal quants without iMatrix, they gate you into their internal quantization repo and model card format, they hide model downloads on your hard drive, they mess with standard APIs for no good reason other than to mess up other backends. I could go on and on.
And if that’s all fine, they’re enshittifying the app with closed code, and pointers to cloud models.
They GIVE LLM inference a bad name, by making it a terrible quality engine that happens to show up in search as the “default.” Hence the comments below of people being unimpressed with local inference. And they sap attention from actual llama.cpp devs, without contributing a single dime. Everyone in the localllama communtity hates their guts, and that’s not even getting into the interpersonal drama they’ve stirred.
They are a leech that’s a net drag to the whole community, that we can’t get rid of because they’re attention grifters. And they’ve gotten worse and worse over time.
It’s more morale to use any cloud API over Ollama, in my eyes. They’re a grift.
EDIT: And, to be clear, I’m not against VC funded downstream stuff.
LM Studio is good! Even though it’s closed source.
Tons of downstream projects are great.
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like one of these: https://huggingface.co/models?other=ik_llama.cpp&sort=modified&search=35B
And speculative decoding like DFlash or MTP (which you can also get specific models for).
EDIT: Wrong link.
Oh, and I just saw you have a 3090.
To get more specific, you can actually run way better models than Qwen 3.5 and Deepseek coder (both of which are very obsolete now). The best that’s practical depends on how much CPU RAM you have, but at the minimum you can do Qwen 3.6 27B, with a more optimal quant like ones here: https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/tree/main
Or Gemma 31B QAT: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF
If you have 128GB CPU RAM, I can upload my custom MiMo 2.5 quant. That should “beat” the cheapest Claude, give or take.
If you have 64GB, I’d suggest a quantization of Step 3.7.
If you have 32GB or 48, I’m not sure. I’d need to look if any “small” MoE is actually better than Qwen 27B now.
https://sleepingrobots.com/dreams/stop-using-ollama/
And that’s not even all of it. Basically they break models in many ways, and they’re slimey Tech Bros.
LM Studio is better, and easy.
If you’re on Nvidia, and want to run optimally, I would use the ik_llama.cpp fork. On AMD, regular llama.cpp. On a Mac, use an MLX runner (Like LM Studio) with an MLX quant (ideally an MLX-DWQ quant).
It’s all pretty technical, and… thats kinda the point. LLMs are just too performance sensitive and too finicky to not have a grasp of how they work. There is no “easy button” to run them without bad results, there can’t be.
But if you don’t have time for that and just want to see if it’s worth it, I’d suggest self hosing your own UI, and trying the dirt cheap APIs of models you can theoretically run on your setup. This will give you a “best case” taste of what they’re capable of.
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
How much CPU RAM do you have?
Did you serve them with ollama?
It’s basically broken, if you did. Try the same models over API, and you’ll see what I mean.
Yep.
I have a RTX 3090 + 128GB CPU RAM.
Currently I run my own custom IQ3_KT quantization of MiMo 2.5 300B, and it’s crazy good. It’s better than API models from not that long ago, and it’s served at about reading speed.
Never thought I’d ever run such a thing on my lowly desktop.
For quick scripts or code assistant, sometimes I use Qwen 27B (another custom quant, currently experimenting with exllama). Or Gemini 12B for messing with image/audio input. But TBH MiMo 2.5 with thinking disabled is smarter than 27B with it.
…And honestly, I use GLM 5.2 API a good bit.
I was lucky enough to get a yearly subscription for like $30, 6 months ago. I do self host the UIs or whatever takes the prompts, though.
Yep.
That’s because DDG is Bing. To be blunt, it’s search is kinda terrible.
Which is the answer to much of “why modern cars are the way they are.”
There’s a bit of a survivor bias for old stuff.
It’s not Android. It’s SailfishOS. With first party support.
And even that aside, I don’t see anything comparable on Aliexpress, hardware wise.
1st-party supported SailfishOS, to be specific.
That’s huge, to me.
This is fine.
But the optics are important. There’s a concerted effort to delegitimize Wikipedia as an information source, as it’s not in Big Tech’s control.
And they don’t have to kill it. They just have to make it less popular than, say, Grokipedia, and every headline like this is a step in that direction.
Hence I have very scientifically minded family who are already saying some strange things about Wikipedia.
It…
Well, it’s not just about “too much ruthless efficiency.”
Zuckerburg fired a lot of brainpower in their ML divisions. He torched their landmark in the “AI” landscape, THE thing that literally set the standard for open weights LLMs (Llama), and hired a bunch of Tech Bro narcissists in their place, because he’s doesn’t have the first clue about how LLMs work, nor how important Llama its wake were.
They were on track to be at the center of open weights LLMs, and Zuck, and ony Zuck, blew it up because he is an ignorant coward. He ran, like he does from every bandwagon he jumps on at the first bump.
Meta still develops PyTorch openly, but we’ll see how long that lasts.
Aside: I’m sick of everyone pretending like these ultra-wealthy industry runners are somehow geniuses. Like Bezos swallowing the “space datacenter” scam whole, Google’s leadership consciously sabotaging their core product, or gestures at Elon Musk’s Twitter page.
They’re making terrible choices. They clearly do not have the techical background to make them. Their decisions aren’t “hard” like the headlines make it out to be, they’re objectively, obviously irrational purely for their own self-interest.