Nothing on the internet is real.
Don’t feed the trolls.
I feel like if there was a dedicated class in schools worldwide, that just did nothing but teach these two things over and over again, the world would be a better place.
…Because the fundamental issue is that people are believing this guy. And feeding him. He is a troll, making cash being a troll; he cannot help himself. But the public maybe possibly can.
I just meant that you have to be cognizant of what went into the quantization.
As an example, a “Q4_K_M” could be too much quantization to be usable on one model, and an inefficient waste of space on the other. Two Q4_K_Ms of the exact same model could be completely different, one totally borked. Or one particular Q4_K_M could excel in one task, but be totally useless for another, even with the exact same settings, when a slightly different sized or type of quantization would excel.
It’s a deep rabbit hole. It’s not random either; there are distinct technical reasons behind every case mentioned above.
And that’s not even at the cutting edge quantization anymore, though what’s “cutting edge” completely depends on your particular hardware and use case.
I’m trying to make this sound daunting on purpose.
Many people have really horrible experience with a default “ollama run” for this exact reason, because the defaults are terrible and the customization is critical to getting coherent, performant output.
Unquantized LLMs, on the other hand, are basically always run the same way: vllm docker image on a big server, official weights. There’s less to “go wrong” trying to squeeze it on hardware with unofficial runtimes and compressors.
https://sleepingrobots.com/dreams/stop-using-ollama/
And this is just the tip of the iceberg for ollama. They’re the same kind of scammy tech bros as OpenAI.
The best setup depends on your hardware. There is no “easy button” unfortunately, quantized LLMs are just too intense and finicky to run without making some informed choices.
It also depends on what you want to do with the LLM. For example, some are too slow or bad at long context for agenic use, some quantizations are great at scripts but terrible outside that, or vice versa.
But LM Studio and Qwen 3.5 35B Q4 is probably the “easiest” flat recommendation I can make.
Or… honestly, just pay $40 for basically unlimited usage for a year from an API, then roll your own frontend.
If you’re running stuff overnight on a homelab, you can get away with using very large models as agents in CPU RAM.
Another option is a hybrid approach: farm “dumb” privacy-insensitive calls to an API you can subscribe to for like $40 a year, and let your local LLM do the orchestrating. This is what I do: I run MiMo locally, but also farm calls out to the exact same LLM in the cloud when appropriate… And I don’t mind one bit. I’m happy to tip them for open sourcing it in the first place.
And Llama and Mistral are ancient history at this point.
The cutting edge of local is lightyears better now. It’s basically where ChatGPT/Anthropic were not that long ago, with a bit less world knowledge because of the size.
The localllama crowd has know this for years.
It happened faster than I expected, though; OpenAI/Anthropic hardly even got the chance to tighten the screws.
How much RAM do you have?
I can run MiMo 2.5 at about 9 tokens/sec, on 128GB RAM, a 7800 and a 3090 in an SFF rig. That’s a fantastic 310B model. I’m requantizing it right now, to see if I can speed it up with Dflash.
Still fantastic models can be run on 64GB or 32GB CPU RAM, as long as you have some GPU. We’re awash in sparse models these days.
Gestures.
This is exactly what I’m talking about.
Did I mention liberalism, capitalism, anarchism or anything? I don’t think so. But now we’re talking “John goddamn fucking Locke” and this thread is barreling towards satisfying Godwin’s Law.
But it’s not just the tankies, apparently. It’s db0 too? Extremists on .world. And yeah, the toxic “snark” subs that Ethan Kelin video points out seem about right.
It’s also that I’ve seen this story before, before I found the Fediverse. Reddit clones that don’t get shut down just wither away when distilled to such an extreme. Lemmy/Piefed would be no different, in spite of everything it has going for it.
Even if we can block most of the Fediverse somehow, what’s the point of staying on a sinking ship?
If the trend holds, commenters are gonna call me out as being insufficiently anti-fascist, again.
AdmiralPatrick went about this the complete wrong way, and was salty as heck. There’s no excusing the blocklist implementation.
But… I don’t disagree with their sentiment.
That comment directed at db0 is abrasive and uncalled for. Yet. Just the other day, I saw a commenter highlight this:
“Users who advocate violence are not moderated so long as the admins agree with the target.” Are they wrong though?
And among the comment maelstrom, db0’s reply was:
We’re not pacifists, no.
And of course my own take that I would rather not kill people was shot down vehemently. To think I almost joined db0 when I first found the Fediverse…
…What the fuck?
I don’t want to be close to a community like that.
One that just endorses calls for explicit violence and dances on people’s graves as long as the admin agrees with the target. One that boosts straight up misinformation as long as it has the “right” message. Lemmy.world seemed like it was on a trajectory to moderate that, but like the Tesseract dev observed, it sure doesn’t feel like that anymore.
Why would I wanna post about niche topics in a place like that?
In a place like this?
…Yeah, I don’t feel welcome here anymore, either. I haven’t for a while. If the trend holds, commenters are gonna call me out as being insufficiently anti-fascist, again.
I’m sick of that.
Maybe I’ll try blocking more spaces, and stick to niche discourse.
But if Lemmy’s current trajectory holds, and the bar for calling for violence is that low and spammed everywhere, I’m most certainly deleting my account, and leaving, too.
It’s still going to be reflected in certain indexes, and market shifts in general, unfortunately.
Yeah, I mean you’re preaching to the choir here.
I’m in individual stocks (that I think are sane) or assets everywhere I can be.
I dunno why yall are celebrating this.
Elon basically gaslighted retail investors, and managers for your retirement funds, into transferring wealth directly to Elon Musk.
Other index funds were forced to buy SpaceX at the IPO price. They literally had no choice, even if they knew they’d lose money. And this includes any overseas ones with exposure to US indices.
This is not something to celebrate. We all just got conned, even if you didn’t buy SpaceX directly.
Funnily enough, OpenAI signed the open weights letter too.
Sam Altman would do literally anything, say anything to make OpenAI look good. He would sacrifice his grandmother if he could, and I’m not one to say that lightly.
I think this is a more global problem of “people now think they’re experts on everything, and shout about it loudly”
Musk is the classic high profile example, but science denialism among the general populace is the low profile one.
And I hear, “well, they always have.”
I dunno… I don’t think the average person in 1926 assumed they were an expert on medicine or physics or finance or whatever. If a doctor, or physicist, or banker told them something, they’d generally nod their head.
Now? “What do they know!?” The notion is offensive, apparently. Clearly their ten minutes of research on Google or ChatGPT refutes 30 years dedicated to centuries-old scientific bodies.
Or maybe they just misjudged some YouTube or Facebook post as an expert, I guess.
Even if I’m wrong there, and people were always like this, the other half is that they weren’t able to shout it into an algorithm that absolutely loves the controversy it generates, and serve it to billions of people automatically.
NO ONE would see this post of Musk, otherwise.
Its still nothing new though. These photos are from 1980, Bruce Davidson:
https://www.magnumphotos.com/arts-culture/society-arts-culture/bruce-davidson-subway-new-york-usa/
Can you edit the post to the original link, OP?
https://baptistnews.com/article/men-only-gyms-seek-to-protect-christian-men-from-lust/
It’s real, but also more critical than you’d think, and worth a read:
This is what Hambrecht and Parsons are doing with their male-only gyms. They’re spreading conspiracies about morality and traditional marriage that cast women as dangerous. They’re promoting men rising up as warrior-kings who are ready to fight evil, and they’re getting rid of women who they claim are ruining everything.
EDIT:
To join Parsons’ Christian manosphere, it’s a bit more intense. Gym members must sign a covenant called “Rule of Life” that requires such beliefs as “marriage as one man and one woman,” or “two genders, given by God at creation.” And it requires strict adherence to “the behavioral and moral standards of the Remnant Gym membership agreement.” Their website says, “Christian authority is enforced through ownership, contracts and the courage to hold the line.”
It’s real. Published one day ago:
https://baptistnews.com/article/men-only-gyms-seek-to-protect-christian-men-from-lust/
The site isn’t what you might think, though. The article is funny, but also a surprisingly good read:
According to the gym’s website, “There was a season of his life marked by homelessness, addiction and spiritual emptiness.” In another video, he admits, “I had a pornography addiction for almost 20 years.”
Mitch Parsons, founder of Youth Group for Men and Remnant Men in Denver, had a similar experience. “I quit four gyms before I built one,” he says on the Remnant Gym website. “I realized the problem wasn’t the gym. It was that no one had built what I actually needed, or what I wanted.”
I know its easy to laugh, but a “porn addiction” can be serious shit. Terry Crews gave some great talks about how it fucked up his life in all sorts of ways.
Also, it’s healthily skeptical:
Also, despite how often Parsons mentions having nakedness shoved in his face at the gyms in Denver, it’s unclear what he means by that. Gym policies across Denver require their members to wear shirts at all times. And Denver public indecency laws prohibit exposing one’s genitals in public.
This is what Hambrecht and Parsons are doing with their male-only gyms. They’re spreading conspiracies about morality and traditional marriage that cast women as dangerous. They’re promoting men rising up as warrior-kings who are ready to fight evil, and they’re getting rid of women who they claim are ruining everything.
It’s got some good writeups, and spicy ones:
https://baptistnews.com/article/why-do-we-keep-sounding-the-alarm/
As the empirical evidence for the threat white Christian nationalism presents to pluralistic democracy has mounted over the past decade, various insider defenders of conservative white Christianity — and even some mainstream white Christian theologians — have alleged the term is simply a cudgel. Those who qualify as Christian nationalists on public opinion surveys, they argue, only want to see a role for their faith in the public square…
Donald Trump is an abuser. He is an adjudicated rapist and predator. Donald Trump is a grifter. He is a well-known con man who gets away with the con because so many don’t want to talk about it. Donald Trump is a liar unlike any liar most of us ever have known. And there are enormous consequences to his perpetual lying. Donald Trump is literally the most dangerous man in the world, and you want to label me with “Trump Derangement Syndrome” for saying that.
Oh that’s perfect!
You can’t realistically run Kimi (as it’s a 1T+ model), but you’re set. There’s a glut of excellent 120B-300B models for you to choose from.
I’m quantizing MiMo 2.5 specifically to cram into 24GB/128GB as I type this. It’s tight, and will use up most of your memory, but it’s fantastically smart and plenty fast. The quantization won’t finish cooking until tomorrow, but I’ll upload it to huggingface then.
But there are others existing quants would fit, like Deepseek Flash IQ3_S: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF/tree/main/UD-IQ3_S
Or… well, any of these! https://huggingface.co/models?num_parameters=min%3A128B%2Cmax%3A256B&library=gguf&sort=modified
The key for you is to run the ik_llama.cpp fork: https://github.com/ikawrakow/ik_llama.cpp/
It’s specifically optimized for hybrid (CPU + GPU) inference on Nvidia desktops; you’ll get MUCH faster speeds than mainline llama.cpp or anything based on it. It also supports some more exotic quantization type; as an example, I’m quantizing MiMo 2.5 as a hybrid quant, with the dense layers at IQ6K/Q8_0 and the sparse experts as an IQ3_KT “trellis” quant type. This should yield a higher fidelity quantization than a typical Q3 GGUF while taking less RAM, at the cost of taking forever to quantize and a slight speed hit.
Some other quantization types (like the KS or R4 types) are specifically configured to be fast on CPU.
There are specialized “quant cookers” that make GGUFs specifically for ik_llama.cpp, like:
https://huggingface.co/ubergarm
https://huggingface.co/AesSedai
https://huggingface.co/sigargv/Laguna-M.1-GGUF
https://huggingface.co/models?other=ik_llama.cpp&sort=modified
Mine will be here once I upload it:
https://huggingface.co/Downtown-Case
For anyone else reading this: none of this is applicable to you if you have an AMD/Intel GPU, or an older Nvidia GPU, or less than a certain amount of RAM, or a non AVX2 CPU or… well, there’s all sorts of caveats.
The optimal runtime is different for everyone. As an example, exllamav3 is WAY better than llama.cpp on modern Nvidia GPUs until you get above a certain amount of CPU RAM; then using all that RAM for hybrid inference makes more sense. On AMD, different backends work better on different GPUs, and… well, you get the point. Basically all LLM running advice is irrelevant without specifics of your hardware, even this post will be obsolete in a month.