Why tho?
This isn’t the smart way, though.
What the homelabbers do (at least before the RAM crisis) is buy Xeon/TR/EPYC boards on the cheap, and then run gaming GPUs for hybrid inference.
This is what I do. I run MiMo 2.5 at 8-10t/s on a 7800X3D/RTX 3090/128GB CPU RAM, more with Dflash. That’s a 300B model: it’s not even in the same class as Qwen 27B, which is what the dev in OP’s article is trying to run.
And this is small-time: setups with 4-8 memory channels can run stuff like Kimi or Deepseek Pro, even faster. Or they can run smaller LLMs with quantization types that are very fast on CPUs, and get crazy speeds.
…And besides, Qwen 27B can run fine on a 4080, with the right framework. It will fit in 16GB as an exl3.
Not that this isn’t a cool hardware hacking project.
…But it’s kind of the wrong approach. It’s about 2 years out of date, as MoEs are king in LLM land now. RAM is horrendously expensive, yes, but so are most used V100s, or used 3090s.
Sure it’s got a lot of VRAM, but the 4080 has five times the compute power.
That’s fine I just need to display pictures of your mom (they are very large) (/s)
Here I was expecting a graphics demo to blow our collective minds but instead I got a story about a local LLM for cheap. It is <current year>. I should have known better.
Were I the author / tech cobbler here, I’d be concerned that too much time with an LLM, local or otherwise, might erode or dull my apparently fairly sharp reasoning and tech skills. (Clarification: Not my sharpness, theirs. I’m a potato.)
Other thoughts: For a minute I thought this whole thing was a tribute to, or a troll in the manner of, that one Redditor that always spun their stories around to being about their dad beating them with jumper cables.
Also, my old PC developed an issue like the warm reboot problem, except with the network interface. I couldn’t just restart, I had to power off and back on. I never did bother to find out whether it was early signs of hardware failure or whether it was an old hardware / newer kernel mismatch.
eBay has some rad Chinese mezzanine boards for these guys too. Nvlink works and everything lol
The last time I saw a passively cooled GPU was the '90s.
I am using PTM sheets and they idle at decent temps, though I did ziptie some high CFM fans behind them lol
Seems like an awful lot of trouble to save $100 not buying a 5060 Ti that also has 16GB.
I had no idea you could get a 16gb card this cheap! TIL.
It’s pretty low performance, though… My card from 8 years ago nearly matches the passmark rating.
its for running ai models
448 GB/s memory vs 900 GB/s and at a cheaper price.
The article talks a lot trash about AMD and ROCm but vulkan works fine too. In fact from a datacenter GPU standpoint there is an AMD option called the V620 available on US EBay that I was able to haggle to $350, with 32GB VRAM, 512GB/s bandwidth, and runs the same Qwen-3.6-27b at about 20t/s. I would argue that’s even more cost effective.
It requires a few of the same fan shenanigans this guy did but there is no need to pull specific past software versions to make it usable in LinuxThis is nice to know. Thank you.
82db? Wild. I guess they don’t really care that much about noise in data centers though.
They mostly don’t, but this is also not how the datacenters cool them.
A datacenter will either have an open water loop, or an all in one taking heat to a more advantagous place for a radiator to be, or at the very least better managed airflow with bigger fans and more specific air baffles.
This thing has no such luxury and has a small area and unknown broader thermal context, so screaming it is to make up for the limitations of the scenario.
If it doesn’t sound like a jet plane taking off, is it really a server at all?
Hey, my home server netbook can play
jet_plane_taking_off.opus.
No they really dont. Big ass fans running 24/7 to help the small fans running 24/7. It all blends into an easily ignored drone though just dont try to have a conversation in there.
What?
Sorry, I think I might have hearing damage from all this constant noise.
The main thing that itches me with the V100 is the fact that given that pascal is about to be EOL, a 2017 card is probably soon next
About to be? Pascal cards no longer work in TrueNAS scale due to Nvidia dropping driver support
The ampere class is soon to fall out of relevance as well.
deleted by creator
I never expected used DC GPUs would make it to eBay.
V100s are ancient in a market obsessed with the very very latest.
The datacenters are unlikely to bother directly with eBay, but they have asset recovery companies that will take the stuff off their hands and seek buyers, including over eBay.
Lots of used DC gear makes it to eBay. This is how homelabbers survive.
I know that and I use that. But recently market for switches and servers dried up. I thought they were shredding DC GPUs too. Most of them die after 3-5 years anyway.
Is nobody extremely tired of the AI slop articles??
The guy is explaining how to hack around GPU fans which were never meant to be run at lower speed through jumper cables and you complain about slop?
This article ISN’T AI slop.
Is nobody exasperatedly tired of comments calling anything remotely related to AI ‘slop??’
Yes.
Lemmy is just by default super AI-hating. I wonder what they felt when Linus Torvalds said that LLM are actually useful tools.
I don’t really care what Linus Torvalds has to say, unless it’s about the Linux kernel, and I don’t think his opinions matter outside of that. Being a great programmer doesn’t mean he’s also an intelligent, socially aware person.
He did an interview with Linus Tech Tips a few months ago, and when asked about artists being “upset about the large scale theft of work” coming from AI, he said “that’s reality, deal with it” and even went on to talk about photographers that are out of work because “you can fake pictures so much better now” while, within the same breath, saying programmers wouldn’t lose their jobs because he wants to have his cake and eat it too. (Obviously I’m paraphrasing a bit)
I do agree AI can be a useful tool, and it’s a position I’ve held since OpenAI was first starting to become recognizable. But anything that’s generated by AI is slop, and I’ll continue to call it such even if nobody else will.
The movie “Spider-Man: Into the Spider-Verse” used AI tools to place random blemishes and marks (that the artists created) on characters faces across multiple frames. I don’t really feel like finding a source for this one, but I’m sure you could fairly easily if you’re that curious. I think that’s an actual use for AI. Streamlining and speeding up the creation process to give creative people more time to do actual creative work. I don’t think asking an LLM to generate or “create” something for you is anything but slop.
Slop isn’t about quality, it’s about the morality. The, again, large scale theft of work required to get to the point of being able to generate something of a passible quality level, plus the destruction of the environment in the process. I think it’s pointless to reject AI tools entirely due to how prominent and widespread they are, but “creating” something with AI will always be slop.
Source for the LTT interview: https://youtu.be/mfv0V1SxbNA
About 33:10
I think broadly most people are less bothered by what the GenAI can do, but more bothered by what experiences are inflicted upon them by other people using the tech.
All well and good if it can so code reviews and code gen at your behest and you can evaluate and get as much as you think you can out of it, but other people are enabled to be amazingly more obnoxious about it.
“Slop” is mostly because people were inclined to do the slop level quality, but were self limited by the reality their poor ideas were roughly as hard to do as much better ideas. Now AI accelerates people that are not concerned with quality or specifics way more than it can accelerate people that care about specifics and quality.
Many people are incapable of nuance.
I think it has some use cases, but it’s like religion: Don’t cram it down my throat, keep it to yourself, and I don’t mind it if it’s on my terms.
Totally get what you mean and I can fully respect that stance. In recent times, it was more like a “let’s bash all the people who think slightly positive about AI”, so kind of the same indoctrination, just the other way around.
We felt like he’s right about it being useful tools but the ethical aspects are hard to ignore. Yes, many ethical aspects.
You’re likely conflating the tools with the implementation or the implementers. An open source model on local hardware has next to zero ethical concerns.
There are MANY ethical concerns about the actions of the big players in AI, their data centers, and to some extent, the original, bootstrapped training data sourcing.
Note a key point of contention is how the training data is used and whether it is effectively discarding copyright. If you invested time making an open source project to do something people appreciate and you get attribution as a result, you may be unhappy that a model trained on your stuff can let a user prompt up an embedded implementation of what your project does without any attribution.
This pretty much applies to all models. No one limited training data to explicitly public domain stuff.
You might not appreciate it, but if it’s posted online then it’s no different from someone else learning to code from reading the project. It’s not making copies of the code, it’s just strengthening the weights on a neural network. Sure, if the code is so obscure that nothing else is like it then it’s possible to get the model to regurgitate some of it due to having so few relevant sources, but it’s very unlikely to be comprehensive enough that it’s violating any copyright. If a court finds that to be the case, for some fictitious example, then I’m certain they can find an agreeable resolution to the isolated case.
However, none of that is justification for just writing off the technology entirely. Pandora’s Box has been opened. The genie isn’t going back in the bottle. You can’t close the barn door, all the cows already escaped. What do you think boycotting it will accomplish? What exactly is the goal by figuratively sticking your fingers in your ears and pretending the models don’t exist?
I have seen this argument before and it doesn’t make sense even in theory.
I used to work at a company that did open source work and also proprietary work with third party closed source code. The company didn’t let anyone who had seen proprietary code contribute to open source, because they felt once a person ‘learned’ from a proprietary codebase, then it’s too risky if similar looking code lands in a project.
Imagine if someone saw the source code for Excel. Then sometime later they notice that Calc didn’t have a feature that Excel did, and contributed an implementation of the feature. Even if they hadn’t been looking directly at the Excel source code in the moment of implementation, you think Microsoft would be so “understanding” when they see someone that once worked on Excel contributing what could be construed as infringing?
The AI companies also seem to acknowledge this, as they have offerings that promise not to use your proprietary code as training fodder. If it is not a risk of infringement, then why would it matter to promise that the proprietary code is kept out of training data? Though it was short lived, why would OpenAI have even made a deal to license Disney material if it’s all fair use anyway? If this sort of stuff is fair game, why do they get so pissy when other companies distill models?
Even as the AI company’s have roughly defended this scenario, their defense should be a cause for concern for users. Generally they say that anything they do with things they can read is ‘fair use’, and when exhibits of clearly infringing outputs are given, they respond with the model only did that because the user’s prompt directed it, and thus the responsibility for infringement should be with the AI user, not the engine that produced the infringing output. So the possibility of an unwitting infringement is possible as the AI companies explicitly say it’s the fault of the user even if it happens.
But we come to your last point, that essentially at this point, the whole thing is ‘too big to fail’ and thus the practical risk is low. Which is true. It’s just a bit disheartening that these companies are given free reign to interpret intellectual property law whichever way is convenient in the moment.
You’re likely conflating the tools with the implementation or the implementers.
I don’t think so, no?
Okay. So, imagine I’m an indie game developer. I’ve got no artistic talent, no funds, and am just making a game that I want to play, not one I think will make any money.
With me so far? I download a free, open source, open weights model on my laptop. I install the open source tools to allow the LLM to read my project files, and I give it a thorough description of the game I’m designing. I work on some aspect in the foreground, while in the background my GPU is utilized to achieve some task I’ve assigned the AI. When it’s done, I review the work, commit the changes, and assign it a new task.
What are the ethical aspects you can’t ignore in my use case?
deleted by creator
Well, yes and no. Even open source models had to consume a lot of power for the training, including the use of non-owned data
Many, many, many things “consume a lot of power” which is both common and benign. A whirlpool tub “consumes a lot of power” for example. So does an electric stove. And the power can be from any source, such as solar, hydro, or wind.
That’s why I specified there’s minor issues that could be argued about the original data sources. However, that bird has flown the coop. There’s no putting that genie back in the bottle, and no way to undo it. You could argue the company’s responsible owe every single person on the planet some form of restitution, which I do, but I don’t think that qualifies as an ethical concern since it’s universal. No one person was harmed more than another, it was all publicly available data - so the public should benefit from the result. Refusing to use it only hampers yourself, at no detriment to those perceived as wrongdoers.
I’m conflicted about it too. But there is a useful software development tool in there. Its a beige mind good at regurgitating cookie-cutter work. Its pretty good at scaffold and cookie cutter unit tests.
I hate how it makes my colleages behave, but I can totally see the legitimate $500M/yr industry worth of software in there, under all the BS.
There’s a Jetbrains or Atlassian size companies worth of product here.
No.
Ah, you got me. I copied the double negative of the original comment. You’re right, the replies here show there are indeed very many people annoyed by the ‘slop’ comments.
Only bootlickers are
🤣
Oh wait, I know this one: “I know you are, but what am I?”
Name-calling. The last bastion of the stubbornly illogical. I can picture you as a toddler, stamping your feet in a huff as you try to think up more names to call people who disagree with you.
Am I meanie poopoo head, too?
Actually if you’re interested in running a local llm it’s quite a good article
Yes but this isnt one of them.
So much so…
