It’s been a while! Let us know what you’re currently working on, what exploded, what cool stuff you found, or just brag that everything is running fine unattended.
I haven’t been doing a lot, renovate auto upgrades for patches and sometimes minor updates work great so far. I got involved in my union to spin up a mastodon server soon, so there’s a new project, but I already run mastodon so it shouldn’t be much of a deal, I just need to get comfortable with the infrastructure. I also set up ntfy but only sent some test notifications and didn’t configure any app yet - planned for Element Chat.
[Meta] I’m not sure how to apply the new tags. In the recent versions of this thread, people sometimes posted about their ollama setup or the like, but I’m not mentioning AI in the post body (except this meta thingy) and it shouldn’t be much of the discussion, but it’s an open post. Maybe curbstickle can chime in to let me know if I should apply one of them.
[AIT] Well, let’s see…nothing really new on the server. Still waiting for something interesting besides *.arr. I did provision a new tower with the following specs:
This, obviously, will be an local AI machine
Nice. Did you have to sell a house to afford the parts?
LOL Not quite, however, I am choking on the 5090 but as I understand, that’s the better option, and 32 GB seems to be the sweetspot for AI. I actually picked up the tower and original RAM for less than $500 USD.
First of all, I mean zero offense with any purchase decision. A 5090 is very good.
…But if I were paying that kind of money, I’d probably get a 4090 and a new motherboard/CPU instead. Maybe a used DRR4 threadripper system.
Hybrid (CPU + GPU) inference is where it’s at these days. It opens up a whole world of huge MoE models, whereas on an 5090 you are stuck with Qwen 27B.
Having a fast CPU, with lots of RAM channels, with full PCIe bandwidth is much more important for that than having a 5090, where a 3090 or 4090 will get the job done.
Even if pure speed is your primary concern, you can tune a sparse 120B model (like Laguna) to run almost as fast as Qwen 27B on a 5090, and get at-least-good results.
It’s more finicky and involved, though. For sure.
Running an LLM on a 5090 is a task. Hybrid CPU + GPU inference is a hobby.
Why you do this to me? LOL Injecting your options. But seriously thank you for the advice. I’m really green in the AI arena. So I’m trying to feel my way around, trying not to spend money on equipment I’ll regret later.
Be aware that you pay a significant performance penalty for going over occulink.
The hit modest for fully offloaded dense models (like Qwen 27B), but dramatic for hybrid inference of big MoEs.
Even my old 3090 got a noticeable performance gain going from a PCIe 3.0 x16 riser to a PCIe 4.0 one.
And that’s why it’s tentative. The reason I’m leaning towards the dock is that, while the 5090 will fit, physically in the case, I am worried about heat dissipation. Even now, when I fire up a moderate model and run it through some paces, I can see the temps rising fairly quickly. So, if I’m going to drop that kind of cabbage on a 5090, the one thing I don’t want to do is watch it turn into a shrink-i-dink.
Duct it!
I have a 400W 3090 with zero case fans.
Its always cool, because I bought like $10 of weather stripping to “seal” its intakes against the edge of an SFF case. So it’s always sucking in ambient air:
You can also severely undervolt a 5090 to like 250-300W, with very little performance impact. Honestly its stock speeds are kind of crazy.
…But again, you aren’t gaining much over a 4090. A 4090 + DDR4 threadripper would be way faster than a 5090 + occulink, for many reasons. And TR CPUs are pretty reasonably priced compared to GPUs these days.
But even if you do go 5090, I’d highly recommend finding some way to shove it in the case and duct some air into its intakes. Its going to be way faster on a PCIe slot.
You could even get a riser and mount it somewhere else in the case, theoretically.
Shit I deleted my post. Yes I have been enjoying some greenery so you just have to hear me out. I have an old countertop ice maker that stripped it’s gears. The refrigeration loop is fine. Pretty wild how small it is. I was wondering…what if I provisioned some brackets on which to mount a muffin fan of unknown CFM, to blow across the coils thru several sheets of mesh of varying gauges to collect moisture drained off to an unknown location, and duct that indirectly into the computer, like a miniature Liebert server room cooling unit. I worry about the moisture tho. Constructing a desiccant wheel seems improbable. I could reheat the cool air upstream or mix warmer air in with the stream to so the outlet air doesn’t stay at a very low dew point and “rewet” downstream components. Perhaps with a small filament heat coil. HMMmmmmm
Shit I deleted my post. Yes I have been enjoying some greenery so you just have to hear me out. I have an old countertop ice maker that stripped it’s gears. The refrigeration loop is fine. Pretty wild how small it is. I was wondering…what if I provisioned some brackets on which to mount a muffin fan of unknown CFM, to blow across the coils thru several sheets of mesh of varying gauges to collect moisture drained off to an unknown location, and duct that indirectly into the computer, like a miniature Liebert server room cooling unit. I worry about the moisture tho. Constructing a desiccant wheel seems improbable. I could reheat the cool air upstream or mix warmer air in with the stream to keep the so the outlet air doesn’t stay at a very low dew point and “rewet” downstream components. Perhaps with a small filament heat coil. HMMmmmmm
deleted by creator