I bought a datacenter GPU that doesn't fit in a normal motherboard, macgyvered the fan with jumper wires, and now I'm running a model that ties with Claude Sonnet 4.6 on benchmarks, all for £200.
Not sure how well that without work even if you could bridge them properly to share their vram, as the latency from the other gpu will be pretty high compared to the local vram. Frame times won’t be that great is my prediction. It works ok for LLMs because they aren’t a realtime compute task like gaming graphics is. If an LLM takes an extra 0.12s to compute its result, you don’t notice, but if a gpu misses a frame deadline by 0.12s, that’s a stutter that represents less than 10 fps.
I believe that’s why earlier attempts at dual gpu systems mostly fizzled out (plus cost concerns). You can get some pure acceleration if you can fit the entire working memory onto both GPUs’ vram (so your total vram is effectively min( GPUA_VRAM, GPUB_VRAM ) rather than GPUA_VRAM + GPUB_VRAM, though that also requires all pixels to be independent of anything calculated on the other GPU, other than maybe post processing effects that could be handled on whichever GPU is handling the display.
That’s not to say that you can’t get extra performance out of multi-gpu setups that don’t just mirror their RAM, but it’s more complicated than “sum of the capabilities of each GPU”.
Yes, that’s basically what the article is about. They run the LLM across both GPUs.
But that’s a feature of llama.cpp. SLI doesn’t really exist any more, and NVlink requires a specific setup, which the 4080 is not part of (the 3090 was the last consumer one, apparently). So you couldn’t pool the VRAM.
Sure it’s got a lot of VRAM, but the 4080 has five times the compute power.
You’re not getting a 4080 for $200 though…
Hmm… I have a 4080 from 2022. I wonder if I can get this thing too, for $200, and use them both? I hate only having 16gb VRAM.
Would be nice if I could use them both at the same time. Like, make the 4080 recognize the other device as additional VRAM
Not sure how well that without work even if you could bridge them properly to share their vram, as the latency from the other gpu will be pretty high compared to the local vram. Frame times won’t be that great is my prediction. It works ok for LLMs because they aren’t a realtime compute task like gaming graphics is. If an LLM takes an extra 0.12s to compute its result, you don’t notice, but if a gpu misses a frame deadline by 0.12s, that’s a stutter that represents less than 10 fps.
I believe that’s why earlier attempts at dual gpu systems mostly fizzled out (plus cost concerns). You can get some pure acceleration if you can fit the entire working memory onto both GPUs’ vram (so your total vram is effectively min( GPUA_VRAM, GPUB_VRAM ) rather than GPUA_VRAM + GPUB_VRAM, though that also requires all pixels to be independent of anything calculated on the other GPU, other than maybe post processing effects that could be handled on whichever GPU is handling the display.
That’s not to say that you can’t get extra performance out of multi-gpu setups that don’t just mirror their RAM, but it’s more complicated than “sum of the capabilities of each GPU”.
Yes, that’s basically what the article is about. They run the LLM across both GPUs.
But that’s a feature of llama.cpp. SLI doesn’t really exist any more, and NVlink requires a specific setup, which the 4080 is not part of (the 3090 was the last consumer one, apparently). So you couldn’t pool the VRAM.
That’s fine I just need to display pictures of your mom (they are very large) (/s)