Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or register to comment.
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
It depends on your hardware. But right now the “cheapest” hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.
I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.
strix halo (128GB)
googles
Jesus christ
Just get the cheapest OEM, Bosgame M5. I know “cheapest” is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don’t want to do AI.
Why wouldn’t you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
I don’t trust it to be accurate; I’d have to reference the sources it pulled from. Extra work for little payoff.
Got it, makes sense. You’d probably have to do some benchmarking first. Depending on how large the source catalog is.
Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I’m also set up with tailscale.
Even ignoring all of the functional issues with LLMs/“AI” related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I’m tired of the wildfires and extreme storms from climate change).
So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.
The server has tailscale installed, so yes I can access it anywhere.
Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it’s output into Claude Pro for vetting and it hasn’t found any flaws in the simple scripts and things I’ve asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that’s all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I’m just a little guy.
Qwen 3.8 27B, running on a RTX 5090. Works so well.
i run k2 7b, it’s delightful!
I tried, but didn’t want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.
I run the AI service (OpenwebUi) myself, but buy the compute on OpenRouter. Self hosting models is not an economic viability for me.
Does OpenRouter have full access to your prompts? What is their privacy policy?
They have a flag Zero Data Retention that cones with the promise that after a call no data is kept on any servers. I’m using that.
It’s contract level trust; I wouldn’t bet my life on it, but overall it’s a compromise I’m willing to make.
I’m building toward exactly this, so take it with that bias. The cheap tier works if you lower expectations. An N150 mini PC with 16GB runs small models through Ollama fine, but 8B models are slow on that class of chip, and you’ll feel it on anything chat-shaped. Where it shines is async work: summarize this, tag that, answer when it’s ready. For phone access I’d keep it LAN-only and use WireGuard or Tailscale to get home rather than opening a port.
Disclosure: I’m working on a small open-source home box that bundles local AI with offline Wikipedia and media. What would make you trust a box like that over a setup you built yourself?
90€/mo to self-host something? Lol
That infographic looks to be for a VPS deployment, which is where the cost would be coming from. They’re comparing to a $9,000+ OpenAI payment.
I’m new to Lemmy so when I tried to post something it said I need an image so I just picked a random one which looked vaguely like self hosting AI
You don’t need an image, and the one you picked makes your post look like an advertisement. You can edit your post to remove it.
I wish I could. I don’t have the hardware for it.
We did the maths. Didn’t seem worthwhile and the payback period was a very long time
I think the value is in the privacy. You can hook it up to knowledge and capabilities you’d never share with a cloud AI. But if your use case is strictly coding, I agree.
True but if you happen to buy broken macs you can just take thier board with all the unified ram it’s become quite the thing recently even Asus duo with 32gb Intel 9 core ultra board’s have been started to be repurposed like mini pcs for ai use. For about 1000usd get the same level of intelligence as o1 high with gemma 26b I think with enough left over for kv cache.
I self-host my own neural network, all within the confines of my own skull.
I’m just glad that this is both Top & Hot comment. 🧠✊
That 33% downtime though
The downtime isn’t the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there’s a context change).
Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.
It’s better than github
It’s scheduled maintenance so it doesn’t count toward downtime. Besides, I’m not trying to use it then anyway.
Scheduled most of the time, anyway.
How much RAM is it using?
Made me think of Johnny Mnemonic. https://www.youtube.com/watch?v=jS2GFqT5ycU
Iirc in a documentary, the brain runs on just 13w of electricity, so the more advanced and efficient AI gets, the less power and therefore resources it’ll need. Maybe Johnny’s world is just that advanced lol
Me also, and I can barely afford to run that
I have a Qwen model running on a Mac Mini with Ollama - it’s reasonable for Home Assistant, responding to iMessages and similar.
