I used to think they were bots. I still do, but I used to, too.
I used to think they were bots. I still do, but I used to, too.
First a caveat/warning - you’ll need a beefy GPU to run larger models, there are some smaller models that perform pretty well.
Adding a medium amount of extra information for you or anyone else that might want to get into running models locally
If you look at https://ollama.com/library?sort=featured you can see models
Model size is measured by parameter count. Generally higher parameter models are better (more “smart”, more accurate) but it’s very challenging/slow to run anything over 25b parameters on consumer GPUs. I tend to find 8-13b parameter models are a sort of sweet spot, the 1-4b parameter models are meant more for really low power devices, they’ll give you OK results for simple requests and summarizing, but they’re not going to wow you.
If you look at the ‘tags’ for the models listed below, you’ll see things like 8b-instruct-q8_0
or 8b-instruct-q4_0
. The q part refers to quantization, or shrinking/compressing a model and the number after that is roughly how aggressively it was compressed. Note the size of each tag and how the size reduces as the quantization gets more aggressive (smaller numbers). You can roughly think of this size number as “how much video ram do I need to run this model”. For me, I try to aim for q8 models, fp16 if they can run in my GPU. I wouldn’t try to use anything below q4 quantization, there seems to be a lot of quality loss below q4. Models can run partially or even fully on a CPU but that’s much slower. Ollama doesn’t yet support these new NPUs found in new laptops/processors, but work is happening there.
It’s a good thing that real open source models are getting good enough to compete with or exceed OpenAI.
I like the game, but agree with the over-tutorialed complaints. They have two difficulty modes, I wish only story mode got all the handholding. I think there’s enough obvious indicators to get you through all the game mechanics.
Coming from c# then typescript and nextjs, rye feels very intuitive and like a nice bridge / gateway drug into python.
Really love arch and the AUR. I’ve been tempted to get nix set up for the rare cases when there’s no AUR package or the AUR package is unmaintained. I figure if there’s no package in the AUR or nixpkgs, it’s probably not worth running.
btop reports some gpu, network and disk information that I don’t think shows up in htop, feels a bit more comprehensive maybe? Both are fine, but I too use btop, it’s nice.
Random trivia: I think btop has been rewritten like 3-5 times now? It’s sort of an inside joke to the point that someone suggested another rewrite from C++ to Rust ( https://github.com/aristocratos/btop/issues/5 ). I guess the guy just likes writing system monitoring console apps.
MAWP - Archer
Taking ollama for instance, either the whole model runs in vram and compute is done on the gpu, or it runs in system ram and compute is done on the cpu. Running models on CPU is horribly slow. You won’t want to do it for large models
LM studio and others allow you to run part of the model on GPU and part on CPU, splitting memory requirements but still pretty slow.
Even the smaller 7B parameter models run pretty slow in CPU and the huge models are orders of magnitude slower
So technically more system ram will let you run some larger models but you will quickly figure out you just don’t want to do it.
FWIW they didn’t merge it, they closed the PR without merging, link to line that still exists on master.
The recent comments are from the announcement of the ladybird browser project which is forked from some browser code from Serenity OS, I guess people are digging into who wrote the code.
Not arguing that the new comments on the PR are good/bad or anything, just a bit of context.
Been 100% linux for like 6-9 months now, these stories make me thankful for finally making the switch.
I’ve tried to make the switch 3-4 times in the past and was stopped by 2 main things:
The experience was so much better this time and I really have no regrets. I don’t imagine I’ll ever run Windows again outside of a VM
Hopefully you’re only forwarding the minimal set of network ports and not all ports/traffic? If so then you’re good, like someone else said if you’ve got a router and it’s forwarding selected traffic then no need for anything else
And what about Dr Pepper Brisket?
Elon “Nick Cannon” Musk
Synology nas are nice. I will say there’s definitely a nice UI there and they generally work well. But there is a good bit of lock-in and there are some really reasonable roll-your-own hardware and software options these days.
If you want something that just works, doesn’t need to be super configurable and is easiest to set up and manage, get a synology. If you don’t mind putting in some work or if you need to really tweak some stuff, roll your own
Most steam games just work. Make sure to go to settings and compatibility and let it use compatibility for all games. Look at something like bottles for a front-end to let you set up and use wine / proton for other launchers, etc….
Rip up the Reddit contract and don’t use that data to train the model. It’s the definition of a garbage in garbage out problem.
Fwiw they’re able to do the same thing by the sound of someone typing a password across the room. Not advocating for fingerprints or anything, just these exotic hacks are everywhere
If you go, definitely stay at Four Seasons Total Landscaping next door, best accommodations around and their convention spaces are great for any press conferences you might need to hastily put together.