I usually connect with my server via ssh in a terminal and run basic commands. What’s a better, more efficient and modern way of doing that? Especially considering ai and documentation along the way? I wonder if there’s a better approach than “connect from remote and act local”. Is there a method to “code local and push to remote”?
I use a fedora server with podman, caddyfile and vi.
Local ansible playbook, easily replayable and documented. This in a git repo and you’re fine.
Maybe a bit more on the overkill side, but I use Talos Linux (declarative K8s distribution), then Terraform for initial setup and FluxCD for everything else (including VMs via Kubevirt). It’s a hell of an initial learning curve, but afterwards I just do my talosctl upgrade and upgrade-k8s from time to time, and don’t have to worry about anything else. Upside is, Kubernetes provides a unified, extensible API for everything, for example:
- reverse proxy via Ingress or Gateway-API
- firewall via NetworkPolicy
- even databases like Postgres through an Kubernetes Operator like CNPG, with a similar simplicity as with the big cloud providers (just a single yaml file with the specs like storage capacity, CPU and memory limits, backup target and schedule etc.)
- it is very scriptable (everything is managed via the API)
- automated image updates via either FluxCD itself or just dependabot/RenovateBot creating PRs to your GitOps repo
Downsides:
- you have to figure out persistent storage (CSI), which is slightly more complicated than just a single filesystem and manually specifying volume mounts like with docker, but if done right, you have a simple interface with powerful capabilities via the Kubernetes API
- there are simple CSI implementations that just expose node local disks via LVM or ZFS, so if you just have a single node, or don’t care about replication, it’s fairly easy to get started
- initial learning curve is quite high, especially if you have no prior experience with container orchestration in general
- definitely overkill if you just want something simple that “just works”
I assume this is on your own physical hardware? What parts do you set up with terraform?
Yes, my own hardware. But before I decided to build my own server last September, I was on a similar setup on a dedicated server at Hetzner, just less powerful (I’m so goddamn glad I bought the memory and storage back then, I probably wouldn’t even get the 192GB DDR5 right now for the price I paid for all memory and storage combined).
Terraform starts at the Talos initial machine config and cluster bootstrap including CNI setup and routing configuration (I have an opnsense in front that handles ingress routing) until FluxCD is fully set up and takes over.
Ingress traffic works via a small Hetzner VPS, and BGP through Wireguard, so there is no DNAT or SNAT between the open Internet and my Kubernetes load balancers and all services see the actual client IP. This took a shit load of time to get right, because wireguard only has an mtu of 1420, but the regular uplink is usually 1500, so pmtu discovery in my load balancers implementation (Cilium) and mss clamping in opnsense had to all be configured
Tailscale service configs for HTTPS ingress and DNS resolution. Podman quadlets / podlets for everything except backup software and tailscale itself. I use the open version of VS code to edit config files in place, but I push them to a private git as well as having them backed up. Storage for app data, media, and configs (everything but OS) is via NFS shares from my gaming PC’s RAID volume. I can run a plenty of apps on an old thinkpad this way. After you configure the first few quadlets with tailscale, it gets real straightforward, but there is a learning curve. Yes, I use SSH, but it is the remote conmection and terminal in Code.
Ansible.
Usually I put handcuffs on her every night, to keep her off of me, and only feed her the highest grade sushi and meat. It keeps her hair shiny, keeps good health, and complaints down to a minimum. I make sure she gets adequate sleep on an expensive mattress while I sleep on a 2 inch futon on the floor.
Oh wait… wrong kind of server.
I ssh into it twice a year to perform updates but otherwise i leave it alone. Server’s been chugging along for like a decade now.
Similar, but you might want to update more frequently with the huge number of critical security bugs being discovered by AI recently
I have unattended upgrades configured
Hmm I need to figure out how to do that
I spent a while getting Ansible to be able to setup and maintain my server(s). And ended up not using it as much as I should, mostly because the computer I used to run Ansible from became a “Steam Machine” so I’m rarely sitting on it with a keyboard now and I don’t want to have personal info and keys on my work computer.
But it was a good learning, and useful while I used it. Now I just use ssh and compose files directly.
self hosted gitlab. each server has a repo with all the docker configs and application configs. also have any scripts for the server itself on there as well, things like required libs, network configs, etc.
I also have a repo dedicated for let’s encrypt SSL that retrieves new certs every month. then on each server is an install script scheduled that pulls the certs down, installs them, and restarts any services automatically.
should anything go wrong, I have a siem monitor that will alert me that a service failed to start etc.
currently running four servers like this with varying degrees of complexity.
I have a git repo with all my docker files, config, notes, etc. I have a main overview file covering the overall system and then a directory for each machine. I edit things on my local machine, then scp and ssh to the various remotes. Once I’m happy with my changes, I commit and push to my Forgejo install on my NAS in case my main computer fails. Secrets, passwords, and keys are in my Vaultwarden.
I don’t use AI for it, but a local model could probably give me the commands to do whatever I wanted based on the repo.
My target is terraform to provide VMs on my miniPC and ansible to configure them. For my nas, probably a basic distro then some ansible stuff to setup stomate.
I’ll take the maverick and share my niche position. I did most if not all my proxmox setup and configuration via complete reliance on LLMs. Now, before anyone says “why would you do that”, and I’ll be straight up. I was high for all of it, and not a small blaze I’m talking regularly stoned for months. And I’ll say it “got me by” for the frame work. It worked? Sorta. Music, Plex, immich, docker, arr stack, podman. Now, I can’t say I fully understand it still and I’ve done a lot of changes since that point last year and I figured it out by asking the model to backtrack what we did and I wont lie it helped me better learn how to question the models for my job, and It did decent documnetation over my server. But again, I was literally high doing it
I have a kubernetes cluster inside of Proxmox vm’s, generally I write a deployment, commit it to a git repo and then use argocd to deploy it in the cluster.
This is not something I would advise to anyone but the clinically insane, but for me there is a certain zen in having everything in git like that. It’s for me the only way I can manage the 60ish applications I run without it having a permanent spot in my brain.
A mix of prayer and bash scripts.
Remove bash scripts and add poorly configured logging and that’s me!
At least you have logging!
It’s good to get comfortable in the command line, including over ssh.
Especially considering ai and documentation along the way?
If I am going to ask AI or Google about something, I just do that to help me find the answer and then apply the answer myself. That way I learn, and can double check the AI isn’t hallucinating, at least on in obvious ways.
As for documentation, I keep a notes folder with detailed notes on manual configuration I’ve done, how and why, and the things I’ve learned along the way. I’ve found it’s both useful to help remember the things I’ve learned, and it is useful to go back to refer to.
I use Gitlab for all configurations ans then I have a Gitlab runner in my K3S cluster so I can deploy locally for free
