I have noticed that my home server is strangely using lots of swap (~5 GB), despite having only a few lightweight processes running and loads of RAM installed (32 GB).
Upon configuring Grafana + Prometheus, I noticed a trend where cache + buffer will progressively increase until swap starts to be used. My system and services combined will use ~8 GB RAM. Upon rebooting, the cache + buffer will start anywhere from 3–10 GB, progressively ramp up to ~25 GB in 1–2h, where swap will start to be needed (~3 GB). See the image attached for reference.
My swap filesystem is on an expensive (to me) SSD, and I would like to reduce its wear by as much as possible. I understand that swap can introduce only minimal wear on SSDs depending on its nature and that it can be harmless, but I am still not sure what is causing this behavior (and why) and whether I should worry about it or not. So I figured I should investigate what is happening here.
My main question is, how can I figure out what is causing this behavior? Is it expected? I am looking for guidance from others who are more experienced than me in the topic.
A little bit about my system:
I am running Debian 12 on an NVMe SSD containing the root partition (btrfs) and docker services. I also have two HDDs, one with persistent data (ext4), and the other with backups (ext4). This is majoritarily a single-user machine. I tried using the following kernel parameters, but it hasn’t helped:
vm.swappiness=10
vm.vfs_cache_pressure=200
My docker services are:
- *arr stack
- jellyfin
- nextcloud
- immich
- open-webui + ollama
- pi-hole
- invidious
- romm
- nginx proxy manager
- grafana + prometheus
- other minor services that I don’t think are doing much (uptime-kuma, stirlingpdf, vaultwarden, etc)
I agree that hitting swap is not ideal, but messing with swappiness sysctl is almost never the answer and very often leads to more problems than it solves.
If you want to tune for workload without needing fairly deep understanding of how Linux manages memory, use a sysctl that sets a whole system behaviour like CPU governor presets.
Not really a matter of opinion, and not true since kernel 5.1 when ssds became first class citizens.
I mean… it’s an issue. OP’s system is swapping the SSD when it clearly should not be doing it. My system had horrible usability issues in some scenarios without a lot of reconfiguration, and thrashed my SSD for no benefit.
The default heuristic is not good for our systems, and OP’s situation is a pretty common scenario. That’s a fact. How is that not an issue?
And support for TRIM and whatever else was added for SSD support is not really the issue, here. That’s working fine.
No, sorry. We don’t know enough to say this. What were the options when configuring ext4? What record size is grafana using? Mariadb or postgres? Is there contention with another db? Is there a mounted NFS path in the same ext4 partition with weird block settings? There are lots of legitimate config reasons for op’s problem, but there are so many parts of op’s setup we don’t know, we similarly can’t possibly know what the issue is.
Swappiness works as intended in most default scheduler configurations and seeing high swap is a symptom, not an issue. It is almost always more productive to find out why the swapping is happening.
Not sure what you’re getting at with TRIM.