In my production systems, and in my home systems, and in my gaming computers, I turn swap off. There’s no good reason to wear out disc writes. In production systems, I’d rather have something fail hard, then get slower and slower and slower falling behind.

Quite frankly, there’s no good reason to have swap on anymore, memory is huge. Most programs don’t need as much as they use. You can set memory pressure in systemd, telling programs to reduce their usage, and giving them individual hard cutoffs, so something like Prometheus can’t take down your whole system just because it’s got a memory leak.

I didn’t look too closely at details, but generally that’s relatively normal the more individual applications you have running, especially with the overhead of them all being virtualized. Often the overhead of running that many virtualized operating systems can be problematic depending on how the container author chose and configured the container’s OS. A lot of people seem to just randomly pick an OS for their containers just so they can distribute their application with docker as easily as possible, without considering the consequences.

Here are the options off the top of my head if disk wear is your primary concern:

1. Turn swap off at the host level. Note, though, that you may end up in bottlenecks if multiple of them do very memory intensive tasks at the same time, like image and video processing, and docker can’t swap out the VMs that are idle to the disk and be able to use the RAM fully for those active tasks. For some of those apps you can configure when they do maintenance tasks and reduce the likelihood.

2. Look at each individual VM in docker and see which are using the most swap and force docker to limit their allowed resources. You can even tell docker to make an application not use swap by setting the memory and memory-swap values to the same value or reduce their swap usage by setting memory-swapiness to a low number, or you can install those applications to the host system and configure the host to reduce swap or disallow swap usage through systemd or whatever rather than running them in docker. You may also find one of the applications you’re running has a memory leak issue. Good OSs often handle that, but who knows what IS the container is running and the docker layer is concealing from the host kernel and not handling quite as well as it being installed on “bare metal” might. Virtualization adds a lot of variables for these kinds of low level resource control scenarios. So just trying it on the host directly might reveal something if you’re interested in digging like I usually am, lol.

3. Get a second, small, inexpensive drive and set that up as the swap partition for the host.

4. Look into what database systems these are using. Often if they each are running their own separate DBMS containers, the DBMS maintenance overhead can cause some additional churn. I have a single instance of PostgreSQL and one of MariaDB running on a separate server that all of my applications connect to. Some of these applications may be not only spinning up a whole VM for the application, but an entirely other VM for their database server. Though I think some of these probably use sqlite which is just a file, I think manynof these use PostgreSQL and/or MySQL/MariaDB by default. This also helps because I only need a single backup for all of my databases. The negative is that if that DB server goes down, then all the apps go down too, but if you already have everything on one server that might not be as big of a deal. I have a lot of smaller servers running smaller subsets of applications based on their resource needs. Helps me conserve electricity rather than running one or two big servers that are almost always highly active. But I enjoy optimizing that kind of stuff, it’s tedious.

Seeing high cache and buffer numbers is usually a good sign, as it means the system is making full use of the memory you have. It’ll automatically deallocate it if something actually needs the memory, but until then it’s using it to store data you might need.

Swap’s a funny one, and whilst you can tune it to an astonishing degree, the decisions the kernel makes aren’t always what you’d want. With this little actual memory usage, compared to the installed RAM, you might be able to run without swap at all, unless you want something like hibernate. If you want to test that, you can run (as root) swapoff -a, which should disable all swap devices, pushing needed pages back in to RAM, but only until you reboot. If it’s stable, you can consider removing the swap partition(s) later.

The amount of swap used is not a good indicative, you need to check if there is a big exchange of data per second/minute. This is the only indicative of an out of memory system.

Sometimes, some regions of data memory “age” in ram without any access for a long periods of time, the the kernel here has two options, it could destroy the region knowing it could recreate it when needed (with some cpu overhead) or moved this to a swap file when the ram structure already in the swap file and release than section.

Which regions are good candidates for this? Buffers, specially in the fs, code region used for processes or even data sections of a long sleeping process…

Checking your data, if those 5gb are created over a long period of time I would not care a lot about it. Remeber how big the swap is, isn’t that important vs real traffic (in or out) to it

Yeah that’s completely normal, looks absolutely fine to me. For more info on how linux uses memory I suggest reading https://www.linuxatemyram.com/

To add: that link suggests looking if your swap usage is changing. Linux is quite happy to stuff some memory that isn’t actually used in swap. If it was right about it not being used you should see the swap usage not changing much, so your SSD isn’t really being used either apart from the initial swap-out. If your swap usage keeps fluctuating wildly there is more going on which might warrant further investigation.

IMO, the easiest way to make your system not swap into some disk you don’t want it to is to not enable any swap partition on that disk.

If it’s using that much RAM for cache, it will survive not having a swap with a minimum impact on performance.

That’s low utilization. If your ram was fully used and stuff was being frequently stopped, that’s when I would be concerned (and when you would experience wear on your SSD).

Right now you have less than a quarter used, and a couple things the kernel decided to swap out. Perfectly normal.

As a suggestion, ditch ollama and setup llama.cpp. it will work fine with openwebui and it’s much more efficient. (Unrelated to the ram/swap issue)

You can enable zram and remove your on disk swap file if you prefer to spare the SSD write cycles…

That is expected and a good thing.

Free ram is wasted ram, so every file touched by any service will be cached and cache will stay there until ram is needed. This means that over time cache and buffers will always grow, because any subsequent access to those will be faster. The moment more ram is needed by programs, those buffers and cache will be freed immediately.

This is also the reason why some swap will be used even if there is free ram. Some stuff and pages can be swapped out without impact on performance.

Look at it this way: free ram is a wasted opportunity to increase performance by caching some file from disk.

No point in messing with swappiness settings, trust me. You want to address the reason this happens in the first place, not disable to reduce swap.

I’ve had a nearly identical scenario at work. The reason? A data verification process that went through 100s of thousands of files to run their checksums and check integrity. This was working as intended, but the system alerts were firing because of that high buffer/cache utilization. So check which processes have highest IO and you might find the answer. Swap is just a side effect.

Huh…

  System load:  1.26                Temperature:           57.0 C
  Usage of /:   20.4% of 454.86GB   Processes:             527
  Memory usage: 44%                 Users logged in:       1
  Swap usage:   0%                  IPv4 address for eno1: 192.168.1.190

Ubuntu 22.04.5 - 32 GB RAM - 53 total containers

cat /proc/sys/vm/swappiness: 60 (pretty much default)

cat /proc/pressure/cpu: some avg10=0.68 avg60=0.62 avg300=0.67 total=960097774
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
cat /proc/pressure/memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
cat /proc/pressure/io: some avg10=1.16 avg60=4.88 avg300=3.85 total=2831425045
full avg10=1.11 avg60=4.54 avg300=3.58 total=2653626377

I disagree with most here.

You need to reduce swappiness even more, and tweak some other variables (like the memory “threshold” Linux starts to swap, and disk cache aggressiveness) to stop Linux from swapping so opportunistically under such a light load.

IMO, Linux is configured for “old” systems by default: slow HDDs, and constrained RAM pools, where disk IO really, really needs caching, and where idle background processes take a large fraction of RAM.

You have superfluous RAM for your workload. And very fast disk IO that isn’t such a hindrance to apps anyway. And a disk you don’t want to wear. This is the opposite scenario: you don’t want Linux to swap unless it absolutely has to.

For reference, this is part of my config. It’s rather niche and you probably shouldn’t use it, but you should consider looking up the variables:

# Keep min reserve reasonable for 8GB usable space
vm.min_free_kbytes = 262144        # Lower absolute minimum to 256MB

vm.watermark_scale_factor = 10     # Lower to 0.1% (which is ~128MB on 128GB)

# Disable watermark boosting completely
vm.watermark_boost_factor = 0

# Normal-ish metadata pressure so desktop doesn't stutter on disk reads
vm.vfs_cache_pressure = 120

# Allow reasonable swapping of inactive anonymous desktop pages, could be lower
vm.swappiness = 10

# Dirty bytes limits to limit caching
vm.dirty_background_bytes = 67108864
vm.dirty_bytes = 268435456

# Disable compaction & proactive scans to stop freezing with large portions of RAM mlocked
vm.compaction_proactiveness = 0
vm.compact_unevictable_allowed = 0

vm.page-cluster = 0 # 4kb pages for SSD

My system still uses RAM as disk cache with this config, it just won’t go out of its way to swap just to keep that cache, especially I lower swappiness to 1-3.

I also have a 1GB zram pool, prioritized over ssd swap. But you should make yours even larger (maybe 4GB?). This will intercept anything that does swap first.

I also start some applications with systemd-run and specify memory caps and swap limits (often forbidding them from swapping entirely).

It makes a night-and-day difference for some workloads on my system, that would otherwise swap pointlessly, just tank performance and even de-stabilize the system.