Sample Header Ad - 728x90

How to prioritize SSD page cache eviction over HDD with slower speed?

2 votes
1 answer
56 views
I have a large slow HDD and a small fast SSD. This is about reads not [RAID](https://unix.stackexchange.com/q/471551/524752) . My desktop grinds to a near-halt when switching back to Firefox or man pages after (re/un)-loading 12+ GiB of Linux kernel build trees and 39 GiB total of different LLMs on the SSD while I only have 31 GiB of RAM:
$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi        10Gi       2.4Gi       1.0Gi        19Gi        20Gi
Swap:             0B          0B          0B
$ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                     8:0    0   1.8T  0 disk  
├─sda1                  8:1    0   512M  0 part  /boot/efi
├─sda2                  8:2    0   1.7G  0 part  /boot
└─sda3                  8:3    0   1.8T  0 part  
  └─sda3_crypt        254:0    0   1.8T  0 crypt 
    ├─vgubuntu-root   254:1    0   1.8T  0 lvm   /
    └─vgubuntu-swap_1 254:2    0   1.9G  0 lvm   
nvme0n1               259:0    0 953.9G  0 disk  
└─nvme0n1p1           259:1    0   100G  0 part  
  └─luks-...          254:3    0   100G  0 crypt /media/...
$ sysctl vm.swappiness
vm.swappiness = 60
The SSD is fast, so I'd rather Linux evict the SSD's page-cached files first. Its uncached read time is seconds anyway. What should stop is eviction of any file under /usr or /home. My man bash and dpkg -S bin/bash return instantly from the page cache, but uncached they take half a minute after exiting the LLMs. More severely, Firefox needs my ~/.mozilla folder for history and cache; with it uncached, waiting for the address bar to work takes minutes. I am looking for an configuration option. systemd-run could set MemoryMax for ktorrent, but I frequently restart llama-server to switch between the ~6 GiB LLMs, and I don't want a separate daemon to keep the cgroup alive. The man and dpkg problems will be fixed when my / moves to the SSD once I sort out fscrypt fears; in the meantime, /usr on tmpfs would leave insufficient available RAM and overlayfs is too much complexity. The LLM workload could, but shouldn't, remount the SSD as a workaround. That leaves the niced kernel build workload still evicting my web browsing one's cache. I looked in /sys/block but couldn't find the right config. [Cgroups v2](https://docs.kernel.org/admin-guide/cgroup-v2.html) has per-device options but only for parallel write workloads (io.max) not for controlling how sequential workloads affect the cache. A [2011 patch](https://lore.kernel.org/lkml/4DFE987E.1070900@jp.fujitsu.com/T/) and a [2023 question](https://unix.stackexchange.com/q/755527/524752) don't see any userspace interface. Which setting can be used to force the SSD's page cache to be evicted before that of the HDD's?
Asked by Daniel T (195 rep)
Apr 13, 2025, 10:44 PM
Last activity: Apr 14, 2025, 02:07 PM