Dynamic memory allocations (kmalloc, vmalloc, ...) in the I/O path of known block drivers
0
votes
0
answers
82
views
I was thinking about proper *swap* solutions for a Linux system, and a very important question arised regarding all of them. As known and confirmed by my experience, a swap device *will* eventually be accessed under an extreme memory pressure situation.
In such cases, an OOM kill is hypothetically near, but actually, since
overcommit_memory
=2
and enough swap space is still *available* in system — the OOM path is unreachable. What happens instead - is that kernel thinks it can swap memory out when needed, but "when needed" is too late. Swap I/O gets submitted to the blockdev driver when the memory pools are already absolutely exhausted. Let the block driver attempt any allocation which is more than some absolute minimum, and the system will be dead locked on memory. Unfortunately, it doesn't even get detected, and the System deadlocked on memory
panic path is never taken. Instead the system freezes forever.
I attempted many experiments to try to track the problem down. I have put code to print stacktrace where block I/O is happening, to find which alloc
-family functions are in the path for popular devices (dm-*
, loop
+nfs
, and other configurations), and how much they allocate.
Yet it didn't help me.
I experience the issues **most often on the "loop dev
+ some FS" setups**, but eventually I have also hit it with dm-vdo
(which is a deduplication and compression device), and few else, when using them for swap.
I thought that the problem could be solved by increasing swappines
and min_free_kbytes
values in vm.\* sysctls, but I still hit the described problems. I had an idea of replacing all the alloc
-like calls with on-stack allocations in the needed I/O path code (since I have 128kb kernel stacks), but that seemed too unclean solution (think, resolving conflicts on pulls from upstream), and not applicable everywhere.
My question is: which of these device drivers rely on (heavy?) dynamic memory allocations in the I/O path?
**Which are known to work fine in the Swap role?
What considerations should be in mind to set up a reliable swap device? Can these devices be stacked on top of each other as much as needed?**
- dm-crypt
- dm-verity
- dm-linear
- loop (--direct-io
) + ext4/fat32/?
- loop + nfs/fuse
- loop (--direct-io
) + btrfs/zfs/bcachefs/?
You can mention personal experience, of course.
Asked by melonfsck - she her
(150 rep)
Nov 12, 2024, 10:14 PM
Last activity: Nov 12, 2024, 10:21 PM
Last activity: Nov 12, 2024, 10:21 PM