swap on zfs blocks system

Over the past few years I have had two laptops exhibiting the same behaviour. One was fixed by re-installation. The other is currently broken. The issue is that as soon as the laptop starts swapping it feels as if it freezes, when sitting at it. Remotely you can see it does not freeze completely: Both run swap on LUKS encrypted partition with zfs on-top. The new system has an additional 8 GB swap, that works as expected: The freezing starts when the 8 GB is used. ~~~ $ cat /proc/swaps Filename Type Size Used Priority /dev/zd16 partition 31457276 3816 -3 /dev/dm-1 partition 8388604 494612 -2 ~~~ top says: ~~~ top - 15:48:30 up 22 min, 4 users, load average: 24.41, 8.42, 4.94 Tasks: 436 total, 13 running, 352 sleeping, 71 stopped, 0 zombie %Cpu(s): 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 62.2 total, 0.4 free, 61.9 used, 0.1 buff/cache GiB Swap: 38.0 total, 29.7 free, 8.3 used. 0.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22433 root 0 -20 0 0 0 R 82.6 0.0 0:17.96 zvol 21398 root 0 -20 0 0 0 R 80.2 0.0 0:18.93 z_wr_int 21395 root 1 -19 0 0 0 R 65.3 0.0 0:13.66 z_wr_iss 22420 root 0 -20 0 0 0 R 60.2 0.0 0:20.14 zvol 445 root 1 -19 0 0 0 R 56.2 0.0 0:25.17 z_wr_iss 21396 root 1 -19 0 0 0 R 45.1 0.0 0:13.17 z_wr_iss 1720 root 20 0 0 0 0 R 1.0 0.0 0:00.21 txg_sync 19644 root 20 0 0 0 0 D 0.8 0.0 0:04.56 kworker/u8:3+even+ 19788 root 20 0 0 0 0 R 0.8 0.0 0:05.43 kworker/u8:6+flus+ ~~~ dmesg --follow says: ~~~ [ 1476.236032] INFO: task vdev_autotrim:502 blocked for more than 122 seconds. [ 1476.236052] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1476.236057] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1476.236060] task:vdev_autotrim state:D stack:0 pid:502 tgid:502 ppid:2 flags:0x00004000 [ 1476.236068] Call Trace: [ 1476.236073] [ 1476.236080] __schedule+0x27c/0x6b0 [ 1476.236093] schedule+0x33/0x110 [ 1476.236100] cv_wait_common+0x102/0x140 [spl] [ 1476.236220] ? __pfx_autoremove_wake_function+0x10/0x10 [ 1476.236230] __cv_wait+0x15/0x30 [spl] [ 1476.236263] vdev_autotrim_wait_kick+0x4d/0xb0 [zfs] [ 1476.237396] vdev_autotrim_thread+0x704/0x7b0 [zfs] [ 1476.238429] ? __pfx_vdev_autotrim_thread+0x10/0x10 [zfs] [ 1476.239451] ? __pfx_thread_generic_wrapper+0x10/0x10 [spl] [ 1476.239503] thread_generic_wrapper+0x5f/0x70 [spl] [ 1476.239540] kthread+0xf2/0x120 [ 1476.239548] ? __pfx_kthread+0x10/0x10 [ 1476.239553] ret_from_fork+0x47/0x70 [ 1476.239560] ? __pfx_kthread+0x10/0x10 [ 1476.239564] ret_from_fork_asm+0x1b/0x30 [ 1476.239573] [ 1599.259263] INFO: task zvol:385 blocked for more than 123 seconds. [ 1599.259282] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.259286] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.259290] task:zvol state:D stack:0 pid:385 tgid:385 ppid:2 flags:0x00004000 [ 1599.259299] Call Trace: [ 1599.259304] [ 1599.259310] __schedule+0x27c/0x6b0 [ 1599.259323] schedule+0x33/0x110 [ 1599.259327] schedule_preempt_disabled+0x15/0x30 [ 1599.259332] rwsem_down_read_slowpath+0x284/0x4d0 [ 1599.259343] ? __list_add+0x17/0x40 [zfs] [ 1599.260419] down_read+0x48/0xd0 [ 1599.260433] dbuf_dirty+0x4b9/0x730 [zfs] [ 1599.261429] dmu_buf_will_dirty_impl+0xd2/0x1b0 [zfs] [ 1599.262416] dmu_buf_will_dirty+0x16/0x30 [zfs] [ 1599.263402] dmu_write_uio_dnode+0x92/0x170 [zfs] [ 1599.264394] zvol_write+0x250/0x400 [zfs] [ 1599.265386] zvol_write_task+0x12/0x30 [zfs] [ 1599.266378] taskq_thread+0x1f6/0x3c0 [spl] [ 1599.266427] ? __pfx_default_wake_function+0x10/0x10 [ 1599.266440] ? __pfx_zvol_write_task+0x10/0x10 [zfs] [ 1599.267440] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 1599.267488] kthread+0xf2/0x120 [ 1599.267496] ? __pfx_kthread+0x10/0x10 [ 1599.267500] ret_from_fork+0x47/0x70 [ 1599.267506] ? __pfx_kthread+0x10/0x10 [ 1599.267510] ret_from_fork_asm+0x1b/0x30 [ 1599.267517] [ 1599.267529] INFO: task txg_sync:492 blocked for more than 123 seconds. [ 1599.267537] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.267542] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.267545] task:txg_sync state:D stack:0 pid:492 tgid:492 ppid:2 flags:0x00004000 [ 1599.267553] Call Trace: [ 1599.267555] [ 1599.267558] __schedule+0x27c/0x6b0 [ 1599.267564] ? default_wake_function+0x1a/0x40 [ 1599.267572] schedule+0x33/0x110 [ 1599.267577] taskq_wait+0x9c/0xd0 [spl] [ 1599.267608] ? __pfx_autoremove_wake_function+0x10/0x10 [ 1599.267615] dmu_objset_sync+0x30d/0x4b0 [zfs] [ 1599.268611] dsl_dataset_sync+0x5e/0x200 [zfs] [ 1599.269610] dsl_pool_sync+0x9b/0x410 [zfs] [ 1599.270697] spa_sync_iterate_to_convergence+0xde/0x220 [zfs] [ 1599.271737] spa_sync+0x321/0x620 [zfs] [ 1599.272767] txg_sync_thread+0x1e7/0x250 [zfs] [ 1599.273791] ? __pfx_txg_sync_thread+0x10/0x10 [zfs] [ 1599.274846] ? __pfx_thread_generic_wrapper+0x10/0x10 [spl] [ 1599.274895] thread_generic_wrapper+0x5f/0x70 [spl] [ 1599.274941] kthread+0xf2/0x120 [ 1599.274949] ? __pfx_kthread+0x10/0x10 [ 1599.274955] ret_from_fork+0x47/0x70 [ 1599.274960] ? __pfx_kthread+0x10/0x10 [ 1599.274965] ret_from_fork_asm+0x1b/0x30 [ 1599.274973] [ 1599.274979] INFO: task vdev_autotrim:502 blocked for more than 246 seconds. [ 1599.274986] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.274992] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.274995] task:vdev_autotrim state:D stack:0 pid:502 tgid:502 ppid:2 flags:0x00004000 [ 1599.275003] Call Trace: [ 1599.275007] [ 1599.275010] __schedule+0x27c/0x6b0 [ 1599.275018] schedule+0x33/0x110 [ 1599.275024] cv_wait_common+0x102/0x140 [spl] [ 1599.275055] ? __pfx_autoremove_wake_function+0x10/0x10 [ 1599.275064] __cv_wait+0x15/0x30 [spl] [ 1599.275094] vdev_autotrim_wait_kick+0x4d/0xb0 [zfs] [ 1599.276136] vdev_autotrim_thread+0x704/0x7b0 [zfs] [ 1599.277156] ? __pfx_vdev_autotrim_thread+0x10/0x10 [zfs] [ 1599.278171] ? __pfx_thread_generic_wrapper+0x10/0x10 [spl] [ 1599.278224] thread_generic_wrapper+0x5f/0x70 [spl] [ 1599.278260] kthread+0xf2/0x120 [ 1599.278268] ? __pfx_kthread+0x10/0x10 [ 1599.278273] ret_from_fork+0x47/0x70 [ 1599.278279] ? __pfx_kthread+0x10/0x10 [ 1599.278283] ret_from_fork_asm+0x1b/0x30 [ 1599.278292] [ 1599.278312] INFO: task vdev_autotrim:1730 blocked for more than 123 seconds. [ 1599.278319] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.278324] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.278327] task:vdev_autotrim state:D stack:0 pid:1730 tgid:1730 ppid:2 flags:0x00004000 [ 1599.278335] Call Trace: [ 1599.278338] [ 1599.278341] __schedule+0x27c/0x6b0 [ 1599.278350] schedule+0x33/0x110 [ 1599.278355] cv_wait_common+0x102/0x140 [spl] [ 1599.278387] ? __pfx_autoremove_wake_function+0x10/0x10 [ 1599.278394] __cv_wait+0x15/0x30 [spl] [ 1599.278425] vdev_autotrim_wait_kick+0x4d/0xb0 [zfs] [ 1599.279467] vdev_autotrim_thread+0x44e/0x7b0 [zfs] [ 1599.280510] ? __pfx_vdev_autotrim_thread+0x10/0x10 [zfs] [ 1599.281544] ? __pfx_thread_generic_wrapper+0x10/0x10 [spl] [ 1599.281595] thread_generic_wrapper+0x5f/0x70 [spl] [ 1599.281632] kthread+0xf2/0x120 [ 1599.281640] ? __pfx_kthread+0x10/0x10 [ 1599.281645] ret_from_fork+0x47/0x70 [ 1599.281650] ? __pfx_kthread+0x10/0x10 [ 1599.281655] ret_from_fork_asm+0x1b/0x30 [ 1599.281663] [ 1599.281707] INFO: task zvol:4168 blocked for more than 123 seconds. [ 1599.281715] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.281719] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.281722] task:zvol state:D stack:0 pid:4168 tgid:4168 ppid:2 flags:0x00004000 [ 1599.281731] Call Trace: [ 1599.281733] [ 1599.281737] __schedule+0x27c/0x6b0 [ 1599.281746] schedule+0x33/0x110 [ 1599.281750] schedule_preempt_disabled+0x15/0x30 [ 1599.281755] rwsem_down_read_slowpath+0x284/0x4d0 [ 1599.281763] down_read+0x48/0xd0 [ 1599.281770] dmu_tx_check_ioerr+0x3f/0x100 [zfs] [ 1599.282813] dmu_tx_count_write+0xe3/0x1d0 [zfs] [ 1599.283850] dmu_tx_hold_write_by_dnode+0x3a/0x60 [zfs] [ 1599.284880] zvol_write+0x225/0x400 [zfs] [ 1599.285892] zvol_write_task+0x12/0x30 [zfs] [ 1599.286921] taskq_thread+0x1f6/0x3c0 [spl] [ 1599.286976] ? __pfx_default_wake_function+0x10/0x10 [ 1599.286988] ? __pfx_zvol_write_task+0x10/0x10 [zfs] [ 1599.287997] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 1599.288047] kthread+0xf2/0x120 [ 1599.288055] ? __pfx_kthread+0x10/0x10 [ 1599.288060] ret_from_fork+0x47/0x70 [ 1599.288066] ? __pfx_kthread+0x10/0x10 [ 1599.288071] ret_from_fork_asm+0x1b/0x30 [ 1599.288079] [ 1599.288131] INFO: task zvol:22411 blocked for more than 123 seconds. [ 1599.288139] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.288143] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.288146] task:zvol state:D stack:0 pid:22411 tgid:22411 ppid:2 flags:0x00004000 [ 1599.288154] Call Trace: [ 1599.288157] [ 1599.288160] __schedule+0x27c/0x6b0 [ 1599.288168] schedule+0x33/0x110 [ 1599.288173] schedule_preempt_disabled+0x15/0x30 [ 1599.288178] rwsem_down_write_slowpath+0x27e/0x550 [ 1599.288186] down_write+0x5c/0x80 [ 1599.288193] dnode_new_blkid+0xf7/0x180 [zfs] [ 1599.289207] dbuf_dirty+0x54f/0x730 [zfs] [ 1599.290202] dmu_buf_will_dirty_impl+0xd2/0x1b0 [zfs] [ 1599.291195] dmu_buf_will_dirty+0x16/0x30 [zfs] [ 1599.292202] dmu_write_uio_dnode+0x92/0x170 [zfs] [ 1599.293198] zvol_write+0x250/0x400 [zfs] [ 1599.294240] zvol_write_task+0x12/0x30 [zfs] [ 1599.295260] taskq_thread+0x1f6/0x3c0 [spl] [ 1599.295311] ? __pfx_default_wake_function+0x10/0x10 [ 1599.295323] ? __pfx_zvol_write_task+0x10/0x10 [zfs] [ 1599.296336] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 1599.296385] kthread+0xf2/0x120 [ 1599.296393] ? __pfx_kthread+0x10/0x10 [ 1599.296398] ret_from_fork+0x47/0x70 [ 1599.296405] ? __pfx_kthread+0x10/0x10 [ 1599.296409] ret_from_fork_asm+0x1b/0x30 [ 1599.296417] [ 1599.296422] INFO: task zvol:22412 blocked for more than 123 seconds. [ 1599.296428] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.296433] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.296436] task:zvol state:D stack:0 pid:22412 tgid:22412 ppid:2 flags:0x00004000 [ 1599.296443] Call Trace: [ 1599.296446] [ 1599.296449] __schedule+0x27c/0x6b0 [ 1599.296458] schedule+0x33/0x110 [ 1599.296462] schedule_preempt_disabled+0x15/0x30 [ 1599.296467] rwsem_down_read_slowpath+0x284/0x4d0 [ 1599.296475] down_read+0x48/0xd0 [ 1599.296481] dmu_tx_check_ioerr+0x3f/0x100 [zfs] [ 1599.297512] dmu_tx_count_write+0xe3/0x1d0 [zfs] [ 1599.298535] dmu_tx_hold_write_by_dnode+0x3a/0x60 [zfs] [ 1599.299563] zvol_write+0x225/0x400 [zfs] [ 1599.300595] zvol_write_task+0x12/0x30 [zfs] [ 1599.301619] taskq_thread+0x1f6/0x3c0 [spl] [ 1599.301670] ? __pfx_default_wake_function+0x10/0x10 [ 1599.301682] ? __pfx_zvol_write_task+0x10/0x10 [zfs] [ 1599.302702] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 1599.302753] kthread+0xf2/0x120 [ 1599.302761] ? __pfx_kthread+0x10/0x10 [ 1599.302766] ret_from_fork+0x47/0x70 [ 1599.302772] ? __pfx_kthread+0x10/0x10 [ 1599.302777] ret_from_fork_asm+0x1b/0x30 [ 1599.302785] [ 1599.302789] INFO: task zvol:22413 blocked for more than 123 seconds. [ 1599.302796] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.302800] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.302804] task:zvol state:D stack:0 pid:22413 tgid:22413 ppid:2 flags:0x00004000 [ 1599.302811] Call Trace: [ 1599.302814] [ 1599.302817] __schedule+0x27c/0x6b0 [ 1599.302825] schedule+0x33/0x110 [ 1599.302829] schedule_preempt_disabled+0x15/0x30 [ 1599.302834] rwsem_down_read_slowpath+0x284/0x4d0 [ 1599.302842] down_read+0x48/0xd0 [ 1599.302847] dmu_tx_check_ioerr+0x3f/0x100 [zfs] [ 1599.303875] dmu_tx_count_write+0xe3/0x1d0 [zfs] [ 1599.304881] dmu_tx_hold_write_by_dnode+0x3a/0x60 [zfs] [ 1599.305969] zvol_write+0x225/0x400 [zfs] [ 1599.306987] zvol_write_task+0x12/0x30 [zfs] [ 1599.308009] taskq_thread+0x1f6/0x3c0 [spl] [ 1599.308058] ? __pfx_default_wake_function+0x10/0x10 [ 1599.308070] ? __pfx_zvol_write_task+0x10/0x10 [zfs] [ 1599.309110] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 1599.309160] kthread+0xf2/0x120 [ 1599.309170] ? __pfx_kthread+0x10/0x10 [ 1599.309175] ret_from_fork+0x47/0x70 [ 1599.309180] ? __pfx_kthread+0x10/0x10 [ 1599.309185] ret_from_fork_asm+0x1b/0x30 [ 1599.309193] [ 1599.309196] INFO: task zvol:22414 blocked for more than 123 seconds. [ 1599.309205] Tainted: P OE 6.8.0-39-generic #39-Ubuntu [ 1599.309209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1599.309212] task:zvol state:D stack:0 pid:22414 tgid:22414 ppid:2 flags:0x00004000 [ 1599.309220] Call Trace: [ 1599.309223] [ 1599.309226] __schedule+0x27c/0x6b0 [ 1599.309234] schedule+0x33/0x110 [ 1599.309239] schedule_preempt_disabled+0x15/0x30 [ 1599.309244] rwsem_down_read_slowpath+0x284/0x4d0 [ 1599.309252] ? __list_add+0x17/0x40 [zfs] [ 1599.310294] down_read+0x48/0xd0 [ 1599.310306] dmu_buf_hold_array_by_dnode+0x4f/0x570 [zfs] [ 1599.311341] ? dsl_dir_tempreserve_space+0x10f/0x160 [zfs] [ 1599.312359] dmu_write_uio_dnode+0x5c/0x170 [zfs] [ 1599.313360] zvol_write+0x250/0x400 [zfs] [ 1599.314393] zvol_write_task+0x12/0x30 [zfs] [ 1599.315434] taskq_thread+0x1f6/0x3c0 [spl] [ 1599.315487] ? __pfx_default_wake_function+0x10/0x10 [ 1599.315498] ? __pfx_zvol_write_task+0x10/0x10 [zfs] [ 1599.316518] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 1599.316568] kthread+0xf2/0x120 [ 1599.316577] ? __pfx_kthread+0x10/0x10 [ 1599.316582] ret_from_fork+0x47/0x70 [ 1599.316587] ? __pfx_kthread+0x10/0x10 [ 1599.316592] ret_from_fork_asm+0x1b/0x30 [ 1599.316601] ~~~ I have the feeling I simply need to do something to /dev/zd16 (aka: /dev/zvol/rpool/swap). (the 8 GB swap is a 8 GB LUKS partition - not zfs). ~~~ $ zfs get all rpool/swap NAME PROPERTY VALUE SOURCE rpool/swap type volume - rpool/swap creation Fri Aug 2 0:39 2024 - rpool/swap used 30.5G - rpool/swap available 176G - rpool/swap referenced 362M - rpool/swap compressratio 1.00x - rpool/swap reservation none default rpool/swap volsize 30G local rpool/swap volblocksize 16K default rpool/swap checksum on default rpool/swap compression lz4 inherited from rpool rpool/swap readonly off default rpool/swap createtxg 1260475 - rpool/swap copies 1 default rpool/swap refreservation 30.5G local rpool/swap guid 13973446581616721601 - rpool/swap primarycache all default rpool/swap secondarycache all default rpool/swap usedbysnapshots 0B - rpool/swap usedbydataset 362M - rpool/swap usedbychildren 0B - rpool/swap usedbyrefreservation 30.1G - rpool/swap logbias latency default rpool/swap objsetid 84726 - rpool/swap dedup off default rpool/swap mlslabel none default rpool/swap sync standard inherited from rpool rpool/swap refcompressratio 1.00x - rpool/swap written 362M - rpool/swap logicalused 364M - rpool/swap logicalreferenced 364M - rpool/swap volmode default default rpool/swap snapshot_limit none default rpool/swap snapshot_count none default rpool/swap snapdev hidden default rpool/swap context none default rpool/swap fscontext none default rpool/swap defcontext none default rpool/swap rootcontext none default rpool/swap redundant_metadata all default rpool/swap encryption aes-256-gcm - rpool/swap keylocation none default rpool/swap keyformat passphrase - rpool/swap pbkdf2iters 100000 - rpool/swap encryptionroot rpool - rpool/swap keystatus available - ~~~