Is it worth the hassle and risk of reformatting an NVME to use 4K blocks on a ZFS pool created with ashift=12?
8
votes
1
answer
491
views
I recently upgraded the NVME drives on my workstation machine, from a pair of Samsung EVO 970 512GB drives to a pair of of Kingston Fury 2TB drives. All went well, and I even converted the machine from old BIOS boot to UEFI boot. No problem.
However, I just noticed that the NVME drives are formatted with 512 byte blocks rather than 4KiB blocks. I mistakenly assumed that they'd be 4K and didn't check.
# nvme list
Node Generic SN Model Namespace Usage Format FW Rev
------------ ---------- ---- ------------------- ---------- ------------- --------- --------
/dev/nvme0n1 /dev/ng0n1 XXXX KINGSTON SFYRD2000G 0x1 2.00TB/2.00TB 512B + 0B EIFK31.7
/dev/nvme1n1 /dev/ng1n1 XXXX KINGSTON SFYRD2000G 0x1 2.00TB/2.00TB 512B + 0B EIFK31.7
# nvme id-ns -H /dev/nvme0n1 | grep Data.Size
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better
I'm using partitions on these drives for GRUB BIOS boot (p1), ESP (p2), an mdadm RAID-1 ext4 /boot filesystem (p3) with lots of space for kernels & ISO images, swap space (p4), L2ARC (p5) and ZIL (p6) for a HDD zfs pool, and the ZFS rootfs (p7).
The BIOS boot partition is obsolete now, since I've switched to UEFI but it resides in otherwise unused space before sector 2048 so isn't important.
They're both partitioned identically.
# gdisk -l /dev/nvme0n1
GPT fdisk (gdisk) version 1.0.10
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/nvme0n1: 3907029168 sectors, 1.8 TiB
Model: KINGSTON SFYRD2000G
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 9E7187C9-3ED2-46EF-A695-E72489F2BEC3
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 8-sector boundaries
Total free space is 143 sectors (71.5 KiB)
Number Start (sector) End (sector) Size Code Name
1 34 2047 1007.0 KiB EF02 BIOS boot partition
2 2048 1050623 512.0 MiB EF00 EFI system partition
3 1050624 8390655 3.5 GiB FD00
4 8390656 142608383 64.0 GiB 8200 Linux swap
5 142608384 276826111 64.0 GiB BF08 Solaris Reserved 2
6 276826112 285214719 4.0 GiB BF09 Solaris Reserved 3
7 285214720 3907028991 1.7 TiB BF00 Solaris root
Anyway, I created the ZFS pool with ashift=12
for 4KiB block sizes, so it's always going to be reading and writing in multiples of 4K at a time.
What I want to know is if there will be a noticeable performance difference if I reformat the NVME drives to use 4K sectors?
I know (roughly) how to do that using the nvme
command while booted from a rescue image, but given the hassle involved and the amount of downtime, and the risk of losing data if I make a mistake or if disaster strikes during one of the periods when the ZFS pool is in a degraded state, I only want to do it if there is a significant benefit...significant, to me, meaning at least a 5 or 10% improvement, not just 1 or 2%.
(I have backups of the root pool - multiple nightly backups in multiple locations - but I'd prefer to avoid restoring from backup)
I don't care about performance for the ESP or /boot partitions. Swap & L2ARC might benefit. The ZIL rarely gets used and probably won't be noticable. The main concern is performance of the zpool partition itself.
Asked by cas
(81932 rep)
Jul 15, 2025, 02:25 PM
Last activity: Jul 15, 2025, 05:23 PM
Last activity: Jul 15, 2025, 05:23 PM