Sample Header Ad - 728x90

Is it worth the hassle and risk of reformatting an NVME to use 4K blocks on a ZFS pool created with ashift=12?

8 votes
1 answer
491 views
I recently upgraded the NVME drives on my workstation machine, from a pair of Samsung EVO 970 512GB drives to a pair of of Kingston Fury 2TB drives. All went well, and I even converted the machine from old BIOS boot to UEFI boot. No problem. However, I just noticed that the NVME drives are formatted with 512 byte blocks rather than 4KiB blocks. I mistakenly assumed that they'd be 4K and didn't check.
# nvme list
Node         Generic    SN   Model               Namespace  Usage         Format    FW Rev
------------ ---------- ---- ------------------- ---------- ------------- --------- --------
/dev/nvme0n1 /dev/ng0n1 XXXX KINGSTON SFYRD2000G 0x1        2.00TB/2.00TB 512B + 0B EIFK31.7
/dev/nvme1n1 /dev/ng1n1 XXXX KINGSTON SFYRD2000G 0x1        2.00TB/2.00TB 512B + 0B EIFK31.7
# nvme id-ns -H /dev/nvme0n1 | grep Data.Size
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better
I'm using partitions on these drives for GRUB BIOS boot (p1), ESP (p2), an mdadm RAID-1 ext4 /boot filesystem (p3) with lots of space for kernels & ISO images, swap space (p4), L2ARC (p5) and ZIL (p6) for a HDD zfs pool, and the ZFS rootfs (p7). The BIOS boot partition is obsolete now, since I've switched to UEFI but it resides in otherwise unused space before sector 2048 so isn't important. They're both partitioned identically.
# gdisk -l /dev/nvme0n1
GPT fdisk (gdisk) version 1.0.10

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/nvme0n1: 3907029168 sectors, 1.8 TiB
Model: KINGSTON SFYRD2000G                     
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 9E7187C9-3ED2-46EF-A695-E72489F2BEC3
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 3907029134
Partitions will be aligned on 8-sector boundaries
Total free space is 143 sectors (71.5 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048         1050623   512.0 MiB   EF00  EFI system partition
   3         1050624         8390655   3.5 GiB     FD00  
   4         8390656       142608383   64.0 GiB    8200  Linux swap
   5       142608384       276826111   64.0 GiB    BF08  Solaris Reserved 2
   6       276826112       285214719   4.0 GiB     BF09  Solaris Reserved 3
   7       285214720      3907028991   1.7 TiB     BF00  Solaris root
Anyway, I created the ZFS pool with ashift=12 for 4KiB block sizes, so it's always going to be reading and writing in multiples of 4K at a time. What I want to know is if there will be a noticeable performance difference if I reformat the NVME drives to use 4K sectors? I know (roughly) how to do that using the nvme command while booted from a rescue image, but given the hassle involved and the amount of downtime, and the risk of losing data if I make a mistake or if disaster strikes during one of the periods when the ZFS pool is in a degraded state, I only want to do it if there is a significant benefit...significant, to me, meaning at least a 5 or 10% improvement, not just 1 or 2%. (I have backups of the root pool - multiple nightly backups in multiple locations - but I'd prefer to avoid restoring from backup) I don't care about performance for the ESP or /boot partitions. Swap & L2ARC might benefit. The ZIL rarely gets used and probably won't be noticable. The main concern is performance of the zpool partition itself.
Asked by cas (81932 rep)
Jul 15, 2025, 02:25 PM
Last activity: Jul 15, 2025, 05:23 PM