FIO reports slower sequential read than the advertised NVMe SSD read bandwidth

# TLDL For the very simple sequential read, FIO reports is much slower than the NVMe SSD sequentail read capability. --- # Main Text Hello everyone, I have been facing an issue while trying to achieve the maximum read bandwidth reported by the vendor for my Samsung 980 Pro 1T NVMe SSD. According to the Samsung product description, the SSD is capable of reaching read bandwidths of around 7 GB/s. However, despite my efforts, I have been unable to achieve this maximum read bandwidth. Current Setup: - SSD: Samsung 980 Pro 1T NVMe SSD - Connection: PCIe 4.0 port - Operating System: Linux Ubuntu Current FIO Script and Results: To test the read performance of the SSD, I have been using the FIO benchmarking tool with the following script:

$ sudo fio --loops=5 --size=1024m --filename=/dev/nvme0n2 --stonewall --ioengine=libaio --direct=1 --zero_buffers=1 --name=Seqread --bs=1024m --iodepth=1 --numjobs=1 --rw=read

Here are the results obtained from running the FIO script:

Seqread: (g=0): rw=read, bs=(R) 1024MiB-1024MiB, (W) 1024MiB-1024MiB, (T) 1024MiB-1024MiB, ioengine=libaio, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1)
Seqread: (groupid=0, jobs=1): err= 0: pid=1504682: Mon Oct 16 09:28:48 2023
  read: IOPS=3, BW=3368MiB/s (3532MB/s)(5120MiB/1520msec)
    slat (msec): min=151, max=314, avg=184.19, stdev=72.71
    clat (msec): min=2, max=149, avg=119.59, stdev=65.39
     lat (msec): min=300, max=316, avg=303.77, stdev= 7.33
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[  148], 40.00th=[  148], 50.00th=[  148], 60.00th=[  148],
     | 70.00th=[  150], 80.00th=[  150], 90.00th=[  150], 95.00th=[  150],
     | 99.00th=[  150], 99.50th=[  150], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  MiB/s): min= 2048, max= 4096, per=81.07%, avg=2730.67, stdev=1182.41, samples=3
   iops        : min=    2, max=    4, avg= 2.67, stdev= 1.15, samples=3
  lat (msec)   : 4=20.00%, 250=80.00%
  cpu          : usr=0.00%, sys=31.47%, ctx=405, majf=0, minf=262156
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3368MiB/s (3532MB/s), 3368MiB/s-3368MiB/s (3532MB/s-3532MB/s), io=5120MiB (5369MB), run=1520-1520msec

Disk stats (read/write):
  nvme0n2: ios=9391/0, merge=0/0, ticks=757218/0, in_queue=757218, util=93.39%

I would greatly appreciate any guidance or suggestions on **how to optimize my FIO script to achieve the expected read bandwidth of around 7 GB/s**. If there are any improvements or modifications that can be made to the script, please let me know. Thank you in advance for your assistance! Please feel free to provide any additional information or insights that may be relevant to the issue at hand. --- Note: It should be PCIe4.0*4:

$ lspci -vv -s 5e:00.0
5e:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
	Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- 
	Kernel driver in use: nvme
	Kernel modules: nvme

$ cat /sys/class/pci_bus/0000\:5e/device/0000\:5e\:00.0/max_link_width 
4
$ cat /sys/class/pci_bus/0000\:5e/device/0000\:5e\:00.0/max_link_speed 
16.0 GT/s PCIe