I'm trying out RDMA on NFS and noticed that it does not seem to work with NFSv4:

[grant@host ~]$ sudo mount -t nfs -o rdma,proto=rdma,vers=4 10.99.99.98:/ifs/rdma-test /mnt/powerscale_rdma
mount.nfs: Protocol family not supported
[grant@host ~]$ sudo mount -t nfs -o rdma,proto=rdma,vers=3 10.99.99.98:/ifs/rdma-test /mnt/powerscale_rdma

nfsv3 loads up and runs just fine, but nfsv4 gets you mount.nfs: Protocol family not supported. [This answer](https://unix.stackexchange.com/a/749996/240147) on [NFS4, insecure, port number, rdma contradiction help](https://unix.stackexchange.com/questions/749990/nfs4-insecure-port-number-rdma-contradiction-help) seems to indicate it could work, but it's not really clear how. Is there a way to run NFSv4 with RDMA?

- With RHEL 8.8 currently, and RHEL 9.x, the latest NFS version is 4.2. - When NFS 4 was introduced, it did away with a few things in NFS3 one of which was multiple port numbers: - *NFS4 **mandates** all traffic now exclusively TCP **uses the single well known port 2049**.* - https://www.snia.org/sites/default/files/SNIA_An_Overview_of_NFSv4-3_0.pdf - you can find more mostly reputable articles stating the same thing. - I have confirmed this by having only TCP 2049 open in firewalld for NFS 4.1 in RHEL 7.9; it does not use port 111 or any other unless you change the default configurations of /etc/nfs.conf or /etc/sysconfig/nfs. And in fact when I did get rdma working (over port 20049) that the rdma protocol specifically bypasses firewalld, an inherent aspect of why rdma saves cpy cycles and is faster i suppose. > The NFS **insecure** option in /etc/exports sets the server to listen to a request from any port on the client. Changing it to 'secure' (default) makes sure that the server will listen to only requests originating from ports 1-1024 of the client. Thus an unauthorized user on the client is kept from starting an NFS dialogue. For reference : https://security.stackexchange.com/questions/246527/what-is-insecure-about-the-insecure-option-of-nfs-exports The default is **secure** vs *insecure* when doing an NFS4 export if neither is mentioned in /etc/exports. With **security rules** it is oftentimes stated *The NFS server must not have the insecure file locking option enabled.*. First with the /etc/exports secure option in play, be default, the *will only operate on secure ports less than 1024` seems to be completely not true since NFS4 runs on port 2049. The number 2049 is greater than 1024... what am I missing? With regards to RDMA which by convention happens on port 20049 there seems to be a little missed fact that one needs to **explicitly** state the **insecure** option in /etc/exports if a mount -o rdma is to be used otherwise the mount always happens as proto=tcp and not proto=rdma with no indication why. I did validate that, using MLNX_OFED_LINUX-23.04-1.1.3.0-rhel8.8-x86_64.iso installed in place of the Redhat InfiniBand Support packages that a mount -o rdma,port=1023 does work with a mount on the client side showing proto=rdma. **However** one must also do (with MLNX only?) an echo rdma 20049 > /proc/sys/nfsd/portlist. Or in the case with secure export an *echo rdma 1023*. Does anyone know how/why these values are not in /proc/sys/nfsd/portlist in the first place and why I must do them manually ? **And then what is the correct way to put those numbers there**, so that after boot my /etc/fstab nfs mounting of my data folder as rdma happens successfully? The MLNX instructional pdf falls short. I have been banging my head against the wall getting RDMA to work, there seems to be a lot of shortcoming with NFS overall, and I have a paid for cluster mgr software that has RDMA placeholders for configuration, but all mounts are always proto=tcp. So if anyone can provide any information on anything described would be helpful, I will + any answer. **Also:** I will end up doing /etc/exports with secure and choose some port number 1023 and below to satisfy security rules. How do I choose a proper number in that range? As ron nobody my understanding was I should never use port numbers below 1000 or 1024 for stuff I set up? **update:** it appears that the /etc/exports parameter of *secure* or *insecure* is inconsequential. What matters is having rdma 20049 in /proc/fs/nfsd/portlist on the nfsserver. With that, or any number, it appears to work with the *secure* exportfs.

in /etc/firewalld/zone/mycustomzone.xml I use the syntax of ` to open port number being a protocol choice of either tcp or udp. For NFSv4 only port 2049 needs to be opened, so when NFS proto=tcp` then open firewall port protocol tcp. And when NFS proto=udp then firewalld protocol=udp. Are there other valid choices besides tcp and udp there for firewalld? Using RHEL 8.7 with NFS v4.2 working, when doing mount it reports my mounts typically having proto=tcp. But when I have RDMA working it says proto=rdma. I know RDMA happens over port 20049 (by default). What is the relationship of RDMA to TCP/UDP, and do I just need to do both in my firewalld xml file where both protocols need to be allowed for RDMA? Or should only one of those two be opened and if so which one? I don't think firewalld supports *protocol=rdma* does it? **update:** - the original title to this post was *NFS proto=rdma and firewalld tcp/udp protocol relationship* - with only firewalld port=22 tcp open and firewalld running on both the nfs-server and nfs-client computers, a mount server:/data /data will fail (because port 2049 tcp is not open) but a mount -o rdma server:/data /data is successful - on the client the rdma mount shows server:/data on /data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.2,local_lock=none,addr=192.168.1.1) - this is in RHEL 8.7 - my observation seems to prove nfs via rdma is not affected by firewalld. is this to be expected given how rdma is supposed to be *direct access* ?