setting up infiniband ib0 in RHEL

running RHEL 7.9 on a few servers that have Mellanox HDR infiniband cards installed. I have installed RHEL 7.9 on each of them from dvd, and chose **server with gui** along with **Infiniband Support** and a handful of other items (the 5 at the bottom). So using Application - System Tools - Settings - Network here are pics of that gui. I get *Connected* but if I do an ip a it shows ib0 state DOWN. This is on a simple LAN, and in the below pic Proxy = None and IPv6 Settings = ignore and I did not set a gateway value for ipv4 settings but I do not think that should matter. - What am I missing or not doing? - Am I required to use MLNX_OFED_LINUX-5.5-1.0.3.2-rhel7.9-x86_64.iso and if so what parts of it? - I had run the MLNX iso's mlnxofedinstall under a previous RHEL install, and had the ib0 network working but the iso's mlnxofedinstall I think was removing existing rdma stuff and was causing my NFS+RDMA problem so I am a little lost. - update: on 1 of my 4 servers I ran mlnxofedinstall --with-nfsrdma and it gets an opensm running and now I have green link lights on all server infiniband's. Is it the opensm that did it? But as a result I somehow lose rdma.service on that server i ran the mlnxofedinstall on and then nukes my changes of getting nfs+rdma working which I really want. Usage: ./mlnxofedinstall [OPTIONS] Installation control: --force Force installation --tmpdir Change tmp directory. Default: /tmp -k|--kernel Default on this system: 3.10.0-1160.49.1.el7.x86_64 -s|--kernel-sources Default on this system: /lib/modules/3.10.0-1160.49.1.el7.x86_64/build -U|--update Update installed version --with-32bit Install 32-bit libraries. Relevant for x86_64 platforms --without-32bit Skip 32-bit libraries installation (Default). Relevant for x86_64 platforms --distro Set Distro name for the running OS (e.g: rhel6.5, sles11sp3). Default: Use auto-detection --skip-distro-check Do not check MLNX_OFED_LINUX vs Distro matching --without-depcheck Run the installation without verifying that all required Distro's packages are installed --check-deps-only Check for missing required Distro's packages and exit --print-distro Print distribution name and exit --add-kernel-support Add kernel support (Run mlnx_add_kernel_support.sh) and install it --add-kernel-support-build-only Add kernel support (Run mlnx_add_kernel_support.sh) and exit --kernel-extra-args '' pass to kernel configure script (single paraeter, space separated) - Pass it along with --add-kernel-support --kmp Install kernel RPMs with KMP support if available - Enabled by default with official MLNX_OFED_LINUX releases - Disabled by default if new MLNX_OFED_LINUX was created with mlnx_add_kernel_support.sh - Passing it along with --add-kernel-support will pass --kmp to mlnx_add_kernel_support.sh --disable-kmp Install kernel RPMs without KMP support if available - Enabled by default if new MLNX_OFED_LINUX was created with mlnx_add_kernel_support.sh --skip-kmp-verify Do not check if the KMP rpms are compatiable with the target kernel --speed-up-kmp Speed up KMP RPMs installation by skipping %post section and running the required operations once for all KMPs --umad-dev-rw Grant non root users read/write permission for umad devices instead of default --umad-dev-na Prevent from non root users read/write access for umad devices. Overrides '--umad-dev-rw' --enable-mlnx_tune Enable Running the mlnx_tune utility --enable-affinity Run mlnx_affinity script upon boot --disable-affinity Disable mlnx_affinity script (Default) --skip-unsupported-devices-check Don't abort if system has an older, unsupported card --enable-opensm Run opensm upon boot --package-install-options RPM install options to use when installing RPM packages (comma separated list) --pre-install- Run given script before given package's install --post-install- Run given script after given package's install Firmware update control: --without-fw-update Skip firmware update --fw-update-only Update firmware. Skip driver installation --force-fw-update Force firmware update --fw-image-dir Firmware images directory to use instead of default package content Package selection: -c|--config Example of the config file can be found under docs --all Install all available packages --bluefield Install BlueField packages --hpc Install minimum packages required for HPC --basic Install minimum packages for basic functionality --dpdk Install minimum packages required for DPDK --ovs-dpdk Install DPDK and OVS packages --vma Install minimum packages required for VMA --xlio Install minimum packages required for XLIO --guest Install minimum packages required by guest OS --hypervisor Install minimum packages required by hypervisor OS --with-fabric-collector Enable installing fabric-collector package --with-nvmf Enable NVMEoF support --with-nfsrdma Enable NFSoRDMA support --without-nfsrdma Disable NFSoRDMA support (Default) Extra package filtering: --kernel-only Install kernel space packages only --user-space-only Filter selected packages and install only User Space packages --without- Do not install package --with- Force installing package Miscellaneous: -h|--help Display this help message and exit --post-start-delay Set openibd POST_START_DELAY parameter in seconds. (Default 0) -p|--print-available Print available packages for current platform And create corresponding ofed.conf file