Mellanox connectx 3 linux

Introduction

Although old, Mellanox ConnectX-3 has been a good card with decent performance and a good price on second-hand markets. This makes it the straightforward choice for a high-performance home or lab network. (If you intend to run InfiniBand on ESXi 7.0 or higher, go for CX4.) Sometimes you would find a card with a strange firmware or one not compatible with your existent gear. Luckily you can change the port configuration on all CX3 cards with a simple firmware flash. Here’s how.

• Flashing firmware may brick your card and I’m not responsible for it
• Do not power off or reboot your computer during firmware flashing

Preparation

I’ll assume you are using Windows Server 2019 with all the latest patches; all programs are available under Linux too so you can apply all the instructions for a Linux server with only minor deviations.

All the download links are for ConnectX-3 non-pro version; If you have a ConnectX-2 or ConnectX-3 Pro, you need to download a different firmware.

Downloads

Download and install Mellanox firmware tools. Install ALL the components (some of them are not selected by default).

Download the ConnectX-3 VPI firmware package. EN firmware can only run Ethernet; VPI firmware run both Ethernet and IB. Thus, VPI firmware is always recommended.

Check Your Card

First of all, we need the card’s machine-dependent device ID. This device ID is used for all the commands below.

Источник

Mellanox NICs in Debian 11 not supported?

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

IamSpartacus

Well-Known Member

Has anyone gotten Mellanox NICs (specifically the ConnectX-3 cards) to work in Debian 11? Their support matrix doesn’t list it.

MLNX_OFED: Firmware — Driver Compatibility Matrix | NVIDIA

fohdeesha

Kaini Industries

They work perfectly and don’t need a driver download from mellanox, debian has and will have the mlx kernel driver included in it like, forever probably. Removing that would be like removing the Intel ixgb driver

Sean Ho

seanho.com

CheatingLiar

New Member

AFAIK, it depends on what you are trying to do. Just using the card in the host to do average network card things should work out of the box as others said. Now if you are going to do SR-IOV, you will need to get the source code and massage it.

ectoplasmosis

Active Member

AFAIK, it depends on what you are trying to do. Just using the card in the host to do average network card things should work out of the box as others said. Now if you are going to do SR-IOV, you will need to get the source code and massage it.

This is incorrect. SR-IOV works fine without having to do or install anything, ‘massage’ or otherwise.

The included kernel module accepts the same commands relating to SR-IOV as any other Mellanox driver. I’m running ConnectX-4/5 SFP28 cards with SR-IOV in several Debian 11 systems without a problem.

efschu3

Active Member

This is incorrect. SR-IOV works fine without having to do or install anything, ‘massage’ or otherwise.

The included kernel module accepts the same commands relating to SR-IOV as any other Mellanox driver. I’m running ConnectX-4/5 SFP28 cards with SR-IOV in several Debian 11 systems without a problem.

This is incorrect, SR-IOV on kernel inbox driver does work well with non-windows-guests. But it does NOT work for windows guests.

ectoplasmosis

Active Member

This is incorrect, SR-IOV on kernel inbox driver does work well with non-windows-guests. But it does NOT work for windows guests.

efschu3

Active Member

ectoplasmosis

Active Member

efschu3

Active Member

Could you pls provide
modinfo mlx4_en

And your exact Version of your Windows Driver?

B/c sriov for Windows guests stoped working with Kernel 4.19 (actualy).

ectoplasmosis

Active Member

Could you pls provide
modinfo mlx4_en

And your exact Version of your Windows Driver?

B/c sriov for Windows guests stoped working with Kernel 4.19 (actualy).

Win10 21H2 guest VMs, WinOFED2 latest version

Works OK, no Code 10/43 errors, all assigned VF devices appear normally within the Windows guest.

Mellanox card is a MCX4121A-ACAT.

efschu3

Active Member

bigfoxtail

New Member

I have a cx341, system is TrueNAS scale,configure SR-IOV dmesg display info
`mlx4_core 0000:c3:00.0: Driver does not support SRIOV configuration via sysfs`

crackelf

Member

Anyone have Mellanox NICs with vf’s in Debian working with the mlx5_core driver?

I’m treating this like the ixgbe driver and not getting far. I see Kernel driver in use: mlx5_core and am trying to create virtual functions in /etc/modprobe.d/mlx5_core.conf with some options like options mlx5_core num_vfs=12 port_type_array=2 probe_vf=12 , but predictably dmesg says it’s ignoring those options as they are «unknown».

Reading the docs over at nvidia show these as valid options, but only for their MLNX_OFED driver. Any tips and or resources here? I’m not finding much for the kernel driver other than the kernel.org docs, but not seeing much about vfs. Is the core driver not enough and I need to enable the EN driver in the kernel? Thanks for any help!

EDIT / ANSWER: echo 8 > /sys/devices/pci0000\:00/0000\:00\:1d.0/0000\:03\:00.0/sriov_numvfs
check this thread for more info

firemeteor

New Member

Win10 21H2 guest VMs, WinOFED2 latest version

Works OK, no Code 10/43 errors, all assigned VF devices appear normally within the Windows guest.

Mellanox card is a MCX4121A-ACAT.

I think I run into a bug that with Linux stock driver (mlx4_en 4.0.0) does not handle promiscuous mode well in SRIOV PF ports.
Switching to the vendor driver (v4.9-4.1.7) immediately solved my problem.
I really hate this driver policy of releasing old && buggy driver to public while keep the up-to-date driver and fixed driver version out-of tree.
And now, the up-to-date version is stuck with ancient kernel version.

Stephan

Well-Known Member

firemeteor

New Member

Thanks for the reminder, Stephan.
Actually I came across that post before but I didn’t realize that this may have anything to do with my issue.
I didn’t try any window guest yet and thus didn’t run into any explicit error code.
My issue is that the NIC just eat my packets in silent when the SR-IOV PF is put into a host side bridge.
If I understand correctly, this setup does not need FDB manipulation which is more about setting up bridge after a VF in the guest.

BTW, when I was on the vendor driver v4.9, I did run into a FDB manipulation related problem that the ‘ip fdb add xxx ‘ command does not seem to make any difference. But anyway I was able to find a workaround to live with it.

I have to say that my experience from this card is pretty bad.
There is no perfect driver that I can use with ease.
For the stock driver I tried to contact the maintainer but got no response yet.
For the vendor driver I’m still trying to build on 5.x kernel without much success.
(I heard from a different forum that newer driver versions which are supposed to not compatible with CX3 may actually work.
But ironically for unknown reason I can’t even build the latest driver which is supposed to compile on my kernel version. )

I’m wondering if I grabbed a X520-DA2 instead of CX3, will it behave any better than the Mellanox ones.

Источник

Обновление драйвера Mellanox ConnectX

Для примера обновлю драйвер сетевого адаптера Mellanox ConnectX-3 Pro EN / HP 544+QSFP (764284-B21) который я установил в сервер HPE DL380 Gen10 с Ubuntu Server 18.04.

Для обновления драйвера необходимо переключится на root пользователя:

Сначала я посмотрел версию установленного драйвера:

modinfo mlx4_core | grep version version: 4.0-0 modinfo mlx4_core | grep ^version:|sed 's/version: * //g' 4.0-0 ethtool -i ens3

Выполнив эти команды на нескольких серверах Ubuntu Server 18.04 я заметил что по умолчанию используется драйвер версии 4.0-0.

Посмотрим видит ли система сетевой адаптер, как он называется и его Part number:

lspci -vvv lspci | grep Mellanox 12:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] lspci -vv -s 12:00.0 | grep "Part number" -A 3 [PN] Part number: 764284-B21 [EC] Engineering changes: A3 [SN] Serial number: CN7644033S [V0] Vendor specific: PCIe FDR x8 13W

Кстати я пробовал установить одновременно сетевые адаптеры на разных чипах (Intel XL710-QDA1 и Mellanox ConnectX-3 / HP 544+QSFP) в сервер HPE DL380 Gen10 и заметил что одновременно они не работают и по команде lspci их не видно.

Потом я посмотрел драйвер на официальном сайте
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
Скачал и распаковал LTS версию MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64.tgz.
Так как в самой новой версии MLNX_OFED_LINUX-5.1-2.5.8.0-ubuntu18.04-x86_64.tgz на момент написания статьи были драйвера начиная от ConnectX-4, а для ConnectX-3 не было.

Распаковал скачанный архив:

tar -xzf MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64.tgz cd MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64

Для запуска процесса обновления необходимо запустить скрипт установки:

В этом случае будет установлен полный пакет утилит и обновлен драйвер. Скрипт обнаружит все сетевые адаптеры Mellanox и предложит выбрать конкретный или все, а также выполнит обновлении прошивки. Убедимся что в файле distro указана верная версия дистрибутива операционной системы.

Но я решил обновить драйвер и установить минимальный набор утилит:

./mlnxofedinstall --upstream-libs --dpdk

Во время установки будут установлены и удалены некоторые пакеты, в случае ошибки их можно установить или удалить вручную, а потом вновь запустить скрипт, например:

apt install tk bison debhelper flex autoconf automake graphviz dpatch gfortran libgfortran4 quilt dkms chrpath tcl m4 swig libnl-route-3-dev ixnfo.com apt remove .

После успешного обновления перезапустим драйвер:

Перезагрузим сервер и убедимся что сеть работает:

Если необходимо только обновить прошивку, то выполним:

./mlnxofedinstall --fw-update-only

Пример создания резервной копии прошивки и конфигурации сетевого адаптера:

flint -d 12:00.0 ri firmware_backup.bin flint -d 12:00.0 dc config_backup.ini

При необходимости закроем порты:

iptables -A INPUT --protocol tcp --in-interface ens2d1 --dport 8888 -j DROP iptables -A INPUT --protocol tcp --in-interface ens2d1 --dport 8889 -j DROP

Модули которые автоматически загружаются при запуске операционной системы:
ConnectX®-2/ConnectX®-3/ConnectX®-3 Pro:
mlx4_en, mlx4_core, mlx4_ib
ConnectX®-4/ConnectX®-4 Lx/ConnectX®-5:
mlx5_core, mlx5_ib

Если необходимо выгрузить модули драйвера, то модуль mlx*_core необходимо выгружать в последнюю очередь, например:

modprobe -r mlx4_ib modprobe -r mlx4_en modprobe -r mlx4_core

modprobe mlx4_core modprobe mlx4_ib modprobe mlx4_en

Приведу список важных пакетов которые устанавливаются скриптом обновления драйвера или через apt install:
rdma-core, libibmad, opensm, opensm-libs, ibutils-libs, ibutils, infiniband-diags, perftest, mstflint

Смотрите также мои статьи:

Источник

install ConnectX-3 on ProxMox v5.x

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

FATeknollogee

Member

Is it possible to install drivers for Mellanox ConnectX-3 in Proxmox v5.x?
The Mellanox site only has drivers for Debian 8.3.
I tried using the 8.3 drivers but I’m getting some errors.

# ./install —distro debian8.3
Log: /tmp/ofed.build.log
Logs dir: /tmp/mlnx-en.31026.logs

Below is the list of mlnx-en packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-en-utils
mlnx-en-dkms
mstflint

This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements.
One or more required packages for installing mlnx-en are missing.
/lib/modules/4.10.1-2-pve/build/scripts is required for the Installation.
Attempting to install the following missing packages:
m4 autoconf swig automake dkms autotools-dev python-libxml2 graphviz debhelper chrpath dpatch quilt libltdl-dev linux-headers-4.10.1-2-pve
Failed command: apt-get install -y m4 autoconf swig automake dkms autotools-dev python-libxml2 graphviz debhelper chrpath dpatch quilt libltdl-dev linux-headers-4.10.1-2-pve

Источник