usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

DPDK error : net_mlx5: port 1 empty mbuf pool

J
jmaloyan@umass.edu
Mon, May 6, 2024 11:56 AM

Hello,

I am currently getting the following error when I try to run dpdk…

net_mlx5: port 1 empty mbuf pool

net_mlx5: port 1 Rx queue allocation failed: Cannot allocate memory

I am wondering if anyone has gotten a similar issue and how they fixed it. I double checked my uhd.conf and made sure my driver was in the right place, but things seem fine. Additionally, I believe dpdk-devbind.py shows the correct output. Below is the full output of the error, my uhd.conf, and dpdk-devbind

sudo uhd_find_devices --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.184,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.5.0.HEAD-0-g471af98f

EAL: Detected 32 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'VA'

EAL: No available hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: PCI device 0000:06:00.0 on NUMA socket -1

EAL: probe driver: 15b3:1017 net_mlx5

EAL: PCI device 0000:06:00.1 on NUMA socket -1

EAL: probe driver: 15b3:1017 net_mlx5

net_mlx5: port 1 empty mbuf pool

net_mlx5: port 1 Rx queue allocation failed: Cannot allocate memory

[ERROR] [DPDK] Port 1: Could not start device

EAL: FATAL: already called initialization.

EAL: already called initialization.

[ERROR] [DPDK] Error with EAL initialization

[ERROR] [UHD] Device discovery error: RuntimeError: DPDK: Failure to start device

[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL initialization

No UHD Devices Found

———————————————————————————————————————-

python3 /usr/bin/dpdk-devbind.py --status

Network devices using kernel driver

---==

0000:00:14.3 'Device 7a70' if=wlp0s20f3 drv=iwlwifi unused=vfio-pci *Active*

0000:06:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp6s0f0np0 drv=mlx5_core unused=vfio-pci *Active*

0000:06:00.1 'MT27800 Family [ConnectX-5] 1017' if=enp6s0f1np1 drv=mlx5_core unused=vfio-pci *Active*

0000:09:00.0 'Device 125c' if=eno2 drv=igc unused=vfio-pci

No 'Baseband' devices detected

==============================

No 'Crypto' devices detected

============================

No 'Eventdev' devices detected

==============================

No 'Mempool' devices detected

=============================

No 'Compress' devices detected

==============================

No 'Misc (rawdev)' devices detected

---==

———————————————————————————————————————————

[use_dpdk=1]

dpdk_mtu=9000

dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/

dpdk_corelist=8,9,10

dpdk_num_mbufs=4095

dpdk_mbuf_cache_size=315

[dpdk_mac=b8:3f:d2:bc:52:8a]

dpdk_lcore = 9

dpdk_ipv4 = 192.168.10.1/24

dpdk_num_desc = 4096

[dpdk_mac=b8:3f:d2:bc:52:8b]

dpdk_lcore = 10

dpdk_ipv4 = 192.168.20.1/24

dpdk_num_desc = 4096

Hello, I am currently getting the following error when I try to run dpdk… `net_mlx5: port 1 empty mbuf pool` `net_mlx5: port 1 Rx queue allocation failed: Cannot allocate memory` I am wondering if anyone has gotten a similar issue and how they fixed it. I double checked my uhd.conf and made sure my driver was in the right place, but things seem fine. Additionally, I believe dpdk-devbind.py shows the correct output. Below is the full output of the error, my uhd.conf, and dpdk-devbind `sudo uhd_find_devices --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.184,use_dpdk=1"` `[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.5.0.HEAD-0-g471af98f` `EAL: Detected 32 lcore(s)` `EAL: Detected 1 NUMA nodes` `EAL: Multi-process socket /var/run/dpdk/rte/mp_socket` `EAL: Selected IOVA mode 'VA'` `EAL: No available hugepages reported in hugepages-1048576kB` `EAL: Probing VFIO support...` `EAL: VFIO support initialized` `EAL: PCI device 0000:06:00.0 on NUMA socket -1` `EAL: probe driver: 15b3:1017 net_mlx5` `EAL: PCI device 0000:06:00.1 on NUMA socket -1` `EAL: probe driver: 15b3:1017 net_mlx5` `net_mlx5: port 1 empty mbuf pool` `net_mlx5: port 1 Rx queue allocation failed: Cannot allocate memory` `[ERROR] [DPDK] Port 1: Could not start device` `EAL: FATAL: already called initialization.` `EAL: already called initialization.` `[ERROR] [DPDK] Error with EAL initialization` `[ERROR] [UHD] Device discovery error: RuntimeError: DPDK: Failure to start device` `[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL initialization` `No UHD Devices Found` ———————————————————————————————————————- `python3 /usr/bin/dpdk-devbind.py --status` `Network devices using kernel driver` `===================================` `0000:00:14.3 'Device 7a70' if=wlp0s20f3 drv=iwlwifi unused=vfio-pci *Active*` `0000:06:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp6s0f0np0 drv=mlx5_core unused=vfio-pci *Active*` `0000:06:00.1 'MT27800 Family [ConnectX-5] 1017' if=enp6s0f1np1 drv=mlx5_core unused=vfio-pci *Active*` `0000:09:00.0 'Device 125c' if=eno2 drv=igc unused=vfio-pci ` `No 'Baseband' devices detected` `==============================` `No 'Crypto' devices detected` `============================` `No 'Eventdev' devices detected` `==============================` `No 'Mempool' devices detected` `=============================` `No 'Compress' devices detected` `==============================` `No 'Misc (rawdev)' devices detected` `===================================` ——————————————————————————————————————————— `[use_dpdk=1]` `dpdk_mtu=9000` `dpdk_driver=/usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/` `dpdk_corelist=8,9,10` `dpdk_num_mbufs=4095` `dpdk_mbuf_cache_size=315` `[dpdk_mac=b8:3f:d2:bc:52:8a]` `dpdk_lcore = 9` `dpdk_ipv4 = 192.168.10.1/24` `dpdk_num_desc = 4096` `[dpdk_mac=b8:3f:d2:bc:52:8b]` `dpdk_lcore = 10` `dpdk_ipv4 = 192.168.20.1/24` `dpdk_num_desc = 4096`
E
ettus@basti.rocks
Mon, May 6, 2024 1:29 PM

Hey there,

can you try to remove the ‘dpdk_num_desc’ options the the uhd.conf file?

I can reproduce this exact error with the line in the config file. Otherwise please provide more info on the device and FPGA image.

Additionally: If i call the uhd_find_devices command with the argument type=x4xx at least the [ERROR] [X300] is no longer there (assuming you have a x4xx device).

Best Regards,
Sebastian

Hey there, can you try to remove the ‘dpdk_num_desc’ options the the uhd.conf file? I can reproduce this exact error with the line in the config file. Otherwise please provide more info on the device and FPGA image. Additionally: If i call the uhd_find_devices command with the argument type=x4xx at least the \[ERROR\] \[X300\] is no longer there (assuming you have a x4xx device). Best Regards,\ Sebastian
J
jmaloyan@umass.edu
Mon, May 6, 2024 1:40 PM

After removing the lines you mentioned it seems it is now able to return successfully…

sudo uhd_find_devices --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.184,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.5.0.HEAD-0-g471af98f

EAL: Detected 32 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'VA'

EAL: No available hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: PCI device 0000:06:00.0 on NUMA socket -1

EAL: probe driver: 15b3:1017 net_mlx5

EAL: PCI device 0000:06:00.1 on NUMA socket -1

EAL: probe driver: 15b3:1017 net_mlx5

--------------------------------------------------

-- UHD Device 0

--------------------------------------------------

Device Address:

serial: 329053B

claimed: False

fpga: CG_400

mgmt_addr: 192.168.1.184

name: ni-x4xx-329053B

product: x410

type: x4xx

Thanks. Also, I am using a custom image, however, the same issue persisted when I was trying with the default CG_400 x410 image

After removing the lines you mentioned it seems it is now able to return successfully… `sudo uhd_find_devices --args "addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.184,use_dpdk=1"` `[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11; UHD_4.5.0.HEAD-0-g471af98f` `EAL: Detected 32 lcore(s)` `EAL: Detected 1 NUMA nodes` `EAL: Multi-process socket /var/run/dpdk/rte/mp_socket` `EAL: Selected IOVA mode 'VA'` `EAL: No available hugepages reported in hugepages-1048576kB` `EAL: Probing VFIO support...` `EAL: VFIO support initialized` `EAL: PCI device 0000:06:00.0 on NUMA socket -1` `EAL: probe driver: 15b3:1017 net_mlx5` `EAL: PCI device 0000:06:00.1 on NUMA socket -1` `EAL: probe driver: 15b3:1017 net_mlx5` `--------------------------------------------------` `-- UHD Device 0` `--------------------------------------------------` `Device Address:` ` serial: 329053B` ` claimed: False` ` fpga: CG_400` ` mgmt_addr: 192.168.1.184` ` name: ni-x4xx-329053B` ` product: x410` ` type: x4xx` Thanks. Also, I am using a custom image, however, the same issue persisted when I was trying with the default CG_400 x410 image