usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

UHD DPDK freezes during initialization

ME
maximilian.engelhardt@tu-ilmenau.de
Fri, Dec 17, 2021 2:27 PM

Hello,

I'm trying to use UHD-DPDK, but fail because the driver freezes during initalization:

UHD_LOG_LEVEL=debug ./benchmark_rate --rx_rate 100e6 --rx_subdev "A:0" --rx_channels 0 --args "mgmt_addr=192.168.30.2,addr=192.168.40.2,use_dpdk=1"\

[INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400; UHD_4.2.0.git-209-gf23ab721
[DEBUG] [PREFS] Loaded user config file /root/.config/uhd.conf
EAL: Detected 32 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Invalid NUMA socket, default to 0
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: No legacy callbacks, legacy socket not created

Here, the output stops, no timeout occurs even after hours.

Without DPDK, so if I omit the argument use_dpdk=1, it works fine. Other DPDK applications also run without problems.
The problem occurs both with the UHD version 4.0.0 (using DPDK 18.11), and with the current state from the git-repository (UHD_4.2.0.git-209-gf23ab721) using DPDK 21.11.

I am using a Mellanox Technologies MCX512A-ACAT to which the USRP X310 is connected with two 10G Ethenet links., of which i tried to use one witch DPDK and the other without. (The documentation suggests to me that this is necessary for the driver to manage the device).

I have attached my uhd.conf configuration file and output of ip addr to this mail.

Does anyone have any idea how I could solve this problem?

Thanks in advance

Max

Hello, I'm trying to use UHD-DPDK, but fail because the driver freezes during initalization: > > UHD_LOG_LEVEL=debug ./benchmark_rate --rx_rate 100e6 --rx_subdev "A:0" --rx_channels 0 --args "mgmt_addr=192.168.30.2,addr=192.168.40.2,use_dpdk=1"\ > \[INFO\] \[UHD\] linux; GNU C++ version 10.2.1 20210110; Boost_107400; UHD_4.2.0.git-209-gf23ab721\ > \[DEBUG\] \[PREFS\] Loaded user config file /root/.config/uhd.conf\ > EAL: Detected 32 lcore(s)\ > EAL: Detected 1 NUMA nodes\ > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket\ > EAL: Selected IOVA mode 'VA'\ > EAL: No available hugepages reported in hugepages-2048kB\ > EAL: Probing VFIO support...\ > EAL: VFIO support initialized\ > EAL: Invalid NUMA socket, default to 0\ > EAL: Invalid NUMA socket, default to 0\ > EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.0 (socket 0)\ > mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.\ > EAL: Invalid NUMA socket, default to 0\ > EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.1 (socket 0)\ > mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.\ > EAL: No legacy callbacks, legacy socket not created Here, the output stops, no timeout occurs even after hours. Without DPDK, so if I omit the argument use_dpdk=1, it works fine. Other DPDK applications also run without problems.\ The problem occurs both with the UHD version 4.0.0 (using DPDK 18.11), and with the current state from the git-repository (UHD_4.2.0.git-209-gf23ab721) using DPDK 21.11. I am using a Mellanox Technologies MCX512A-ACAT to which the USRP X310 is connected with two 10G Ethenet links., of which i tried to use one witch DPDK and the other without. (The documentation suggests to me that this is necessary for the driver to manage the device).\ \ I have attached my uhd.conf configuration file and output of ip addr to this mail. Does anyone have any idea how I could solve this problem? Thanks in advance Max
RK
Rob Kossler
Fri, Dec 17, 2021 2:38 PM

Hi Max,
I have one suggestion, but have very low confidence that it will help. I
think that the Ettus documentation that talks about needing a non-DPDK
connection for device management might not apply to the X310 but rather
only to MPM-based devices.  So, I think you can keep both ports under DPDK
control.  And, I'm wondering in your command line if perhaps you should
just omit specifying "mgmt_addr" such that it will become default.  Perhaps
try a few cases: (1) "addr=192.168.30.2"; (2) "addr=192.168.40.2"; (3)
"addr=192.168.30.2,second_addr=192.168.40.2".
Rob

On Fri, Dec 17, 2021 at 9:28 AM maximilian.engelhardt@tu-ilmenau.de wrote:

Hello,

I'm trying to use UHD-DPDK, but fail because the driver freezes during
initalization:

UHD_LOG_LEVEL=debug ./benchmark_rate --rx_rate 100e6 --rx_subdev "A:0"

--rx_channels 0 --args "mgmt_addr=192.168.30.2,addr=192.168.40.2,use_dpdk=1"
[INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400;
UHD_4.2.0.git-209-gf23ab721
[DEBUG] [PREFS] Loaded user config file /root/.config/uhd.conf
EAL: Detected 32 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Invalid NUMA socket, default to 0
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: No legacy callbacks, legacy socket not created

Here, the output stops, no timeout occurs even after hours.

Without DPDK, so if I omit the argument use_dpdk=1, it works fine. Other
DPDK applications also run without problems.
The problem occurs both with the UHD version 4.0.0 (using DPDK 18.11), and
with the current state from the git-repository
(UHD_4.2.0.git-209-gf23ab721) using DPDK 21.11.

I am using a Mellanox Technologies MCX512A-ACAT to which the USRP X310 is
connected with two 10G Ethenet links., of which i tried to use one witch
DPDK and the other without. (The documentation suggests to me that this is
necessary for the driver to manage the device).

I have attached my uhd.conf configuration file and output of ip addr to
this mail.

Does anyone have any idea how I could solve this problem?

Thanks in advance

Max


USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com

Hi Max, I have one suggestion, but have very low confidence that it will help. I think that the Ettus documentation that talks about needing a non-DPDK connection for device management might not apply to the X310 but rather only to MPM-based devices. So, I think you can keep both ports under DPDK control. And, I'm wondering in your command line if perhaps you should just omit specifying "mgmt_addr" such that it will become default. Perhaps try a few cases: (1) "addr=192.168.30.2"; (2) "addr=192.168.40.2"; (3) "addr=192.168.30.2,second_addr=192.168.40.2". Rob On Fri, Dec 17, 2021 at 9:28 AM <maximilian.engelhardt@tu-ilmenau.de> wrote: > Hello, > > I'm trying to use UHD-DPDK, but fail because the driver freezes during > initalization: > > > UHD_LOG_LEVEL=debug ./benchmark_rate --rx_rate 100e6 --rx_subdev "A:0" > --rx_channels 0 --args "mgmt_addr=192.168.30.2,addr=192.168.40.2,use_dpdk=1" > [INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400; > UHD_4.2.0.git-209-gf23ab721 > [DEBUG] [PREFS] Loaded user config file /root/.config/uhd.conf > EAL: Detected 32 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: No available hugepages reported in hugepages-2048kB > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: Invalid NUMA socket, default to 0 > EAL: Invalid NUMA socket, default to 0 > EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.0 (socket 0) > mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000. > EAL: Invalid NUMA socket, default to 0 > EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.1 (socket 0) > mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000. > EAL: No legacy callbacks, legacy socket not created > > Here, the output stops, no timeout occurs even after hours. > > Without DPDK, so if I omit the argument use_dpdk=1, it works fine. Other > DPDK applications also run without problems. > The problem occurs both with the UHD version 4.0.0 (using DPDK 18.11), and > with the current state from the git-repository > (UHD_4.2.0.git-209-gf23ab721) using DPDK 21.11. > > I am using a Mellanox Technologies MCX512A-ACAT to which the USRP X310 is > connected with two 10G Ethenet links., of which i tried to use one witch > DPDK and the other without. (The documentation suggests to me that this is > necessary for the driver to manage the device). > > I have attached my uhd.conf configuration file and output of ip addr to > this mail. > > Does anyone have any idea how I could solve this problem? > > Thanks in advance > > Max > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >
JW
Jovian Wysocki
Fri, Dec 17, 2021 4:40 PM

Max,

I believe you might be running into an issue of loading too many drivers within dpdk.
Try running this script with sudo privileges. (Look over the script first to see what it’s doing)
Then change your file dpdk_driver location to /usr/local/lib/dpdk-pmds/

Here is an excerpt from the example file.
;dpdk_driver is the -d flag for the DPDK EAL. If DPDK doesn't pick up the driver for your NIC
;automatically, you may need this argument to point it to the folder where it can find the drivers
;Note that DPDK will attempt to load everything in that folder as a driver, so you may want to
;create a separate folder with symlinks to the librte_pmd_* and librte_mempool_* libraries.
dpdk_driver=/usr/local/lib/dpdk-pmds/

[NI]
Jovian Wysocki
Software Engineer
Aerospace/Defense/Government Business
Cell: +15127182366  |  ni.comhttps://www.ni.com/
[https://www.ni.com/content/dam/web/scene7/email/signature/email-sig-banner.png]https://www.ni.com/r/zf03uz

DISCLAIMER: The attached Code is provided As Is. It has not been tested or validated as a product, for use in a deployed application or system, or for use in hazardous environments. You assume all risks for use of the Code and use of the Code is subject to the Sample Code License Terms which can be found at: http://ni.com/samplecodelicense

NI will only perform services based on its understanding and condition that the goods or services (i) are not for the use in the production or development of any item produced, purchased, or ordered by any entity with a footnote 1 designation in the license requirement column of Supplement No. 4 to Part 744, U.S. Export Administration Regulations and (ii) such a company is not a party to the transaction.  If our understanding is incorrect, please notify us immediately because a specific authorization may be required from the U.S. Commerce Department before the transaction may proceed further.

From: Rob Kossler rkossler@nd.edu
Sent: Friday, December 17, 2021 8:38 AM
To: maximilian.engelhardt@tu-ilmenau.de
Cc: usrp-users@lists.ettus.com
Subject: [EXTERNAL] [USRP-users] Re: UHD DPDK freezes during initialization

Hi Max,
I have one suggestion, but have very low confidence that it will help. I think that the Ettus documentation that talks about needing a non-DPDK connection for device management might not apply to the X310 but rather only to MPM-based devices.  So, I think you can keep both ports under DPDK control.  And, I'm wondering in your command line if perhaps you should just omit specifying "mgmt_addr" such that it will become default.  Perhaps try a few cases: (1) "addr=192.168.30.2"; (2) "addr=192.168.40.2"; (3) "addr=192.168.30.2,second_addr=192.168.40.2".
Rob

On Fri, Dec 17, 2021 at 9:28 AM <maximilian.engelhardt@tu-ilmenau.demailto:maximilian.engelhardt@tu-ilmenau.de> wrote:

Hello,

I'm trying to use UHD-DPDK, but fail because the driver freezes during initalization:

UHD_LOG_LEVEL=debug ./benchmark_rate --rx_rate 100e6 --rx_subdev "A:0" --rx_channels 0 --args "mgmt_addr=192.168.30.2,addr=192.168.40.2,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400; UHD_4.2.0.git-209-gf23ab721
[DEBUG] [PREFS] Loaded user config file /root/.config/uhd.conf
EAL: Detected 32 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Invalid NUMA socket, default to 0
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: No legacy callbacks, legacy socket not created

Here, the output stops, no timeout occurs even after hours.

Without DPDK, so if I omit the argument use_dpdk=1, it works fine. Other DPDK applications also run without problems.
The problem occurs both with the UHD version 4.0.0 (using DPDK 18.11), and with the current state from the git-repository (UHD_4.2.0.git-209-gf23ab721) using DPDK 21.11.

I am using a Mellanox Technologies MCX512A-ACAT to which the USRP X310 is connected with two 10G Ethenet links., of which i tried to use one witch DPDK and the other without. (The documentation suggests to me that this is necessary for the driver to manage the device).

I have attached my uhd.conf configuration file and output of ip addr to this mail.

Does anyone have any idea how I could solve this problem?

Thanks in advance

Max


USRP-users mailing list -- usrp-users@lists.ettus.commailto:usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.commailto:usrp-users-leave@lists.ettus.com

INTERNAL - NI CONFIDENTIAL

Max, I believe you might be running into an issue of loading too many drivers within dpdk. Try running this script with sudo privileges. (Look over the script first to see what it’s doing) Then change your file dpdk_driver location to /usr/local/lib/dpdk-pmds/ Here is an excerpt from the example file. ;dpdk_driver is the -d flag for the DPDK EAL. If DPDK doesn't pick up the driver for your NIC ;automatically, you may need this argument to point it to the folder where it can find the drivers ;Note that DPDK will attempt to load _everything_ in that folder as a driver, so you may want to ;create a separate folder with symlinks to the librte_pmd_* and librte_mempool_* libraries. dpdk_driver=/usr/local/lib/dpdk-pmds/ [NI] Jovian Wysocki Software Engineer Aerospace/Defense/Government Business Cell: +15127182366 | ni.com<https://www.ni.com/> [https://www.ni.com/content/dam/web/scene7/email/signature/email-sig-banner.png]<https://www.ni.com/r/zf03uz> DISCLAIMER: The attached Code is provided As Is. It has not been tested or validated as a product, for use in a deployed application or system, or for use in hazardous environments. You assume all risks for use of the Code and use of the Code is subject to the Sample Code License Terms which can be found at: http://ni.com/samplecodelicense NI will only perform services based on its understanding and condition that the goods or services (i) are not for the use in the production or development of any item produced, purchased, or ordered by any entity with a footnote 1 designation in the license requirement column of Supplement No. 4 to Part 744, U.S. Export Administration Regulations and (ii) such a company is not a party to the transaction. If our understanding is incorrect, please notify us immediately because a specific authorization may be required from the U.S. Commerce Department before the transaction may proceed further. From: Rob Kossler <rkossler@nd.edu> Sent: Friday, December 17, 2021 8:38 AM To: maximilian.engelhardt@tu-ilmenau.de Cc: usrp-users@lists.ettus.com Subject: [EXTERNAL] [USRP-users] Re: UHD DPDK freezes during initialization Hi Max, I have one suggestion, but have very low confidence that it will help. I think that the Ettus documentation that talks about needing a non-DPDK connection for device management might not apply to the X310 but rather only to MPM-based devices. So, I think you can keep both ports under DPDK control. And, I'm wondering in your command line if perhaps you should just omit specifying "mgmt_addr" such that it will become default. Perhaps try a few cases: (1) "addr=192.168.30.2"; (2) "addr=192.168.40.2"; (3) "addr=192.168.30.2,second_addr=192.168.40.2". Rob On Fri, Dec 17, 2021 at 9:28 AM <maximilian.engelhardt@tu-ilmenau.de<mailto:maximilian.engelhardt@tu-ilmenau.de>> wrote: Hello, I'm trying to use UHD-DPDK, but fail because the driver freezes during initalization: > UHD_LOG_LEVEL=debug ./benchmark_rate --rx_rate 100e6 --rx_subdev "A:0" --rx_channels 0 --args "mgmt_addr=192.168.30.2,addr=192.168.40.2,use_dpdk=1" [INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400; UHD_4.2.0.git-209-gf23ab721 [DEBUG] [PREFS] Loaded user config file /root/.config/uhd.conf EAL: Detected 32 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.0 (socket 0) mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000. EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:08:00.1 (socket 0) mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000. EAL: No legacy callbacks, legacy socket not created Here, the output stops, no timeout occurs even after hours. Without DPDK, so if I omit the argument use_dpdk=1, it works fine. Other DPDK applications also run without problems. The problem occurs both with the UHD version 4.0.0 (using DPDK 18.11), and with the current state from the git-repository (UHD_4.2.0.git-209-gf23ab721) using DPDK 21.11. I am using a Mellanox Technologies MCX512A-ACAT to which the USRP X310 is connected with two 10G Ethenet links., of which i tried to use one witch DPDK and the other without. (The documentation suggests to me that this is necessary for the driver to manage the device). I have attached my uhd.conf configuration file and output of ip addr to this mail. Does anyone have any idea how I could solve this problem? Thanks in advance Max _______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com<mailto:usrp-users@lists.ettus.com> To unsubscribe send an email to usrp-users-leave@lists.ettus.com<mailto:usrp-users-leave@lists.ettus.com> INTERNAL - NI CONFIDENTIAL