On 02/01/2021 06:15 PM, Rob Kossler via USRP-users wrote:

This would seem to indicate that something very horrible went wrong in
the build or DPDK--creating ELF headers that aren't correct for the
type of hardware they're being used on.

I'm not convinced anything went wrong in the build of DPDK.  The build
didn't indicate any problems.  But I wouldn't be surprised to find out that
things aren't being installed where they are expected.

In any case, I realized a mistake in my original post which accounts for
the discrepancy in the location of the library file mentioned in the
error.  I had been fooling with the uhd.conf setting for "dkdk_driver"
which I had changed on one system but not the other.

And most recently, I accidentally commented out this setting completely and
things seem to be better.  The new error is the following.  Not sure why
I'm getting this error message.  Suggestions welcome.

EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
[ERROR] [DPDK] No available DPDK devices (ports) found!
EAL: FATAL: already called initialization.
EAL: already called initialization.

On Mon, Feb 1, 2021 at 6:23 PM Marcus D. Leech via USRP-users <
usrp-users@lists.ettus.com> wrote:

Unfortunately that’s one I can’t help with. I don’t have any DPDK capable machines at my disposal.

I vaguely recall that you have to be root for DPDK to work. Perhaps that’s the issue?

Sent from my iPhone

I'm running as root.  Here is the full command I'm using with my N310 and
the first part of the resulting output. Note that the ERROR is now a UHD
generated error message rather than an error message from the DPDK libs.

One thing I'm looking into is the "vfio-pci" module.  The instructions say
to load this via "modprobe vfio-pci" prior to binding the NIC.  But, I
didn't have to do this and it seems that this module might now be built-in
such that it is not necessary. In any case, the "dpdk-devbind" seems to
work fine so maybe I am barking up the wrong tree.

root@irisheyes9-Z240-SFF:~# benchmark_rate --rx_rate 62.5e6 --tx_rate
62.5e6 --channels="0,1,2,3"
--args="mgmt_addr=192.168.1.74,addr=192.168.61.2,use_dpdk=1"

[INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
UHD_4.0.0.0-50-ge520e3ff
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
[ERROR] [DPDK] No available DPDK devices (ports) found!
EAL: FATAL: already called initialization.
EAL: already called initialization.

On Mon, Feb 1, 2021 at 7:44 PM Marcus D Leech patchvonbraun@gmail.com
wrote:

BTW, here is the output of the dpdk-devbind status. It all looks good to me.

root@irisheyes9-Z240-SFF:~# dpdk-devbind -s

Network devices using DPDK-compatible driver

---===========
0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1584' drv=vfio-pci
unused=i40e
0000:03:00.1 'Ethernet Controller XL710 for 40GbE QSFP+ 1584' drv=vfio-pci
unused=i40e

Network devices using kernel driver

---==
0000:00:1f.6 'Ethernet Connection (2) I219-LM 15b7' if=eno1 drv=e1000e
unused=vfio-pci Active
0000:03:00.2 'Ethernet Controller XL710 for 40GbE QSFP+ 1584' if=ens4f2
drv=i40e unused=vfio-pci Active
0000:03:00.3 'Ethernet Controller XL710 for 40GbE QSFP+ 1584' if=ens4f3
drv=i40e unused=vfio-pci Active

On Mon, Feb 1, 2021 at 7:51 PM Rob Kossler rkossler@nd.edu wrote:

Has anyone successfully used DPDK with Ubuntu 20.04, UHD 4.0, Intel XL710
NIC, and N310 (or X310)?

On Mon, Feb 1, 2021 at 7:54 PM Rob Kossler rkossler@nd.edu wrote:

On Mon, Feb 1, 2021 at 9:02 PM Rob Kossler via USRP-users
usrp-users@lists.ettus.com wrote:

If I remember correctly, I believe DPDK tries to dlopen() everything
in the directory specified by the dpdk_driver parameter in the DPDK
section of uhd.conf, leading to a lot of errors similar to yours
('Invalid ELF header' and the like). Having the correct collection of
.so files in that directory is key.

What's worked for me in the past when using DPDK with an Intel XL710
is creating a directory (I used /usr/local/lib/dpdk-pmds) and copying
a specific set of DPDK .so files into this directory:

• librte_mempool_ring.so
• librte_pdump.so (I think this one is optional--I had been trying to
  get packet dumps from DPDK a while back)
• librte_pmd_i40e.so
• librte_pmd_ixgbe.so (may be optional?)
• librte_pmd_pcap.so (this one is also optional, I think)
• librte_pmd_ring.so

(Symlinking to the actual libraries wherever they get installed
instead of copying them into the directory would probably work as
well.)

Then, make sure that the dpdk-driver key in the [use_dpdk=1] section
of uhd.conf points to that directory:
dpdk_driver = /usr/local/lib/dpdk-pmds

Hopefully that will resolve the issue and get you a little further
down the road.

Best regards,
Aaron

Hi Aaron,
This did indeed help.  Now I am able to run ONCE successfully.  After that
I get an error.  Same behavior on both systems.  Not yet sure how to clear
the error.  I played with dpdk_link_timeout and even tried resetting the
N310 using "overlay rm n310 && overlay add n310 && systemctl restart
usrp-hwd".  But no luck.
Rob

// First run succeeds
root@irisheyes5-hp-z240-sff:~# uhd_image_loader
--args="addr=192.168.1.88,type=n3xx,fpga=XG"
[INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
UHD_4.0.0.0-50-ge520e3ff
[INFO] [MPMD] Initializing 1 device(s) in parallel with args:
mgmt_addr=192.168.1.88,type=n3xx,product=n310,serial=3144673,claimed=False,skip_init=1
[WARNING] [MPM.RPCServer] A timeout event occured!
[INFO] [MPMD] Claimed device without full initialization.
[INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.
[INFO] [MPM.PeriphManager] Updating component fpga' [INFO] [MPM.PeriphManager] Updating component dts'
[INFO] [MPM.RPCServer] Resetting peripheral manager.
[INFO] [MPM.PeriphManager] Device serial number: 3144673
[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).
[INFO] [MPM.PeriphManager] init() called with device args
`clock_source=internal,time_source=internal'.
[INFO] [MPMD IMAGE LOADER] Update component function succeeded.
root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
--rx_rate=62.5e6 --channels="0,1,2,3"
--args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"

[INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
UHD_4.0.0.0-50-ge520e3ff
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL:  using IOMMU type 1 (Type 1)
EAL: PCI device 0000:03:00.1 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:03:00.2 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:03:00.3 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
[00:00:00.000152] Creating the usrp device with:
use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args:
mgmt_addr=192.168.1.88,type=n3xx,product=n310,serial=3144673,claimed=False,use_dpdk=1,addr=192.168.60.2
[INFO] [MPM.PeriphManager] init() called with device args
`mgmt_addr=192.168.1.88,product=n310,use_dpdk=1,clock_source=internal,time_source=internal'.
Using Device: Single USRP:
Device: N300-Series Device
Mboard 0: n310
RX Channel: 0
RX DSP: 0
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 1
RX DSP: 1
RX Dboard: A
RX Subdev: Magnesium
RX Channel: 2
RX DSP: 2
RX Dboard: B
RX Subdev: Magnesium
RX Channel: 3
RX DSP: 3
RX Dboard: B
RX Subdev: Magnesium
TX Channel: 0
TX DSP: 0
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 1
TX DSP: 1
TX Dboard: A
TX Subdev: Magnesium
TX Channel: 2
TX DSP: 2
TX Dboard: B
TX Subdev: Magnesium
TX Channel: 3
TX DSP: 3
TX Dboard: B
TX Subdev: Magnesium

[00:00:03.21715319] Setting device timestamp to 0...
[INFO] [MULTI_USRP]    1) catch time transition at pps edge
[INFO] [MULTI_USRP]    2) set times next pps (synchronously)
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
[WARNING] [0/Radio#1] Attempting to set tick rate to 0. Skipping.
[WARNING] [0/Radio#1] Attempting to set tick rate to 0. Skipping.
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1989
[00:00:04.907401082] Testing receive rate 62.500000 Msps on 4 channels
[00:00:04.914615576] Testing transmit rate 62.500000 Msps on 4 channels
[00:00:15.167869894] Benchmark complete.

Benchmark rate summary:
Num received samples:    2549794336
Num dropped samples:      0
Num overruns detected:    0
Num transmitted samples:  2499910452
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected:  0
Num late commands:        0
Num timeouts (Tx):        0
Num timeouts (Rx):        0

Done!

// Second run fails
root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
--rx_rate=62.5e6 --channels="0,1,2,3"
--args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"

[INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
UHD_4.0.0.0-50-ge520e3ff
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL:  using IOMMU type 1 (Type 1)
EAL: PCI device 0000:03:00.1 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:03:00.2 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:03:00.3 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
[ERROR] [DPDK] All DPDK links did not report as up!
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [UHD] Device discovery error: RuntimeError: DPDK: All DPDK links
did not report as up!
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
initialization
[00:00:00.000122] Creating the usrp device with:
use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL
initialization
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
initialization
Error: LookupError: KeyError: No devices found for ----->
Device Address:
use_dpdk: 1
mgmt_addr: 192.168.1.88
addr: 192.168.60.2

// Third run fails
root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
--rx_rate=62.5e6 --channels="0,1,2,3"
--args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"

[INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
UHD_4.0.0.0-50-ge520e3ff
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL:  using IOMMU type 1 (Type 1)
EAL: PCI device 0000:03:00.1 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:03:00.2 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:03:00.3 on NUMA socket -1
EAL:  Invalid NUMA socket, default to 0
EAL:  probe driver: 8086:1584 net_i40e
[ERROR] [DPDK] All DPDK links did not report as up!
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [UHD] Device discovery error: RuntimeError: DPDK: All DPDK links
did not report as up!
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
initialization
[00:00:00.000148] Creating the usrp device with:
use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL
initialization
EAL: FATAL: already called initialization.
EAL: already called initialization.
[ERROR] [DPDK] Error with EAL initialization
[ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
initialization
Error: LookupError: KeyError: No devices found for ----->
Device Address:
use_dpdk: 1
mgmt_addr: 192.168.1.88
addr: 192.168.60.2

On Tue, Feb 2, 2021 at 11:53 AM Aaron Rossetto via USRP-users <
usrp-users@lists.ettus.com> wrote:

I notice in the second and subsequent runs, you get this message from UHD:

[ERROR] [DPDK] All DPDK links did not report as up!

One of the other issues I've noticed with DPDK (and unfortunately
don't have an answer for) is that link detection seems to have issues.
I'm not sure if this is an XL710-specific problem or whether it's more
widespread, but I added some code to try to mitigate things somewhat
in commit eada49e4d. This commit checks the link status at
250-millisecond intervals for up to the link status timeout (default 1
second) in case the links take a while to register as up. One thing
you could try is overriding the default link status timeout and
increasing the value, which you can do by adding a dpdk_link_timeout=X
line to the [use_dpdk=1] section of your uhd.conf file, where X is the
new timeout in number of milliseconds.

Best regards,
Aaron

On Tue, Feb 2, 2021 at 1:47 PM Rob Kossler rkossler@nd.edu wrote: