Hi Marcus,

Anyone else able to look at this with ability to Tx on x310?

I have looked at all the counters on the server, and it I don’t see anything unusual.
Let me know if you need me to up/down the interface (to reset) and provide you some counters….

Here is more info:

~$ sudo ethtool -i eno2
driver: ixgbe
version: 5.19.0-1010-nvidia-lowlatency
firmware-version: 0x80001743, 1.3082.0
expansion-rom-version: 
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

~$ lspci -v -s 000:04:00.1
04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
        DeviceName: Intel Ethernet X550 #2
        Subsystem: Super Micro Computer Inc Ethernet Controller 10G X550T
        Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0
        Memory at 200fff400000 (64-bit, prefetchable) [size=4M]
        Memory at 200fffc00000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at 9a100000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

Good catch!!!

I updated it now, but still have the same result…printing S’s when directly connected to the powerful server.

I repeated the test on the USRP PC (2974) with the same results, printed U’s.

For the server, I cleared the NIC stats, then ran the program for a few seconds.  Here is the resulting statistics for the NIC, and I don’t see anything abnormal.  It’s my understanding that for intel NICS, fdir_miss is normal.

Do you see anything abnormal?  Any chance someone else could try this .cpp file?  I appreciate the help…. I not sure what to try next.

sudo ethtool -S eno2
NIC statistics:
     rx_packets: 146651
     tx_packets: 1065751
     rx_bytes: 12030774
     tx_bytes: 8403361197
     rx_pkts_nic: 146651
     tx_pkts_nic: 1065750
     rx_bytes_nic: 12617378
     tx_bytes_nic: 8407627498
     lsc_int: 2
     tx_busy: 0
     non_eop_descs: 20
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 0
     broadcast: 4
     rx_no_buffer_count: 0
     collisions: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     hw_rsc_aggregated: 0
     hw_rsc_flushed: 0
     fdir_match: 0
     fdir_miss: 146646
     fdir_overflow: 0
     rx_fifo_errors: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_length_errors: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     tx_flow_control_xon: 0
     rx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_flow_control_xoff: 0
     rx_csum_offload_errors: 0
     alloc_rx_page: 326466
     alloc_rx_page_failed: 0
     alloc_rx_buff_failed: 0
     rx_no_dma_resources: 0
     os2bmc_rx_by_bmc: 0
     os2bmc_tx_by_bmc: 0
     os2bmc_tx_by_host: 0
     os2bmc_rx_by_host: 0
     tx_hwtstamp_timeouts: 0
     tx_hwtstamp_skipped: 0
     rx_hwtstamp_cleared: 0
     tx_ipsec: 0
     rx_ipsec: 0
     fcoe_bad_fccrc: 0
     rx_fcoe_dropped: 0
     rx_fcoe_packets: 0
     rx_fcoe_dwords: 0
     fcoe_noddp: 0
     fcoe_noddp_ext_buff: 0
     tx_fcoe_packets: 0
     tx_fcoe_dwords: 0
     tx_queue_0_packets: 0
     tx_queue_0_bytes: 0
     tx_queue_1_packets: 0
     tx_queue_1_bytes: 0
     tx_queue_2_packets: 45
     tx_queue_2_bytes: 6177
     tx_queue_3_packets: 0
     tx_queue_3_bytes: 0
     tx_queue_4_packets: 0
     tx_queue_4_bytes: 0
     tx_queue_5_packets: 0
     tx_queue_5_bytes: 0
     tx_queue_6_packets: 0
     tx_queue_6_bytes: 0
     tx_queue_7_packets: 0
     tx_queue_7_bytes: 0
     tx_queue_8_packets: 0
     tx_queue_8_bytes: 0
     tx_queue_9_packets: 0
     tx_queue_9_bytes: 0
     tx_queue_10_packets: 6
     tx_queue_10_bytes: 500
     tx_queue_11_packets: 8
     tx_queue_11_bytes: 676
     tx_queue_12_packets: 0
     tx_queue_12_bytes: 0
     tx_queue_13_packets: 0
     tx_queue_13_bytes: 0
     tx_queue_14_packets: 0
     tx_queue_14_bytes: 0
     tx_queue_15_packets: 0
     tx_queue_15_bytes: 0
     tx_queue_16_packets: 0
     tx_queue_16_bytes: 0
     tx_queue_17_packets: 0
     tx_queue_17_bytes: 0
     tx_queue_18_packets: 0
     tx_queue_18_bytes: 0
     tx_queue_19_packets: 0
     tx_queue_19_bytes: 0
     tx_queue_20_packets: 0
     tx_queue_20_bytes: 0
     tx_queue_21_packets: 0
     tx_queue_21_bytes: 0
     tx_queue_22_packets: 0
     tx_queue_22_bytes: 0
     tx_queue_23_packets: 0
     tx_queue_23_bytes: 0
     tx_queue_24_packets: 1
     tx_queue_24_bytes: 58
     tx_queue_25_packets: 0
     tx_queue_25_bytes: 0
     tx_queue_26_packets: 0
     tx_queue_26_bytes: 0
     tx_queue_27_packets: 0
     tx_queue_27_bytes: 0
     tx_queue_28_packets: 0
     tx_queue_28_bytes: 0
     tx_queue_29_packets: 0
     tx_queue_29_bytes: 0
     tx_queue_30_packets: 0
     tx_queue_30_bytes: 0
     tx_queue_31_packets: 0
     tx_queue_31_bytes: 0
     tx_queue_32_packets: 0
     tx_queue_32_bytes: 0
     tx_queue_33_packets: 0
     tx_queue_33_bytes: 0
     tx_queue_34_packets: 0
     tx_queue_34_bytes: 0
     tx_queue_35_packets: 0
     tx_queue_35_bytes: 0
     tx_queue_36_packets: 0
     tx_queue_36_bytes: 0
     tx_queue_37_packets: 0
     tx_queue_37_bytes: 0
     tx_queue_38_packets: 0
     tx_queue_38_bytes: 0
     tx_queue_39_packets: 0
     tx_queue_39_bytes: 0
     tx_queue_40_packets: 0
     tx_queue_40_bytes: 0
     tx_queue_41_packets: 0
     tx_queue_41_bytes: 0
     tx_queue_42_packets: 0
     tx_queue_42_bytes: 0
     tx_queue_43_packets: 0
     tx_queue_43_bytes: 0
     tx_queue_44_packets: 0
     tx_queue_44_bytes: 0
     tx_queue_45_packets: 0
     tx_queue_45_bytes: 0
     tx_queue_46_packets: 0
     tx_queue_46_bytes: 0
     tx_queue_47_packets: 0
     tx_queue_47_bytes: 0
     tx_queue_48_packets: 0
     tx_queue_48_bytes: 0
     tx_queue_49_packets: 0
     tx_queue_49_bytes: 0
     tx_queue_50_packets: 0
     tx_queue_50_bytes: 0
     tx_queue_51_packets: 0
     tx_queue_51_bytes: 0
     tx_queue_52_packets: 0
     tx_queue_52_bytes: 0
     tx_queue_53_packets: 0
     tx_queue_53_bytes: 0
     tx_queue_54_packets: 0
     tx_queue_54_bytes: 0
     tx_queue_55_packets: 0
     tx_queue_55_bytes: 0
     tx_queue_56_packets: 4
     tx_queue_56_bytes: 468
     tx_queue_57_packets: 0
     tx_queue_57_bytes: 0
     tx_queue_58_packets: 0
     tx_queue_58_bytes: 0
     tx_queue_59_packets: 0
     tx_queue_59_bytes: 0
     tx_queue_60_packets: 1061658
     tx_queue_60_bytes: 8403020004
     tx_queue_61_packets: 4029
     tx_queue_61_bytes: 333314
     tx_queue_62_packets: 0
     tx_queue_62_bytes: 0
     tx_queue_63_packets: 0
     tx_queue_63_bytes: 0
     rx_queue_0_packets: 5
     rx_queue_0_bytes: 300
     rx_queue_1_packets: 0
     rx_queue_1_bytes: 0
     rx_queue_2_packets: 0
     rx_queue_2_bytes: 0
     rx_queue_3_packets: 0
     rx_queue_3_bytes: 0
     rx_queue_4_packets: 0
     rx_queue_4_bytes: 0
     rx_queue_5_packets: 0
     rx_queue_5_bytes: 0
     rx_queue_6_packets: 0
     rx_queue_6_bytes: 0
     rx_queue_7_packets: 0
     rx_queue_7_bytes: 0
     rx_queue_8_packets: 0
     rx_queue_8_bytes: 0
     rx_queue_9_packets: 0
     rx_queue_9_bytes: 0
     rx_queue_10_packets: 0
     rx_queue_10_bytes: 0
     rx_queue_11_packets: 0
     rx_queue_11_bytes: 0
     rx_queue_12_packets: 0
     rx_queue_12_bytes: 0
     rx_queue_13_packets: 0
     rx_queue_13_bytes: 0
     rx_queue_14_packets: 0
     rx_queue_14_bytes: 0
     rx_queue_15_packets: 0
     rx_queue_15_bytes: 0
     rx_queue_16_packets: 0
     rx_queue_16_bytes: 0
     rx_queue_17_packets: 0
     rx_queue_17_bytes: 0
     rx_queue_18_packets: 0
     rx_queue_18_bytes: 0
     rx_queue_19_packets: 0
     rx_queue_19_bytes: 0
     rx_queue_20_packets: 146646
     rx_queue_20_bytes: 12030474
     rx_queue_21_packets: 0
     rx_queue_21_bytes: 0
     rx_queue_22_packets: 0
     rx_queue_22_bytes: 0
     rx_queue_23_packets: 0
     rx_queue_23_bytes: 0
     rx_queue_24_packets: 0
     rx_queue_24_bytes: 0
     rx_queue_25_packets: 0
     rx_queue_25_bytes: 0
     rx_queue_26_packets: 0
     rx_queue_26_bytes: 0
     rx_queue_27_packets: 0
     rx_queue_27_bytes: 0
     rx_queue_28_packets: 0
     rx_queue_28_bytes: 0
     rx_queue_29_packets: 0
     rx_queue_29_bytes: 0
     rx_queue_30_packets: 0
     rx_queue_30_bytes: 0
     rx_queue_31_packets: 0
     rx_queue_31_bytes: 0
     rx_queue_32_packets: 0
     rx_queue_32_bytes: 0
     rx_queue_33_packets: 0
     rx_queue_33_bytes: 0
     rx_queue_34_packets: 0
     rx_queue_34_bytes: 0
     rx_queue_35_packets: 0
     rx_queue_35_bytes: 0
     rx_queue_36_packets: 0
     rx_queue_36_bytes: 0
     rx_queue_37_packets: 0
     rx_queue_37_bytes: 0
     rx_queue_38_packets: 0
     rx_queue_38_bytes: 0
     rx_queue_39_packets: 0
     rx_queue_39_bytes: 0
     rx_queue_40_packets: 0
     rx_queue_40_bytes: 0
     rx_queue_41_packets: 0
     rx_queue_41_bytes: 0
     rx_queue_42_packets: 0
     rx_queue_42_bytes: 0
     rx_queue_43_packets: 0
     rx_queue_43_bytes: 0
     rx_queue_44_packets: 0
     rx_queue_44_bytes: 0
     rx_queue_45_packets: 0
     rx_queue_45_bytes: 0
     rx_queue_46_packets: 0
     rx_queue_46_bytes: 0
     rx_queue_47_packets: 0
     rx_queue_47_bytes: 0
     rx_queue_48_packets: 0
     rx_queue_48_bytes: 0
     rx_queue_49_packets: 0
     rx_queue_49_bytes: 0
     rx_queue_50_packets: 0
     rx_queue_50_bytes: 0
     rx_queue_51_packets: 0
     rx_queue_51_bytes: 0
     rx_queue_52_packets: 0
     rx_queue_52_bytes: 0
     rx_queue_53_packets: 0
     rx_queue_53_bytes: 0
     rx_queue_54_packets: 0
     rx_queue_54_bytes: 0
     rx_queue_55_packets: 0
     rx_queue_55_bytes: 0
     rx_queue_56_packets: 0
     rx_queue_56_bytes: 0
     rx_queue_57_packets: 0
     rx_queue_57_bytes: 0
     rx_queue_58_packets: 0
     rx_queue_58_bytes: 0
     rx_queue_59_packets: 0
     rx_queue_59_bytes: 0
     rx_queue_60_packets: 0
     rx_queue_60_bytes: 0
     rx_queue_61_packets: 0
     rx_queue_61_bytes: 0
     rx_queue_62_packets: 0
     rx_queue_62_bytes: 0
     rx_queue_63_packets: 0
     rx_queue_63_bytes: 0
     tx_pb_0_pxon: 0
     tx_pb_0_pxoff: 0
     tx_pb_1_pxon: 0
     tx_pb_1_pxoff: 0
     tx_pb_2_pxon: 0
     tx_pb_2_pxoff: 0
     tx_pb_3_pxon: 0
     tx_pb_3_pxoff: 0
     tx_pb_4_pxon: 0
     tx_pb_4_pxoff: 0
     tx_pb_5_pxon: 0
     tx_pb_5_pxoff: 0
     tx_pb_6_pxon: 0
     tx_pb_6_pxoff: 0
     tx_pb_7_pxon: 0
     tx_pb_7_pxoff: 0
     rx_pb_0_pxon: 0
     rx_pb_0_pxoff: 0
     rx_pb_1_pxon: 0
     rx_pb_1_pxoff: 0
     rx_pb_2_pxon: 0
     rx_pb_2_pxoff: 0
     rx_pb_3_pxon: 0
     rx_pb_3_pxoff: 0
     rx_pb_4_pxon: 0
     rx_pb_4_pxoff: 0
     rx_pb_5_pxon: 0
     rx_pb_5_pxoff: 0
     rx_pb_6_pxon: 0
     rx_pb_6_pxoff: 0
     rx_pb_7_pxon: 0
     rx_pb_7_pxoff: 0

Here is some interesting info that I could not explain.

• Ran on server at 200Msps, got S’s with uhd thread priority at 0.90

• Ran on server at 100Msps, got S’s with uhd thread priority at 0.90

• Ran on server at 6.25Msps, got S’s with uhd thread priority at 0.90

Why would I get S’s at 6.25Msps boggles my mind.  This is the 192.168.30.2 address.

Then on the USRP PC (I think you called it a SOM), keeping mind this is a 2974, with IP 192.168.40.2 address.  Remember this is an internal switch.

• Ran on USRP PC at 200Msps, got U’s with uhd thread priority at 0.90

• Ran on USRP PC at 100Msps, no U’s after 12 min with uhd thread priority at 0.90

Notice NO U’s for 100Msps.

While doing this, I noticed some odd behavior.  With Tx running on 40.2 (the USRP PC), I could not start a test on 30.2 (on the server).  30.2 would give an error

Error: LookupError: KeyError: No devices found for ----->

Empty Device Address

Also doing this:  uhd_usrp_probe

[INFO] [UHD] linux; GNU C++ version 11.3.0; Boost_107400; UHD_4.4.0

Error: LookupError: KeyError: No devices found for ----->

Empty Device Address

They should run independently?

I went back and looked at the 2974…. Got it.  It’s one x310 with 2 interfaces (30.2, 40.2), so it would make sense that the second instance doesn’t work.  Thank you.

In regards to the 8000 MTU, I set this up on the server (directly connected to 30.2), I still get the same trickle of S’s at low sps of 6.25M.

Here are the results of an Rx at 200e6, a Tx at 200e6, Tx 6.25e6, Tx 1.0e6.  So strange that I get out of sequence at 6.25

~/uhd_versions/uhd_4.4.0.0/host/build/examples$ ./benchmark_rate --args="type=x300,addr=192.168.30.2" --rx_rate=200e6 --duration=30

[INFO] [UHD] linux; GNU C++ version 11.4.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[00:00:00.000198] Creating the usrp device with: type=x300,addr=192.168.30.2...
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Using Device: Single USRP:
  Device: X-Series Device
  Mboard 0: X310
  RX Channel: 0
    RX DSP: 0
    RX Dboard: A
    RX Subdev: UBX RX
  RX Channel: 1
    RX DSP: 1
    RX Dboard: B
    RX Subdev: UBX RX
  TX Channel: 0
    TX DSP: 0
    TX Dboard: A
    TX Subdev: UBX TX
  TX Channel: 1
    TX DSP: 1
    TX Dboard: B
    TX Subdev: UBX TX

[00:00:03.370079690] Setting device timestamp to 0...
[00:00:03.372464052] Testing receive rate 200.000000 Msps on 1 channels
[00:00:33.373075031] Benchmark complete.


Benchmark rate summary:
  Num received samples:     6000007360
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  0
  Num sequence errors (Tx): 0
  Num sequence errors (Rx): 0
  Num underruns detected:   0
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

~/uhd_versions/uhd_4.4.0.0/host/build/examples$ ./benchmark_rate --args="type=x300,addr=192.168.30.2" --tx_rate=200e6 --duration=30

[INFO] [UHD] linux; GNU C++ version 11.4.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[00:00:00.000157] Creating the usrp device with: type=x300,addr=192.168.30.2...
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Using Device: Single USRP:
  Device: X-Series Device
  Mboard 0: X310
  RX Channel: 0
    RX DSP: 0
    RX Dboard: A
    RX Subdev: UBX RX
  RX Channel: 1
    RX DSP: 1
    RX Dboard: B
    RX Subdev: UBX RX
  TX Channel: 0
    TX DSP: 0
    TX Dboard: A
    TX Subdev: UBX TX
  TX Channel: 1
    TX DSP: 1
    TX Dboard: B
    TX Subdev: UBX TX

[00:00:03.415515038] Setting device timestamp to 0...
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1996
[00:00:03.418182017] Testing transmit rate 200.000000 Msps on 1 channels
USSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS[00:00:33.419713489] Benchmark complete.


Benchmark rate summary:
  Num received samples:     0
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  6000235480
  Num sequence errors (Tx): 76
  Num sequence errors (Rx): 0
  Num underruns detected:   1
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

~/uhd_versions/uhd_4.4.0.0/host/build/examples$ ./benchmark_rate --args="type=x300,addr=192.168.30.2" --tx_rate=6.25e6 --duration=30

[INFO] [UHD] linux; GNU C++ version 11.4.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[00:00:00.000187] Creating the usrp device with: type=x300,addr=192.168.30.2...
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Using Device: Single USRP:
  Device: X-Series Device
  Mboard 0: X310
  RX Channel: 0
    RX DSP: 0
    RX Dboard: A
    RX Subdev: UBX RX
  RX Channel: 1
    RX DSP: 1
    RX Dboard: B
    RX Subdev: UBX RX
  TX Channel: 0
    TX DSP: 0
    TX Dboard: A
    TX Subdev: UBX TX
  TX Channel: 1
    TX DSP: 1
    TX Dboard: B
    TX Subdev: UBX TX

[00:00:03.350475486] Setting device timestamp to 0...
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1996
[00:00:03.353129760] Testing transmit rate 6.250000 Msps on 1 channels
SSSSS[00:00:33.375330149] Benchmark complete.


Benchmark rate summary:
  Num received samples:     0
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  187641964
  Num sequence errors (Tx): 5
  Num sequence errors (Rx): 0
  Num underruns detected:   0
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

~/uhd_versions/uhd_4.4.0.0/host/build/examples$ ./benchmark_rate --args="type=x300,addr=192.168.30.2" --tx_rate=1e6 --duration=30

[INFO] [UHD] linux; GNU C++ version 11.4.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[00:00:00.000162] Creating the usrp device with: type=x300,addr=192.168.30.2...
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Using Device: Single USRP:
  Device: X-Series Device
  Mboard 0: X310
  RX Channel: 0
    RX DSP: 0
    RX Dboard: A
    RX Subdev: UBX RX
  RX Channel: 1
    RX DSP: 1
    RX Dboard: B
    RX Subdev: UBX RX
  TX Channel: 0
    TX DSP: 0
    TX Dboard: A
    TX Subdev: UBX TX
  TX Channel: 1
    TX DSP: 1
    TX Dboard: B
    TX Subdev: UBX TX

[00:00:03.351625101] Setting device timestamp to 0...
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Setting TX spp to 1996
[00:00:03.354355919] Testing transmit rate 1.000000 Msps on 1 channels
[00:00:33.473151517] Benchmark complete.


Benchmark rate summary:
  Num received samples:     0
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  30117644
  Num sequence errors (Tx): 0
  Num sequence errors (Rx): 0
  Num underruns detected:   0
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

I also got out of sequence errors at 1e6, I just had to wait 60 seconds instead of 30.

Hi,

Thanks for the support on this.  We purchased a second unit.  The problem on the 2nd unit is no longer present after installing the same SFP + cable, and connecting to the same server running the same program.

Thanks…