usrp-users@lists.ettus.com
Discussion and technical support related to USRP, UHD, RFNoC
View all threadsx310 Not Sending Jumbo Packets (>8000 bytes) as 1 Packet on 10GigE
We have followed all instructions on the website and searched the forums with no solutions. I have provided much evidence below that USRP/UHD driver is not behaving properly in this case.
We are trying to send a 8192 byte packet to the x310 with the latest UHD 4.6 (FPGA 39.2) which also didn’t seem to work with UHD 4.3 using the 10GigE interface. We technically have a USRP-2974, but I think it is irrelevant.
The 8192 data is being fragmented into 2 packets (Length 8042 and 266 on the wire using tcpdump).
You will be able to demonstrate very similar behavior by making the following change to host/examples/tx_samples_c.c (which sends a Length 8034 and 258 on the the wire).
// Set up buffer
#if 1
samps_per_buff = 2048; // (~8192 bytes)
#else
EXECUTE_OR_GOTO(
free_tx_streamer, uhd_tx_streamer_max_num_samps(tx_streamer, &samps_per_buff))
fprintf(stderr, “Buffer size in samples: %zu\n”, samps_per_buff);#endif
Please note, the interface has been configured to MTU 9000 along with the other changes mentioned in https://files.ettus.com/manual/page_usrp_x3x0_config.html
enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000inet 192.168.40.1 netmask 255.255.255.0 broadcast 192.168.40.255
When I run the program, you can see an indication of a USRP problem “Maximum frame size: 8000 bytes.”\
: ~/uhd-4.6.0.0/host/build/examples$ ./tx_samples_c\
Creating USRP with args “”...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.6.0.0-1-ga9f0b4c7
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Setting TX Rate: 200000000.000000...
Actual TX Rate: 200000000.000000...
Setting TX Gain: 0.000000 db...
Actual TX Gain: 0.000000...
Setting TX frequency: 2000.000000 MHz...
Actual TX frequency: 2000.000000 MHz...
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Press Ctrl+C to stop streaming...
**I have change parameters (**send_frame_size) to force the tx packet to 9000 in USRP stream, but I get this message which makes no sense. Also tried 8500, same complaint.
[WARNING] [X300] You requested a send frame size of (9000) but your NIC’s max frame size is (8000).Please verify your NIC’s MTU setting using ‘ip link’ or set the send_frame_size argument appropriately.UHD will use the auto-detected max frame size for this connection.
Interesting enough, at 8000 it does not complain. Of course we are fragmenting because 8192 bytes packet.
------
I have confirmed that ping with larger size packet doesn’t work.
cjohnson@USRP-1:~$ ping -s 8200 192.168.40.2
PING 192.168.40.2 (192.168.40.2) 8200(8228) bytes of data.
^C
--- 192.168.40.2 ping statistics ---
33 packets transmitted, 0 received, 100% packet loss, time 32758ms
On 17/06/2024 19:54, cjohnson@serranosystems.com wrote:
We have followed all instructions on the website and searched the
forums with no solutions. I have provided much evidence below that
USRP/UHD driver is not behaving properly in this case.
We are trying to send a 8192 byte packet to the x310 with the latest
UHD 4.6 (FPGA 39.2) which also didn’t seem to work with UHD 4.3 using
the 10GigE interface. We technically have a USRP-2974, but I think it
is irrelevant.
The 8192 data is being fragmented into 2 packets (Length 8042 and 266
on the wire using tcpdump).
You will be able to demonstrate very similar behavior by making the
following change to host/examples/tx_samples_c.c (which sends a Length
8034 and 258 on the the wire).
// Set up buffer
#if 1
samps_per_buff = 2048; // (~8192 bytes)
#else
EXECUTE_OR_GOTO(
free_tx_streamer, uhd_tx_streamer_max_num_samps(tx_streamer,
&samps_per_buff))
fprintf(stderr, “Buffer size in samples: %zu\n”, samps_per_buff);#endif
Please note, the interface has been configured to MTU 9000 along with
the other changes mentioned in
https://files.ettus.com/manual/page_usrp_x3x0_config.html
enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000inet
192.168.40.1 netmask 255.255.255.0 broadcast 192.168.40.255
When I run the program, you can see an indication of a USRP
problem “Maximum frame size: 8000 bytes.”
: ~/uhd-4.6.0.0/host/build/examples$ ./tx_samples_c
Creating USRP with args “”...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100;
UHD_4.6.0.0-1-ga9f0b4c7
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Setting TX Rate: 200000000.000000...
Actual TX Rate: 200000000.000000...
Setting TX Gain: 0.000000 db...
Actual TX Gain: 0.000000...
Setting TX frequency: 2000.000000 MHz...
Actual TX frequency: 2000.000000 MHz...
[WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
Press Ctrl+C to stop streaming...
I have change parameters (send_frame_size) to force the tx packet
to 9000 in USRP stream, but I get this message which makes no sense.
Also tried 8500, same complaint.
[WARNING] [X300] You requested a send frame size of (9000) but your
NIC’s max frame size is (8000).Please verify your NIC’s MTU setting
using ‘ip link’ or set the send_frame_size argument appropriately.UHD
will use the auto-detected max frame size for this connection.
Interesting enough, at 8000 it does not complain. Of course we are
fragmenting because 8192 bytes packet.
I have confirmed that ping with larger size packet doesn’t work.
cjohnson@USRP-1:~$ ping -s 8200 192.168.40.2
PING 192.168.40.2 (192.168.40.2) 8200(8228) bytes of data.
^C
--- 192.168.40.2 ping statistics ---
33 packets transmitted, 0 received, 100% packet loss, time 32758ms
USRP-users mailing list --usrp-users@lists.ettus.com
To unsubscribe send an email tousrp-users-leave@lists.ettus.com
I have run into a similar issue, years ago, with a 1Gig interface
blindly accepting a jumbo-frame MTU request, but
actually being unable to in reality honor it.
Now, that would be unusual with a 10GiG card--what card type do you
have? Are you running direct, or through a
switch?
It is the switch that is internal to you hardware on USRP-2974.
You can see the switch below (from your documentation). I’m sending from the SBC (Single board computer) shown in the diagram.
On 17/06/2024 20:29, cjohnson@serranosystems.com wrote:
It is the switch that is internal to you hardware on USRP-2974.
You can see the switch below (from your documentation). I’m sending
from the SBC (Single board computer) shown in the diagram.
Right. I'd forgotten about the 2974, which has historically been
supported by other parts of the organization, and I'm
not familiar with it.
You might try opening an official support ticket for it, assuming that
you have a support contract, or the device is
under the "initial bring-up" provisions of the NI product support
arrangements:
On 17/06/2024 20:29, cjohnson@serranosystems.com wrote:
It is the switch that is internal to you hardware on USRP-2974.
You can see the switch below (from your documentation). I’m sending
from the SBC (Single board computer) shown in the diagram.
https://kb.ettus.com/File:2974_blk_dia_hiLevel_v01.png
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Interestingly enough, I just tried a ping -s 8100 on my X310, and then a
ping -s 8105, and the 8105 failed. Looking at
previous log messages for my application it also claims 8000 byte
maximum frame size, so this may be a limitation
in the PHY/MAC implementation in the X310 FPGA.
Hi all,
8192 bytes is the maximum frame size supported by the FPGA, but that's
including all headers. So the actual RF data payload that you can send is
less than this, typically closer to 8000 bytes. I often set my SPP to 1996
out of habit since this is usually a pretty safe number with jumbo frames.
The driver attempts to determine the biggest MTU supported automatically.
Thanks,
Wade
On Mon, Jun 17, 2024, 7:47 PM Marcus D. Leech patchvonbraun@gmail.com
wrote:
On 17/06/2024 20:29, cjohnson@serranosystems.com wrote:
It is the switch that is internal to you hardware on USRP-2974.
You can see the switch below (from your documentation). I’m sending
from the SBC (Single board computer) shown in the diagram.
https://kb.ettus.com/File:2974_blk_dia_hiLevel_v01.png
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Interestingly enough, I just tried a ping -s 8100 on my X310, and then a
ping -s 8105, and the 8105 failed. Looking at
previous log messages for my application it also claims 8000 byte
maximum frame size, so this may be a limitation
in the PHY/MAC implementation in the X310 FPGA.
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-leave@lists.ettus.com
Actually, you can probably expect that the ping -s with jumbo frames to fail, since the FPGA stack is very limited and most likely hasn’t implemented ICMP for jumbo frames.
However, the limitation of 8000 seems to be an artificial limit of the UHD code, as it I have been able to demonstrate it can send > 8000, but not at my requested value due to the way the driver is written. You can see in the packet trace (if you do a tcpdump) is sends larger than 8000.
See “static const size_t DATA_FRAME_MAX_SIZE = 8000; // CHDR packet size in bytes” in x300_defaults.hpp and what appears to be some bad logic in determine_max_frame_size() (setting packet size to min(tx,rx)).
I hacked the code and have seen the Tx >= 8136, but don’t understand it enough yet.
You might also need to reach out to the FPGA team to actually see what the limitation is or is not. Documentation says to set MTU to 9000, but UHD driver has some issues in implementation.
I hacked the code to set the DATA_FRAME_MAX_SIZE = 8500, half-way between 8000 and MTU 9000.
Then I made sure to set the send_frame_size=recv_frame_size=8700, which is equal, and over the MAX_SIZE.It still picks 8144. You can see below.
I added a bunch of debug of the intermediate variables.
$~/uhd-4.6.0.0/host/build/examples$ ./tx_samples_c -a "type=x300,addr=192.168.40.2,send_frame_size=8700,recv_frame_size=8700"
Creating USRP with args "type=x300,addr=192.168.40.2,send_frame_size=8700,recv_frame_size=8700"...
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.6.0.0-1-ga9f0b4c7
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] user_frame_size.send_frame_size: 8700
[INFO] [X300] x300::DATA_FRAME_MAX_SIZE: 8500
[INFO] [X300] std::min(user_frame_size.send_frame_size, x300::DATA_FRAME_MAX_SIZE): 8500
[INFO] [X300] max_send_frame_size: 8500
[INFO] [X300] min_recv_frame_size: 8144
[INFO] [X300] min_send_frame_size: 8144
[INFO] [X300] Maximum frame size: 8144 bytes.
[WARNING] [X300] You requested a receive frame size of (8700) but your NIC's max frame size is (8144).Please verify your NIC's MTU setting using 'ip link' or set the recv_frame_size argument appropriately.UHD will use the auto-detected max frame size for this connection.
[WARNING] [X300] You requested a send frame size of (8700) but your NIC's max frame size is (8144).Please verify your NIC's MTU setting using 'ip link' or set the send_frame_size argument appropriately.UHD will use the auto-detected max frame size for this connection.
[WARNING] [X300] For the 192.168.40.2 connection, UHD recommends a send frame size of at least 8500 for best
performance, but your configuration will only allow 8144.This may negatively impact your maximum achievable sample rate.
Check the MTU on the interface and/or the send_frame_size argument.
[WARNING] [X300] For the 192.168.40.2 connection, UHD recommends a receive frame size of at least 8500 for best
performance, but your configuration will only allow 8144.This may negatively impact your maximum achievable sample rate.
Check the MTU on the interface and/or the recv_frame_size argument.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a
[INFO] [X300] Radio 1x clock: 200 MHz
Setting TX Rate: 200000000.000000...
Actual TX Rate: 200000000.000000...
Setting TX Gain: 0.000000 db...
On 17/06/2024 21:16, cjohnson@serranosystems.com wrote:
I hacked the code to set the DATA_FRAME_MAX_SIZE = 8500, half-way
between 8000 and MTU 9000.
Then I made sure to set the send_frame_size=recv_frame_size=8700,
which is equal, and over the MAX_SIZE.It still picks 8144. You can see
below.
I added a bunch of debug of the intermediate variables.
$~/uhd-4.6.0.0/host/build/examples$ ./tx_samples_c -a
"type=x300,addr=192.168.40.2,send_frame_size=8700,recv_frame_size=8700"
|Creating USRP with args
"type=x300,addr=192.168.40.2,send_frame_size=8700,recv_frame_size=8700"...|
|[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100;
UHD_4.6.0.0-1-ga9f0b4c7|
|[INFO] [X300] X300 initialization sequence...|
|[INFO] [X300] user_frame_size.send_frame_size: 8700|
|[INFO] [X300] x300::DATA_FRAME_MAX_SIZE: 8500|
|[INFO] [X300] std::min(user_frame_size.send_frame_size,
x300::DATA_FRAME_MAX_SIZE): 8500|
|[INFO] [X300] max_send_frame_size: 8500|
|[INFO] [X300] min_recv_frame_size: 8144|
|[INFO] [X300] min_send_frame_size: 8144|
|[INFO] [X300] Maximum frame size: 8144 bytes.|
|[WARNING] [X300] You requested a receive frame size of (8700) but
your NIC's max frame size is (8144).Please verify your NIC's MTU
setting using 'ip link' or set the recv_frame_size argument
appropriately.UHD will use the auto-detected max frame size for this
connection.|
|[WARNING] [X300] You requested a send frame size of (8700) but your
NIC's max frame size is (8144).Please verify your NIC's MTU setting
using 'ip link' or set the send_frame_size argument appropriately.UHD
will use the auto-detected max frame size for this connection.|
|[WARNING] [X300] For the 192.168.40.2 connection, UHD recommends a
send frame size of at least 8500 for best|
|performance, but your configuration will only allow 8144.This may
negatively impact your maximum achievable sample rate.|
|Check the MTU on the interface and/or the send_frame_size argument.|
|[WARNING] [X300] For the 192.168.40.2 connection, UHD recommends a
receive frame size of at least 8500 for best|
|performance, but your configuration will only allow 8144.This may
negatively impact your maximum achievable sample rate.|
|Check the MTU on the interface and/or the recv_frame_size argument.|
|[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929a|
|[INFO] [X300] Radio 1x clock: 200 MHz|
|Setting TX Rate: 200000000.000000...|
|Actual TX Rate: 200000000.000000...|
|Setting TX Gain: 0.000000 db...|
USRP-users mailing list --usrp-users@lists.ettus.com
To unsubscribe send an email tousrp-users-leave@lists.ettus.com
Well, as Wade pointed out, the MTU limit in the FPGA is 8192. I don't
know whether the UHD 'stack' is using PMTUD, or
some other technique. With PMTUD (Path MTU Discovery), even if your
NIC MTU is 9000, with PMTUD, things will
be configured for the largest MTU that the path can actually
support. In this case, with the FPGA supporting a max
of 8192, the numbers quoted make sense...
Thanks for all the great info Marcus & Wade