[USRP-users] about 10 Gig ethernet configuration

Marcus Müller marcus.mueller at ettus.com
Sun Oct 25 05:22:40 EDT 2015


Hi Sanjoy,

Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth2 9000 0 1084 0 0 0 1153 0 0 0 BMRU


You get 1153 packets that were dropped due to not being picked up on RX
side.
I was first not quite sure who really defines this value, whether it's
the network hardware or the linux kernel itself, but after reading
netstat code and linux kernel code, it seems that netstat kind of
mislabels what is called "rx_fifo_errors" in the kernel as "RX-OVR".
That didn't make things more intuitive researching (because of course
there's also a overrun field in the stats struct in the netdev in the
kernel... ).
The good news: rx_fifo_errors is not something the Intel ixgbe driver
modifies -- which means it doesn't seem to be a hardware counter on
packets that weren't fetched from the card.

So, could you share what

sysctl net.core.rmem_max net.core.wmem_max   


says? It's the maximum memory that your linux might allocate for receive
and transmit buffers on the card.

sudo sysctl -w net.core.rmem_max $(( 512 * 2**20 )) 
sudo sysctl -w net.core.wmem_max $(( 512 * 2**20 )) 

should set the sizes to 512MB (cave: only til next reboot). Could you
try with that?

Best regards,
Marcus

On 24.10.2015 18:28, Sanjoy Basak wrote:
> Hi Marcus,
> Thanks a lot for the quick reply. The kernel version is 3.19.0-31-generic
>
> I just checked again at 30s, 25MSps. I got 4 D and 5 D for 2
> consecutive initiations.
>
> *@30sec*
>
> Setting RX Rate: 25.000000 Msps...
>
> Actual RX Rate: 25.000000 Msps...
>
> Setting RX Freq: 2000.000000 MHz...
>
> Actual RX Freq: 2000.000000 MHz...
>
> Waiting for "lo_locked": ++++++++++ locked.
>
> Press Ctrl + C to stop streaming...
>
> DGot an overflow indication. Please consider the following:
>
> Your write medium must sustain a rate of 100.000000MB/s.
>
> Dropped samples will not be written to the file.
>
> Please modify this example for your purposes.
>
> This message will not appear again.
>
> DDDD
>
> Done!
>
>
> *@240 sec*
>
> Setting RX Rate: 25.000000 Msps...
>
> Actual RX Rate: 25.000000 Msps...
>
> Setting RX Freq: 2000.000000 MHz...
>
> Actual RX Freq: 2000.000000 MHz...
>
> Waiting for "lo_locked": ++++++++++ locked.
>
> Press Ctrl + C to stop streaming...
>
> DGot an overflow indication. Please consider the following:
>
> Your write medium must sustain a rate of 100.000000MB/s.
>
> Dropped samples will not be written to the file.
>
> Please modify this example for your purposes.
>
> This message will not appear again.
>
> DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
>
> Done!
>
>
> This is the netstat right after restarting the computer
>
> netstat -i eth2
>
> Kernel Interface table
>
> Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
>
> eth0 1500 0 9 0 0 0 26 0 0 0 BMRU
>
> eth1 1500 0 0 0 0 0 0 0 0 0 BMU
>
> eth2 9000 0 1084 0 0 0 1153 0 0 0 BMRU
>
> lo 65536 0 177 0 0 0 177 0 0 0 L
>
>
> And this is netstat after
>
> ./rx_samples_to_file --duration 240 --rate 200e6 --freq=2e9 --file
> /dev/null
>
> which gave a lot of D (did not count how many but a lot)
>
> Kernel Interface table
>
> Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
>
> eth0 1500 0 94 0 0 0 30 0 0 0 BMRU
>
> eth1 1500 0 0 0 0 0 0 0 0 0 BMU
>
> eth2 9000 0 24190524 936 0 0 355025 0 0 0 BMRU
>
> lo 65536 0 183 0 0 0 183 0 0 0 LRU
>
>
> Best regards
>
> Sanjoy
>
>
>
>
> On Sat, Oct 24, 2015 at 5:31 PM, Marcus Müller
> <marcus.mueller at ettus.com <mailto:marcus.mueller at ettus.com>> wrote:
>
>     Hi Sanjoy,
>
>     Two observations:
>     Your routing table looks fine.
>
>     Then, these rx_samples... results are interesting. Basically, 30s
>     of 200MS/s are 8 times the amount of packets as 30s at 25MS/s;
>     however the former produces 53 sequence errors, and the latter
>     only 3. That's off by a factor of 2. To verify, could you try 240s
>     at 25MS/s?
>     If this holds, the sequence errors would seem to be load-related.
>     Kernel version currently used (uname -r)?
>     Let's analyze where things go missing. Could you compare the
>     output of "netstat -i eth2" before and after a rx_samples_to_file
>     run that produced lots of D?
>
>     Best regards,
>     Marcus
>
>
>     Am 24. Oktober 2015 15:58:28 MESZ, schrieb Sanjoy Basak
>     <sanjoybasak14 at gmail.com <mailto:sanjoybasak14 at gmail.com>>:
>
>         Hi Micheal,
>         I made the cpu governor to performance. This is always
>         performance now, does not change. We did not buy the cable
>         from ettus. 
>         We bought DeLOCK Twinaxial-Kabel SFP+ M SFP+ M 3,0m SFF-8431
>         86222 from a store here.
>
>         Hi Rob,
>         Thanks for the command. I am not really sure which cable you
>         used. But I am still having drop of packets. I don't have any
>         other cable right now to test. 
>
>
>         Hi Marcus,
>         I tried your suggestions. I put the output here.
>
>         *ip route *
>
>         192.168.40.0/24 <http://192.168.40.0/24> dev eth2 proto kernel
>         scope link src 192.168.40.1 metric 1
>
>
>         I don't get any drop at this.
>
>         rx_samples_to_file --rate 200e6 --file /dev/null --nsamps $((
>         2 * 1000  * 1000 ))
>
>         Setting RX Rate: 200.000000 Msps...
>
>         Actual RX Rate: 200.000000 Msps...
>
>         Setting RX Freq: 0.000000 MHz...
>
>         Actual RX Freq: 340.000000 MHz...
>
>         Waiting for "lo_locked": ++++++++++ locked.
>
>         Done!
>
>
>         However with
>
>         ./rx_samples_to_file --duration 30 --rate 200e6 --freq=2e9
>         --file /dev/null
>
>         Setting RX Rate: 200.000000 Msps...
>
>         Actual RX Rate: 200.000000 Msps...
>
>         Setting RX Freq: 2000.000000 MHz...
>
>         Actual RX Freq: 2000.000000 MHz...
>
>         Waiting for "lo_locked": ++++++++++ locked.
>
>         Press Ctrl + C to stop streaming...
>
>         DGot an overflow indication. Please consider the following:
>
>         Your write medium must sustain a rate of 800.000000MB/s.
>
>         Dropped samples will not be written to the file.
>
>         Please modify this example for your purposes.
>
>         This message will not appear again.
>
>         DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
>
>         Done!
>
>
>         Even at 25 MSps I got drop of packet
>
>         ./rx_samples_to_file --duration 30 --rate 25e6 --freq=2e9
>         --file /dev/null
>
>         Setting RX Rate: 25.000000 Msps...
>
>         Actual RX Rate: 25.000000 Msps...
>
>         Setting RX Freq: 2000.000000 MHz...
>
>         Actual RX Freq: 2000.000000 MHz...
>
>         Waiting for "lo_locked": ++++++++++ locked.
>
>         Press Ctrl + C to stop streaming...
>
>         DGot an overflow indication. Please consider the following:
>
>           Your write medium must sustain a rate of 100.000000MB/s.
>
>           Dropped samples will not be written to the file.
>
>           Please modify this example for your purposes.
>
>           This message will not appear again.
>
>         DD
>
>         Done!
>
>
>         But with  1 Gig interface with 25 MSps I am not getting any drop
>
>         ./rx_samples_to_file --duration 30 --rate 25e6 --freq=2e9
>         --file /dev/null
>
>         Setting RX Rate: 25.000000 Msps...
>
>         Actual RX Rate: 25.000000 Msps...
>
>         Setting RX Freq: 2000.000000 MHz...
>
>         Actual RX Freq: 2000.000000 MHz...
>
>         Waiting for "lo_locked": ++++++++++ locked.
>
>         Press Ctrl + C to stop streaming...
>
>         Done!
>
>
>         I hope raid is not making an issue here, as I don't see any
>         indication when I tried with 1 Gig interface.
>
>         Please let me know from  seeing the output what you think
>         making the trouble. 
>
>
>         Best regards
>         Sanjoy
>
>
>         On Sat, Oct 24, 2015 at 1:48 AM, Michael West
>         <michael.west at ettus.com <mailto:michael.west at ettus.com>> wrote:
>
>             I was corrected regarding the 10 GbE copper cables.  The
>             long copper cables were failing our tests due to a bug in
>             the IP used for the 10 GbE in the FPGA image that has
>             since been resolved.  We are currently using cables up to
>             3m with no problems.  We have found some 5m cables work
>             and some don't depending on the quality of the cable.  The
>             10 GbE cables and NICs sold on the Ettus website all work
>             well.
>
>             Regards,
>             Michael
>
>
>
>
>             On Fri, Oct 23, 2015 at 3:47 PM, Marcus Müller
>             <marcus.mueller at ettus.com
>             <mailto:marcus.mueller at ettus.com>> wrote:
>
>                 So to understand this
>
>                 TX much more stable than RX, and RX showing not a
>                 single "O" (a proper overflow) but a lot of "D" packet
>                 errors (which are, in fact, either very serious packet
>                 losses, or package reordering).
>                 It's probably the cable, or your network stack might
>                 be up to no good.
>
>                 Could you share the output of
>
>                 ip route
>
>                 Also, it's possible that some default firewall rule
>                 makes your CPU break a sweat; you could try to bypass
>                 firewall for the eth2 device
>
>                 sudo iptables -A INPUT -i eth2 -j ACCEPT
>
>                 (this is *not* permanent).
>                 If that doesn't help, maybe inspecting the packets
>                 captured with "wireshark" might help. For example, set
>                 up wireshark to listen to eth2, power up the USRP; you
>                 should see a few packets being exchanged on the interface.
>                 Then run
>
>                 rx_samples_to_file --rate 200e6 --file /dev/null
>                 --nsamps $(( 2 * 1000  * 1000 ))
>
>                 then stop the capture. Did you see "D"s? If yes, I'd
>                 like to have a look at the captured data; please save
>                 it. We'll figure out a way to exchange it.
>
>                 Best regards,
>                 Marcus
>                 On 23.10.2015 21:21, Sanjoy Basak wrote:
>>                 Hi Michael, Neel and Marcus,
>>
>>                 Thanks a lot for the suggestions. I tried each of those. 
>>                 I updated the kernel to 3.19.0-31-generic. 
>>
>>                 CPU governors, I set to performance as instruction
>>                 given in this link. 
>>                 http://files.ettus.com/manual/page_usrp_x3x0_config.html#x3x0cfg_hostpc_pwr
>>                 Now, after setting it to performance and restarting,
>>                 if I check with cpu-freq, I find all governers are
>>                 set at performance. But after a while(5/10 mins
>>                 later) if I check again cpu-freq info, I see all
>>                 governers are set to powersave. 
>>                 Could you please tell me why and how to configure it
>>                 properly?
>>
>>                 I tried with benchmark_rate to check the performance.
>>                 With tx-rate till 200 MSps I do not get any
>>                 overflow/underflow. However with rx-rate 25/50/100
>>                 MSps I get drop of packets. I made a direct
>>                 connection from computer to the X310. No switch is used.
>>
>>                 On the contrary with shorter(copper) cable at 1Gig
>>                 interface I do not get any drop of packets at 25MSps
>>                 (on both tx_rate and rx_rate). 
>>                 Is 3m copper cable really bad? 
>>
>>                 We don't have fibre cable right now. We already
>>                 ordered. I hope we will get it next week and check
>>                 how it is. We don't have any SFP+ 10 Gig interface.
>>                 So can't really test 10 gig interface with short 10
>>                 gig copper cable.
>>
>>                 These are the benchmark_rate results for 10 Gig interface
>>
>>
>>                 Testing receive rate 25.000000 Msps on 1 channels
>>                 DDDDDDDD
>>                 Benchmark rate summary:
>>                   Num received samples:    249845308
>>                   Num dropped samples:     15968
>>                   Num overflows detected:  0
>>                   Num transmitted samples: 0
>>                   Num sequence errors:     0
>>                   Num underflows detected: 0
>>
>>
>>                 Testing receive rate 100.000000 Msps on 1 channels 
>>                 DDDDDDDDDDDDDDDDDDDDD 
>>                 Benchmark rate summary: 
>>                   Num received samples:    999934124 
>>                   Num dropped samples:     41916 
>>                   Num overflows detected:  0 
>>                   Num transmitted samples: 0 
>>                   Num sequence errors:     0 
>>                   Num underflows detected: 0
>>
>>                  I also used this command
>>                 sudo ethtool -C eth2 rx-usecs 16 rx-frames 20
>>
>>                 Please let me know what you think occurring the
>>                 problem and whether replacing copper cable with short
>>                 fibre cable solves the problem.
>>
>>                 Best regards
>>                 Sanjoy
>>
>>
>>                 On Wed, Oct 21, 2015 at 12:27 PM, Marcus Müller
>>                 <usrp-users at lists.ettus.com
>>                 <mailto:usrp-users at lists.ettus.com>> wrote:
>>
>>                     Hi Sanjoy,
>>
>>                     furthermore, and I found that to be crucial on my
>>                     system, you should make sure that your 10GE card
>>                     doesn't generate an interrupt for every packet it
>>                     receives (your application will pull these
>>                     packets as fast as possible, anyway).
>>                     On linux, you'd do
>>
>>                     sudo ethtool -c {name of ethernet interface, e.g.
>>                     eth1}
>>
>>                     to see the coalescing options available with your
>>                     device and kernel version,
>>                     and modify a setting using
>>
>>                     sudo ethtool -C {name of ethernet interface, e.g.
>>                     eth1} {name of setting} {new value of setting}
>>
>>                     For example, I set "rx_usecs" (microseconds to
>>                     wait between triggering an interrupt) to 16, and
>>                     "rx_frames" to 20 or so -- your milage might
>>                     vary, depending on your application, OS and also
>>                     a few hardware factors.
>>
>>                     Best regards,
>>                     Marcus
>>
>>
>>                     On 19.10.2015 21:56, Michael West via USRP-users
>>                     wrote:
>>>                     Also, check the CPU governor and make sure it is
>>>                     set to "performance".
>>>
>>>                     Your output indicates frame errors on the
>>>                     interface, which could be due to the cable.  If
>>>                     possible, try using a shorter copper cable or
>>>                     switch to a fibre cable.  With 10 GbE over
>>>                     copper, the shorter the better.  If you need the
>>>                     length, you should be using fibre.
>>>
>>>                     If you still have issues, please provide a
>>>                     little more information:  Are you connected
>>>                     directly with the X310 or through a switch?  How
>>>                     are you testing the performance?  Are you using
>>>                     your own application or benchmark_rate?  If your
>>>                     own application, what is it doing with the
>>>                     received data (processing it with the same
>>>                     thread or different thread, saving to disk, etc...)?
>>>
>>>                     Regards,
>>>                     Michael
>>>
>>>                     On Mon, Oct 19, 2015 at 12:51 PM, Neel Pandeya
>>>                     via USRP-users <usrp-users at lists.ettus.com
>>>                     <mailto:usrp-users at lists.ettus.com>> wrote:
>>>
>>>                         Hello Sanjoy:
>>>
>>>                         That Intel X520 10GbE card with Ubuntu
>>>                         14.04.3 should be working well for you.
>>>
>>>                         Could you try upgrading to kernel 3.16 or
>>>                         3.19, and let us know your results?
>>>
>>>                         To upgrade to kernel 3.16, run "sudo apt-get
>>>                         install linux-generic-lts-utopic".
>>>
>>>                         To upgrade to kernel 3.19, run "sudo apt-get
>>>                         install linux-generic-lts-vivid".
>>>
>>>                         After the upgrade, be sure to reboot the system.
>>>
>>>                         Also, check that you have the CPU governors
>>>                         set to "performance", and that you have the
>>>                         highest clock rate set.
>>>
>>>                         --Neel
>>>
>>>
>>>
>>>                         On 19 October 2015 at 12:03, Sanjoy Basak
>>>                         via USRP-users <usrp-users at lists.ettus.com
>>>                         <mailto:usrp-users at lists.ettus.com>> wrote:
>>>
>>>                             Hi Experts,
>>>                             We recently bought a 10 Gig ethernet
>>>                             card (X510-DA2) and Twinaxial-Kabel SFP+
>>>                             M SFP+ M 3,0m. I installed it in our pc
>>>                             (OS: Ubuntu Mate 14.04.3 LTS) and can
>>>                             communicate with X310/SBX-120 properly.
>>>                             However, the performance we are getting
>>>                             is quite poor. I am getting dropped
>>>                             packets (D O) mostly at 25,50 MSps and
>>>                             goes quite crazy at 100 MSps and
>>>                             sometimes even at lower sample rates. 
>>>                             Previously I tested with 1 Gig ethernet,
>>>                             at 25 MSps I could stream without any
>>>                             drop/late packet or underrun. 
>>>                             The CPU uses or network uses is also not
>>>                             reaching 100%. 
>>>
>>>                             Could you please tell me what to
>>>                             check/correct so I can stream at 100
>>>                             MSps without any error?
>>>
>>>                             I am putting the ifconfig and ethtool
>>>                             results and the kernel
>>>                             version(3.16.0-30-generic)
>>>
>>>                             eth2      Link encap:Ethernet  HWaddr
>>>                             90:e2:ba:a6:a9:dd  
>>>                                       inet6 addr:
>>>                             fe80::92e2:baff:fea6:a9dd/64 Scope:Link
>>>                                       UP BROADCAST MULTICAST
>>>                              MTU:9000  Metric:1
>>>                                       RX packets:4530726 errors:146
>>>                             dropped:0 overruns:0 frame:146
>>>                                       TX packets:6811057 errors:0
>>>                             dropped:0 overruns:0 carrier:0
>>>                                       collisions:0 txqueuelen:1000 
>>>                                       RX bytes:26331306938 (26.3 GB)
>>>                              TX bytes:35856233935 (35.8 GB)
>>>
>>>                             Settings for eth2:
>>>                             Supported ports: [ FIBRE ]
>>>                             Supported link modes:   10000baseT/Full 
>>>                             Supported pause frame use: No
>>>                             Supports auto-negotiation: No
>>>                             Advertised link modes:  10000baseT/Full 
>>>                             Advertised pause frame use: No
>>>                             Advertised auto-negotiation: No
>>>                             Speed: 10000Mb/s
>>>                             Duplex: Full
>>>                             Port: Other
>>>                             PHYAD: 0
>>>                             Transceiver: external
>>>                             Auto-negotiation: off
>>>                             Cannot get wake-on-lan settings:
>>>                             Operation not permitted
>>>                             Current message level: 0x00000007 (7)
>>>                                   drv probe link
>>>                             Link detected: yes
>>>
>>>                             Our computer config is:
>>>                             Processor: 12*Intel(R) Xeon (R) CPU
>>>                             E5-2620 v3 @2.40 GHz
>>>                             RAM: 65871 MB
>>>                             SSD raid 5
>>>
>>>
>>>                             Best regards
>>>                             Sanjoy
>>>
>>>                             _______________________________________________
>>>                             USRP-users mailing list
>>>                             USRP-users at lists.ettus.com
>>>                             <mailto:USRP-users at lists.ettus.com>
>>>                             http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>
>>>
>>>
>>>                         _______________________________________________
>>>                         USRP-users mailing list
>>>                         USRP-users at lists.ettus.com
>>>                         <mailto:USRP-users at lists.ettus.com>
>>>                         http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>
>>>
>>>
>>>
>>>                     _______________________________________________
>>>                     USRP-users mailing list
>>>                     USRP-users at lists.ettus.com
>>>                     <mailto:USRP-users at lists.ettus.com>
>>>                     http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>>
>>                     _______________________________________________
>>                     USRP-users mailing list
>>                     USRP-users at lists.ettus.com
>>                     <mailto:USRP-users at lists.ettus.com>
>>                     http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>>
>
>
>
>
>     -- 
>     Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/attachments/20151025/290b2261/attachment-0002.html>


More information about the USRP-users mailing list