[USRP-users] about 10 Gig ethernet configuration

Marcus Müller marcus.mueller at ettus.com
Fri Oct 23 18:47:01 EDT 2015


So to understand this

TX much more stable than RX, and RX showing not a single "O" (a proper
overflow) but a lot of "D" packet errors (which are, in fact, either
very serious packet losses, or package reordering).
It's probably the cable, or your network stack might be up to no good.

Could you share the output of

ip route

Also, it's possible that some default firewall rule makes your CPU break
a sweat; you could try to bypass firewall for the eth2 device

sudo iptables -A INPUT -i eth2 -j ACCEPT

(this is *not* permanent).
If that doesn't help, maybe inspecting the packets captured with
"wireshark" might help. For example, set up wireshark to listen to eth2,
power up the USRP; you should see a few packets being exchanged on the
interface.
Then run

rx_samples_to_file --rate 200e6 --file /dev/null --nsamps $(( 2 * 1000 
* 1000 ))

then stop the capture. Did you see "D"s? If yes, I'd like to have a look
at the captured data; please save it. We'll figure out a way to exchange it.

Best regards,
Marcus
On 23.10.2015 21:21, Sanjoy Basak wrote:
> Hi Michael, Neel and Marcus,
>
> Thanks a lot for the suggestions. I tried each of those. 
> I updated the kernel to 3.19.0-31-generic. 
>
> CPU governors, I set to performance as instruction given in this link. 
> http://files.ettus.com/manual/page_usrp_x3x0_config.html#x3x0cfg_hostpc_pwr
> Now, after setting it to performance and restarting, if I check with
> cpu-freq, I find all governers are set at performance. But after a
> while(5/10 mins later) if I check again cpu-freq info, I see all
> governers are set to powersave. 
> Could you please tell me why and how to configure it properly?
>
> I tried with benchmark_rate to check the performance. With tx-rate
> till 200 MSps I do not get any overflow/underflow. However with
> rx-rate 25/50/100 MSps I get drop of packets. I made a direct
> connection from computer to the X310. No switch is used.
>
> On the contrary with shorter(copper) cable at 1Gig interface I do not
> get any drop of packets at 25MSps (on both tx_rate and rx_rate). 
> Is 3m copper cable really bad? 
>
> We don't have fibre cable right now. We already ordered. I hope we
> will get it next week and check how it is. We don't have any SFP+ 10
> Gig interface. So can't really test 10 gig interface with short 10 gig
> copper cable.
>
> These are the benchmark_rate results for 10 Gig interface
>
>
> Testing receive rate 25.000000 Msps on 1 channels
> DDDDDDDD
> Benchmark rate summary:
>   Num received samples:    249845308
>   Num dropped samples:     15968
>   Num overflows detected:  0
>   Num transmitted samples: 0
>   Num sequence errors:     0
>   Num underflows detected: 0
>
>
> Testing receive rate 100.000000 Msps on 1 channels 
> DDDDDDDDDDDDDDDDDDDDD 
> Benchmark rate summary: 
>   Num received samples:    999934124 
>   Num dropped samples:     41916 
>   Num overflows detected:  0 
>   Num transmitted samples: 0 
>   Num sequence errors:     0 
>   Num underflows detected: 0
>
>  I also used this command
> sudo ethtool -C eth2 rx-usecs 16 rx-frames 20
>
> Please let me know what you think occurring the problem and
> whether replacing copper cable with short fibre cable solves the problem.
>
> Best regards
> Sanjoy
>
>
> On Wed, Oct 21, 2015 at 12:27 PM, Marcus Müller
> <usrp-users at lists.ettus.com <mailto:usrp-users at lists.ettus.com>> wrote:
>
>     Hi Sanjoy,
>
>     furthermore, and I found that to be crucial on my system, you
>     should make sure that your 10GE card doesn't generate an interrupt
>     for every packet it receives (your application will pull these
>     packets as fast as possible, anyway).
>     On linux, you'd do
>
>     sudo ethtool -c {name of ethernet interface, e.g. eth1}
>
>     to see the coalescing options available with your device and
>     kernel version,
>     and modify a setting using
>
>     sudo ethtool -C {name of ethernet interface, e.g. eth1} {name of
>     setting} {new value of setting}
>
>     For example, I set "rx_usecs" (microseconds to wait between
>     triggering an interrupt) to 16, and "rx_frames" to 20 or so --
>     your milage might vary, depending on your application, OS and also
>     a few hardware factors.
>
>     Best regards,
>     Marcus
>
>
>     On 19.10.2015 21:56, Michael West via USRP-users wrote:
>>     Also, check the CPU governor and make sure it is set to
>>     "performance".
>>
>>     Your output indicates frame errors on the interface, which could
>>     be due to the cable.  If possible, try using a shorter copper
>>     cable or switch to a fibre cable.  With 10 GbE over copper, the
>>     shorter the better.  If you need the length, you should be using
>>     fibre.
>>
>>     If you still have issues, please provide a little more
>>     information:  Are you connected directly with the X310 or through
>>     a switch?  How are you testing the performance?  Are you using
>>     your own application or benchmark_rate?  If your own application,
>>     what is it doing with the received data (processing it with the
>>     same thread or different thread, saving to disk, etc...)?
>>
>>     Regards,
>>     Michael
>>
>>     On Mon, Oct 19, 2015 at 12:51 PM, Neel Pandeya via USRP-users
>>     <usrp-users at lists.ettus.com <mailto:usrp-users at lists.ettus.com>>
>>     wrote:
>>
>>         Hello Sanjoy:
>>
>>         That Intel X520 10GbE card with Ubuntu 14.04.3 should be
>>         working well for you.
>>
>>         Could you try upgrading to kernel 3.16 or 3.19, and let us
>>         know your results?
>>
>>         To upgrade to kernel 3.16, run "sudo apt-get install
>>         linux-generic-lts-utopic".
>>
>>         To upgrade to kernel 3.19, run "sudo apt-get install
>>         linux-generic-lts-vivid".
>>
>>         After the upgrade, be sure to reboot the system.
>>
>>         Also, check that you have the CPU governors set to
>>         "performance", and that you have the highest clock rate set.
>>
>>         --Neel
>>
>>
>>
>>         On 19 October 2015 at 12:03, Sanjoy Basak via USRP-users
>>         <usrp-users at lists.ettus.com
>>         <mailto:usrp-users at lists.ettus.com>> wrote:
>>
>>             Hi Experts,
>>             We recently bought a 10 Gig ethernet card (X510-DA2)
>>             and Twinaxial-Kabel SFP+ M SFP+ M 3,0m. I installed it in
>>             our pc (OS: Ubuntu Mate 14.04.3 LTS) and can communicate
>>             with X310/SBX-120 properly. However, the performance we
>>             are getting is quite poor. I am getting dropped packets
>>             (D O) mostly at 25,50 MSps and goes quite crazy at 100
>>             MSps and sometimes even at lower sample rates. 
>>             Previously I tested with 1 Gig ethernet, at 25 MSps I
>>             could stream without any drop/late packet or underrun. 
>>             The CPU uses or network uses is also not reaching 100%. 
>>
>>             Could you please tell me what to check/correct so I can
>>             stream at 100 MSps without any error?
>>
>>             I am putting the ifconfig and ethtool results and the
>>             kernel version(3.16.0-30-generic)
>>
>>             eth2      Link encap:Ethernet  HWaddr 90:e2:ba:a6:a9:dd  
>>                       inet6 addr: fe80::92e2:baff:fea6:a9dd/64 Scope:Link
>>                       UP BROADCAST MULTICAST  MTU:9000  Metric:1
>>                       RX packets:4530726 errors:146 dropped:0
>>             overruns:0 frame:146
>>                       TX packets:6811057 errors:0 dropped:0
>>             overruns:0 carrier:0
>>                       collisions:0 txqueuelen:1000 
>>                       RX bytes:26331306938 (26.3 GB)  TX
>>             bytes:35856233935 (35.8 GB)
>>
>>             Settings for eth2:
>>             Supported ports: [ FIBRE ]
>>             Supported link modes:   10000baseT/Full 
>>             Supported pause frame use: No
>>             Supports auto-negotiation: No
>>             Advertised link modes:  10000baseT/Full 
>>             Advertised pause frame use: No
>>             Advertised auto-negotiation: No
>>             Speed: 10000Mb/s
>>             Duplex: Full
>>             Port: Other
>>             PHYAD: 0
>>             Transceiver: external
>>             Auto-negotiation: off
>>             Cannot get wake-on-lan settings: Operation not permitted
>>             Current message level: 0x00000007 (7)
>>                   drv probe link
>>             Link detected: yes
>>
>>             Our computer config is:
>>             Processor: 12*Intel(R) Xeon (R) CPU E5-2620 v3 @2.40 GHz
>>             RAM: 65871 MB
>>             SSD raid 5
>>
>>
>>             Best regards
>>             Sanjoy
>>
>>             _______________________________________________
>>             USRP-users mailing list
>>             USRP-users at lists.ettus.com
>>             <mailto:USRP-users at lists.ettus.com>
>>             http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>>
>>
>>         _______________________________________________
>>         USRP-users mailing list
>>         USRP-users at lists.ettus.com <mailto:USRP-users at lists.ettus.com>
>>         http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>>
>>
>>
>>     _______________________________________________
>>     USRP-users mailing list
>>     USRP-users at lists.ettus.com <mailto:USRP-users at lists.ettus.com>
>>     http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
>
>     _______________________________________________
>     USRP-users mailing list
>     USRP-users at lists.ettus.com <mailto:USRP-users at lists.ettus.com>
>     http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/attachments/20151024/1089e044/attachment-0002.html>


More information about the USRP-users mailing list