[USRP-users] about 10 Gig ethernet configuration

Marcus Müller marcus.mueller at ettus.com
Sat Oct 24 11:31:49 EDT 2015


Hi Sanjoy,

Two observations:
Your routing table looks fine.

Then, these rx_samples... results are interesting. Basically, 30s of 200MS/s are 8 times the amount of packets as 30s at 25MS/s; however the former produces 53 sequence errors, and the latter only 3. That's off by a factor of 2. To verify, could you try 240s at 25MS/s? 
If this holds, the sequence errors would seem to be load-related. Kernel version currently used (uname -r)?
Let's analyze where things go missing. Could you compare the output of "netstat -i eth2" before and after a rx_samples_to_file run that produced lots of D?

Best regards,
Marcus

Am 24. Oktober 2015 15:58:28 MESZ, schrieb Sanjoy Basak <sanjoybasak14 at gmail.com>:
>Hi Micheal,
>I made the cpu governor to performance. This is always performance now,
>does not change. We did not buy the cable from ettus.
>We bought DeLOCK Twinaxial-Kabel SFP+ M SFP+ M 3,0m SFF-8431 86222 from
>a
>store here.
>
>Hi Rob,
>Thanks for the command. I am not really sure which cable you used. But
>I am
>still having drop of packets. I don't have any other cable right now to
>test.
>
>
>Hi Marcus,
>I tried your suggestions. I put the output here.
>
>*ip route *
>
>192.168.40.0/24 dev eth2 proto kernel scope link src 192.168.40.1
>metric 1
>
>
>I don't get any drop at this.
>
>rx_samples_to_file --rate 200e6 --file /dev/null --nsamps $(( 2 * 1000 
>*
>1000 ))
>
>Setting RX Rate: 200.000000 Msps...
>
>Actual RX Rate: 200.000000 Msps...
>
>Setting RX Freq: 0.000000 MHz...
>
>Actual RX Freq: 340.000000 MHz...
>
>Waiting for "lo_locked": ++++++++++ locked.
>
>Done!
>
>
>However with
>
>./rx_samples_to_file --duration 30 --rate 200e6 --freq=2e9 --file
>/dev/null
>
>Setting RX Rate: 200.000000 Msps...
>
>Actual RX Rate: 200.000000 Msps...
>
>Setting RX Freq: 2000.000000 MHz...
>
>Actual RX Freq: 2000.000000 MHz...
>
>Waiting for "lo_locked": ++++++++++ locked.
>
>Press Ctrl + C to stop streaming...
>
>DGot an overflow indication. Please consider the following:
>
>Your write medium must sustain a rate of 800.000000MB/s.
>
>Dropped samples will not be written to the file.
>
>Please modify this example for your purposes.
>
>This message will not appear again.
>
>DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
>
>Done!
>
>
>Even at 25 MSps I got drop of packet
>
>./rx_samples_to_file --duration 30 --rate 25e6 --freq=2e9 --file
>/dev/null
>
>Setting RX Rate: 25.000000 Msps...
>
>Actual RX Rate: 25.000000 Msps...
>
>Setting RX Freq: 2000.000000 MHz...
>
>Actual RX Freq: 2000.000000 MHz...
>
>Waiting for "lo_locked": ++++++++++ locked.
>
>Press Ctrl + C to stop streaming...
>
>DGot an overflow indication. Please consider the following:
>
>  Your write medium must sustain a rate of 100.000000MB/s.
>
>  Dropped samples will not be written to the file.
>
>  Please modify this example for your purposes.
>
>  This message will not appear again.
>
>DD
>
>Done!
>
>
>But with  1 Gig interface with 25 MSps I am not getting any drop
>
>./rx_samples_to_file --duration 30 --rate 25e6 --freq=2e9 --file
>/dev/null
>
>Setting RX Rate: 25.000000 Msps...
>
>Actual RX Rate: 25.000000 Msps...
>
>Setting RX Freq: 2000.000000 MHz...
>
>Actual RX Freq: 2000.000000 MHz...
>
>Waiting for "lo_locked": ++++++++++ locked.
>
>Press Ctrl + C to stop streaming...
>
>Done!
>
>I hope raid is not making an issue here, as I don't see any indication
>when
>I tried with 1 Gig interface.
>
>Please let me know from  seeing the output what you think making the
>trouble.
>
>
>Best regards
>Sanjoy
>
>
>On Sat, Oct 24, 2015 at 1:48 AM, Michael West <michael.west at ettus.com>
>wrote:
>
>> I was corrected regarding the 10 GbE copper cables.  The long copper
>> cables were failing our tests due to a bug in the IP used for the 10
>GbE in
>> the FPGA image that has since been resolved.  We are currently using
>cables
>> up to 3m with no problems.  We have found some 5m cables work and
>some
>> don't depending on the quality of the cable.  The 10 GbE cables and
>NICs
>> sold on the Ettus website all work well.
>>
>> Regards,
>> Michael
>>
>>
>>
>>
>> On Fri, Oct 23, 2015 at 3:47 PM, Marcus Müller
><marcus.mueller at ettus.com>
>> wrote:
>>
>>> So to understand this
>>>
>>> TX much more stable than RX, and RX showing not a single "O" (a
>proper
>>> overflow) but a lot of "D" packet errors (which are, in fact, either
>very
>>> serious packet losses, or package reordering).
>>> It's probably the cable, or your network stack might be up to no
>good.
>>>
>>> Could you share the output of
>>>
>>> ip route
>>>
>>> Also, it's possible that some default firewall rule makes your CPU
>break
>>> a sweat; you could try to bypass firewall for the eth2 device
>>>
>>> sudo iptables -A INPUT -i eth2 -j ACCEPT
>>>
>>> (this is *not* permanent).
>>> If that doesn't help, maybe inspecting the packets captured with
>>> "wireshark" might help. For example, set up wireshark to listen to
>eth2,
>>> power up the USRP; you should see a few packets being exchanged on
>the
>>> interface.
>>> Then run
>>>
>>> rx_samples_to_file --rate 200e6 --file /dev/null --nsamps $(( 2 *
>1000  *
>>> 1000 ))
>>>
>>> then stop the capture. Did you see "D"s? If yes, I'd like to have a
>look
>>> at the captured data; please save it. We'll figure out a way to
>exchange it.
>>>
>>> Best regards,
>>> Marcus
>>> On 23.10.2015 21:21, Sanjoy Basak wrote:
>>>
>>> Hi Michael, Neel and Marcus,
>>>
>>> Thanks a lot for the suggestions. I tried each of those.
>>> I updated the kernel to 3.19.0-31-generic.
>>>
>>> CPU governors, I set to performance as instruction given in this
>link.
>>>
>>>
>http://files.ettus.com/manual/page_usrp_x3x0_config.html#x3x0cfg_hostpc_pwr
>>> Now, after setting it to performance and restarting, if I check with
>>> cpu-freq, I find all governers are set at performance. But after a
>>> while(5/10 mins later) if I check again cpu-freq info, I see all
>governers
>>> are set to powersave.
>>> Could you please tell me why and how to configure it properly?
>>>
>>> I tried with benchmark_rate to check the performance. With tx-rate
>till
>>> 200 MSps I do not get any overflow/underflow. However with rx-rate
>>> 25/50/100 MSps I get drop of packets. I made a direct connection
>from
>>> computer to the X310. No switch is used.
>>>
>>> On the contrary with shorter(copper) cable at 1Gig interface I do
>not get
>>> any drop of packets at 25MSps (on both tx_rate and rx_rate).
>>> Is 3m copper cable really bad?
>>>
>>> We don't have fibre cable right now. We already ordered. I hope we
>will
>>> get it next week and check how it is. We don't have any SFP+ 10 Gig
>>> interface. So can't really test 10 gig interface with short 10 gig
>copper
>>> cable.
>>>
>>> These are the benchmark_rate results for 10 Gig interface
>>>
>>>
>>> Testing receive rate 25.000000 Msps on 1 channels
>>> DDDDDDDD
>>> Benchmark rate summary:
>>>   Num received samples:    249845308
>>>   Num dropped samples:     15968
>>>   Num overflows detected:  0
>>>   Num transmitted samples: 0
>>>   Num sequence errors:     0
>>>   Num underflows detected: 0
>>>
>>>
>>> Testing receive rate 100.000000 Msps on 1 channels
>>> DDDDDDDDDDDDDDDDDDDDD
>>> Benchmark rate summary:
>>>   Num received samples:    999934124
>>>   Num dropped samples:     41916
>>>   Num overflows detected:  0
>>>   Num transmitted samples: 0
>>>   Num sequence errors:     0
>>>   Num underflows detected: 0
>>>
>>>  I also used this command
>>> sudo ethtool -C eth2 rx-usecs 16 rx-frames 20
>>>
>>> Please let me know what you think occurring the problem and
>>> whether replacing copper cable with short fibre cable solves the
>problem.
>>>
>>> Best regards
>>> Sanjoy
>>>
>>>
>>> On Wed, Oct 21, 2015 at 12:27 PM, Marcus Müller <
>>> usrp-users at lists.ettus.com> wrote:
>>>
>>>> Hi Sanjoy,
>>>>
>>>> furthermore, and I found that to be crucial on my system, you
>should
>>>> make sure that your 10GE card doesn't generate an interrupt for
>every
>>>> packet it receives (your application will pull these packets as
>fast as
>>>> possible, anyway).
>>>> On linux, you'd do
>>>>
>>>> sudo ethtool -c {name of ethernet interface, e.g. eth1}
>>>>
>>>> to see the coalescing options available with your device and kernel
>>>> version,
>>>> and modify a setting using
>>>>
>>>> sudo ethtool -C {name of ethernet interface, e.g. eth1} {name of
>>>> setting} {new value of setting}
>>>>
>>>> For example, I set "rx_usecs" (microseconds to wait between
>triggering
>>>> an interrupt) to 16, and "rx_frames" to 20 or so -- your milage
>might vary,
>>>> depending on your application, OS and also a few hardware factors.
>>>>
>>>> Best regards,
>>>> Marcus
>>>>
>>>>
>>>> On 19.10.2015 21:56, Michael West via USRP-users wrote:
>>>>
>>>> Also, check the CPU governor and make sure it is set to
>"performance".
>>>>
>>>> Your output indicates frame errors on the interface, which could be
>due
>>>> to the cable.  If possible, try using a shorter copper cable or
>switch to a
>>>> fibre cable.  With 10 GbE over copper, the shorter the better.  If
>you need
>>>> the length, you should be using fibre.
>>>>
>>>> If you still have issues, please provide a little more information:
> Are
>>>> you connected directly with the X310 or through a switch?  How are
>you
>>>> testing the performance?  Are you using your own application or
>>>> benchmark_rate?  If your own application, what is it doing with the
>>>> received data (processing it with the same thread or different
>thread,
>>>> saving to disk, etc...)?
>>>>
>>>> Regards,
>>>> Michael
>>>>
>>>> On Mon, Oct 19, 2015 at 12:51 PM, Neel Pandeya via USRP-users <
>>>> usrp-users at lists.ettus.com> wrote:
>>>>
>>>>> Hello Sanjoy:
>>>>>
>>>>> That Intel X520 10GbE card with Ubuntu 14.04.3 should be working
>well
>>>>> for you.
>>>>>
>>>>> Could you try upgrading to kernel 3.16 or 3.19, and let us know
>your
>>>>> results?
>>>>>
>>>>> To upgrade to kernel 3.16, run "sudo apt-get install
>>>>> linux-generic-lts-utopic".
>>>>>
>>>>> To upgrade to kernel 3.19, run "sudo apt-get install
>>>>> linux-generic-lts-vivid".
>>>>>
>>>>> After the upgrade, be sure to reboot the system.
>>>>>
>>>>> Also, check that you have the CPU governors set to "performance",
>and
>>>>> that you have the highest clock rate set.
>>>>>
>>>>> --Neel
>>>>>
>>>>>
>>>>>
>>>>> On 19 October 2015 at 12:03, Sanjoy Basak via USRP-users <
>>>>> <usrp-users at lists.ettus.com>usrp-users at lists.ettus.com> wrote:
>>>>>
>>>>>> Hi Experts,
>>>>>> We recently bought a 10 Gig ethernet card (X510-DA2) and
>Twinaxial-Kabel
>>>>>> SFP+ M SFP+ M 3,0m. I installed it in our pc (OS: Ubuntu Mate
>14.04.3 LTS)
>>>>>> and can communicate with X310/SBX-120 properly. However, the
>performance we
>>>>>> are getting is quite poor. I am getting dropped packets (D O)
>mostly at
>>>>>> 25,50 MSps and goes quite crazy at 100 MSps and sometimes even at
>lower
>>>>>> sample rates.
>>>>>> Previously I tested with 1 Gig ethernet, at 25 MSps I could
>stream
>>>>>> without any drop/late packet or underrun.
>>>>>> The CPU uses or network uses is also not reaching 100%.
>>>>>>
>>>>>> Could you please tell me what to check/correct so I can stream at
>100
>>>>>> MSps without any error?
>>>>>>
>>>>>> I am putting the ifconfig and ethtool results and the kernel
>version(
>>>>>> 3.16.0-30-generic)
>>>>>>
>>>>>> eth2      Link encap:Ethernet  HWaddr 90:e2:ba:a6:a9:dd
>>>>>>           inet6 addr: fe80::92e2:baff:fea6:a9dd/64 Scope:Link
>>>>>>           UP BROADCAST MULTICAST  MTU:9000  Metric:1
>>>>>>           RX packets:4530726 errors:146 dropped:0 overruns:0
>frame:146
>>>>>>           TX packets:6811057 errors:0 dropped:0 overruns:0
>carrier:0
>>>>>>           collisions:0 txqueuelen:1000
>>>>>>           RX bytes:26331306938 (26.3 GB)  TX bytes:35856233935
>(35.8
>>>>>> GB)
>>>>>>
>>>>>> Settings for eth2:
>>>>>> Supported ports: [ FIBRE ]
>>>>>> Supported link modes:   10000baseT/Full
>>>>>> Supported pause frame use: No
>>>>>> Supports auto-negotiation: No
>>>>>> Advertised link modes:  10000baseT/Full
>>>>>> Advertised pause frame use: No
>>>>>> Advertised auto-negotiation: No
>>>>>> Speed: 10000Mb/s
>>>>>> Duplex: Full
>>>>>> Port: Other
>>>>>> PHYAD: 0
>>>>>> Transceiver: external
>>>>>> Auto-negotiation: off
>>>>>> Cannot get wake-on-lan settings: Operation not permitted
>>>>>> Current message level: 0x00000007 (7)
>>>>>>       drv probe link
>>>>>> Link detected: yes
>>>>>>
>>>>>> Our computer config is:
>>>>>> Processor: 12*Intel(R) Xeon (R) CPU E5-2620 v3 @2.40 GHz
>>>>>> RAM: 65871 MB
>>>>>> SSD raid 5
>>>>>>
>>>>>>
>>>>>> Best regards
>>>>>> Sanjoy
>>>>>>
>>>>>> _______________________________________________
>>>>>> USRP-users mailing list
>>>>>> USRP-users at lists.ettus.com
>>>>>>
>http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> USRP-users mailing list
>>>>> USRP-users at lists.ettus.com
>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> USRP-users mailing
>listUSRP-users at lists.ettus.comhttp://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> USRP-users mailing list
>>>> USRP-users at lists.ettus.com
>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>
>>>>
>>>
>>>
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/attachments/20151024/864f07a1/attachment-0002.html>


More information about the USRP-users mailing list