[USRP-users] about 10 Gig ethernet configuration

Sanjoy Basak sanjoybasak14 at gmail.com
Sat Oct 24 09:58:28 EDT 2015


Hi Micheal,
I made the cpu governor to performance. This is always performance now,
does not change. We did not buy the cable from ettus.
We bought DeLOCK Twinaxial-Kabel SFP+ M SFP+ M 3,0m SFF-8431 86222 from a
store here.

Hi Rob,
Thanks for the command. I am not really sure which cable you used. But I am
still having drop of packets. I don't have any other cable right now to
test.


Hi Marcus,
I tried your suggestions. I put the output here.

*ip route *

192.168.40.0/24 dev eth2 proto kernel scope link src 192.168.40.1 metric 1


I don't get any drop at this.

rx_samples_to_file --rate 200e6 --file /dev/null --nsamps $(( 2 * 1000  *
1000 ))

Setting RX Rate: 200.000000 Msps...

Actual RX Rate: 200.000000 Msps...

Setting RX Freq: 0.000000 MHz...

Actual RX Freq: 340.000000 MHz...

Waiting for "lo_locked": ++++++++++ locked.

Done!


However with

./rx_samples_to_file --duration 30 --rate 200e6 --freq=2e9 --file /dev/null

Setting RX Rate: 200.000000 Msps...

Actual RX Rate: 200.000000 Msps...

Setting RX Freq: 2000.000000 MHz...

Actual RX Freq: 2000.000000 MHz...

Waiting for "lo_locked": ++++++++++ locked.

Press Ctrl + C to stop streaming...

DGot an overflow indication. Please consider the following:

Your write medium must sustain a rate of 800.000000MB/s.

Dropped samples will not be written to the file.

Please modify this example for your purposes.

This message will not appear again.

DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

Done!


Even at 25 MSps I got drop of packet

./rx_samples_to_file --duration 30 --rate 25e6 --freq=2e9 --file /dev/null

Setting RX Rate: 25.000000 Msps...

Actual RX Rate: 25.000000 Msps...

Setting RX Freq: 2000.000000 MHz...

Actual RX Freq: 2000.000000 MHz...

Waiting for "lo_locked": ++++++++++ locked.

Press Ctrl + C to stop streaming...

DGot an overflow indication. Please consider the following:

  Your write medium must sustain a rate of 100.000000MB/s.

  Dropped samples will not be written to the file.

  Please modify this example for your purposes.

  This message will not appear again.

DD

Done!


But with  1 Gig interface with 25 MSps I am not getting any drop

./rx_samples_to_file --duration 30 --rate 25e6 --freq=2e9 --file /dev/null

Setting RX Rate: 25.000000 Msps...

Actual RX Rate: 25.000000 Msps...

Setting RX Freq: 2000.000000 MHz...

Actual RX Freq: 2000.000000 MHz...

Waiting for "lo_locked": ++++++++++ locked.

Press Ctrl + C to stop streaming...

Done!

I hope raid is not making an issue here, as I don't see any indication when
I tried with 1 Gig interface.

Please let me know from  seeing the output what you think making the
trouble.


Best regards
Sanjoy


On Sat, Oct 24, 2015 at 1:48 AM, Michael West <michael.west at ettus.com>
wrote:

> I was corrected regarding the 10 GbE copper cables.  The long copper
> cables were failing our tests due to a bug in the IP used for the 10 GbE in
> the FPGA image that has since been resolved.  We are currently using cables
> up to 3m with no problems.  We have found some 5m cables work and some
> don't depending on the quality of the cable.  The 10 GbE cables and NICs
> sold on the Ettus website all work well.
>
> Regards,
> Michael
>
>
>
>
> On Fri, Oct 23, 2015 at 3:47 PM, Marcus Müller <marcus.mueller at ettus.com>
> wrote:
>
>> So to understand this
>>
>> TX much more stable than RX, and RX showing not a single "O" (a proper
>> overflow) but a lot of "D" packet errors (which are, in fact, either very
>> serious packet losses, or package reordering).
>> It's probably the cable, or your network stack might be up to no good.
>>
>> Could you share the output of
>>
>> ip route
>>
>> Also, it's possible that some default firewall rule makes your CPU break
>> a sweat; you could try to bypass firewall for the eth2 device
>>
>> sudo iptables -A INPUT -i eth2 -j ACCEPT
>>
>> (this is *not* permanent).
>> If that doesn't help, maybe inspecting the packets captured with
>> "wireshark" might help. For example, set up wireshark to listen to eth2,
>> power up the USRP; you should see a few packets being exchanged on the
>> interface.
>> Then run
>>
>> rx_samples_to_file --rate 200e6 --file /dev/null --nsamps $(( 2 * 1000  *
>> 1000 ))
>>
>> then stop the capture. Did you see "D"s? If yes, I'd like to have a look
>> at the captured data; please save it. We'll figure out a way to exchange it.
>>
>> Best regards,
>> Marcus
>> On 23.10.2015 21:21, Sanjoy Basak wrote:
>>
>> Hi Michael, Neel and Marcus,
>>
>> Thanks a lot for the suggestions. I tried each of those.
>> I updated the kernel to 3.19.0-31-generic.
>>
>> CPU governors, I set to performance as instruction given in this link.
>>
>> http://files.ettus.com/manual/page_usrp_x3x0_config.html#x3x0cfg_hostpc_pwr
>> Now, after setting it to performance and restarting, if I check with
>> cpu-freq, I find all governers are set at performance. But after a
>> while(5/10 mins later) if I check again cpu-freq info, I see all governers
>> are set to powersave.
>> Could you please tell me why and how to configure it properly?
>>
>> I tried with benchmark_rate to check the performance. With tx-rate till
>> 200 MSps I do not get any overflow/underflow. However with rx-rate
>> 25/50/100 MSps I get drop of packets. I made a direct connection from
>> computer to the X310. No switch is used.
>>
>> On the contrary with shorter(copper) cable at 1Gig interface I do not get
>> any drop of packets at 25MSps (on both tx_rate and rx_rate).
>> Is 3m copper cable really bad?
>>
>> We don't have fibre cable right now. We already ordered. I hope we will
>> get it next week and check how it is. We don't have any SFP+ 10 Gig
>> interface. So can't really test 10 gig interface with short 10 gig copper
>> cable.
>>
>> These are the benchmark_rate results for 10 Gig interface
>>
>>
>> Testing receive rate 25.000000 Msps on 1 channels
>> DDDDDDDD
>> Benchmark rate summary:
>>   Num received samples:    249845308
>>   Num dropped samples:     15968
>>   Num overflows detected:  0
>>   Num transmitted samples: 0
>>   Num sequence errors:     0
>>   Num underflows detected: 0
>>
>>
>> Testing receive rate 100.000000 Msps on 1 channels
>> DDDDDDDDDDDDDDDDDDDDD
>> Benchmark rate summary:
>>   Num received samples:    999934124
>>   Num dropped samples:     41916
>>   Num overflows detected:  0
>>   Num transmitted samples: 0
>>   Num sequence errors:     0
>>   Num underflows detected: 0
>>
>>  I also used this command
>> sudo ethtool -C eth2 rx-usecs 16 rx-frames 20
>>
>> Please let me know what you think occurring the problem and
>> whether replacing copper cable with short fibre cable solves the problem.
>>
>> Best regards
>> Sanjoy
>>
>>
>> On Wed, Oct 21, 2015 at 12:27 PM, Marcus Müller <
>> usrp-users at lists.ettus.com> wrote:
>>
>>> Hi Sanjoy,
>>>
>>> furthermore, and I found that to be crucial on my system, you should
>>> make sure that your 10GE card doesn't generate an interrupt for every
>>> packet it receives (your application will pull these packets as fast as
>>> possible, anyway).
>>> On linux, you'd do
>>>
>>> sudo ethtool -c {name of ethernet interface, e.g. eth1}
>>>
>>> to see the coalescing options available with your device and kernel
>>> version,
>>> and modify a setting using
>>>
>>> sudo ethtool -C {name of ethernet interface, e.g. eth1} {name of
>>> setting} {new value of setting}
>>>
>>> For example, I set "rx_usecs" (microseconds to wait between triggering
>>> an interrupt) to 16, and "rx_frames" to 20 or so -- your milage might vary,
>>> depending on your application, OS and also a few hardware factors.
>>>
>>> Best regards,
>>> Marcus
>>>
>>>
>>> On 19.10.2015 21:56, Michael West via USRP-users wrote:
>>>
>>> Also, check the CPU governor and make sure it is set to "performance".
>>>
>>> Your output indicates frame errors on the interface, which could be due
>>> to the cable.  If possible, try using a shorter copper cable or switch to a
>>> fibre cable.  With 10 GbE over copper, the shorter the better.  If you need
>>> the length, you should be using fibre.
>>>
>>> If you still have issues, please provide a little more information:  Are
>>> you connected directly with the X310 or through a switch?  How are you
>>> testing the performance?  Are you using your own application or
>>> benchmark_rate?  If your own application, what is it doing with the
>>> received data (processing it with the same thread or different thread,
>>> saving to disk, etc...)?
>>>
>>> Regards,
>>> Michael
>>>
>>> On Mon, Oct 19, 2015 at 12:51 PM, Neel Pandeya via USRP-users <
>>> usrp-users at lists.ettus.com> wrote:
>>>
>>>> Hello Sanjoy:
>>>>
>>>> That Intel X520 10GbE card with Ubuntu 14.04.3 should be working well
>>>> for you.
>>>>
>>>> Could you try upgrading to kernel 3.16 or 3.19, and let us know your
>>>> results?
>>>>
>>>> To upgrade to kernel 3.16, run "sudo apt-get install
>>>> linux-generic-lts-utopic".
>>>>
>>>> To upgrade to kernel 3.19, run "sudo apt-get install
>>>> linux-generic-lts-vivid".
>>>>
>>>> After the upgrade, be sure to reboot the system.
>>>>
>>>> Also, check that you have the CPU governors set to "performance", and
>>>> that you have the highest clock rate set.
>>>>
>>>> --Neel
>>>>
>>>>
>>>>
>>>> On 19 October 2015 at 12:03, Sanjoy Basak via USRP-users <
>>>> <usrp-users at lists.ettus.com>usrp-users at lists.ettus.com> wrote:
>>>>
>>>>> Hi Experts,
>>>>> We recently bought a 10 Gig ethernet card (X510-DA2) and Twinaxial-Kabel
>>>>> SFP+ M SFP+ M 3,0m. I installed it in our pc (OS: Ubuntu Mate 14.04.3 LTS)
>>>>> and can communicate with X310/SBX-120 properly. However, the performance we
>>>>> are getting is quite poor. I am getting dropped packets (D O) mostly at
>>>>> 25,50 MSps and goes quite crazy at 100 MSps and sometimes even at lower
>>>>> sample rates.
>>>>> Previously I tested with 1 Gig ethernet, at 25 MSps I could stream
>>>>> without any drop/late packet or underrun.
>>>>> The CPU uses or network uses is also not reaching 100%.
>>>>>
>>>>> Could you please tell me what to check/correct so I can stream at 100
>>>>> MSps without any error?
>>>>>
>>>>> I am putting the ifconfig and ethtool results and the kernel version(
>>>>> 3.16.0-30-generic)
>>>>>
>>>>> eth2      Link encap:Ethernet  HWaddr 90:e2:ba:a6:a9:dd
>>>>>           inet6 addr: fe80::92e2:baff:fea6:a9dd/64 Scope:Link
>>>>>           UP BROADCAST MULTICAST  MTU:9000  Metric:1
>>>>>           RX packets:4530726 errors:146 dropped:0 overruns:0 frame:146
>>>>>           TX packets:6811057 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:1000
>>>>>           RX bytes:26331306938 (26.3 GB)  TX bytes:35856233935 (35.8
>>>>> GB)
>>>>>
>>>>> Settings for eth2:
>>>>> Supported ports: [ FIBRE ]
>>>>> Supported link modes:   10000baseT/Full
>>>>> Supported pause frame use: No
>>>>> Supports auto-negotiation: No
>>>>> Advertised link modes:  10000baseT/Full
>>>>> Advertised pause frame use: No
>>>>> Advertised auto-negotiation: No
>>>>> Speed: 10000Mb/s
>>>>> Duplex: Full
>>>>> Port: Other
>>>>> PHYAD: 0
>>>>> Transceiver: external
>>>>> Auto-negotiation: off
>>>>> Cannot get wake-on-lan settings: Operation not permitted
>>>>> Current message level: 0x00000007 (7)
>>>>>       drv probe link
>>>>> Link detected: yes
>>>>>
>>>>> Our computer config is:
>>>>> Processor: 12*Intel(R) Xeon (R) CPU E5-2620 v3 @2.40 GHz
>>>>> RAM: 65871 MB
>>>>> SSD raid 5
>>>>>
>>>>>
>>>>> Best regards
>>>>> Sanjoy
>>>>>
>>>>> _______________________________________________
>>>>> USRP-users mailing list
>>>>> USRP-users at lists.ettus.com
>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> USRP-users mailing list
>>>> USRP-users at lists.ettus.com
>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> USRP-users mailing listUSRP-users at lists.ettus.comhttp://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>
>>>
>>>
>>> _______________________________________________
>>> USRP-users mailing list
>>> USRP-users at lists.ettus.com
>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/attachments/20151024/e0f62492/attachment-0002.html>


More information about the USRP-users mailing list