[USRP-users] tx_streamer and timed_transmit fail depending on multi_usrp configuration.

Michael R. Freedman mfreedman at ll.mit.edu
Wed Mar 27 14:57:23 EDT 2019


Hello,

      I have a need to have a separate tx_streamer for each channel in a 
multi_usrp using timed transmission.  I am seeing bizarre results that 
at this point I can't explain.  I have tried 3.10, 3.13 and the latest 
release 3.14 all with different results.  I have attached some sample 
code that illustrates the issue.  It is a non-useable example and 
doesn't match the exact scenario that I am using, but it is however 
simple and displays the same behavior that I am seeing.


System Configuration :

6 Ettus X310s rev6 with UBX-40 v2 boards
Ettus Octoclock with GPS antenna using internal pps and ref
Netgear XSM4324S-100NES 10GigE switch configured for 9000MTU


Tried both a :
Mellanox MT27710 dual 10GigE Card
On Board Intel X550 10GigE

Tried both Fiberstore cables as well as Ettus Cables

Server is a Supermicro X121DPX-T based system with dual Intel Xeon Gold 
6154 18core processors
128GB memory

Linux Mint 19.1 with 4.15.0-20-generic kernel
CPU governor is disabled.


ethtool -g enp24s0f0
Ring parameters for enp24s0f0:
Pre-set maximums:
RX:        8192
RX Mini:    0
RX Jumbo:    0
TX:        8192
Current hardware settings:
RX:        8192
RX Mini:    0
RX Jumbo:    0
TX:        8192


sysctl.conf file
et.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv6.conf.all.disable_ipv6 = 1
fs.inotify.max_user_watches = 65536
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
net.core.rmem_default=2097152
net.core.rmem_max = 33554432
net.core.wmem_default = 1048576
net.core.wmem_max = 33554432
net.core.netdev_max_backlog = 100000
net.ipv4.ipfrag_high_threshold = 8388608
net.ipv4.ipfrag_high_thresh = 8388608
kernel.sem = 4096 1024000 4096 4096
kernel.shmmni = 8192
kernel.shmall = 3221225472
kernel.shmmax = 17179869184
kernel.msgmni = 16384
kernel.msgmax = 65536
kernel.msgmnb = 65536
fs.mqueue.msg_max = 512
fs.mqueue.queues_max = 256

These particular results are from 3.14.0.0 :

Successful :
./bin/tx_timed_samples --args=addr0=192.168.40.100 --subdev="A:0" 
--channels="0" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples --args=addr0=192.168.40.100 --subdev="A:0 B:0" 
--channels="0" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples --args=addr0=192.168.40.100,addr1=192.168.40.101 
--subdev="A:0 B:0" --channels="0" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples --args=addr0=192.168.40.100,addr1=192.168.40.101 
--subdev="A:0 B:0" --channels="2" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples 
--args=addr0=192.168.40.100,addr1=192.168.40.101,addr2=192.168.40.102 
--subdev="A:0 B:0" --channels="0" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples 
--args=addr0=192.168.40.100,addr1=192.168.40.101,addr2=192.168.40.102 
--subdev="A:0 B:0" --channels="4" --nsamp 10000 --rate 2000000 --wire="sc16"

Late packets but still says success :
./bin/tx_timed_samples --args=addr0=192.168.40.100 --subdev="B:0" 
--channels="0" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples --args=addr0=192.168.40.100 --subdev="A:0 B:0" 
--channels="1" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples 
--args=addr0=192.168.40.100,addr1=192.168.40.101,addr2=192.168.40.102 
--subdev="A:0 B:0" --channels="3" --nsamp 10000 --rate 2000000 --wire="sc16"

Fail : ( never gets acks )
./bin/tx_timed_samples --args=addr0=192.168.40.100,addr1=192.168.40.101 
--subdev="A:0 B:0" --channels="1" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples --args=addr0=192.168.40.100,addr1=192.168.40.101 
--subdev="A:0 B:0" --channels="3" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples 
--args=addr0=192.168.40.100,addr1=192.168.40.101,addr2=192.168.40.102 
--subdev="A:0 B:0" --channels="1" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples 
--args=addr0=192.168.40.100,addr1=192.168.40.101,addr2=192.168.40.102 
--subdev="A:0 B:0" --channels="2" --nsamp 10000 --rate 2000000 --wire="sc16"
./bin/tx_timed_samples 
--args=addr0=192.168.40.100,addr1=192.168.40.101,addr2=192.168.40.102 
--subdev="A:0 B:0" --channels="5" --nsamp 10000 --rate 2000000 --wire="sc16"

Moving the address order moves the problem as expected.  While I haven't 
run this particular program on anything but 3.14.0.0, with the 
application that I am writing
I see this behavior on 3.14.  On 3.13 I can write to channel 0 but get 
Time_Code_Errors on channel 1 in a single USRP setup.  With multiple 
USRPs I never get ack messages
on channel 1.  On 3.10 I seem to get the ack messages but have underunns 
and late messages.  On 3.13 I have little success until I lower the MTU 
size down below 6000.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: tx_timed_samples.cpp
Type: text/x-c++src
Size: 8983 bytes
Desc: not available
URL: <http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/attachments/20190327/3bb7dce3/attachment.cpp>


More information about the USRP-users mailing list