usrp-users@lists.ettus.com

Discussion and technical support related to USRP, UHD, RFNoC

View all threads

Several issues with UHD 3.10.1.0 and X300

SM
Serge Malo
Tue, Dec 6, 2016 7:39 PM

Hi all,

We have started the process of upgrading the UHD version in our application
to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate", so it
will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000, and
added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s before I
restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could make in
order to make sure the radio is ready for a new process? In our old UHD
version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed. unexpected
FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to reproduce
it with a simple script calling "benchmark_rate" in a loop under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result file
"doit_res.txt")
We have clients and partners using our application with the older version
of UHD who reported this error happening about 1% of the time. The issue
seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate to
execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed:
EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a
sequence error on the 2nd loop, but all samples are transmitted (no error
message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

Hi all, We have started the process of upgrading the UHD version in our application to UHD-003-010-001-000, but we are facing several issues. I have reproduced those issues with the UHD example "benchmark_rate", so it will be easier to find solutions or workarounds. Configuration: O/S: Windows 10 and Ubuntu 14.04LTS. X300 P/N: 156485E-1DL S/N: 30DB7E8 FPGA image: usrp_x300_fpga_HG.bit 10GbE connection with Intel X520-DA2 card. CPU: Intel core i7 4790K. I have recompiled UHD myself from the tag release_003_010_001_000, and added a patch to fix the timed commands (patch given by Derek) Issue #1: Can't find X300 when restarting application. When benchmark_rate process ends normally, I have to wait 5-10s before I restart it again, otherwise I get this error: Error: LookupError: KeyError: No devices found for -----> Device Address: addr: 192.168.40.2 Q: At the end of a transmission, is there call or a poll we could make in order to make sure the radio is ready for a new process? In our old UHD version, this was not happening. This issue happens on Windows and Ubuntu. Issue #2: Error message "x300_dac_ctrl: front-end sync failed. unexpected FIFO depth [0x1f]" This error happens only about 1% of the time, but I was able to reproduce it with a simple script calling "benchmark_rate" in a loop under Ubuntu. See attached script "doit.sh". In my case, the error occurred 1/100 at Loop #23 (See attached result file "doit_res.txt") We have clients and partners using our application with the older version of UHD who reported this error happening about 1% of the time. The issue seems to still be present in UHD 3.10.1.0. Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to external source. This error happened 3 times out of my 100 iteration test. See attached result file "doit_res.txt" Issue #4: Can't restart the streaming within a process I have tried to add a loop inside the source code of benchmark_rate to execute the test several times. However, I get this error message on the 2nd loop: Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - AssertionError: buff->size() > 0 in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool) at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a sequence error on the 2nd loop, but all samples are transmitted (no error message) See attached benchmark_rate_modified.cpp Please, let me know if there is better/cleaner way to do this "repeat loop". Thanks in advance for your help! Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol>
MW
Michael West
Tue, Dec 6, 2016 11:11 PM

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate", so
it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000, and
added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s before I
restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could make in
order to make sure the radio is ready for a new process? In our old UHD
version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed. unexpected
FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to reproduce
it with a simple script calling "benchmark_rate" in a loop under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result file
"doit_res.txt")
We have clients and partners using our application with the older version
of UHD who reported this error happening about 1% of the time. The issue
seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate to
execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed:
EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a
sequence error on the 2nd loop, but all samples are transmitted (no error
message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat
loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Serge, Thank you for the detailed information and examples. I am working on reproducing the issues with the information you have provided. I will let you know what I find. Regards, Michael On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < usrp-users@lists.ettus.com> wrote: > Hi all, > > We have started the process of upgrading the UHD version in our > application to UHD-003-010-001-000, but we are facing several issues. > I have reproduced those issues with the UHD example "benchmark_rate", so > it will be easier to find solutions or workarounds. > > Configuration: > O/S: Windows 10 and Ubuntu 14.04LTS. > X300 P/N: 156485E-1DL S/N: 30DB7E8 > FPGA image: usrp_x300_fpga_HG.bit > 10GbE connection with Intel X520-DA2 card. > CPU: Intel core i7 4790K. > I have recompiled UHD myself from the tag release_003_010_001_000, and > added a patch to fix the timed commands (patch given by Derek) > > Issue #1: Can't find X300 when restarting application. > When benchmark_rate process ends normally, I have to wait 5-10s before I > restart it again, otherwise I get this error: > Error: LookupError: KeyError: No devices found for -----> > Device Address: > addr: 192.168.40.2 > Q: At the end of a transmission, is there call or a poll we could make in > order to make sure the radio is ready for a new process? In our old UHD > version, this was not happening. > This issue happens on Windows and Ubuntu. > > > Issue #2: Error message "x300_dac_ctrl: front-end sync failed. unexpected > FIFO depth [0x1f]" > This error happens only about 1% of the time, but I was able to reproduce > it with a simple script calling "benchmark_rate" in a loop under Ubuntu. > See attached script "doit.sh". > In my case, the error occurred 1/100 at Loop #23 (See attached result file > "doit_res.txt") > We have clients and partners using our application with the older version > of UHD who reported this error happening about 1% of the time. The issue > seems to still be present in UHD 3.10.1.0. > > > Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to > external source. > This error happened 3 times out of my 100 iteration test. > See attached result file "doit_res.txt" > > > Issue #4: Can't restart the streaming within a process > I have tried to add a loop inside the source code of benchmark_rate to > execute the test several times. However, I get this error message on the > 2nd loop: > Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: > EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: > EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - > AssertionError: buff->size() > 0 > in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool) > at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 > > NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a > sequence error on the 2nd loop, but all samples are transmitted (no error > message) > See attached benchmark_rate_modified.cpp > Please, let me know if there is better/cleaner way to do this "repeat > loop". > > > Thanks in advance for your help! > Serge > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > _______________________________________________ > USRP-users mailing list > USRP-users@lists.ettus.com > http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com > >
MW
Michael West
Wed, Dec 7, 2016 6:29 PM

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but not
with the head of the maint branch.  So, I believe that one has been fixed.
I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate", so
it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000, and
added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s before I
restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could make in
order to make sure the radio is ready for a new process? In our old UHD
version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed. unexpected
FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to reproduce
it with a simple script calling "benchmark_rate" in a loop under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result
file "doit_res.txt")
We have clients and partners using our application with the older version
of UHD who reported this error happening about 1% of the time. The issue
seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate to
execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed:
EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a
sequence error on the 2nd loop, but all samples are transmitted (no error
message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat
loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Serge, I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but not with the head of the maint branch. So, I believe that one has been fixed. I am still working on reproducing the other 3 issues. Regards, Michael On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > Thank you for the detailed information and examples. I am working on > reproducing the issues with the information you have provided. I will let > you know what I find. > > Regards, > Michael > > On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < > usrp-users@lists.ettus.com> wrote: > >> Hi all, >> >> We have started the process of upgrading the UHD version in our >> application to UHD-003-010-001-000, but we are facing several issues. >> I have reproduced those issues with the UHD example "benchmark_rate", so >> it will be easier to find solutions or workarounds. >> >> Configuration: >> O/S: Windows 10 and Ubuntu 14.04LTS. >> X300 P/N: 156485E-1DL S/N: 30DB7E8 >> FPGA image: usrp_x300_fpga_HG.bit >> 10GbE connection with Intel X520-DA2 card. >> CPU: Intel core i7 4790K. >> I have recompiled UHD myself from the tag release_003_010_001_000, and >> added a patch to fix the timed commands (patch given by Derek) >> >> Issue #1: Can't find X300 when restarting application. >> When benchmark_rate process ends normally, I have to wait 5-10s before I >> restart it again, otherwise I get this error: >> Error: LookupError: KeyError: No devices found for -----> >> Device Address: >> addr: 192.168.40.2 >> Q: At the end of a transmission, is there call or a poll we could make in >> order to make sure the radio is ready for a new process? In our old UHD >> version, this was not happening. >> This issue happens on Windows and Ubuntu. >> >> >> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. unexpected >> FIFO depth [0x1f]" >> This error happens only about 1% of the time, but I was able to reproduce >> it with a simple script calling "benchmark_rate" in a loop under Ubuntu. >> See attached script "doit.sh". >> In my case, the error occurred 1/100 at Loop #23 (See attached result >> file "doit_res.txt") >> We have clients and partners using our application with the older version >> of UHD who reported this error happening about 1% of the time. The issue >> seems to still be present in UHD 3.10.1.0. >> >> >> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to >> external source. >> This error happened 3 times out of my 100 iteration test. >> See attached result file "doit_res.txt" >> >> >> Issue #4: Can't restart the streaming within a process >> I have tried to add a loop inside the source code of benchmark_rate to >> execute the test several times. However, I get this error message on the >> 2nd loop: >> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: >> EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >> AssertionError: buff->size() > 0 >> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool) >> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 >> >> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a >> sequence error on the 2nd loop, but all samples are transmitted (no error >> message) >> See attached benchmark_rate_modified.cpp >> Please, let me know if there is better/cleaner way to do this "repeat >> loop". >> >> >> Thanks in advance for your help! >> Serge >> >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 <(514)%20294-4017> >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> _______________________________________________ >> USRP-users mailing list >> USRP-users@lists.ettus.com >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >> >> >
SM
Serge Malo
Wed, Dec 7, 2016 8:35 PM

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but not
with the head of the maint branch.  So, I believe that one has been fixed.
I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate", so
it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000, and
added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s before I
restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could make
in order to make sure the radio is ready for a new process? In our old UHD
version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result
file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate to
execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed:
EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a
sequence error on the 2nd loop, but all samples are transmitted (no error
message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat
loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Thanks for the update Michael, I will try the head of the maint branch on my side too as soon as possible. Regards, Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but not > with the head of the maint branch. So, I believe that one has been fixed. > I am still working on reproducing the other 3 issues. > > Regards, > Michael > > On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> > wrote: > >> Hi Serge, >> >> Thank you for the detailed information and examples. I am working on >> reproducing the issues with the information you have provided. I will let >> you know what I find. >> >> Regards, >> Michael >> >> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >> usrp-users@lists.ettus.com> wrote: >> >>> Hi all, >>> >>> We have started the process of upgrading the UHD version in our >>> application to UHD-003-010-001-000, but we are facing several issues. >>> I have reproduced those issues with the UHD example "benchmark_rate", so >>> it will be easier to find solutions or workarounds. >>> >>> Configuration: >>> O/S: Windows 10 and Ubuntu 14.04LTS. >>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>> FPGA image: usrp_x300_fpga_HG.bit >>> 10GbE connection with Intel X520-DA2 card. >>> CPU: Intel core i7 4790K. >>> I have recompiled UHD myself from the tag release_003_010_001_000, and >>> added a patch to fix the timed commands (patch given by Derek) >>> >>> Issue #1: Can't find X300 when restarting application. >>> When benchmark_rate process ends normally, I have to wait 5-10s before I >>> restart it again, otherwise I get this error: >>> Error: LookupError: KeyError: No devices found for -----> >>> Device Address: >>> addr: 192.168.40.2 >>> Q: At the end of a transmission, is there call or a poll we could make >>> in order to make sure the radio is ready for a new process? In our old UHD >>> version, this was not happening. >>> This issue happens on Windows and Ubuntu. >>> >>> >>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>> unexpected FIFO depth [0x1f]" >>> This error happens only about 1% of the time, but I was able to >>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>> Ubuntu. >>> See attached script "doit.sh". >>> In my case, the error occurred 1/100 at Loop #23 (See attached result >>> file "doit_res.txt") >>> We have clients and partners using our application with the older >>> version of UHD who reported this error happening about 1% of the time. The >>> issue seems to still be present in UHD 3.10.1.0. >>> >>> >>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to >>> external source. >>> This error happened 3 times out of my 100 iteration test. >>> See attached result file "doit_res.txt" >>> >>> >>> Issue #4: Can't restart the streaming within a process >>> I have tried to add a loop inside the source code of benchmark_rate to >>> execute the test several times. However, I get this error message on the >>> 2nd loop: >>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: >>> EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>> AssertionError: buff->size() > 0 >>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool) >>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 >>> >>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get a >>> sequence error on the 2nd loop, but all samples are transmitted (no error >>> message) >>> See attached benchmark_rate_modified.cpp >>> Please, let me know if there is better/cleaner way to do this "repeat >>> loop". >>> >>> >>> Thanks in advance for your help! >>> Serge >>> >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 <(514)%20294-4017> >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> _______________________________________________ >>> USRP-users mailing list >>> USRP-users@lists.ettus.com >>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>> >>> >> >
MW
Michael West
Thu, Dec 8, 2016 12:56 AM

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried running
the script on UHD 3.10.1.0 and the head of the maint branch without any
issue.  I am even using a rev 5 device to match the X300 hardware revision
of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these failures
    when using the internal reference?
  2. Do you see these failures on other devices or just this one device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo serge.malo@skydelsolutions.com
wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but
not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate",
so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000, and
added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s before
I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could make
in order to make sure the radio is ready for a new process? In our old UHD
version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result
file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate to
execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed:
EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get
a sequence error on the 2nd loop, but all samples are transmitted (no error
message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat
loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Serge, I cannot seem to reproduce the issue #2 or issue #3. I have tried running the script on UHD 3.10.1.0 and the head of the maint branch without any issue. I am even using a rev 5 device to match the X300 hardware revision of your device. There are 2 things I am curious about: 1) What is the external reference being used? Do you see these failures when using the internal reference? 2) Do you see these failures on other devices or just this one device? I will be looking into issue #4 tomorrow. Regards, Michael On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <serge.malo@skydelsolutions.com> wrote: > Thanks for the update Michael, > I will try the head of the maint branch on my side too as soon as possible. > > Regards, > Serge > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> wrote: > >> Hi Serge, >> >> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but >> not with the head of the maint branch. So, I believe that one has been >> fixed. I am still working on reproducing the other 3 issues. >> >> Regards, >> Michael >> >> On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> >> wrote: >> >>> Hi Serge, >>> >>> Thank you for the detailed information and examples. I am working on >>> reproducing the issues with the information you have provided. I will let >>> you know what I find. >>> >>> Regards, >>> Michael >>> >>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>> usrp-users@lists.ettus.com> wrote: >>> >>>> Hi all, >>>> >>>> We have started the process of upgrading the UHD version in our >>>> application to UHD-003-010-001-000, but we are facing several issues. >>>> I have reproduced those issues with the UHD example "benchmark_rate", >>>> so it will be easier to find solutions or workarounds. >>>> >>>> Configuration: >>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>> FPGA image: usrp_x300_fpga_HG.bit >>>> 10GbE connection with Intel X520-DA2 card. >>>> CPU: Intel core i7 4790K. >>>> I have recompiled UHD myself from the tag release_003_010_001_000, and >>>> added a patch to fix the timed commands (patch given by Derek) >>>> >>>> Issue #1: Can't find X300 when restarting application. >>>> When benchmark_rate process ends normally, I have to wait 5-10s before >>>> I restart it again, otherwise I get this error: >>>> Error: LookupError: KeyError: No devices found for -----> >>>> Device Address: >>>> addr: 192.168.40.2 >>>> Q: At the end of a transmission, is there call or a poll we could make >>>> in order to make sure the radio is ready for a new process? In our old UHD >>>> version, this was not happening. >>>> This issue happens on Windows and Ubuntu. >>>> >>>> >>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>> unexpected FIFO depth [0x1f]" >>>> This error happens only about 1% of the time, but I was able to >>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>> Ubuntu. >>>> See attached script "doit.sh". >>>> In my case, the error occurred 1/100 at Loop #23 (See attached result >>>> file "doit_res.txt") >>>> We have clients and partners using our application with the older >>>> version of UHD who reported this error happening about 1% of the time. The >>>> issue seems to still be present in UHD 3.10.1.0. >>>> >>>> >>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to >>>> external source. >>>> This error happened 3 times out of my 100 iteration test. >>>> See attached result file "doit_res.txt" >>>> >>>> >>>> Issue #4: Can't restart the streaming within a process >>>> I have tried to add a loop inside the source code of benchmark_rate to >>>> execute the test several times. However, I get this error message on the >>>> 2nd loop: >>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: >>>> EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>> AssertionError: buff->size() > 0 >>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const bool) >>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 >>>> >>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get >>>> a sequence error on the 2nd loop, but all samples are transmitted (no error >>>> message) >>>> See attached benchmark_rate_modified.cpp >>>> Please, let me know if there is better/cleaner way to do this "repeat >>>> loop". >>>> >>>> >>>> Thanks in advance for your help! >>>> Serge >>>> >>>> >>>> __________________ >>>> *Serge Malo * >>>> CDO & Co-founder, Skydel Solutions >>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>> www.skydelsolutions.com >>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>> >>>> _______________________________________________ >>>> USRP-users mailing list >>>> USRP-users@lists.ettus.com >>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>> >>>> >>> >> >
SM
Serge Malo
Thu, Dec 8, 2016 5:44 PM

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the problem is
fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of UHD,
I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems with
internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems with
external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have reported
issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I will run
overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got the
same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried running
the script on UHD 3.10.1.0 and the head of the maint branch without any
issue.  I am even using a rev 5 device to match the X300 hardware revision
of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these failures
    when using the internal reference?
  2. Do you see these failures on other devices or just this one device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <serge.malo@skydelsolutions.
com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but
not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate",
so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000, and
added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s before
I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could make
in order to make sure the radio is ready for a new process? In our old UHD
version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result
file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate to
execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get
a sequence error on the 2nd loop, but all samples are transmitted (no error
message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat
loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hello Michael, Here are more details and results from my side: Issue #1: I have recompiled UHD with "Head of the maint branch", and the problem is fixed under Ubuntu. Great! However, I'm facing a new issue under Windows 10 with this version of UHD, I can't connect to the X300 at all, see issue #5 below. Issues #2 and #3 -I have tried UHD 003.010.001.000 with internal clock: the problems happened again at about the same rate (issue #2 1/100; issue #3 2/100). See attached result file "doit_res2_intclk.txt" -I have tried UHD "head of maint": I did not reproduce the problems with internal clock (0/100). -I have tried UHD "head of maint": I did not reproduce the problems with external clock (0/100). -My external clock is a Wenzel OCXO -I have only tried one X300 device so far. But some clients have reported issue #2 with our older version of UHD. So, issues "seem" to be fixed with the "head of maint" version. I will run overnight tests to get more results. Issue #5: I have recompiled UHD "head of maint" under Windows 10. When I use benchmark_rate, I get the error message below. I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got the same error message. Creating the usrp device with: addr=192.168.60.2... -- X300 initialization sequence... -- Determining maximum frame size... 8000 bytes. -- Setup basic communication... -- Loading values from EEPROM... -- Setup RF frontend clocking... -- Radio 1x clock:200 -- Creating WSA UDP transport for 192.168.60.2:49153 Error: AssertionError: block_def in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class uhd::device_addr_t,enum uhd::endianness_t) at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 Do you see this problem on your side too? Thanks again, Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > I cannot seem to reproduce the issue #2 or issue #3. I have tried running > the script on UHD 3.10.1.0 and the head of the maint branch without any > issue. I am even using a rev 5 device to match the X300 hardware revision > of your device. > > There are 2 things I am curious about: > > 1) What is the external reference being used? Do you see these failures > when using the internal reference? > 2) Do you see these failures on other devices or just this one device? > > I will be looking into issue #4 tomorrow. > > Regards, > Michael > > On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <serge.malo@skydelsolutions. > com> wrote: > >> Thanks for the update Michael, >> I will try the head of the maint branch on my side too as soon as >> possible. >> >> Regards, >> Serge >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 <(514)%20294-4017> >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> wrote: >> >>> Hi Serge, >>> >>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but >>> not with the head of the maint branch. So, I believe that one has been >>> fixed. I am still working on reproducing the other 3 issues. >>> >>> Regards, >>> Michael >>> >>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> Thank you for the detailed information and examples. I am working on >>>> reproducing the issues with the information you have provided. I will let >>>> you know what I find. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>> usrp-users@lists.ettus.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> We have started the process of upgrading the UHD version in our >>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>> I have reproduced those issues with the UHD example "benchmark_rate", >>>>> so it will be easier to find solutions or workarounds. >>>>> >>>>> Configuration: >>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>> 10GbE connection with Intel X520-DA2 card. >>>>> CPU: Intel core i7 4790K. >>>>> I have recompiled UHD myself from the tag release_003_010_001_000, and >>>>> added a patch to fix the timed commands (patch given by Derek) >>>>> >>>>> Issue #1: Can't find X300 when restarting application. >>>>> When benchmark_rate process ends normally, I have to wait 5-10s before >>>>> I restart it again, otherwise I get this error: >>>>> Error: LookupError: KeyError: No devices found for -----> >>>>> Device Address: >>>>> addr: 192.168.40.2 >>>>> Q: At the end of a transmission, is there call or a poll we could make >>>>> in order to make sure the radio is ready for a new process? In our old UHD >>>>> version, this was not happening. >>>>> This issue happens on Windows and Ubuntu. >>>>> >>>>> >>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>> unexpected FIFO depth [0x1f]" >>>>> This error happens only about 1% of the time, but I was able to >>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>> Ubuntu. >>>>> See attached script "doit.sh". >>>>> In my case, the error occurred 1/100 at Loop #23 (See attached result >>>>> file "doit_res.txt") >>>>> We have clients and partners using our application with the older >>>>> version of UHD who reported this error happening about 1% of the time. The >>>>> issue seems to still be present in UHD 3.10.1.0. >>>>> >>>>> >>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to >>>>> external source. >>>>> This error happened 3 times out of my 100 iteration test. >>>>> See attached result file "doit_res.txt" >>>>> >>>>> >>>>> Issue #4: Can't restart the streaming within a process >>>>> I have tried to add a loop inside the source code of benchmark_rate to >>>>> execute the test several times. However, I get this error message on the >>>>> 2nd loop: >>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>> AssertionError: buff->size() > 0 >>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>> bool) >>>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 >>>>> >>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I get >>>>> a sequence error on the 2nd loop, but all samples are transmitted (no error >>>>> message) >>>>> See attached benchmark_rate_modified.cpp >>>>> Please, let me know if there is better/cleaner way to do this "repeat >>>>> loop". >>>>> >>>>> >>>>> Thanks in advance for your help! >>>>> Serge >>>>> >>>>> >>>>> __________________ >>>>> *Serge Malo * >>>>> CDO & Co-founder, Skydel Solutions >>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>> www.skydelsolutions.com >>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>> >>>>> _______________________________________________ >>>>> USRP-users mailing list >>>>> USRP-users@lists.ettus.com >>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>> >>>>> >>>> >>> >> >
MW
Michael West
Thu, Dec 8, 2016 10:42 PM

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2 and
#3, I am not able to debug them.  The fact that the issues disappear when
using the head of maint is encouraging, but I cannot explain why.  If you
need an answer as to why, I would need to have you ship the unit to me so I
can try to reproduce the problems and debug them here.  Please contact me
off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to find the
XML files defining the RFNoC blocks.  The files may have been installed in
the incorrect path (we are checking the Windows installers now).  If you
built and installed the 64-bit version of UHD, please check for the folder
C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program
Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the problem is
fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of UHD,
I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems with
internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems with
external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have reported
issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I will run
overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got the
same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these failures
    when using the internal reference?
  2. Do you see these failures on other devices or just this one device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but
not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example "benchmark_rate",
so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000,
and added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could
make in order to make sure the radio is ready for a new process? In our old
UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached result
file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate
to execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I
get a sequence error on the 2nd loop, but all samples are transmitted (no
error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this "repeat
loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Serge, Since I do not have a unit here that I can use to reproduce issues #2 and #3, I am not able to debug them. The fact that the issues disappear when using the head of maint is encouraging, but I cannot explain why. If you need an answer as to why, I would need to have you ship the unit to me so I can try to reproduce the problems and debug them here. Please contact me off-list if we need to do that. Regarding Issue #5, the error is produced by UHD not being able to find the XML files defining the RFNoC blocks. The files may have been installed in the incorrect path (we are checking the Windows installers now). If you built and installed the 64-bit version of UHD, please check for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program Files\UHD\share\uhd\rfnoc. Regards, Michael On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <serge.malo@skydelsolutions.com> wrote: > Hello Michael, > > Here are more details and results from my side: > > Issue #1: > I have recompiled UHD with "Head of the maint branch", and the problem is > fixed under Ubuntu. Great! > However, I'm facing a new issue under Windows 10 with this version of UHD, > I can't connect to the X300 at all, see issue #5 below. > > Issues #2 and #3 > -I have tried UHD 003.010.001.000 with internal clock: the problems > happened again at about the same rate (issue #2 1/100; issue #3 2/100). See > attached result file "doit_res2_intclk.txt" > -I have tried UHD "head of maint": I did not reproduce the problems with > internal clock (0/100). > -I have tried UHD "head of maint": I did not reproduce the problems with > external clock (0/100). > -My external clock is a Wenzel OCXO > -I have only tried one X300 device so far. But some clients have reported > issue #2 with our older version of UHD. > So, issues "seem" to be fixed with the "head of maint" version. I will run > overnight tests to get more results. > > Issue #5: > I have recompiled UHD "head of maint" under Windows 10. When I use > benchmark_rate, I get the error message below. > I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got the > same error message. > > Creating the usrp device with: addr=192.168.60.2... > -- X300 initialization sequence... > -- Determining maximum frame size... 8000 bytes. > -- Setup basic communication... > -- Loading values from EEPROM... > -- Setup RF frontend clocking... > -- Radio 1x clock:200 > -- Creating WSA UDP transport for 192.168.60.2:49153 > Error: AssertionError: block_def > in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned > __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class > uhd::device_addr_t,enum uhd::endianness_t) > at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 > > Do you see this problem on your side too? > > Thanks again, > Serge > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> wrote: > >> Hi Serge, >> >> I cannot seem to reproduce the issue #2 or issue #3. I have tried >> running the script on UHD 3.10.1.0 and the head of the maint branch without >> any issue. I am even using a rev 5 device to match the X300 hardware >> revision of your device. >> >> There are 2 things I am curious about: >> >> 1) What is the external reference being used? Do you see these failures >> when using the internal reference? >> 2) Do you see these failures on other devices or just this one device? >> >> I will be looking into issue #4 tomorrow. >> >> Regards, >> Michael >> >> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >> serge.malo@skydelsolutions.com> wrote: >> >>> Thanks for the update Michael, >>> I will try the head of the maint branch on my side too as soon as >>> possible. >>> >>> Regards, >>> Serge >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 <(514)%20294-4017> >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but >>>> not with the head of the maint branch. So, I believe that one has been >>>> fixed. I am still working on reproducing the other 3 issues. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> Thank you for the detailed information and examples. I am working on >>>>> reproducing the issues with the information you have provided. I will let >>>>> you know what I find. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>> usrp-users@lists.ettus.com> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We have started the process of upgrading the UHD version in our >>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>> I have reproduced those issues with the UHD example "benchmark_rate", >>>>>> so it will be easier to find solutions or workarounds. >>>>>> >>>>>> Configuration: >>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>> CPU: Intel core i7 4790K. >>>>>> I have recompiled UHD myself from the tag release_003_010_001_000, >>>>>> and added a patch to fix the timed commands (patch given by Derek) >>>>>> >>>>>> Issue #1: Can't find X300 when restarting application. >>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>> before I restart it again, otherwise I get this error: >>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>> Device Address: >>>>>> addr: 192.168.40.2 >>>>>> Q: At the end of a transmission, is there call or a poll we could >>>>>> make in order to make sure the radio is ready for a new process? In our old >>>>>> UHD version, this was not happening. >>>>>> This issue happens on Windows and Ubuntu. >>>>>> >>>>>> >>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>> unexpected FIFO depth [0x1f]" >>>>>> This error happens only about 1% of the time, but I was able to >>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>> Ubuntu. >>>>>> See attached script "doit.sh". >>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached result >>>>>> file "doit_res.txt") >>>>>> We have clients and partners using our application with the older >>>>>> version of UHD who reported this error happening about 1% of the time. The >>>>>> issue seems to still be present in UHD 3.10.1.0. >>>>>> >>>>>> >>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to >>>>>> external source. >>>>>> This error happened 3 times out of my 100 iteration test. >>>>>> See attached result file "doit_res.txt" >>>>>> >>>>>> >>>>>> Issue #4: Can't restart the streaming within a process >>>>>> I have tried to add a loop inside the source code of benchmark_rate >>>>>> to execute the test several times. However, I get this error message on the >>>>>> 2nd loop: >>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>> AssertionError: buff->size() > 0 >>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>> bool) >>>>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>> >>>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I >>>>>> get a sequence error on the 2nd loop, but all samples are transmitted (no >>>>>> error message) >>>>>> See attached benchmark_rate_modified.cpp >>>>>> Please, let me know if there is better/cleaner way to do this "repeat >>>>>> loop". >>>>>> >>>>>> >>>>>> Thanks in advance for your help! >>>>>> Serge >>>>>> >>>>>> >>>>>> __________________ >>>>>> *Serge Malo * >>>>>> CDO & Co-founder, Skydel Solutions >>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>> www.skydelsolutions.com >>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>> >>>>>> _______________________________________________ >>>>>> USRP-users mailing list >>>>>> USRP-users@lists.ettus.com >>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>>> >>>>>> >>>>> >>>> >>> >> >
SM
Serge Malo
Fri, Dec 9, 2016 2:05 PM

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on Windows
too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of maint"
during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you know the
results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000 executions) on
your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2 and
#3, I am not able to debug them.  The fact that the issues disappear when
using the head of maint is encouraging, but I cannot explain why.  If you
need an answer as to why, I would need to have you ship the unit to me so I
can try to reproduce the problems and debug them here.  Please contact me
off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to find
the XML files defining the RFNoC blocks.  The files may have been installed
in the incorrect path (we are checking the Windows installers now).  If you
built and installed the 64-bit version of UHD, please check for the folder
C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program
Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <serge.malo@skydelsolutions.com

wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the problem
is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of
UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems with
internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems with
external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have reported
issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I will
run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got
the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but
not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working on
reproducing the issues with the information you have provided.  I will let
you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000,
and added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could
make in order to make sure the radio is ready for a new process? In our old
UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to
external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate
to execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp
:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I
get a sequence error on the 2nd loop, but all samples are transmitted (no
error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hello Michael, Issue #5: My bad! this is not an issue, I can use "head of maint" on Windows now. Issue #1: I confirm this is fixed with the "head of maint" UHD version on Windows too. Great! Issues #2 and #3: Unfortunately, I was able to reproduce the problem with "head of maint" during my overnight on Ubuntu: Issue #2 happened 4 times out of 948 executions Issue #3 happened 9 times out of 948 executions But under Windows 10, I had the next results: Issue #2 happened 0 time out of 520 executions Issue #3 happened 1 time out of 520 executions I will try to reproduce the problem with another X300 and let you know the results. I will also try to see if the issue #3 occurs with an Octoclock-G. Can you confirm you don't see this problem at all ( 0 / 1000 executions) on your side (this test can be done over one night). Issue #4 The problem still happens with UHD "head of maint" under Windows. Were you able to reproduce this on your side? Thanks again, Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > Since I do not have a unit here that I can use to reproduce issues #2 and > #3, I am not able to debug them. The fact that the issues disappear when > using the head of maint is encouraging, but I cannot explain why. If you > need an answer as to why, I would need to have you ship the unit to me so I > can try to reproduce the problems and debug them here. Please contact me > off-list if we need to do that. > > Regarding Issue #5, the error is produced by UHD not being able to find > the XML files defining the RFNoC blocks. The files may have been installed > in the incorrect path (we are checking the Windows installers now). If you > built and installed the 64-bit version of UHD, please check for the folder > C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program > Files\UHD\share\uhd\rfnoc. > > Regards, > Michael > > On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <serge.malo@skydelsolutions.com > > wrote: > >> Hello Michael, >> >> Here are more details and results from my side: >> >> Issue #1: >> I have recompiled UHD with "Head of the maint branch", and the problem >> is fixed under Ubuntu. Great! >> However, I'm facing a new issue under Windows 10 with this version of >> UHD, I can't connect to the X300 at all, see issue #5 below. >> >> Issues #2 and #3 >> -I have tried UHD 003.010.001.000 with internal clock: the problems >> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >> attached result file "doit_res2_intclk.txt" >> -I have tried UHD "head of maint": I did not reproduce the problems with >> internal clock (0/100). >> -I have tried UHD "head of maint": I did not reproduce the problems with >> external clock (0/100). >> -My external clock is a Wenzel OCXO >> -I have only tried one X300 device so far. But some clients have reported >> issue #2 with our older version of UHD. >> So, issues "seem" to be fixed with the "head of maint" version. I will >> run overnight tests to get more results. >> >> Issue #5: >> I have recompiled UHD "head of maint" under Windows 10. When I use >> benchmark_rate, I get the error message below. >> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got >> the same error message. >> >> Creating the usrp device with: addr=192.168.60.2... >> -- X300 initialization sequence... >> -- Determining maximum frame size... 8000 bytes. >> -- Setup basic communication... >> -- Loading values from EEPROM... >> -- Setup RF frontend clocking... >> -- Radio 1x clock:200 >> -- Creating WSA UDP transport for 192.168.60.2:49153 >> Error: AssertionError: block_def >> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >> uhd::device_addr_t,enum uhd::endianness_t) >> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >> >> Do you see this problem on your side too? >> >> Thanks again, >> Serge >> >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 <(514)%20294-4017> >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> wrote: >> >>> Hi Serge, >>> >>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>> any issue. I am even using a rev 5 device to match the X300 hardware >>> revision of your device. >>> >>> There are 2 things I am curious about: >>> >>> 1) What is the external reference being used? Do you see these >>> failures when using the internal reference? >>> 2) Do you see these failures on other devices or just this one device? >>> >>> I will be looking into issue #4 tomorrow. >>> >>> Regards, >>> Michael >>> >>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>> serge.malo@skydelsolutions.com> wrote: >>> >>>> Thanks for the update Michael, >>>> I will try the head of the maint branch on my side too as soon as >>>> possible. >>>> >>>> Regards, >>>> Serge >>>> >>>> __________________ >>>> *Serge Malo * >>>> CDO & Co-founder, Skydel Solutions >>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>> www.skydelsolutions.com >>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>> >>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, but >>>>> not with the head of the maint branch. So, I believe that one has been >>>>> fixed. I am still working on reproducing the other 3 issues. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> >>>>> wrote: >>>>> >>>>>> Hi Serge, >>>>>> >>>>>> Thank you for the detailed information and examples. I am working on >>>>>> reproducing the issues with the information you have provided. I will let >>>>>> you know what I find. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>> usrp-users@lists.ettus.com> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We have started the process of upgrading the UHD version in our >>>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>>> I have reproduced those issues with the UHD example >>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>> >>>>>>> Configuration: >>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>> CPU: Intel core i7 4790K. >>>>>>> I have recompiled UHD myself from the tag release_003_010_001_000, >>>>>>> and added a patch to fix the timed commands (patch given by Derek) >>>>>>> >>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>>> before I restart it again, otherwise I get this error: >>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>> Device Address: >>>>>>> addr: 192.168.40.2 >>>>>>> Q: At the end of a transmission, is there call or a poll we could >>>>>>> make in order to make sure the radio is ready for a new process? In our old >>>>>>> UHD version, this was not happening. >>>>>>> This issue happens on Windows and Ubuntu. >>>>>>> >>>>>>> >>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>> unexpected FIFO depth [0x1f]" >>>>>>> This error happens only about 1% of the time, but I was able to >>>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>>> Ubuntu. >>>>>>> See attached script "doit.sh". >>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>> result file "doit_res.txt") >>>>>>> We have clients and partners using our application with the older >>>>>>> version of UHD who reported this error happening about 1% of the time. The >>>>>>> issue seems to still be present in UHD 3.10.1.0. >>>>>>> >>>>>>> >>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock to >>>>>>> external source. >>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>> See attached result file "doit_res.txt" >>>>>>> >>>>>>> >>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>> I have tried to add a loop inside the source code of benchmark_rate >>>>>>> to execute the test several times. However, I get this error message on the >>>>>>> 2nd loop: >>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>>> AssertionError: buff->size() > 0 >>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>> bool) >>>>>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp >>>>>>> :206 >>>>>>> >>>>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I >>>>>>> get a sequence error on the 2nd loop, but all samples are transmitted (no >>>>>>> error message) >>>>>>> See attached benchmark_rate_modified.cpp >>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>> "repeat loop". >>>>>>> >>>>>>> >>>>>>> Thanks in advance for your help! >>>>>>> Serge >>>>>>> >>>>>>> >>>>>>> __________________ >>>>>>> *Serge Malo * >>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>> www.skydelsolutions.com >>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> USRP-users mailing list >>>>>>> USRP-users@lists.ettus.com >>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
SM
Serge Malo
Mon, Dec 12, 2016 6:55 PM

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did not
show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and see
with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we will
put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows, and the
problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will workaround the
problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD" version. Neel
once told us this was not recommended, but its the only way I found to be
able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old UHD"
version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, version
of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994 samples,
but now it returns only 364 samples. I still see "Determining frame size...
8000 bytes" in UHD's output. Is this expected? I see the same value (364
samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on Windows
too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of maint"
during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you know the
results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000 executions)
on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2 and
#3, I am not able to debug them.  The fact that the issues disappear when
using the head of maint is encouraging, but I cannot explain why.  If you
need an answer as to why, I would need to have you ship the unit to me so I
can try to reproduce the problems and debug them here.  Please contact me
off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to find
the XML files defining the RFNoC blocks.  The files may have been installed
in the incorrect path (we are checking the Windows installers now).  If you
built and installed the 64-bit version of UHD, please check for the folder
C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program
Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <serge.malo@skydelsolutions.co
m> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the problem
is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of
UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems with
internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems with
external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I will
run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got
the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release,
but not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working
on reproducing the issues with the information you have provided.  I will
let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000,
and added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could
make in order to make sure the radio is ready for a new process? In our old
UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock
to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of benchmark_rate
to execute the test several times. However, I get this error message on the
2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp
:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I
get a sequence error on the 2nd loop, but all samples are transmitted (no
error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hello Michael, I have some good news today, and 2 outstanding questions at the end. Issue #2: we have tested with 2 other X300 under Windows, and the problem did not show up. So, on our side, the problem only happens 0.5% of the time, with one device, under Ubuntu. We will implement a "retry" mechanism to workaround this problem, and see with our clients if this is Ok with them. If they have a X300 with which the issue happens more frequently, we will put them in touch with you. Issue #3 We have tested with 2 other X300 and an octoclock-G under Windows, and the problem showed up 4/1000 for one device and 0/1000 for the other. Again, the problem happens less than 1% of time, so we will workaround the problem by implementing a retry mechanism. Issue #4 I have found a workaround that seems functional: When I wish to restart the streaming, I simply delete (reset) the multi_usrp object and redo the complete initialization. In fact, this is the workaround I was using with my "old UHD" version. Neel once told us this was not recommended, but its the only way I found to be able to restart the streaming without quitting the application. The original issue is still reproducible 100% of the time with benchmark_rate under Windows. Issue #6 (We had discussed this earlier): Spectrum deformation with the "old UHD" version. I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, version of December 1st 2016). Question A: The function "tx_streamer::get_max_num_samps" used to return 1994 samples, but now it returns only 364 samples. I still see "Determining frame size... 8000 bytes" in UHD's output. Is this expected? I see the same value (364 samples) with benchmark_rate. Question B: Creating the mutli_usrp object now takes 15-20 seconds, while it took around 10s with the "old UHD" version I had. Is there a way to reduce this initialization time? Thanks again for your help! Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.com> wrote: > Hello Michael, > > > Issue #5: > My bad! this is not an issue, I can use "head of maint" on Windows now. > > Issue #1: > I confirm this is fixed with the "head of maint" UHD version on Windows > too. Great! > > Issues #2 and #3: > Unfortunately, I was able to reproduce the problem with "head of maint" > during my overnight on Ubuntu: > Issue #2 happened 4 times out of 948 executions > Issue #3 happened 9 times out of 948 executions > But under Windows 10, I had the next results: > Issue #2 happened 0 time out of 520 executions > Issue #3 happened 1 time out of 520 executions > I will try to reproduce the problem with another X300 and let you know the > results. > I will also try to see if the issue #3 occurs with an Octoclock-G. > Can you confirm you don't see this problem at all ( 0 / 1000 executions) > on your side (this test can be done over one night). > > Issue #4 > The problem still happens with UHD "head of maint" under Windows. > Were you able to reproduce this on your side? > > Thanks again, > Serge > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> wrote: > >> Hi Serge, >> >> Since I do not have a unit here that I can use to reproduce issues #2 and >> #3, I am not able to debug them. The fact that the issues disappear when >> using the head of maint is encouraging, but I cannot explain why. If you >> need an answer as to why, I would need to have you ship the unit to me so I >> can try to reproduce the problems and debug them here. Please contact me >> off-list if we need to do that. >> >> Regarding Issue #5, the error is produced by UHD not being able to find >> the XML files defining the RFNoC blocks. The files may have been installed >> in the incorrect path (we are checking the Windows installers now). If you >> built and installed the 64-bit version of UHD, please check for the folder >> C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program >> Files\UHD\share\uhd\rfnoc. >> >> Regards, >> Michael >> >> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <serge.malo@skydelsolutions.co >> m> wrote: >> >>> Hello Michael, >>> >>> Here are more details and results from my side: >>> >>> Issue #1: >>> I have recompiled UHD with "Head of the maint branch", and the problem >>> is fixed under Ubuntu. Great! >>> However, I'm facing a new issue under Windows 10 with this version of >>> UHD, I can't connect to the X300 at all, see issue #5 below. >>> >>> Issues #2 and #3 >>> -I have tried UHD 003.010.001.000 with internal clock: the problems >>> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >>> attached result file "doit_res2_intclk.txt" >>> -I have tried UHD "head of maint": I did not reproduce the problems with >>> internal clock (0/100). >>> -I have tried UHD "head of maint": I did not reproduce the problems with >>> external clock (0/100). >>> -My external clock is a Wenzel OCXO >>> -I have only tried one X300 device so far. But some clients have >>> reported issue #2 with our older version of UHD. >>> So, issues "seem" to be fixed with the "head of maint" version. I will >>> run overnight tests to get more results. >>> >>> Issue #5: >>> I have recompiled UHD "head of maint" under Windows 10. When I use >>> benchmark_rate, I get the error message below. >>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got >>> the same error message. >>> >>> Creating the usrp device with: addr=192.168.60.2... >>> -- X300 initialization sequence... >>> -- Determining maximum frame size... 8000 bytes. >>> -- Setup basic communication... >>> -- Loading values from EEPROM... >>> -- Setup RF frontend clocking... >>> -- Radio 1x clock:200 >>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>> Error: AssertionError: block_def >>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>> uhd::device_addr_t,enum uhd::endianness_t) >>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>> >>> Do you see this problem on your side too? >>> >>> Thanks again, >>> Serge >>> >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 <(514)%20294-4017> >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>>> any issue. I am even using a rev 5 device to match the X300 hardware >>>> revision of your device. >>>> >>>> There are 2 things I am curious about: >>>> >>>> 1) What is the external reference being used? Do you see these >>>> failures when using the internal reference? >>>> 2) Do you see these failures on other devices or just this one device? >>>> >>>> I will be looking into issue #4 tomorrow. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>> serge.malo@skydelsolutions.com> wrote: >>>> >>>>> Thanks for the update Michael, >>>>> I will try the head of the maint branch on my side too as soon as >>>>> possible. >>>>> >>>>> Regards, >>>>> Serge >>>>> >>>>> __________________ >>>>> *Serge Malo * >>>>> CDO & Co-founder, Skydel Solutions >>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>> www.skydelsolutions.com >>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>> >>>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>>>> wrote: >>>>> >>>>>> Hi Serge, >>>>>> >>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, >>>>>> but not with the head of the maint branch. So, I believe that one has been >>>>>> fixed. I am still working on reproducing the other 3 issues. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Serge, >>>>>>> >>>>>>> Thank you for the detailed information and examples. I am working >>>>>>> on reproducing the issues with the information you have provided. I will >>>>>>> let you know what I find. >>>>>>> >>>>>>> Regards, >>>>>>> Michael >>>>>>> >>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> We have started the process of upgrading the UHD version in our >>>>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>> I have reproduced those issues with the UHD example >>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>> >>>>>>>> Configuration: >>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>> CPU: Intel core i7 4790K. >>>>>>>> I have recompiled UHD myself from the tag release_003_010_001_000, >>>>>>>> and added a patch to fix the timed commands (patch given by Derek) >>>>>>>> >>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>>>> before I restart it again, otherwise I get this error: >>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>> Device Address: >>>>>>>> addr: 192.168.40.2 >>>>>>>> Q: At the end of a transmission, is there call or a poll we could >>>>>>>> make in order to make sure the radio is ready for a new process? In our old >>>>>>>> UHD version, this was not happening. >>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>> >>>>>>>> >>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>>> unexpected FIFO depth [0x1f]" >>>>>>>> This error happens only about 1% of the time, but I was able to >>>>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>>>> Ubuntu. >>>>>>>> See attached script "doit.sh". >>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>>> result file "doit_res.txt") >>>>>>>> We have clients and partners using our application with the older >>>>>>>> version of UHD who reported this error happening about 1% of the time. The >>>>>>>> issue seems to still be present in UHD 3.10.1.0. >>>>>>>> >>>>>>>> >>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock >>>>>>>> to external source. >>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>> See attached result file "doit_res.txt" >>>>>>>> >>>>>>>> >>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>> I have tried to add a loop inside the source code of benchmark_rate >>>>>>>> to execute the test several times. However, I get this error message on the >>>>>>>> 2nd loop: >>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>>>> AssertionError: buff->size() > 0 >>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>> bool) >>>>>>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp >>>>>>>> :206 >>>>>>>> >>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I >>>>>>>> get a sequence error on the 2nd loop, but all samples are transmitted (no >>>>>>>> error message) >>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>> "repeat loop". >>>>>>>> >>>>>>>> >>>>>>>> Thanks in advance for your help! >>>>>>>> Serge >>>>>>>> >>>>>>>> >>>>>>>> __________________ >>>>>>>> *Serge Malo * >>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>> www.skydelsolutions.com >>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> USRP-users mailing list >>>>>>>> USRP-users@lists.ettus.com >>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
MW
Michael West
Mon, Dec 12, 2016 7:10 PM

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently being
addressed.  The maximum number of samples issue should be fixed in the next
release of UHD.  The slower initialization is being looked at, but may take
more time to completely resolve.  Please let me know if it is significantly
impacting your application and I can see what I can do to elevate the
priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <serge.malo@skydelsolutions.com

wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did not
show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and see
with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we will
put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows, and the
problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will workaround the
problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD" version.
Neel once told us this was not recommended, but its the only way I found to
be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old UHD"
version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on Windows
too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of maint"
during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you know
the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000 executions)
on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2
and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to find
the XML files defining the RFNoC blocks.  The files may have been installed
in the incorrect path (we are checking the Windows installers now).  If you
built and installed the 64-bit version of UHD, please check for the folder
C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program
Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the problem
is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of
UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems
with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems
with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I will
run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got
the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release,
but not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com

wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working
on reproducing the issues with the information you have provided.  I will
let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag release_003_010_001_000,
and added a patch to fix the timed commands (patch given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could
make in order to make sure the radio is ready for a new process? In our old
UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock
to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp
:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I
get a sequence error on the 2nd loop, but all samples are transmitted (no
error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

Hi Serge, I will set up longer term tests for issues #2 and #3 to see if I can reproduce and debug on my end. Issue #4 seems like a legitimate bug, so let me spend some time digging into that this week. Regarding your questions, they are known issues that are currently being addressed. The maximum number of samples issue should be fixed in the next release of UHD. The slower initialization is being looked at, but may take more time to completely resolve. Please let me know if it is significantly impacting your application and I can see what I can do to elevate the priority. Regards, Michael On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <serge.malo@skydelsolutions.com > wrote: > Hello Michael, > > I have some good news today, and 2 outstanding questions at the end. > > Issue #2: > we have tested with 2 other X300 under Windows, and the problem did not > show up. > So, on our side, the problem only happens 0.5% of the time, with one > device, under Ubuntu. > We will implement a "retry" mechanism to workaround this problem, and see > with our clients if this is Ok with them. > If they have a X300 with which the issue happens more frequently, we will > put them in touch with you. > > Issue #3 > We have tested with 2 other X300 and an octoclock-G under Windows, and the > problem showed up 4/1000 for one device and 0/1000 for the other. > Again, the problem happens less than 1% of time, so we will workaround the > problem by implementing a retry mechanism. > > Issue #4 > I have found a workaround that seems functional: > When I wish to restart the streaming, I simply delete (reset) the > multi_usrp object and redo the complete initialization. > In fact, this is the workaround I was using with my "old UHD" version. > Neel once told us this was not recommended, but its the only way I found to > be able to restart the streaming without quitting the application. > The original issue is still reproducible 100% of the time with > benchmark_rate under Windows. > > Issue #6 > (We had discussed this earlier): Spectrum deformation with the "old UHD" > version. > I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, > version of December 1st 2016). > > Question A: > The function "tx_streamer::get_max_num_samps" used to return 1994 > samples, but now it returns only 364 samples. I still see "Determining > frame size... 8000 bytes" in UHD's output. Is this expected? I see the same > value (364 samples) with benchmark_rate. > > Question B: > Creating the mutli_usrp object now takes 15-20 seconds, while it took > around 10s with the "old UHD" version I had. Is there a way to reduce this > initialization time? > > Thanks again for your help! > Serge > > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.com> > wrote: > >> Hello Michael, >> >> >> Issue #5: >> My bad! this is not an issue, I can use "head of maint" on Windows now. >> >> Issue #1: >> I confirm this is fixed with the "head of maint" UHD version on Windows >> too. Great! >> >> Issues #2 and #3: >> Unfortunately, I was able to reproduce the problem with "head of maint" >> during my overnight on Ubuntu: >> Issue #2 happened 4 times out of 948 executions >> Issue #3 happened 9 times out of 948 executions >> But under Windows 10, I had the next results: >> Issue #2 happened 0 time out of 520 executions >> Issue #3 happened 1 time out of 520 executions >> I will try to reproduce the problem with another X300 and let you know >> the results. >> I will also try to see if the issue #3 occurs with an Octoclock-G. >> Can you confirm you don't see this problem at all ( 0 / 1000 executions) >> on your side (this test can be done over one night). >> >> Issue #4 >> The problem still happens with UHD "head of maint" under Windows. >> Were you able to reproduce this on your side? >> >> Thanks again, >> Serge >> >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> wrote: >> >>> Hi Serge, >>> >>> Since I do not have a unit here that I can use to reproduce issues #2 >>> and #3, I am not able to debug them. The fact that the issues disappear >>> when using the head of maint is encouraging, but I cannot explain why. If >>> you need an answer as to why, I would need to have you ship the unit to me >>> so I can try to reproduce the problems and debug them here. Please contact >>> me off-list if we need to do that. >>> >>> Regarding Issue #5, the error is produced by UHD not being able to find >>> the XML files defining the RFNoC blocks. The files may have been installed >>> in the incorrect path (we are checking the Windows installers now). If you >>> built and installed the 64-bit version of UHD, please check for the folder >>> C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program >>> Files\UHD\share\uhd\rfnoc. >>> >>> Regards, >>> Michael >>> >>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>> serge.malo@skydelsolutions.com> wrote: >>> >>>> Hello Michael, >>>> >>>> Here are more details and results from my side: >>>> >>>> Issue #1: >>>> I have recompiled UHD with "Head of the maint branch", and the problem >>>> is fixed under Ubuntu. Great! >>>> However, I'm facing a new issue under Windows 10 with this version of >>>> UHD, I can't connect to the X300 at all, see issue #5 below. >>>> >>>> Issues #2 and #3 >>>> -I have tried UHD 003.010.001.000 with internal clock: the problems >>>> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >>>> attached result file "doit_res2_intclk.txt" >>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>> with internal clock (0/100). >>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>> with external clock (0/100). >>>> -My external clock is a Wenzel OCXO >>>> -I have only tried one X300 device so far. But some clients have >>>> reported issue #2 with our older version of UHD. >>>> So, issues "seem" to be fixed with the "head of maint" version. I will >>>> run overnight tests to get more results. >>>> >>>> Issue #5: >>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>> benchmark_rate, I get the error message below. >>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got >>>> the same error message. >>>> >>>> Creating the usrp device with: addr=192.168.60.2... >>>> -- X300 initialization sequence... >>>> -- Determining maximum frame size... 8000 bytes. >>>> -- Setup basic communication... >>>> -- Loading values from EEPROM... >>>> -- Setup RF frontend clocking... >>>> -- Radio 1x clock:200 >>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>> Error: AssertionError: block_def >>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>> uhd::device_addr_t,enum uhd::endianness_t) >>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>> >>>> Do you see this problem on your side too? >>>> >>>> Thanks again, >>>> Serge >>>> >>>> >>>> __________________ >>>> *Serge Malo * >>>> CDO & Co-founder, Skydel Solutions >>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>> www.skydelsolutions.com >>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>> >>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>>>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>>>> any issue. I am even using a rev 5 device to match the X300 hardware >>>>> revision of your device. >>>>> >>>>> There are 2 things I am curious about: >>>>> >>>>> 1) What is the external reference being used? Do you see these >>>>> failures when using the internal reference? >>>>> 2) Do you see these failures on other devices or just this one device? >>>>> >>>>> I will be looking into issue #4 tomorrow. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>> serge.malo@skydelsolutions.com> wrote: >>>>> >>>>>> Thanks for the update Michael, >>>>>> I will try the head of the maint branch on my side too as soon as >>>>>> possible. >>>>>> >>>>>> Regards, >>>>>> Serge >>>>>> >>>>>> __________________ >>>>>> *Serge Malo * >>>>>> CDO & Co-founder, Skydel Solutions >>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>> www.skydelsolutions.com >>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>> >>>>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Serge, >>>>>>> >>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, >>>>>>> but not with the head of the maint branch. So, I believe that one has been >>>>>>> fixed. I am still working on reproducing the other 3 issues. >>>>>>> >>>>>>> Regards, >>>>>>> Michael >>>>>>> >>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West <michael.west@ettus.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Serge, >>>>>>>> >>>>>>>> Thank you for the detailed information and examples. I am working >>>>>>>> on reproducing the issues with the information you have provided. I will >>>>>>>> let you know what I find. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> We have started the process of upgrading the UHD version in our >>>>>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>> >>>>>>>>> Configuration: >>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>> I have recompiled UHD myself from the tag release_003_010_001_000, >>>>>>>>> and added a patch to fix the timed commands (patch given by Derek) >>>>>>>>> >>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>>>>> before I restart it again, otherwise I get this error: >>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>> Device Address: >>>>>>>>> addr: 192.168.40.2 >>>>>>>>> Q: At the end of a transmission, is there call or a poll we could >>>>>>>>> make in order to make sure the radio is ready for a new process? In our old >>>>>>>>> UHD version, this was not happening. >>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>> >>>>>>>>> >>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>>>> unexpected FIFO depth [0x1f]" >>>>>>>>> This error happens only about 1% of the time, but I was able to >>>>>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>>>>> Ubuntu. >>>>>>>>> See attached script "doit.sh". >>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>>>> result file "doit_res.txt") >>>>>>>>> We have clients and partners using our application with the older >>>>>>>>> version of UHD who reported this error happening about 1% of the time. The >>>>>>>>> issue seems to still be present in UHD 3.10.1.0. >>>>>>>>> >>>>>>>>> >>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock >>>>>>>>> to external source. >>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>> >>>>>>>>> >>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>> message on the 2nd loop: >>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>>>>> AssertionError: buff->size() > 0 >>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>> bool) >>>>>>>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp >>>>>>>>> :206 >>>>>>>>> >>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, I >>>>>>>>> get a sequence error on the 2nd loop, but all samples are transmitted (no >>>>>>>>> error message) >>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>> "repeat loop". >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks in advance for your help! >>>>>>>>> Serge >>>>>>>>> >>>>>>>>> >>>>>>>>> __________________ >>>>>>>>> *Serge Malo * >>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>> www.skydelsolutions.com >>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> USRP-users mailing list >>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
MW
Michael West
Mon, Dec 12, 2016 11:07 PM

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is expected.  I
believe the failure on Windows may be due to the default socket buffer size
being too small.  Try increasing the default receive socket buffer size (to
2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently being
addressed.  The maximum number of samples issue should be fixed in the next
release of UHD.  The slower initialization is being looked at, but may take
more time to completely resolve.  Please let me know if it is significantly
impacting your application and I can see what I can do to elevate the
priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <serge.malo@skydelsolutions.
com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did not
show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and see
with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we will
put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows, and
the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will workaround
the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD" version.
Neel once told us this was not recommended, but its the only way I found to
be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old UHD"
version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on Windows
too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of maint"
during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you know
the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000 executions)
on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2
and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to find
the XML files defining the RFNoC blocks.  The files may have been installed
in the incorrect path (we are checking the Windows installers now).  If you
built and installed the 64-bit version of UHD, please check for the folder
C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program
Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of
UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems
with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems
with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I will
run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got
the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release,
but not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am working
on reproducing the issues with the information you have provided.  I will
let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we could
make in order to make sure the radio is ready for a new process? In our old
UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the older
version of UHD who reported this error happening about 1% of the time. The
issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock
to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp
:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu,
I get a sequence error on the 2nd loop, but all samples are transmitted (no
error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hi Serge, Regarding Issue #4, the sequence error is a known issue and is expected. I believe the failure on Windows may be due to the default socket buffer size being too small. Try increasing the default receive socket buffer size (to 2MB) and let me know if you still see the failures. Regards, Michael On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > I will set up longer term tests for issues #2 and #3 to see if I can > reproduce and debug on my end. Issue #4 seems like a legitimate bug, so > let me spend some time digging into that this week. > > Regarding your questions, they are known issues that are currently being > addressed. The maximum number of samples issue should be fixed in the next > release of UHD. The slower initialization is being looked at, but may take > more time to completely resolve. Please let me know if it is significantly > impacting your application and I can see what I can do to elevate the > priority. > > Regards, > Michael > > On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <serge.malo@skydelsolutions. > com> wrote: > >> Hello Michael, >> >> I have some good news today, and 2 outstanding questions at the end. >> >> Issue #2: >> we have tested with 2 other X300 under Windows, and the problem did not >> show up. >> So, on our side, the problem only happens 0.5% of the time, with one >> device, under Ubuntu. >> We will implement a "retry" mechanism to workaround this problem, and see >> with our clients if this is Ok with them. >> If they have a X300 with which the issue happens more frequently, we will >> put them in touch with you. >> >> Issue #3 >> We have tested with 2 other X300 and an octoclock-G under Windows, and >> the problem showed up 4/1000 for one device and 0/1000 for the other. >> Again, the problem happens less than 1% of time, so we will workaround >> the problem by implementing a retry mechanism. >> >> Issue #4 >> I have found a workaround that seems functional: >> When I wish to restart the streaming, I simply delete (reset) the >> multi_usrp object and redo the complete initialization. >> In fact, this is the workaround I was using with my "old UHD" version. >> Neel once told us this was not recommended, but its the only way I found to >> be able to restart the streaming without quitting the application. >> The original issue is still reproducible 100% of the time with >> benchmark_rate under Windows. >> >> Issue #6 >> (We had discussed this earlier): Spectrum deformation with the "old UHD" >> version. >> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >> version of December 1st 2016). >> >> Question A: >> The function "tx_streamer::get_max_num_samps" used to return 1994 >> samples, but now it returns only 364 samples. I still see "Determining >> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >> value (364 samples) with benchmark_rate. >> >> Question B: >> Creating the mutli_usrp object now takes 15-20 seconds, while it took >> around 10s with the "old UHD" version I had. Is there a way to reduce this >> initialization time? >> >> Thanks again for your help! >> Serge >> >> >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 <(514)%20294-4017> >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.com> >> wrote: >> >>> Hello Michael, >>> >>> >>> Issue #5: >>> My bad! this is not an issue, I can use "head of maint" on Windows now. >>> >>> Issue #1: >>> I confirm this is fixed with the "head of maint" UHD version on Windows >>> too. Great! >>> >>> Issues #2 and #3: >>> Unfortunately, I was able to reproduce the problem with "head of maint" >>> during my overnight on Ubuntu: >>> Issue #2 happened 4 times out of 948 executions >>> Issue #3 happened 9 times out of 948 executions >>> But under Windows 10, I had the next results: >>> Issue #2 happened 0 time out of 520 executions >>> Issue #3 happened 1 time out of 520 executions >>> I will try to reproduce the problem with another X300 and let you know >>> the results. >>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>> Can you confirm you don't see this problem at all ( 0 / 1000 executions) >>> on your side (this test can be done over one night). >>> >>> Issue #4 >>> The problem still happens with UHD "head of maint" under Windows. >>> Were you able to reproduce this on your side? >>> >>> Thanks again, >>> Serge >>> >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> Since I do not have a unit here that I can use to reproduce issues #2 >>>> and #3, I am not able to debug them. The fact that the issues disappear >>>> when using the head of maint is encouraging, but I cannot explain why. If >>>> you need an answer as to why, I would need to have you ship the unit to me >>>> so I can try to reproduce the problems and debug them here. Please contact >>>> me off-list if we need to do that. >>>> >>>> Regarding Issue #5, the error is produced by UHD not being able to find >>>> the XML files defining the RFNoC blocks. The files may have been installed >>>> in the incorrect path (we are checking the Windows installers now). If you >>>> built and installed the 64-bit version of UHD, please check for the folder >>>> C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to C:\Program >>>> Files\UHD\share\uhd\rfnoc. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>> serge.malo@skydelsolutions.com> wrote: >>>> >>>>> Hello Michael, >>>>> >>>>> Here are more details and results from my side: >>>>> >>>>> Issue #1: >>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>> problem is fixed under Ubuntu. Great! >>>>> However, I'm facing a new issue under Windows 10 with this version of >>>>> UHD, I can't connect to the X300 at all, see issue #5 below. >>>>> >>>>> Issues #2 and #3 >>>>> -I have tried UHD 003.010.001.000 with internal clock: the problems >>>>> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >>>>> attached result file "doit_res2_intclk.txt" >>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>> with internal clock (0/100). >>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>> with external clock (0/100). >>>>> -My external clock is a Wenzel OCXO >>>>> -I have only tried one X300 device so far. But some clients have >>>>> reported issue #2 with our older version of UHD. >>>>> So, issues "seem" to be fixed with the "head of maint" version. I will >>>>> run overnight tests to get more results. >>>>> >>>>> Issue #5: >>>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>>> benchmark_rate, I get the error message below. >>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I got >>>>> the same error message. >>>>> >>>>> Creating the usrp device with: addr=192.168.60.2... >>>>> -- X300 initialization sequence... >>>>> -- Determining maximum frame size... 8000 bytes. >>>>> -- Setup basic communication... >>>>> -- Loading values from EEPROM... >>>>> -- Setup RF frontend clocking... >>>>> -- Radio 1x clock:200 >>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>> Error: AssertionError: block_def >>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>> >>>>> Do you see this problem on your side too? >>>>> >>>>> Thanks again, >>>>> Serge >>>>> >>>>> >>>>> __________________ >>>>> *Serge Malo * >>>>> CDO & Co-founder, Skydel Solutions >>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>> www.skydelsolutions.com >>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>> >>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>>> wrote: >>>>> >>>>>> Hi Serge, >>>>>> >>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>>>>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>>>>> any issue. I am even using a rev 5 device to match the X300 hardware >>>>>> revision of your device. >>>>>> >>>>>> There are 2 things I am curious about: >>>>>> >>>>>> 1) What is the external reference being used? Do you see these >>>>>> failures when using the internal reference? >>>>>> 2) Do you see these failures on other devices or just this one >>>>>> device? >>>>>> >>>>>> I will be looking into issue #4 tomorrow. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>> >>>>>>> Thanks for the update Michael, >>>>>>> I will try the head of the maint branch on my side too as soon as >>>>>>> possible. >>>>>>> >>>>>>> Regards, >>>>>>> Serge >>>>>>> >>>>>>> __________________ >>>>>>> *Serge Malo * >>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>> www.skydelsolutions.com >>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>> >>>>>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Serge, >>>>>>>> >>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, >>>>>>>> but not with the head of the maint branch. So, I believe that one has been >>>>>>>> fixed. I am still working on reproducing the other 3 issues. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>> michael.west@ettus.com> wrote: >>>>>>>> >>>>>>>>> Hi Serge, >>>>>>>>> >>>>>>>>> Thank you for the detailed information and examples. I am working >>>>>>>>> on reproducing the issues with the information you have provided. I will >>>>>>>>> let you know what I find. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Michael >>>>>>>>> >>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> We have started the process of upgrading the UHD version in our >>>>>>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>> >>>>>>>>>> Configuration: >>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>> given by Derek) >>>>>>>>>> >>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>>>>>> before I restart it again, otherwise I get this error: >>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>> Device Address: >>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>> Q: At the end of a transmission, is there call or a poll we could >>>>>>>>>> make in order to make sure the radio is ready for a new process? In our old >>>>>>>>>> UHD version, this was not happening. >>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>>>>> unexpected FIFO depth [0x1f]" >>>>>>>>>> This error happens only about 1% of the time, but I was able to >>>>>>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>>>>>> Ubuntu. >>>>>>>>>> See attached script "doit.sh". >>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>>>>> result file "doit_res.txt") >>>>>>>>>> We have clients and partners using our application with the older >>>>>>>>>> version of UHD who reported this error happening about 1% of the time. The >>>>>>>>>> issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to lock >>>>>>>>>> to external source. >>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>> message on the 2nd loop: >>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>>>>>> AssertionError: buff->size() > 0 >>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>> bool) >>>>>>>>>> at ...\uhd_003_010_001_000-Patch1\host\lib\rfnoc\ctrl_iface.cpp >>>>>>>>>> :206 >>>>>>>>>> >>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, >>>>>>>>>> I get a sequence error on the 2nd loop, but all samples are transmitted (no >>>>>>>>>> error message) >>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>>> "repeat loop". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks in advance for your help! >>>>>>>>>> Serge >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> __________________ >>>>>>>>>> *Serge Malo * >>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>> www.skydelsolutions.com >>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> USRP-users mailing list >>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>> us.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
SM
Serge Malo
Wed, Dec 14, 2016 3:47 PM

Hello Michael,

Issue #4:
I have tried to increase the socket buffer size (with the send_buff_size
argument), but I'm getting the next error (see complete output attached):
UU
UHD Error:
x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out

Issue "Maximum number of samples":
I'm worried that this will negatively impact the performance of the Tx
streaming (causing more frequent under-runs), but I have not tested enough
yet.
You said it should be available in the next UHD release. Let me know if
this could be fixed in the "maint" branch, so I could pull this earlier.
Do you have a timeframe for the next UHD version?

Issue "Initialization time":
This is not a bug issue for us, but any improvement in the next UHD release
will be appreciated.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 12 December 2016 at 18:07, Michael West michael.west@ettus.com wrote:

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is expected.
I believe the failure on Windows may be due to the default socket buffer
size being too small.  Try increasing the default receive socket buffer
size (to 2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently being
addressed.  The maximum number of samples issue should be fixed in the next
release of UHD.  The slower initialization is being looked at, but may take
more time to completely resolve.  Please let me know if it is significantly
impacting your application and I can see what I can do to elevate the
priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did not
show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and
see with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we
will put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows, and
the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will workaround
the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD" version.
Neel once told us this was not recommended, but its the only way I found to
be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old UHD"
version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on Windows
too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of maint"
during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you know
the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000
executions) on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2
and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to
find the XML files defining the RFNoC blocks.  The files may have been
installed in the incorrect path (we are checking the Windows installers
now).  If you built and installed the 64-bit version of UHD, please check
for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to
C:\Program Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version of
UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems
with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems
with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I
will run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I
got the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0 release,
but not with the head of the maint branch.  So, I believe that one has been
fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am
working on reproducing the issues with the information you have provided.
I will let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we
could make in order to make sure the radio is ready for a new process? In
our old UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the
older version of UHD who reported this error happening about 1% of the
time. The issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to
lock to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1
\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under Ubuntu,
I get a sequence error on the 2nd loop, but all samples are transmitted (no
error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hello Michael, Issue #4: I have tried to increase the socket buffer size (with the send_buff_size argument), but I'm getting the next error (see complete output attached): UU UHD Error: x300 fw communication failure #1 EnvironmentError: IOError: x300 fw poke32 - reply timed out Issue "Maximum number of samples": I'm worried that this will negatively impact the performance of the Tx streaming (causing more frequent under-runs), but I have not tested enough yet. You said it should be available in the next UHD release. Let me know if this could be fixed in the "maint" branch, so I could pull this earlier. Do you have a timeframe for the next UHD version? Issue "Initialization time": This is not a bug issue for us, but any improvement in the next UHD release will be appreciated. Thanks, Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 12 December 2016 at 18:07, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > Regarding Issue #4, the sequence error is a known issue and is expected. > I believe the failure on Windows may be due to the default socket buffer > size being too small. Try increasing the default receive socket buffer > size (to 2MB) and let me know if you still see the failures. > > Regards, > Michael > > On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com> > wrote: > >> Hi Serge, >> >> I will set up longer term tests for issues #2 and #3 to see if I can >> reproduce and debug on my end. Issue #4 seems like a legitimate bug, so >> let me spend some time digging into that this week. >> >> Regarding your questions, they are known issues that are currently being >> addressed. The maximum number of samples issue should be fixed in the next >> release of UHD. The slower initialization is being looked at, but may take >> more time to completely resolve. Please let me know if it is significantly >> impacting your application and I can see what I can do to elevate the >> priority. >> >> Regards, >> Michael >> >> On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo < >> serge.malo@skydelsolutions.com> wrote: >> >>> Hello Michael, >>> >>> I have some good news today, and 2 outstanding questions at the end. >>> >>> Issue #2: >>> we have tested with 2 other X300 under Windows, and the problem did not >>> show up. >>> So, on our side, the problem only happens 0.5% of the time, with one >>> device, under Ubuntu. >>> We will implement a "retry" mechanism to workaround this problem, and >>> see with our clients if this is Ok with them. >>> If they have a X300 with which the issue happens more frequently, we >>> will put them in touch with you. >>> >>> Issue #3 >>> We have tested with 2 other X300 and an octoclock-G under Windows, and >>> the problem showed up 4/1000 for one device and 0/1000 for the other. >>> Again, the problem happens less than 1% of time, so we will workaround >>> the problem by implementing a retry mechanism. >>> >>> Issue #4 >>> I have found a workaround that seems functional: >>> When I wish to restart the streaming, I simply delete (reset) the >>> multi_usrp object and redo the complete initialization. >>> In fact, this is the workaround I was using with my "old UHD" version. >>> Neel once told us this was not recommended, but its the only way I found to >>> be able to restart the streaming without quitting the application. >>> The original issue is still reproducible 100% of the time with >>> benchmark_rate under Windows. >>> >>> Issue #6 >>> (We had discussed this earlier): Spectrum deformation with the "old UHD" >>> version. >>> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >>> version of December 1st 2016). >>> >>> Question A: >>> The function "tx_streamer::get_max_num_samps" used to return 1994 >>> samples, but now it returns only 364 samples. I still see "Determining >>> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >>> value (364 samples) with benchmark_rate. >>> >>> Question B: >>> Creating the mutli_usrp object now takes 15-20 seconds, while it took >>> around 10s with the "old UHD" version I had. Is there a way to reduce this >>> initialization time? >>> >>> Thanks again for your help! >>> Serge >>> >>> >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 <(514)%20294-4017> >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.com> >>> wrote: >>> >>>> Hello Michael, >>>> >>>> >>>> Issue #5: >>>> My bad! this is not an issue, I can use "head of maint" on Windows now. >>>> >>>> Issue #1: >>>> I confirm this is fixed with the "head of maint" UHD version on Windows >>>> too. Great! >>>> >>>> Issues #2 and #3: >>>> Unfortunately, I was able to reproduce the problem with "head of maint" >>>> during my overnight on Ubuntu: >>>> Issue #2 happened 4 times out of 948 executions >>>> Issue #3 happened 9 times out of 948 executions >>>> But under Windows 10, I had the next results: >>>> Issue #2 happened 0 time out of 520 executions >>>> Issue #3 happened 1 time out of 520 executions >>>> I will try to reproduce the problem with another X300 and let you know >>>> the results. >>>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>>> Can you confirm you don't see this problem at all ( 0 / 1000 >>>> executions) on your side (this test can be done over one night). >>>> >>>> Issue #4 >>>> The problem still happens with UHD "head of maint" under Windows. >>>> Were you able to reproduce this on your side? >>>> >>>> Thanks again, >>>> Serge >>>> >>>> >>>> __________________ >>>> *Serge Malo * >>>> CDO & Co-founder, Skydel Solutions >>>> Cell: 1-514-294-4017 >>>> www.skydelsolutions.com >>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>> >>>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> Since I do not have a unit here that I can use to reproduce issues #2 >>>>> and #3, I am not able to debug them. The fact that the issues disappear >>>>> when using the head of maint is encouraging, but I cannot explain why. If >>>>> you need an answer as to why, I would need to have you ship the unit to me >>>>> so I can try to reproduce the problems and debug them here. Please contact >>>>> me off-list if we need to do that. >>>>> >>>>> Regarding Issue #5, the error is produced by UHD not being able to >>>>> find the XML files defining the RFNoC blocks. The files may have been >>>>> installed in the incorrect path (we are checking the Windows installers >>>>> now). If you built and installed the 64-bit version of UHD, please check >>>>> for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to >>>>> C:\Program Files\UHD\share\uhd\rfnoc. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>>> serge.malo@skydelsolutions.com> wrote: >>>>> >>>>>> Hello Michael, >>>>>> >>>>>> Here are more details and results from my side: >>>>>> >>>>>> Issue #1: >>>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>>> problem is fixed under Ubuntu. Great! >>>>>> However, I'm facing a new issue under Windows 10 with this version of >>>>>> UHD, I can't connect to the X300 at all, see issue #5 below. >>>>>> >>>>>> Issues #2 and #3 >>>>>> -I have tried UHD 003.010.001.000 with internal clock: the problems >>>>>> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >>>>>> attached result file "doit_res2_intclk.txt" >>>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>>> with internal clock (0/100). >>>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>>> with external clock (0/100). >>>>>> -My external clock is a Wenzel OCXO >>>>>> -I have only tried one X300 device so far. But some clients have >>>>>> reported issue #2 with our older version of UHD. >>>>>> So, issues "seem" to be fixed with the "head of maint" version. I >>>>>> will run overnight tests to get more results. >>>>>> >>>>>> Issue #5: >>>>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>>>> benchmark_rate, I get the error message below. >>>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I >>>>>> got the same error message. >>>>>> >>>>>> Creating the usrp device with: addr=192.168.60.2... >>>>>> -- X300 initialization sequence... >>>>>> -- Determining maximum frame size... 8000 bytes. >>>>>> -- Setup basic communication... >>>>>> -- Loading values from EEPROM... >>>>>> -- Setup RF frontend clocking... >>>>>> -- Radio 1x clock:200 >>>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>>> Error: AssertionError: block_def >>>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>>> >>>>>> Do you see this problem on your side too? >>>>>> >>>>>> Thanks again, >>>>>> Serge >>>>>> >>>>>> >>>>>> __________________ >>>>>> *Serge Malo * >>>>>> CDO & Co-founder, Skydel Solutions >>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>> www.skydelsolutions.com >>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>> >>>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Serge, >>>>>>> >>>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>>>>>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>>>>>> any issue. I am even using a rev 5 device to match the X300 hardware >>>>>>> revision of your device. >>>>>>> >>>>>>> There are 2 things I am curious about: >>>>>>> >>>>>>> 1) What is the external reference being used? Do you see these >>>>>>> failures when using the internal reference? >>>>>>> 2) Do you see these failures on other devices or just this one >>>>>>> device? >>>>>>> >>>>>>> I will be looking into issue #4 tomorrow. >>>>>>> >>>>>>> Regards, >>>>>>> Michael >>>>>>> >>>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>> >>>>>>>> Thanks for the update Michael, >>>>>>>> I will try the head of the maint branch on my side too as soon as >>>>>>>> possible. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Serge >>>>>>>> >>>>>>>> __________________ >>>>>>>> *Serge Malo * >>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>> www.skydelsolutions.com >>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>> >>>>>>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Serge, >>>>>>>>> >>>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 release, >>>>>>>>> but not with the head of the maint branch. So, I believe that one has been >>>>>>>>> fixed. I am still working on reproducing the other 3 issues. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Michael >>>>>>>>> >>>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Serge, >>>>>>>>>> >>>>>>>>>> Thank you for the detailed information and examples. I am >>>>>>>>>> working on reproducing the issues with the information you have provided. >>>>>>>>>> I will let you know what I find. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Michael >>>>>>>>>> >>>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> We have started the process of upgrading the UHD version in our >>>>>>>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>>> >>>>>>>>>>> Configuration: >>>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>>> given by Derek) >>>>>>>>>>> >>>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>>>>>>> before I restart it again, otherwise I get this error: >>>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>>> Device Address: >>>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>>> Q: At the end of a transmission, is there call or a poll we >>>>>>>>>>> could make in order to make sure the radio is ready for a new process? In >>>>>>>>>>> our old UHD version, this was not happening. >>>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>>>>>> unexpected FIFO depth [0x1f]" >>>>>>>>>>> This error happens only about 1% of the time, but I was able to >>>>>>>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>>>>>>> Ubuntu. >>>>>>>>>>> See attached script "doit.sh". >>>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>>>>>> result file "doit_res.txt") >>>>>>>>>>> We have clients and partners using our application with the >>>>>>>>>>> older version of UHD who reported this error happening about 1% of the >>>>>>>>>>> time. The issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to >>>>>>>>>>> lock to external source. >>>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>>> message on the 2nd loop: >>>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>>>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>>>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>>>>>>> AssertionError: buff->size() > 0 >>>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>>> bool) >>>>>>>>>>> at ...\uhd_003_010_001_000-Patch1 >>>>>>>>>>> \host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>>>>>>> >>>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under Ubuntu, >>>>>>>>>>> I get a sequence error on the 2nd loop, but all samples are transmitted (no >>>>>>>>>>> error message) >>>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>>>> "repeat loop". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks in advance for your help! >>>>>>>>>>> Serge >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> __________________ >>>>>>>>>>> *Serge Malo * >>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> USRP-users mailing list >>>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>>> us.com >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
MW
Michael West
Wed, Dec 14, 2016 6:37 PM

Hi Serge,

Issue #4:
That is progress.  It looks like the radio control responses are now
getting received with the larger socket buffer, but somewhere along the
line the firmware control transport is having issues.  To confirm, the
error is only on Windows and Ubuntu works fine, right?  Also, can you check
to see if you see the issue if you modify the code to use a single
multi_usrp object instead of re-creating it?

Issue "Maximum number of samples":
The fix requires a new FPGA image, so it is not likely to be available
until the next release.  I will see what I can do to get you early access
if possible.

Issue "initialization time":
Believe me, we all want a shorter initialization time.  I'm sure it will
improve over time.

Regards,
Michael

On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #4:
I have tried to increase the socket buffer size (with the send_buff_size
argument), but I'm getting the next error (see complete output attached):
UU
UHD Error:
x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out

Issue "Maximum number of samples":
I'm worried that this will negatively impact the performance of the Tx
streaming (causing more frequent under-runs), but I have not tested enough
yet.
You said it should be available in the next UHD release. Let me know if
this could be fixed in the "maint" branch, so I could pull this earlier.
Do you have a timeframe for the next UHD version?

Issue "Initialization time":
This is not a bug issue for us, but any improvement in the next UHD
release will be appreciated.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 12 December 2016 at 18:07, Michael West michael.west@ettus.com wrote:

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is expected.
I believe the failure on Windows may be due to the default socket buffer
size being too small.  Try increasing the default receive socket buffer
size (to 2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently being
addressed.  The maximum number of samples issue should be fixed in the next
release of UHD.  The slower initialization is being looked at, but may take
more time to completely resolve.  Please let me know if it is significantly
impacting your application and I can see what I can do to elevate the
priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did not
show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and
see with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we
will put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows, and
the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will workaround
the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD" version.
Neel once told us this was not recommended, but its the only way I found to
be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old
UHD" version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.com

wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on
Windows too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of
maint" during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you know
the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000
executions) on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues #2
and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to
find the XML files defining the RFNoC blocks.  The files may have been
installed in the incorrect path (we are checking the Windows installers
now).  If you built and installed the 64-bit version of UHD, please check
for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to
C:\Program Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version
of UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems
with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems
with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I
will run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I
got the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0
release, but not with the head of the maint branch.  So, I believe that one
has been fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am
working on reproducing the issues with the information you have provided.
I will let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in our
application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait 5-10s
before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we
could make in order to make sure the radio is ready for a new process? In
our old UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able to
reproduce it with a simple script calling "benchmark_rate" in a loop under
Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the
older version of UHD who reported this error happening about 1% of the
time. The issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to
lock to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32()
failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed:
EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet -
AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1
\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under
Ubuntu, I get a sequence error on the 2nd loop, but all samples are
transmitted (no error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hi Serge, Issue #4: That is progress. It looks like the radio control responses are now getting received with the larger socket buffer, but somewhere along the line the firmware control transport is having issues. To confirm, the error is only on Windows and Ubuntu works fine, right? Also, can you check to see if you see the issue if you modify the code to use a single multi_usrp object instead of re-creating it? Issue "Maximum number of samples": The fix requires a new FPGA image, so it is not likely to be available until the next release. I will see what I can do to get you early access if possible. Issue "initialization time": Believe me, we all want a shorter initialization time. I'm sure it will improve over time. Regards, Michael On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo <serge.malo@skydelsolutions.com> wrote: > Hello Michael, > > Issue #4: > I have tried to increase the socket buffer size (with the send_buff_size > argument), but I'm getting the next error (see complete output attached): > UU > UHD Error: > x300 fw communication failure #1 > EnvironmentError: IOError: x300 fw poke32 - reply timed out > > Issue "Maximum number of samples": > I'm worried that this will negatively impact the performance of the Tx > streaming (causing more frequent under-runs), but I have not tested enough > yet. > You said it should be available in the next UHD release. Let me know if > this could be fixed in the "maint" branch, so I could pull this earlier. > Do you have a timeframe for the next UHD version? > > Issue "Initialization time": > This is not a bug issue for us, but any improvement in the next UHD > release will be appreciated. > > Thanks, > Serge > > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 12 December 2016 at 18:07, Michael West <michael.west@ettus.com> wrote: > >> Hi Serge, >> >> Regarding Issue #4, the sequence error is a known issue and is expected. >> I believe the failure on Windows may be due to the default socket buffer >> size being too small. Try increasing the default receive socket buffer >> size (to 2MB) and let me know if you still see the failures. >> >> Regards, >> Michael >> >> On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com> >> wrote: >> >>> Hi Serge, >>> >>> I will set up longer term tests for issues #2 and #3 to see if I can >>> reproduce and debug on my end. Issue #4 seems like a legitimate bug, so >>> let me spend some time digging into that this week. >>> >>> Regarding your questions, they are known issues that are currently being >>> addressed. The maximum number of samples issue should be fixed in the next >>> release of UHD. The slower initialization is being looked at, but may take >>> more time to completely resolve. Please let me know if it is significantly >>> impacting your application and I can see what I can do to elevate the >>> priority. >>> >>> Regards, >>> Michael >>> >>> On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo < >>> serge.malo@skydelsolutions.com> wrote: >>> >>>> Hello Michael, >>>> >>>> I have some good news today, and 2 outstanding questions at the end. >>>> >>>> Issue #2: >>>> we have tested with 2 other X300 under Windows, and the problem did not >>>> show up. >>>> So, on our side, the problem only happens 0.5% of the time, with one >>>> device, under Ubuntu. >>>> We will implement a "retry" mechanism to workaround this problem, and >>>> see with our clients if this is Ok with them. >>>> If they have a X300 with which the issue happens more frequently, we >>>> will put them in touch with you. >>>> >>>> Issue #3 >>>> We have tested with 2 other X300 and an octoclock-G under Windows, and >>>> the problem showed up 4/1000 for one device and 0/1000 for the other. >>>> Again, the problem happens less than 1% of time, so we will workaround >>>> the problem by implementing a retry mechanism. >>>> >>>> Issue #4 >>>> I have found a workaround that seems functional: >>>> When I wish to restart the streaming, I simply delete (reset) the >>>> multi_usrp object and redo the complete initialization. >>>> In fact, this is the workaround I was using with my "old UHD" version. >>>> Neel once told us this was not recommended, but its the only way I found to >>>> be able to restart the streaming without quitting the application. >>>> The original issue is still reproducible 100% of the time with >>>> benchmark_rate under Windows. >>>> >>>> Issue #6 >>>> (We had discussed this earlier): Spectrum deformation with the "old >>>> UHD" version. >>>> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >>>> version of December 1st 2016). >>>> >>>> Question A: >>>> The function "tx_streamer::get_max_num_samps" used to return 1994 >>>> samples, but now it returns only 364 samples. I still see "Determining >>>> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >>>> value (364 samples) with benchmark_rate. >>>> >>>> Question B: >>>> Creating the mutli_usrp object now takes 15-20 seconds, while it took >>>> around 10s with the "old UHD" version I had. Is there a way to reduce this >>>> initialization time? >>>> >>>> Thanks again for your help! >>>> Serge >>>> >>>> >>>> >>>> __________________ >>>> *Serge Malo * >>>> CDO & Co-founder, Skydel Solutions >>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>> www.skydelsolutions.com >>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>> >>>> On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.com >>>> > wrote: >>>> >>>>> Hello Michael, >>>>> >>>>> >>>>> Issue #5: >>>>> My bad! this is not an issue, I can use "head of maint" on Windows now. >>>>> >>>>> Issue #1: >>>>> I confirm this is fixed with the "head of maint" UHD version on >>>>> Windows too. Great! >>>>> >>>>> Issues #2 and #3: >>>>> Unfortunately, I was able to reproduce the problem with "head of >>>>> maint" during my overnight on Ubuntu: >>>>> Issue #2 happened 4 times out of 948 executions >>>>> Issue #3 happened 9 times out of 948 executions >>>>> But under Windows 10, I had the next results: >>>>> Issue #2 happened 0 time out of 520 executions >>>>> Issue #3 happened 1 time out of 520 executions >>>>> I will try to reproduce the problem with another X300 and let you know >>>>> the results. >>>>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>>>> Can you confirm you don't see this problem at all ( 0 / 1000 >>>>> executions) on your side (this test can be done over one night). >>>>> >>>>> Issue #4 >>>>> The problem still happens with UHD "head of maint" under Windows. >>>>> Were you able to reproduce this on your side? >>>>> >>>>> Thanks again, >>>>> Serge >>>>> >>>>> >>>>> __________________ >>>>> *Serge Malo * >>>>> CDO & Co-founder, Skydel Solutions >>>>> Cell: 1-514-294-4017 >>>>> www.skydelsolutions.com >>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>> >>>>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>>>> wrote: >>>>> >>>>>> Hi Serge, >>>>>> >>>>>> Since I do not have a unit here that I can use to reproduce issues #2 >>>>>> and #3, I am not able to debug them. The fact that the issues disappear >>>>>> when using the head of maint is encouraging, but I cannot explain why. If >>>>>> you need an answer as to why, I would need to have you ship the unit to me >>>>>> so I can try to reproduce the problems and debug them here. Please contact >>>>>> me off-list if we need to do that. >>>>>> >>>>>> Regarding Issue #5, the error is produced by UHD not being able to >>>>>> find the XML files defining the RFNoC blocks. The files may have been >>>>>> installed in the incorrect path (we are checking the Windows installers >>>>>> now). If you built and installed the 64-bit version of UHD, please check >>>>>> for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to >>>>>> C:\Program Files\UHD\share\uhd\rfnoc. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>> >>>>>>> Hello Michael, >>>>>>> >>>>>>> Here are more details and results from my side: >>>>>>> >>>>>>> Issue #1: >>>>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>>>> problem is fixed under Ubuntu. Great! >>>>>>> However, I'm facing a new issue under Windows 10 with this version >>>>>>> of UHD, I can't connect to the X300 at all, see issue #5 below. >>>>>>> >>>>>>> Issues #2 and #3 >>>>>>> -I have tried UHD 003.010.001.000 with internal clock: the problems >>>>>>> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >>>>>>> attached result file "doit_res2_intclk.txt" >>>>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>>>> with internal clock (0/100). >>>>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>>>> with external clock (0/100). >>>>>>> -My external clock is a Wenzel OCXO >>>>>>> -I have only tried one X300 device so far. But some clients have >>>>>>> reported issue #2 with our older version of UHD. >>>>>>> So, issues "seem" to be fixed with the "head of maint" version. I >>>>>>> will run overnight tests to get more results. >>>>>>> >>>>>>> Issue #5: >>>>>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>>>>> benchmark_rate, I get the error message below. >>>>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I >>>>>>> got the same error message. >>>>>>> >>>>>>> Creating the usrp device with: addr=192.168.60.2... >>>>>>> -- X300 initialization sequence... >>>>>>> -- Determining maximum frame size... 8000 bytes. >>>>>>> -- Setup basic communication... >>>>>>> -- Loading values from EEPROM... >>>>>>> -- Setup RF frontend clocking... >>>>>>> -- Radio 1x clock:200 >>>>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>>>> Error: AssertionError: block_def >>>>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>>>> >>>>>>> Do you see this problem on your side too? >>>>>>> >>>>>>> Thanks again, >>>>>>> Serge >>>>>>> >>>>>>> >>>>>>> __________________ >>>>>>> *Serge Malo * >>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>> www.skydelsolutions.com >>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>> >>>>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Serge, >>>>>>>> >>>>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>>>>>>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>>>>>>> any issue. I am even using a rev 5 device to match the X300 hardware >>>>>>>> revision of your device. >>>>>>>> >>>>>>>> There are 2 things I am curious about: >>>>>>>> >>>>>>>> 1) What is the external reference being used? Do you see these >>>>>>>> failures when using the internal reference? >>>>>>>> 2) Do you see these failures on other devices or just this one >>>>>>>> device? >>>>>>>> >>>>>>>> I will be looking into issue #4 tomorrow. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>> >>>>>>>>> Thanks for the update Michael, >>>>>>>>> I will try the head of the maint branch on my side too as soon as >>>>>>>>> possible. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Serge >>>>>>>>> >>>>>>>>> __________________ >>>>>>>>> *Serge Malo * >>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>> www.skydelsolutions.com >>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>> >>>>>>>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Serge, >>>>>>>>>> >>>>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 >>>>>>>>>> release, but not with the head of the maint branch. So, I believe that one >>>>>>>>>> has been fixed. I am still working on reproducing the other 3 issues. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Michael >>>>>>>>>> >>>>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Serge, >>>>>>>>>>> >>>>>>>>>>> Thank you for the detailed information and examples. I am >>>>>>>>>>> working on reproducing the issues with the information you have provided. >>>>>>>>>>> I will let you know what I find. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Michael >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> We have started the process of upgrading the UHD version in our >>>>>>>>>>>> application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>>>> >>>>>>>>>>>> Configuration: >>>>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>>>> given by Derek) >>>>>>>>>>>> >>>>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>>>> When benchmark_rate process ends normally, I have to wait 5-10s >>>>>>>>>>>> before I restart it again, otherwise I get this error: >>>>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>>>> Device Address: >>>>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>>>> Q: At the end of a transmission, is there call or a poll we >>>>>>>>>>>> could make in order to make sure the radio is ready for a new process? In >>>>>>>>>>>> our old UHD version, this was not happening. >>>>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>>>>>>> unexpected FIFO depth [0x1f]" >>>>>>>>>>>> This error happens only about 1% of the time, but I was able to >>>>>>>>>>>> reproduce it with a simple script calling "benchmark_rate" in a loop under >>>>>>>>>>>> Ubuntu. >>>>>>>>>>>> See attached script "doit.sh". >>>>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>>>>>>> result file "doit_res.txt") >>>>>>>>>>>> We have clients and partners using our application with the >>>>>>>>>>>> older version of UHD who reported this error happening about 1% of the >>>>>>>>>>>> time. The issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to >>>>>>>>>>>> lock to external source. >>>>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>>>> message on the 2nd loop: >>>>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() >>>>>>>>>>>> failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: >>>>>>>>>>>> EnvironmentError: IOError: Block ctrl (CE_01_Port_40) no response packet - >>>>>>>>>>>> AssertionError: buff->size() > 0 >>>>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>>>> bool) >>>>>>>>>>>> at ...\uhd_003_010_001_000-Patch1 >>>>>>>>>>>> \host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>>>>>>>> >>>>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under >>>>>>>>>>>> Ubuntu, I get a sequence error on the 2nd loop, but all samples are >>>>>>>>>>>> transmitted (no error message) >>>>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>>>>> "repeat loop". >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance for your help! >>>>>>>>>>>> Serge >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> __________________ >>>>>>>>>>>> *Serge Malo * >>>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> USRP-users mailing list >>>>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>>>> us.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
SM
Serge Malo
Thu, Dec 15, 2016 1:42 AM

Hello Michael,

Issue #4:
Here's what I observed:
Windows, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Error
Windows, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!
Ubuntu, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Ok (but Sequence Error)
Ubuntu, restarting the streaming by re-creating the multi_usrp and then the
streamer: Ok!

I will use the solution "re-creating the multi_usrp and then the streamer"
in my application. This means we have to go through the init sequence every
time we start the streaming.

Let me know if I can help by running other tests.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 14 December 2016 at 13:37, Michael West michael.west@ettus.com wrote:

Hi Serge,

Issue #4:
That is progress.  It looks like the radio control responses are now
getting received with the larger socket buffer, but somewhere along the
line the firmware control transport is having issues.  To confirm, the
error is only on Windows and Ubuntu works fine, right?  Also, can you check
to see if you see the issue if you modify the code to use a single
multi_usrp object instead of re-creating it?

Issue "Maximum number of samples":
The fix requires a new FPGA image, so it is not likely to be available
until the next release.  I will see what I can do to get you early access
if possible.

Issue "initialization time":
Believe me, we all want a shorter initialization time.  I'm sure it will
improve over time.

Regards,
Michael

On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo <serge.malo@skydelsolutions.
com> wrote:

Hello Michael,

Issue #4:
I have tried to increase the socket buffer size (with the send_buff_size
argument), but I'm getting the next error (see complete output attached):
UU
UHD Error:
x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out

Issue "Maximum number of samples":
I'm worried that this will negatively impact the performance of the Tx
streaming (causing more frequent under-runs), but I have not tested enough
yet.
You said it should be available in the next UHD release. Let me know if
this could be fixed in the "maint" branch, so I could pull this earlier.
Do you have a timeframe for the next UHD version?

Issue "Initialization time":
This is not a bug issue for us, but any improvement in the next UHD
release will be appreciated.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 12 December 2016 at 18:07, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is
expected.  I believe the failure on Windows may be due to the default
socket buffer size being too small.  Try increasing the default receive
socket buffer size (to 2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently
being addressed.  The maximum number of samples issue should be fixed in
the next release of UHD.  The slower initialization is being looked at, but
may take more time to completely resolve.  Please let me know if it is
significantly impacting your application and I can see what I can do to
elevate the priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did
not show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and
see with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we
will put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows, and
the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will workaround
the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD" version.
Neel once told us this was not recommended, but its the only way I found to
be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old
UHD" version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.co
m> wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows
now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on
Windows too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of
maint" during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you
know the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000
executions) on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues
#2 and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to
find the XML files defining the RFNoC blocks.  The files may have been
installed in the incorrect path (we are checking the Windows installers
now).  If you built and installed the 64-bit version of UHD, please check
for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to
C:\Program Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version
of UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the problems
happened again at about the same rate (issue #2 1/100; issue #3 2/100). See
attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the problems
with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the problems
with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I
will run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I
got the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have tried
running the script on UHD 3.10.1.0 and the head of the maint branch without
any issue.  I am even using a rev 5 device to match the X300 hardware
revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon as
possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com

wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0
release, but not with the head of the maint branch.  So, I believe that one
has been fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am
working on reproducing the issues with the information you have provided.
I will let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in
our application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait
5-10s before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we
could make in order to make sure the radio is ready for a new process? In
our old UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync failed.
unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able
to reproduce it with a simple script calling "benchmark_rate" in a loop
under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See attached
result file "doit_res.txt")
We have clients and partners using our application with the
older version of UHD who reported this error happening about 1% of the
time. The issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to
lock to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0]
user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0]
sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40)
no response packet - AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1
\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under
Ubuntu, I get a sequence error on the 2nd loop, but all samples are
transmitted (no error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hello Michael, Issue #4: Here's what I observed: Windows, restarting the streaming by re-creating the streamer (as the modified benchmark_rate.cpp I provided): Error Windows, restarting the streaming by re-creating the multi_usrp and then the streamer: Ok! Ubuntu, restarting the streaming by re-creating the streamer (as the modified benchmark_rate.cpp I provided): Ok (but Sequence Error) Ubuntu, restarting the streaming by re-creating the multi_usrp and then the streamer: Ok! I will use the solution "re-creating the multi_usrp and then the streamer" in my application. This means we have to go through the init sequence every time we start the streaming. Let me know if I can help by running other tests. Thanks, Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 14 December 2016 at 13:37, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > Issue #4: > That is progress. It looks like the radio control responses are now > getting received with the larger socket buffer, but somewhere along the > line the firmware control transport is having issues. To confirm, the > error is only on Windows and Ubuntu works fine, right? Also, can you check > to see if you see the issue if you modify the code to use a single > multi_usrp object instead of re-creating it? > > Issue "Maximum number of samples": > The fix requires a new FPGA image, so it is not likely to be available > until the next release. I will see what I can do to get you early access > if possible. > > Issue "initialization time": > Believe me, we all want a shorter initialization time. I'm sure it will > improve over time. > > Regards, > Michael > > On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo <serge.malo@skydelsolutions. > com> wrote: > >> Hello Michael, >> >> Issue #4: >> I have tried to increase the socket buffer size (with the send_buff_size >> argument), but I'm getting the next error (see complete output attached): >> UU >> UHD Error: >> x300 fw communication failure #1 >> EnvironmentError: IOError: x300 fw poke32 - reply timed out >> >> Issue "Maximum number of samples": >> I'm worried that this will negatively impact the performance of the Tx >> streaming (causing more frequent under-runs), but I have not tested enough >> yet. >> You said it should be available in the next UHD release. Let me know if >> this could be fixed in the "maint" branch, so I could pull this earlier. >> Do you have a timeframe for the next UHD version? >> >> Issue "Initialization time": >> This is not a bug issue for us, but any improvement in the next UHD >> release will be appreciated. >> >> Thanks, >> Serge >> >> >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 <(514)%20294-4017> >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> On 12 December 2016 at 18:07, Michael West <michael.west@ettus.com> >> wrote: >> >>> Hi Serge, >>> >>> Regarding Issue #4, the sequence error is a known issue and is >>> expected. I believe the failure on Windows may be due to the default >>> socket buffer size being too small. Try increasing the default receive >>> socket buffer size (to 2MB) and let me know if you still see the failures. >>> >>> Regards, >>> Michael >>> >>> On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> I will set up longer term tests for issues #2 and #3 to see if I can >>>> reproduce and debug on my end. Issue #4 seems like a legitimate bug, so >>>> let me spend some time digging into that this week. >>>> >>>> Regarding your questions, they are known issues that are currently >>>> being addressed. The maximum number of samples issue should be fixed in >>>> the next release of UHD. The slower initialization is being looked at, but >>>> may take more time to completely resolve. Please let me know if it is >>>> significantly impacting your application and I can see what I can do to >>>> elevate the priority. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo < >>>> serge.malo@skydelsolutions.com> wrote: >>>> >>>>> Hello Michael, >>>>> >>>>> I have some good news today, and 2 outstanding questions at the end. >>>>> >>>>> Issue #2: >>>>> we have tested with 2 other X300 under Windows, and the problem did >>>>> not show up. >>>>> So, on our side, the problem only happens 0.5% of the time, with one >>>>> device, under Ubuntu. >>>>> We will implement a "retry" mechanism to workaround this problem, and >>>>> see with our clients if this is Ok with them. >>>>> If they have a X300 with which the issue happens more frequently, we >>>>> will put them in touch with you. >>>>> >>>>> Issue #3 >>>>> We have tested with 2 other X300 and an octoclock-G under Windows, and >>>>> the problem showed up 4/1000 for one device and 0/1000 for the other. >>>>> Again, the problem happens less than 1% of time, so we will workaround >>>>> the problem by implementing a retry mechanism. >>>>> >>>>> Issue #4 >>>>> I have found a workaround that seems functional: >>>>> When I wish to restart the streaming, I simply delete (reset) the >>>>> multi_usrp object and redo the complete initialization. >>>>> In fact, this is the workaround I was using with my "old UHD" version. >>>>> Neel once told us this was not recommended, but its the only way I found to >>>>> be able to restart the streaming without quitting the application. >>>>> The original issue is still reproducible 100% of the time with >>>>> benchmark_rate under Windows. >>>>> >>>>> Issue #6 >>>>> (We had discussed this earlier): Spectrum deformation with the "old >>>>> UHD" version. >>>>> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >>>>> version of December 1st 2016). >>>>> >>>>> Question A: >>>>> The function "tx_streamer::get_max_num_samps" used to return 1994 >>>>> samples, but now it returns only 364 samples. I still see "Determining >>>>> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >>>>> value (364 samples) with benchmark_rate. >>>>> >>>>> Question B: >>>>> Creating the mutli_usrp object now takes 15-20 seconds, while it took >>>>> around 10s with the "old UHD" version I had. Is there a way to reduce this >>>>> initialization time? >>>>> >>>>> Thanks again for your help! >>>>> Serge >>>>> >>>>> >>>>> >>>>> __________________ >>>>> *Serge Malo * >>>>> CDO & Co-founder, Skydel Solutions >>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>> www.skydelsolutions.com >>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>> >>>>> On 9 December 2016 at 09:05, Serge Malo <serge.malo@skydelsolutions.co >>>>> m> wrote: >>>>> >>>>>> Hello Michael, >>>>>> >>>>>> >>>>>> Issue #5: >>>>>> My bad! this is not an issue, I can use "head of maint" on Windows >>>>>> now. >>>>>> >>>>>> Issue #1: >>>>>> I confirm this is fixed with the "head of maint" UHD version on >>>>>> Windows too. Great! >>>>>> >>>>>> Issues #2 and #3: >>>>>> Unfortunately, I was able to reproduce the problem with "head of >>>>>> maint" during my overnight on Ubuntu: >>>>>> Issue #2 happened 4 times out of 948 executions >>>>>> Issue #3 happened 9 times out of 948 executions >>>>>> But under Windows 10, I had the next results: >>>>>> Issue #2 happened 0 time out of 520 executions >>>>>> Issue #3 happened 1 time out of 520 executions >>>>>> I will try to reproduce the problem with another X300 and let you >>>>>> know the results. >>>>>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>>>>> Can you confirm you don't see this problem at all ( 0 / 1000 >>>>>> executions) on your side (this test can be done over one night). >>>>>> >>>>>> Issue #4 >>>>>> The problem still happens with UHD "head of maint" under Windows. >>>>>> Were you able to reproduce this on your side? >>>>>> >>>>>> Thanks again, >>>>>> Serge >>>>>> >>>>>> >>>>>> __________________ >>>>>> *Serge Malo * >>>>>> CDO & Co-founder, Skydel Solutions >>>>>> Cell: 1-514-294-4017 >>>>>> www.skydelsolutions.com >>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>> >>>>>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Serge, >>>>>>> >>>>>>> Since I do not have a unit here that I can use to reproduce issues >>>>>>> #2 and #3, I am not able to debug them. The fact that the issues disappear >>>>>>> when using the head of maint is encouraging, but I cannot explain why. If >>>>>>> you need an answer as to why, I would need to have you ship the unit to me >>>>>>> so I can try to reproduce the problems and debug them here. Please contact >>>>>>> me off-list if we need to do that. >>>>>>> >>>>>>> Regarding Issue #5, the error is produced by UHD not being able to >>>>>>> find the XML files defining the RFNoC blocks. The files may have been >>>>>>> installed in the incorrect path (we are checking the Windows installers >>>>>>> now). If you built and installed the 64-bit version of UHD, please check >>>>>>> for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to >>>>>>> C:\Program Files\UHD\share\uhd\rfnoc. >>>>>>> >>>>>>> Regards, >>>>>>> Michael >>>>>>> >>>>>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>> >>>>>>>> Hello Michael, >>>>>>>> >>>>>>>> Here are more details and results from my side: >>>>>>>> >>>>>>>> Issue #1: >>>>>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>>>>> problem is fixed under Ubuntu. Great! >>>>>>>> However, I'm facing a new issue under Windows 10 with this version >>>>>>>> of UHD, I can't connect to the X300 at all, see issue #5 below. >>>>>>>> >>>>>>>> Issues #2 and #3 >>>>>>>> -I have tried UHD 003.010.001.000 with internal clock: the problems >>>>>>>> happened again at about the same rate (issue #2 1/100; issue #3 2/100). See >>>>>>>> attached result file "doit_res2_intclk.txt" >>>>>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>>>>> with internal clock (0/100). >>>>>>>> -I have tried UHD "head of maint": I did not reproduce the problems >>>>>>>> with external clock (0/100). >>>>>>>> -My external clock is a Wenzel OCXO >>>>>>>> -I have only tried one X300 device so far. But some clients have >>>>>>>> reported issue #2 with our older version of UHD. >>>>>>>> So, issues "seem" to be fixed with the "head of maint" version. I >>>>>>>> will run overnight tests to get more results. >>>>>>>> >>>>>>>> Issue #5: >>>>>>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>>>>>> benchmark_rate, I get the error message below. >>>>>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I >>>>>>>> got the same error message. >>>>>>>> >>>>>>>> Creating the usrp device with: addr=192.168.60.2... >>>>>>>> -- X300 initialization sequence... >>>>>>>> -- Determining maximum frame size... 8000 bytes. >>>>>>>> -- Setup basic communication... >>>>>>>> -- Loading values from EEPROM... >>>>>>>> -- Setup RF frontend clocking... >>>>>>>> -- Radio 1x clock:200 >>>>>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>>>>> Error: AssertionError: block_def >>>>>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>>>>> >>>>>>>> Do you see this problem on your side too? >>>>>>>> >>>>>>>> Thanks again, >>>>>>>> Serge >>>>>>>> >>>>>>>> >>>>>>>> __________________ >>>>>>>> *Serge Malo * >>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>> www.skydelsolutions.com >>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>> >>>>>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Serge, >>>>>>>>> >>>>>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have tried >>>>>>>>> running the script on UHD 3.10.1.0 and the head of the maint branch without >>>>>>>>> any issue. I am even using a rev 5 device to match the X300 hardware >>>>>>>>> revision of your device. >>>>>>>>> >>>>>>>>> There are 2 things I am curious about: >>>>>>>>> >>>>>>>>> 1) What is the external reference being used? Do you see these >>>>>>>>> failures when using the internal reference? >>>>>>>>> 2) Do you see these failures on other devices or just this one >>>>>>>>> device? >>>>>>>>> >>>>>>>>> I will be looking into issue #4 tomorrow. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Michael >>>>>>>>> >>>>>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the update Michael, >>>>>>>>>> I will try the head of the maint branch on my side too as soon as >>>>>>>>>> possible. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Serge >>>>>>>>>> >>>>>>>>>> __________________ >>>>>>>>>> *Serge Malo * >>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>> www.skydelsolutions.com >>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>> >>>>>>>>>> On 7 December 2016 at 13:29, Michael West <michael.west@ettus.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi Serge, >>>>>>>>>>> >>>>>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 >>>>>>>>>>> release, but not with the head of the maint branch. So, I believe that one >>>>>>>>>>> has been fixed. I am still working on reproducing the other 3 issues. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Michael >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Serge, >>>>>>>>>>>> >>>>>>>>>>>> Thank you for the detailed information and examples. I am >>>>>>>>>>>> working on reproducing the issues with the information you have provided. >>>>>>>>>>>> I will let you know what I find. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Michael >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> We have started the process of upgrading the UHD version in >>>>>>>>>>>>> our application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>>>>> >>>>>>>>>>>>> Configuration: >>>>>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>>>>> given by Derek) >>>>>>>>>>>>> >>>>>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>>>>> When benchmark_rate process ends normally, I have to wait >>>>>>>>>>>>> 5-10s before I restart it again, otherwise I get this error: >>>>>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>>>>> Device Address: >>>>>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>>>>> Q: At the end of a transmission, is there call or a poll we >>>>>>>>>>>>> could make in order to make sure the radio is ready for a new process? In >>>>>>>>>>>>> our old UHD version, this was not happening. >>>>>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync failed. >>>>>>>>>>>>> unexpected FIFO depth [0x1f]" >>>>>>>>>>>>> This error happens only about 1% of the time, but I was able >>>>>>>>>>>>> to reproduce it with a simple script calling "benchmark_rate" in a loop >>>>>>>>>>>>> under Ubuntu. >>>>>>>>>>>>> See attached script "doit.sh". >>>>>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See attached >>>>>>>>>>>>> result file "doit_res.txt") >>>>>>>>>>>>> We have clients and partners using our application with the >>>>>>>>>>>>> older version of UHD who reported this error happening about 1% of the >>>>>>>>>>>>> time. The issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to >>>>>>>>>>>>> lock to external source. >>>>>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>>>>> message on the 2nd loop: >>>>>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>> user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>> sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) >>>>>>>>>>>>> no response packet - AssertionError: buff->size() > 0 >>>>>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>>>>> bool) >>>>>>>>>>>>> at ...\uhd_003_010_001_000-Patch1 >>>>>>>>>>>>> \host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>>>>>>>>> >>>>>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under >>>>>>>>>>>>> Ubuntu, I get a sequence error on the 2nd loop, but all samples are >>>>>>>>>>>>> transmitted (no error message) >>>>>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>>>>>> "repeat loop". >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks in advance for your help! >>>>>>>>>>>>> Serge >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> __________________ >>>>>>>>>>>>> *Serge Malo * >>>>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> USRP-users mailing list >>>>>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>>>>> us.com >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
SM
Serge Malo
Thu, Dec 15, 2016 5:06 PM

Hello Michael,

I have another issue with UHD 3.10.1.0: under-runs at 2x50MSPS under
Windows 10 64-bit.

With benchmark_rate, I can't use a TX sampling rate of 50MSPS on 2 channels
under Windows without having tons of under-runs.
I am using benchmark_rate with the options "--tx_cpu=sc16 --channels=0,1".

  • I was able to 2x50MSPS with my old version of UHD (and also 2x60MSPS with
    a mcr of 120MHz)
  • I am able to use 2x50MSPS and 2x60MSPS under Ubuntu with UHD 3.10.1.0,
    with the same PC (same BIOS settings).

I have tried UHD compiled by Ettus (MSVC 2013, Boost 1.56) and the one I
have recompiled from maint branch (Dec 1st). I get the same results:
2x25MSPS: Ok
2x50MSPS: Under-runs
2x60MSPS: Under-runs
2x100MSP: Under-runs

I have set my BIOS settings such that the CPU clock rate is always stable
at 100% (4GHz).
In the  X520-DA2 driver settings:

  • I have enabled Jumbo Packets
  • I have increased Tx and Rx buffer size to maximum
  • I have set interrupt Moderation Rate to "Extreme", but "Adaptive" or
    "Off" don't see to have better results.
    I have set the "FastSendDatagramThreshold" Registry Key to 9000

I am wondering if this performance drop is related to the "max_num_samps"
value being limited to 364 samples in this UHD version.
Do you see the same results on Windows on your end?

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 14 December 2016 at 20:42, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #4:
Here's what I observed:
Windows, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Error
Windows, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!
Ubuntu, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Ok (but Sequence Error)
Ubuntu, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!

I will use the solution "re-creating the multi_usrp and then the streamer"
in my application. This means we have to go through the init sequence every
time we start the streaming.

Let me know if I can help by running other tests.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 14 December 2016 at 13:37, Michael West michael.west@ettus.com wrote:

Hi Serge,

Issue #4:
That is progress.  It looks like the radio control responses are now
getting received with the larger socket buffer, but somewhere along the
line the firmware control transport is having issues.  To confirm, the
error is only on Windows and Ubuntu works fine, right?  Also, can you check
to see if you see the issue if you modify the code to use a single
multi_usrp object instead of re-creating it?

Issue "Maximum number of samples":
The fix requires a new FPGA image, so it is not likely to be available
until the next release.  I will see what I can do to get you early access
if possible.

Issue "initialization time":
Believe me, we all want a shorter initialization time.  I'm sure it will
improve over time.

Regards,
Michael

On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Issue #4:
I have tried to increase the socket buffer size (with the send_buff_size
argument), but I'm getting the next error (see complete output attached):
UU
UHD Error:
x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out

Issue "Maximum number of samples":
I'm worried that this will negatively impact the performance of the Tx
streaming (causing more frequent under-runs), but I have not tested enough
yet.
You said it should be available in the next UHD release. Let me know if
this could be fixed in the "maint" branch, so I could pull this earlier.
Do you have a timeframe for the next UHD version?

Issue "Initialization time":
This is not a bug issue for us, but any improvement in the next UHD
release will be appreciated.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 12 December 2016 at 18:07, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is
expected.  I believe the failure on Windows may be due to the default
socket buffer size being too small.  Try increasing the default receive
socket buffer size (to 2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently
being addressed.  The maximum number of samples issue should be fixed in
the next release of UHD.  The slower initialization is being looked at, but
may take more time to completely resolve.  Please let me know if it is
significantly impacting your application and I can see what I can do to
elevate the priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did
not show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and
see with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we
will put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows,
and the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will
workaround the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD"
version. Neel once told us this was not recommended, but its the only way I
found to be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old
UHD" version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows
now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on
Windows too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of
maint" during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you
know the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000
executions) on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues
#2 and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to
find the XML files defining the RFNoC blocks.  The files may have been
installed in the incorrect path (we are checking the Windows installers
now).  If you built and installed the 64-bit version of UHD, please check
for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to
C:\Program Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version
of UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the
problems happened again at about the same rate (issue #2 1/100; issue #3
2/100). See attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the
problems with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the
problems with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I
will run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I
got the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have
tried running the script on UHD 3.10.1.0 and the head of the maint branch
without any issue.  I am even using a rev 5 device to match the X300
hardware revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon
as possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0
release, but not with the head of the maint branch.  So, I believe that one
has been fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am
working on reproducing the issues with the information you have provided.
I will let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in
our application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait
5-10s before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we
could make in order to make sure the radio is ready for a new process? In
our old UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync
failed. unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able
to reproduce it with a simple script calling "benchmark_rate" in a loop
under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See
attached result file "doit_res.txt")
We have clients and partners using our application with the
older version of UHD who reported this error happening about 1% of the
time. The issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to
lock to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0]
user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0]
sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40)
no response packet - AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1
\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under
Ubuntu, I get a sequence error on the 2nd loop, but all samples are
transmitted (no error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hello Michael, I have another issue with UHD 3.10.1.0: under-runs at 2x50MSPS under Windows 10 64-bit. With benchmark_rate, I can't use a TX sampling rate of 50MSPS on 2 channels under Windows without having tons of under-runs. I am using benchmark_rate with the options "--tx_cpu=sc16 --channels=0,1". - I was able to 2x50MSPS with my old version of UHD (and also 2x60MSPS with a mcr of 120MHz) - I am able to use 2x50MSPS and 2x60MSPS under Ubuntu with UHD 3.10.1.0, with the same PC (same BIOS settings). I have tried UHD compiled by Ettus (MSVC 2013, Boost 1.56) and the one I have recompiled from maint branch (Dec 1st). I get the same results: 2x25MSPS: Ok 2x50MSPS: Under-runs 2x60MSPS: Under-runs 2x100MSP: Under-runs I have set my BIOS settings such that the CPU clock rate is always stable at 100% (4GHz). In the X520-DA2 driver settings: - I have enabled Jumbo Packets - I have increased Tx and Rx buffer size to maximum - I have set interrupt Moderation Rate to "Extreme", but "Adaptive" or "Off" don't see to have better results. I have set the "FastSendDatagramThreshold" Registry Key to 9000 I am wondering if this performance drop is related to the "max_num_samps" value being limited to 364 samples in this UHD version. Do you see the same results on Windows on your end? Regards, Serge __________________ *Serge Malo * CDO & Co-founder, Skydel Solutions Cell: 1-514-294-4017 www.skydelsolutions.com Twitter: @skydelsol <https://twitter.com/skydelsol> On 14 December 2016 at 20:42, Serge Malo <serge.malo@skydelsolutions.com> wrote: > Hello Michael, > > Issue #4: > Here's what I observed: > Windows, restarting the streaming by re-creating the streamer (as the > modified benchmark_rate.cpp I provided): Error > Windows, restarting the streaming by re-creating the multi_usrp and then > the streamer: Ok! > Ubuntu, restarting the streaming by re-creating the streamer (as the > modified benchmark_rate.cpp I provided): Ok (but Sequence Error) > Ubuntu, restarting the streaming by re-creating the multi_usrp and then > the streamer: Ok! > > I will use the solution "re-creating the multi_usrp and then the streamer" > in my application. This means we have to go through the init sequence every > time we start the streaming. > > Let me know if I can help by running other tests. > > Thanks, > Serge > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 14 December 2016 at 13:37, Michael West <michael.west@ettus.com> wrote: > >> Hi Serge, >> >> Issue #4: >> That is progress. It looks like the radio control responses are now >> getting received with the larger socket buffer, but somewhere along the >> line the firmware control transport is having issues. To confirm, the >> error is only on Windows and Ubuntu works fine, right? Also, can you check >> to see if you see the issue if you modify the code to use a single >> multi_usrp object instead of re-creating it? >> >> Issue "Maximum number of samples": >> The fix requires a new FPGA image, so it is not likely to be available >> until the next release. I will see what I can do to get you early access >> if possible. >> >> Issue "initialization time": >> Believe me, we all want a shorter initialization time. I'm sure it will >> improve over time. >> >> Regards, >> Michael >> >> On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo < >> serge.malo@skydelsolutions.com> wrote: >> >>> Hello Michael, >>> >>> Issue #4: >>> I have tried to increase the socket buffer size (with the send_buff_size >>> argument), but I'm getting the next error (see complete output attached): >>> UU >>> UHD Error: >>> x300 fw communication failure #1 >>> EnvironmentError: IOError: x300 fw poke32 - reply timed out >>> >>> Issue "Maximum number of samples": >>> I'm worried that this will negatively impact the performance of the Tx >>> streaming (causing more frequent under-runs), but I have not tested enough >>> yet. >>> You said it should be available in the next UHD release. Let me know if >>> this could be fixed in the "maint" branch, so I could pull this earlier. >>> Do you have a timeframe for the next UHD version? >>> >>> Issue "Initialization time": >>> This is not a bug issue for us, but any improvement in the next UHD >>> release will be appreciated. >>> >>> Thanks, >>> Serge >>> >>> >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 <(514)%20294-4017> >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> On 12 December 2016 at 18:07, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> Regarding Issue #4, the sequence error is a known issue and is >>>> expected. I believe the failure on Windows may be due to the default >>>> socket buffer size being too small. Try increasing the default receive >>>> socket buffer size (to 2MB) and let me know if you still see the failures. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> I will set up longer term tests for issues #2 and #3 to see if I can >>>>> reproduce and debug on my end. Issue #4 seems like a legitimate bug, so >>>>> let me spend some time digging into that this week. >>>>> >>>>> Regarding your questions, they are known issues that are currently >>>>> being addressed. The maximum number of samples issue should be fixed in >>>>> the next release of UHD. The slower initialization is being looked at, but >>>>> may take more time to completely resolve. Please let me know if it is >>>>> significantly impacting your application and I can see what I can do to >>>>> elevate the priority. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo < >>>>> serge.malo@skydelsolutions.com> wrote: >>>>> >>>>>> Hello Michael, >>>>>> >>>>>> I have some good news today, and 2 outstanding questions at the end. >>>>>> >>>>>> Issue #2: >>>>>> we have tested with 2 other X300 under Windows, and the problem did >>>>>> not show up. >>>>>> So, on our side, the problem only happens 0.5% of the time, with one >>>>>> device, under Ubuntu. >>>>>> We will implement a "retry" mechanism to workaround this problem, and >>>>>> see with our clients if this is Ok with them. >>>>>> If they have a X300 with which the issue happens more frequently, we >>>>>> will put them in touch with you. >>>>>> >>>>>> Issue #3 >>>>>> We have tested with 2 other X300 and an octoclock-G under Windows, >>>>>> and the problem showed up 4/1000 for one device and 0/1000 for the other. >>>>>> Again, the problem happens less than 1% of time, so we will >>>>>> workaround the problem by implementing a retry mechanism. >>>>>> >>>>>> Issue #4 >>>>>> I have found a workaround that seems functional: >>>>>> When I wish to restart the streaming, I simply delete (reset) the >>>>>> multi_usrp object and redo the complete initialization. >>>>>> In fact, this is the workaround I was using with my "old UHD" >>>>>> version. Neel once told us this was not recommended, but its the only way I >>>>>> found to be able to restart the streaming without quitting the application. >>>>>> The original issue is still reproducible 100% of the time with >>>>>> benchmark_rate under Windows. >>>>>> >>>>>> Issue #6 >>>>>> (We had discussed this earlier): Spectrum deformation with the "old >>>>>> UHD" version. >>>>>> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >>>>>> version of December 1st 2016). >>>>>> >>>>>> Question A: >>>>>> The function "tx_streamer::get_max_num_samps" used to return 1994 >>>>>> samples, but now it returns only 364 samples. I still see "Determining >>>>>> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >>>>>> value (364 samples) with benchmark_rate. >>>>>> >>>>>> Question B: >>>>>> Creating the mutli_usrp object now takes 15-20 seconds, while it took >>>>>> around 10s with the "old UHD" version I had. Is there a way to reduce this >>>>>> initialization time? >>>>>> >>>>>> Thanks again for your help! >>>>>> Serge >>>>>> >>>>>> >>>>>> >>>>>> __________________ >>>>>> *Serge Malo * >>>>>> CDO & Co-founder, Skydel Solutions >>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>> www.skydelsolutions.com >>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>> >>>>>> On 9 December 2016 at 09:05, Serge Malo < >>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>> >>>>>>> Hello Michael, >>>>>>> >>>>>>> >>>>>>> Issue #5: >>>>>>> My bad! this is not an issue, I can use "head of maint" on Windows >>>>>>> now. >>>>>>> >>>>>>> Issue #1: >>>>>>> I confirm this is fixed with the "head of maint" UHD version on >>>>>>> Windows too. Great! >>>>>>> >>>>>>> Issues #2 and #3: >>>>>>> Unfortunately, I was able to reproduce the problem with "head of >>>>>>> maint" during my overnight on Ubuntu: >>>>>>> Issue #2 happened 4 times out of 948 executions >>>>>>> Issue #3 happened 9 times out of 948 executions >>>>>>> But under Windows 10, I had the next results: >>>>>>> Issue #2 happened 0 time out of 520 executions >>>>>>> Issue #3 happened 1 time out of 520 executions >>>>>>> I will try to reproduce the problem with another X300 and let you >>>>>>> know the results. >>>>>>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>>>>>> Can you confirm you don't see this problem at all ( 0 / 1000 >>>>>>> executions) on your side (this test can be done over one night). >>>>>>> >>>>>>> Issue #4 >>>>>>> The problem still happens with UHD "head of maint" under Windows. >>>>>>> Were you able to reproduce this on your side? >>>>>>> >>>>>>> Thanks again, >>>>>>> Serge >>>>>>> >>>>>>> >>>>>>> __________________ >>>>>>> *Serge Malo * >>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>> Cell: 1-514-294-4017 >>>>>>> www.skydelsolutions.com >>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>> >>>>>>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Serge, >>>>>>>> >>>>>>>> Since I do not have a unit here that I can use to reproduce issues >>>>>>>> #2 and #3, I am not able to debug them. The fact that the issues disappear >>>>>>>> when using the head of maint is encouraging, but I cannot explain why. If >>>>>>>> you need an answer as to why, I would need to have you ship the unit to me >>>>>>>> so I can try to reproduce the problems and debug them here. Please contact >>>>>>>> me off-list if we need to do that. >>>>>>>> >>>>>>>> Regarding Issue #5, the error is produced by UHD not being able to >>>>>>>> find the XML files defining the RFNoC blocks. The files may have been >>>>>>>> installed in the incorrect path (we are checking the Windows installers >>>>>>>> now). If you built and installed the 64-bit version of UHD, please check >>>>>>>> for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to >>>>>>>> C:\Program Files\UHD\share\uhd\rfnoc. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>> >>>>>>>>> Hello Michael, >>>>>>>>> >>>>>>>>> Here are more details and results from my side: >>>>>>>>> >>>>>>>>> Issue #1: >>>>>>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>>>>>> problem is fixed under Ubuntu. Great! >>>>>>>>> However, I'm facing a new issue under Windows 10 with this version >>>>>>>>> of UHD, I can't connect to the X300 at all, see issue #5 below. >>>>>>>>> >>>>>>>>> Issues #2 and #3 >>>>>>>>> -I have tried UHD 003.010.001.000 with internal clock: the >>>>>>>>> problems happened again at about the same rate (issue #2 1/100; issue #3 >>>>>>>>> 2/100). See attached result file "doit_res2_intclk.txt" >>>>>>>>> -I have tried UHD "head of maint": I did not reproduce the >>>>>>>>> problems with internal clock (0/100). >>>>>>>>> -I have tried UHD "head of maint": I did not reproduce the >>>>>>>>> problems with external clock (0/100). >>>>>>>>> -My external clock is a Wenzel OCXO >>>>>>>>> -I have only tried one X300 device so far. But some clients have >>>>>>>>> reported issue #2 with our older version of UHD. >>>>>>>>> So, issues "seem" to be fixed with the "head of maint" version. I >>>>>>>>> will run overnight tests to get more results. >>>>>>>>> >>>>>>>>> Issue #5: >>>>>>>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>>>>>>> benchmark_rate, I get the error message below. >>>>>>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I >>>>>>>>> got the same error message. >>>>>>>>> >>>>>>>>> Creating the usrp device with: addr=192.168.60.2... >>>>>>>>> -- X300 initialization sequence... >>>>>>>>> -- Determining maximum frame size... 8000 bytes. >>>>>>>>> -- Setup basic communication... >>>>>>>>> -- Loading values from EEPROM... >>>>>>>>> -- Setup RF frontend clocking... >>>>>>>>> -- Radio 1x clock:200 >>>>>>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>>>>>> Error: AssertionError: block_def >>>>>>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>>>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>>>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>>>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>>>>>> >>>>>>>>> Do you see this problem on your side too? >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> Serge >>>>>>>>> >>>>>>>>> >>>>>>>>> __________________ >>>>>>>>> *Serge Malo * >>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>> www.skydelsolutions.com >>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>> >>>>>>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Serge, >>>>>>>>>> >>>>>>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have >>>>>>>>>> tried running the script on UHD 3.10.1.0 and the head of the maint branch >>>>>>>>>> without any issue. I am even using a rev 5 device to match the X300 >>>>>>>>>> hardware revision of your device. >>>>>>>>>> >>>>>>>>>> There are 2 things I am curious about: >>>>>>>>>> >>>>>>>>>> 1) What is the external reference being used? Do you see these >>>>>>>>>> failures when using the internal reference? >>>>>>>>>> 2) Do you see these failures on other devices or just this one >>>>>>>>>> device? >>>>>>>>>> >>>>>>>>>> I will be looking into issue #4 tomorrow. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Michael >>>>>>>>>> >>>>>>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks for the update Michael, >>>>>>>>>>> I will try the head of the maint branch on my side too as soon >>>>>>>>>>> as possible. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Serge >>>>>>>>>>> >>>>>>>>>>> __________________ >>>>>>>>>>> *Serge Malo * >>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>> >>>>>>>>>>> On 7 December 2016 at 13:29, Michael West < >>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Serge, >>>>>>>>>>>> >>>>>>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 >>>>>>>>>>>> release, but not with the head of the maint branch. So, I believe that one >>>>>>>>>>>> has been fixed. I am still working on reproducing the other 3 issues. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Michael >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Serge, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for the detailed information and examples. I am >>>>>>>>>>>>> working on reproducing the issues with the information you have provided. >>>>>>>>>>>>> I will let you know what I find. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Michael >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have started the process of upgrading the UHD version in >>>>>>>>>>>>>> our application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Configuration: >>>>>>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>>>>>> given by Derek) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>>>>>> When benchmark_rate process ends normally, I have to wait >>>>>>>>>>>>>> 5-10s before I restart it again, otherwise I get this error: >>>>>>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>>>>>> Device Address: >>>>>>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>>>>>> Q: At the end of a transmission, is there call or a poll we >>>>>>>>>>>>>> could make in order to make sure the radio is ready for a new process? In >>>>>>>>>>>>>> our old UHD version, this was not happening. >>>>>>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync >>>>>>>>>>>>>> failed. unexpected FIFO depth [0x1f]" >>>>>>>>>>>>>> This error happens only about 1% of the time, but I was able >>>>>>>>>>>>>> to reproduce it with a simple script calling "benchmark_rate" in a loop >>>>>>>>>>>>>> under Ubuntu. >>>>>>>>>>>>>> See attached script "doit.sh". >>>>>>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See >>>>>>>>>>>>>> attached result file "doit_res.txt") >>>>>>>>>>>>>> We have clients and partners using our application with the >>>>>>>>>>>>>> older version of UHD who reported this error happening about 1% of the >>>>>>>>>>>>>> time. The issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to >>>>>>>>>>>>>> lock to external source. >>>>>>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>>>>>> message on the 2nd loop: >>>>>>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>>> user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>>> sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) >>>>>>>>>>>>>> no response packet - AssertionError: buff->size() > 0 >>>>>>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>>>>>> bool) >>>>>>>>>>>>>> at ...\uhd_003_010_001_000-Patch1 >>>>>>>>>>>>>> \host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>>>>>>>>>> >>>>>>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under >>>>>>>>>>>>>> Ubuntu, I get a sequence error on the 2nd loop, but all samples are >>>>>>>>>>>>>> transmitted (no error message) >>>>>>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>>>>>>> "repeat loop". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks in advance for your help! >>>>>>>>>>>>>> Serge >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> __________________ >>>>>>>>>>>>>> *Serge Malo * >>>>>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> USRP-users mailing list >>>>>>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>>>>>> us.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
MW
Michael West
Thu, Dec 15, 2016 6:21 PM

Hi Serge,

Issue #4:
Thanks for the additional information.  My testing on Ubuntu yields the
same result as yours.  I will do more testing on Windows and see if I can
reproduce the error and track down the root cause (I'm currently suspecting
something with the udp_wsa_zero_copy transport, since that is used only in
Windows).  The sequence error on Ubuntu is a known issue and we are working
on it.  I'm glad you have a good method to use for your application.  We
will continue to work on fixing the issues you have pointed out.

Issue "Maximum number of samples":
The fix is not yet complete for this.  But the lower packet size does not
impact performance in any way if the sample rate is <=50 Msps, so it will
only be an issue if your application needs a TX sample rate greater than
that.

Thanks again for all the great feedback!

Regards,
Michael

On Wed, Dec 14, 2016 at 5:42 PM, Serge Malo serge.malo@skydelsolutions.com
wrote:

Hello Michael,

Issue #4:
Here's what I observed:
Windows, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Error
Windows, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!
Ubuntu, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Ok (but Sequence Error)
Ubuntu, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!

I will use the solution "re-creating the multi_usrp and then the streamer"
in my application. This means we have to go through the init sequence every
time we start the streaming.

Let me know if I can help by running other tests.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 14 December 2016 at 13:37, Michael West michael.west@ettus.com wrote:

Hi Serge,

Issue #4:
That is progress.  It looks like the radio control responses are now
getting received with the larger socket buffer, but somewhere along the
line the firmware control transport is having issues.  To confirm, the
error is only on Windows and Ubuntu works fine, right?  Also, can you check
to see if you see the issue if you modify the code to use a single
multi_usrp object instead of re-creating it?

Issue "Maximum number of samples":
The fix requires a new FPGA image, so it is not likely to be available
until the next release.  I will see what I can do to get you early access
if possible.

Issue "initialization time":
Believe me, we all want a shorter initialization time.  I'm sure it will
improve over time.

Regards,
Michael

On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Issue #4:
I have tried to increase the socket buffer size (with the send_buff_size
argument), but I'm getting the next error (see complete output attached):
UU
UHD Error:
x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out

Issue "Maximum number of samples":
I'm worried that this will negatively impact the performance of the Tx
streaming (causing more frequent under-runs), but I have not tested enough
yet.
You said it should be available in the next UHD release. Let me know if
this could be fixed in the "maint" branch, so I could pull this earlier.
Do you have a timeframe for the next UHD version?

Issue "Initialization time":
This is not a bug issue for us, but any improvement in the next UHD
release will be appreciated.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 12 December 2016 at 18:07, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is
expected.  I believe the failure on Windows may be due to the default
socket buffer size being too small.  Try increasing the default receive
socket buffer size (to 2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently
being addressed.  The maximum number of samples issue should be fixed in
the next release of UHD.  The slower initialization is being looked at, but
may take more time to completely resolve.  Please let me know if it is
significantly impacting your application and I can see what I can do to
elevate the priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did
not show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem, and
see with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we
will put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows,
and the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will
workaround the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD"
version. Neel once told us this was not recommended, but its the only way I
found to be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old
UHD" version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it took
around 10s with the "old UHD" version I had. Is there a way to reduce this
initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows
now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on
Windows too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of
maint" during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you
know the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000
executions) on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues
#2 and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to
find the XML files defining the RFNoC blocks.  The files may have been
installed in the incorrect path (we are checking the Windows installers
now).  If you built and installed the 64-bit version of UHD, please check
for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to
C:\Program Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this version
of UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the
problems happened again at about the same rate (issue #2 1/100; issue #3
2/100). See attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the
problems with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the
problems with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I
will run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I use
benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I
got the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West michael.west@ettus.com
wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have
tried running the script on UHD 3.10.1.0 and the head of the maint branch
without any issue.  I am even using a rev 5 device to match the X300
hardware revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon
as possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0
release, but not with the head of the maint branch.  So, I believe that one
has been fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am
working on reproducing the issues with the information you have provided.
I will let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in
our application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait
5-10s before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we
could make in order to make sure the radio is ready for a new process? In
our old UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync
failed. unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able
to reproduce it with a simple script calling "benchmark_rate" in a loop
under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See
attached result file "doit_res.txt")
We have clients and partners using our application with the
older version of UHD who reported this error happening about 1% of the
time. The issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to
lock to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0]
user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0]
sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40)
no response packet - AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1
\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under
Ubuntu, I get a sequence error on the 2nd loop, but all samples are
transmitted (no error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do this
"repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hi Serge, Issue #4: Thanks for the additional information. My testing on Ubuntu yields the same result as yours. I will do more testing on Windows and see if I can reproduce the error and track down the root cause (I'm currently suspecting something with the udp_wsa_zero_copy transport, since that is used only in Windows). The sequence error on Ubuntu is a known issue and we are working on it. I'm glad you have a good method to use for your application. We will continue to work on fixing the issues you have pointed out. Issue "Maximum number of samples": The fix is not yet complete for this. But the lower packet size does not impact performance in any way if the sample rate is <=50 Msps, so it will only be an issue if your application needs a TX sample rate greater than that. Thanks again for all the great feedback! Regards, Michael On Wed, Dec 14, 2016 at 5:42 PM, Serge Malo <serge.malo@skydelsolutions.com> wrote: > Hello Michael, > > Issue #4: > Here's what I observed: > Windows, restarting the streaming by re-creating the streamer (as the > modified benchmark_rate.cpp I provided): Error > Windows, restarting the streaming by re-creating the multi_usrp and then > the streamer: Ok! > Ubuntu, restarting the streaming by re-creating the streamer (as the > modified benchmark_rate.cpp I provided): Ok (but Sequence Error) > Ubuntu, restarting the streaming by re-creating the multi_usrp and then > the streamer: Ok! > > I will use the solution "re-creating the multi_usrp and then the streamer" > in my application. This means we have to go through the init sequence every > time we start the streaming. > > Let me know if I can help by running other tests. > > Thanks, > Serge > > > __________________ > *Serge Malo * > CDO & Co-founder, Skydel Solutions > Cell: 1-514-294-4017 <(514)%20294-4017> > www.skydelsolutions.com > Twitter: @skydelsol <https://twitter.com/skydelsol> > > On 14 December 2016 at 13:37, Michael West <michael.west@ettus.com> wrote: > >> Hi Serge, >> >> Issue #4: >> That is progress. It looks like the radio control responses are now >> getting received with the larger socket buffer, but somewhere along the >> line the firmware control transport is having issues. To confirm, the >> error is only on Windows and Ubuntu works fine, right? Also, can you check >> to see if you see the issue if you modify the code to use a single >> multi_usrp object instead of re-creating it? >> >> Issue "Maximum number of samples": >> The fix requires a new FPGA image, so it is not likely to be available >> until the next release. I will see what I can do to get you early access >> if possible. >> >> Issue "initialization time": >> Believe me, we all want a shorter initialization time. I'm sure it will >> improve over time. >> >> Regards, >> Michael >> >> On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo < >> serge.malo@skydelsolutions.com> wrote: >> >>> Hello Michael, >>> >>> Issue #4: >>> I have tried to increase the socket buffer size (with the send_buff_size >>> argument), but I'm getting the next error (see complete output attached): >>> UU >>> UHD Error: >>> x300 fw communication failure #1 >>> EnvironmentError: IOError: x300 fw poke32 - reply timed out >>> >>> Issue "Maximum number of samples": >>> I'm worried that this will negatively impact the performance of the Tx >>> streaming (causing more frequent under-runs), but I have not tested enough >>> yet. >>> You said it should be available in the next UHD release. Let me know if >>> this could be fixed in the "maint" branch, so I could pull this earlier. >>> Do you have a timeframe for the next UHD version? >>> >>> Issue "Initialization time": >>> This is not a bug issue for us, but any improvement in the next UHD >>> release will be appreciated. >>> >>> Thanks, >>> Serge >>> >>> >>> >>> __________________ >>> *Serge Malo * >>> CDO & Co-founder, Skydel Solutions >>> Cell: 1-514-294-4017 <(514)%20294-4017> >>> www.skydelsolutions.com >>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>> >>> On 12 December 2016 at 18:07, Michael West <michael.west@ettus.com> >>> wrote: >>> >>>> Hi Serge, >>>> >>>> Regarding Issue #4, the sequence error is a known issue and is >>>> expected. I believe the failure on Windows may be due to the default >>>> socket buffer size being too small. Try increasing the default receive >>>> socket buffer size (to 2MB) and let me know if you still see the failures. >>>> >>>> Regards, >>>> Michael >>>> >>>> On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> I will set up longer term tests for issues #2 and #3 to see if I can >>>>> reproduce and debug on my end. Issue #4 seems like a legitimate bug, so >>>>> let me spend some time digging into that this week. >>>>> >>>>> Regarding your questions, they are known issues that are currently >>>>> being addressed. The maximum number of samples issue should be fixed in >>>>> the next release of UHD. The slower initialization is being looked at, but >>>>> may take more time to completely resolve. Please let me know if it is >>>>> significantly impacting your application and I can see what I can do to >>>>> elevate the priority. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo < >>>>> serge.malo@skydelsolutions.com> wrote: >>>>> >>>>>> Hello Michael, >>>>>> >>>>>> I have some good news today, and 2 outstanding questions at the end. >>>>>> >>>>>> Issue #2: >>>>>> we have tested with 2 other X300 under Windows, and the problem did >>>>>> not show up. >>>>>> So, on our side, the problem only happens 0.5% of the time, with one >>>>>> device, under Ubuntu. >>>>>> We will implement a "retry" mechanism to workaround this problem, and >>>>>> see with our clients if this is Ok with them. >>>>>> If they have a X300 with which the issue happens more frequently, we >>>>>> will put them in touch with you. >>>>>> >>>>>> Issue #3 >>>>>> We have tested with 2 other X300 and an octoclock-G under Windows, >>>>>> and the problem showed up 4/1000 for one device and 0/1000 for the other. >>>>>> Again, the problem happens less than 1% of time, so we will >>>>>> workaround the problem by implementing a retry mechanism. >>>>>> >>>>>> Issue #4 >>>>>> I have found a workaround that seems functional: >>>>>> When I wish to restart the streaming, I simply delete (reset) the >>>>>> multi_usrp object and redo the complete initialization. >>>>>> In fact, this is the workaround I was using with my "old UHD" >>>>>> version. Neel once told us this was not recommended, but its the only way I >>>>>> found to be able to restart the streaming without quitting the application. >>>>>> The original issue is still reproducible 100% of the time with >>>>>> benchmark_rate under Windows. >>>>>> >>>>>> Issue #6 >>>>>> (We had discussed this earlier): Spectrum deformation with the "old >>>>>> UHD" version. >>>>>> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >>>>>> version of December 1st 2016). >>>>>> >>>>>> Question A: >>>>>> The function "tx_streamer::get_max_num_samps" used to return 1994 >>>>>> samples, but now it returns only 364 samples. I still see "Determining >>>>>> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >>>>>> value (364 samples) with benchmark_rate. >>>>>> >>>>>> Question B: >>>>>> Creating the mutli_usrp object now takes 15-20 seconds, while it took >>>>>> around 10s with the "old UHD" version I had. Is there a way to reduce this >>>>>> initialization time? >>>>>> >>>>>> Thanks again for your help! >>>>>> Serge >>>>>> >>>>>> >>>>>> >>>>>> __________________ >>>>>> *Serge Malo * >>>>>> CDO & Co-founder, Skydel Solutions >>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>> www.skydelsolutions.com >>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>> >>>>>> On 9 December 2016 at 09:05, Serge Malo < >>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>> >>>>>>> Hello Michael, >>>>>>> >>>>>>> >>>>>>> Issue #5: >>>>>>> My bad! this is not an issue, I can use "head of maint" on Windows >>>>>>> now. >>>>>>> >>>>>>> Issue #1: >>>>>>> I confirm this is fixed with the "head of maint" UHD version on >>>>>>> Windows too. Great! >>>>>>> >>>>>>> Issues #2 and #3: >>>>>>> Unfortunately, I was able to reproduce the problem with "head of >>>>>>> maint" during my overnight on Ubuntu: >>>>>>> Issue #2 happened 4 times out of 948 executions >>>>>>> Issue #3 happened 9 times out of 948 executions >>>>>>> But under Windows 10, I had the next results: >>>>>>> Issue #2 happened 0 time out of 520 executions >>>>>>> Issue #3 happened 1 time out of 520 executions >>>>>>> I will try to reproduce the problem with another X300 and let you >>>>>>> know the results. >>>>>>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>>>>>> Can you confirm you don't see this problem at all ( 0 / 1000 >>>>>>> executions) on your side (this test can be done over one night). >>>>>>> >>>>>>> Issue #4 >>>>>>> The problem still happens with UHD "head of maint" under Windows. >>>>>>> Were you able to reproduce this on your side? >>>>>>> >>>>>>> Thanks again, >>>>>>> Serge >>>>>>> >>>>>>> >>>>>>> __________________ >>>>>>> *Serge Malo * >>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>> Cell: 1-514-294-4017 >>>>>>> www.skydelsolutions.com >>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>> >>>>>>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Serge, >>>>>>>> >>>>>>>> Since I do not have a unit here that I can use to reproduce issues >>>>>>>> #2 and #3, I am not able to debug them. The fact that the issues disappear >>>>>>>> when using the head of maint is encouraging, but I cannot explain why. If >>>>>>>> you need an answer as to why, I would need to have you ship the unit to me >>>>>>>> so I can try to reproduce the problems and debug them here. Please contact >>>>>>>> me off-list if we need to do that. >>>>>>>> >>>>>>>> Regarding Issue #5, the error is produced by UHD not being able to >>>>>>>> find the XML files defining the RFNoC blocks. The files may have been >>>>>>>> installed in the incorrect path (we are checking the Windows installers >>>>>>>> now). If you built and installed the 64-bit version of UHD, please check >>>>>>>> for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to >>>>>>>> C:\Program Files\UHD\share\uhd\rfnoc. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Michael >>>>>>>> >>>>>>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>> >>>>>>>>> Hello Michael, >>>>>>>>> >>>>>>>>> Here are more details and results from my side: >>>>>>>>> >>>>>>>>> Issue #1: >>>>>>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>>>>>> problem is fixed under Ubuntu. Great! >>>>>>>>> However, I'm facing a new issue under Windows 10 with this version >>>>>>>>> of UHD, I can't connect to the X300 at all, see issue #5 below. >>>>>>>>> >>>>>>>>> Issues #2 and #3 >>>>>>>>> -I have tried UHD 003.010.001.000 with internal clock: the >>>>>>>>> problems happened again at about the same rate (issue #2 1/100; issue #3 >>>>>>>>> 2/100). See attached result file "doit_res2_intclk.txt" >>>>>>>>> -I have tried UHD "head of maint": I did not reproduce the >>>>>>>>> problems with internal clock (0/100). >>>>>>>>> -I have tried UHD "head of maint": I did not reproduce the >>>>>>>>> problems with external clock (0/100). >>>>>>>>> -My external clock is a Wenzel OCXO >>>>>>>>> -I have only tried one X300 device so far. But some clients have >>>>>>>>> reported issue #2 with our older version of UHD. >>>>>>>>> So, issues "seem" to be fixed with the "head of maint" version. I >>>>>>>>> will run overnight tests to get more results. >>>>>>>>> >>>>>>>>> Issue #5: >>>>>>>>> I have recompiled UHD "head of maint" under Windows 10. When I use >>>>>>>>> benchmark_rate, I get the error message below. >>>>>>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but I >>>>>>>>> got the same error message. >>>>>>>>> >>>>>>>>> Creating the usrp device with: addr=192.168.60.2... >>>>>>>>> -- X300 initialization sequence... >>>>>>>>> -- Determining maximum frame size... 8000 bytes. >>>>>>>>> -- Setup basic communication... >>>>>>>>> -- Loading values from EEPROM... >>>>>>>>> -- Setup RF frontend clocking... >>>>>>>>> -- Radio 1x clock:200 >>>>>>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>>>>>> Error: AssertionError: block_def >>>>>>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>>>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>>>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>>>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>>>>>> >>>>>>>>> Do you see this problem on your side too? >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> Serge >>>>>>>>> >>>>>>>>> >>>>>>>>> __________________ >>>>>>>>> *Serge Malo * >>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>> www.skydelsolutions.com >>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>> >>>>>>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Serge, >>>>>>>>>> >>>>>>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have >>>>>>>>>> tried running the script on UHD 3.10.1.0 and the head of the maint branch >>>>>>>>>> without any issue. I am even using a rev 5 device to match the X300 >>>>>>>>>> hardware revision of your device. >>>>>>>>>> >>>>>>>>>> There are 2 things I am curious about: >>>>>>>>>> >>>>>>>>>> 1) What is the external reference being used? Do you see these >>>>>>>>>> failures when using the internal reference? >>>>>>>>>> 2) Do you see these failures on other devices or just this one >>>>>>>>>> device? >>>>>>>>>> >>>>>>>>>> I will be looking into issue #4 tomorrow. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Michael >>>>>>>>>> >>>>>>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks for the update Michael, >>>>>>>>>>> I will try the head of the maint branch on my side too as soon >>>>>>>>>>> as possible. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Serge >>>>>>>>>>> >>>>>>>>>>> __________________ >>>>>>>>>>> *Serge Malo * >>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>> >>>>>>>>>>> On 7 December 2016 at 13:29, Michael West < >>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Serge, >>>>>>>>>>>> >>>>>>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 >>>>>>>>>>>> release, but not with the head of the maint branch. So, I believe that one >>>>>>>>>>>> has been fixed. I am still working on reproducing the other 3 issues. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Michael >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Serge, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for the detailed information and examples. I am >>>>>>>>>>>>> working on reproducing the issues with the information you have provided. >>>>>>>>>>>>> I will let you know what I find. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Michael >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have started the process of upgrading the UHD version in >>>>>>>>>>>>>> our application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Configuration: >>>>>>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>>>>>> given by Derek) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>>>>>> When benchmark_rate process ends normally, I have to wait >>>>>>>>>>>>>> 5-10s before I restart it again, otherwise I get this error: >>>>>>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>>>>>> Device Address: >>>>>>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>>>>>> Q: At the end of a transmission, is there call or a poll we >>>>>>>>>>>>>> could make in order to make sure the radio is ready for a new process? In >>>>>>>>>>>>>> our old UHD version, this was not happening. >>>>>>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync >>>>>>>>>>>>>> failed. unexpected FIFO depth [0x1f]" >>>>>>>>>>>>>> This error happens only about 1% of the time, but I was able >>>>>>>>>>>>>> to reproduce it with a simple script calling "benchmark_rate" in a loop >>>>>>>>>>>>>> under Ubuntu. >>>>>>>>>>>>>> See attached script "doit.sh". >>>>>>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See >>>>>>>>>>>>>> attached result file "doit_res.txt") >>>>>>>>>>>>>> We have clients and partners using our application with the >>>>>>>>>>>>>> older version of UHD who reported this error happening about 1% of the >>>>>>>>>>>>>> time. The issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to >>>>>>>>>>>>>> lock to external source. >>>>>>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>>>>>> message on the 2nd loop: >>>>>>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>>> user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>>> sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) >>>>>>>>>>>>>> no response packet - AssertionError: buff->size() > 0 >>>>>>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>>>>>> bool) >>>>>>>>>>>>>> at ...\uhd_003_010_001_000-Patch1 >>>>>>>>>>>>>> \host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>>>>>>>>>> >>>>>>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under >>>>>>>>>>>>>> Ubuntu, I get a sequence error on the 2nd loop, but all samples are >>>>>>>>>>>>>> transmitted (no error message) >>>>>>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>>>>>> Please, let me know if there is better/cleaner way to do this >>>>>>>>>>>>>> "repeat loop". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks in advance for your help! >>>>>>>>>>>>>> Serge >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> __________________ >>>>>>>>>>>>>> *Serge Malo * >>>>>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> USRP-users mailing list >>>>>>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>>>>>> us.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
MW
Michael West
Thu, Dec 15, 2016 6:56 PM

Hi Serge,

The TX performance is a major item being worked on right now for the UHD
3.10.x.x code base.  The UHD-3.9.LTS branch can do 2 TX channels at 100
Msps each, but it usually takes some tuning of system parameters (like you
have done) to get there.  The addition of the DRAM in the UHD 3.10.x.x is
supposed to ease that, but we are still working out the kinks.  I have been
told that the latest unreleased changes support 2xTX up to 50 Msps, so the
next release of UHD 3.10.x.x will achieve at least that.

Regards,
Michael

On Thu, Dec 15, 2016 at 10:21 AM, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Issue #4:
Thanks for the additional information.  My testing on Ubuntu yields the
same result as yours.  I will do more testing on Windows and see if I can
reproduce the error and track down the root cause (I'm currently suspecting
something with the udp_wsa_zero_copy transport, since that is used only in
Windows).  The sequence error on Ubuntu is a known issue and we are working
on it.  I'm glad you have a good method to use for your application.  We
will continue to work on fixing the issues you have pointed out.

Issue "Maximum number of samples":
The fix is not yet complete for this.  But the lower packet size does not
impact performance in any way if the sample rate is <=50 Msps, so it will
only be an issue if your application needs a TX sample rate greater than
that.

Thanks again for all the great feedback!

Regards,
Michael

On Wed, Dec 14, 2016 at 5:42 PM, Serge Malo <serge.malo@skydelsolutions.
com> wrote:

Hello Michael,

Issue #4:
Here's what I observed:
Windows, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Error
Windows, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!
Ubuntu, restarting the streaming by re-creating the streamer (as the
modified benchmark_rate.cpp I provided): Ok (but Sequence Error)
Ubuntu, restarting the streaming by re-creating the multi_usrp and then
the streamer: Ok!

I will use the solution "re-creating the multi_usrp and then the
streamer" in my application. This means we have to go through the init
sequence every time we start the streaming.

Let me know if I can help by running other tests.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 14 December 2016 at 13:37, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Issue #4:
That is progress.  It looks like the radio control responses are now
getting received with the larger socket buffer, but somewhere along the
line the firmware control transport is having issues.  To confirm, the
error is only on Windows and Ubuntu works fine, right?  Also, can you check
to see if you see the issue if you modify the code to use a single
multi_usrp object instead of re-creating it?

Issue "Maximum number of samples":
The fix requires a new FPGA image, so it is not likely to be available
until the next release.  I will see what I can do to get you early access
if possible.

Issue "initialization time":
Believe me, we all want a shorter initialization time.  I'm sure it will
improve over time.

Regards,
Michael

On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Issue #4:
I have tried to increase the socket buffer size (with the
send_buff_size argument), but I'm getting the next error (see complete
output attached):
UU
UHD Error:
x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out

Issue "Maximum number of samples":
I'm worried that this will negatively impact the performance of the Tx
streaming (causing more frequent under-runs), but I have not tested enough
yet.
You said it should be available in the next UHD release. Let me know if
this could be fixed in the "maint" branch, so I could pull this earlier.
Do you have a timeframe for the next UHD version?

Issue "Initialization time":
This is not a bug issue for us, but any improvement in the next UHD
release will be appreciated.

Thanks,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 12 December 2016 at 18:07, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Regarding Issue #4, the sequence error is a known issue and is
expected.  I believe the failure on Windows may be due to the default
socket buffer size being too small.  Try increasing the default receive
socket buffer size (to 2MB) and let me know if you still see the failures.

Regards,
Michael

On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com

wrote:

Hi Serge,

I will set up longer term tests for issues #2 and #3 to see if I can
reproduce and debug on my end.  Issue #4 seems like a legitimate bug, so
let me spend some time digging into that this week.

Regarding your questions, they are known issues that are currently
being addressed.  The maximum number of samples issue should be fixed in
the next release of UHD.  The slower initialization is being looked at, but
may take more time to completely resolve.  Please let me know if it is
significantly impacting your application and I can see what I can do to
elevate the priority.

Regards,
Michael

On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

I have some good news today, and 2 outstanding questions at the end.

Issue #2:
we have tested with 2 other X300 under Windows, and the problem did
not show up.
So, on our side, the problem only happens 0.5% of the time, with one
device, under Ubuntu.
We will implement a "retry" mechanism to workaround this problem,
and see with our clients if this is Ok with them.
If they have a X300 with which the issue happens more frequently, we
will put them in touch with you.

Issue #3
We have tested with 2 other X300 and an octoclock-G under Windows,
and the problem showed up 4/1000 for one device and 0/1000 for the other.
Again, the problem happens less than 1% of time, so we will
workaround the problem by implementing a retry mechanism.

Issue #4
I have found a workaround that seems functional:
When I wish to restart the streaming, I simply delete (reset) the
multi_usrp object and redo the complete initialization.
In fact, this is the workaround I was using with my "old UHD"
version. Neel once told us this was not recommended, but its the only way I
found to be able to restart the streaming without quitting the application.
The original issue is still reproducible 100% of the time with
benchmark_rate under Windows.

Issue #6
(We had discussed this earlier): Spectrum deformation with the "old
UHD" version.
I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch,
version of December 1st 2016).

Question A:
The function "tx_streamer::get_max_num_samps" used to return 1994
samples, but now it returns only 364 samples. I still see "Determining
frame size... 8000 bytes" in UHD's output. Is this expected? I see the same
value (364 samples) with benchmark_rate.

Question B:
Creating the mutli_usrp object now takes 15-20 seconds, while it
took around 10s with the "old UHD" version I had. Is there a way to reduce
this initialization time?

Thanks again for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 9 December 2016 at 09:05, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Issue #5:
My bad! this is not an issue, I can use "head of maint" on Windows
now.

Issue #1:
I confirm this is fixed with the "head of maint" UHD version on
Windows too. Great!

Issues #2 and #3:
Unfortunately, I was able to reproduce the problem with "head of
maint" during my overnight on Ubuntu:
Issue #2 happened 4 times out of 948 executions
Issue #3 happened 9 times out of 948 executions
But under Windows 10, I had the next results:
Issue #2 happened 0 time out of 520 executions
Issue #3 happened 1 time out of 520 executions
I will try to reproduce the problem with another X300 and let you
know the results.
I will also try to see if the issue #3 occurs with an Octoclock-G.
Can you confirm you don't see this problem at all ( 0 / 1000
executions) on your side (this test can be done over one night).

Issue #4
The problem still happens with UHD "head of maint" under Windows.
Were you able to reproduce this on your side?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 8 December 2016 at 17:42, Michael West michael.west@ettus.com
wrote:

Hi Serge,

Since I do not have a unit here that I can use to reproduce issues
#2 and #3, I am not able to debug them.  The fact that the issues disappear
when using the head of maint is encouraging, but I cannot explain why.  If
you need an answer as to why, I would need to have you ship the unit to me
so I can try to reproduce the problems and debug them here.  Please contact
me off-list if we need to do that.

Regarding Issue #5, the error is produced by UHD not being able to
find the XML files defining the RFNoC blocks.  The files may have been
installed in the incorrect path (we are checking the Windows installers
now).  If you built and installed the 64-bit version of UHD, please check
for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to
C:\Program Files\UHD\share\uhd\rfnoc.

Regards,
Michael

On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Hello Michael,

Here are more details and results from my side:

Issue #1:
I have recompiled UHD with  "Head of the maint branch", and the
problem is fixed under Ubuntu. Great!
However, I'm facing a new issue under Windows 10 with this
version of UHD, I can't connect to the X300 at all, see issue #5 below.

Issues #2 and #3
-I have tried UHD 003.010.001.000 with internal clock: the
problems happened again at about the same rate (issue #2 1/100; issue #3
2/100). See attached result file "doit_res2_intclk.txt"
-I have tried UHD "head of maint": I did not reproduce the
problems with internal clock (0/100).
-I have tried UHD "head of maint": I did not reproduce the
problems with external clock (0/100).
-My external clock is a Wenzel OCXO
-I have only tried one X300 device so far. But some clients have
reported issue #2 with our older version of UHD.
So, issues "seem" to be fixed with the "head of maint" version. I
will run overnight tests to get more results.

Issue #5:
I have recompiled UHD "head of maint" under Windows 10. When I
use benchmark_rate, I get the error message below.
I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but
I got the same error message.

Creating the usrp device with: addr=192.168.60.2...
-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- Creating WSA UDP transport for 192.168.60.2:49153
Error: AssertionError: block_def
in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned
__int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class
uhd::device_addr_t,enum uhd::endianness_t)
at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149

Do you see this problem on your side too?

Thanks again,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com

wrote:

Hi Serge,

I cannot seem to reproduce the issue #2 or issue #3.  I have
tried running the script on UHD 3.10.1.0 and the head of the maint branch
without any issue.  I am even using a rev 5 device to match the X300
hardware revision of your device.

There are 2 things I am curious about:

  1. What is the external reference being used?  Do you see these
    failures when using the internal reference?
  2. Do you see these failures on other devices or just this one
    device?

I will be looking into issue #4 tomorrow.

Regards,
Michael

On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo <
serge.malo@skydelsolutions.com> wrote:

Thanks for the update Michael,
I will try the head of the maint branch on my side too as soon
as possible.

Regards,
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol

On 7 December 2016 at 13:29, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

I was able to reproduce issue #1 with the tagged 3.10.1.0
release, but not with the head of the maint branch.  So, I believe that one
has been fixed.  I am still working on reproducing the other 3 issues.

Regards,
Michael

On Tue, Dec 6, 2016 at 3:11 PM, Michael West <
michael.west@ettus.com> wrote:

Hi Serge,

Thank you for the detailed information and examples.  I am
working on reproducing the issues with the information you have provided.
I will let you know what I find.

Regards,
Michael

On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users <
usrp-users@lists.ettus.com> wrote:

Hi all,

We have started the process of upgrading the UHD version in
our application to UHD-003-010-001-000, but we are facing several issues.
I have reproduced those issues with the UHD example
"benchmark_rate", so it will be easier to find solutions or workarounds.

Configuration:
O/S: Windows 10 and Ubuntu 14.04LTS.
X300 P/N: 156485E-1DL S/N: 30DB7E8
FPGA image: usrp_x300_fpga_HG.bit
10GbE connection with Intel X520-DA2 card.
CPU: Intel core i7 4790K.
I have recompiled UHD myself from the tag
release_003_010_001_000, and added a patch to fix the timed commands (patch
given by Derek)

Issue #1: Can't find X300 when restarting application.
When benchmark_rate process ends normally, I have to wait
5-10s before I restart it again, otherwise I get this error:
Error: LookupError: KeyError: No devices found for ----->
Device Address:
addr: 192.168.40.2
Q: At the end of a transmission, is there call or a poll we
could make in order to make sure the radio is ready for a new process? In
our old UHD version, this was not happening.
This issue happens on Windows and Ubuntu.

Issue #2: Error message "x300_dac_ctrl: front-end sync
failed. unexpected FIFO depth [0x1f]"
This error happens only about 1% of the time, but I was able
to reproduce it with a simple script calling "benchmark_rate" in a loop
under Ubuntu.
See attached script "doit.sh".
In my case, the error occurred 1/100 at Loop #23 (See
attached result file "doit_res.txt")
We have clients and partners using our application with the
older version of UHD who reported this error happening about 1% of the
time. The issue seems to still be present in UHD 3.10.1.0.

Issue #3: Error: RuntimeError: Reference Clock PLL failed to
lock to external source.
This error happened 3 times out of my 100 iteration test.
See attached result file "doit_res.txt"

Issue #4: Can't restart the streaming within a process
I have tried to add a loop inside the source code of
benchmark_rate to execute the test several times. However, I get this error
message on the 2nd loop:
Error: EnvironmentError: IOError: [0/Radio_0]
user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0]
sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40)
no response packet - AssertionError: buff->size() > 0
in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const
bool)
at ...\uhd_003_010_001_000-Patch1
\host\lib\rfnoc\ctrl_iface.cpp:206

NOTE: This issue #4 only happens under Windows 10. Under
Ubuntu, I get a sequence error on the 2nd loop, but all samples are
transmitted (no error message)
See attached benchmark_rate_modified.cpp
Please, let me know if there is better/cleaner way to do
this "repeat loop".

Thanks in advance for your help!
Serge


*Serge Malo *
CDO & Co-founder, Skydel Solutions
Cell: 1-514-294-4017 <(514)%20294-4017>
www.skydelsolutions.com
Twitter: @skydelsol https://twitter.com/skydelsol


USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett
us.com

Hi Serge, The TX performance is a major item being worked on right now for the UHD 3.10.x.x code base. The UHD-3.9.LTS branch can do 2 TX channels at 100 Msps each, but it usually takes some tuning of system parameters (like you have done) to get there. The addition of the DRAM in the UHD 3.10.x.x is supposed to ease that, but we are still working out the kinks. I have been told that the latest unreleased changes support 2xTX up to 50 Msps, so the next release of UHD 3.10.x.x will achieve at least that. Regards, Michael On Thu, Dec 15, 2016 at 10:21 AM, Michael West <michael.west@ettus.com> wrote: > Hi Serge, > > Issue #4: > Thanks for the additional information. My testing on Ubuntu yields the > same result as yours. I will do more testing on Windows and see if I can > reproduce the error and track down the root cause (I'm currently suspecting > something with the udp_wsa_zero_copy transport, since that is used only in > Windows). The sequence error on Ubuntu is a known issue and we are working > on it. I'm glad you have a good method to use for your application. We > will continue to work on fixing the issues you have pointed out. > > Issue "Maximum number of samples": > The fix is not yet complete for this. But the lower packet size does not > impact performance in any way if the sample rate is <=50 Msps, so it will > only be an issue if your application needs a TX sample rate greater than > that. > > Thanks again for all the great feedback! > > Regards, > Michael > > On Wed, Dec 14, 2016 at 5:42 PM, Serge Malo <serge.malo@skydelsolutions. > com> wrote: > >> Hello Michael, >> >> Issue #4: >> Here's what I observed: >> Windows, restarting the streaming by re-creating the streamer (as the >> modified benchmark_rate.cpp I provided): Error >> Windows, restarting the streaming by re-creating the multi_usrp and then >> the streamer: Ok! >> Ubuntu, restarting the streaming by re-creating the streamer (as the >> modified benchmark_rate.cpp I provided): Ok (but Sequence Error) >> Ubuntu, restarting the streaming by re-creating the multi_usrp and then >> the streamer: Ok! >> >> I will use the solution "re-creating the multi_usrp and then the >> streamer" in my application. This means we have to go through the init >> sequence every time we start the streaming. >> >> Let me know if I can help by running other tests. >> >> Thanks, >> Serge >> >> >> __________________ >> *Serge Malo * >> CDO & Co-founder, Skydel Solutions >> Cell: 1-514-294-4017 <(514)%20294-4017> >> www.skydelsolutions.com >> Twitter: @skydelsol <https://twitter.com/skydelsol> >> >> On 14 December 2016 at 13:37, Michael West <michael.west@ettus.com> >> wrote: >> >>> Hi Serge, >>> >>> Issue #4: >>> That is progress. It looks like the radio control responses are now >>> getting received with the larger socket buffer, but somewhere along the >>> line the firmware control transport is having issues. To confirm, the >>> error is only on Windows and Ubuntu works fine, right? Also, can you check >>> to see if you see the issue if you modify the code to use a single >>> multi_usrp object instead of re-creating it? >>> >>> Issue "Maximum number of samples": >>> The fix requires a new FPGA image, so it is not likely to be available >>> until the next release. I will see what I can do to get you early access >>> if possible. >>> >>> Issue "initialization time": >>> Believe me, we all want a shorter initialization time. I'm sure it will >>> improve over time. >>> >>> Regards, >>> Michael >>> >>> On Wed, Dec 14, 2016 at 7:47 AM, Serge Malo < >>> serge.malo@skydelsolutions.com> wrote: >>> >>>> Hello Michael, >>>> >>>> Issue #4: >>>> I have tried to increase the socket buffer size (with the >>>> send_buff_size argument), but I'm getting the next error (see complete >>>> output attached): >>>> UU >>>> UHD Error: >>>> x300 fw communication failure #1 >>>> EnvironmentError: IOError: x300 fw poke32 - reply timed out >>>> >>>> Issue "Maximum number of samples": >>>> I'm worried that this will negatively impact the performance of the Tx >>>> streaming (causing more frequent under-runs), but I have not tested enough >>>> yet. >>>> You said it should be available in the next UHD release. Let me know if >>>> this could be fixed in the "maint" branch, so I could pull this earlier. >>>> Do you have a timeframe for the next UHD version? >>>> >>>> Issue "Initialization time": >>>> This is not a bug issue for us, but any improvement in the next UHD >>>> release will be appreciated. >>>> >>>> Thanks, >>>> Serge >>>> >>>> >>>> >>>> __________________ >>>> *Serge Malo * >>>> CDO & Co-founder, Skydel Solutions >>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>> www.skydelsolutions.com >>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>> >>>> On 12 December 2016 at 18:07, Michael West <michael.west@ettus.com> >>>> wrote: >>>> >>>>> Hi Serge, >>>>> >>>>> Regarding Issue #4, the sequence error is a known issue and is >>>>> expected. I believe the failure on Windows may be due to the default >>>>> socket buffer size being too small. Try increasing the default receive >>>>> socket buffer size (to 2MB) and let me know if you still see the failures. >>>>> >>>>> Regards, >>>>> Michael >>>>> >>>>> On Mon, Dec 12, 2016 at 11:10 AM, Michael West <michael.west@ettus.com >>>>> > wrote: >>>>> >>>>>> Hi Serge, >>>>>> >>>>>> I will set up longer term tests for issues #2 and #3 to see if I can >>>>>> reproduce and debug on my end. Issue #4 seems like a legitimate bug, so >>>>>> let me spend some time digging into that this week. >>>>>> >>>>>> Regarding your questions, they are known issues that are currently >>>>>> being addressed. The maximum number of samples issue should be fixed in >>>>>> the next release of UHD. The slower initialization is being looked at, but >>>>>> may take more time to completely resolve. Please let me know if it is >>>>>> significantly impacting your application and I can see what I can do to >>>>>> elevate the priority. >>>>>> >>>>>> Regards, >>>>>> Michael >>>>>> >>>>>> On Mon, Dec 12, 2016 at 10:55 AM, Serge Malo < >>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>> >>>>>>> Hello Michael, >>>>>>> >>>>>>> I have some good news today, and 2 outstanding questions at the end. >>>>>>> >>>>>>> Issue #2: >>>>>>> we have tested with 2 other X300 under Windows, and the problem did >>>>>>> not show up. >>>>>>> So, on our side, the problem only happens 0.5% of the time, with one >>>>>>> device, under Ubuntu. >>>>>>> We will implement a "retry" mechanism to workaround this problem, >>>>>>> and see with our clients if this is Ok with them. >>>>>>> If they have a X300 with which the issue happens more frequently, we >>>>>>> will put them in touch with you. >>>>>>> >>>>>>> Issue #3 >>>>>>> We have tested with 2 other X300 and an octoclock-G under Windows, >>>>>>> and the problem showed up 4/1000 for one device and 0/1000 for the other. >>>>>>> Again, the problem happens less than 1% of time, so we will >>>>>>> workaround the problem by implementing a retry mechanism. >>>>>>> >>>>>>> Issue #4 >>>>>>> I have found a workaround that seems functional: >>>>>>> When I wish to restart the streaming, I simply delete (reset) the >>>>>>> multi_usrp object and redo the complete initialization. >>>>>>> In fact, this is the workaround I was using with my "old UHD" >>>>>>> version. Neel once told us this was not recommended, but its the only way I >>>>>>> found to be able to restart the streaming without quitting the application. >>>>>>> The original issue is still reproducible 100% of the time with >>>>>>> benchmark_rate under Windows. >>>>>>> >>>>>>> Issue #6 >>>>>>> (We had discussed this earlier): Spectrum deformation with the "old >>>>>>> UHD" version. >>>>>>> I confirm that this problem is fixed in UHD 3.10.1.0 (maint branch, >>>>>>> version of December 1st 2016). >>>>>>> >>>>>>> Question A: >>>>>>> The function "tx_streamer::get_max_num_samps" used to return 1994 >>>>>>> samples, but now it returns only 364 samples. I still see "Determining >>>>>>> frame size... 8000 bytes" in UHD's output. Is this expected? I see the same >>>>>>> value (364 samples) with benchmark_rate. >>>>>>> >>>>>>> Question B: >>>>>>> Creating the mutli_usrp object now takes 15-20 seconds, while it >>>>>>> took around 10s with the "old UHD" version I had. Is there a way to reduce >>>>>>> this initialization time? >>>>>>> >>>>>>> Thanks again for your help! >>>>>>> Serge >>>>>>> >>>>>>> >>>>>>> >>>>>>> __________________ >>>>>>> *Serge Malo * >>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>> www.skydelsolutions.com >>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>> >>>>>>> On 9 December 2016 at 09:05, Serge Malo < >>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>> >>>>>>>> Hello Michael, >>>>>>>> >>>>>>>> >>>>>>>> Issue #5: >>>>>>>> My bad! this is not an issue, I can use "head of maint" on Windows >>>>>>>> now. >>>>>>>> >>>>>>>> Issue #1: >>>>>>>> I confirm this is fixed with the "head of maint" UHD version on >>>>>>>> Windows too. Great! >>>>>>>> >>>>>>>> Issues #2 and #3: >>>>>>>> Unfortunately, I was able to reproduce the problem with "head of >>>>>>>> maint" during my overnight on Ubuntu: >>>>>>>> Issue #2 happened 4 times out of 948 executions >>>>>>>> Issue #3 happened 9 times out of 948 executions >>>>>>>> But under Windows 10, I had the next results: >>>>>>>> Issue #2 happened 0 time out of 520 executions >>>>>>>> Issue #3 happened 1 time out of 520 executions >>>>>>>> I will try to reproduce the problem with another X300 and let you >>>>>>>> know the results. >>>>>>>> I will also try to see if the issue #3 occurs with an Octoclock-G. >>>>>>>> Can you confirm you don't see this problem at all ( 0 / 1000 >>>>>>>> executions) on your side (this test can be done over one night). >>>>>>>> >>>>>>>> Issue #4 >>>>>>>> The problem still happens with UHD "head of maint" under Windows. >>>>>>>> Were you able to reproduce this on your side? >>>>>>>> >>>>>>>> Thanks again, >>>>>>>> Serge >>>>>>>> >>>>>>>> >>>>>>>> __________________ >>>>>>>> *Serge Malo * >>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>> Cell: 1-514-294-4017 >>>>>>>> www.skydelsolutions.com >>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>> >>>>>>>> On 8 December 2016 at 17:42, Michael West <michael.west@ettus.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Serge, >>>>>>>>> >>>>>>>>> Since I do not have a unit here that I can use to reproduce issues >>>>>>>>> #2 and #3, I am not able to debug them. The fact that the issues disappear >>>>>>>>> when using the head of maint is encouraging, but I cannot explain why. If >>>>>>>>> you need an answer as to why, I would need to have you ship the unit to me >>>>>>>>> so I can try to reproduce the problems and debug them here. Please contact >>>>>>>>> me off-list if we need to do that. >>>>>>>>> >>>>>>>>> Regarding Issue #5, the error is produced by UHD not being able to >>>>>>>>> find the XML files defining the RFNoC blocks. The files may have been >>>>>>>>> installed in the incorrect path (we are checking the Windows installers >>>>>>>>> now). If you built and installed the 64-bit version of UHD, please check >>>>>>>>> for the folder C:\Program Files(x86)\UHD\share\uhd\rfnoc and move it to >>>>>>>>> C:\Program Files\UHD\share\uhd\rfnoc. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Michael >>>>>>>>> >>>>>>>>> On Thu, Dec 8, 2016 at 9:44 AM, Serge Malo < >>>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>>> >>>>>>>>>> Hello Michael, >>>>>>>>>> >>>>>>>>>> Here are more details and results from my side: >>>>>>>>>> >>>>>>>>>> Issue #1: >>>>>>>>>> I have recompiled UHD with "Head of the maint branch", and the >>>>>>>>>> problem is fixed under Ubuntu. Great! >>>>>>>>>> However, I'm facing a new issue under Windows 10 with this >>>>>>>>>> version of UHD, I can't connect to the X300 at all, see issue #5 below. >>>>>>>>>> >>>>>>>>>> Issues #2 and #3 >>>>>>>>>> -I have tried UHD 003.010.001.000 with internal clock: the >>>>>>>>>> problems happened again at about the same rate (issue #2 1/100; issue #3 >>>>>>>>>> 2/100). See attached result file "doit_res2_intclk.txt" >>>>>>>>>> -I have tried UHD "head of maint": I did not reproduce the >>>>>>>>>> problems with internal clock (0/100). >>>>>>>>>> -I have tried UHD "head of maint": I did not reproduce the >>>>>>>>>> problems with external clock (0/100). >>>>>>>>>> -My external clock is a Wenzel OCXO >>>>>>>>>> -I have only tried one X300 device so far. But some clients have >>>>>>>>>> reported issue #2 with our older version of UHD. >>>>>>>>>> So, issues "seem" to be fixed with the "head of maint" version. I >>>>>>>>>> will run overnight tests to get more results. >>>>>>>>>> >>>>>>>>>> Issue #5: >>>>>>>>>> I have recompiled UHD "head of maint" under Windows 10. When I >>>>>>>>>> use benchmark_rate, I get the error message below. >>>>>>>>>> I have tried MSVC 2013+Boost 1.57, and MSVC 2015+Boost 1.62, but >>>>>>>>>> I got the same error message. >>>>>>>>>> >>>>>>>>>> Creating the usrp device with: addr=192.168.60.2... >>>>>>>>>> -- X300 initialization sequence... >>>>>>>>>> -- Determining maximum frame size... 8000 bytes. >>>>>>>>>> -- Setup basic communication... >>>>>>>>>> -- Loading values from EEPROM... >>>>>>>>>> -- Setup RF frontend clocking... >>>>>>>>>> -- Radio 1x clock:200 >>>>>>>>>> -- Creating WSA UDP transport for 192.168.60.2:49153 >>>>>>>>>> Error: AssertionError: block_def >>>>>>>>>> in void __cdecl uhd::usrp::device3_impl::enumerate_rfnoc_blocks(unsigned >>>>>>>>>> __int64,unsigned __int64,unsigned __int64,const class uhd::sid_t &,class >>>>>>>>>> uhd::device_addr_t,enum uhd::endianness_t) >>>>>>>>>> at ...\uhd-maint\host\lib\usrp\device3\device3_impl.cpp:149 >>>>>>>>>> >>>>>>>>>> Do you see this problem on your side too? >>>>>>>>>> >>>>>>>>>> Thanks again, >>>>>>>>>> Serge >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> __________________ >>>>>>>>>> *Serge Malo * >>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>> www.skydelsolutions.com >>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>> >>>>>>>>>> On 7 December 2016 at 19:56, Michael West <michael.west@ettus.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi Serge, >>>>>>>>>>> >>>>>>>>>>> I cannot seem to reproduce the issue #2 or issue #3. I have >>>>>>>>>>> tried running the script on UHD 3.10.1.0 and the head of the maint branch >>>>>>>>>>> without any issue. I am even using a rev 5 device to match the X300 >>>>>>>>>>> hardware revision of your device. >>>>>>>>>>> >>>>>>>>>>> There are 2 things I am curious about: >>>>>>>>>>> >>>>>>>>>>> 1) What is the external reference being used? Do you see these >>>>>>>>>>> failures when using the internal reference? >>>>>>>>>>> 2) Do you see these failures on other devices or just this one >>>>>>>>>>> device? >>>>>>>>>>> >>>>>>>>>>> I will be looking into issue #4 tomorrow. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Michael >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 7, 2016 at 12:35 PM, Serge Malo < >>>>>>>>>>> serge.malo@skydelsolutions.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks for the update Michael, >>>>>>>>>>>> I will try the head of the maint branch on my side too as soon >>>>>>>>>>>> as possible. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Serge >>>>>>>>>>>> >>>>>>>>>>>> __________________ >>>>>>>>>>>> *Serge Malo * >>>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>>> >>>>>>>>>>>> On 7 December 2016 at 13:29, Michael West < >>>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Serge, >>>>>>>>>>>>> >>>>>>>>>>>>> I was able to reproduce issue #1 with the tagged 3.10.1.0 >>>>>>>>>>>>> release, but not with the head of the maint branch. So, I believe that one >>>>>>>>>>>>> has been fixed. I am still working on reproducing the other 3 issues. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Michael >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 6, 2016 at 3:11 PM, Michael West < >>>>>>>>>>>>> michael.west@ettus.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Serge, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for the detailed information and examples. I am >>>>>>>>>>>>>> working on reproducing the issues with the information you have provided. >>>>>>>>>>>>>> I will let you know what I find. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Michael >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Dec 6, 2016 at 11:39 AM, Serge Malo via USRP-users < >>>>>>>>>>>>>> usrp-users@lists.ettus.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have started the process of upgrading the UHD version in >>>>>>>>>>>>>>> our application to UHD-003-010-001-000, but we are facing several issues. >>>>>>>>>>>>>>> I have reproduced those issues with the UHD example >>>>>>>>>>>>>>> "benchmark_rate", so it will be easier to find solutions or workarounds. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Configuration: >>>>>>>>>>>>>>> O/S: Windows 10 and Ubuntu 14.04LTS. >>>>>>>>>>>>>>> X300 P/N: 156485E-1DL S/N: 30DB7E8 >>>>>>>>>>>>>>> FPGA image: usrp_x300_fpga_HG.bit >>>>>>>>>>>>>>> 10GbE connection with Intel X520-DA2 card. >>>>>>>>>>>>>>> CPU: Intel core i7 4790K. >>>>>>>>>>>>>>> I have recompiled UHD myself from the tag >>>>>>>>>>>>>>> release_003_010_001_000, and added a patch to fix the timed commands (patch >>>>>>>>>>>>>>> given by Derek) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issue #1: Can't find X300 when restarting application. >>>>>>>>>>>>>>> When benchmark_rate process ends normally, I have to wait >>>>>>>>>>>>>>> 5-10s before I restart it again, otherwise I get this error: >>>>>>>>>>>>>>> Error: LookupError: KeyError: No devices found for -----> >>>>>>>>>>>>>>> Device Address: >>>>>>>>>>>>>>> addr: 192.168.40.2 >>>>>>>>>>>>>>> Q: At the end of a transmission, is there call or a poll we >>>>>>>>>>>>>>> could make in order to make sure the radio is ready for a new process? In >>>>>>>>>>>>>>> our old UHD version, this was not happening. >>>>>>>>>>>>>>> This issue happens on Windows and Ubuntu. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issue #2: Error message "x300_dac_ctrl: front-end sync >>>>>>>>>>>>>>> failed. unexpected FIFO depth [0x1f]" >>>>>>>>>>>>>>> This error happens only about 1% of the time, but I was able >>>>>>>>>>>>>>> to reproduce it with a simple script calling "benchmark_rate" in a loop >>>>>>>>>>>>>>> under Ubuntu. >>>>>>>>>>>>>>> See attached script "doit.sh". >>>>>>>>>>>>>>> In my case, the error occurred 1/100 at Loop #23 (See >>>>>>>>>>>>>>> attached result file "doit_res.txt") >>>>>>>>>>>>>>> We have clients and partners using our application with the >>>>>>>>>>>>>>> older version of UHD who reported this error happening about 1% of the >>>>>>>>>>>>>>> time. The issue seems to still be present in UHD 3.10.1.0. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issue #3: Error: RuntimeError: Reference Clock PLL failed to >>>>>>>>>>>>>>> lock to external source. >>>>>>>>>>>>>>> This error happened 3 times out of my 100 iteration test. >>>>>>>>>>>>>>> See attached result file "doit_res.txt" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issue #4: Can't restart the streaming within a process >>>>>>>>>>>>>>> I have tried to add a loop inside the source code of >>>>>>>>>>>>>>> benchmark_rate to execute the test several times. However, I get this error >>>>>>>>>>>>>>> message on the 2nd loop: >>>>>>>>>>>>>>> Error: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>>>> user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] >>>>>>>>>>>>>>> sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) >>>>>>>>>>>>>>> no response packet - AssertionError: buff->size() > 0 >>>>>>>>>>>>>>> in unsigned __int64 __cdecl ctrl_iface_impl::wait_for_ack(const >>>>>>>>>>>>>>> bool) >>>>>>>>>>>>>>> at ...\uhd_003_010_001_000-Patch1 >>>>>>>>>>>>>>> \host\lib\rfnoc\ctrl_iface.cpp:206 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> NOTE: This issue #4 only happens under Windows 10. Under >>>>>>>>>>>>>>> Ubuntu, I get a sequence error on the 2nd loop, but all samples are >>>>>>>>>>>>>>> transmitted (no error message) >>>>>>>>>>>>>>> See attached benchmark_rate_modified.cpp >>>>>>>>>>>>>>> Please, let me know if there is better/cleaner way to do >>>>>>>>>>>>>>> this "repeat loop". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks in advance for your help! >>>>>>>>>>>>>>> Serge >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> __________________ >>>>>>>>>>>>>>> *Serge Malo * >>>>>>>>>>>>>>> CDO & Co-founder, Skydel Solutions >>>>>>>>>>>>>>> Cell: 1-514-294-4017 <(514)%20294-4017> >>>>>>>>>>>>>>> www.skydelsolutions.com >>>>>>>>>>>>>>> Twitter: @skydelsol <https://twitter.com/skydelsol> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> USRP-users mailing list >>>>>>>>>>>>>>> USRP-users@lists.ettus.com >>>>>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ett >>>>>>>>>>>>>>> us.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >