[USRP-users] Receiving IO Block Error when Using RFNoC Split Stream

EJ Kreinar ejkreinar at gmail.com
Thu Nov 14 18:11:38 EST 2019


Cool!

I'll point out that your new build actually got past the old failure point
you noted before, which was immediately after attempting to interact with
the SplitStream RFNOC block!

```
[INFO] [0/SplitStream_0] Initializing block control (NOC ID:
0x5757000000000000)
[ERROR] [UHD] Exception caught in safe-call.
[...etc...]
```

I second Nick's thought. Try adding a FIFO after the second split-stream
port.

Though on further inspection, I actually doubt this particular application
would work at all, since it looks like you're trying to push 56 MHz
through the RFNoC FPGA->ARM transport and then through ZMQ. The E310
definitely cannot handle that sort of load, and you might get undefined
behavior in RFNOC fosphor because the underflow will propagate back to the
RFNoC Radio and momentarily stop the RF stream before restarting...

EJ

On Thu, Nov 14, 2019 at 6:05 PM Nick Foster <bistromath at gmail.com> wrote:

> I also haven't tested to see if this is a gr-ettus limitation or a UHD
> limitation.
>
> On Thu, Nov 14, 2019 at 3:04 PM Nick Foster <bistromath at gmail.com> wrote:
>
>> You can't have heterogenous output destinations on RFNoC blocks -- i.e.,
>> you can't send one output to the host and one output to another RFNoC block.
>>
>> I'm not sure why this limitation exists as there is no architectural
>> reason I can see.
>>
>> Nick
>>
>> On Thu, Nov 14, 2019 at 12:35 PM Jonathan Lockhart <jlockhartrt at gmail.com>
>> wrote:
>>
>>> Greetings EJ,
>>>
>>> So went and dug out the E312 b/c I couldn't wait and unfortunately that
>>> didn't resolve the issue I was experiencing. I cherry picked the new
>>> split_stream block, I am using the same flow graph as provided before, but
>>> when it goes to run on the E312 I get the following error.
>>>
>>> Traceback (most recent call last):
>>>   File "rfnoc_fosphor_radio_network_usrp.py", line 281, in <module>
>>>     main()
>>>   File "rfnoc_fosphor_radio_network_usrp.py", line 271, in main
>>>     tb.start()
>>>   File
>>> "/home/root/localinstall/usr/lib/python2.7/site-packages/gnuradio/gr/top_block.py",
>>> line 109, in start
>>>     top_block_start_unlocked(self._impl, max_noutput_items)
>>>   File
>>> "/home/root/localinstall/usr/lib/python2.7/site-packages/gnuradio/gr/runtime_swig.py",
>>> line 3671, in top_block_start_unlocked
>>>     return _runtime_swig.top_block_start_unlocked(*args, **kwargs)
>>> RuntimeError: uhd_rfnoc_SplitStream(9): missing connection from output
>>> port 0
>>>
>>> Looks like something is missing from the split stream. I assume it needs
>>> to be fixed in the verilog.
>>>
>>> I attached a screenshot of the new, cleaned up GNU radio file. I had to
>>> remove a FIFO and switch to using a throttle b/c I wasn't able to get it
>>> all to fit in the bit file.
>>>
>>> Regards,
>>> Jon
>>>
>>> On Wed, Nov 13, 2019 at 10:46 AM Jonathan Lockhart <
>>> jlockhartrt at gmail.com> wrote:
>>>
>>>> Greetings EJ,
>>>>
>>>> Thanks for the info. I haven't had time to grab the new block as my dev
>>>> box is packed for moving this weekend. I will get it downloaded and try it
>>>> as soon as I can.
>>>>
>>>> Regards,
>>>> Jon
>>>>
>>>> On Sat, Nov 9, 2019 at 9:48 AM EJ Kreinar <ejkreinar at gmail.com> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Just want to chime in since I recently went through the upgrade
>>>>> process to UHD-3.14...
>>>>>
>>>>> Can you confirm you're using uhd-3.14? If so, this is actually a
>>>>> split stream fpga bug caused by the commit Jon referred to, not the "not
>>>>> enough bandwidth" problem.
>>>>>
>>>>> Fortunately, rather than referring the old commit, the bug seems to
>>>>> have been fixed in October on the master branch: https://
>>>>> github.com/EttusResearch/fpga/commit/1102779f49d44c9e8b88ce7251d203eb62ae26c9
>>>>> (but not yet ported onto 3.14)
>>>>>
>>>>> I just cherry-picked 1102779f onto my UHD-3.14 and it cleaned it up
>>>>> for me.
>>>>>
>>>>> I assume this will eventually make it to the UHD-3.14 branch? But if
>>>>> not the cherry pick works fine
>>>>>
>>>>> EJ
>>>>>
>>>>>
>>>>> On Fri, Nov 8, 2019, 2:43 PM Jonathan Lockhart via USRP-users <
>>>>> usrp-users at lists.ettus.com> wrote:
>>>>>
>>>>>> Jonathon,
>>>>>>
>>>>>> I will give that a try and see if it works.
>>>>>>
>>>>>>
>>>>>> Nick,
>>>>>>
>>>>>> If the revert on Split_Stream fails, I will try switching from ce_clk
>>>>>> to bus_clk and give that a try.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Jon
>>>>>>
>>>>>> On Fri, Nov 8, 2019 at 1:52 PM Nick Foster <bistromath at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Jonathon (Pendlum), correct me if I'm wrong, but I think this is the
>>>>>>> usual split-stream-demands-more-bandwidth-than-RFNoC-bus-allows problem.
>>>>>>>
>>>>>>> Jonathan (Lockhart), if I'm right, then in your
>>>>>>> rfnoc_ce_auto_inst_e310.v, if you change the ce_clk/ce_rst in the
>>>>>>> noc_block_split_stream instantiation to bus_clk/bus_rst, this may fix the
>>>>>>> problem.
>>>>>>>
>>>>>>> Nick
>>>>>>>
>>>>>>> On Fri, Nov 8, 2019 at 10:20 AM Jonathon Pendlum <
>>>>>>> jonathon.pendlum at ettus.com> wrote:
>>>>>>>
>>>>>>>> Hi Jon,
>>>>>>>>
>>>>>>>> Can you try reverting commit e39334fe on the fpga repo and
>>>>>>>> rebuilding your bitstream? Let me know if that makes a difference or not.
>>>>>>>>
>>>>>>>> Jonathon
>>>>>>>>
>>>>>>>> On Sat, Nov 9, 2019 at 12:13 AM Jonathan Lockhart via USRP-users <
>>>>>>>> usrp-users at lists.ettus.com> wrote:
>>>>>>>>
>>>>>>>>> Greetings Nick,
>>>>>>>>>
>>>>>>>>> Here is the screenshot of my GR flow graph, though it shouldn't
>>>>>>>>> matter as the Split_Stream RFNoC Block I believe is a failure of the python
>>>>>>>>> or Verilog but the solutions in the link I sent in my previous email did
>>>>>>>>> not resolve the issue.
>>>>>>>>>
>>>>>>>>> [image: Screenshot from 2019-11-08 10-06-50.png]
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Jon
>>>>>>>>>
>>>>>>>>> On Thu, Nov 7, 2019 at 5:33 PM Nick Foster <bistromath at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Can you be more specific about what your flowgraph looks like?
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 7, 2019 at 2:22 PM Jonathan Lockhart via USRP-users <
>>>>>>>>>> usrp-users at lists.ettus.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Greetings,
>>>>>>>>>>>
>>>>>>>>>>> I was wondering if anyone had encountered the following error
>>>>>>>>>>> and had a way to fix it?
>>>>>>>>>>>
>>>>>>>>>>> [INFO] [UHD] linux; GNU C++ version 4.9.2; Boost_105700;
>>>>>>>>>>> UHD_3.14.1.HEAD-0-g0347a6d8
>>>>>>>>>>> [INFO] [E300] Loading FPGA image:
>>>>>>>>>>> /home/root/localinstall/e300.bit...
>>>>>>>>>>> [INFO] [E300] FPGA image loaded
>>>>>>>>>>> [INFO] [E300] Detecting internal GPS
>>>>>>>>>>> .... [INFO] [E300] GPSDO found
>>>>>>>>>>> [INFO] [E300] Initializing core control (global registers)...
>>>>>>>>>>>
>>>>>>>>>>> [INFO] [E300] Performing register loopback test...
>>>>>>>>>>> [INFO] [E300] Register loopback test passed
>>>>>>>>>>> [INFO] [0/Radio_0] Initializing block control (NOC ID:
>>>>>>>>>>> 0x12AD100000000000)
>>>>>>>>>>> [WARNING] [RFNOC] Can't find a block controller for key
>>>>>>>>>>> SplitStream, using default block controller!
>>>>>>>>>>> [INFO] [0/SplitStream_0] Initializing block control (NOC ID:
>>>>>>>>>>> 0x5757000000000000)
>>>>>>>>>>> [ERROR] [UHD] Exception caught in safe-call.
>>>>>>>>>>>   in ctrl_iface_impl<_endianness>::~ctrl_iface_impl() [with
>>>>>>>>>>> uhd::endianness_t _endianness = (uhd::endianness_t)1u]
>>>>>>>>>>>   at
>>>>>>>>>>> /home/jon/rfnoc_3.14.1.1/src/uhd/host/lib/rfnoc/ctrl_iface.cpp:52
>>>>>>>>>>> this->send_cmd_pkt(0, 0, true); -> EnvironmentError: IOError:
>>>>>>>>>>> Block ctrl (CE_01_Port_21) no response packet - AssertionError: bool(buff)
>>>>>>>>>>>   in uint64_t ctrl_iface_impl<_endianness>::wait_for_ack(bool,
>>>>>>>>>>> double) [with uhd::endianness_t _endianness = (uhd::endianness_t)1u;
>>>>>>>>>>> uint64_t = long long unsigned int]
>>>>>>>>>>>   at
>>>>>>>>>>> /home/jon/rfnoc_3.14.1.1/src/uhd/host/lib/rfnoc/ctrl_iface.cpp:142
>>>>>>>>>>>
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>   File "rfnoc_fosphor_network_usrp.py", line 283, in <module>
>>>>>>>>>>>     main()
>>>>>>>>>>>   File "rfnoc_fosphor_network_usrp.py", line 272, in main
>>>>>>>>>>>     tb = top_block_cls(fft_size=options.fft_size,
>>>>>>>>>>> fpga_path=options.fpga_path, freq=options.freq, gain=options.gain,
>>>>>>>>>>> host_ip_addr=options.host_ip_addr, samp_rate=options.samp_rate,
>>>>>>>>>>> tdecay=options.tdecay, trise=options.trise)
>>>>>>>>>>>   File "rfnoc_fosphor_network_usrp.py", line 43, in __init__
>>>>>>>>>>>     self.device3 = variable_uhd_device3_0 =
>>>>>>>>>>> ettus.device3(uhd.device_addr_t( ",".join(('type=e3x0',
>>>>>>>>>>> "master_clock_rate=%d,fpga=%s" % (samp_rate,fpga_path))) ))
>>>>>>>>>>>   File
>>>>>>>>>>> "/home/root/localinstall/usr/lib/python2.7/site-packages/ettus/ettus_swig.py",
>>>>>>>>>>> line 1307, in make
>>>>>>>>>>>     return _ettus_swig.device3_make(*args, **kwargs)
>>>>>>>>>>> RuntimeError: EnvironmentError: IOError: [0/SplitStream_0]
>>>>>>>>>>> sr_read64() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_21)
>>>>>>>>>>> no response packet - AssertionError: bool(buff)
>>>>>>>>>>>   in uint64_t ctrl_iface_impl<_endianness>::wait_for_ack(bool,
>>>>>>>>>>> double) [with uhd::endianness_t _endianness = (uhd::endianness_t)1u;
>>>>>>>>>>> uint64_t = long long unsigned int]
>>>>>>>>>>>   at
>>>>>>>>>>> /home/jon/rfnoc_3.14.1.1/src/uhd/host/lib/rfnoc/ctrl_iface.cpp:142
>>>>>>>>>>>
>>>>>>>>>>> This is only occurring when I use the split stream block in
>>>>>>>>>>> RFNoC. I have tried the fixes in [1] but unfortunately they have not
>>>>>>>>>>> helped. Also, fix 1, I can't seem to before b/c I am missing the file
>>>>>>>>>>> rfnoc_ce_auto_inst_<device>.v  as shown from the output when
>>>>>>>>>>> attempting a "find" command in Ubuntu.
>>>>>>>>>>>
>>>>>>>>>>> find: ‘rfnoc_ce_auto_inst_e310.v’: No such file or directory
>>>>>>>>>>>
>>>>>>>>>>> I ran it on both ~/* and /* with no luck.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Jon Lockhart
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://kb.ettus.com/RFNoC#Why_do_I_have_a_command_timeout_error.3F
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> USRP-users mailing list
>>>>>>>>>>> USRP-users at lists.ettus.com
>>>>>>>>>>>
>>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>> USRP-users mailing list
>>>>>>>>> USRP-users at lists.ettus.com
>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>> USRP-users mailing list
>>>>>> USRP-users at lists.ettus.com
>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/attachments/20191114/ee45f6da/attachment.html>


More information about the USRP-users mailing list