Hi all!

I’m an FPGA developer, dragged into the Yocto world a few years ago with the move to Zynq and ZynqMP architectures.  As a research group, we mainly develop on Xilinx dev boards like the ZCU102 MPSoC, and ZCU111 RFSoC.

Having recent success adding the Xillinx Deep-Learning Processor (DPU) to the FPGA fabric, building the Vitis-AI libraries into the rootfs, and accelerating ML applications on various models, we decided to move things over to an x410 to take advantage of the RFSoC in a complete SDR product.

The FPGA design was straight forward, and the Make-based FPGA build scripts, which I am typically not a fan of, was quite well thought out, worked wonderfully, and was easy to modify to customize the flow, and add changes to the block design.  With a little digging, it was also fairly straightforward to incorporate the changes into the XSA/device tree for use in building the rootfs.

Incorporating the Vitis-AI libraries and DPU drivers into the KAS/Kirkstone/Mender/Titanium build system is another story.  Long story short, after the typical, lengthy, Yocto debug process, it ultimately fails at what seems to be a kernel incompatibility between the mainline kernel and the Vitis-AI requirements.  This particular time, it manifests as syntax errors in the zocl module compilation, though I suspect it is actually a cascading series of failures that can not be solved one at a time.

Researching the failure on the Xilinx forum leads to assertions that certain parts of the Vitis-AI libraries (as well as many other Xilinx applications that exercise features of the FPGA) require the Xilinx fork of the Linux Kernel, linux-xlnx.

Has anyone attempted to switch kernels in a UHD system, or to integrate the linux-xlnx kernel features into the UHD kernel?

Hello,

I ported USRP X410 code to ZCU111. As the NI/Ettus kernel didn’t work straight away (i.e. because of device-tree and bootloader issues) I started from booting with a linux-xlnx kernel that I had from Petalinux together with device-tree for ZCU111.

I was able to use that kernel to some point, but I don’t remember at the moment if I used it to run any FPGA bitstream. Probably not. Very soon I switched to NI/Ettus kernel by patching it only with necessary changes (automatic ‘git bisect’ was indispensable tool that made finding crucial commits from linux-xlnx kernel). The rationale was that I wanted to keep changes in relation to the mainline UHD minimal. But it also seemed that NI/Ettus kernel needed very few changes to be taken from linux-xlnx in comparison to the other way around (there are for example ethernet drivers from NI for 10Gb / 100Gb Ethernet ports that are implemented in FPGA). I don’t know - your situation might be different.

All of the kernel patches - both from NI and me are here:
https://github.com/ptrkrysik/meta-ettus/tree/x411/meta-titanium/recipes-kernel/linux/files

So you can have a look and decide for yourself which path is better for you.

Best Regards,
Piotr Krysik

Hi Piotr,

Thanks!  That actually does help quite a bit.  You hit perfectly on my main dilemma: Is it more straightforward to patch the linux-xlnx kernel with the UHD mods, or do I patch the UHD kernel with the linux-xlnx mods?

The fact that you’ve successfully added linux-xlnx features to the UHD kernel to support the  requirements of both the ZCU111 and USRP code make my decision easy.  I was hoping that I could avoid having to find a way to stitch the linux-xlnx kernel into the OS to replace a kernel that looks highly customized.  I’ll certainly check out your repo for examples, and I’ll share the Vitis-AI-related meta-ettus patches on my github as well.

Great ‘git bisect’ tip!  I have been avoiding it for years, for some unknown reason, so I suppose it’s time to see what it can do. :-)

Thanks again for pointing me in the right direction!

Mike

Hi Marcus!   

Hahaha  I thoroughly enjoyed the rant!    I think you are correct about Make and its variations.   Indispensable seems an understatement for something that is so pervasive in software development.   As I mentioned, I am primarily an FPGA developer.   At some point, when the Zynq and ZynqMP SoCs were released, the software community seemed to (initially, at least) scatter when they saw "FPGA" in the name, and I found myself inadvertently "volun-told" to be responsible for building Linux releases with custom drivers for my hardware. :-)  It was horrifying! :-)

You don't see Make involved in FPGA builds as frequently as you'd expect, considering the popularity of SoCs these days.   As a self-proclaimed "Crusty Old Hardware Guy", I'm not permitted to actually say out loud that I think Make is a good addition to the FPGA development flow... ;-)  ...but I have to admit that I've been pleasantly surprised by how easily all aspects of a build can be automated, once the initial setup is done.

I think what keeps me from jumping in completely, is that many aspects of FPGA development are still very much grounded in Hardware Development principles and techniques, processes in which a GUI is supremely helpful.   At the end of the day, I still need to see a schematic, block diagram, or wave-forms.   To that end, however, with the x410 UHD build, I was impressed by how straight-forward it was to modify the Make files and build Tcl scripts to create the project, build the IP, and open it in the GUI for me to inspect and continue in my normal FPGA-dev flow.

Hahaha   I tried to keep this short, but I apparently reply to rants with...more ranting.   :-)

Thanks for the reply!

Mike

Well put.  The abstraction really is at the heart of it, isn’t it?  Perhaps that’s why Make was adopted by the hardware world more slowly.  Over the last decade or two, FPGAs have gotten so large, and the external connectivity options so numerous (1GE, 10GE, 40G, 100G), that hardware designs routinely have multiple flavors, differentiated only by the source code included in each.  Hahaha  Make was inevitable at that point.  I have to admit, the x4xx UHD build system is a pretty good example of how to combine Make and Tcl for FPGA design.

But the crusty old hardware guy in me is still hopeful that schematic entry of FPGA designs will come back! :-)

Hi Piotr,

I looked through the code in your repo, and it’s a perfect template for the types of patches and mods I need to make.  I had seen the x410_defconfig file in the kernel recipes and was wondering if I could edit it, or if it was automatically generated.  Seeing that you added a few Xilinx-specific CONFIG_ options was very useful.  I think I have a good place to start in diffing the two kernels and adding the Xilinx-specific mods to UHD to support the DPU drivers.

If we were on stack exchange, I would definitely mark this as answered! :-)  I’ll update the thread as things progress.

Many thanks,

Mike

Hi Mike,

I can see that one thing have changed since I did that patching. At the time most recent linux-xlnx was based on linux 5.10 and USRP X410 used 5.15.

Now the situation is different: most recent linux-xlnx is 6.6 and USRP still uses the same kernel.

Of course you don’t have to rely on most recent linux-xlnx kernel, but just the version for which DPU is supported. But it’s additional factor that requires consideration.

Best Regards,
Piotr Krysik

Hello Mike,

I don’t know what your preference regarding setup is, but for me it was crucial to be able to make changes to the rootfs and kernel quickly and to be able to remotely reset the device.

This was especially important when dealing with kernel and bootloader. As I didn’t have much experience with editing kernel and u-boot sources, it was indispensable to be able to check my changes and applied patches quickly (the additional difficulty was that I didn’t have a X410 to see in action how things should work).

My setup consisted of NFS server hosting rootfs and TFTP server for kernel. The bootloader was loaded through JTAG. This way I for example was able to make a script that: compiled and installed new kernel with the board turned off or in not working state, tried to boot it, check if system started correctly and return the result to git bisect. This way (after some fine-tuning) I was able to run ‘git bisect’ and wait for it to find a commit where the board started to work. If you are interested how to configure what I’ve described - ask.

With this approach you could for example locate when DPU support that you need was added. Your case seems to be better in one regard though: as you know what you are looking for  you can also look at commit messages and the code. In my case I didn’t know what was source of problems - the board just didn’t boot or (in another case) loading ‘nixge’ driver for 10Gb/s ports crashed the board.

Piotr