Hi all,

 we've had a bunch of great discussions on the packaging of FPGA images
 (binaries) and on packaging UHD in general. Since there were many
 different discussions, I'd like to summarize them all in one big email.
 Thanks to Rick, Maitland, and Phil for providing feedback.

 If you just use UHD via git, none of this affects you.

 ## Release tags

 Going forward, we'll change the format of the release tags. Before, it
 was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is
 a much more standard format. Personally, I'm happy to make this change
 -- we were hesitant to change it for consistency's sake, but it seems no
 one actually cares about the old format anyway. If you're pulling from
 github, this means you get nice, reproducible URLs like this:
 https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
 <https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0>

 ...and the tarball for the sources:
 https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz
 <https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz>

 ...and so on. I'd be interested to see who cares about the old tag
 format. For now, we'll double-tag (and we might re-tag old versions,
 too, if there's interest) to keep things easy.

 ## Getting FPGA images for releases

 We have two paths here. First, we have the uhd_images_downloader format
 (I'll talk more about that below). You can grab images from the
 images/manifest.txt file, but I understand that's cumbersome.

 So, on top of the uhd_images_downloader format, we're considering
 bringing back an archive per release, but we're still figuring out the
 details of this.

 We have for the sake of feedback uploaded an archive here:
 https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
 <https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0>

 I'd love to know if this format is suitable for people, and if things
 are missing. If this is all everyone needs, we'll find a way to automate
 this and make it error-proof.

 However, this means that you will be downloading *all* default images
 (currently, that's over 200 MB after unpacking), some of which rarely or
 never change (e.g., the USRP1 image will always be the same).
 If you want to tailor the images you download (e.g., you might only care
 about the embedded images), pulling the individual packages might be a
 better option. The problem is, we don't have an easy way of identifying
 which ones there are without requiring to run a Python script at package
 build time.
 Something like this works, if you have access to the UHD source code
 (bash code):

 SRC_URI=http://files.ettus.com/binaries/cache/`cat
 <http://files.ettus.com/binaries/cache/`cat> images/manifest.txt |
 grep -v '#' | awk -F' ' '{print $3}' | grep b200mini`

 ...and it would limit the images to the b200mini images, but it's not
 exactly handy. If people have suggestions, I'd be curious to hear them.

 ## FPGA Versions

 There was a question why we don't version our FPGA images (e.g., why
 does the X300 image not have version 1.2.3, and we say this version
 works with UHD 3.11.0.0). There are several problems with this: First,
 we have over 40 FPGA images (not all are packaged in the default image).
 Versioning them all correctly is already a hard logistical issue.
 Second, how do we create meaningful version numbers? The FPGA images
 consist of many subcomponents, all of which would need to go into an
 image version. The X300 and X310 images, for example, should always have
 similar versions when they have similar behaviour. But what if we fix
 something only for X300? Then, both the X310 and X300 images have 5
 variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do
 we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA,
 XA} in sync? Then of course, we have components on all X-Series that we
 also use on the E310. So that makes the version-number-space even more
 higher-dimensional. And so on.
 Finally, we consider the FPGA source code part of UHD. That means, for
 version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA
 repository that matches the source code. The images built off of this
 commit are the ones that matter, and so they already have a version (the
 UHD version, in this case, 3.11.0.0).
 For any given UHD commit, the correct set of FPGA images is the one
 tracked in images/manifest.txt.

 ## The new images downloader

 If you're used to using uhd_images_downloader, nothing changes for you.
 Here's the contents of an older email explaining the motivation and
 changes in greater detail.
 Note that we'll start using this tool to distribute even more things
 going forward. We have a bunch of things where building binaries is
 getting more and more complicated for end users, and we'll try and make
 it easy to update devices.


 ```
 Subject: [UHD] Changes to uhd_images_downloader
 Date: Tue, 23 Jan 2018 15:33:14 -0800
 From: Martin Braun <martin.braun@ettus.com
 <mailto:martin.braun@ettus.com>>

 For the next UHD version, we will be updating uhd_images_downloader. The
 problem with the old uhd_images_downloader is that it refers to one big
 zip file for all the images. As we add more and more devices, we not
 only have to update the zip file more often, it also becomes bigger and
 bigger. If you're running uhd_images_downloader on an E310, it's most
 likely that you only want the E310 images, and not the megabytes of
 other FPGA images.

 The new tool fixes all those issues by breaking up the zip file into
 more, smaller zip files. On top of that, the new tool has a bunch of
 other improvements:

 - It maintains an inventory of downloaded files, and skips downloading
 them again if they didn't change. This means if we roll out a fix for
 the X300 only, and you run uhd_images_downloader without any arguments,
 it won't try and update anything else. Should you delete the inventory,
 it'll simply get everything, like it did before.
 - You can now filter by type. Use the -t switch to match against a
 regular expression, e.g. '-t b200mini' will only download a b200mini
 file.
 - We'll be adding more image types soon that are not FPGA images, such
 as filesystem updates.
 - 'uhd_images_downloader -l' will list all the targets you can download.

 In general, this should improve things and make other things easier.
 However, since we had to basically rewrite the tool, which had otherwise
 matured, it's likely that the new tool has shortcomings. We'd like to
 hear from you regarding those.

 There are a few problems we are already tracking:
 - We currently have no archives other than zip files. We're intending to
 add .tar.xz in the future, and you will be able to use
 uhd_images_downloader to fetch those (assuming your system can unzip xz
 archives)
 - The tool output changed, and we're still fine tuning the output.


 If you're a package maintainer, you probably hard-coded the URL for the
 FPGA images in your package creation script.
 We track the URLs of the individual archives, along with their SHA256
 hash, in a manifest that is checked into git:
 https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt
 <https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt>

 The base URL for all those is http://files.ettus.com/binaries/cache
 <http://files.ettus.com/binaries/cache>, so
 http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip
 <http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip>
 will fetch the USRP1 FPGA images.

 We're happy for any feedback -- ideally before we release the next
 version.

 Cheers,
 Martin
 ```



 Hopefully this is helpful. Again, I'd like to thank everyone for
 providing feedback.

 Cheers,
 Martin



 _______________________________________________
 Discuss-gnuradio mailing list
 Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org>
 https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
 <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>

On 03/08/2018 11:33 AM, Sid Hayn wrote: > https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt > > This address is invalid, assuming you meant this one? > > https://github.com/EttusResearch/uhd/blob/master/images/manifest.txt Yes! Thanks for pointing that out. -- M > > Thanks, > Zero > > On Wed, Mar 7, 2018 at 8:12 PM, Martin Braun <martin.braun@ettus.com > <mailto:martin.braun@ettus.com>> wrote: > > Hi all, > > we've had a bunch of great discussions on the packaging of FPGA images > (binaries) and on packaging UHD in general. Since there were many > different discussions, I'd like to summarize them all in one big email. > Thanks to Rick, Maitland, and Phil for providing feedback. > > If you just use UHD via git, none of this affects you. > > ## Release tags > > Going forward, we'll change the format of the release tags. Before, it > was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is > a much more standard format. Personally, I'm happy to make this change > -- we were hesitant to change it for consistency's sake, but it seems no > one actually cares about the old format anyway. If you're pulling from > github, this means you get nice, reproducible URLs like this: > https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0 > <https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0> > > ...and the tarball for the sources: > https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz > <https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz> > > ...and so on. I'd be interested to see who cares about the old tag > format. For now, we'll double-tag (and we might re-tag old versions, > too, if there's interest) to keep things easy. > > ## Getting FPGA images for releases > > We have two paths here. First, we have the uhd_images_downloader format > (I'll talk more about that below). You can grab images from the > images/manifest.txt file, but I understand that's cumbersome. > > So, on top of the uhd_images_downloader format, we're considering > bringing back an archive per release, but we're still figuring out the > details of this. > > We have for the sake of feedback uploaded an archive here: > https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0 > <https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0> > > I'd love to know if this format is suitable for people, and if things > are missing. If this is all everyone needs, we'll find a way to automate > this and make it error-proof. > > However, this means that you will be downloading *all* default images > (currently, that's over 200 MB after unpacking), some of which rarely or > never change (e.g., the USRP1 image will always be the same). > If you want to tailor the images you download (e.g., you might only care > about the embedded images), pulling the individual packages might be a > better option. The problem is, we don't have an easy way of identifying > which ones there are without requiring to run a Python script at package > build time. > Something like this works, if you have access to the UHD source code > (bash code): > > SRC_URI=http://files.ettus.com/binaries/cache/`cat > <http://files.ettus.com/binaries/cache/`cat> images/manifest.txt | > grep -v '#' | awk -F' ' '{print $3}' | grep b200mini` > > ...and it would limit the images to the b200mini images, but it's not > exactly handy. If people have suggestions, I'd be curious to hear them. > > ## FPGA Versions > > There was a question why we don't version our FPGA images (e.g., why > does the X300 image not have version 1.2.3, and we say this version > works with UHD 3.11.0.0). There are several problems with this: First, > we have over 40 FPGA images (not all are packaged in the default image). > Versioning them all correctly is already a hard logistical issue. > Second, how do we create meaningful version numbers? The FPGA images > consist of many subcomponents, all of which would need to go into an > image version. The X300 and X310 images, for example, should always have > similar versions when they have similar behaviour. But what if we fix > something only for X300? Then, both the X310 and X300 images have 5 > variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do > we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA, > XA} in sync? Then of course, we have components on all X-Series that we > also use on the E310. So that makes the version-number-space even more > higher-dimensional. And so on. > Finally, we consider the FPGA source code part of UHD. That means, for > version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA > repository that matches the source code. The images built off of this > commit are the ones that matter, and so they already have a version (the > UHD version, in this case, 3.11.0.0). > For any given UHD commit, the correct set of FPGA images is the one > tracked in images/manifest.txt. > > ## The new images downloader > > If you're used to using uhd_images_downloader, nothing changes for you. > Here's the contents of an older email explaining the motivation and > changes in greater detail. > Note that we'll start using this tool to distribute even more things > going forward. We have a bunch of things where building binaries is > getting more and more complicated for end users, and we'll try and make > it easy to update devices. > > > ``` > Subject: [UHD] Changes to uhd_images_downloader > Date: Tue, 23 Jan 2018 15:33:14 -0800 > From: Martin Braun <martin.braun@ettus.com > <mailto:martin.braun@ettus.com>> > > For the next UHD version, we will be updating uhd_images_downloader. The > problem with the old uhd_images_downloader is that it refers to one big > zip file for all the images. As we add more and more devices, we not > only have to update the zip file more often, it also becomes bigger and > bigger. If you're running uhd_images_downloader on an E310, it's most > likely that you only want the E310 images, and not the megabytes of > other FPGA images. > > The new tool fixes all those issues by breaking up the zip file into > more, smaller zip files. On top of that, the new tool has a bunch of > other improvements: > > - It maintains an inventory of downloaded files, and skips downloading > them again if they didn't change. This means if we roll out a fix for > the X300 only, and you run uhd_images_downloader without any arguments, > it won't try and update anything else. Should you delete the inventory, > it'll simply get everything, like it did before. > - You can now filter by type. Use the -t switch to match against a > regular expression, e.g. '-t b200mini' will only download a b200mini > file. > - We'll be adding more image types soon that are not FPGA images, such > as filesystem updates. > - 'uhd_images_downloader -l' will list all the targets you can download. > > In general, this should improve things and make other things easier. > However, since we had to basically rewrite the tool, which had otherwise > matured, it's likely that the new tool has shortcomings. We'd like to > hear from you regarding those. > > There are a few problems we are already tracking: > - We currently have no archives other than zip files. We're intending to > add .tar.xz in the future, and you will be able to use > uhd_images_downloader to fetch those (assuming your system can unzip xz > archives) > - The tool output changed, and we're still fine tuning the output. > > > If you're a package maintainer, you probably hard-coded the URL for the > FPGA images in your package creation script. > We track the URLs of the individual archives, along with their SHA256 > hash, in a manifest that is checked into git: > https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt > <https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt> > > The base URL for all those is http://files.ettus.com/binaries/cache > <http://files.ettus.com/binaries/cache>, so > http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip > <http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip> > will fetch the USRP1 FPGA images. > > We're happy for any feedback -- ideally before we release the next > version. > > Cheers, > Martin > ``` > > > > Hopefully this is helpful. Again, I'd like to thank everyone for > providing feedback. > > Cheers, > Martin > > > > _______________________________________________ > Discuss-gnuradio mailing list > Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio > <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio> > >