Introducing Gentoo Build Publisher

Gentoo Linux is a source-based rolling-release meta-distribution that one can twist and mold into pretty much anything one likes. That's just a verbose way of saying Gentoo is awesome.

Over the years I have been interested in mixing Gentoo's source-based DNA with continuous integration. For a while I've had most of my physical, virtual, and container Gentoo systems' packages built by Jenkins and have been mostly happy with the results except for a few situations (outlined here). For the past couple of months I have worked on the next iteration of the idea and have come up with what I call Gentoo Build Publisher, or "gbp" for short. Build Publisher.

Gentoo Build Publisher is a combination of best practices and a Django application that allows one to build all one's systems' Gentoo packages in a CI/CD (Jenkins) instance which gives following advantages:

  • The emerge --sync, emerge --update @world cycle installs only binary packages that were built specifically for your machine.

  • CI/CD pulls your repos (portage & overlays) and only starts a build when they have changed.

  • Your machines' portage configuration (/etc/portage`, repos, world file, etc.) is kept in version control and is synced to your machines the same way your packages and portage tree are. This also makes pushing the same configuration to multiple machines easy.

  • Binary packages are built inside of server-less, root-less containers using buildah.

  • All builds are not "published" until they are completed and successful.

  • It is possible to publish older builds and have systems go back in time. This is useful, for example, if an upgrade to a package is buggy or the newest version isn't yet desirable.

  • It is capable to see only the packages what will be installed for a particular system before doing an emerge --sync.

  • Jenkins can be configured to automatically publish the last successful build, or builds can be published manually.

The Gentoo Build Publisher web app has a REST interface and (soon) a command-line app.

$ gbp list lighthouse
...
[  ]  1177 06/06/21 11:13:23
[ N]  1178 06/06/21 12:14:47
[PN]  1179 06/06/21 13:18:14
[  ]  1180 06/06/21 14:13:21
[ N]  1181 06/06/21 15:38:46
[ N]  1182 06/06/21 16:15:35

In the above example, my machine's name is lighthouse Build 1182 is the latest successful build and 1179 is the currently published one (P) The N means there are user notes for the build. User notes can be added for a build in the Django Admin. However GBP automatically adds a note to a newly pulled build if it contains packages that the previous build did not. Since 1180 has no notes I can presume that it has no new packages and so it may not be worth publishing. Build 1181 looks like it may have a new package:

$ gbp status lighthouse 1181
Build: lighthouse/1181
Submitted: Sun Jun  6 15:38:46 2021 -0700
Completed: Sun Jun  6 15:41:01 2021 -0700
Published: False

    Packages built:

    * app-accessibility/at-spi2-core-2.40.2-1
    * app-arch/gnome-autoar-0.3.3-1
    * dev-lang/vala-0.52.4-1
    * dev-libs/vala-common-0.52.4-1
    * gnome-base/librsvg-2.50.7-1
    * gnome-base/nautilus-40.2-1
    * gnome-extra/evolution-data-server-3.40.2-1
    * gnome-extra/gnome-calendar-40.2-1
    * gnome-extra/sushi-3.38.1-1
    * gnome-extra/yelp-40.2-1
    * gnome-extra/yelp-xsl-40.2-1
    * mail-client/evolution-3.40.2-1
    * media-gfx/eog-40.2-1
    * net-libs/libmbim-1.24.8-1
    * net-libs/libqmi-1.28.6-1
    * net-misc/modemmanager-1.16.6-1
    * sys-apps/osinfo-db-20210531-1
    * www-client/epiphany-40.2-1
    * x11-libs/vte-0.64.2-1
    * x11-terms/gnome-terminal-3.40.2-1

It is also possible to see the differences between two specific builds:

$ gbp diff lighthouse 1179 1182
diff -r lighthouse/1179 lighthouse/1182
--- a/lighthouse/1179 Sun Jun  6 13:18:14 2021 -0700
+++ b/lighthouse/1182 Sun Jun  6 16:15:35 2021 -0700
-app-accessibility/at-spi2-core-2.40.1-3
+app-accessibility/at-spi2-core-2.40.2-1
-app-arch/gnome-autoar-0.3.2-2
+app-arch/gnome-autoar-0.3.3-1
-dev-lang/vala-0.52.3-2
+dev-lang/vala-0.52.4-1
-dev-libs/vala-common-0.52.3-2
+dev-libs/vala-common-0.52.4-1
-gnome-base/librsvg-2.50.6-1
+gnome-base/librsvg-2.50.7-1
-gnome-base/nautilus-40.1-5
+gnome-base/nautilus-40.2-1
-gnome-extra/evolution-data-server-3.40.1-4
+gnome-extra/evolution-data-server-3.40.2-1
-gnome-extra/gnome-calendar-40.1-3
+gnome-extra/gnome-calendar-40.2-1
-gnome-extra/sushi-3.38.0-4
+gnome-extra/sushi-3.38.1-1
-gnome-extra/yelp-40.0-3
+gnome-extra/yelp-40.2-1
-gnome-extra/yelp-xsl-40.0-3
+gnome-extra/yelp-xsl-40.2-1
-mail-client/evolution-3.40.1-2
+mail-client/evolution-3.40.2-1
-media-gfx/eog-40.1-3
+media-gfx/eog-40.2-1
-net-libs/libmbim-1.24.6-3
+net-libs/libmbim-1.24.8-1
-net-libs/libqmi-1.28.4-1
+net-libs/libqmi-1.28.6-1
-net-misc/modemmanager-1.16.4-3
+net-misc/modemmanager-1.16.6-1
-sys-apps/osinfo-db-20210426-3
+sys-apps/osinfo-db-20210531-1
-www-client/epiphany-40.1-3
+www-client/epiphany-40.2-1
-x11-libs/libX11-1.7.1-1
+x11-libs/libX11-1.7.2-1
-x11-libs/vte-0.64.1-5
+x11-libs/vte-0.64.2-1
-x11-misc/compose-tables-1.7.1-1
+x11-misc/compose-tables-1.7.2-1
-x11-terms/gnome-terminal-3.40.1-2
+x11-terms/gnome-terminal-3.40.2-1

The above example shows (in diff like format) that the difference between the two builds showing both the outgoing and incoming packages.

Publishing a newer (or older) build is simple:

$ git publish lighthouse 1182

Here's an example of what all my current GBP builds look like in Jenkins:

Jenkins

As can be seen above, there is a "gbp" job that builds my Gentoo Build Publisher instance.

Next time I will write about how one can install one's own gbp instance.

For any questions, comments, or feedback feel free to contact me on GitHub.

Pages