Debian release packaging for MOOS-IvP as a test case for Open Ocean Software DevSecOps pipelines

DevSecOps
1

Background

One of our main goals at Open Ocean Software will be to better support the flow of code from Git to end user operations.

One aspect of this is automated binary builds, such as Debian packages. The MOOS-IvP community has expressed interest in providing such packages to the public, and I think this would be a great testing ground for how to do this well, decide on a good separation of labor between member project and Open Ocean, and as much as possible reuse this for future member projects with similar needs.

Existing MOOS-IvP packages on GobySoft

I build .deb packages for the active LTS releases of Ubuntu and the oldstable/stable Debian releases. I build for amd64 (64-bit x86), armhf (32-bit ARM), arm64 (64-bit ARM) architectures, which supports all the computer hardware that I currently encounter. ARM is currently cross-built on amd64.

Using the existing packages (packages.gobysoft.org)

I’ve attached a script (install_gobysoft_ubuntu_repo.sh) that will install (on Ubuntu) the correct GPG signing key (to /usr/share/keyrings/gobysoft-archive-keyring.gpg) and the apt sources to /etc/apt/sources.list.d/gobysoft_release.list.

After running this once you can install the desired packages (some or all of):

sudo apt install {packages}

where `{packages}`:

MOOS

- moosdb10 - MOOSDB binary

- libmoos10-dev - MOOS libraries (dev: headers and SONAME symlinks)

- libmoos10 - MOOS libraries (runtime)

IvP:

- moos-ivp-apps - The MOOS-IvP multi-objective decision engine (pHelmIvP) and related tools for autonomous marine vehicles, except GUI applications.

- moos-ivp-gui - Graphical tools for the MOOS-IvP multi-objective decision engine.

- libmoos-ivp-dev - Developers’ package for the libraries included with the MOOS-IvP multi-objective decision engine (and related tools) for autonomous marine vehicles.

- libmoos-ivp - Libraries included with the MOOS-IvP multi-objective decision engine (and related tools) for autonomous marine vehicles.

moos-ivp-apps and moos-ivp-gui are split out so that headless systems (e.g., vehicles) that don’t need the FLTK dependency don’t have to install it.

*-dev libraries are split out in typical Debian format so that runtime systems don’t install all the headers and non-SONAME library symlinks, but downstream developers using MOOS-IvP projects can install them.

How these packages are built

I build the Core MOOS and MOOS-IvP packages separately, as I still consider these to be independent projects (something we should resolve as MOOS in MOOS-IvP is increasingly an incompatible fork):

Debian sources repos:

- Core MOOS: GitHub - GobySoft/core-moos-debian: Debian packages for core MOOS · GitHub

- [MOOS-]IvP: GitHub - GobySoft/ivp-debian · GitHub

These packages are built automatically on changes to the above git repos using CircleCI (.circleci/config.yml) in both repos.

These CircleCI builds use a clean Docker container to perform the build (using dpkg-buildpackage ultimately). The Docker images are built from my repo here:

- GitHub - GobySoft/circleci-debian-build: Configuration for Debian package building on CircleCI · GitHub

Other resources

2

install_gobysoft_ubuntu_repo.sh:

!/usr/bin/env bash                                                                                                                                                    
set -e                                                                                                                                                                 
                                                                                                                                                                       
# ---- Use sudo only if not already root ----                                                                                                                          
if [ "$(id -u)" -eq 0 ]; then                                                                                                                                          
    SUDO=""                                                                                                                                                            
elif command -v sudo >/dev/null 2>&1; then                                                                                                                             
    SUDO="sudo"                                                                                                                                                        
else                                                                                                                                                                   
    echo "This script requires root or sudo."                                                                                                                          
    exit 1                                                                                                                                                             
fi                                                                                                                                                                     
                                                                                                                                                                       
# ---- Determine Ubuntu codename (noble, jammy, etc.) ----                                                                                                             
. /etc/os-release                                                                                                                                                      
CODENAME="$VERSION_CODENAME"                                                                                                                                           
                                                                                                                                                                       
if [ -z "$CODENAME" ]; then                                                                                                                                            
    echo "Could not determine Ubuntu codename."                                                                                                                        
    exit 1                                                                                                                                                             
fi                                                                                                                                                                     
                                                                                                                                                                       
# ---- Requirements for bare Ubuntu 24.04 ----                                                                                                                         
$SUDO apt-get update                                                                                                                                                   
$SUDO apt-get install -y \                                                                                                                                             
    ca-certificates \                                                                                                                                                  
    curl \                                                                                                                                                             
    gnupg                                                                                                                                                              
                                                                                                                                                                       
# ---- Install GobySoft repository key ----                                                                                                                            
$SUDO install -m 0755 -d /usr/share/keyrings                                                                                                                           
                                                                                                                                                                       
curl -fsSL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x19478082E2F8D3FE" \                                                                                
  | $SUDO gpg --dearmor -o /usr/share/keyrings/gobysoft-archive-keyring.gpg                                                                                            
                                                                                                                                                                       
$SUDO chmod 0644 /usr/share/keyrings/gobysoft-archive-keyring.gpg                                                                                                      
                                                                                                                                                                       
# ---- Add repository ----                                                                                                                                             
echo "deb [signed-by=/usr/share/keyrings/gobysoft-archive-keyring.gpg] http://packages.gobysoft.org/ubuntu/release/ ${CODENAME}/" \                                    
  | $SUDO tee /etc/apt/sources.list.d/gobysoft_release.list > /dev/null                                                                                                
                                                                                                                                                                       
# ---- Update package lists ----                                                                                                                                       
$SUDO apt-get update
3

My question for the community are:

  1. Does MOOS-IvP want to take over the debian configuration file directory? Or does this transition to an Open Ocean Software product?
  2. How do we build these? Continue with CircleCI, or use Github Actions? Or host our own using AWS or similar?
  3. Who hosts the output .deb files and metadata? Do we want a packages.oceansoft.org repository where these are pushed, or do we make a packages.moos-ivp.org that we then synchronize (perhaps after some review?) to packages.oceansoft.org?
    1. How do we ensure that the packages are what we believe they are (e.g., GPG signing keys)? This is important for trust from downstream users.
    2. What division of repositories do we have (“components” in Ubuntu lingo)? GobySoft does continuous (every build of the main branch) and release (tagged releases). We could add a beta repo (between continuous and release) and/or a test repo in here as well if that made sense. Or we could add a stable that is less updated than release that only gets minor or patch updates.
  4. Do we like the existing allocation of packages from MOOS-IvP (that is, MOOS-IvP source package creates the Debian packages listed above:moos-ivp-apps, moos-ivp-gui, libmoos-ivp and libmoos-ivp-dev)?
  5. How is MOOS supported? Do we continue to build MOOS from upstream core-moos or do we build it from the MOOS-IvP packages?
  6. Which Ubuntu / Debian distro releases do we target? Do we package the latest IvP release for all supported distro releases or only the latest?

Please let me know what you think and if there’s enough momentum I will arrange a meeting.

4

My 2-cents:

Does MOOS-IvP want to take over the debian configuration file directory? Or does this transition to an Open Ocean Software product?

I’d say moos-ivp can take over the definitions. I’m working on cleaning moos-ivp code itself to make this effort easier.

Mid-to-long term, I’d like to do some larger change on the source side to meet Debian best practices. Right now I’m focusing on making each package one LICENSE. Since not all libraries are uniformly GPL or LGPL, I’ve also been toying around with making each lib its own package. May be overkill, but if I’m splitting some up I may as well split them all.

I’d be interested in getting the moos/ivp packages in Debian upstream if there were user interest for that. But following Best Practices makes our lives easier at the very least.

How do we build these? Continue with CircleCI, or use Github Actions?

I’m most familiar with Github Actions and Gitlab CI, but Circle CI works as well. If we make a Ubuntu PPA, Launchpad would build it, so whatever we write should be CI-agnostic. Again, being OSS we can use some free community minutes here.

Who hosts the output .deb files and metadata?

For a v0, I’ve just planned on attaching the deb file to the tagged GitHub release, but I’d like to get it into some proper apt repo. A PPA was my first thought since that solves both the hosting and GPG key management for us (plus, we don’t need to spend on our own bandwidth & hosting costs, human and technical), but if oceansoft wants to be the apt repo publisher I wouldn’t be opposed to it.

What division of repositories do we have (“components” in Ubuntu lingo)? GobySoft does continuous (every build of the main branch) and release (tagged releases). We could add a beta repo (between continuous and release) and/or a test repo in here as well if that made sense. Or we could add a stable that is less updated than release that only gets minor or patch updates.

I like having continuous for the latest bleeding edge builds from git. I’d propose stable for any tagged release. I’ve also seen the pattern of a component per major version - so maybe one for 22, 24, 26, etc?

Do we like the existing allocation of packages from MOOS-IvP?

Not 100% sure what this means :sweat_smile:

How is MOOS supported? Do we continue to build MOOS from upstream core-moos or do we build it from the MOOS-IvP packages?

We’ve spoken with Paul about officially taking over the MOOS repo control, but not much progress has been made on that recently. The differences between the two are very minimal, and we send upstream most of our patches, but they still aren’t exactly 1:1. To afford us some flexibility, I’ve been planning on making the MOOS core packages all -mit variants, which Provide a virtual package moos. That way, if someone truly wanted to, they could replace our build with the Oxford build or their own core moos.

The default in moos-ivp would be to Require moos, and recommend moos-mit.

Which Ubuntu / Debian distro releases do we target?

LTS (plus the previous LTS for a while)? Each major IvP release at MOOS-DaWG typically coincides with a Ubuntu LTS release a few months before. For my Docker images, I don’t want to force a bump of both a major ubuntu LTS and major MOOS bump at the same time, so I wait a few months until the Lab adopts the newest Ubuntu, then start targeting that.

A CI matrix is cheap, so as long as there aren’t critical features we need exclusively in newer versions (which I sincerely doubt - we’re still not using C++11 everywhere), I’d be fine building for n-1.

5

What I meant here is do we like the breakout of packages from the MOOS-IvP source repo into APT packages moos-ivp-apps, moos-ivp-gui, libmoos-ivp-dev, and libmoos-ivp or do we want more (or less) granularity?

6

Please don’t put each lib in its own package- that seems overkill. I think MOOS-IvP already builds way too many libraries (is there a real technical reason we can’t just build libmoos-ivp.so and be done?? - and maybe also libmoos-ivp-gui.so to split out the FLTK dependency?) There are circular dependencies in the existing forest of libraries that MOOS-IvP builds.

7

I like this idea and I think it’s a good way to move forward here.

8

When I last used it the PPA infrastructure was a bit creaky and dated. It also only builds for Ubuntu and I wanted to have Debian support as well. You can certainly publish to a PPA in addition to whatever we (Open Ocean) decide to spin up - it uses the same debian configuration as you know. I also have a vision that a packages.oceansoft.org might eventually serve ready-to-use Docker images, RPi filesystems, VBox virtual machines, etc. in a “choose from a menu and build” format including packages from one or more member projects as the end-user desires.