GPG signing APT repository in Spacewalk

After following my article on how to register Ubuntu and Debian clients with Spacewalk you might have noticed a APT warning

WARNING: The following packages cannot be authenticated.

This happens because the APT repositories in Spacewalk are not GPG signed. You can still install the packages but have to acknowledge it by entering “yes”.

For various reasons it would be better to have the repository signed. Read on after the jump how I got that working.

Background information

… about what turns an APT repo into a signed repo:
– Sign each package individually
– Dont sign each package but the entire repo at once

As all? major repo maintainers (Ubuntu and Debian) use the second approach and we are mirroring them, we will mimic them.
Two repository metadata files are necessary and missing: Release and Release.gpg

(Here is the Debian wiki about secure-apt explaining all this in detail https://wiki.debian.org/SecureApt)

Release

Contains information about the content of the repo, such as architecture, but most important for us: The MD5 / SHA checksums for the Packages and Packages.gz files which contain checksums for each package.

Release.gpg

Contains the GPG signature for the “Release” file.

Chain of Trust

APT uses the checksums in the Packages file to verify each package. It also uses the checksum in the Release file to verify the Packages file.
For verifying the Release file itself it uses GPG against the Release.gpg file. The fitting GPG pubkey needs to be imported in APT first.

Preparation

Creating the files Release and Release.gpg is really simple. What turned out to be a bit of a mess was the wrongly formatted /etc/apt/sources.list.d/spacewalk.list that gets generated by apt-transport-spacewalk package based on the channels the system is subscribed to.

# Old / bad format with all "components" in one line
deb spacewalk://spacewalk.xxx.lan channels: main precise-spacewalk-client precise-security precise-updates

Which only worked because of some “creative” magic in /usr/lib/apt/methods/spacewalk

# New format based on
# deb http://site.example.com/debian distribution component1 component2
# where distribution is sth like precise, precise-updates, trusty, trusty-backports etc.. and component fixed to repodata
# see https://wiki.debian.org/SourcesList
deb spacewalk://spacewalk.xxx.lan precise repodata
deb spacewalk://spacewalk.xxx.lan precise-updates repodata
deb spacewalk://spacewalk.xxx.lan precise-security repodata

If we dont change the format, the Release / Release.gpg file will only be queried for one of our channels. Find the required patch for /usr/lib/apt-spacewalk/pre_invoke.py here as I submitted it to the Bugtracker.

Next up we need to patch /usr/lib/apt/methods/spacewalk , diff is here

Unfortunately we also need to patch a single line in the Spacewalk Server package to allow the download of our two new metadata files, get it here and afterwards do

spacewalk-service restart

Now we fixed everything needed so we can finally GPG sign our repo.

Signing the Repo

Ok first of all we need a GPG key. On Spacewalk server it needs to be imported to the GPG keyring. On the clients only the public key needs to be imported to APT.

# Generate a key, answer:
#     4) RSA sign only
#     0 expires never
#     Real Name: Spacewalk, email, comment: For GPG signing APT repos
gpg --gen-key
# Export the key parts
gpg --export-secret-keys --armor spacewalk-admins@holidaycheck.com > spacewalk.gpg.privkey
gpg --armor --export spacewalk-admins@holidaycheck.com > spacewalk.gpg.pubkey
# Import the public key on ALL clients
apt-key add spacewalk.gpg.pubkey

Now we need to sign our repos. I created a handy bash script for that purpose that can be used as cronjob until this gets implemented in the Spacewalk server repo generation.
Take this script here and run it in every folder in /var/cache/rhn/repodata/
For each channel you have one folder in there like precise, precise-updates, precise-backports….

cd /var/cache/rhn/repodata/precise
/opt/spacewalk/secureApt.sh precise precise
cd /var/cache/rhn/repodata/precise-security
/opt/spacewalk/secureApt.sh precise precise-security
cd /var/cache/rhn/repodata/precise-backports
/opt/spacewalk/secureApt.sh precise precise-backports

This creates a Release and Release.gpg file for every channel based on the Packages and Packages.gz files. These can change after every repo sync so you need to run the secureApt cron right after every repo sync !

If you now apt-get update on your clients, you should see the files being downloaded and when installing packages you shouldnt get warnings about unauthenticated packages.

4# apt-get update
Apt-Spacewalk: Updating sources.list
Get:1 spacewalk://spacewalk.xxx.lan precise Release.gpg [490 B]
Get:2 spacewalk://spacewalk.xxx.lan precise-security Release.gpg [490 B]
Get:3 spacewalk://spacewalk.xxx.lan precise-updates Release.gpg [490 B]
Get:4 spacewalk://spacewalk.xxx.lan precise Release [927 B]
Get:5 spacewalk://spacewalk.xxx.lan precise-security Release [932 B]
Get:6 spacewalk://spacewalk.xxx.lan precise-updates Release [935 B]
Get:7 spacewalk://spacewalk.xxx.lan precise/repodata amd64 Packages [1,890 kB]
Get:8 spacewalk://spacewalk.xxx.lan precise/repodata i386 Packages [1,890 kB]
Ign spacewalk://spacewalk.xxx.lan precise/repodata TranslationIndex
Ign spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata TranslationIndex
Get:9 spacewalk://spacewalk.xxx.lan precise-security/repodata amd64 Packages [724 kB]
Get:10 spacewalk://spacewalk.xxx.lan precise-security/repodata i386 Packages [724 kB]
Ign spacewalk://spacewalk.xxx.lan precise-security/repodata TranslationIndex
Get:11 spacewalk://spacewalk.xxx.lan precise-updates/repodata amd64 Packages [1,364 kB]
Get:12 spacewalk://spacewalk.xxx.lan precise-updates/repodata i386 Packages [1,364 kB]
Ign spacewalk://spacewalk.xxx.lan precise-updates/repodata TranslationIndex
Get:13 spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata amd64 Packages [1,238 B]
Get:14 spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata i386 Packages [1,238 B]
....
Fetched 7,964 kB in 6s (1,229 kB/s)
Reading package lists... Done

Now, enjoy your signed repo and report back any bugs you might notice.

    • Derek
    • December 29th, 2015 8:47pm

    After applying the patches and restarting the spacewalk service. I ran an apt-get update and saw that the spacewalk.list file was now in the new format. I then tried installing a package as a test and got this:

    $ sudo apt-get install unzip
    Reading package lists… Done
    Building dependency tree
    Reading state information… Done
    You might want to run ‘apt-get -f install’ to correct these:
    The following packages have unmet dependencies:
    apt-xapian-index : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    dh-python : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    language-selector-common : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    lsb-release : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    python-apt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-chardet : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-dbus : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-debian : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-gi : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-gobject-2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-libxml2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-newt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-openssl : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-requests : Depends: python:any (= 2.7.5-5~) but it is not installable
    python-six : Depends: python:any (>= 2.7.5-5~) but it is not installable
    Depends: python:any (= 2.7.5-5~) but it is not installable
    Depends: python:any (= 2.7.1-0ubuntu2) but it is not installable
    python3-commandnotfound : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    python3-distupgrade : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    python3-update-manager : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    ubuntu-release-upgrader-core : Depends: python3:any (>= 3.2~) but it is not installable
    ufw : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    update-manager-core : Depends: python3:any (>= 3.2~) but it is not installable
    E: Unmet dependencies. Try ‘apt-get -f install’ with no packages (or specify a solution).

    Any guesses?
    -thanks

    • mike
    • June 20th, 2016 12:55pm

    Hi Phil,

    Thank you for sharing this info. We are trying to use Spacewalk 2.4 for our Ubuntu 16.04 VM’s so that we can run updates.

    Per your blog, I was able to sync the APT repo to spacewalk server, sign the repo with GPG, register the Ubuntu client to spacewalk without any issues.

    When I tried to run the apt-get update on the Ubuntu 16.04 client, I get the following error:

    root@ubuntu:~# apt-get clean
    root@buuntu:~# apt-get update
    Apt-Spacewalk: Updating sources.list
    Ign:1 spacewalk://x.x.x.x xenial-main InRelease
    Get:2 spacewalk://x.x.x.x xenial-main Release [934 B]
    Reading package lists… Error!
    E: Problem parsing dependency Breaks
    E: Error occurred while processing kmod (NewVersion2)
    E: Problem with MergeList /var/lib/apt/lists/23.253.248.26_dists_xenial-main_repodata_binary-amd64_Packages
    E: The package lists or status file could not be parsed or opened.
    rootubuntu:~#

    Any help/pointers would be very much appreciated.

    Thank You

      • phil
      • June 20th, 2016 1:04pm

      Hi Mike,
      Unfortunately I didnt have the chance to work with SW 2.4 and Ubuntu16 clients yet, my “wisdom” ends with 2.3 and Ubuntu12/14. So from my point of view you are bleeding edge 😉
      So I would try getting 2.4 + Ubuntu16 working (including package install via APT) without GPG-signed repos first. If thats working, play around with GPG. Thats rather hackish anyways as it also requires server-side patches :/
      I’d appreciate any feedback if you solve anything around this 🙂

  1. No trackbacks yet.