GPG signing APT repository in Spacewalk
After following my article on how to register Ubuntu and Debian clients with Spacewalk you might have noticed a APT warning
This happens because the APT repositories in Spacewalk are not GPG signed. You can still install the packages but have to acknowledge it by entering “yes”.
For various reasons it would be better to have the repository signed. Read on after the jump how I got that working.
Background information
… about what turns an APT repo into a signed repo:
– Sign each package individually
– Dont sign each package but the entire repo at once
As all? major repo maintainers (Ubuntu and Debian) use the second approach and we are mirroring them, we will mimic them.
Two repository metadata files are necessary and missing: Release and Release.gpg
(Here is the Debian wiki about secure-apt explaining all this in detail https://wiki.debian.org/SecureApt)
Release
Contains information about the content of the repo, such as architecture, but most important for us: The MD5 / SHA checksums for the Packages and Packages.gz files which contain checksums for each package.
Release.gpg
Contains the GPG signature for the “Release” file.
Chain of Trust
APT uses the checksums in the Packages file to verify each package. It also uses the checksum in the Release file to verify the Packages file.
For verifying the Release file itself it uses GPG against the Release.gpg file. The fitting GPG pubkey needs to be imported in APT first.
Preparation
Creating the files Release and Release.gpg is really simple. What turned out to be a bit of a mess was the wrongly formatted /etc/apt/sources.list.d/spacewalk.list that gets generated by apt-transport-spacewalk package based on the channels the system is subscribed to.
deb spacewalk://spacewalk.xxx.lan channels: main precise-spacewalk-client precise-security precise-updates
Which only worked because of some “creative” magic in /usr/lib/apt/methods/spacewalk
# deb http://site.example.com/debian distribution component1 component2
# where distribution is sth like precise, precise-updates, trusty, trusty-backports etc.. and component fixed to repodata
# see https://wiki.debian.org/SourcesList
deb spacewalk://spacewalk.xxx.lan precise repodata
deb spacewalk://spacewalk.xxx.lan precise-updates repodata
deb spacewalk://spacewalk.xxx.lan precise-security repodata
If we dont change the format, the Release / Release.gpg file will only be queried for one of our channels. Find the required patch for /usr/lib/apt-spacewalk/pre_invoke.py here as I submitted it to the Bugtracker.
Next up we need to patch /usr/lib/apt/methods/spacewalk , diff is here
Unfortunately we also need to patch a single line in the Spacewalk Server package to allow the download of our two new metadata files, get it here and afterwards do
Now we fixed everything needed so we can finally GPG sign our repo.
Signing the Repo
Ok first of all we need a GPG key. On Spacewalk server it needs to be imported to the GPG keyring. On the clients only the public key needs to be imported to APT.
# 4) RSA sign only
# 0 expires never
# Real Name: Spacewalk, email, comment: For GPG signing APT repos
gpg --gen-key
# Export the key parts
gpg --export-secret-keys --armor spacewalk-admins@holidaycheck.com > spacewalk.gpg.privkey
gpg --armor --export spacewalk-admins@holidaycheck.com > spacewalk.gpg.pubkey
# Import the public key on ALL clients
apt-key add spacewalk.gpg.pubkey
Now we need to sign our repos. I created a handy bash script for that purpose that can be used as cronjob until this gets implemented in the Spacewalk server repo generation.
Take this script here and run it in every folder in /var/cache/rhn/repodata/
For each channel you have one folder in there like precise, precise-updates, precise-backports….
/opt/spacewalk/secureApt.sh precise precise
cd /var/cache/rhn/repodata/precise-security
/opt/spacewalk/secureApt.sh precise precise-security
cd /var/cache/rhn/repodata/precise-backports
/opt/spacewalk/secureApt.sh precise precise-backports
This creates a Release and Release.gpg file for every channel based on the Packages and Packages.gz files. These can change after every repo sync so you need to run the secureApt cron right after every repo sync !
If you now apt-get update on your clients, you should see the files being downloaded and when installing packages you shouldnt get warnings about unauthenticated packages.
Apt-Spacewalk: Updating sources.list
Get:1 spacewalk://spacewalk.xxx.lan precise Release.gpg [490 B]
Get:2 spacewalk://spacewalk.xxx.lan precise-security Release.gpg [490 B]
Get:3 spacewalk://spacewalk.xxx.lan precise-updates Release.gpg [490 B]
Get:4 spacewalk://spacewalk.xxx.lan precise Release [927 B]
Get:5 spacewalk://spacewalk.xxx.lan precise-security Release [932 B]
Get:6 spacewalk://spacewalk.xxx.lan precise-updates Release [935 B]
Get:7 spacewalk://spacewalk.xxx.lan precise/repodata amd64 Packages [1,890 kB]
Get:8 spacewalk://spacewalk.xxx.lan precise/repodata i386 Packages [1,890 kB]
Ign spacewalk://spacewalk.xxx.lan precise/repodata TranslationIndex
Ign spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata TranslationIndex
Get:9 spacewalk://spacewalk.xxx.lan precise-security/repodata amd64 Packages [724 kB]
Get:10 spacewalk://spacewalk.xxx.lan precise-security/repodata i386 Packages [724 kB]
Ign spacewalk://spacewalk.xxx.lan precise-security/repodata TranslationIndex
Get:11 spacewalk://spacewalk.xxx.lan precise-updates/repodata amd64 Packages [1,364 kB]
Get:12 spacewalk://spacewalk.xxx.lan precise-updates/repodata i386 Packages [1,364 kB]
Ign spacewalk://spacewalk.xxx.lan precise-updates/repodata TranslationIndex
Get:13 spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata amd64 Packages [1,238 B]
Get:14 spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata i386 Packages [1,238 B]
....
Fetched 7,964 kB in 6s (1,229 kB/s)
Reading package lists... Done
Now, enjoy your signed repo and report back any bugs you might notice.
After applying the patches and restarting the spacewalk service. I ran an apt-get update and saw that the spacewalk.list file was now in the new format. I then tried installing a package as a test and got this:
$ sudo apt-get install unzip
Reading package lists… Done
Building dependency tree
Reading state information… Done
You might want to run ‘apt-get -f install’ to correct these:
The following packages have unmet dependencies:
apt-xapian-index : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
dh-python : Depends: python3:any (>= 3.3.2-2~) but it is not installable
language-selector-common : Depends: python3:any (>= 3.3.2-2~) but it is not installable
lsb-release : Depends: python3:any (>= 3.3.2-2~) but it is not installable
python-apt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-chardet : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-dbus : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-debian : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-gi : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-gobject-2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-libxml2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-newt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-openssl : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-requests : Depends: python:any (= 2.7.5-5~) but it is not installable
python-six : Depends: python:any (>= 2.7.5-5~) but it is not installable
Depends: python:any (= 2.7.5-5~) but it is not installable
Depends: python:any (= 2.7.1-0ubuntu2) but it is not installable
python3-commandnotfound : Depends: python3:any (>= 3.3.2-2~) but it is not installable
python3-distupgrade : Depends: python3:any (>= 3.3.2-2~) but it is not installable
python3-update-manager : Depends: python3:any (>= 3.3.2-2~) but it is not installable
ubuntu-release-upgrader-core : Depends: python3:any (>= 3.2~) but it is not installable
ufw : Depends: python3:any (>= 3.3.2-2~) but it is not installable
update-manager-core : Depends: python3:any (>= 3.2~) but it is not installable
E: Unmet dependencies. Try ‘apt-get -f install’ with no packages (or specify a solution).
Any guesses?
-thanks
This sounds like this bug here https://bugzilla.redhat.com/show_bug.cgi?id=1243387 with a suggested workaround by “mhofmann” in comments of /spacewalk/registering-ubuntu-and-debian-servers-with-spacewalk
Hi Phil,
Thank you for sharing this info. We are trying to use Spacewalk 2.4 for our Ubuntu 16.04 VM’s so that we can run updates.
Per your blog, I was able to sync the APT repo to spacewalk server, sign the repo with GPG, register the Ubuntu client to spacewalk without any issues.
When I tried to run the apt-get update on the Ubuntu 16.04 client, I get the following error:
root@ubuntu:~# apt-get clean
root@buuntu:~# apt-get update
Apt-Spacewalk: Updating sources.list
Ign:1 spacewalk://x.x.x.x xenial-main InRelease
Get:2 spacewalk://x.x.x.x xenial-main Release [934 B]
Reading package lists… Error!
E: Problem parsing dependency Breaks
E: Error occurred while processing kmod (NewVersion2)
E: Problem with MergeList /var/lib/apt/lists/23.253.248.26_dists_xenial-main_repodata_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.
rootubuntu:~#
Any help/pointers would be very much appreciated.
Thank You
Hi Mike,
Unfortunately I didnt have the chance to work with SW 2.4 and Ubuntu16 clients yet, my “wisdom” ends with 2.3 and Ubuntu12/14. So from my point of view you are bleeding edge 😉
So I would try getting 2.4 + Ubuntu16 working (including package install via APT) without GPG-signed repos first. If thats working, play around with GPG. Thats rather hackish anyways as it also requires server-side patches :/
I’d appreciate any feedback if you solve anything around this 🙂
What is the actual RedHat bug you have these filed under? I have created updated patches for this that work with the current CentOS Spacewalk packages (2.6.78-1) and I would like to upload them. I also have new patches for the Ubuntu side as well, but the backend-server patches need to be applied first.
oh, awesome to hear! see https://bugzilla.redhat.com/show_bug.cgi?id=1198723
I also link to some patches attached to that bugreport in this article here
I can’t seem to log in to the RedHat bugzilla, reset what I think is my account, or even create an account. I keep on getting “You submitted changes to createaccount.cgi with an invalid token, which may indicate that someone tried to abuse you, for instance by making you click on a URL which redirected you here without your consent.
Are you sure you want to commit these changes?” and clicking yes just redisplays the same message. I tried 3 browsers, so I doubt it’s on my end, although there’s a small chance our firewall might be blocking a non-HTTP-based Javascript action.
You might be hitting this problem.
https://bugzilla.redhat.com/show_bug.cgi?id=1226329
Hi Phil,
I’m hitting a new issue here and not sure what to to next.
Running “apt-get update” on trusty clients end with this result…
Ign spacewalk://SPACEWALK-URL trusty InRelease
E: Invalid message from method spacewalk: SSL exception (-1, ‘Unexpected EOF’)
102 Status
Message: Waiting for headers
URI: spacewalk://SPACEWALK-URL/dists/trusty-security/InRelease
Precise systems are running fine though. Well even Trusty was working when I left the office on Friday.
Running or removing and running secureApt script again, does not resolve the issue.
Not sure what file it’s complaining about: “Unexpected EOF”.
Thank you in advance
So for this particular issue i had to remove /XMLRPC from the apt spacewalk.list file
from: spacewalk://spacewalk.hc.lan/XMLRPC channels: ….
to: spacewalk://spacewalk.hc.lan channels:
Seems to be working now.