GPG signing APT repository in Spacewalk
After following my article on how to register Ubuntu and Debian clients with Spacewalk you might have noticed a APT warning
This happens because the APT repositories in Spacewalk are not GPG signed. You can still install the packages but have to acknowledge it by entering “yes”.
For various reasons it would be better to have the repository signed. Read on after the jump how I got that working.
Background information
… about what turns an APT repo into a signed repo:
– Sign each package individually
– Dont sign each package but the entire repo at once
As all? major repo maintainers (Ubuntu and Debian) use the second approach and we are mirroring them, we will mimic them.
Two repository metadata files are necessary and missing: Release and Release.gpg
(Here is the Debian wiki about secure-apt explaining all this in detail https://wiki.debian.org/SecureApt)
Release
Contains information about the content of the repo, such as architecture, but most important for us: The MD5 / SHA checksums for the Packages and Packages.gz files which contain checksums for each package.
Release.gpg
Contains the GPG signature for the “Release” file.
Chain of Trust
APT uses the checksums in the Packages file to verify each package. It also uses the checksum in the Release file to verify the Packages file.
For verifying the Release file itself it uses GPG against the Release.gpg file. The fitting GPG pubkey needs to be imported in APT first.
Preparation
Creating the files Release and Release.gpg is really simple. What turned out to be a bit of a mess was the wrongly formatted /etc/apt/sources.list.d/spacewalk.list that gets generated by apt-transport-spacewalk package based on the channels the system is subscribed to.
deb spacewalk://spacewalk.xxx.lan channels: main precise-spacewalk-client precise-security precise-updates
Which only worked because of some “creative” magic in /usr/lib/apt/methods/spacewalk
# deb http://site.example.com/debian distribution component1 component2
# where distribution is sth like precise, precise-updates, trusty, trusty-backports etc.. and component fixed to repodata
# see https://wiki.debian.org/SourcesList
deb spacewalk://spacewalk.xxx.lan precise repodata
deb spacewalk://spacewalk.xxx.lan precise-updates repodata
deb spacewalk://spacewalk.xxx.lan precise-security repodata
If we dont change the format, the Release / Release.gpg file will only be queried for one of our channels. Find the required patch for /usr/lib/apt-spacewalk/pre_invoke.py here as I submitted it to the Bugtracker.
Next up we need to patch /usr/lib/apt/methods/spacewalk , diff is here
Unfortunately we also need to patch a single line in the Spacewalk Server package to allow the download of our two new metadata files, get it here and afterwards do
Now we fixed everything needed so we can finally GPG sign our repo.
Signing the Repo
Ok first of all we need a GPG key. On Spacewalk server it needs to be imported to the GPG keyring. On the clients only the public key needs to be imported to APT.
# 4) RSA sign only
# 0 expires never
# Real Name: Spacewalk, email, comment: For GPG signing APT repos
gpg --gen-key
# Export the key parts
gpg --export-secret-keys --armor spacewalk-admins@holidaycheck.com > spacewalk.gpg.privkey
gpg --armor --export spacewalk-admins@holidaycheck.com > spacewalk.gpg.pubkey
# Import the public key on ALL clients
apt-key add spacewalk.gpg.pubkey
Now we need to sign our repos. I created a handy bash script for that purpose that can be used as cronjob until this gets implemented in the Spacewalk server repo generation.
Take this script here and run it in every folder in /var/cache/rhn/repodata/
For each channel you have one folder in there like precise, precise-updates, precise-backports….
/opt/spacewalk/secureApt.sh precise precise
cd /var/cache/rhn/repodata/precise-security
/opt/spacewalk/secureApt.sh precise precise-security
cd /var/cache/rhn/repodata/precise-backports
/opt/spacewalk/secureApt.sh precise precise-backports
This creates a Release and Release.gpg file for every channel based on the Packages and Packages.gz files. These can change after every repo sync so you need to run the secureApt cron right after every repo sync !
If you now apt-get update on your clients, you should see the files being downloaded and when installing packages you shouldnt get warnings about unauthenticated packages.
Apt-Spacewalk: Updating sources.list
Get:1 spacewalk://spacewalk.xxx.lan precise Release.gpg [490 B]
Get:2 spacewalk://spacewalk.xxx.lan precise-security Release.gpg [490 B]
Get:3 spacewalk://spacewalk.xxx.lan precise-updates Release.gpg [490 B]
Get:4 spacewalk://spacewalk.xxx.lan precise Release [927 B]
Get:5 spacewalk://spacewalk.xxx.lan precise-security Release [932 B]
Get:6 spacewalk://spacewalk.xxx.lan precise-updates Release [935 B]
Get:7 spacewalk://spacewalk.xxx.lan precise/repodata amd64 Packages [1,890 kB]
Get:8 spacewalk://spacewalk.xxx.lan precise/repodata i386 Packages [1,890 kB]
Ign spacewalk://spacewalk.xxx.lan precise/repodata TranslationIndex
Ign spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata TranslationIndex
Get:9 spacewalk://spacewalk.xxx.lan precise-security/repodata amd64 Packages [724 kB]
Get:10 spacewalk://spacewalk.xxx.lan precise-security/repodata i386 Packages [724 kB]
Ign spacewalk://spacewalk.xxx.lan precise-security/repodata TranslationIndex
Get:11 spacewalk://spacewalk.xxx.lan precise-updates/repodata amd64 Packages [1,364 kB]
Get:12 spacewalk://spacewalk.xxx.lan precise-updates/repodata i386 Packages [1,364 kB]
Ign spacewalk://spacewalk.xxx.lan precise-updates/repodata TranslationIndex
Get:13 spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata amd64 Packages [1,238 B]
Get:14 spacewalk://spacewalk.xxx.lan precise-spacewalk-client/repodata i386 Packages [1,238 B]
....
Fetched 7,964 kB in 6s (1,229 kB/s)
Reading package lists... Done
Now, enjoy your signed repo and report back any bugs you might notice.
After applying the patches and restarting the spacewalk service. I ran an apt-get update and saw that the spacewalk.list file was now in the new format. I then tried installing a package as a test and got this:
$ sudo apt-get install unzip
Reading package lists… Done
Building dependency tree
Reading state information… Done
You might want to run ‘apt-get -f install’ to correct these:
The following packages have unmet dependencies:
apt-xapian-index : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
dh-python : Depends: python3:any (>= 3.3.2-2~) but it is not installable
language-selector-common : Depends: python3:any (>= 3.3.2-2~) but it is not installable
lsb-release : Depends: python3:any (>= 3.3.2-2~) but it is not installable
python-apt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-chardet : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-dbus : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-debian : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-gi : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-gobject-2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-libxml2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-newt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-openssl : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
python-requests : Depends: python:any (= 2.7.5-5~) but it is not installable
python-six : Depends: python:any (>= 2.7.5-5~) but it is not installable
Depends: python:any (= 2.7.5-5~) but it is not installable
Depends: python:any (= 2.7.1-0ubuntu2) but it is not installable
python3-commandnotfound : Depends: python3:any (>= 3.3.2-2~) but it is not installable
python3-distupgrade : Depends: python3:any (>= 3.3.2-2~) but it is not installable
python3-update-manager : Depends: python3:any (>= 3.3.2-2~) but it is not installable
ubuntu-release-upgrader-core : Depends: python3:any (>= 3.2~) but it is not installable
ufw : Depends: python3:any (>= 3.3.2-2~) but it is not installable
update-manager-core : Depends: python3:any (>= 3.2~) but it is not installable
E: Unmet dependencies. Try ‘apt-get -f install’ with no packages (or specify a solution).
Any guesses?
-thanks
This sounds like this bug here https://bugzilla.redhat.com/show_bug.cgi?id=1243387 with a suggested workaround by “mhofmann” in comments of http://www.devops-blog.net/spacewalk/registering-ubuntu-and-debian-servers-with-spacewalk
Hi Phil,
Thank you for sharing this info. We are trying to use Spacewalk 2.4 for our Ubuntu 16.04 VM’s so that we can run updates.
Per your blog, I was able to sync the APT repo to spacewalk server, sign the repo with GPG, register the Ubuntu client to spacewalk without any issues.
When I tried to run the apt-get update on the Ubuntu 16.04 client, I get the following error:
root@ubuntu:~# apt-get clean
root@buuntu:~# apt-get update
Apt-Spacewalk: Updating sources.list
Ign:1 spacewalk://x.x.x.x xenial-main InRelease
Get:2 spacewalk://x.x.x.x xenial-main Release [934 B]
Reading package lists… Error!
E: Problem parsing dependency Breaks
E: Error occurred while processing kmod (NewVersion2)
E: Problem with MergeList /var/lib/apt/lists/23.253.248.26_dists_xenial-main_repodata_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.
rootubuntu:~#
Any help/pointers would be very much appreciated.
Thank You
Hi Mike,
Unfortunately I didnt have the chance to work with SW 2.4 and Ubuntu16 clients yet, my “wisdom” ends with 2.3 and Ubuntu12/14. So from my point of view you are bleeding edge 😉
So I would try getting 2.4 + Ubuntu16 working (including package install via APT) without GPG-signed repos first. If thats working, play around with GPG. Thats rather hackish anyways as it also requires server-side patches :/
I’d appreciate any feedback if you solve anything around this 🙂
What is the actual RedHat bug you have these filed under? I have created updated patches for this that work with the current CentOS Spacewalk packages (2.6.78-1) and I would like to upload them. I also have new patches for the Ubuntu side as well, but the backend-server patches need to be applied first.
oh, awesome to hear! see https://bugzilla.redhat.com/show_bug.cgi?id=1198723
I also link to some patches attached to that bugreport in this article here
I can’t seem to log in to the RedHat bugzilla, reset what I think is my account, or even create an account. I keep on getting “You submitted changes to createaccount.cgi with an invalid token, which may indicate that someone tried to abuse you, for instance by making you click on a URL which redirected you here without your consent.
Are you sure you want to commit these changes?” and clicking yes just redisplays the same message. I tried 3 browsers, so I doubt it’s on my end, although there’s a small chance our firewall might be blocking a non-HTTP-based Javascript action.
You might be hitting this problem.
https://bugzilla.redhat.com/show_bug.cgi?id=1226329
Hi Phil,
I’m hitting a new issue here and not sure what to to next.
Running “apt-get update” on trusty clients end with this result…
Ign spacewalk://SPACEWALK-URL trusty InRelease
E: Invalid message from method spacewalk: SSL exception (-1, ‘Unexpected EOF’)
102 Status
Message: Waiting for headers
URI: spacewalk://SPACEWALK-URL/dists/trusty-security/InRelease
Precise systems are running fine though. Well even Trusty was working when I left the office on Friday.
Running or removing and running secureApt script again, does not resolve the issue.
Not sure what file it’s complaining about: “Unexpected EOF”.
Thank you in advance
So for this particular issue i had to remove /XMLRPC from the apt spacewalk.list file
from: spacewalk://spacewalk.hc.lan/XMLRPC channels: ….
to: spacewalk://spacewalk.hc.lan channels:
Seems to be working now.
I get an error stating that the GPG signature is invalid, even though as far as I can tell everything matches up:
# apt update
Apt-Spacewalk: Updating sources.list
Ign:1 spacewalk://spacewalk.server extras-ubuntu InRelease
[…]
Reading package lists… Done
W: GPG error: spacewalk://spacewalk.server extras-ubuntu Release:
The following signatures were invalid: 41CDF527725B5CD68EA405AA27D22AF72385D175
The key is imported to apt-key
# apt-key list
/etc/apt/trusted.gpg
——————–
pub rsa2048 2018-03-23 [SC]
41CD F527 725B 5CD6 8EA4 05AA 27D2 2AF7 2385 D175
uid [ unknown] Spacewalk (For GPG signing APT repos)
And the Release file is, in fact, signed with the correct key:
# gpg -k
/root/.gnupg/pubring.gpg
————————
pub 2048R/2385D175 2018-03-23
uid Spacewalk (For GPG signing APT repos)
# gpg ./Release.gpg
Detached signature.
Please enter name of data file: Release
gpg: Signature made Fri 23 Mar 2018 10:43:50 AM EDT using RSA key ID 2385D175
gpg: Good signature from “Spacewalk (For GPG signing APT repos) ”
Any ideas?
I am getting the same error about the GPG signature being invalid, but everything checks out. Were you able to resolve this? does anyone have any ideas?
Nope, never resolved it. I simply accepted that, at present, this isn’t a thing that works based on the information that is readily available.
This is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850839: gpg is signing the Release file with SHA1 by default but apt-get doesn’t accept that any more. Adding –personal-digest-preferences SHA256 to the gpg –sign call in secureApt.sh fixes it.
maybe the key/signature is sha1 instead of sha256?
Hi!
I tried using your script to sign on Ubuntu 16 repository. I get the Release file and all seems well, expect that client tells me
—
W: The repository ‘spacewalk://<> ubuntu-16 Release’ does not have a Release file.
—
Also for some reason after modifying the scripts I’m getting this error but only once. I’m not sure if this is related or is it because I tried to tackle missing “Multi-Arch”-data issues at the same time.
—
E: Failed to fetch spacewalk://<>/dists/ubuntu-16/repodata/binary-amd64/Packages 404 Not Found
—
I think I can somehow find out what is wrong with the later one but any pointers about the Release file?
apt now requires that the Date in Release files be in UTC.
so securApt.sh should be updated to reflect that when it creates the Release file.
this should work for securApt.sh
DATE=`date –utc “+%a, %d %b %Y %H:%M:%S “`
GPG_PASS=’Kiewitc1ff11c1p’
HEADER=”Origin: Ubuntu
Label: Ubuntu
Suite: $2
Version: 12.04
Codename: $1
Date: ${DATE}UTC
if you like, you can do a PR here https://github.com/philicious/spacewalk-scripts
I’m no longer working with Spacewalk and cannot easily test it
Great work and thanks Phil.
I am working with Spacewalk 2.9 and try to get ubuntu Repos working. Sync is working fine.. added channels using example from https://www.redhat.com/archives/spacewalk-list/2017-September/msg00042.html and client registration also worked and with Phils patches in /usr/lib/apt-spacewalk/pre_invoke.py and /usr/lib/apt/methods/spacewalk apt update leaves correct /etc/apt/sources.list.d/spacewalk.list file.
When I run apt update 2nd time I get this error:
root@ubuntu1604-server:~# apt update
Apt-Spacewalk: Updating sources.list
WARNING:root:could not open file ‘/etc/apt/sources.list’
Ign:1 spacewalk://spacewalk.mydomain.de xenial-backports InRelease
Ign:2 spacewalk://spacewalk.mydomain.de xenial-security InRelease
Ign:3 spacewalk://spacewalk.mydomain.de xenial-updates InRelease
Ign:4 spacewalk://spacewalk.mydomain.de xenial InRelease
Get:5 spacewalk://spacewalk.mydomain.de xenial-backports Release [495 B]
Get:6 spacewalk://spacewalk.mydomain.de xenial-security Release [494 B]
Get:7 spacewalk://spacewalk.mydomain.de xenial-updates Release [493 B]
Get:8 spacewalk://spacewalk.mydomain.de xenial Release [485 B]
Get:9 spacewalk://spacewalk.mydomain.de xenial-backports Release.gpg [490 B]
Get:10 spacewalk://spacewalk.mydomain.de xenial-security Release.gpg [490 B]
Get:11 spacewalk://spacewalk.mydomain.de xenial-updates Release.gpg [490 B]
Get:12 spacewalk://spacewalk.mydomain.de xenial Release.gpg [490 B]
Reading package lists… Done
W: spacewalk://spacewalk.mydomain.de/dists/xenial-backports/Release.gpg: Signature by key D554E63E9E88FEDCCF9D530FF446FCB1E3DF6327 uses weak digest algorithm (SHA1)
W: spacewalk://spacewalk.mydomain.de/dists/xenial-security/Release.gpg: Signature by key D554E63E9E88FEDCCF9D530FF446FCB1E3DF6327 uses weak digest algorithm (SHA1)
W: spacewalk://spacewalk.mydomain.de/dists/xenial-updates/Release.gpg: Signature by key D554E63E9E88FEDCCF9D530FF446FCB1E3DF6327 uses weak digest algorithm (SHA1)
W: spacewalk://spacewalk.mydomain.de/dists/xenial/Release.gpg: Signature by key D554E63E9E88FEDCCF9D530FF446FCB1E3DF6327 uses weak digest algorithm (SHA1)
E: Failed to fetch spacewalk://spacewalk.mydomain.de/dists/xenial-backports/Release
E: Failed to fetch spacewalk://spacewalk.mydomain.de/dists/xenial-security/Release
E: Failed to fetch spacewalk://spacewalk.mydomain.de/dists/xenial-updates/Release
E: Failed to fetch spacewalk://spacewalk.mydomain.de/dists/xenial/Release
E: Some index files failed to download. They have been ignored, or old ones used instead.
Also on the Server side if running secureAPT.sh there are some files missing as it seems. For example the Packages file.
The Directories in /var/cache/rhn/repodata were not created automaticaly. I added them mkdir xenial xenial-updates… and runned modified secureAPT.sh (replaced 12 by 16 and Distro Name)
I really don’t know which part of the documentation I am missing- Read anything twice but now I have no glue how to get forward-
Any help is welcome.
Yours,
Marc