GPG signing APT repository in Spacewalk

After following my article on how to register Ubuntu and Debian clients with Spacewalk you might have noticed a APT warning

WARNING: The following packages cannot be authenticated.

This happens because the APT repositories in Spacewalk are not GPG signed. You can still install the packages but have to acknowledge it by entering “yes”.

For various reasons it would be better to have the repository signed. Read on after the jump how I got that working.

Background information

… about what turns an APT repo into a signed repo:
– Sign each package individually
– Dont sign each package but the entire repo at once

As all? major repo maintainers (Ubuntu and Debian) use the second approach and we are mirroring them, we will mimic them.
Two repository metadata files are necessary and missing: Release and Release.gpg

(Here is the Debian wiki about secure-apt explaining all this in detail


Contains information about the content of the repo, such as architecture, but most important for us: The MD5 / SHA checksums for the Packages and Packages.gz files which contain checksums for each package.


Contains the GPG signature for the “Release” file.

Chain of Trust

APT uses the checksums in the Packages file to verify each package. It also uses the checksum in the Release file to verify the Packages file.
For verifying the Release file itself it uses GPG against the Release.gpg file. The fitting GPG pubkey needs to be imported in APT first.


Creating the files Release and Release.gpg is really simple. What turned out to be a bit of a mess was the wrongly formatted /etc/apt/sources.list.d/spacewalk.list that gets generated by apt-transport-spacewalk package based on the channels the system is subscribed to.

# Old / bad format with all "components" in one line
deb spacewalk:// channels: main precise-spacewalk-client precise-security precise-updates

Which only worked because of some “creative” magic in /usr/lib/apt/methods/spacewalk

# New format based on
# deb distribution component1 component2
# where distribution is sth like precise, precise-updates, trusty, trusty-backports etc.. and component fixed to repodata
# see
deb spacewalk:// precise repodata
deb spacewalk:// precise-updates repodata
deb spacewalk:// precise-security repodata

If we dont change the format, the Release / Release.gpg file will only be queried for one of our channels. Find the required patch for /usr/lib/apt-spacewalk/ here as I submitted it to the Bugtracker.

Next up we need to patch /usr/lib/apt/methods/spacewalk , diff is here

Unfortunately we also need to patch a single line in the Spacewalk Server package to allow the download of our two new metadata files, get it here and afterwards do

spacewalk-service restart

Now we fixed everything needed so we can finally GPG sign our repo.

Signing the Repo

Ok first of all we need a GPG key. On Spacewalk server it needs to be imported to the GPG keyring. On the clients only the public key needs to be imported to APT.

# Generate a key, answer:
#     4) RSA sign only
#     0 expires never
#     Real Name: Spacewalk, email, comment: For GPG signing APT repos
gpg --gen-key
# Export the key parts
gpg --export-secret-keys --armor > spacewalk.gpg.privkey
gpg --armor --export > spacewalk.gpg.pubkey
# Import the public key on ALL clients
apt-key add spacewalk.gpg.pubkey

Now we need to sign our repos. I created a handy bash script for that purpose that can be used as cronjob until this gets implemented in the Spacewalk server repo generation.
Take this script here and run it in every folder in /var/cache/rhn/repodata/
For each channel you have one folder in there like precise, precise-updates, precise-backports….

cd /var/cache/rhn/repodata/precise
/opt/spacewalk/ precise precise
cd /var/cache/rhn/repodata/precise-security
/opt/spacewalk/ precise precise-security
cd /var/cache/rhn/repodata/precise-backports
/opt/spacewalk/ precise precise-backports

This creates a Release and Release.gpg file for every channel based on the Packages and Packages.gz files. These can change after every repo sync so you need to run the secureApt cron right after every repo sync !

If you now apt-get update on your clients, you should see the files being downloaded and when installing packages you shouldnt get warnings about unauthenticated packages.

4# apt-get update
Apt-Spacewalk: Updating sources.list
Get:1 spacewalk:// precise Release.gpg [490 B]
Get:2 spacewalk:// precise-security Release.gpg [490 B]
Get:3 spacewalk:// precise-updates Release.gpg [490 B]
Get:4 spacewalk:// precise Release [927 B]
Get:5 spacewalk:// precise-security Release [932 B]
Get:6 spacewalk:// precise-updates Release [935 B]
Get:7 spacewalk:// precise/repodata amd64 Packages [1,890 kB]
Get:8 spacewalk:// precise/repodata i386 Packages [1,890 kB]
Ign spacewalk:// precise/repodata TranslationIndex
Ign spacewalk:// precise-spacewalk-client/repodata TranslationIndex
Get:9 spacewalk:// precise-security/repodata amd64 Packages [724 kB]
Get:10 spacewalk:// precise-security/repodata i386 Packages [724 kB]
Ign spacewalk:// precise-security/repodata TranslationIndex
Get:11 spacewalk:// precise-updates/repodata amd64 Packages [1,364 kB]
Get:12 spacewalk:// precise-updates/repodata i386 Packages [1,364 kB]
Ign spacewalk:// precise-updates/repodata TranslationIndex
Get:13 spacewalk:// precise-spacewalk-client/repodata amd64 Packages [1,238 B]
Get:14 spacewalk:// precise-spacewalk-client/repodata i386 Packages [1,238 B]
Fetched 7,964 kB in 6s (1,229 kB/s)
Reading package lists... Done

Now, enjoy your signed repo and report back any bugs you might notice.

    • Derek
    • December 29th, 2015 8:47pm

    After applying the patches and restarting the spacewalk service. I ran an apt-get update and saw that the spacewalk.list file was now in the new format. I then tried installing a package as a test and got this:

    $ sudo apt-get install unzip
    Reading package lists… Done
    Building dependency tree
    Reading state information… Done
    You might want to run ‘apt-get -f install’ to correct these:
    The following packages have unmet dependencies:
    apt-xapian-index : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    dh-python : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    language-selector-common : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    lsb-release : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    python-apt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-chardet : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-dbus : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-debian : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-gi : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-gobject-2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-libxml2 : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-newt : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-openssl : Depends: python:any (>= 2.7.1-0ubuntu2) but it is not installable
    python-requests : Depends: python:any (= 2.7.5-5~) but it is not installable
    python-six : Depends: python:any (>= 2.7.5-5~) but it is not installable
    Depends: python:any (= 2.7.5-5~) but it is not installable
    Depends: python:any (= 2.7.1-0ubuntu2) but it is not installable
    python3-commandnotfound : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    python3-distupgrade : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    python3-update-manager : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    ubuntu-release-upgrader-core : Depends: python3:any (>= 3.2~) but it is not installable
    ufw : Depends: python3:any (>= 3.3.2-2~) but it is not installable
    update-manager-core : Depends: python3:any (>= 3.2~) but it is not installable
    E: Unmet dependencies. Try ‘apt-get -f install’ with no packages (or specify a solution).

    Any guesses?

    • mike
    • June 20th, 2016 12:55pm

    Hi Phil,

    Thank you for sharing this info. We are trying to use Spacewalk 2.4 for our Ubuntu 16.04 VM’s so that we can run updates.

    Per your blog, I was able to sync the APT repo to spacewalk server, sign the repo with GPG, register the Ubuntu client to spacewalk without any issues.

    When I tried to run the apt-get update on the Ubuntu 16.04 client, I get the following error:

    root@ubuntu:~# apt-get clean
    root@buuntu:~# apt-get update
    Apt-Spacewalk: Updating sources.list
    Ign:1 spacewalk://x.x.x.x xenial-main InRelease
    Get:2 spacewalk://x.x.x.x xenial-main Release [934 B]
    Reading package lists… Error!
    E: Problem parsing dependency Breaks
    E: Error occurred while processing kmod (NewVersion2)
    E: Problem with MergeList /var/lib/apt/lists/
    E: The package lists or status file could not be parsed or opened.

    Any help/pointers would be very much appreciated.

    Thank You

      • phil
      • June 20th, 2016 1:04pm

      Hi Mike,
      Unfortunately I didnt have the chance to work with SW 2.4 and Ubuntu16 clients yet, my “wisdom” ends with 2.3 and Ubuntu12/14. So from my point of view you are bleeding edge 😉
      So I would try getting 2.4 + Ubuntu16 working (including package install via APT) without GPG-signed repos first. If thats working, play around with GPG. Thats rather hackish anyways as it also requires server-side patches :/
      I’d appreciate any feedback if you solve anything around this 🙂

        • Paul-Andre Panon
        • June 28th, 2017 5:30pm

        What is the actual RedHat bug you have these filed under? I have created updated patches for this that work with the current CentOS Spacewalk packages (2.6.78-1) and I would like to upload them. I also have new patches for the Ubuntu side as well, but the backend-server patches need to be applied first.

          • phil
          • June 28th, 2017 7:30pm

          oh, awesome to hear! see

          I also link to some patches attached to that bugreport in this article here

            • Paul-Andre Panon
            • June 28th, 2017 9:34pm

            I can’t seem to log in to the RedHat bugzilla, reset what I think is my account, or even create an account. I keep on getting “You submitted changes to createaccount.cgi with an invalid token, which may indicate that someone tried to abuse you, for instance by making you click on a URL which redirected you here without your consent.
            Are you sure you want to commit these changes?” and clicking yes just redisplays the same message. I tried 3 browsers, so I doubt it’s on my end, although there’s a small chance our firewall might be blocking a non-HTTP-based Javascript action.

      • Ben
      • May 15th, 2017 6:36am

      You might be hitting this problem.

    • eking
    • August 7th, 2017 3:43pm

    Hi Phil,

    I’m hitting a new issue here and not sure what to to next.

    Running “apt-get update” on trusty clients end with this result…

    Ign spacewalk://SPACEWALK-URL trusty InRelease
    E: Invalid message from method spacewalk: SSL exception (-1, ‘Unexpected EOF’)
    102 Status
    Message: Waiting for headers
    URI: spacewalk://SPACEWALK-URL/dists/trusty-security/InRelease

    Precise systems are running fine though. Well even Trusty was working when I left the office on Friday.

    Running or removing and running secureApt script again, does not resolve the issue.

    Not sure what file it’s complaining about: “Unexpected EOF”.

    Thank you in advance

      • eking
      • August 30th, 2017 1:03pm

      So for this particular issue i had to remove /XMLRPC from the apt spacewalk.list file

      from: spacewalk://spacewalk.hc.lan/XMLRPC channels: ….
      to: spacewalk://spacewalk.hc.lan channels:

      Seems to be working now.

    • ngst
    • March 23rd, 2018 4:53pm

    I get an error stating that the GPG signature is invalid, even though as far as I can tell everything matches up:
    # apt update
    Apt-Spacewalk: Updating sources.list
    Ign:1 spacewalk://spacewalk.server extras-ubuntu InRelease
    Reading package lists… Done
    W: GPG error: spacewalk://spacewalk.server extras-ubuntu Release:
    The following signatures were invalid: 41CDF527725B5CD68EA405AA27D22AF72385D175

    The key is imported to apt-key

    # apt-key list
    pub rsa2048 2018-03-23 [SC]
    41CD F527 725B 5CD6 8EA4 05AA 27D2 2AF7 2385 D175
    uid [ unknown] Spacewalk (For GPG signing APT repos)

    And the Release file is, in fact, signed with the correct key:

    # gpg -k
    pub 2048R/2385D175 2018-03-23
    uid Spacewalk (For GPG signing APT repos)

    # gpg ./Release.gpg
    Detached signature.
    Please enter name of data file: Release
    gpg: Signature made Fri 23 Mar 2018 10:43:50 AM EDT using RSA key ID 2385D175
    gpg: Good signature from “Spacewalk (For GPG signing APT repos) ”

    Any ideas?

      • cpg
      • May 23rd, 2018 5:19pm

      I am getting the same error about the GPG signature being invalid, but everything checks out. Were you able to resolve this? does anyone have any ideas?

        • ngst
        • June 12th, 2018 2:11pm

        Nope, never resolved it. I simply accepted that, at present, this isn’t a thing that works based on the information that is readily available.

        • michael
        • November 16th, 2018 12:50pm

        This is gpg is signing the Release file with SHA1 by default but apt-get doesn’t accept that any more. Adding –personal-digest-preferences SHA256 to the gpg –sign call in fixes it.

    • Thomas
    • August 2nd, 2018 10:50am

    maybe the key/signature is sha1 instead of sha256?

    • Joomeli
    • August 8th, 2018 6:45am


    I tried using your script to sign on Ubuntu 16 repository. I get the Release file and all seems well, expect that client tells me

    W: The repository ‘spacewalk://<> ubuntu-16 Release’ does not have a Release file.

    Also for some reason after modifying the scripts I’m getting this error but only once. I’m not sure if this is related or is it because I tried to tackle missing “Multi-Arch”-data issues at the same time.

    E: Failed to fetch spacewalk://<>/dists/ubuntu-16/repodata/binary-amd64/Packages 404 Not Found

    I think I can somehow find out what is wrong with the later one but any pointers about the Release file?

    • khoi
    • March 1st, 2019 9:32pm

    apt now requires that the Date in Release files be in UTC.

    so should be updated to reflect that when it creates the Release file.

    • khoi
    • March 1st, 2019 9:37pm

    this should work for
    DATE=`date –utc “+%a, %d %b %Y %H:%M:%S “`

    HEADER=”Origin: Ubuntu
    Label: Ubuntu
    Suite: $2
    Version: 12.04
    Codename: $1
    Date: ${DATE}UTC

  1. No trackbacks yet.