GPG signing RPMs with Sigul Signing Server & Koji integration

When you are building your own RPMs and distributing them either on your own infrastructure or to the public, you should consider signing them with a GPG key. That way the client machines that install your RPMs can verify the integrity and authenticity of what they are installing.

GPG signing can either be done manually, which is fairly easy but unhandy or you can use a way more automated and solid way, using the Sigul Signing Server by Fedora.

Sigul keeps the private keys used for signing on its server and they arent accesible by the clients who want to sign RPMs. All requests by Sigul Clients to Sigul Server are sent over the Sigul Bridge which relays them. This allows signing RPMs from various machines, without having access to actual keys being used. So you never communicate directly with the Server which can and should be isolated from the rest of the world and only allow connections from/to the Bridge in the Firwall.

Lets start with installing the necessary packages. You can run all 3 components on the same server or on different machines. The setup remains the same.

#we need a modified version for gnupg1 as gnupg2 doesnt work with sigul ->
yum install pexpect python-fedora python-nss python-sqlalchemy
rpm -Uvh sigul-0.100-0.1.el6.noarch.rpm
rpm -Uvh gnupg1-1.4.11-3.el6.x86_64.rpm

Make sure that hostnames for Sigul Bridge and Server are resolvable before proceeding ! Either by DNS or /etc/hosts. Also use two distinct hostnames even if its the same machine !

Bridge Setup

We are going to need an NSS database to hold the certificates

su sigul
#create new NSS database
certutil -d /var/lib/sigul -N
#create Sigul CA  which will be used on all Sigul components. Change CN to whatever you like
certutil -d /var/lib/sigul -S -n my-sigul-ca -s 'CN=My Sigul' -t CT,, -x -v 120
# create Bridge cert. CN has to be the hostname that your bridge will be using and needs to be resolvable !
certutil -d /var/lib/sigul -S -n sigul-bridge-cert -s '' -c my-sigul-ca -t u,, -v 120

Now for some config, su back to root and open up /etc/sigul/bridge.conf

# Path to a directory containing a NSS database
; nss-dir: @localstatedir@/lib/sigul
# Password for accessing the NSS database.  If not specified, the bridge will
# ask on startup
nss-password: nss-password-you-entered-earlier

Just in case permissions got mangled

chown sigul: /var/lib/sigul/*.db

Now start the Bridge

service sigul_bridge start
chkconfig sigul_bridge on

If the bridge doesnt startup, run it with sigul_bridge -v -v which runs it in debug mode.

Finally export the generated CA and copy it to the Sigul Server. Then delete it from the bridge !

pk12util -d /var/lib/sigul/ -o myca.p12 -n my-sigul-ca


Server Setup

Ok now that we have the Bridge up and running, we need to setup the Server which does the actual signing for us.

certutil -d /var/lib/sigul -N
# import the CA from bridge, rm .p12 file afterwards
pk12util -d /var/lib/sigul -i myca.p12
# set trust on CA
certutil -d /var/lib/sigul/ -M -n my-sigul-ca -t CT

Now we create a cert for the server. You still need this even if its the same host as the bridge.

# CN is your server hostname. Make sure its resolvable.
certutil -d /var/lib/sigul/ -S -n sigul-server-cert -s '' -c laterpay-sigul-ca -t u,, -v 120

Now some config we need to set in /etc/sigul/server.conf. There are more options you may have to set if you used different cert names or file paths.

# Host name of the publically acessible bridge to clients

# Path to a directory containing a NSS database
; nss-dir: @localstatedir@/lib/sigul
# Password for accessing the NSS database.  If not specified, the server will
# ask on startup
nss-password: nss-password-you-entered-earlier

And some initial configuration as root

#create server DB that holds users and keys
# add admin, name = siguladmin
# startup
service sigul_server start
chkconfig sigul_server on

If the service doesnt start, run is as sigul_server -v -v which will give you debug infos.

 Client Setup

You can have several Clients running. They will all communicate to the Bridge which relays to the Server.

certutil -d /var/lib/sigul -N
# import the CA from bridge, rm .p12 file afterwards
pk12util -d /var/lib/sigul -i myca.p12
# set trust on CA
certutil -d /var/lib/sigul/ -M -n my-sigul-ca -t CT

Now we create a cert for the client. You still need this even if its the same host as the bridge.

# CN is your sigul username
certutil -d /var/lib/sigul/ -S -n sigul-client-cert -s 'CN=admin' -c my-sigul-ca -t u,, -v 120

Edit /etc/sigul/client.conf

# Host name of the publically acessible bridge to the server
# Port on which the bridge expects client connections
; bridge-port: 44334
# Nickname of the client's certificate in the NSS database specified below
; client-cert-nickname: sigul-client-cert
# Host name of the private server, used for verifying its certificate
# Name of the invoking user, defaults to login name on the local machine
user-name: siguladmin

# Path to a directory containing a NSS database
nss-dir: /var/lib/sigul
# Password for accessing the NSS database.  If not specified, the client will
# ask on startup
; nss-password is not specified by default, but sigul_setup_client stores
; it in ~/.sigul/client.conf
nss-password: nss-password-you-entered-earlier

Now we need to import the private part of a GPG key which will be used for signing. If you dont have one, you can easily create one. Read here

sigul import-key my-gpg-key-name /root/RPM-GPG-KEY-private

# verify its there
sigul list-keys

At this point signing RPMs with Sigul should be working. Verify it with a local RPM before we move on integrating it with Koji

# this will give you signed.rpm file
sigul -v -v sign-rpm -o signed.rpm my-gpg-key-name nginx-1.2.7-1.el6.ngx.x86_64.rpm

# verifying signed packages
rpm --import /root/RPM-GPG-KEY-public
rpm -Kvv signed.rpm


Koji integration

Now we are going to integrate Sigul with Koji in a very automated way.

(We are working on the Bridge now, besides the client.conf thats living on the client)

edit /etc/sigul/bridge.conf and client.conf and set

# Config file used to connect to the Koji hub. Should be the one used by Kojiweb
koji-config: ~/.koji/config

You can put the Koji config and certs anywhere where Sigul can read them. We just need to put all necessary files there. I simply chose ~/.koji/ = /var/lib/sigul/.koji

mkdir /var/lib/sigul/.koji
cp /etc/koji.conf /var/lib/sigul/.koji/config
cp /etc/pki/koji/kojiweb.pem /var/lib/sigul/.koji/kojiweb.crt
cp /home/kojiadmin/.koji/serverca.crt /var/lib/sigul/.koji/
cp /home/kojiadmin/.koji/clientca.crt /var/lib/sigul/.koji/
chown -R sigul: /var/lib/sigul/.koji/

In /var/lib/sigul/.koji/config set

cert = ~/.koji/kojiweb.crt

Finally we can test the signing of an RPM existing in Koji

sigul sign-rpm -o signed.rpm --v3-signature my-gpg-key-name rubygem-rubyforge-1.0.5-1.fc11.noarch.rpm

Once you run the above one time successfully, the siguladmin user is created in Koji, now add the “sign” permission to it so you can upload the signatures again:

su kojiadmin
koji grant-permission admin siguladmin

Now, for signing and automatically storing the signed RPM back to Koji

sigul sign-rpm --koji-only --store-in-koji --v3-signature my-gpg-key-name rubygem-rubyforge-1.0.5-1.fc11.noarch.rpm
# alternative write locally and manually import
sigul sign-rpm -o rubygem-signed.rpm --v3-signature my-gpg-key-name rubygem-rubyforge-1.0.5-1.fc11.noarch.rpm
su kojiadmin
koji import-sig ./rubgygem-signed.rpm
koji write-signed-rpm --all GPG-KEY-ID # key id needs to be lowercased ! -> Koji bug


Automating it

Now this all nice but still annoying because you have to issue a command for signing and importing to Koji manually after every RPM build. Lets automate this because we are lazy and dont wanna repeat ourselves. I wrote a small Koji plugin for that purpose.

# On Koji-Hub host: enabling plugins in Koji first if you havent done so already
yum install koji-hub-plugins

edit  /etc/koji-hub/hub.conf and enable plugin and pluginpath

## Koji hub plugins
## The path where plugins are found
PluginPath = /usr/lib/koji-hub-plugins
## A space-separated list of plugins to load
Plugins = sigul_sign

Put to /usr/lib/koji-hub-plugins and edit its config near the top . It logs to /var/log/httpd/error_log
Also it tags the RPM for the testing repo. I you only have one repo, set “testing_tag” to that one instead.

The plugin runs as user kojid therefore needs “sign” permission

su kojiadmin
koji grant-permission admin kojid

Add apache to group sigul and

# otherwise the plugin cant run the sigul command
chmod -R g=rw /var/lib/sigul
chmod g=rwx /var/lib/sigul

Restart apache and after every RPM build in Koji, it should be automatically signed and imported back to Koji.

Only thing left is to edit mash config to also process signed RPMs

    • Andrew Miskell
    • March 2nd, 2016 5:29pm

    I tried to sign with sign for koji and I set everything up as described in the article but when I run:

    sigul sign-rpm –koji-only –store-in-koji –v3-signature my-gpg-key-name foo.rpm

    I get an EOF error and when I look in the logs for sigul-bridge, I see the following error.

    2016-03-02 17:26:19,345 WARNING: Error working with request data: Koji connection failed: CN=kojiweb,OU=xxx,O=xxx,ST=California,C=US is not authorized to login other users

    Any idea what’s causing this?

      • phil
      • March 31st, 2016 3:37pm

      Sorry for the late response but I only saw your comment now. Koji is very picky when it comes to the certificate setup. So I assume you have a problem there. I’d review and double-check all certs and users.

    • Corin
    • April 6th, 2016 4:32pm


    I’ve setup koji and sigul on the same machine and koji works fine (80 odd package builds already) and sigul can sign any rpm I give it. Sigul also has access to koji but whenever I try to sign an rpm with –koji-only and –store-in-koji it signs the rpm and then gets an EOF and in the bridge logs it shows Required field rpm-release missing. The Release field in the rpm info shows 4.el6 for this particular one I keep testing with for example.

    Any ideas?

      • phil
      • April 6th, 2016 4:40pm

      Hey, I’m afraid but I dont have/use Koji anymore and your error doesnt ring a bell. However these articles should work. What I would suggest is trying to sign and then upload manually. Also use high verbosity where possible.

  1. No trackbacks yet.