Building RPM from Git with Koji

If you have followed my other articles about Koji, you should have a fully working setup now. However its not very handy to only build local SRPM.
Fortunately Koji can build RPMs by spec-files and Makefiles which it gets from a Git repo or other SCM. Read on to learn how you get that going.

In /etc/kojid/kojid.conf

allowed_scms=github.com:/github-username/*:no

;using any other command instead of "make sources". Example showing "fedpkg sources"
;allowed_scms=github.com:/github-username/*:no:fedpkg,sources

This allows all Github repos owned by the Github account “github-username”.

When building out of an SCM, Koji expects there will be a Makefile in the project root that has a ‘sources’ target. Koji will call ‘make sources’ on the checked out files.
This target simply needs to download all the sources for the SRPM that are not already included in the SCM repository.

If the repo is private, you also need to add a SSH public key to the Github user and the private key to /root/.ssh/id_rsa on the Koji server.
Check with ssl_add -l if its loaded correctly.

My prefered folder structure for this repo has all RPM specs in a subfolder by their name and in there version-folders. Then in there the spec and Makefile.
e.g.: jetty/8.1.9/jetty-8.1.9.spec

The Makefile is easy. Pay attention to the indentation as Makefiles NEED tabs and not spaces !!

dir = $(shell pwd)
version = $(notdir $(dir))
package = jetty

sources:
    spectool -g $(package)-$(version).spec

spec files are found on the internet for nearly every application. However you have to modify some of them to fit your needs.

So finally we can build from Git now like this

#koji build [tag] [scheme]://[user]@[hostname]/[path/to/repository]?[path/to/project]#[revision]

koji build dist-centos6 --scratch 'git+ssh://git@github.com/github-username/reponame.git?jetty/8.1.9/#HEAD

There are also other types of SCM you can use like SVN and CVS

cvs://, cvs+ssh://
git://, git+http://, git+https://, git+rsync://, git+ssh://
svn://, svn+http://, svn+https://, svn+ssh://

Read on Getting RPM built by Koji into YUM-able repo


Some thoughts where this can lead to:

Using Githubs webhooks feature combined with a really simple httpd script, you can have Koji build automatically from Git after every commit. Or if a special commit message is used.
As long as I dont write an article about that, I just point you here https://github.com/philicious/koji-scripts-and-plugins/blob/master/koji-github-webhook/koji-github-tornado.py

    • Tomasz K.
    • February 13th, 2014 11:11am

    Thank you for your another knowledge sharing article! I’ve managed to set up Koji and build a package. Now it’s time for git. By the way, have you every tried gitlab? Is it good option for maintaining spec files?

      • phil
      • February 13th, 2014 1:26pm

      No never tried Gitlab with Koji but any ssh Git repo should work. Also other SCM types. I’ll add them to the psost

        • Tomasz K.
        • February 13th, 2014 2:07pm

        OK, thnaks. :-)

    • Tomasz K.
    • February 14th, 2014 3:18pm

    Hi,
    I have another question – don’t we need to add ssh key of kojibuilder user to github, so that it can be authorised to clone the repo?

      • phil
      • February 14th, 2014 3:27pm

      Add public key to the Github account and private key to /root/.ssh/id_rsa on the Koji builder host. At least thats how I have it working.
      Feel free to try out different ways and report :)

    • Tomasz K.
    • February 17th, 2014 2:09pm

    Hi,

    I had to switch mockuser to root, because when build via SCM koji made dirs in /var/lib/mock as root.root instead of kojibuilder.mock (like in the case of building SRPM).

    But anyways, I have another problem – now koji is able to clone the git repository but it is unable to build the sources because it lacks of sources in root/rpmbuild/SOURCES/ directory. Shouldn’t it download the sources by itself? There is address to sources in spec file.

    “When building out of an SCM, Koji expects there will be a Makefile in the project root that has a ‘sources’ target. Koji will call ‘make sources’ on the checked out files.
    This target simply needs to download all the sources for the SRPM that are not already included in the SCM repository.”

    What do you mean in this paragraph? What should I write in Makefile if the file sources-1.2.tar.gz is not in the repository, but somewhere on ftp and this address is given in spec file?

    Btw. I’m testing koji building by using this repo: http://pkgs.fedoraproject.org/cgit/BitchX.git/tree/
    I’ve just cloned it to my gitlab and added Makefile provided by you in the article.

      • phil
      • February 17th, 2014 2:30pm

      Fedora uses a sligthly different approach. They dont use standard command “make sources” but “fedpkg sources” in Koji config.
      My Makefile will be called with “make sources” (=Koji default) and it will run the “sources” target. Therefore “spectool -g $(package)-$(version).spec” is run (see http://linuxmanpages.net/manpages/fedora16/man1/spectool.1.html). spectool -g will download the sources specified in spec file (e.g Source0: http://www.bitchx.ca/%{name}-%{version}.tar.gz)

      However in the sources target you can do anything to get the sources. like wget, curl, ftp, execute bash scripts or fixing up archive name with “mv master master.tgz”. By any means the sources target has to take care of bringing in the sources as an archive with correct file extension.
      spectool -g is a nice way because spec file is used anyways for rpm building later.

    • Tomasz K.
    • February 17th, 2014 2:33pm

    Ok, I get it. “spectool -g $(package)-$(version).spec” downloads the tarball, but still I don’t know why koji complains that there is no tarball in SOURCES directory? Should I modify the spec file? Maybe I should use the one used by CentOS/ScientificLinux or some other distribution? BTW. Do you have some good source of spec-files/Makefiles for koji? Or rather I have to prepare or those files by my own?

      • phil
      • February 17th, 2014 2:42pm

      tried the BitchX spec with spectool -g and BitchX-1.2.1.tar.gz is downloaded. Check mock build dirs for the specfile itself and the tar.gz
      For troubleshooting try running spectool -g as the kojibuilder user in that dir. Maybe some firewall/permission problem.
      Spec files you find on the internet are usually fine. Only sometimes you have to modify them a little.
      Makefile stays the same. Sometimes when downloading from git repos you have to do a mv master master.tgz

    • Tomasz K.
    • February 17th, 2014 2:53pm

    [kojiadmin@koji ~]$ koji build dist-linux6 –scratch ‘git+ssh://git@git.example.com/root/bitchx-spec-file.git?#HEAD’
    Created task: 145
    Task info: http://koji.example.com/koji/taskinfo?taskID=145
    Watching tasks (this may be safely interrupted)…
    145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD): free
    145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD): free -> open (kojid)
    146 buildSRPMFromSCM (/root/bitchx-spec-file.git:HEAD): free
    146 buildSRPMFromSCM (/root/bitchx-spec-file.git:HEAD): free -> open (kojid)
    146 buildSRPMFromSCM (/root/bitchx-spec-file.git:HEAD): open (kojid) -> FAILED: BuildError: error building srpm, mock exited with status 1; see build.log for more information
    0 free 1 open 0 done 1 failed
    145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD): open (kojid) -> FAILED: BuildError: error building srpm, mock exited with status 1; see build.log for more information
    0 free 0 open 0 done 2 failed

    145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD) failed

    [root@koji root]# cat /var/lib/mock/dist-linux6-build-58-13/result/build.log
    Mock Version: 1.1.32
    Mock Version: 1.1.32
    Mock Version: 1.1.32
    ENTER do(['bash', '--login', '-c', 'rpmbuild -bs --target x86_64 --nodeps builddir/build/SPECS/BitchX.spec'], False, ‘/var/lib/mock/dist-linux6-build-58-13/root/’, None, 86400, True, False, 0, 135, None, False, {‘LANG’: ‘en_US.UTF-8′, ‘TERM’: ‘vt100′, ‘SHELL’: ‘/bin/bash’, ‘HOSTNAME’: ‘mock’, ‘PROMPT_COMMAND’: ‘echo -n “”‘, ‘HOME’: ‘/builddir’, ‘PATH’: ‘/usr/bin:/bin:/usr/sbin:/sbin’}, logger=)
    Executing command: ['bash', '--login', '-c', 'rpmbuild -bs --target x86_64 --nodeps builddir/build/SPECS/BitchX.spec'] with env {‘LANG’: ‘en_US.UTF-8′, ‘TERM’: ‘vt100′, ‘SHELL’: ‘/bin/bash’, ‘HOSTNAME’: ‘mock’, ‘PROMPT_COMMAND’: ‘echo -n “”‘, ‘HOME’: ‘/builddir’, ‘PATH’: ‘/usr/bin:/bin:/usr/sbin:/sbin’}
    error: File /builddir/build/SOURCES/BitchX-1.2.1.tar.gz: No such file or directory
    Building target platforms: x86_64
    Building for target x86_64
    Child return code was: 1
    EXCEPTION: Command failed. See logs for output.
    # ['bash', '--login', '-c', 'rpmbuild -bs --target x86_64 --nodeps builddir/build/SPECS/BitchX.spec']
    Traceback (most recent call last):
    File “/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py”, line 70, in trace
    result = func(*args, **kw)
    File “/usr/lib/python2.6/site-packages/mockbuild/util.py”, line 359, in do
    raise mockbuild.exception.Error, (“Command failed. See logs for output.\n # %s” % (command,), child.returncode)
    Error: Command failed. See logs for output.
    # ['bash', '--login', '-c', 'rpmbuild -bs --target x86_64 --nodeps builddir/build/SPECS/BitchX.spec']
    LEAVE do –> EXCEPTION RAISED

    Do you know what is going on here? Why it can’t download the source tarball?

      • phil
      • February 17th, 2014 2:58pm

      check other logs too. checking out repo and running “make sources” is done before mock runs. So the sources should be already there and you wont find the real error in the build.log but in the other logs. (reading koji logs can be a confusing and complicated thing)

  1. No trackbacks yet.