Building RPM from Git with Koji
If you have followed my other articles about Koji, you should have a fully working setup now. However its not very handy to only build local SRPM.
Fortunately Koji can build RPMs by spec-files and Makefiles which it gets from a Git repo or other SCM. Read on to learn how you get that going.
In /etc/kojid/kojid.conf
;using any other command instead of "make sources". Example showing "fedpkg sources"
;allowed_scms=github.com:/github-username/*:no:fedpkg,sources
This allows all Github repos owned by the Github account “github-username”.
When building out of an SCM, Koji expects there will be a Makefile in the project root that has a ‘sources’ target. Koji will call ‘make sources’ on the checked out files.
This target simply needs to download all the sources for the SRPM that are not already included in the SCM repository.
If the repo is private, you also need to add a SSH public key to the Github user and the private key to /root/.ssh/id_rsa on the Koji server.
Check with ssl_add -l if its loaded correctly.
My prefered folder structure for this repo has all RPM specs in a subfolder by their name and in there version-folders. Then in there the spec and Makefile.
e.g.: jetty/8.1.9/jetty-8.1.9.spec
The Makefile is easy. Pay attention to the indentation as Makefiles NEED tabs and not spaces !!
version = $(notdir $(dir))
package = jetty
sources:
spectool -g $(package)-$(version).spec
spec files are found on the internet for nearly every application. However you have to modify some of them to fit your needs.
So finally we can build from Git now like this
koji build dist-centos6 --scratch 'git+ssh://git@github.com/github-username/reponame.git?jetty/8.1.9/#HEAD
There are also other types of SCM you can use like SVN and CVS
git://, git+http://, git+https://, git+rsync://, git+ssh://
svn://, svn+http://, svn+https://, svn+ssh://
Read on Getting RPM built by Koji into YUM-able repo
Some thoughts where this can lead to:
Using Githubs webhooks feature combined with a really simple httpd script, you can have Koji build automatically from Git after every commit. Or if a special commit message is used.
As long as I dont write an article about that, I just point you here https://github.com/philicious/koji-scripts-and-plugins/blob/master/koji-github-webhook/koji-github-tornado.py
Thank you for your another knowledge sharing article! I’ve managed to set up Koji and build a package. Now it’s time for git. By the way, have you every tried gitlab? Is it good option for maintaining spec files?
No never tried Gitlab with Koji but any ssh Git repo should work. Also other SCM types. I’ll add them to the psost
OK, thnaks. 🙂
Hi,
I have another question – don’t we need to add ssh key of kojibuilder user to github, so that it can be authorised to clone the repo?
Add public key to the Github account and private key to /root/.ssh/id_rsa on the Koji builder host. At least thats how I have it working.
Feel free to try out different ways and report 🙂
Hi,
I had to switch mockuser to root, because when build via SCM koji made dirs in /var/lib/mock as root.root instead of kojibuilder.mock (like in the case of building SRPM).
But anyways, I have another problem – now koji is able to clone the git repository but it is unable to build the sources because it lacks of sources in root/rpmbuild/SOURCES/ directory. Shouldn’t it download the sources by itself? There is address to sources in spec file.
“When building out of an SCM, Koji expects there will be a Makefile in the project root that has a ‘sources’ target. Koji will call ‘make sources’ on the checked out files.
This target simply needs to download all the sources for the SRPM that are not already included in the SCM repository.”
What do you mean in this paragraph? What should I write in Makefile if the file sources-1.2.tar.gz is not in the repository, but somewhere on ftp and this address is given in spec file?
Btw. I’m testing koji building by using this repo: http://pkgs.fedoraproject.org/cgit/BitchX.git/tree/
I’ve just cloned it to my gitlab and added Makefile provided by you in the article.
Fedora uses a sligthly different approach. They dont use standard command “make sources” but “fedpkg sources” in Koji config.
My Makefile will be called with “make sources” (=Koji default) and it will run the “sources” target. Therefore “spectool -g $(package)-$(version).spec” is run (see http://linuxmanpages.net/manpages/fedora16/man1/spectool.1.html). spectool -g will download the sources specified in spec file (e.g Source0: http://www.bitchx.ca/%{name}-%{version}.tar.gz)
However in the sources target you can do anything to get the sources. like wget, curl, ftp, execute bash scripts or fixing up archive name with “mv master master.tgz”. By any means the sources target has to take care of bringing in the sources as an archive with correct file extension.
spectool -g is a nice way because spec file is used anyways for rpm building later.
Ok, I get it. “spectool -g $(package)-$(version).spec” downloads the tarball, but still I don’t know why koji complains that there is no tarball in SOURCES directory? Should I modify the spec file? Maybe I should use the one used by CentOS/ScientificLinux or some other distribution? BTW. Do you have some good source of spec-files/Makefiles for koji? Or rather I have to prepare or those files by my own?
tried the BitchX spec with spectool -g and BitchX-1.2.1.tar.gz is downloaded. Check mock build dirs for the specfile itself and the tar.gz
For troubleshooting try running spectool -g as the kojibuilder user in that dir. Maybe some firewall/permission problem.
Spec files you find on the internet are usually fine. Only sometimes you have to modify them a little.
Makefile stays the same. Sometimes when downloading from git repos you have to do a mv master master.tgz
[kojiadmin@koji ~]$ koji build dist-linux6 –scratch ‘git+ssh://git@git.example.com/root/bitchx-spec-file.git?#HEAD’
Created task: 145
Task info: http://koji.example.com/koji/taskinfo?taskID=145
Watching tasks (this may be safely interrupted)…
145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD): free
145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD): free -> open (kojid)
146 buildSRPMFromSCM (/root/bitchx-spec-file.git:HEAD): free
146 buildSRPMFromSCM (/root/bitchx-spec-file.git:HEAD): free -> open (kojid)
146 buildSRPMFromSCM (/root/bitchx-spec-file.git:HEAD): open (kojid) -> FAILED: BuildError: error building srpm, mock exited with status 1; see build.log for more information
0 free 1 open 0 done 1 failed
145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD): open (kojid) -> FAILED: BuildError: error building srpm, mock exited with status 1; see build.log for more information
0 free 0 open 0 done 2 failed
145 build (dist-linux6, /root/bitchx-spec-file.git:HEAD) failed
[root@koji root]# cat /var/lib/mock/dist-linux6-build-58-13/result/build.log
Mock Version: 1.1.32
Mock Version: 1.1.32
Mock Version: 1.1.32
ENTER do([‘bash’, ‘–login’, ‘-c’, ‘rpmbuild -bs –target x86_64 –nodeps builddir/build/SPECS/BitchX.spec’], False, ‘/var/lib/mock/dist-linux6-build-58-13/root/’, None, 86400, True, False, 0, 135, None, False, {‘LANG’: ‘en_US.UTF-8’, ‘TERM’: ‘vt100’, ‘SHELL’: ‘/bin/bash’, ‘HOSTNAME’: ‘mock’, ‘PROMPT_COMMAND’: ‘echo -n “”‘, ‘HOME’: ‘/builddir’, ‘PATH’: ‘/usr/bin:/bin:/usr/sbin:/sbin’}, logger=)
Executing command: [‘bash’, ‘–login’, ‘-c’, ‘rpmbuild -bs –target x86_64 –nodeps builddir/build/SPECS/BitchX.spec’] with env {‘LANG’: ‘en_US.UTF-8’, ‘TERM’: ‘vt100’, ‘SHELL’: ‘/bin/bash’, ‘HOSTNAME’: ‘mock’, ‘PROMPT_COMMAND’: ‘echo -n “”‘, ‘HOME’: ‘/builddir’, ‘PATH’: ‘/usr/bin:/bin:/usr/sbin:/sbin’}
error: File /builddir/build/SOURCES/BitchX-1.2.1.tar.gz: No such file or directory
Building target platforms: x86_64
Building for target x86_64
Child return code was: 1
EXCEPTION: Command failed. See logs for output.
# [‘bash’, ‘–login’, ‘-c’, ‘rpmbuild -bs –target x86_64 –nodeps builddir/build/SPECS/BitchX.spec’]
Traceback (most recent call last):
File “/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py”, line 70, in trace
result = func(*args, **kw)
File “/usr/lib/python2.6/site-packages/mockbuild/util.py”, line 359, in do
raise mockbuild.exception.Error, (“Command failed. See logs for output.\n # %s” % (command,), child.returncode)
Error: Command failed. See logs for output.
# [‘bash’, ‘–login’, ‘-c’, ‘rpmbuild -bs –target x86_64 –nodeps builddir/build/SPECS/BitchX.spec’]
LEAVE do –> EXCEPTION RAISED
Do you know what is going on here? Why it can’t download the source tarball?
check other logs too. checking out repo and running “make sources” is done before mock runs. So the sources should be already there and you wont find the real error in the build.log but in the other logs. (reading koji logs can be a confusing and complicated thing)