Debian Package Continuous Integration
Notes on problems encountered in performing Continuous Integration builds of Debian packages, usually caused by impedance mismatches between CI and Debian build processes.
Output files in parent directory
The output from a build is placed in .., which may not be regarded by the CI server as a directory that the build process should place files in. For example, Jenkins clones to jenkins/workspace/$JOB_NAME; files written to jenkins/workspace will be left behind after the build finishes.
Background: the Debian build process expects the result of debian/rules build to be placed in ., and be removed by debian/rules clean. debian/rules binary produces the binary packages, typically by copying the build result to debian/$binarypackage and then invoking dpkg-build --build. While dpkg-build can write the assembled binary packages anywhere, by default dh_builddeb tells it to put them in .. by default. This can be overridden, but goes against the assumption made by higher level tools (dpkg-buildpackage and its wrappers).
Solutions:
Clone your repo to a subdirectory of the CI build directory, and invoke debian/rules from inside this directory.
Start your build with mkdir dummy && (GLOBIGNORE=.:..:dummy && mv * dummy)
Start your build with mkdir dummy && sudo mount --bind "$PWD" dummy && cd dummy (cute, use if you're in a VM or if your build is run under a separate mount_namespaces(7) )
References:
.git directory removed from source package
The .git directory, and other files related to version control, are not included in source packages by default. This leads to problems when the build process relies on these files (e.g., in order to run git describe) and the build occurs in an isolated environment where they are not available, such as the one provided by pbuilder.
Background: the traditional Debian maintenance workflow evolved around the practice of combining the release of a program with Debian-specific modifications, and uploading the resulting source package to the Debian archive. This workflow does not rely on the version control system used by the software's author. This is necessary because:
- The upstream repository may be very large and the full history is not of interest to a package maintainer
- Upstream may not use Git 😛
- Upstream may use a centralized version control system that you cannot use
- Upstream may use a proprietary version control system
- Upstream may not use a version control system at all!
When your CI system uses a tool such as pbuilder to build your package in an isolated environment, a source package is produced with dpkg-source --build and is then passed to pbuilder build. Since you are packaging software built directly out of your repository, rather than working from an archive from upstream's release, dpkg-source produces a 'native' source package. This is simply an archive of your repository, but with certain files removed, among them the files used by your version control system. The files to be removed are determined by a regular expression that can be viewed by running dpkg-source --help and viewing the description of the -I option. As a result, when your source package is extracted inside the build environment, the .git directory will be missing.
Solutions:
- Don't rely on version control information at build time
Tell dpkg-source to produce a 3.0 (git) format source package, which includes the .git directory, by putting 3.0 (git) in debian/source/format
Tell dpkg-source to include the .git directory when it produces a source package, by adding tar-ignore = dummy-value-please-ignore to debian/source/options
Use a different method to transport your repository into an isolated build environment (probably means not using pbuilder and family, though there is a --use-pdebuild-internal option to pdebuild which tries to run clean inside the chroot--needs investigation)
- Do not build in an isolated environment
References:
Version numbers
A package's version number is determined by the first entry in the debian/changelog file, which is inconvenient to keep updated in version control.
Background: the debian/changelog file is traditionally used to record the history of changes made to the Debian packaging of a piece of software. When the Debian packaging is being maintained alongside the software itself, the file may be redundant. As an alternative to maintaining it in version control, it is possible generate it before starting the build.
The most important part of the generated changelog will be the version number of the first entry. The format is quite flexible, but it should only ever increase. Ideally it would be generated from information in the source code being built, for instance from a file containing the version of the release, but maintaining such a file can also be inconvenient. If care is taken, it is possible to use information extracted from the version control system instead. Care must be taken when using commit ids which do not monotonically increase; if they are to be included then they should be placed after the build number so that they don't cause successive builds to be ordered randomly. Instead of the build number, the date & time of the build in big-endian format can be used.
In this example, the Python module version number is used. In this case, this version number is compatible with the requirements for Debian package versions from Policy, but if it was not then further processing would be required. In particular, if the Python module version included a hyphen -, it would have to be converted to some other character to prevent dpkg-source from building a non-native package.
%: dh $@ clean: rm -f debian/changelog; \ EDITOR=true VISUAL=true \ dch --create \ --package=foopkg \ --newversion='$(shell python3 setup.py --version)' \ 'Auto-generated changelog entry; do not commit'; \ dh clean
Solutions:
Generate debian/changelog when cloning your repository (but this becomes an extra step that you have to remember to do when cloning the repository yourself)
Generate debian/changelog from version information taken from a file in your repository in the clean command in debian/rules
Generate debian/changelog from version control information in the clean command in debian/rules (but bear in mind that the .git directory will not be included in generated source packages by default)
References: