Debian Package Continuous Integration

Notes on problems encountered in performing Continuous Integration builds of Debian packages, usually caused by impedance mismatches between CI and Debian build processes.

Output files in parent directory

The output from a build is placed in .., which may not be regarded by the CI server as a directory that the build process should place files in. For example, Jenkins clones to jenkins/workspace/$JOB_NAME; files written to jenkins/workspace will be left behind after the build finishes.

Background: the Debian build process expects the result of debian/rules build to be placed in ., and be removed by debian/rules clean. debian/rules binary produces the binary packages, typically by copying the build result to debian/$binarypackage and then invoking dpkg-build --build. While dpkg-build can write the assembled binary packages anywhere, by default dh_builddeb tells it to put them in .. by default. This can be overridden, but goes against the assumption made by higher level tools (dpkg-buildpackage and its wrappers).



.git directory removed from source package

The .git directory, and other files related to version control, are not included in source packages by default. This leads to problems when the build process relies on these files (e.g., in order to run git describe) and the build occurs in an isolated environment where they are not available, such as the one provided by pbuilder.

Background: the traditional Debian maintenance workflow evolved around the practice of combining the release of a program with Debian-specific modifications, and uploading the resulting source package to the Debian archive. This workflow does not rely on the version control system used by the software's author. This is necessary because:

  1. The upstream repository may be very large and the full history is not of interest to a package maintainer
  2. Upstream may not use Git 😛
  3. Upstream may use a centralized version control system that you cannot use
  4. Upstream may use a proprietary version control system
  5. Upstream may not use a version control system at all!

When your CI system uses a tool such as pbuilder to build your package in an isolated environment, a source package is produced with dpkg-source --build and is then passed to pbuilder build. Since you are packaging software built directly out of your repository, rather than working from an archive from upstream's release, dpkg-source produces a 'native' source package. This is simply an archive of your repository, but with certain files removed, among them the files used by your version control system. The files to be removed are determined by a regular expression that can be viewed by running dpkg-source --help and viewing the description of the -I option. As a result, when your source package is extracted inside the build environment, the .git directory will be missing.



Version numbers

A package's version number is determined by the first entry in the debian/changelog file, which is inconvenient to keep updated in version control.

Background: the debian/changelog file is traditionally used to record the history of changes made to the Debian packaging of a piece of software. When the Debian packaging is being maintained alongside the software itself, the file may be redundant. As an alternative to maintaining it in version control, it is possible generate it before starting the build.

The most important part of the generated changelog will be the version number of the first entry. The format is quite flexible, but it should only ever increase. Ideally it would be generated from information in the source code being built, for instance from a file containing the version of the release, but maintaining such a file can also be inconvenient. If care is taken, it is possible to use information extracted from the version control system instead. Care must be taken when using commit ids which do not monotonically increase; if they are to be included then they should be placed after the build number so that they don't cause successive builds to be ordered randomly. Instead of the build number, the date & time of the build in big-endian format can be used.

In this example, the Python module version number is used. In this case, this version number is compatible with the requirements for Debian package versions from Policy, but if it was not then further processing would be required. In particular, if the Python module version included a hyphen -, it would have to be converted to some other character to prevent dpkg-source from building a non-native package.



CategoryTechnote DebianPackageContinuousIntegration (last edited 2017-03-14 15:16:53 by sam)

© Sam Morris <>.
Content may be distributed and modified providing this notice is preserved.