pbuilder firewall setup

For many years I have maintained firewalls on my servers that prevent processes from being able to connect to the outside world, with a specific whitelist of exceptions. This is easy to do using the excellent ferm:

• dmain (ip ip6) {
    table filter {
      chain OUTPUT {
        policy DROP;
  
        mod state {
          state INVALID DROP;
          state (ESTABLISHED RELATED) ACCEPT;
        }
  
        outerface lo {
          ACCEPT;
        }
  
        outerface ens3 {
          # APT
          proto tcp dport www mod owner uid-owner root ACCEPT;
  
          # DNS queries
          proto (tdp udp) dport domain mod owner uid-owner bind ACCEPT;
  
          # SMTP
          proto tcp mod owner Debian-exim ACCEPT;
  
          # imagine more rules here!
        }
  
        mod limit {
          limit 1/second limit-burst 10 NFLOG nflog-group 0 nflog-prefix 'out';
          limit 10/second limit-burst 100 REJECT reject-with icmp-admin-prohibited;
        }
      }
    }
  }

And here's the result:

• $ sudo -u www-robots nc -v google.co.uk 80
  nc: connect to google.co.uk port 80 (tcp) failed: Permission denied
  nc: connect to google.co.uk port 80 (tcp) failed: No route to host
  
  $ grep -w "UID=$(id -u www-robots)" /var/log/ulog/syslogemu
  Mar 14 13:33:58 traxus out IN= OUT=ens3 MAC= SRC=2a03:b0c0:0:1010::64:e001 DST=2a00:1450:400c:c07::5e LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=441093 PROTO=TCP SPT=42848 DPT=80 SEQ=3603527347 ACK=0 WINDOW=28800 SYN URGP=0 UID=116 GID=123 MARK=0 
  Mar 14 13:33:58 traxus out IN= OUT=ens3 MAC= SRC=2a03:b0c0:0:1010::64:e001 DST=2a00:1450:400c:c07::5e LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=532265 PROTO=TCP SPT=42848 DPT=80 SEQ=3603527347 ACK=0 WINDOW=28800 SYN URGP=0 UID=116 GID=123 MARK=0 
  Mar 14 13:33:59 traxus out IN= OUT=ens3 MAC= SRC=37.139.10.94 DST=74.125.133.94 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=49354 DF PROTO=TCP SPT=48450 DPT=80 SEQ=1696519290 ACK=0 WINDOW=29200 SYN URGP=0 UID=116 GID=123 MARK=0 
  Mar 14 13:33:59 traxus out IN= OUT=ens3 MAC= SRC=37.139.10.94 DST=74.125.133.94 LEN=60 TOS=00 PREC=0x00 TTL=64 ID=49355 DF PROTO=TCP SPT=48450 DPT=80 SEQ=1696519290 ACK=0 WINDOW=29200 SYN URGP=0 UID=116 GID=123 MARK=0 

Most of the matching rules use the owner match extension to ensure that only processes running with the correct user id are able to connect out. Unfortunately, I ran into a problem when running apt 1.4 inside a pbuilder chroot.

The problem

Attempts to set up a new pbuilder chroot using Debian 9 ("stretch") failed with a strange error from apt:

• E: The repository 'http://deb.debian.org/debian stretch Release' does no longer have a Release file.

This was caused by the firewall blocking the connection:

• Mar 13 21:32:18 traxus out IN= OUT=ens3 MAC= SRC=2a03:b0c0:0:1010::64:e001 DST=2001:41c8:1000:21::21:4 LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=682342 PROTO=TCP SPT=52752 DPT=80 SEQ=3677167127 ACK=0 WINDOW=28800 SYN URGP=0 UID=100 GID=65534 MARK=0 

But why is the connection being attempted by a process with such a strange uid?

• $ getent passwd 100
  uuidd:x:100:101::/run/uuidd:/bin/false

The gid from the log message doesn't even match uuidd's group, but rather:

• $ getent group 65534
  nogroup:x:65534:

It turns out that these are only the values from /etc/passwd and /etc/group from outside the chroot set up by pbuilder. On the inside, we see something else:

• $ getent passwd 100
  _apt:x:100:65534::/nonexistent:/bin/false

It turns out that since version 1.1~exp3, apt has used an unprivileged user for tasks such as downloading packages, and so I will have to figure out how to allow this user to pass through the firewall.

The obvious solution would be to allow outgoing connections for uid 100, but this is not ideal:

• It would allow the uuidd user outside of the chroot to bypass the firewall as well

• The uid 100 inside the chroot is not necessarily fixed; policy states that uid range 1000-999 is for dynamically allocated system users and groups, assigned when a package is installed

A better solution would be to allow the firewall to recognize that that a packet originates from a process running within a pbuilder chroot, and permit it to pass.

• This doesn't mean that processes running inside the chroot have free access to the outside world, however, because pbuilder itself ensures the package build process itself runs in a separate network namespace; only the chroot preparation phase where apt runs is permitted network access.

The solution

The net_cls cgroup can be used to assign a 'class identifier' (classid) to any packet transmitted by a process within a cgroup. iptables can then be configured to match on this classid. I have chosen the arbitrary classid 10:10 for this example.

First, I needed to make sure that the net_cls hierarchy was mounted during boot. This was the easiest step because I am using systemd which mounts all available cgroup controllers by default:

• $ findmnt /sys/fs/cgroup/net_cls
  TARGET                          SOURCE FSTYPE OPTIONS
  /sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio

Next I needed some firewall rules. I added support for the cgroup match module to ferm, and updated my rules with the following:

• chain OUTPUT {
    ...
    outerface ens3 {
      ...
      # Identifies packets from a pbuilder chroot
      proto tcp mod cgroup cgroup 10:10 ACCEPT;
      ...
    }
    ...
  }

  If you're not using ferm, you can add the equivalent rule by hand: iptables -A ... -m cgroup --cgroup 1048592; annoyingly iptables only accepts the classid in base 10, so it must be calculated by using major and minor as the upper and lower halves of a 32-bit integer, and converting to decimal.

Then I modified the systemd unit for my Buildbot worker to create the /pbuilder cgroup within the net_cls hierarchy and assign the 10:10 classid to processes within it.

• $ cat /etc/systemd/system/buildworker.service
  ...
  [Service]
  PermissionsStartOnly=true
  ExecStartPre=/usr/bin/cgcreate -g net_cls:pbuilder
  ExecStartPre=/usr/bin/cgset -r net_cls.classid=0x00100010 pbuilder
  ExecStopPost=/usr/bin/cgdelete net_cls:pbuilder
  ...

  net_cls.classid is calculated by using the major part of the classid as the upper 16 bits, and the minor part as the lower 16 bits of a 32-bit integer. At least this time we don't have to convert to decimal!

The final piece of the puzzle is to ensure that the Buildbot worker actaully runs pbuilder build inside the net_cls hiarchy's /pbuilder cgroup when it kicks off a package build. This is done with the cgexec command:

• cgexec -g net_cls:pbuilder pdebuild ...

CategoryTechnote