+++
title = "poudriere in jails with zfs"
date = "2016-06-26T16:30:00+00:00"
author = "Gibheer"
draft = false
+++

There are tons of tutorials out there on how to get poudriere running in a jail.
But most of them have in common, that they either miss options or have too many
of them.
So what I try with this, is to get the most condensed version of the whole
process to get poudriere up and running, serving the generated packages.

In the following for 4 sections, we create a jail with the name *poudriere1*,
using partitions `tank/jails/poudriere1` and `tank/jails/ppoudriere1/data`. For
connectivity the jail will get the IP *192.168.1.10*.

## setting up zfs

The first step is to create both ZFS partitions on the host system with the
following commands:

```
$ zfs create -o mountpoint=/jails/poudriere1 tank/jails/poudriere1
$ zfs create -o jailed=on tank/jails/poudriere1
```

The option *jailed=on* makes the partition completely available for the jail to
manipulate, so that poudriere can create new partitions. This also makes it
unavailable for the host system to mount as the jail can change the mountpoint.

## system preparations

In addition to the ZFS partitions, we also need a separate network address to
listen on, so that we can make the packages available using nginx.

For this, we define a new network interface lo1. This is then used by the jail
to add its IP.

To make this work, place the following line in rc.conf:

```
cloned_interfaces="lo1"
```

and start the new interface with the command

```
service netif cloneup
```

Next the jail must be able to reach the internet. As I had pf already in place,
I added the following NAT rule

```
nat on em0 inet from 192.168.1.0/24 to any -> (em0)
```

which redirects all traffic from 192.168.1.0/24 to the external interface.

Reload pf to make this change working

```
$ service pf reload
```

But to make this work, the host must also be told to do packet forwarding in the
`rc.conf`

```
gateway_enable="YES"
```

After that restart the routing service using

```
$ service routing restart
```

## configuring the jail

The next step is to configure the jail, so that it can start and run poudriere.
This is done in `/etc/jail.conf`. The following section shows the settings
needed and a short explanation. More can be looked up in `main 8 jail`.

```
poudriere1 {
  # first we set the permissions for the jail
  # enforce_statfs allows the jail to get information about the mountpoint. With
  # 1 it is able to see its root mountpoint and below. This is needed to be able
  # to mount any file system.
  enforce_statfs=1;
  # This option allows the jail to mount file systems.
  allow.mount;
  # The following options enable mounting the specific file systems, needed to
  # get poudriere running.
  allow.mount.devfs;
  # nullfs is used for remounting the ports tree into the child jails.
  allow.mount.nullfs;
  # tmpfs can be disabled when poudriere is told to not use it.
  allow.mount.tmpfs;
  allow.mount.procfs;
  # This is needed to mount ZFS file systems.
  allow.mount.zfs;
  # As poudriere is using chflags, the jails needs to be allowed its usage.
  allow.chflags;
  # This option needs to be set, as poudriere grants that permission its jails.
  allow.raw_sockets;
  allow.socket_af;
  allow.sysvipc;
  # Allow this jail to run its own child childs up to the number.
  children.max=20;

  # Set a hostname visiable through jls.
  host.hostname = "$name";
  # Set the path to the jails root directory.
  path = "/jails/$name";

  # Automatically mount and unmount the dev file system, needed for ZFS and also
  # used in poudriere jails.
  mount.devfs;
  # Set the IPs to use. 192.168.1.10 is handled automatically, localhost
  # is reused from the host system. 
  ip4.addr=lo1|192.168.1.10, 127.0.0.1;
  ip6.addr=::1;

  # Boot up the jail at start using the RC system. This enables the use of rc.conf.
  exec.start += "/bin/sh /etc/rc";
  # After the jail is started, grant the ZFS partition to the jail, so that it
  # can manage the work partition itself.
  exec.poststart += "zfs jail $name rpool/jails/$name";
  # On stopping the jail, go through the RC system.
  exec.stop += "/bin/sh /etc/rc.shutdown";
  # This option makes sure, that the jail is running without any environment
  # variables set.
  exec.clean;
}
```

## installing and starting the jail

To install the jail we need to fetch a release, extract it into the jail root,
make some last adjustments and then start it up.

To fetch a release the following command can be used (adjust the version to
your need):

```
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE/base.txz -o /tmp/base.txz
```

Then extract the base package into the root using the following command

```
$ tar -xf /tmp/base.txz -C /jails/poudriere1
```

Next the timezone has to be set, the resolv.conf copied and the hostname set.

```
$ echo 'hostname="poudriere1"' > /jails/poudriere1/etc/rc.conf
$ cp /etc/localtime /jails/poudriere1/etc/localtime
$ cp /etc/resolv.conf /jails/poudriere1/etc/resolv.conf
```

Instead of using the resolv.conf of the host system it would also be possible to
use unbound on the host and use that as the DNS server in the jail.

With that done, we can now start the jail using the command:

```
$ jail -c poudriere1
```

If the command fails with the message that it can't find `/jails/poudriere1`,
check that the ZFS partition hasn't *jailed* set to *on* and that it is mounted
in the correct place.

Accessing the jail using *jexec* it is possible to check if the basic setup works.

```
$ jexec poudriere1
root@poudriere1:/ # echo 'GET index.html' | nc zero-knowledge.org 80
root@poudriere1:/ # zfs list
```

If you do not get html output, check if the interface has the IPs defined and
that you set up the routing correctly.
If the ZFS partitions are missing, check if the permissions are set in */etc/jail.conf*.

## configuring poudriere

To install poudriere, build it from ports or install it through pkg.

```
root@poudriere1/ # portsnap fetch extract
root@poudriere1/ # cd /usr/ports/ports-mgmt/poudriere
root@poudriere1/ # make install clean
```

Before starting with setting up poudriere, we have to mount the work directory
somewhere where we can actually use it:

```
root@poudriere1/ # zfs set mountpoint=/poudriere tank/jails/poudriere1/work
root@poudriere1/ # zfs mount tank/jails/poudriere1/work
```

Now we can set up poudriere the environment. Change the following settings in
`/usr/local/etc/poudriere.conf`:

```
ZPOOL=tank
# relative to the zpool
ZROOTFS=/jails/poudriere1/work
BASEFS=/poudriere
# enable when you have set mount.allow.tmpfs in the jail.conf
USE_TMPFS=yes
# size in GB to allow for the ram drive
TMPFS_LIMIT=2
# set to no when you have the linux driver enabled
NOLINUX=yes
# set to no when you do not want to keep old versions of packages around
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=10
PRESERVE_TIMESTAMP=yes
BUILD_AS_NON_ROOT=yes
```

With that done, we can build the first jail for poudriere to work with. I mostly
follow the [FreeBSD handbook](https://www.freebsd.org/doc/handbook/ports-poudriere.html)

```
root@poudriere1/ # # create a jail with the 10.2-RELEASE
root@poudriere1/ # poudriere jail -c -j 102amd64 -v 10.2-RELEASE
root@poudriere1/ # # list available jails
root@poudriere1/ # poudriere jail -l
```

If there is a problem with the jail creation, you can run the command using *-x*
to get the debug output.

```
poudriere -x jail -c -j 102amd64 -v 10.2-RELEASE
```

If it happens that you get the error `Unable to execute id(1) in jail.` a
permission is missing in `/etc/jail.conf`.
To find out which is missing, check the debug output for the jail command. All
permissions are added on the command line, so it is easier to compare the
list of permissions with what poudriere wants to grant its jails.

Next we create the ports tree for poudriere to use:

```
root@poudriere1/ # # create a new ports tree
root@poudriere1/ # poudriere ports -c -p local
root@poudriere1/ # # list the installed port trees
root@poudriere1/ # poudriere ports -l
```

The next step is to create the list of packages poudriere should build into
`/usr/local/etc/poudriere.d/base-pkglist`:

```
ports-mgmt/pkg
www/nginx
```

It is also possible to use sets, for example for build options. The next code
would be the make.conf for the base set, when placed in `/usr/local/etc/base-make.conf`:

```
OPTIONS_UNSET=DOCS EXAMPLES X11 DOCBOOK NLS CUPS
DEFAULT_VERSIONS+=ssl=openssl
DEFAULT_VERSIONS+=pgsql=9.5
```

Using these files, the ports can be configured using the command:

```
root@poudriere1/ # poudriere options -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
```

To start a bulk run, which build all packages in the list, use the bulk command

```
root@poudriere1/ # poudriere bulk -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
```

You can find the created packages in the directory `/poudriere/data/packages`.

## configuration of nginx in the jail

The configuration of nginx in the jail is done in a moment.

For that nginx has to be installed. Using the freshly built packages us the
following command (adjust the path according to your setup):

```
pkg install /poudriere/data/packages/102amd64-local-base/All/nginx-1.10.1.2.txz
```

After that you can configure nginx in the file `/usr/local/etc/nginx/nginx.conf`.

The server configuration needs adjustment and nginx must be told where the data
resides:

```
server {
  listen      192.168.1.10:80;
  server_name 192.168.1.10;

  location / {
    root      /poudriere/data/packages;
    autoindex on;
  }
}
```

This will create an automatic index of the directory content and make it
available for download. With this, it can be consumed by pkg on other systems.

If you also want to serve the logs, you can enable them with the following code

```
location /logs {
  root      /poudriere/data/logs/bulk;
  autoindex on;
}
```

## configuration of nginx outside of the jail

To forward incoming requests to the jail nginx instance, the following location
option can be used:

```
location / {
  proxy_pass http://192.168.1.10:80;
  include proxy_params;
}
```

## more information

This should help to get things up and running. If you need further information,
please see the following man pages

* [man jail](https://www.freebsd.org/cgi/man.cgi?jail)
* [man jail.conf](https://www.freebsd.org/cgi/man.cgi?jail.conf)
* [man poudriere](https://www.freebsd.org/cgi/man.cgi?poudriere)
* [man zfs](https://www.freebsd.org/cgi/man.cgi?zfs)

There is also good documentation found on

* [FreeBSD handbook ports section](https://www.freebsd.org/doc/handbook/ports.html)
* [FreeBSD handbook jails section](https://www.freebsd.org/doc/handbook/jails.html)
* [nginx](http://nginx.org/en/docs/)

There are also some tools to run jails, instead of making it raw like I did in
this entry.

* [cbsb](https://www.bsdstore.ru/en/about.html)
* [ezjail](https://erdgeist.org/arts/software/ezjail/)
* [iocage (unsupported from 10.3 onwards)](https://github.com/iocage/iocage)