add post about poudriere in a jail
This commit is contained in:
parent
5f2b36542c
commit
3cde894140
|
@ -0,0 +1,366 @@
|
||||||
|
+++
|
||||||
|
title = "poudriere in jails with zfs"
|
||||||
|
date = "2016-06-26T16:30:00+00:00"
|
||||||
|
author = "Gibheer"
|
||||||
|
draft = false
|
||||||
|
+++
|
||||||
|
|
||||||
|
There are tons of tutorials out there on how to get poudriere running in a jail.
|
||||||
|
But most of them have in common, that they either miss options or have too many
|
||||||
|
of them.
|
||||||
|
So what I try with this, is to get the most condensed version of the whole
|
||||||
|
process to get poudriere up and running, serving the generated packages.
|
||||||
|
|
||||||
|
In the following for 4 sections, we create a jail with the name *poudriere1*,
|
||||||
|
using partitions `tank/jails/poudriere1` and `tank/jails/ppoudriere1/data`. For
|
||||||
|
connectivity the jail will get the IP *192.168.1.10*.
|
||||||
|
|
||||||
|
## setting up zfs
|
||||||
|
|
||||||
|
The first step is to create both ZFS partitions on the host system with the
|
||||||
|
following commands:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ zfs create -o mountpoint=/jails/poudriere1 tank/jails/poudriere1
|
||||||
|
$ zfs create -o jailed=on tank/jails/poudriere1
|
||||||
|
```
|
||||||
|
|
||||||
|
The option *jailed=on* makes the partition completely available for the jail to
|
||||||
|
manipulate, so that poudriere can create new partitions. This also makes it
|
||||||
|
unavailable for the host system to mount as the jail can change the mountpoint.
|
||||||
|
|
||||||
|
## system preparations
|
||||||
|
|
||||||
|
In addition to the ZFS partitions, we also need a separate network address to
|
||||||
|
listen on, so that we can make the packages available using nginx.
|
||||||
|
|
||||||
|
For this, we define a new network interface lo1. This is then used by the jail
|
||||||
|
to add its IP.
|
||||||
|
|
||||||
|
To make this work, place the following line in rc.conf:
|
||||||
|
|
||||||
|
```
|
||||||
|
cloned_interfaces="lo1"
|
||||||
|
```
|
||||||
|
|
||||||
|
and start the new interface with the command
|
||||||
|
|
||||||
|
```
|
||||||
|
service netif cloneup
|
||||||
|
```
|
||||||
|
|
||||||
|
Next the jail must be able to reach the internet. As I had pf already in place,
|
||||||
|
I added the following NAT rule
|
||||||
|
|
||||||
|
```
|
||||||
|
nat on em0 inet from 192.168.1.0/24 to any -> (em0)
|
||||||
|
```
|
||||||
|
|
||||||
|
which redirects all traffic from 192.168.1.0/24 to the external interface.
|
||||||
|
|
||||||
|
Reload pf to make this change working
|
||||||
|
|
||||||
|
```
|
||||||
|
$ service pf reload
|
||||||
|
```
|
||||||
|
|
||||||
|
But to make this work, the host must also be told to do packet forwarding in the
|
||||||
|
`rc.conf`
|
||||||
|
|
||||||
|
```
|
||||||
|
gateway_enable="YES"
|
||||||
|
```
|
||||||
|
|
||||||
|
After that restart the routing service using
|
||||||
|
|
||||||
|
```
|
||||||
|
$ service routing restart
|
||||||
|
```
|
||||||
|
|
||||||
|
## configuring the jail
|
||||||
|
|
||||||
|
The next step is to configure the jail, so that it can start and run poudriere.
|
||||||
|
This is done in `/etc/jail.conf`. The following section shows the settings
|
||||||
|
needed and a short explanation. More can be looked up in `main 8 jail`.
|
||||||
|
|
||||||
|
```
|
||||||
|
poudriere1 {
|
||||||
|
# first we set the permissions for the jail
|
||||||
|
# enforce_statfs allows the jail to get information about the mountpoint. With
|
||||||
|
# 1 it is able to see its root mountpoint and below. This is needed to be able
|
||||||
|
# to mount any file system.
|
||||||
|
enforce_statfs=1;
|
||||||
|
# This option allows the jail to mount file systems.
|
||||||
|
allow.mount;
|
||||||
|
# The following options enable mounting the specific file systems, needed to
|
||||||
|
# get poudriere running.
|
||||||
|
allow.mount.devfs;
|
||||||
|
# nullfs is used for remounting the ports tree into the child jails.
|
||||||
|
allow.mount.nullfs;
|
||||||
|
# tmpfs can be disabled when poudriere is told to not use it.
|
||||||
|
allow.mount.tmpfs;
|
||||||
|
allow.mount.procfs;
|
||||||
|
# This is needed to mount ZFS file systems.
|
||||||
|
allow.mount.zfs;
|
||||||
|
# As poudriere is using chflags, the jails needs to be allowed its usage.
|
||||||
|
allow.chflags;
|
||||||
|
# This option needs to be set, as poudriere grants that permission its jails.
|
||||||
|
allow.raw_sockets;
|
||||||
|
allow.socket_af;
|
||||||
|
allow.sysvipc;
|
||||||
|
# Allow this jail to run its own child childs up to the number.
|
||||||
|
children.max=20;
|
||||||
|
|
||||||
|
# Set a hostname visiable through jls.
|
||||||
|
host.hostname = "$name";
|
||||||
|
# Set the path to the jails root directory.
|
||||||
|
path = "/jails/$name";
|
||||||
|
|
||||||
|
# Automatically mount and unmount the dev file system, needed for ZFS and also
|
||||||
|
# used in poudriere jails.
|
||||||
|
mount.devfs;
|
||||||
|
# Set the IPs to use. 192.168.1.10 is handled automatically, localhost
|
||||||
|
# is reused from the host system.
|
||||||
|
ip4.addr=lo1|192.168.1.10, 127.0.0.1;
|
||||||
|
ip6.addr=::1;
|
||||||
|
|
||||||
|
# Boot up the jail at start using the RC system. This enables the use of rc.conf.
|
||||||
|
exec.start += "/bin/sh /etc/rc";
|
||||||
|
# After the jail is started, grant the ZFS partition to the jail, so that it
|
||||||
|
# can manage the work partition itself.
|
||||||
|
exec.poststart += "zfs jail $name rpool/jails/$name";
|
||||||
|
# On stopping the jail, go through the RC system.
|
||||||
|
exec.stop += "/bin/sh /etc/rc.shutdown";
|
||||||
|
# This option makes sure, that the jail is running without any environment
|
||||||
|
# variables set.
|
||||||
|
exec.clean;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## installing and starting the jail
|
||||||
|
|
||||||
|
To install the jail we need to fetch a release, extract it into the jail root,
|
||||||
|
make some last adjustments and then start it up.
|
||||||
|
|
||||||
|
To fetch a release the following command can be used (adjust the version to
|
||||||
|
your need):
|
||||||
|
|
||||||
|
```
|
||||||
|
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE/base.txz -o /tmp/base.txz
|
||||||
|
```
|
||||||
|
|
||||||
|
Then extract the base package into the root using the following command
|
||||||
|
|
||||||
|
```
|
||||||
|
$ tar -xf /tmp/base.txz -C /jails/poudriere1
|
||||||
|
```
|
||||||
|
|
||||||
|
Next the timezone has to be set, the resolv.conf copied and the hostname set.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ echo 'hostname="poudriere1"' > /jails/poudriere1/etc/rc.conf
|
||||||
|
$ cp /etc/localtime /jails/poudriere1/etc/localtime
|
||||||
|
$ cp /etc/resolv.conf /jails/poudriere1/etc/resolv.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Instead of using the resolv.conf of the host system it would also be possible to
|
||||||
|
use unbound on the host and use that as the DNS server in the jail.
|
||||||
|
|
||||||
|
With that done, we can now start the jail using the command:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ jail -c poudriere1
|
||||||
|
```
|
||||||
|
|
||||||
|
If the command fails with the message that it can't find `/jails/poudriere1`,
|
||||||
|
check that the ZFS partition hasn't *jailed* set to *on* and that it is mounted
|
||||||
|
in the correct place.
|
||||||
|
|
||||||
|
Accessing the jail using *jexec* it is possible to check if the basic setup works.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ jexec poudriere1
|
||||||
|
root@poudriere1:/ # echo 'GET index.html' | nc zero-knowledge.org 80
|
||||||
|
root@poudriere1:/ # zfs list
|
||||||
|
```
|
||||||
|
|
||||||
|
If you do not get html output, check if the interface has the IPs defined and
|
||||||
|
that you set up the routing correctly.
|
||||||
|
If the ZFS partitions are missing, check if the permissions are set in */etc/jail.conf*.
|
||||||
|
|
||||||
|
## configuring poudriere
|
||||||
|
|
||||||
|
To install poudriere, build it from ports or install it through pkg.
|
||||||
|
|
||||||
|
```
|
||||||
|
root@poudriere1/ # portsnap fetch extract
|
||||||
|
root@poudriere1/ # cd /usr/ports/ports-mgmt/poudriere
|
||||||
|
root@poudriere1/ # make install clean
|
||||||
|
```
|
||||||
|
|
||||||
|
Before starting with setting up poudriere, we have to mount the work directory
|
||||||
|
somewhere where we can actually use it:
|
||||||
|
|
||||||
|
```
|
||||||
|
root@poudriere1/ # zfs set mountpoint=/poudriere tank/jails/poudriere1/work
|
||||||
|
root@poudriere1/ # zfs mount tank/jails/poudriere1/work
|
||||||
|
```
|
||||||
|
|
||||||
|
Now we can set up poudriere the environment. Change the following settings in
|
||||||
|
`/usr/local/etc/poudriere.conf`:
|
||||||
|
|
||||||
|
```
|
||||||
|
ZPOOL=tank
|
||||||
|
# relative to the zpool
|
||||||
|
ZROOTFS=/jails/poudriere1/work
|
||||||
|
BASEFS=/poudriere
|
||||||
|
# enable when you have set mount.allow.tmpfs in the jail.conf
|
||||||
|
USE_TMPFS=yes
|
||||||
|
# size in GB to allow for the ram drive
|
||||||
|
TMPFS_LIMIT=2
|
||||||
|
# set to no when you have the linux driver enabled
|
||||||
|
NOLINUX=yes
|
||||||
|
# set to no when you do not want to keep old versions of packages around
|
||||||
|
KEEP_OLD_PACKAGES=yes
|
||||||
|
KEEP_OLD_PACKAGES_COUNT=10
|
||||||
|
PRESERVE_TIMESTAMP=yes
|
||||||
|
BUILD_AS_NON_ROOT=yes
|
||||||
|
```
|
||||||
|
|
||||||
|
With that done, we can build the first jail for poudriere to work with. I mostly
|
||||||
|
follow the [FreeBSD handbook](https://www.freebsd.org/doc/handbook/ports-poudriere.html)
|
||||||
|
|
||||||
|
```
|
||||||
|
root@poudriere1/ # # create a jail with the 10.2-RELEASE
|
||||||
|
root@poudriere1/ # poudriere jail -c -j 102amd64 -v 10.2-RELEASE
|
||||||
|
root@poudriere1/ # # list available jails
|
||||||
|
root@poudriere1/ # poudriere jail -l
|
||||||
|
```
|
||||||
|
|
||||||
|
If there is a problem with the jail creation, you can run the command using *-x*
|
||||||
|
to get the debug output.
|
||||||
|
|
||||||
|
```
|
||||||
|
poudriere -x jail -c -j 102amd64 -v 10.2-RELEASE
|
||||||
|
```
|
||||||
|
|
||||||
|
If it happens that you get the error `Unable to execute id(1) in jail.` a
|
||||||
|
permission is missing in `/etc/jail.conf`.
|
||||||
|
To find out which is missing, check the debug output for the jail command. All
|
||||||
|
permissions are added on the command line, so it is easier to compare the
|
||||||
|
list of permissions with what poudriere wants to grant its jails.
|
||||||
|
|
||||||
|
Next we create the ports tree for poudriere to use:
|
||||||
|
|
||||||
|
```
|
||||||
|
root@poudriere1/ # # create a new ports tree
|
||||||
|
root@poudriere1/ # poudriere ports -c -p local
|
||||||
|
root@poudriere1/ # # list the installed port trees
|
||||||
|
root@poudriere1/ # poudriere ports -l
|
||||||
|
```
|
||||||
|
|
||||||
|
The next step is to create the list of packages poudriere should build into
|
||||||
|
`/usr/local/etc/poudriere.d/base-pkglist`:
|
||||||
|
|
||||||
|
```
|
||||||
|
ports-mgmt/pkg
|
||||||
|
www/nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
It is also possible to use sets, for example for build options. The next code
|
||||||
|
would be the make.conf for the base set, when placed in `/usr/local/etc/base-make.conf`:
|
||||||
|
|
||||||
|
```
|
||||||
|
OPTIONS_UNSET=DOCS EXAMPLES X11 DOCBOOK NLS CUPS
|
||||||
|
DEFAULT_VERSIONS+=ssl=openssl
|
||||||
|
DEFAULT_VERSIONS+=pgsql=9.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Using these files, the ports can be configured using the command:
|
||||||
|
|
||||||
|
```
|
||||||
|
root@poudriere1/ # poudriere options -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
|
||||||
|
```
|
||||||
|
|
||||||
|
To start a bulk run, which build all packages in the list, use the bulk command
|
||||||
|
|
||||||
|
```
|
||||||
|
root@poudriere1/ # poudriere bulk -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
|
||||||
|
```
|
||||||
|
|
||||||
|
You can find the created packages in the directory `/poudriere/data/packages`.
|
||||||
|
|
||||||
|
## configuration of nginx in the jail
|
||||||
|
|
||||||
|
The configuration of nginx in the jail is done in a moment.
|
||||||
|
|
||||||
|
For that nginx has to be installed. Using the freshly built packages us the
|
||||||
|
following command (adjust the path according to your setup):
|
||||||
|
|
||||||
|
```
|
||||||
|
pkg install /poudriere/data/packages/102amd64-local-base/All/nginx-1.10.1.2.txz
|
||||||
|
```
|
||||||
|
|
||||||
|
After that you can configure nginx in the file `/usr/local/etc/nginx/nginx.conf`.
|
||||||
|
|
||||||
|
The server configuration needs adjustment and nginx must be told where the data
|
||||||
|
resides:
|
||||||
|
|
||||||
|
```
|
||||||
|
server {
|
||||||
|
listen 192.168.1.10:80;
|
||||||
|
server_name 192.168.1.10;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
root /poudriere/data/packages;
|
||||||
|
autoindex on;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This will create an automatic index of the directory content and make it
|
||||||
|
available for download. With this, it can be consumed by pkg on other systems.
|
||||||
|
|
||||||
|
If you also want to serve the logs, you can enable them with the following code
|
||||||
|
|
||||||
|
```
|
||||||
|
location /logs {
|
||||||
|
root /poudriere/data/logs/bulk;
|
||||||
|
autoindex on;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## configuration of nginx outside of the jail
|
||||||
|
|
||||||
|
To forward incoming requests to the jail nginx instance, the following location
|
||||||
|
option can be used:
|
||||||
|
|
||||||
|
```
|
||||||
|
location / {
|
||||||
|
proxy_pass http://192.168.1.10:80;
|
||||||
|
include proxy_params;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## more information
|
||||||
|
|
||||||
|
This should help to get things up and running. If you need further information,
|
||||||
|
please see the following man pages
|
||||||
|
|
||||||
|
* [man jail](https://www.freebsd.org/cgi/man.cgi?jail)
|
||||||
|
* [man jail.conf](https://www.freebsd.org/cgi/man.cgi?jail.conf)
|
||||||
|
* [man poudriere](https://www.freebsd.org/cgi/man.cgi?poudriere)
|
||||||
|
* [man zfs](https://www.freebsd.org/cgi/man.cgi?zfs)
|
||||||
|
|
||||||
|
There is also good documentation found on
|
||||||
|
|
||||||
|
* [FreeBSD handbook ports section](https://www.freebsd.org/doc/handbook/ports.html)
|
||||||
|
* [FreeBSD handbook jails section](https://www.freebsd.org/doc/handbook/jails.html)
|
||||||
|
* [nginx](http://nginx.org/en/docs/)
|
||||||
|
|
||||||
|
There are also some tools to run jails, instead of making it raw like I did in
|
||||||
|
this entry.
|
||||||
|
|
||||||
|
* [cbsb](https://www.bsdstore.ru/en/about.html)
|
||||||
|
* [ezjail](https://erdgeist.org/arts/software/ezjail/)
|
||||||
|
* [iocage (unsupported from 10.3 onwards)](https://github.com/iocage/iocage)
|
Loading…
Reference in New Issue