add post about poudriere in a jail
This commit is contained in:
parent
5f2b36542c
commit
3cde894140
|
@ -0,0 +1,366 @@
|
|||
+++
|
||||
title = "poudriere in jails with zfs"
|
||||
date = "2016-06-26T16:30:00+00:00"
|
||||
author = "Gibheer"
|
||||
draft = false
|
||||
+++
|
||||
|
||||
There are tons of tutorials out there on how to get poudriere running in a jail.
|
||||
But most of them have in common, that they either miss options or have too many
|
||||
of them.
|
||||
So what I try with this, is to get the most condensed version of the whole
|
||||
process to get poudriere up and running, serving the generated packages.
|
||||
|
||||
In the following for 4 sections, we create a jail with the name *poudriere1*,
|
||||
using partitions `tank/jails/poudriere1` and `tank/jails/ppoudriere1/data`. For
|
||||
connectivity the jail will get the IP *192.168.1.10*.
|
||||
|
||||
## setting up zfs
|
||||
|
||||
The first step is to create both ZFS partitions on the host system with the
|
||||
following commands:
|
||||
|
||||
```
|
||||
$ zfs create -o mountpoint=/jails/poudriere1 tank/jails/poudriere1
|
||||
$ zfs create -o jailed=on tank/jails/poudriere1
|
||||
```
|
||||
|
||||
The option *jailed=on* makes the partition completely available for the jail to
|
||||
manipulate, so that poudriere can create new partitions. This also makes it
|
||||
unavailable for the host system to mount as the jail can change the mountpoint.
|
||||
|
||||
## system preparations
|
||||
|
||||
In addition to the ZFS partitions, we also need a separate network address to
|
||||
listen on, so that we can make the packages available using nginx.
|
||||
|
||||
For this, we define a new network interface lo1. This is then used by the jail
|
||||
to add its IP.
|
||||
|
||||
To make this work, place the following line in rc.conf:
|
||||
|
||||
```
|
||||
cloned_interfaces="lo1"
|
||||
```
|
||||
|
||||
and start the new interface with the command
|
||||
|
||||
```
|
||||
service netif cloneup
|
||||
```
|
||||
|
||||
Next the jail must be able to reach the internet. As I had pf already in place,
|
||||
I added the following NAT rule
|
||||
|
||||
```
|
||||
nat on em0 inet from 192.168.1.0/24 to any -> (em0)
|
||||
```
|
||||
|
||||
which redirects all traffic from 192.168.1.0/24 to the external interface.
|
||||
|
||||
Reload pf to make this change working
|
||||
|
||||
```
|
||||
$ service pf reload
|
||||
```
|
||||
|
||||
But to make this work, the host must also be told to do packet forwarding in the
|
||||
`rc.conf`
|
||||
|
||||
```
|
||||
gateway_enable="YES"
|
||||
```
|
||||
|
||||
After that restart the routing service using
|
||||
|
||||
```
|
||||
$ service routing restart
|
||||
```
|
||||
|
||||
## configuring the jail
|
||||
|
||||
The next step is to configure the jail, so that it can start and run poudriere.
|
||||
This is done in `/etc/jail.conf`. The following section shows the settings
|
||||
needed and a short explanation. More can be looked up in `main 8 jail`.
|
||||
|
||||
```
|
||||
poudriere1 {
|
||||
# first we set the permissions for the jail
|
||||
# enforce_statfs allows the jail to get information about the mountpoint. With
|
||||
# 1 it is able to see its root mountpoint and below. This is needed to be able
|
||||
# to mount any file system.
|
||||
enforce_statfs=1;
|
||||
# This option allows the jail to mount file systems.
|
||||
allow.mount;
|
||||
# The following options enable mounting the specific file systems, needed to
|
||||
# get poudriere running.
|
||||
allow.mount.devfs;
|
||||
# nullfs is used for remounting the ports tree into the child jails.
|
||||
allow.mount.nullfs;
|
||||
# tmpfs can be disabled when poudriere is told to not use it.
|
||||
allow.mount.tmpfs;
|
||||
allow.mount.procfs;
|
||||
# This is needed to mount ZFS file systems.
|
||||
allow.mount.zfs;
|
||||
# As poudriere is using chflags, the jails needs to be allowed its usage.
|
||||
allow.chflags;
|
||||
# This option needs to be set, as poudriere grants that permission its jails.
|
||||
allow.raw_sockets;
|
||||
allow.socket_af;
|
||||
allow.sysvipc;
|
||||
# Allow this jail to run its own child childs up to the number.
|
||||
children.max=20;
|
||||
|
||||
# Set a hostname visiable through jls.
|
||||
host.hostname = "$name";
|
||||
# Set the path to the jails root directory.
|
||||
path = "/jails/$name";
|
||||
|
||||
# Automatically mount and unmount the dev file system, needed for ZFS and also
|
||||
# used in poudriere jails.
|
||||
mount.devfs;
|
||||
# Set the IPs to use. 192.168.1.10 is handled automatically, localhost
|
||||
# is reused from the host system.
|
||||
ip4.addr=lo1|192.168.1.10, 127.0.0.1;
|
||||
ip6.addr=::1;
|
||||
|
||||
# Boot up the jail at start using the RC system. This enables the use of rc.conf.
|
||||
exec.start += "/bin/sh /etc/rc";
|
||||
# After the jail is started, grant the ZFS partition to the jail, so that it
|
||||
# can manage the work partition itself.
|
||||
exec.poststart += "zfs jail $name rpool/jails/$name";
|
||||
# On stopping the jail, go through the RC system.
|
||||
exec.stop += "/bin/sh /etc/rc.shutdown";
|
||||
# This option makes sure, that the jail is running without any environment
|
||||
# variables set.
|
||||
exec.clean;
|
||||
}
|
||||
```
|
||||
|
||||
## installing and starting the jail
|
||||
|
||||
To install the jail we need to fetch a release, extract it into the jail root,
|
||||
make some last adjustments and then start it up.
|
||||
|
||||
To fetch a release the following command can be used (adjust the version to
|
||||
your need):
|
||||
|
||||
```
|
||||
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE/base.txz -o /tmp/base.txz
|
||||
```
|
||||
|
||||
Then extract the base package into the root using the following command
|
||||
|
||||
```
|
||||
$ tar -xf /tmp/base.txz -C /jails/poudriere1
|
||||
```
|
||||
|
||||
Next the timezone has to be set, the resolv.conf copied and the hostname set.
|
||||
|
||||
```
|
||||
$ echo 'hostname="poudriere1"' > /jails/poudriere1/etc/rc.conf
|
||||
$ cp /etc/localtime /jails/poudriere1/etc/localtime
|
||||
$ cp /etc/resolv.conf /jails/poudriere1/etc/resolv.conf
|
||||
```
|
||||
|
||||
Instead of using the resolv.conf of the host system it would also be possible to
|
||||
use unbound on the host and use that as the DNS server in the jail.
|
||||
|
||||
With that done, we can now start the jail using the command:
|
||||
|
||||
```
|
||||
$ jail -c poudriere1
|
||||
```
|
||||
|
||||
If the command fails with the message that it can't find `/jails/poudriere1`,
|
||||
check that the ZFS partition hasn't *jailed* set to *on* and that it is mounted
|
||||
in the correct place.
|
||||
|
||||
Accessing the jail using *jexec* it is possible to check if the basic setup works.
|
||||
|
||||
```
|
||||
$ jexec poudriere1
|
||||
root@poudriere1:/ # echo 'GET index.html' | nc zero-knowledge.org 80
|
||||
root@poudriere1:/ # zfs list
|
||||
```
|
||||
|
||||
If you do not get html output, check if the interface has the IPs defined and
|
||||
that you set up the routing correctly.
|
||||
If the ZFS partitions are missing, check if the permissions are set in */etc/jail.conf*.
|
||||
|
||||
## configuring poudriere
|
||||
|
||||
To install poudriere, build it from ports or install it through pkg.
|
||||
|
||||
```
|
||||
root@poudriere1/ # portsnap fetch extract
|
||||
root@poudriere1/ # cd /usr/ports/ports-mgmt/poudriere
|
||||
root@poudriere1/ # make install clean
|
||||
```
|
||||
|
||||
Before starting with setting up poudriere, we have to mount the work directory
|
||||
somewhere where we can actually use it:
|
||||
|
||||
```
|
||||
root@poudriere1/ # zfs set mountpoint=/poudriere tank/jails/poudriere1/work
|
||||
root@poudriere1/ # zfs mount tank/jails/poudriere1/work
|
||||
```
|
||||
|
||||
Now we can set up poudriere the environment. Change the following settings in
|
||||
`/usr/local/etc/poudriere.conf`:
|
||||
|
||||
```
|
||||
ZPOOL=tank
|
||||
# relative to the zpool
|
||||
ZROOTFS=/jails/poudriere1/work
|
||||
BASEFS=/poudriere
|
||||
# enable when you have set mount.allow.tmpfs in the jail.conf
|
||||
USE_TMPFS=yes
|
||||
# size in GB to allow for the ram drive
|
||||
TMPFS_LIMIT=2
|
||||
# set to no when you have the linux driver enabled
|
||||
NOLINUX=yes
|
||||
# set to no when you do not want to keep old versions of packages around
|
||||
KEEP_OLD_PACKAGES=yes
|
||||
KEEP_OLD_PACKAGES_COUNT=10
|
||||
PRESERVE_TIMESTAMP=yes
|
||||
BUILD_AS_NON_ROOT=yes
|
||||
```
|
||||
|
||||
With that done, we can build the first jail for poudriere to work with. I mostly
|
||||
follow the [FreeBSD handbook](https://www.freebsd.org/doc/handbook/ports-poudriere.html)
|
||||
|
||||
```
|
||||
root@poudriere1/ # # create a jail with the 10.2-RELEASE
|
||||
root@poudriere1/ # poudriere jail -c -j 102amd64 -v 10.2-RELEASE
|
||||
root@poudriere1/ # # list available jails
|
||||
root@poudriere1/ # poudriere jail -l
|
||||
```
|
||||
|
||||
If there is a problem with the jail creation, you can run the command using *-x*
|
||||
to get the debug output.
|
||||
|
||||
```
|
||||
poudriere -x jail -c -j 102amd64 -v 10.2-RELEASE
|
||||
```
|
||||
|
||||
If it happens that you get the error `Unable to execute id(1) in jail.` a
|
||||
permission is missing in `/etc/jail.conf`.
|
||||
To find out which is missing, check the debug output for the jail command. All
|
||||
permissions are added on the command line, so it is easier to compare the
|
||||
list of permissions with what poudriere wants to grant its jails.
|
||||
|
||||
Next we create the ports tree for poudriere to use:
|
||||
|
||||
```
|
||||
root@poudriere1/ # # create a new ports tree
|
||||
root@poudriere1/ # poudriere ports -c -p local
|
||||
root@poudriere1/ # # list the installed port trees
|
||||
root@poudriere1/ # poudriere ports -l
|
||||
```
|
||||
|
||||
The next step is to create the list of packages poudriere should build into
|
||||
`/usr/local/etc/poudriere.d/base-pkglist`:
|
||||
|
||||
```
|
||||
ports-mgmt/pkg
|
||||
www/nginx
|
||||
```
|
||||
|
||||
It is also possible to use sets, for example for build options. The next code
|
||||
would be the make.conf for the base set, when placed in `/usr/local/etc/base-make.conf`:
|
||||
|
||||
```
|
||||
OPTIONS_UNSET=DOCS EXAMPLES X11 DOCBOOK NLS CUPS
|
||||
DEFAULT_VERSIONS+=ssl=openssl
|
||||
DEFAULT_VERSIONS+=pgsql=9.5
|
||||
```
|
||||
|
||||
Using these files, the ports can be configured using the command:
|
||||
|
||||
```
|
||||
root@poudriere1/ # poudriere options -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
|
||||
```
|
||||
|
||||
To start a bulk run, which build all packages in the list, use the bulk command
|
||||
|
||||
```
|
||||
root@poudriere1/ # poudriere bulk -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist
|
||||
```
|
||||
|
||||
You can find the created packages in the directory `/poudriere/data/packages`.
|
||||
|
||||
## configuration of nginx in the jail
|
||||
|
||||
The configuration of nginx in the jail is done in a moment.
|
||||
|
||||
For that nginx has to be installed. Using the freshly built packages us the
|
||||
following command (adjust the path according to your setup):
|
||||
|
||||
```
|
||||
pkg install /poudriere/data/packages/102amd64-local-base/All/nginx-1.10.1.2.txz
|
||||
```
|
||||
|
||||
After that you can configure nginx in the file `/usr/local/etc/nginx/nginx.conf`.
|
||||
|
||||
The server configuration needs adjustment and nginx must be told where the data
|
||||
resides:
|
||||
|
||||
```
|
||||
server {
|
||||
listen 192.168.1.10:80;
|
||||
server_name 192.168.1.10;
|
||||
|
||||
location / {
|
||||
root /poudriere/data/packages;
|
||||
autoindex on;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This will create an automatic index of the directory content and make it
|
||||
available for download. With this, it can be consumed by pkg on other systems.
|
||||
|
||||
If you also want to serve the logs, you can enable them with the following code
|
||||
|
||||
```
|
||||
location /logs {
|
||||
root /poudriere/data/logs/bulk;
|
||||
autoindex on;
|
||||
}
|
||||
```
|
||||
|
||||
## configuration of nginx outside of the jail
|
||||
|
||||
To forward incoming requests to the jail nginx instance, the following location
|
||||
option can be used:
|
||||
|
||||
```
|
||||
location / {
|
||||
proxy_pass http://192.168.1.10:80;
|
||||
include proxy_params;
|
||||
}
|
||||
```
|
||||
|
||||
## more information
|
||||
|
||||
This should help to get things up and running. If you need further information,
|
||||
please see the following man pages
|
||||
|
||||
* [man jail](https://www.freebsd.org/cgi/man.cgi?jail)
|
||||
* [man jail.conf](https://www.freebsd.org/cgi/man.cgi?jail.conf)
|
||||
* [man poudriere](https://www.freebsd.org/cgi/man.cgi?poudriere)
|
||||
* [man zfs](https://www.freebsd.org/cgi/man.cgi?zfs)
|
||||
|
||||
There is also good documentation found on
|
||||
|
||||
* [FreeBSD handbook ports section](https://www.freebsd.org/doc/handbook/ports.html)
|
||||
* [FreeBSD handbook jails section](https://www.freebsd.org/doc/handbook/jails.html)
|
||||
* [nginx](http://nginx.org/en/docs/)
|
||||
|
||||
There are also some tools to run jails, instead of making it raw like I did in
|
||||
this entry.
|
||||
|
||||
* [cbsb](https://www.bsdstore.ru/en/about.html)
|
||||
* [ezjail](https://erdgeist.org/arts/software/ezjail/)
|
||||
* [iocage (unsupported from 10.3 onwards)](https://github.com/iocage/iocage)
|
Loading…
Reference in New Issue