From 3cde89414025eee34208392d4fee88a81acca159 Mon Sep 17 00:00:00 2001 From: Gibheer Date: Fri, 8 Jul 2016 21:41:33 +0200 Subject: [PATCH] add post about poudriere in a jail --- content/post/126.md | 366 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 366 insertions(+) create mode 100644 content/post/126.md diff --git a/content/post/126.md b/content/post/126.md new file mode 100644 index 0000000..ddda154 --- /dev/null +++ b/content/post/126.md @@ -0,0 +1,366 @@ ++++ +title = "poudriere in jails with zfs" +date = "2016-06-26T16:30:00+00:00" +author = "Gibheer" +draft = false ++++ + +There are tons of tutorials out there on how to get poudriere running in a jail. +But most of them have in common, that they either miss options or have too many +of them. +So what I try with this, is to get the most condensed version of the whole +process to get poudriere up and running, serving the generated packages. + +In the following for 4 sections, we create a jail with the name *poudriere1*, +using partitions `tank/jails/poudriere1` and `tank/jails/ppoudriere1/data`. For +connectivity the jail will get the IP *192.168.1.10*. + +## setting up zfs + +The first step is to create both ZFS partitions on the host system with the +following commands: + +``` +$ zfs create -o mountpoint=/jails/poudriere1 tank/jails/poudriere1 +$ zfs create -o jailed=on tank/jails/poudriere1 +``` + +The option *jailed=on* makes the partition completely available for the jail to +manipulate, so that poudriere can create new partitions. This also makes it +unavailable for the host system to mount as the jail can change the mountpoint. + +## system preparations + +In addition to the ZFS partitions, we also need a separate network address to +listen on, so that we can make the packages available using nginx. + +For this, we define a new network interface lo1. This is then used by the jail +to add its IP. + +To make this work, place the following line in rc.conf: + +``` +cloned_interfaces="lo1" +``` + +and start the new interface with the command + +``` +service netif cloneup +``` + +Next the jail must be able to reach the internet. As I had pf already in place, +I added the following NAT rule + +``` +nat on em0 inet from 192.168.1.0/24 to any -> (em0) +``` + +which redirects all traffic from 192.168.1.0/24 to the external interface. + +Reload pf to make this change working + +``` +$ service pf reload +``` + +But to make this work, the host must also be told to do packet forwarding in the +`rc.conf` + +``` +gateway_enable="YES" +``` + +After that restart the routing service using + +``` +$ service routing restart +``` + +## configuring the jail + +The next step is to configure the jail, so that it can start and run poudriere. +This is done in `/etc/jail.conf`. The following section shows the settings +needed and a short explanation. More can be looked up in `main 8 jail`. + +``` +poudriere1 { + # first we set the permissions for the jail + # enforce_statfs allows the jail to get information about the mountpoint. With + # 1 it is able to see its root mountpoint and below. This is needed to be able + # to mount any file system. + enforce_statfs=1; + # This option allows the jail to mount file systems. + allow.mount; + # The following options enable mounting the specific file systems, needed to + # get poudriere running. + allow.mount.devfs; + # nullfs is used for remounting the ports tree into the child jails. + allow.mount.nullfs; + # tmpfs can be disabled when poudriere is told to not use it. + allow.mount.tmpfs; + allow.mount.procfs; + # This is needed to mount ZFS file systems. + allow.mount.zfs; + # As poudriere is using chflags, the jails needs to be allowed its usage. + allow.chflags; + # This option needs to be set, as poudriere grants that permission its jails. + allow.raw_sockets; + allow.socket_af; + allow.sysvipc; + # Allow this jail to run its own child childs up to the number. + children.max=20; + + # Set a hostname visiable through jls. + host.hostname = "$name"; + # Set the path to the jails root directory. + path = "/jails/$name"; + + # Automatically mount and unmount the dev file system, needed for ZFS and also + # used in poudriere jails. + mount.devfs; + # Set the IPs to use. 192.168.1.10 is handled automatically, localhost + # is reused from the host system. + ip4.addr=lo1|192.168.1.10, 127.0.0.1; + ip6.addr=::1; + + # Boot up the jail at start using the RC system. This enables the use of rc.conf. + exec.start += "/bin/sh /etc/rc"; + # After the jail is started, grant the ZFS partition to the jail, so that it + # can manage the work partition itself. + exec.poststart += "zfs jail $name rpool/jails/$name"; + # On stopping the jail, go through the RC system. + exec.stop += "/bin/sh /etc/rc.shutdown"; + # This option makes sure, that the jail is running without any environment + # variables set. + exec.clean; +} +``` + +## installing and starting the jail + +To install the jail we need to fetch a release, extract it into the jail root, +make some last adjustments and then start it up. + +To fetch a release the following command can be used (adjust the version to +your need): + +``` +$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE/base.txz -o /tmp/base.txz +``` + +Then extract the base package into the root using the following command + +``` +$ tar -xf /tmp/base.txz -C /jails/poudriere1 +``` + +Next the timezone has to be set, the resolv.conf copied and the hostname set. + +``` +$ echo 'hostname="poudriere1"' > /jails/poudriere1/etc/rc.conf +$ cp /etc/localtime /jails/poudriere1/etc/localtime +$ cp /etc/resolv.conf /jails/poudriere1/etc/resolv.conf +``` + +Instead of using the resolv.conf of the host system it would also be possible to +use unbound on the host and use that as the DNS server in the jail. + +With that done, we can now start the jail using the command: + +``` +$ jail -c poudriere1 +``` + +If the command fails with the message that it can't find `/jails/poudriere1`, +check that the ZFS partition hasn't *jailed* set to *on* and that it is mounted +in the correct place. + +Accessing the jail using *jexec* it is possible to check if the basic setup works. + +``` +$ jexec poudriere1 +root@poudriere1:/ # echo 'GET index.html' | nc zero-knowledge.org 80 +root@poudriere1:/ # zfs list +``` + +If you do not get html output, check if the interface has the IPs defined and +that you set up the routing correctly. +If the ZFS partitions are missing, check if the permissions are set in */etc/jail.conf*. + +## configuring poudriere + +To install poudriere, build it from ports or install it through pkg. + +``` +root@poudriere1/ # portsnap fetch extract +root@poudriere1/ # cd /usr/ports/ports-mgmt/poudriere +root@poudriere1/ # make install clean +``` + +Before starting with setting up poudriere, we have to mount the work directory +somewhere where we can actually use it: + +``` +root@poudriere1/ # zfs set mountpoint=/poudriere tank/jails/poudriere1/work +root@poudriere1/ # zfs mount tank/jails/poudriere1/work +``` + +Now we can set up poudriere the environment. Change the following settings in +`/usr/local/etc/poudriere.conf`: + +``` +ZPOOL=tank +# relative to the zpool +ZROOTFS=/jails/poudriere1/work +BASEFS=/poudriere +# enable when you have set mount.allow.tmpfs in the jail.conf +USE_TMPFS=yes +# size in GB to allow for the ram drive +TMPFS_LIMIT=2 +# set to no when you have the linux driver enabled +NOLINUX=yes +# set to no when you do not want to keep old versions of packages around +KEEP_OLD_PACKAGES=yes +KEEP_OLD_PACKAGES_COUNT=10 +PRESERVE_TIMESTAMP=yes +BUILD_AS_NON_ROOT=yes +``` + +With that done, we can build the first jail for poudriere to work with. I mostly +follow the [FreeBSD handbook](https://www.freebsd.org/doc/handbook/ports-poudriere.html) + +``` +root@poudriere1/ # # create a jail with the 10.2-RELEASE +root@poudriere1/ # poudriere jail -c -j 102amd64 -v 10.2-RELEASE +root@poudriere1/ # # list available jails +root@poudriere1/ # poudriere jail -l +``` + +If there is a problem with the jail creation, you can run the command using *-x* +to get the debug output. + +``` +poudriere -x jail -c -j 102amd64 -v 10.2-RELEASE +``` + +If it happens that you get the error `Unable to execute id(1) in jail.` a +permission is missing in `/etc/jail.conf`. +To find out which is missing, check the debug output for the jail command. All +permissions are added on the command line, so it is easier to compare the +list of permissions with what poudriere wants to grant its jails. + +Next we create the ports tree for poudriere to use: + +``` +root@poudriere1/ # # create a new ports tree +root@poudriere1/ # poudriere ports -c -p local +root@poudriere1/ # # list the installed port trees +root@poudriere1/ # poudriere ports -l +``` + +The next step is to create the list of packages poudriere should build into +`/usr/local/etc/poudriere.d/base-pkglist`: + +``` +ports-mgmt/pkg +www/nginx +``` + +It is also possible to use sets, for example for build options. The next code +would be the make.conf for the base set, when placed in `/usr/local/etc/base-make.conf`: + +``` +OPTIONS_UNSET=DOCS EXAMPLES X11 DOCBOOK NLS CUPS +DEFAULT_VERSIONS+=ssl=openssl +DEFAULT_VERSIONS+=pgsql=9.5 +``` + +Using these files, the ports can be configured using the command: + +``` +root@poudriere1/ # poudriere options -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist +``` + +To start a bulk run, which build all packages in the list, use the bulk command + +``` +root@poudriere1/ # poudriere bulk -j 102amd64 -p local -z base -f /usr/local/etc/poudriere.d/base-pkglist +``` + +You can find the created packages in the directory `/poudriere/data/packages`. + +## configuration of nginx in the jail + +The configuration of nginx in the jail is done in a moment. + +For that nginx has to be installed. Using the freshly built packages us the +following command (adjust the path according to your setup): + +``` +pkg install /poudriere/data/packages/102amd64-local-base/All/nginx-1.10.1.2.txz +``` + +After that you can configure nginx in the file `/usr/local/etc/nginx/nginx.conf`. + +The server configuration needs adjustment and nginx must be told where the data +resides: + +``` +server { + listen 192.168.1.10:80; + server_name 192.168.1.10; + + location / { + root /poudriere/data/packages; + autoindex on; + } +} +``` + +This will create an automatic index of the directory content and make it +available for download. With this, it can be consumed by pkg on other systems. + +If you also want to serve the logs, you can enable them with the following code + +``` +location /logs { + root /poudriere/data/logs/bulk; + autoindex on; +} +``` + +## configuration of nginx outside of the jail + +To forward incoming requests to the jail nginx instance, the following location +option can be used: + +``` +location / { + proxy_pass http://192.168.1.10:80; + include proxy_params; +} +``` + +## more information + +This should help to get things up and running. If you need further information, +please see the following man pages + +* [man jail](https://www.freebsd.org/cgi/man.cgi?jail) +* [man jail.conf](https://www.freebsd.org/cgi/man.cgi?jail.conf) +* [man poudriere](https://www.freebsd.org/cgi/man.cgi?poudriere) +* [man zfs](https://www.freebsd.org/cgi/man.cgi?zfs) + +There is also good documentation found on + +* [FreeBSD handbook ports section](https://www.freebsd.org/doc/handbook/ports.html) +* [FreeBSD handbook jails section](https://www.freebsd.org/doc/handbook/jails.html) +* [nginx](http://nginx.org/en/docs/) + +There are also some tools to run jails, instead of making it raw like I did in +this entry. + +* [cbsb](https://www.bsdstore.ru/en/about.html) +* [ezjail](https://erdgeist.org/arts/software/ezjail/) +* [iocage (unsupported from 10.3 onwards)](https://github.com/iocage/iocage)