+++
title = "leaving FreeBSD for Archlinux"
date = "2022-08-30T21:00:00+00:00"
author = "Gibheer"
+++

After close to 10 years running FreeBSD and trying to make it work for how I
like to run my services, I am now finally giving up.

It mostly comes down to the way I want to run jails and how there is still no
proper tooling for FreeBSD to make it work.

But before I get into more details, some history to make clear where I am coming
from and why I want to run things how I want to do.

2008 - 2012 - the OpenSolaris Epoch
--------------------------------

Back in 2008 I was introduced to OpenSolaris. It was a very stable OS with ZFS,
zones, a virtual network stack, resource management and service management framework.

The tooling in many cases was just very intuitive to use and the man pages full
of examples on how to use them.

Sadly Sun was bought by Oracle and OpenSolaris was killed near instantly.
Forks were created in Illumos distributions, but the momentum was mostly lost.

Despite that I set up a server running OpenIndiana in 2010 for a project on a
mixed used server and was running without issues for more than 2 years.
Many applications were split into different zones, resources were pinned to
different containers to make sure everything had room to breath, network was
managed via the virtual network stack and flows to ensure ssh traffic was always
possible.

2012 - 2022 - the FreeBSD era
-----------------------------

In 2012 I was a bit lost, because Illumos almost ground to a halt regarding
development and support from other open source projects.
I liked every part of OpenSolaris/OpenIndiana, most of all its stability in
every load I could throw at it.
The only other distribution that offered even part of that stability was FreeBSD.

FreeBSD had jails, which I saw first around the same time I found OpenSolaris
and it supported ZFS in a very fresh release.

So I started migrating all my stuff to a new server now running FreeBSD. In some
cases it get very impressive tools like gpart, camcontrol, mfiutil and most of
all pf.

But FreeBSD was missing a properly working virtual network stack and making jails
work mostly consisted either of using ezjail or completely manual management via
jail.conf.
I fiddled around with jails to make it even a bit how I used zones before but to
no avail.

I left the services running directly on the root system for some years and tried
again multiple times in the last 10 years.
Some things changed, e.g. multiple jail management tools were created, but none
was quite as close to what zonecfg/zoneadm could do. The virtual network stack
evolved but even in 2020 I still had crashes when using epair devices.

At one point I even set up a huge virtual network lab with virtual routers and
failover, BGP, OSPF and the works. At that time I wrote [carp_failover](https://git.zero-knowledge.org/gibheer/carp_failover),
a small tool to make carp failovers work with jails with only epair devices.

But after close to 10 years of waiting, something else happend - systemd.

2012 till now - systemd getting better
--------------------------------------

In [2012 Archlinux was migrated to systemd](https://archlinux.org/news/end-of-initscripts-support/).
Since then systemd gained a lot of things. The most important for me though
is systemd-networkd and systemd-machined.

Both these components make network and container management much easier to use
on linux. Both of these tools need config files and then do what is told them to do.

The most interesting part is, that it is pretty close in how zone and network
management worked in OpenSolaris (not from the actual config file format, but
how the components interact).

It took systemd a long time, but it is now at a point where the tooling is close
to being on par with the OpenSolaris tools I knew.

Now FreeBSD also has a very good init system in `rc`. Sadly FreeBSD didn't
recognize systemd as a hint, that other tooling apart from the init system is also
important.

2022 - what now?
----------------

Now in 2022, after trying for 3 years to migrate my FreeBSD and porting it to a
setup similar to my OpenIndiana setup, I am giving up.
FreeBSD doesn't offer the tooling that I need to make this work.

I can't run jails with resource limits, configured in multiple text files with
the network architecture I want without investing massive amounts of time.

What I decided to do was to go with Archlinux. For one it provides the software
I need without patches as best they can and update in a very timely manner.

With OpenZFS providing ZFS modules and systemd providing the container tooling,
there is just nothing left on FreeBSD that could make me stay and invest more
time. My private live just doesn't allow that anymore.

Will I miss anything from FreeBSD? Yes, pf. Probably the best firewall tool that
I have ever used.
The same is true for gpart which, at one point, I even used to rescue one of my
linux systems.
Both will stay in my memory in the same happy place as zoneadm and dladm and
many other OpenSolaris tools.

Now I will start to move all my stuff onto my Archlinux server, which already
contains a couple containers, nicely separated and secured via systemd-machined.

I systemd without flaws? No, but the tooling I require and need to make my stuff
work just works.

My old FreeBSD server now has an uptime of 504 days and I am afraid to reboot it.
My new server has an uptime of 2 days and is running Archlinux for half a year.

Thank you FreeBSD for all your hard work. I will remember you.
