add new post 'leaving FreeBSD for Archlinux'

content/index.md

@@ -8,6 +8,7 @@ template = "index.html"
 This blog is maintained by [Gibheer](/author/Gibheer) and [Stormwind](/author/Stormwind)
 about various topics.
 
+ * [leaving FreeBSD for Archlinux](post/128.html)
  * [link summary 2016/07/08](post/127.html)
  * [poudriere in jails with zfs](post/126.html)
  * [gotchas with IPs and Jails](post/125.html)

content/post/128.md

@@ -0,0 +1,123 @@
++++
+title = "leaving FreeBSD for Archlinux"
+date = "2022-08-30T21:00:00+00:00"
+author = "Gibheer"
++++
+
+After close to 10 years running FreeBSD and trying to make it work for how I
+like to run my services, I am now finally giving up.
+
+It mostly comes down to the way I want to run jails and how there is still no
+proper tooling for FreeBSD to make it work.
+
+But before I get into more details, some history to make clear where I am coming
+from and why I want to run things how I want to do.
+
+2008 - 2012 - the OpenSolaris Epoch
+--------------------------------
+
+Back in 2008 I was introduced to OpenSolaris. It was a very stable OS with ZFS,
+zones, a virtual network stack, resource management and service management framework.
+
+The tooling in many cases was just very intuitive to use and the man pages full
+of examples on how to use them.
+
+Sadly Sun was bought by Oracle and OpenSolaris was killed near instantly.
+Forks were created in Illumos distributions, but the momentum was mostly lost.
+
+Despite that I set up a server running OpenIndiana in 2010 for a project on a
+mixed used server and was running without issues for more than 2 years.
+Many applications were split into different zones, resources were pinned to
+different containers to make sure everything had room to breath, network was
+managed via the virtual network stack and flows to ensure ssh traffic was always
+possible.
+
+2012 - 2022 - the FreeBSD era
+-----------------------------
+
+In 2012 I was a bit lost, because Illumos almost ground to a halt regarding
+development and support from other open source projects.
+I liked every part of OpenSolaris/OpenIndiana, most of all its stability in
+every load I could throw at it.
+The only other distribution that offered even part of that stability was FreeBSD.
+
+FreeBSD had jails, which I saw first around the same time I found OpenSolaris
+and it supported ZFS in a very fresh release.
+
+So I started migrating all my stuff to a new server now running FreeBSD. In some
+cases it get very impressive tools like gpart, camcontrol, mfiutil and most of
+all pf.
+
+But FreeBSD was missing a properly working virtual network stack and making jails
+work mostly consisted either of using ezjail or completely manual management via
+jail.conf.
+I fiddled around with jails to make it even a bit how I used zones before but to
+no avail.
+
+I left the services running directly on the root system for some years and tried
+again multiple times in the last 10 years.
+Some things changed, e.g. multiple jail management tools were created, but none
+was quite as close to what zonecfg/zoneadm could do. The virtual network stack
+evolved but even in 2020 I still had crashes when using epair devices.
+
+At one point I even set up a huge virtual network lab with virtual routers and
+failover, BGP, OSPF and the works. At that time I wrote [carp_failover](https://git.zero-knowledge.org/gibheer/carp_failover),
+a small tool to make carp failovers work with jails with only epair devices.
+
+But after close to 10 years of waiting, something else happend - systemd.
+
+2012 till now - systemd getting better
+--------------------------------------
+
+In [2012 Archlinux was migrated to systemd](https://archlinux.org/news/end-of-initscripts-support/).
+Since then systemd gained a lot of things. The most important for me though
+is systemd-networkd and systemd-machined.
+
+Both these components make network and container management much easier to use
+on linux. Both of these tools need config files and then do what is told them to do.
+
+The most interesting part is, that it is pretty close in how zone and network
+management worked in OpenSolaris (not from the actual config file format, but
+how the components interact).
+
+It took systemd a long time, but it is now at a point where the tooling is close
+to being on par with the OpenSolaris tools I knew.
+
+Now FreeBSD also has a very good init system in `rc`. Sadly FreeBSD didn't
+recognize systemd as a hint, that other tooling apart from the init system is also
+important.
+
+2022 - what now?
+----------------
+
+Now in 2022, after trying for 3 years to migrate my FreeBSD and porting it to a
+setup similar to my OpenIndiana setup, I am giving up.
+FreeBSD doesn't offer the tooling that I need to make this work.
+
+I can't run jails with resource limits, configured in multiple text files with
+the network architecture I want without investing massive amounts of time.
+
+What I decided to do was to go with Archlinux. For one it provides the software
+I need without patches as best they can and update in a very timely manner.
+
+With OpenZFS providing ZFS modules and systemd providing the container tooling,
+there is just nothing left on FreeBSD that could make me stay and invest more
+time. My private live just doesn't allow that anymore.
+
+Will I miss anything from FreeBSD? Yes, pf. Probably the best firewall tool that
+I have ever used.
+The same is true for gpart which, at one point, I even used to rescue one of my
+linux systems.
+Both will stay in my memory in the same happy place as zoneadm and dladm and
+many other OpenSolaris tools.
+
+Now I will start to move all my stuff onto my Archlinux server, which already
+contains a couple containers, nicely separated and secured via systemd-machined.
+
+I systemd without flaws? No, but the tooling I require and need to make my stuff
+work just works.
+
+My old FreeBSD server now has an uptime of 504 days and I am afraid to reboot it.
+My new server has an uptime of 2 days and is running Archlinux for half a year.
+
+Thank you FreeBSD for all your hard work. I will remember you.