About the lab

last updated

Here there be servers. This is a homelab I built by hand, predominantly “pre-AI” (funny what a watershed that’s become). It’s also a direct result of blood, sweat, and tears (all three literal and a story for another time). It consists of one main server running Proxmox, a backup box running Proxmox Backup Server, an office PC running OPNsense, and a segmented home network.

For the past two-ish years my setup was rock solid as long as I didn’t touch it. I set up every container and service by hand, and it ran with relative reliability until it came time for a new piece of the puzzle. Cracks started showing and I found myself feeling like each new service required an increasing amount of effort to get working with everything else.

To learn new tools as well as make my homelab more resilient and flexible, I’ve made it my mission to refactor things one piece at a time. This involves things like: moving manual installs to Ansible as the opportunities present themselves, being more intentional and consistent with my documentation (as much as one can be), and closing the many, many gaps I find along the way. This blog is the record of that work and a testament to the countless hours I have invested in a poorly cooled stack of rust, scrap, and duct tape.

The map

The router and firewall, with its VLANs, leads to the main node. On the main node a reverse proxy, with internal names and HTTPS, sits in front of four groups: media, home and life, dev and AI, and infrastructure (DNS, file shares, the Tailscale router). Every container and VM on the main node is backed up nightly to the backup server.
Select the diagram to open it full size.

Hardware

MachineWhat it isJob
Main nodeAMD Ryzen, 64 GB RAM, Intel Arc A380, ZFS pools on SSD and HDDRuns every service, as LXC containers and VMs
Backup serverAn OptiPlex 3050 (i5-7500)Proxmox Backup Server
Router / firewallAn OptiPlex 7040 (6th-gen i5) running OPNsenseRouting, firewall, VLANs
SwitchManaged, PoECarries the VLANs
Wi-FiOne Wi-Fi 7 access pointHopefully self-explanatory…

Network

The network is split into VLANs: trusted devices, the lab, IoT, work, the rest of the household, guests, and a DMZ. The firewall then decides what can talk to what.

I assign every service an internal name using a DNS rewrite, and the reverse proxy serves them over HTTPS with a wildcard certificate for an internal domain. This means I never type (or memorize) an IP address or click through a certificate warning.

There are no port forwards and no tunnels, so nothing is exposed to the internet. Away from home, I connect over Tailscale. While not in the spirit of “own everything” and FOSS, it is ridiculously easy to set up and use. Another added benefit is that it’s relatively painless (ymmv) to get other people on board. A few friends and family can reach select services through Tailscale sharing, and only the containers those services run on, not the rest of the network.

Four ways in. Trusted devices and the rest of the household connect through the router and firewall box, which gives each group its own VLAN. Trusted devices reach the reverse proxy on the main node, which fronts media and everything else; the household reaches media through one firewall rule. Away from home, I come in over Tailscale through a subnet router on the main node to the reverse proxy. Friends and family reach the media containers through Tailscale sharing.
Select the diagram to open it full size.

What runs on it

~thirty containers, in four groups:

How it’s run

How it got here

I estimate this whole thing started around the beginning of 2022. It has gone through many iterations since, and I have nuked everything to start from scratch on more than one occasion. Since then, my current lab (roughly v3 I’d say) has been a Ship of Theseus: replaced one harvested part or eBay purchase at a time. Like most long-lived systems, this lab has held up with varying degrees of grace and a number of shouted curses.

It may sound lame, but this homelab has been an important part of my life and something that has helped me grow and grown alongside me. We’ve been through 4 apartments together (so far), and I deeply believe that my life would be in a very different place had I not started this hobby.