Hi! I’ve done the 100nt revamp of my homelab so I want to do a small post to describe and document how everything is doing and where I wanna go since I think this is relatively interesting.
First and foremost, do not forget what I want my homelab for. My homelab is excessive, overkill, baroque... Do I need a Kubernetes cluster? No, obviously not. Do I learn a lot by having and maintaining it? Hell, yes!
My homelab is optimized for learning. I try stuff, I am wild, I am naive and crazy. Sorry, not sorry.
Obviously it can be done better. I expect no less.
The sections are in no particular order.
🔌Hardware
I have a small Thinkcentre m90q kubernetes cluster. It is 3 machines. All are a tiny bit different but they work fine together anyway.
Next to that I have my good old Qnap with 4x8TB disks in RAID5.
I 3d printed a new homebrew NAS. I want probably to ditch RAID for it. RAID is not a backup. I learnt that the wrong way. RAID only provides availability and just some extra reliability. I have to do the backups anyway and not having the data available in NAS for some hours is something I can definitely live with. Specially since I am moving the data that needs to be available to Longhorn volumes in Kubernetes. More on this later.
The new NAS is the fourth Proxmox node you see in the screenshot.
This one will also double down as Proxmox Backup Server.
☎️ Networking
I moved production to Hetzner but I don’t want to use straight their platform. Maybe I am a bit cheap but I rented a Hetzner server and installed OPNSense on it.
Since I took that path in my servers I bought a small pfSense box and installed OPNSense on it. I use it to create a small Wireguard network so I can see my production virtual machines locally to ease debugging.
This means that at home I can do ssh myuser@192.168.2.123 and I will see the machine as if it was here. This is a bit more nuisance to set up at router level but it simplifies a lot the configuration of the continuous integration and deployment. I used to have VPN at computer level but setting this in my router makes everything far easier.
Security is a concern anyway. I know now my home becomes a gigantic security issue 🥲
The future
I use Wireguard tunnels to access my network from the outside but I want to start giving access to some friends. I will probably have to fight somehow for the IP address spaces and I don't know how to properly solve that yet.
Operating systems
I rely on Proxmox and bare Debian a lot. I know Proxmox is just software over Debian but you know what I mean. I am starting to have all my servers in plain Debian. Indeed moving to a homebrew NAS will mean moving from QNap's propietary thing to Proxmox with Debian VMs.
I have, anyway exceptions. Since Ghost recommends Ubuntu is is using it underneath.
General tools
So you can get a good view of the mess I am in I keep a Homepage installation with links to a lot of stuff here. It looks a lot because it is a lot but I think in a couple of iterations I can get a super fun start page. At the moment the mess looks like this.
You don't have to be super smart to notice I am mixing hosted stuff with remote stuff. I need a couple of iterations on this. I could throw some HTML and that is it but for some reason I am using homepage for this.
Also I have some concerns on storing secrets for homepage. The way it is done is not kubernetes secrets friendly so I have to play a bit here.
Anyway since it is not exposed to the internets I can live with it for a while.
Development tools
ArgoCD
Since I have a k8s cluster I need to have something to deploy stuff there. I am starting to have so many things that I need to keep everything versioned in version control.
Forgejo
I am moving some repos to Forgejo and using GitHub just for a backup. I am using forgejo actions (which are mostly GitHub compatible by the way) since I don’t need super complex workflows for anything.
Harbor
A simple docker image registry.
Longhorn
For the important storage I've been using longhorn last months. It is a very cool way of having storage redundancy in my K8S nodes.
Also the volumes in longhorn are the ones I consider critical so backups are regularly made to a Backblaze b2 bucket.
Prometheus and Grafana
One of the cool things of using Kubernetes is that you can use a Grafana operator to host the Json of the dashboard of every service in their own repo so you can compose the final picture based on the dashboards you have stored in every application repository.
It is not like I have absolutely everything under control yet but it is a small work in progress.
Development station
I still keep my dotfiles alongside my neovim config so you can see here how I work. I use XMonad in my main computer but I cannot consider myself a power user.
I host this at my Forgejo but it is doubled down in Github so it is easier to share with anyone.
Media consumption
Plex
A good old Plex. I could move to an Open source solution but I am too lazy to relinquish on my lifetime license.
Audiobookshelf
My audiobook host. I don't use subscription services like Audible so I have this so I can synchronize the position in the book from my phone and the computer.
Miniflux
My trustworthy RSS reader. I use it for blogs but it doubles down as a notificator for new releases of software I use. If it is hosted on Github or the like you can subscribe to the releases feed to know when an update is posted.
Also I use its integration with telegram to have notifications on my mobile phone when something changes.
ArchiveBox
ArchiveBox is Pocket on steroids and self hosted. You send it URLs and he archives them. Simple as that. It generates PDF files for everything which are the view I often use.
General tooling
Plausible
I have a plausible instance for analytics for this blog. It is not something extremely important but it is better than selling my visitor's data to Google.
Ghost
This blog is hosted on Ghost. Since now I have a remote Proxmox I can get small VMs for cheap so this sounds like a better way to ensure I continue writing.
Klipper
To control my filament 3d printer I have a small Creality sonic pad. This acts as a klipper server that has a webcam connected so I can see what is going on in the printer.
That is mostly it. It is a lot and it is overkill but it is extremely fun.