Welcome, this is a blog post I've been procrastinating on, but it is the cornerstone to the whole topic of my blog. This post will detail my k3s installation for my home server - this post intends to describe exactly how I have a self-hosted Kubernetes cluster. This post is a mega-post, so to speak. In order to fully detail my k3s setup, it depends on a few other posts detailing the peripherals I've installed to my k3s cluster that are still integral for it to function.
First, a quick overview: What is k3s? Well, it's a single binary Kubernetes install, that is, the one binary includes all the parts of the Kubernetes master and node, and it provides a slick way of configuring this binary in order to create your own clusters with ease. K3s is a Rancher product, although for the purposes of this tutorial, we will not be installing this cluster within Rancher or Docker, instead, this will be a bare-metal install.
Hardware
My hardware is nothing fancy, but, k3s is lightweight and works great on last decade hardware 😜.
I've named all my computers after Norse gods, and so I'll reference each one by that name when applicable!
thor - An old Lenovo workstation, rocking an Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz and 8gb of DDR2 RAM
odin - A hand-me-down custom built mini form factor PC with an Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz and 16gb of DDR3 RAM
loki - An HP Compaq I picked up from craigslist for $50, with an Intel(R) Core(TM) i5 CPU 650 @ 3.20GHz and 16gb of DDR3 RAM.
I replaced each HD with a 240gb SSD, for about $30 each from Amazon, and I've installed Ubuntu 20.04LTS on each of these computers. If I had to guess, I'm about $300 deep in hardware and equipment, but this setup is working excellently for me!
I have an additional Rasbperry Pi 3B+ that I've converted to an NFS server. It's name is heimdall. It's sole purpose is a backup location for my Kubernetes volume solution, so that I have less of a chance of losing critical data - this has an old 320gb spinning external HD that the NFS drive. This piece of my home setup will not be documented at this time beyond the fact that I have an NFS server for these backups.
Getting Started
First things first, this blog is going to make the assumption that you have 1 or more Linux based computers configured and ready to go. If you are following my guide in totality, there are few things you need to run directly on each server to prepare the nodes themselves to function with Longhorn - so be sure to pull up this tutorial and pay attention to the commands you will need for each node.
Regardless of how you're setting this all up, the first thing you want to do is update your system.
sudo apt update \
&& sudo apt upgrade -yThere are a few tools that I like for remote management of my cluster. I keep this minimal and light, but be sure to install the following tools:
sudo apt install openssh-server -y \
&& sudo apt install curl -y \
&& sudo apt install htop -yopenssh-server allows us to ssh in later, so that we can remotely manage each server from our LAN. curl is required for installing k3s. htop is a visual top command, to give you an idea of how your server is doing at a point in time.
Preparing k3s
Before installing k3s (which is really easy), we want to create the k3s config file on our first (or only) master node. We will be going from the Rancher docs here. For simplicity, create this file at /etc/rancher/k3s/config.yaml
Here are the contents of mine:
node-name: thor
write-kubeconfig-mode: 644
write-kubeconfig: /etc/rancher/k3s/kubeconfig
tls-san:
- "thor.local"
disable:
- "traefik"
- "local-storage"
- "servicelb"
token: {secret-token-prefix}Let's break down each part!
The node-name declares how this node will appear with kubectl get nodes. Thor is my only Kubernetes master. When I started, it was the only computer I had, and it's been reliable so far. If you have 3 computers at the beginning, you should strongly consider running with the embedded etcd database for high-availability mode. I did not, and to migrate to this now would be starting over. This is something I'll tackle if and when Thor dies.
The kubeconfig properties are important, as this instructs k3s to generate your kubeconfig file which you will need if you want to manage your Kubernetes cluster from another computer. I manage mine from my laptop.
The tls-san declares a domain name that Kubernetes will allow kubectl commands from. It will always include localhost. It was important for me to add thor.local so that I can manage my cluster from my LAN. You will need to add the mDNS name of your master server(s) here.
the disable property is straight forward, although for everything that we disable, we will need to provide a replacement. This is where the uber-blog bit comes into play - each of these replacements are detailed, and worthy of their own blog post.
- traefik - k3s ships with Traefik v1. I replaced this with Traefik v2, which has wonderful Kubernetes support.
- local-storage - By default, k3s only has the ability to mount temporary folders, which are deleted when the pod dies. This isn't good for anything stateful (which I run several stateful things), so to fix this, I replaced it with Longhorn.
- servicelb - this is the external networking component of Kubernetes, and servicelb is only capable of binding to the public LAN IP addresses of the host nodes themselves. This isn't good enough for me, since I want some services accessible on their own IP addresses and therefore their own mDNS host names on my LAN, and so I replaced this with MetalLB.
lastly, token for part of the key for connecting other servers to this master, whether they are other masters or worker nodes.
Installing k3s
With that config file created, you're ready to head over to https://k3s.io/ and run the installation command.
curl -sfL https://get.k3s.io | sh -After a short amount of time, you should have your first Kubernetes node and master up and ready to go!
When that command is done, you can copy the file at /etc/rancher/k3s/kubeconfig to your laptop or external computer you want to manage your Kubernetes cluster from. Treat this file like a password, if someone has this file combined with network access to the Kubernetes API server (hosted on port 6443 by default), they have admin access to your Kubernetes cluster. For this reason, I only expose my Kubernetes server on my local LAN, and I do not make port 6443 public on my router. I can only manage my home cluster when I'm at home.
Adding a Second (and Third, and beyond) Node
You are going to love this part of the tutorial.
Firstly, we need to find our secret token that is created by the master node, which will allow us to connect additional nodes.
You can find this token at /var/lib/rancher/k3s/server/node-token. Setting the K3S_URL environment variable automatically sets k3s to worker mode. Best of all, this command starts a service, so it auto-restarts when you reboot your server.
You'll want to set K3S_NODE_NAME to a unique node name of your choice. I set my other 2 node names to match the server name (Norse Gods and all 😂)
curl -sfL https://get.k3s.io \
| K3S_URL=https://{IP_ADDRESS_OF_MASTER}:6443 \
K3S_TOKEN={GENERATED_TOKEN} \
K3S_NODE_NAME=odin sh -
Adding worker nodes to your Home Kubernetes cluster is literally that easy!
You can see additional configuration options here.
At this point, you have an installed, multi-node k3s cluster that you also have kubectl access to. If you are following my tutorial, which includes the disabled bits, your cluster is not yet ready for the limelight. Be sure to continue the uber-blog series by next installing (in order, due to dependencies)
Be sure to drop a like, or even a tip if you followed along with this k3s install! I love my home server, it's opened up a whole new world of side-projects for myself since I now have a cheap place to host them! (I'm only paying for electricity on a stack of cheap hardware!)
