Jimmy B. - Setting Up Gitpod on k3s Using the NEW Installer

As of December 8, 2021 Gitpod no longer supports self-hosting the Gitpod software. You can read more about this here. In short Gitpod no longer sells licenses and official support for the self-hosted product. You can still however setup a self-hosted instance of Gitpod. Check out my new post about setting up your own self-hosted Gitpod instance here.

For my new Gitpod instance I used 4 VMs. The first VM is going to be the master/controller node and the other three will be my worker nodes.

I created my controller node with 2 CPU cores, 4GBs of RAM and 80GBs of disk space. I used Ubuntu Server 20.04.3 for the operating system. During the install process I setup my partitions as follows:

/boot - 1G
/ - remaining space

Once the install of Ubuntu completed I started setting the system up as follows:

Setting Up the Kubernetes Controller

For whatever reason Ubuntu doesn't allow '.' in the hostname when going through the installation process so the first step is to fix the hostname in /etc/hostname. I set mine to kubemaster.mydomain.com. Then run hostname -F /etc/hostname.
This step is optional. My Kubernetes cluster controller and work nodes all have private IP addresses. I've allocated the IP address block 192.168.102.1 - 192.168.102.253 to the cluster. This is not required if your systems have public IPs and resolvable hostnames. I did this so each of the systems in my cluster know how to properly communicate with each other. My hosts file looked like this:
```
127.0.1.1 kubemaster.mydomain.com kubemaster
192.168.103.1 kubemaster.mydomain.com
192.168.103.2 kubenode1.mydomain.com
192.168.103.3 kubenode2.mydomain.com
192.168.103.4 kubenode3.mydomain.com
```
Next set the timezone of the system to use your timezone ( in my case it's America/Chicago) - dpkg-reconfigure tzdata.
Run apt -y update && apt -y upgrade to update the software on the system.

Now we'll install MariaDB 10.5. This is used to store some information from k3s.

apt -y install curl software-properties-common dirmngr
apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
add-apt-repository 'deb [arch=amd64,arm64,ppc64el,s390x] https://mirror.rackspace.com/mariadb/repo/10.5/ubuntu focal main'
apt -y install mariadb-server mariadb-client
systemctl status mariadb
systemctl enable mariadb
mysql_secure_installation

Create the database for k3s. You'll want to login using the root MySQL password you set when you ran mysql_secure_installation in the previous step.

mysql -u root -p
CREATE DATABASE k3s CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'k3s'@'localhost' IDENTIFIED BY '$PASSWORD';
GRANT ALL PRIVILEGES ON k3s.* TO 'k3s'@'localhost';
FLUSH PRIVILEGES;
quit

Make sure to actually change out $PASSWORD with a secure password.

I install k3s:

curl -sfL https://get.k3s.io | sh -s - --datastore-endpoint="mysql://k3s:$PASSWORD@tcp(localhost:3306)/k3s" --write-kubeconfig-mode 644 --disable traefik --disable servicelb

Make sure you swap out $PASSWORD with the password you set for the k3s MySQL user in the previous step.

Edit /etc/fstab and comment out the line for swap.
```
# /swap.img none swap sw 0 0
```
Reboot the server (reboot). Once rebooted, log back into the system and run free -mh to make sure it shows swap like so:
```
Swap:            0B          0B          0B
```
If it does, then remove the /swap.img file.
Next we need to grab the k3s node-token and save it somewhere safe. It'll be a long string of characters and numbers.
```
cat /var/lib/rancher/k3s/server/node-token
```
I also have another system that I connect to my cluster from. If you do too grab the configuration file from the cluster:
```
cat /etc/rancher/k3s/k3s.yaml
```
Take the contents there and place it into the /home/user/.kube/config file if you're on Linux or /Users/user/.kube/config file if on macOS. You may need to create the .kube directory first. You'll need to edit the file to point to the IP address of your Kubernetes controller. You should find a line inside the config file that looks like:
```
server: https://127.0.0.1:6443
```
You'll want to change 127.0.0.1 to the IP or hostname of your controller. Save and exit the file.

Run kubectl get nodes. This should return something like:

 NAME                        STATUS   ROLES                  AGE     VERSION
 kubemaster.mydomain.com    Ready    control-plane,master   4d23h   v1.21.7+k3s1

Setting Up the Kubernetes Worker Nodes

The next three VMs are going to be our worker nodes where pods can be spun up and such. I created each of these with 8 CPU cores, 8GBs of RAM and 250GBs of disk space. I've used Ubuntu Server 20.04.3 on these as well. During the install process I setup my partitions as follows:

/boot - 1G
/ - remaining space

Once the operating system has finished installing we can go through the following steps to finish setting up the system.

First I fix the hostname in /etc/hostname. I set mine to kubenodeX.mydomain.com where 'X' is a number. So for example, on my first worker node, the hostname is kubenode1.mydomain.com, the second worker is kubenode2.mydomain.com and so on. Once that file has been edited and saved I run hostname -F /etc/hostname to set the hostname.
This step is optional. My Kubernetes cluster controller and work nodes all have private IP addresses. I've allocated the IP address block 192.168.102.1 - 192.168.102.253 to the cluster. This is not required if your systems have public IPs and resolvable hostnames. I did this so each of the systems in my cluster know how to properly communicate with each other. My hosts file looked like this:
```
127.0.1.1 kubemaster.mydomain.com kubemaster
192.168.103.1 kubemaster.mydomain.com
192.168.103.2 kubenode1.mydomain.com
192.168.103.3 kubenode2.mydomain.com
192.168.103.4 kubenode3.mydomain.com
```
Next set the system time to use your timezone (in my case it's America/Chicago) - dpkg-reconfigure tzdata.
Run apt -y update && apt -y upgrade to update all the packages on the system.

Next we're going to install k3s in agent mode:

export K3S_URL=https://kubemaster.mydomain.com:6443
export K3S_TOKEN=K104887s5p9182394ydc31c4988f6761844fe71e54ee93f6f64a76dsa87df800c86::server:39aef067sa87d8as9d6d7fb981db4
curl -fsL https://get.k3s.io | K3S_URL=$K3S_URL K3S_TOKEN=$K3S_TOKEN sh -s agent

Make sure to set K3S_URL properly. You can use the hostname or IP address of your controller node here. Make sure you also set the K3S_TOKEN with the node-token you got from your controller node.

Edit /etc/fstab and comment out the line for swap.
```
# /swap.img none swap sw 0 0
```
Reboot the server (reboot). Once rebooted, log back into the system and run free -mh to make sure it shows swap like so:
```
Swap:    0B    0B    0B
```
If it does, then remove the /swap.img file.

You'll repeat these 7 steps for each of your Kubernetes worker nodes.

Installing Rancher

This section is 100% optional. I like to have Rancher as it provides me a visual overview of my Kubernetes cluster as well as I can manage the cluster in Rancher as well. I should note that I have Traefik running which handles fetching and managing the TLS certificate to my Rancher instance.

From the system which has access to your Kubernetes cluster via kubectl, create a new namespace in your cluster. This is a requirement of Rancher.
```
kubectl create namespace cattle-system
```
Next we'll need to install the Helm chart for the latest version(s) of Rancher:
```
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
```
As noted before I already have Traefik setup and handling TLS certificates so this is the command I use to deploy Rancher to my Kubernetes cluster:
```
helm install rancher rancher-latest/rancher \
    --namespace cattle-system \
    --set hostname=rancher.mydomain.com \
    --set tls=external
```
This will deploy the latest version of Rancher to your Kubernetes cluster. You can run the following command to check the progress of the deployment:
```
kubectl -n cattle-system rollout status deploy/rancher
```
Once the deployment is complete you will likely need to assign an external IP to the Rancher service. Make sure you assign an IP to the service where Rancher is actually deployed. You can use a command like:
```
kubectl patch svc rancher -p '{"spec":{"externalIPs":["192.168.103.10"]}}' -n cattle-system
```
Now you should be able to visit https://rancher.yourdomain.com.

Installing Gitpod

Yay, we made it this far! Now we can get to installing Gitpod! For the following you need to be on a system that has access to your Kubernetes cluster (via kubectl), and that has Docker installed.

First we're going to add labels to our Kubenetes worker nodes. These are required for Gitpod. The following command may be different for you, but you'll want to give the hostnames of all of your worker nodes in the for i in bit. For me this looked like:
```
for i in kubenode1.mydomain.com kubenode2.mydomain.com kubenode3.mydomain.com ; do kubectl label node $i gitpod.io/workload_meta=true gitpod.io/workload_ide=true gitpod.io/workload_workspace_services=true gitpod.io/workload_workspace_regular=true gitpod.io/workload_workspace_headless=true ; done
```
This will add all the necessary labels to your worker nodes in a single command.
Next we need to build the pre-build installer. First visit werft and pick a build you want to run. For example gitpod-build-main.2071 - main.2071. We'll use the following commands to build the installer:
```
docker create -ti --name installer eu.gcr.io/gitpod-core-dev/build/installer:main.2071
docker cp installer:/app/installer ./installer
docker rm -f installer
```
Note that the build is there on the end of that first command.
The next command will generate a base configuration file:
```
./installer init > gitpod.config.yaml
```
From here you can open your gitpod.conf.yaml file to customize. At minimum make sure to set the domain: option to be your domain and update the workspace.runtime.containerdRuntimeDir and workspace.runtime.containerdSocket values.
```
containerdRuntimeDir: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
containerdSocket: /run/k3s/containerd/containerd.sock
```
I also setup the authProviders: section as I have a GitHub Enterprise instance. It should be noted the authProviders may be moved to a secret in the future.

You can also setup external cluster dependencies such as an external Docker registry, database, and object storage. In order to use external dependencies you'll need to set the inCluster setting for them to true. For example if using an external database, the database section would look like:
```
database:
  inCluster: false
  external:
    certificate:
      kind: secret
      name: database-token
```
Your database-token secret needs to have the following key/pair values:
- encryptionKeys - database encryption key
- host - IP or URL of the database
- password - database password
- port - database port, usually 3306
- username - database username

Next we'll need to install cert-manager into our Kubernetes cluster. This is required even if you're providing your own TLS certificate for Gitpod. cert-manager will be used to generate certificates for internal Gitpod services.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade \
    --atomic \
    --cleanup-on-fail \
    --create-namespace \
    --install \
    --namespace='cert-manager' \
    --reset-values \
    --set installCRDs=true \
    --set 'extraArgs={--dns01-recursive-nameservers-only=true,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}' \
    --wait \
    cert-manager \
    jetstack/cert-manager

Next I create a new namespace in Kubernetes for Gitpod. You don't have to do this, Gitpod will use the namespace 'default' by default. I'm sort of an organizational freak so I prefer to keep everything Gitpod in its own namespace. The following command creates a new namespace, gitpod.
```
kubectl create namespace gitpod
```
Since I have my own TLS certificate for Gitpod I manually created the https-certificates secret in Kubernetes. If you're familiar with doing this via the command-line go for it! I used Rancher. Of note, you must ensure your TLS certificate contains domain.com, *.domain.com and *.ws.domain.com. If you'd like to create the 'https-certificates' secret via Rancher you may following these steps:

First I bring up https://rancher.mydomain.com, and login as necessary. You should see a screen that has the following:

Click on 'local'. From the sidebar on the left side click on 'Storage' and then 'Secrets'.

Then click on the blue 'Create' button.

On the page that comes up with 4-5 boxes, click on the TLS Certificate box.

On the next page ensure the Namespace is set to 'gitpod' and the Name is set to 'https-certificates'.

Fill in the Private Key and Certificate files accordingly. I used a certificate from Let's Encrypt so I pasted in the contents of the fullchain.pem file into the Certificate field. Hit the blue 'Create' button when you're set.
Next validate the Gitpod configuration file:
```
./installer validate config --config gitpod.config.yaml
```
Hopefully everything looks good here and "valid": true is returned.
Next we need to check that our Kubernetes cluster is setup properly for Gitpod:
```
./installer validate cluster --kubeconfig ~/.kube/config --config gitpod.config.yaml
```
You may see any error about the https-certificates secret not being found, but that should be fine to ignore.
Render the Gitpod YAML file. This is what will be used to deploy Gitpod to your Kubernetes cluster.
```
./installer render --config gitpod.config.yaml --namespace gitpod > gitpod.yaml
```
TIME TO DEPLOY GITPOD!!
```
kubectl apply -f gitpod.yaml
```
You can watch as things get setup and deployed in Rancher or on the command-line you can run:
```
watch -n5 kubectl get all -n gitpod
```
or if you just want to watch the pods:
```
watch -n5 kubectl get pods -n gitpod
```
If everything is all good and happy, you should see all of the pods should show as Running.
If you run kubectl get svc -n gitpod you may notice the proxy service doesn't have an External IP. This is normal since we don't have anything running to assign out external IPs at the moment.
```
proxy    LoadBalancer    10.43.230.187    <none>    80:32673/TCP,443:32262/TCP,9500:32178/TCP    4d19h
```

Installing MetalLB

This step is 100% optional as well if you already have a service that assigns external IPs. If you don't and want something quick and easy then let's have a look at MetalLB.

Again from our system which has access to our Kubernetes cluster (via kubectl) we'll setup MetalLB.

Install MetaLB via Manifests:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

This will install MetalLB under the metallb-system namespace.

This next step we can do either via Rancher or on the command-line. I've done it via Rancher myself but the choice is yours. In Rancher I click on the 'Import YAML' button found in the upper right corner:

Next I paste in the following as a template:
```
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.240-192.168.1.250
```
I changed the last line to be the IP range that I want MetalLB to hand out. In my case this is 192.168.102.3-192.168.102.253. Make sure to select 'metallb-system' from the 'Default Namespace' dropdown menu. Hit the blue 'Import' button when you're all set.
I've also setup IP blocks on my worker nodes. I gave 192.168.102.10 - 192.168.102.19 to kubenode1.mydomain.com, 192.168.102.20 - 192.168.102.29 to kubenode2.mydomain.com, and 192.168.102.30 - 192.168.102.39 to kubenode3.mydomain.com. I set that up within their /etc/netplan/00-installer-config.yaml file.
Back on the command-line if you run kubectl get svc -n gitpod you should see the proxy service now has an IP address:
```
proxy    LoadBalancer    10.43.230.187    192.168.103.11    80:32673/TCP,443:32262/TCP,9500:32178/TCP    4d19h
```