blog/content/posts/2019-12-03-kubernetes-setup.md

---
title: Single node Kubernetes setup with kubeadm and cert-manager
date: 2019-12-03
author: James McDonald
type: post
draft: true
categories:
 - Tech
---

This is an update of my [previous post](/posts/2019-04-04-kubernetes-setup/)
about how to set up a single node Kubernetes cluster. As noted below, this
is an evolving technology and the details of setting it up change over time.

This follows
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ for the
initial cluster setup. You should look there for more detail. I strongly
recommend that you read about the various projects and verify that this
information is still valid. Things change pretty quickly. I've deliberately not
included a lot of detail in the post beyond what I did and the commands to use.

# Requirements

You can work around these, but it's recommended to have at least:

* 4GB RAM
* 2 cores

# Decisions

Kubernetes is an open ended platform which lets you make a lot of decisions
yourself. Unfortunately it also requires you to make them (see projects like
k3s to avoid some of that). I decided to use:

* Debian stable (buster), because why would you ever not use Debian?
* Docker as the container runtime. Next time I'll try something else (CRI-O),
  but Docker is familiar so it's easy to diagnose any issues.
* Weave Net for networking. I had some trouble with Flannel, and Weave also 
  supports NetworkPolicy if you want to tinker with that.
* No Helm. Helm is great, but it hides away details and I'm interested in
  details.
* nginx for ingress. There are a bunch of possibilities, but nginx is the
  common choice and integrates nicely with lots of other things.
* Let's Encrypt certificates validated by Cloudflare DNS. I use Cloudflare
  already, so this was an easy choice. DNS validation allows for wildcards and
  for internal hosts.
* Single node. There are a lot of cool things about Kubernetes that you don't
  get with a single node, but what I'm setting up here is for home. You can
  easily add more nodes by following the instructions `kubeadm` gives you when
  it runs.

# Enable net.bridge.bridge-nf-call-iptables

This is required by Weave and other networking options. You can read
more at
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network

This was actually already set on my machine, but it doesn't hurt to be explicit.

```
cat > /etc/sysctl.d/20-bridge-nf.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
```

# Install Docker with recommended settings
```
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
apt-get update
apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg2
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
echo 'deb [arch=amd64] https://download.docker.com/linux/debian buster stable' > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y --no-install-recommends docker-ce
```

The `--no-install-recommends` will avoid pulling in stuff you don't need,
including the aufs DKMS package.

# Switch to legacy iptables

Debian 10 (buster) and newer Ubuntu come with a new iptables that uses
netfilter. This is not currently supported by kubeadm, so you have to switch to
the legacy iptables.

```
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy
```

If you don't do this you will get the same firewall rule added every few
seconds until the network becomes unusably slow. Note that there is an
`iptables` 1.8.3 package in buster-backports which may fix this problem.
Personally I'm using legacy for the moment.

# Install Kubernetes components
```
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
```

The `xenial` in the APT source is correct. That's the repo they seem to
update, and these are Go binaries anyway so they're self-contained.

# Run `kubeadm` to set up the cluster

```
kubeadm init
```

That's it. Neat, huh?

There is still a bunch of work to do to make the cluster actually useful. You
can do most of the rest of this as a non-root user. Follow the instructions
`kubeadm` gave you to copy the credential **as your regular user**.

```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

And as a handy extra tip, you'll want completion:

```
source <(kubectl completion bash)
```

# Install Flannel for pod networking
```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
kubectl get pods --all-namespaces
```

You should see coredns pods come to life if all is well.

# Untaint the master so you can run pods
```
kubectl taint nodes --all node-role.kubernetes.io/master-
```

At this point you can run pods and expose them with services. If that's all you
need, you're done! Next I set up an nginx-ingress and cert-manager to allow for
hostname-based HTTPS ingress with Let's Encrypt certificates.

# Set up nginx-ingress

I set up the nginx-ingress with host networking so I could expose my cluster's
services via ports 80 and 443 on the host. You can also expose it via a NodePort or
LoadBalancer, but this worked well for my simple setup.

```
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
cat > nginx-host-networking.yaml <<EOF
spec:
  template:
    spec:
      hostNetwork: true
EOF
kubectl -n ingress-nginx patch deployment nginx-ingress-controller --patch="$(<nginx-host-networking.yaml)"
```

You can now create Ingresses to make your services accessible externally. If
you expose them via HTTP they should just work. HTTPS will work too, but with a
self-signed certificate.

# Set up cert-manager

This is a bit fiddly, which isn't helped by the fact cert-manager's
documentation is a bit out of date. Luckily, I read the source so you don't
have to!

The cert-manager tool is pretty neat. It lets you provision Let's Encrypt
certificates either manually (useful for a fallback wildcard certificate) or
automatically based on annotation on your Ingress.

I have overridden the nameservers used for verifying dns01 records because I
use split-horizon DNS.

```
kubectl create ns cert-manager
kubectl config set-context --current --namespace=cert-manager
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/cert-manager.yaml
cat > external-dns.yaml <<EOF
# Add dns01 recursive nameservers
spec:
  template:
    spec:
      containers:
        - name: cert-manager
          args:
          - --cluster-resource-namespace=\$(POD_NAMESPACE)
          - --leader-election-namespace=\$(POD_NAMESPACE)
          - --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
EOF

kubectl -n cert-manager patch deployment cert-manager --patch="$(<external-dns.yaml)"
```

# Set up a ClusterIssuer to issue certificates

You'll need your Cloudflare API key to put into a Kubernetes secret.

```
kubectl -n cert-manager create secret generic cloudflare-api-key --from-literal=api-key=XXXXX
cat > clusterissuer-cf.yaml <<EOF
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@example.com

    privateKeySecretRef:
      name: letsencrypt

    dns01:
      providers:
        - name: cf-dns
          cloudflare:
            email: you@example.com
            apiKeySecretRef:
              name: cloudflare-api-key
              key: api-key
EOF
kubectl apply -f clusterissuer-cf.yaml
```

# Create a wildcard certificate

I created a wildcard certificate in the ingress-nginx namespace so I can use it
as the default.

```
cat > wildcard-cert.yaml <<EOF
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: ingress-nginx
spec:
  secretName: wildcard-example-com
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt
  commonName: '*.example.com'
  dnsNames:
    - example.com
  acme:
    config:
      - dns01:
          provider: cf-dns
        domains:
          - '*.example.com'
          - 'example.com'
EOF
kubectl apply -f wildcard-cert.yaml
```

You can watch the progress of the certificate generation. Tab completion should find the names of your orders and challenges for you.

```
kubectl describe order ...
kubectl describe challenge ...
```

Set this as the default certificate for the ingress controller as follows:
```
kubectl -n ingress-nginx edit deployment nginx-ingress-controller
```



Add:

```
        - --default-ssl-certificate=$(POD_NAMESPACE)/wildcard-example-com
```

to the list of args to the `nginx-ingress-controller` container. You can do
this with a patch if you prefer.

# Try it out!

You should have DNS set up for, in this example, `*.example.com` pointing to you Kubernetes host.

Deploy a simple nginx service to test things out.

```
kubectl create ns testland
kubectl config set-context --current --namespace=testland
kubectl create deployment nginx --image nginx
kubectl expose deployment nginx --port 80
cat > nginx-ingress.yaml <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
spec:
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80
  tls:
  - hosts:
    - nginx.example.com
EOF
kubectl apply -f nginx-ingress.yaml
```

You should be able to visit http://nginx.example.com/ in your browser and get redirected to HTTPS with a real, live certificate.

I'll run through the process again to check for any errors and update this post.