diff --git a/content/posts/2019-12-03-kubernetes-setup.md b/content/posts/2019-12-03-kubernetes-setup.md
new file mode 100644
index 0000000..4564ead
--- /dev/null
+++ b/content/posts/2019-12-03-kubernetes-setup.md
@@ -0,0 +1,340 @@
+---
+title: Single node Kubernetes setup with kubeadm and cert-manager
+date: 2019-12-03
+author: James McDonald
+type: post
+draft: true
+categories:
+ - Tech
+---
+
+This is an update of my [previous post](/posts/2019-04-04-kubernetes-setup/)
+about how to set up a single node Kubernetes cluster. As noted below, this
+is an evolving technology and the details of setting it up change over time.
+
+This follows
+https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ for the
+initial cluster setup. You should look there for more detail. I strongly
+recommend that you read about the various projects and verify that this
+information is still valid. Things change pretty quickly. I've deliberately not
+included a lot of detail in the post beyond what I did and the commands to use.
+
+# Requirements
+
+You can work around these, but it's recommended to have at least:
+
+* 4GB RAM
+* 2 cores
+
+# Decisions
+
+Kubernetes is an open ended platform which lets you make a lot of decisions
+yourself. Unfortunately it also requires you to make them (see projects like
+k3s to avoid some of that). I decided to use:
+
+* Debian stable (buster), because why would you ever not use Debian?
+* Docker as the container runtime. Next time I'll try something else (CRI-O),
+  but Docker is familiar so it's easy to diagnose any issues.
+* Weave Net for networking. I had some trouble with Flannel, and Weave also 
+  supports NetworkPolicy if you want to tinker with that.
+* No Helm. Helm is great, but it hides away details and I'm interested in
+  details.
+* nginx for ingress. There are a bunch of possibilities, but nginx is the
+  common choice and integrates nicely with lots of other things.
+* Let's Encrypt certificates validated by Cloudflare DNS. I use Cloudflare
+  already, so this was an easy choice. DNS validation allows for wildcards and
+  for internal hosts.
+* Single node. There are a lot of cool things about Kubernetes that you don't
+  get with a single node, but what I'm setting up here is for home. You can
+  easily add more nodes by following the instructions `kubeadm` gives you when
+  it runs.
+
+# Enable net.bridge.bridge-nf-call-iptables
+
+This is required by Weave and other networking options. You can read
+more at
+https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
+
+This was actually already set on my machine, but it doesn't hurt to be explicit.
+
+```
+cat > /etc/sysctl.d/20-bridge-nf.conf <<EOF
+net.bridge.bridge-nf-call-iptables = 1
+EOF
+sysctl --system
+```
+
+# Install Docker with recommended settings
+```
+mkdir /etc/docker
+cat > /etc/docker/daemon.json <<EOF
+{
+  "exec-opts": ["native.cgroupdriver=systemd"],
+  "log-driver": "json-file",
+  "log-opts": {
+    "max-size": "100m"
+  },
+  "storage-driver": "overlay2"
+}
+EOF
+apt-get update
+apt-get install -y \
+    apt-transport-https \
+    ca-certificates \
+    curl \
+    gnupg2
+curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+echo 'deb [arch=amd64] https://download.docker.com/linux/debian buster stable' > /etc/apt/sources.list.d/docker.list
+apt-get update
+apt-get install -y --no-install-recommends docker-ce
+```
+
+The `--no-install-recommends` will avoid pulling in stuff you don't need,
+including the aufs DKMS package.
+
+# Switch to legacy iptables
+
+Debian 10 (buster) and newer Ubuntu come with a new iptables that uses
+netfilter. This is not currently supported by kubeadm, so you have to switch to
+the legacy iptables.
+
+```
+sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
+sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
+sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy
+```
+
+If you don't do this you will get the same firewall rule added every few
+seconds until the network becomes unusably slow. Note that there is an
+`iptables` 1.8.3 package in buster-backports which may fix this problem.
+Personally I'm using legacy for the moment.
+
+# Install Kubernetes components
+```
+curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
+cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
+deb https://apt.kubernetes.io/ kubernetes-xenial main
+EOF
+apt-get update
+apt-get install -y kubelet kubeadm kubectl
+```
+
+The `xenial` in the APT source is correct. That's the repo they seem to
+update, and these are Go binaries anyway so they're self-contained.
+
+# Run `kubeadm` to set up the cluster
+
+```
+kubeadm init
+```
+
+That's it. Neat, huh?
+
+There is still a bunch of work to do to make the cluster actually useful. You
+can do most of the rest of this as a non-root user. Follow the instructions
+`kubeadm` gave you to copy the credential **as your regular user**.
+
+```
+mkdir -p $HOME/.kube
+sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+sudo chown $(id -u):$(id -g) $HOME/.kube/config
+```
+
+And as a handy extra tip, you'll want completion:
+
+```
+source <(kubectl completion bash)
+```
+
+# Install Flannel for pod networking
+```
+kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
+kubectl get pods --all-namespaces
+```
+
+You should see coredns pods come to life if all is well.
+
+# Untaint the master so you can run pods
+```
+kubectl taint nodes --all node-role.kubernetes.io/master-
+```
+
+At this point you can run pods and expose them with services. If that's all you
+need, you're done! Next I set up an nginx-ingress and cert-manager to allow for
+hostname-based HTTPS ingress with Let's Encrypt certificates.
+
+# Set up nginx-ingress
+
+I set up the nginx-ingress with host networking so I could expose my cluster's
+services via ports 80 and 443 on the host. You can also expose it via a NodePort or
+LoadBalancer, but this worked well for my simple setup.
+
+```
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
+cat > nginx-host-networking.yaml <<EOF
+spec:
+  template:
+    spec:
+      hostNetwork: true
+EOF
+kubectl -n ingress-nginx patch deployment nginx-ingress-controller --patch="$(<nginx-host-networking.yaml)"
+```
+
+You can now create Ingresses to make your services accessible externally. If
+you expose them via HTTP they should just work. HTTPS will work too, but with a
+self-signed certificate.
+
+# Set up cert-manager
+
+This is a bit fiddly, which isn't helped by the fact cert-manager's
+documentation is a bit out of date. Luckily, I read the source so you don't
+have to!
+
+The cert-manager tool is pretty neat. It lets you provision Let's Encrypt
+certificates either manually (useful for a fallback wildcard certificate) or
+automatically based on annotation on your Ingress.
+
+I have overridden the nameservers used for verifying dns01 records because I
+use split-horizon DNS.
+
+```
+kubectl create ns cert-manager
+kubectl config set-context --current --namespace=cert-manager
+kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/cert-manager.yaml
+cat > external-dns.yaml <<EOF
+# Add dns01 recursive nameservers
+spec:
+  template:
+    spec:
+      containers:
+        - name: cert-manager
+          args:
+          - --cluster-resource-namespace=\$(POD_NAMESPACE)
+          - --leader-election-namespace=\$(POD_NAMESPACE)
+          - --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
+EOF
+
+kubectl -n cert-manager patch deployment cert-manager --patch="$(<external-dns.yaml)"
+```
+
+# Set up a ClusterIssuer to issue certificates
+
+You'll need your Cloudflare API key to put into a Kubernetes secret.
+
+```
+kubectl -n cert-manager create secret generic cloudflare-api-key --from-literal=api-key=XXXXX
+cat > clusterissuer-cf.yaml <<EOF
+apiVersion: certmanager.k8s.io/v1alpha1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: you@example.com
+
+    privateKeySecretRef:
+      name: letsencrypt
+
+    dns01:
+      providers:
+        - name: cf-dns
+          cloudflare:
+            email: you@example.com
+            apiKeySecretRef:
+              name: cloudflare-api-key
+              key: api-key
+EOF
+kubectl apply -f clusterissuer-cf.yaml
+```
+
+# Create a wildcard certificate
+
+I created a wildcard certificate in the ingress-nginx namespace so I can use it
+as the default.
+
+```
+cat > wildcard-cert.yaml <<EOF
+apiVersion: certmanager.k8s.io/v1alpha1
+kind: Certificate
+metadata:
+  name: wildcard-example-com
+  namespace: ingress-nginx
+spec:
+  secretName: wildcard-example-com
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt
+  commonName: '*.example.com'
+  dnsNames:
+    - example.com
+  acme:
+    config:
+      - dns01:
+          provider: cf-dns
+        domains:
+          - '*.example.com'
+          - 'example.com'
+EOF
+kubectl apply -f wildcard-cert.yaml
+```
+
+You can watch the progress of the certificate generation. Tab completion should find the names of your orders and challenges for you.
+
+```
+kubectl describe order ...
+kubectl describe challenge ...
+```
+
+Set this as the default certificate for the ingress controller as follows:
+```
+kubectl -n ingress-nginx edit deployment nginx-ingress-controller
+```
+
+
+
+Add:
+
+```
+        - --default-ssl-certificate=$(POD_NAMESPACE)/wildcard-example-com
+```
+
+to the list of args to the `nginx-ingress-controller` container. You can do
+this with a patch if you prefer.
+
+# Try it out!
+
+You should have DNS set up for, in this example, `*.example.com` pointing to you Kubernetes host.
+
+Deploy a simple nginx service to test things out.
+
+```
+kubectl create ns testland
+kubectl config set-context --current --namespace=testland
+kubectl create deployment nginx --image nginx
+kubectl expose deployment nginx --port 80
+cat > nginx-ingress.yaml <<EOF
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: nginx
+spec:
+  rules:
+  - host: nginx.example.com
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: nginx
+          servicePort: 80
+  tls:
+  - hosts:
+    - nginx.example.com
+EOF
+kubectl apply -f nginx-ingress.yaml
+```
+
+You should be able to visit http://nginx.example.com/ in your browser and get redirected to HTTPS with a real, live certificate.
+
+I'll run through the process again to check for any errors and update this post.