james/blog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add draft kubernetes post

Browse Source
This commit is contained in:
James McDonald
2020-06-10 19:20:25 +02:00
parent 40c0d2c5e0
commit 89512ac775
1 changed files with 340 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/posts/2019-12-03-kubernetes-setup.md
+340
View File
@@ -0,0 +1,340 @@
---
title: Single node Kubernetes setup with kubeadm and cert-manager
date: 2019-12-03
author: James McDonald
type: post
draft: true
categories:
- Tech
---
This is an update of my [previous post](/posts/2019-04-04-kubernetes-setup/)
about how to set up a single node Kubernetes cluster. As noted below, this
is an evolving technology and the details of setting it up change over time.
This follows
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ for the
initial cluster setup. You should look there for more detail. I strongly
recommend that you read about the various projects and verify that this
information is still valid. Things change pretty quickly. I've deliberately not
included a lot of detail in the post beyond what I did and the commands to use.
# Requirements
You can work around these, but it's recommended to have at least:
* 4GB RAM
* 2 cores
# Decisions
Kubernetes is an open ended platform which lets you make a lot of decisions
yourself. Unfortunately it also requires you to make them (see projects like
k3s to avoid some of that). I decided to use:
* Debian stable (buster), because why would you ever not use Debian?
* Docker as the container runtime. Next time I'll try something else (CRI-O),
but Docker is familiar so it's easy to diagnose any issues.
* Weave Net for networking. I had some trouble with Flannel, and Weave also
supports NetworkPolicy if you want to tinker with that.
* No Helm. Helm is great, but it hides away details and I'm interested in
details.
* nginx for ingress. There are a bunch of possibilities, but nginx is the
common choice and integrates nicely with lots of other things.
* Let's Encrypt certificates validated by Cloudflare DNS. I use Cloudflare
already, so this was an easy choice. DNS validation allows for wildcards and
for internal hosts.
* Single node. There are a lot of cool things about Kubernetes that you don't
get with a single node, but what I'm setting up here is for home. You can
easily add more nodes by following the instructions `kubeadm` gives you when
it runs.
# Enable net.bridge.bridge-nf-call-iptables
This is required by Weave and other networking options. You can read
more at
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
This was actually already set on my machine, but it doesn't hurt to be explicit.
```
cat > /etc/sysctl.d/20-bridge-nf.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
```
# Install Docker with recommended settings
```
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
apt-get update
apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg2
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
echo 'deb [arch=amd64] https://download.docker.com/linux/debian buster stable' > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y --no-install-recommends docker-ce
```
The `--no-install-recommends` will avoid pulling in stuff you don't need,
including the aufs DKMS package.
# Switch to legacy iptables
Debian 10 (buster) and newer Ubuntu come with a new iptables that uses
netfilter. This is not currently supported by kubeadm, so you have to switch to
the legacy iptables.
```
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy
```
If you don't do this you will get the same firewall rule added every few
seconds until the network becomes unusably slow. Note that there is an
`iptables` 1.8.3 package in buster-backports which may fix this problem.
Personally I'm using legacy for the moment.
# Install Kubernetes components
```
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
```
The `xenial` in the APT source is correct. That's the repo they seem to
update, and these are Go binaries anyway so they're self-contained.
# Run `kubeadm` to set up the cluster
```
kubeadm init
```
That's it. Neat, huh?
There is still a bunch of work to do to make the cluster actually useful. You
can do most of the rest of this as a non-root user. Follow the instructions
`kubeadm` gave you to copy the credential **as your regular user**.
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
And as a handy extra tip, you'll want completion:
```
source <(kubectl completion bash)
```
# Install Flannel for pod networking
```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
kubectl get pods --all-namespaces
```
You should see coredns pods come to life if all is well.
# Untaint the master so you can run pods
```
kubectl taint nodes --all node-role.kubernetes.io/master-
```
At this point you can run pods and expose them with services. If that's all you
need, you're done! Next I set up an nginx-ingress and cert-manager to allow for
hostname-based HTTPS ingress with Let's Encrypt certificates.
# Set up nginx-ingress
I set up the nginx-ingress with host networking so I could expose my cluster's
services via ports 80 and 443 on the host. You can also expose it via a NodePort or
LoadBalancer, but this worked well for my simple setup.
```
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
cat > nginx-host-networking.yaml <<EOF
spec:
template:
spec:
hostNetwork: true
EOF
kubectl -n ingress-nginx patch deployment nginx-ingress-controller --patch="$(<nginx-host-networking.yaml)"
```
You can now create Ingresses to make your services accessible externally. If
you expose them via HTTP they should just work. HTTPS will work too, but with a
self-signed certificate.
# Set up cert-manager
This is a bit fiddly, which isn't helped by the fact cert-manager's
documentation is a bit out of date. Luckily, I read the source so you don't
have to!
The cert-manager tool is pretty neat. It lets you provision Let's Encrypt
certificates either manually (useful for a fallback wildcard certificate) or
automatically based on annotation on your Ingress.
I have overridden the nameservers used for verifying dns01 records because I
use split-horizon DNS.
```
kubectl create ns cert-manager
kubectl config set-context --current --namespace=cert-manager
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/cert-manager.yaml
cat > external-dns.yaml <<EOF
# Add dns01 recursive nameservers
spec:
template:
spec:
containers:
- name: cert-manager
args:
- --cluster-resource-namespace=\$(POD_NAMESPACE)
- --leader-election-namespace=\$(POD_NAMESPACE)
- --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
EOF
kubectl -n cert-manager patch deployment cert-manager --patch="$(<external-dns.yaml)"
```
# Set up a ClusterIssuer to issue certificates
You'll need your Cloudflare API key to put into a Kubernetes secret.
```
kubectl -n cert-manager create secret generic cloudflare-api-key --from-literal=api-key=XXXXX
cat > clusterissuer-cf.yaml <<EOF
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: you@example.com
privateKeySecretRef:
name: letsencrypt
dns01:
providers:
- name: cf-dns
cloudflare:
email: you@example.com
apiKeySecretRef:
name: cloudflare-api-key
key: api-key
EOF
kubectl apply -f clusterissuer-cf.yaml
```
# Create a wildcard certificate
I created a wildcard certificate in the ingress-nginx namespace so I can use it
as the default.
```
cat > wildcard-cert.yaml <<EOF
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: wildcard-example-com
namespace: ingress-nginx
spec:
secretName: wildcard-example-com
issuerRef:
kind: ClusterIssuer
name: letsencrypt
commonName: '*.example.com'
dnsNames:
- example.com
acme:
config:
- dns01:
provider: cf-dns
domains:
- '*.example.com'
- 'example.com'
EOF
kubectl apply -f wildcard-cert.yaml
```
You can watch the progress of the certificate generation. Tab completion should find the names of your orders and challenges for you.
```
kubectl describe order ...
kubectl describe challenge ...
```
Set this as the default certificate for the ingress controller as follows:
```
kubectl -n ingress-nginx edit deployment nginx-ingress-controller
```
Add:
```
- --default-ssl-certificate=$(POD_NAMESPACE)/wildcard-example-com
```
to the list of args to the `nginx-ingress-controller` container. You can do
this with a patch if you prefer.
# Try it out!
You should have DNS set up for, in this example, `*.example.com` pointing to you Kubernetes host.
Deploy a simple nginx service to test things out.
```
kubectl create ns testland
kubectl config set-context --current --namespace=testland
kubectl create deployment nginx --image nginx
kubectl expose deployment nginx --port 80
cat > nginx-ingress.yaml <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
spec:
rules:
- host: nginx.example.com
http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
tls:
- hosts:
- nginx.example.com
EOF
kubectl apply -f nginx-ingress.yaml
```
You should be able to visit http://nginx.example.com/ in your browser and get redirected to HTTPS with a real, live certificate.
I'll run through the process again to check for any errors and update this post.