80 lines
5.2 KiB
Markdown
80 lines
5.2 KiB
Markdown
# Kubernetes-Home-Lab
|
|
|
|
A repository for the Ansible playbooks used in my Youtube video series. In this series I go over configuring my Raspberry Pi 4 cluster as a Kubernetes cluster using Kubeadm. I also automate the process as much as I can using these Ansible playbooks.
|
|
|
|
Each playbook role comes with a README file outling the manual commands as well if Ansible is not somethign the user wishes to use for this proces .
|
|
|
|
Link to the youtube series:
|
|
- Part 1: [Design, Hardware, and Installing the OS](https://www.youtube.com/watch?v=s017P0ns-YY&t=439s)
|
|
- Part 2: [Configuring Bare Metal Hosts](https://www.youtube.com/watch?v=sDSuAPoM5iQ&t=505s)
|
|
- Part 3: [Configuring Pre-requisites to run Kubeadm](https://www.youtube.com/watch?v=BvMEXcJe-bs)
|
|
- Part 3a: [My Thought Process When Building Palybooks](https://www.youtube.com/watch?v=gO8OMoW5VLo&t=2027s)
|
|
- Part 4: [Deploying an NGINX Load Balancer with Ansible](https://www.youtube.com/watch?v=4W8cwgPJKrw&t=222s)
|
|
- Part 5: [Creating Encryption Secrets for Kubernetes Cluster with Ansible](https://www.youtube.com/watch?v=DkkJviaWklY&t=162s)
|
|
- Part 6: [Creating a CA and certificates for Kubernetes Cluster with Ansible](https://www.youtube.com/watch?v=l-gOIylwsWo)
|
|
- Part 7: [Deploying Kubeadm and Joining Nodes](https://www.youtube.com/watch?v=qjJOABXe1JA&t=2s)
|
|
- Part 8: [Creating an NFS server for persistent Storage](https://www.youtube.com/watch?v=L97Z5In3KXQ)
|
|
|
|
|
|
## Architecure and Versions
|
|
|
|
- Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1029-raspi aarch64)
|
|
- Kubeadm, kubectl, kubelet v1.21.0
|
|
- containerd .io v1.4.4
|
|
- Flannel ( k8s networking solution)
|
|
- Openssl v1.1.1f
|
|
|
|
## Networking
|
|
-Cluster CIDR: IPs for pods - 10.240.0.0/16
|
|
-Worker Nodes Pod CIDR: Sepcific CIDR within the Cluster CIDR for one worker node. Multiple worker nodes pod CIDRs cannot overlap ( controlled by Flannel )
|
|
|
|
## Known Issues
|
|
|
|
###### Issue:
|
|
- Kubernetes-dashboard pods stuck in creating
|
|
###### Error:
|
|
- Warning FailedCreatePodSandBox 13s (x6 over 79s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1cce51808f870550bc01b3a8899e1fa13cad6372fa465a8e4c459a18ec8c51ea": failed to set bridge addr: "cni0" already has an IP address different from 10.240.1.1/24
|
|
###### Fix:
|
|
- This happens when you destroy flannel and deploy it again sometimes. Go to the nodes the kubernetes-dashboard hosts are hosted on and run `sudo ip link set cni0 down` and then `sudo brctl delbr cni0` and wait a few minutes.
|
|
|
|
###### Issue:
|
|
- Flannel containers CrashLoppBackoff
|
|
###### Error:
|
|
- Error registering network: failed to configure interface flannel.1: failed to ensure address of interface flannel.1: link has incompatible addresses. Remove additional addresses and try again.
|
|
###### Fix:
|
|
- This happens when you destroy flannel and deploy it again sometimes. On the nodes in question run `sudo ip link delete flannel.1` and then wait a few minutes for flannel pods to restart.
|
|
|
|
|
|
###### Issue:
|
|
- kubectl not working with error:
|
|
###### Error:
|
|
- The connection to the server localhost:8080 was refused - did you specify the right host or port?
|
|
###### Fix:
|
|
- This means the admin.conf for your cluster doesnt exist in that users $HOME/.kube directory. If Kubeadm did NOT finish the init this wont be available yet. If it did finish the init, simply place it in the directory `cp /etc/kubernetes/admin.conf $HOME/.kube/`
|
|
|
|
###### Issue:
|
|
- kubelet cannot connect to API server address:6443
|
|
###### Error:
|
|
- This shows up under `journalctl -u kubelet` or `systemctl status kubelet` as connection refused to the kube-api-server address on port 6443.
|
|
###### Fix:
|
|
- This one can happen for multiple reasons, but the best way to get the bottom of it is to first verify the kube-api container is healthy `crictl ps -a` will show you the running containers on the mastern node. If the kube-api one is in a state other than "running" you can do a `crictl logs < kube-api container ID >` to get more info about why the container is not creating. It may be due to a bad option or syntax passed into the config used in `kubeadm init --config`.
|
|
- It may also be due to firewall issues. Check to make sure port 6443 is accesible.
|
|
|
|
###### Issue:
|
|
- Nodes not ready when using `kubectl get nodes`
|
|
###### Error:
|
|
- Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
|
|
###### Fix:
|
|
- Did you install networking such as Netweave, flannel, calico, etc? If not this is needed.
|
|
- Did you delete /opt/cni/ and now its empty? You should see a "bin" directory. If you did delete it, then reinstall `kubernetes-cni`.
|
|
- Try and restart containerd as well then check for the cni0 interface.
|
|
|
|
## Troubleshooting
|
|
###### Get logs from a container using crictl
|
|
- `crictl ps -a` to list running containers and their ID
|
|
- `crictl logs < container ID >` to view logs for the container
|
|
- `kubectl get nodes` will list the nodes and their READY or NOT READY status in the cluster
|
|
- `kubectl describe node < node >` will show more information about the node and possible reveal any issues
|
|
- `kubectl get pods -n kube-system -o wide` A widely used command to display all pods in the kube-system namespace and their status
|
|
|