147 lines
4.8 KiB
Markdown
147 lines
4.8 KiB
Markdown
# A simple Kubernetes load balancer
|
|
|
|
Configures nginx to forward connections to your node IPs.
|
|
Services should be declared as NodePort, which means that they
|
|
open a port on all nodes. When the request lands on any node,
|
|
it is forwarded to the correct pod via the network mesh kubernetes
|
|
is using. In theory, there is one a hop penalty.
|
|
|
|
But lets be honest. You're running with a single LB, probably on a
|
|
GCE free tier N1 VM. That extra hop doesn't matter.
|
|
|
|
## Config
|
|
|
|
Configure nginx to do what you want, test it. Use any Node IP for your testing.
|
|
This will become the 'template_dir' in the argument to the LB.
|
|
|
|
Move that directory (i.e., `/etc/nginx`) to somewhere new,
|
|
(i.e. `/etc/nginx-template/`).
|
|
|
|
Make a workspace directory for this tool; it will write configs to this folder
|
|
before updating the symlink you created above. It needs to be persistent so on
|
|
server reboot the service starts ok (i.e., `mkdir /var/skubelb/`).
|
|
|
|
Create a symlink in the workspace which will point to the 'active' configuration;
|
|
this will be updated by the tool (i.e., `ln -s /etc/nginx-template /var/skubelb/nginx`).
|
|
Make a symlink from the old config directory to that symlink (i.e.,
|
|
`ln -s /var/skubelb/nginx /etc/nginx`). Two layers are symlinks are used so we can
|
|
have a non-root user run the rool when we setup the service.
|
|
|
|
Make sure the user running the tool has read access to the template folder, read-write
|
|
access to the workspace folder and config symlink.
|
|
|
|
Run the server with a command like:
|
|
|
|
```sh
|
|
skubelb --needle some_node_ip \
|
|
--workspace_dir /var/skubelb \
|
|
--config_symlink /var/skubelb/nginx \
|
|
--template_dir /etc/nginx-template
|
|
--listen 0.0.0.0:8888
|
|
```
|
|
|
|
Replacing `some_node_ip` with the node IP you used during the initial setup.
|
|
|
|
Next, configure the Kubernetes nodes to POST `http://loadbalancer:8888/register/${NODE_IP}` when
|
|
they started, and DELETE `http://loadbalancer:8888/register/${NODE_IP}` when they shutdown. The easiest
|
|
way to do this is with a daemonset; see the example below, or in daemon_set.yaml.
|
|
|
|
#### Running as a system service
|
|
|
|
Setup a user to run the service; make that user
|
|
with `useradd -M skubelb`, Prevent logins with `usermod -L skubelb`.
|
|
|
|
Make a workspace dir, `mkdir /var/skubelb/`, and give access to the
|
|
daemon user, `chown skubelb:skubelb /var/skubelb/`.
|
|
|
|
Add the systemd config to `/etc/systemd/system/skubelb.service`:
|
|
|
|
```toml
|
|
[Unit]
|
|
Description=Simple Kubernetes Load Balancer
|
|
After=network.target
|
|
StartLimitIntervalSec=0
|
|
|
|
[Service]
|
|
Type=simple
|
|
Restart=always
|
|
RestartSec=1
|
|
User=skubelb
|
|
ExecStart=/usr/local/bin/skubelb --needle some_node_ip \
|
|
--workspace-dir /var/skubelb \
|
|
--config-symlink /var/skubelb/nginx \
|
|
--template-dir /etc/nginx-template \
|
|
--listen 0.0.0.0:8888 \
|
|
--reload-cmd '/usr/bin/sudo systemctl reload nginx'
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
Make sure you update `--needle some_node_ip` with something
|
|
like `--needle 123.44.55.123`. The IP of node you tested with.
|
|
|
|
### Sample Kubernets configuration
|
|
|
|
Deploy this [daemon set](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)
|
|
to your cluster, replacing `lb_address` with the address of your load balancer.
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: skubelb
|
|
namespace: skubelb
|
|
labels:
|
|
k8s-app: skubelb
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
name: skubelb
|
|
template:
|
|
metadata:
|
|
labels:
|
|
name: skubelb
|
|
spec:
|
|
tolerations:
|
|
# these tolerations are to have the daemonset runnable on control plane nodes
|
|
# remove them if your control plane nodes should not run pods
|
|
- key: node-role.kubernetes.io/control-plane
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
- key: node-role.kubernetes.io/master
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
containers:
|
|
- name: skubelb
|
|
image: alpine/curl:latest
|
|
env:
|
|
- name: NODE_IP
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: status.hostIP
|
|
command: ['sh', '-c', 'echo "Wait for heat death of universe" && sleep 999999d']
|
|
lifecycle:
|
|
preStop:
|
|
exec:
|
|
command:
|
|
- sh
|
|
- "-c"
|
|
- "curl -X POST 10.128.0.2:8888/register/${NODE_IP}"
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- sh
|
|
- "-c"
|
|
- "curl -X POST 10.128.0.2:8888/register/${NODE_IP}"
|
|
resources:
|
|
limits:
|
|
memory: 200Mi
|
|
requests:
|
|
cpu: 10m
|
|
memory: 100Mi
|
|
terminationGracePeriodSeconds: 30
|
|
```
|
|
|
|
NOTE: you should need to make an entry in the firewall to allow this request through. It is very important that the firewall entry has a source filter; it should only be allowed from the Kubernetes cluster. Nginx will forward traffic to any host that registers, and this could easily become a MitM vulnerability.
|