skubelb/README.md

# A simple Kubernetes load balancer

Configures nginx to forward connections to your node IPs.
Services should be declared as NodePort, which means that they
open a port on all nodes. When the request lands on any node,
it is forwarded to the correct pod via the network mesh kubernetes
is using. In theory, there is one a hop penalty.

But lets be honest. You're running with a single LB, probably on a
GCE free tier N1 VM. That extra hop doesn't matter.

## Config

Configure nginx to do what you want, test it. Use any Node IP for your testing.
This will become the 'template_dir' in the argument to the LB.

Move that directory (i.e., `/etc/nginx`) to somewhere new,
(i.e. `/etc/nginx-template/`).

Make a workspace directory for this tool; it will write configs to this folder
before updating the symlink you created above. It needs to be persistent so on
server reboot the service starts ok (i.e., `mkdir /var/skubelb/`).

Create a symlink in the workspace which will point to the 'active' configuration;
this will be updated by the tool (i.e., `ln -s /etc/nginx-template /var/skubelb/nginx`).
Make a symlink from the old config directory to that symlink (i.e., 
`ln -s /var/skubelb/nginx /etc/nginx`). Two layers are symlinks are used so we can
have a non-root user run the rool when we setup the service.

Make sure the user running the tool has read access to the template folder, read-write
access to the workspace folder and config symlink.

Run the server with a command like:

```sh
skubelb --needle some_node_ip \
    --workspace_dir /var/skubelb \
    --config_symlink /var/skubelb/nginx \
    --template_dir /etc/nginx-template
    --listen 0.0.0.0:8888
```

Replacing `some_node_ip` with the node IP you used during the initial setup.

Next, configure the Kubernetes nodes to POST `http://loadbalancer:8888/register/${NODE_IP}` when
they started, and DELETE `http://loadbalancer:8888/register/${NODE_IP}` when they shutdown. The easiest
way to do this is with a daemonset; see the example below, or in daemon_set.yaml.

#### Running as a system service

Setup a user to run the service; make that user
with `useradd -M skubelb`, Prevent logins with `usermod -L skubelb`.

Make a workspace dir, `mkdir /var/skubelb/`, and give access to the
daemon user, `chown skubelb:skubelb /var/skubelb/`.

Add the systemd config to `/etc/systemd/system/skubelb.service`:

```toml
[Unit]
Description=Simple Kubernetes Load Balancer
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=skubelb
ExecStart=/usr/local/bin/skubelb --needle some_node_ip \
    --workspace-dir /var/skubelb \
    --config-symlink /var/skubelb/nginx \
    --template-dir /etc/nginx-template \
    --listen 0.0.0.0:8888 \
    --reload-cmd '/usr/bin/sudo systemctl reload nginx'

[Install]
WantedBy=multi-user.target
```

Make sure you update `--needle some_node_ip` with something
like `--needle 123.44.55.123`. The IP of node you tested with.

### Sample Kubernets configuration

Deploy this [daemon set](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)
to your cluster, replacing `lb_address` with the address of your load balancer.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: skubelb
  namespace: skubelb
  labels:
    k8s-app: skubelb
spec:
  selector:
    matchLabels:
      name: skubelb
  template:
    metadata:
      labels:
        name: skubelb
    spec:
      tolerations:
      # these tolerations are to have the daemonset runnable on control plane nodes
      # remove them if your control plane nodes should not run pods
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: skubelb
        image: alpine/curl:latest
        env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        command: ['sh', '-c', 'echo "Wait for heat death of universe" && sleep 999999d']
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - "-c"
              - "curl -X POST 10.128.0.2:8888/register/${NODE_IP}"
          postStart:
            exec:
              command:
              - sh
              - "-c"
              - "curl -X POST 10.128.0.2:8888/register/${NODE_IP}"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 10m
            memory: 100Mi
      terminationGracePeriodSeconds: 30
```

NOTE: you should need to make an entry in the firewall to allow this request through. It is very important that the firewall entry has a source filter; it should only be allowed from the Kubernetes cluster. Nginx will forward traffic to any host that registers, and this could easily become a MitM vulnerability.

## Other tips

### Use 'upstream' in nginx

Do this:

```
upstream hosts {
        server 10.182.0.36:30004;
        server 10.182.0.39:30004;
}
server {
        server_name git.tipsy.codes tipsy.codes;

        location / {
                proxy_pass http://hosts;
        }
}
```

Rather than just writing out the IP in the proxy_pass.

### visudo to only allow the nginx reload command

Use `sudo visudo` to update the sudoers file and add this line:

```
skubelb ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
```

This will prevent the user from running commands other than reload.