Configures nginx to forward connections to your node IPs. Services should be declared as NodePort, which means that they open a port on all nodes. When the request lands on any node, it is forwarded to the correct pod via the network mesh kubernetes is using. In theory, there is one a hop penalty.

But lets be honest. You're running with a single LB, probably on a GCE free tier N1 VM. That extra hop doesn't matter.

Config

Configure nginx to do what you want, test it. Use any Node IP for your testing. This will become the 'template_dir' in the argument to the LB.

Move that directory (i.e., /etc/nginx) to somewhere new, (i.e. /etc/nginx-template/).

Make a workspace directory for this tool; it will write configs to this folder before updating the symlink you created above. It needs to be persistent so on server reboot the service starts ok (i.e., mkdir /var/skubelb/).

Create a symlink in the workspace which will point to the 'active' configuration; this will be updated by the tool (i.e., ln -s /etc/nginx-template /var/skubelb/nginx). Make a symlink from the old config directory to that symlink (i.e., ln -s /var/skubelb/nginx /etc/nginx). Two layers are symlinks are used so we can have a non-root user run the rool when we setup the service.

Make sure the user running the tool has read access to the template folder, read-write access to the workspace folder and config symlink.

Run the server with a command like:

skubelb --needle some_node_ip \
    --workspace_dir /var/skubelb \
    --config_symlink /var/skubelb/nginx \
    --template_dir /etc/nginx-template
    --listen 0.0.0.0:8888

Replacing some_node_ip with the node IP you used during the initial setup.

Next, configure the Kubernetes nodes to POST http://loadbalancer:8888/register/${NODE_IP} when they started, and DELETE http://loadbalancer:8888/register/${NODE_IP} when they shutdown. The easiest way to do this is with a daemonset; see the example below, or in daemon_set.yaml.

Running as a system service

Setup a user to run the service; make that user with useradd -M skubelb, Prevent logins with usermod -L skubelb.

Make a workspace dir, mkdir /var/skubelb/, and give access to the daemon user, chown skubelb:skubelb /var/skubelb/.

Add the systemd config to /etc/systemd/system/skubelb.service:

[Unit]
Description=Simple Kubernetes Load Balancer
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=skubelb
ExecStart=/usr/local/bin/skubelb --needle some_node_ip \
    --workspace-dir /var/skubelb \
    --config-symlink /var/skubelb/nginx \
    --template-dir /etc/nginx-template \
    --listen 0.0.0.0:8888 \
    --reload-cmd '/usr/bin/sudo systemctl reload nginx'

[Install]
WantedBy=multi-user.target

Make sure you update --needle some_node_ip with something like --needle 123.44.55.123. The IP of node you tested with.

Sample Kubernets configuration

Deploy this daemon set to your cluster, replacing lb_address with the address of your load balancer.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: skubelb
  namespace: skubelb
  labels:
    k8s-app: skubelb
spec:
  selector:
    matchLabels:
      name: skubelb
  template:
    metadata:
      labels:
        name: skubelb
    spec:
      tolerations:
      # these tolerations are to have the daemonset runnable on control plane nodes
      # remove them if your control plane nodes should not run pods
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: skubelb
        image: alpine/curl:latest
        env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        command: ['sh', '-c', 'echo "Wait for heat death of universe" && sleep 999999d']
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - "-c"
              - "curl -X POST 10.128.0.2:8888/register/${NODE_IP}"
          postStart:
            exec:
              command:
              - sh
              - "-c"
              - "curl -X POST 10.128.0.2:8888/register/${NODE_IP}"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 10m
            memory: 100Mi
      terminationGracePeriodSeconds: 30

NOTE: you should need to make an entry in the firewall to allow this request through. It is very important that the firewall entry has a source filter; it should only be allowed from the Kubernetes cluster. Nginx will forward traffic to any host that registers, and this could easily become a MitM vulnerability.

Other tips

Use 'upstream' in nginx

Do this:

upstream hosts {
        server 10.182.0.36:30004;
        server 10.182.0.39:30004;
}
server {
        server_name git.tipsy.codes tipsy.codes;

        location / {
                proxy_pass http://hosts;
        }
}

Rather than just writing out the IP in the proxy_pass.

visudo to only allow the nginx reload command

Use sudo visudo to update the sudoers file and add this line:

skubelb ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx

This will prevent the user from running commands other than reload.