Load balancers

What a balancer is

A load balancer is one of your servers, given the balancer role when you create it, running Nginx in front of a pool of your other servers. It owns the domains, holds the certificate, and forwards every request to one of the backends over a private network the team owns. That network is a WireGuard mesh, so a packet reaches a backend only after it has been decrypted under a known peer's key.

A balancer hosts no sites of its own. Site creation on it is refused. It runs Nginx and holds certificates, so it passes every web-stack check, but it serves nothing except the pool.

A backend is a server, not an address

You add a backend by choosing a server and, optionally, a weight. There is no address field. Provisore reads the server's address from its membership of the balancer's private network, and the file the balancer renders cannot express a host, a scheme, or a second port. A backend is on the mesh by construction.

The API refuses these by name:

• The balancer's own server, which cannot forward to itself.
• A server provisioned without Nginx, which serves no HTTP for the balancer to forward to.
• A server that is not a member of the balancer's private network, or whose membership has not finished converging. Wait for the network, then add it.
• A server already in the pool, and any pool that would exceed 32 backends.

Pool settings are one value each, for the whole pool: the backend port (80 by default), the algorithm, how many failed forwards in a row take a backend out of rotation and for how long, the health check path, and the drain window. Weights are 1 to 100 per backend.

Domains          shop.example.com, www.shop.example.com   # first one names the certificate
Algorithm        round_robin | least_connections | ip_hash
Backend port     80
Health path      /
Drain window     30s                                      # 0–600
Backends         up to 32, weight 1–100 each

TLS terminates at the balancer

Once a domain's DNS points at the balancer, an HTTP-01 challenge for that domain arrives at the balancer and is forwarded to whichever backend the pool picks, while certbot wrote the token on the one backend it ran on. One request in N validates, and which one changes on every retry.

So per-backend certificates cannot be issued or renewed behind a pool. Issuance appears to work often enough to look flaky, and renewal stops working weeks later. The certificate belongs to the balancer, is issued there, and renews there. The challenge is answered from the balancer's own directory and never proxied, and the force-HTTPS redirect is written so that it does not catch the challenge on its way past.

That leaves one balancer per set of domains. Two balancers serving the same names does not work, because an HTTP-01 challenge lands on one of them and the token was written on the other. Escaping that needs DNS-01 validation for a balancer, which is refused today and is not built.

Removing a backend drains it

Removing a backend does not delete it straight away. It stays in the configuration marked down, so Nginx stops choosing it at the reload and the requests already on it finish. A reload never cuts a connection in flight. The row is dropped once the drain window has passed, by a scheduled apply. If the balancer cannot be reached at that moment the backend stays down, taking no traffic, and is tried again later.

A backend can also be disabled, which is the same "in the file, marked down" state with no deadline. Neither state can be changed on a backend that is mid-drain or mid-deploy. Enabling a deploying backend would put a half-swapped release in front of traffic, and disabling a draining one would mean the removal never finishes.

A deploying backend leaves the pool

When you deploy to a server that is in a pool, its backends are marked deploying before the deploy starts and returned to the pool only after the deployment succeeds. A deploy that fails its own health check leaves its backend out of rotation.

None of that can fail a deploy. If a balancer cannot be reached, the deploy log says the server stayed in its pools and may answer errors while the release is swapped, and the deploy runs anyway. Refusing to deploy because one balancer was unreachable would turn one broken balancer into a fleet-wide freeze.

The origins stop answering directly

Once every live site on a backend is fronted by a balancer, that server's Nginx binds its private-network address and its public ports 80 and 443 close. Otherwise the origin would be a second front door: the same site, reachable at the server's public IP, on a certificate the balancer now owns and the origin no longer renews, bypassing the pool and ignoring what the balancer says about draining or deploying.

A server with one fronted site and one direct site keeps its public ports, because closing them would take the direct site down. Your applications still see the real client. The balancer sets the forwarded headers, and the origin trusts them from the balancer's private address and nowhere else.

What the health column means

Two different things keep a broken backend from taking traffic, and only one of them is a number on a page. Nginx's own passive detection is the mechanism. It acts inside the request path, on the connection that just failed, with no probe interval to wait out. What Provisore adds is a report: one request per backend from the balancer, run at the end of every apply and again every five minutes, so the app can say which backend is down.

A reading has an age. Past fifteen minutes it is shown as unknown rather than carried forward, in both directions. An hours-old failure is no evidence that a backend is broken now, and an hours-old success is no evidence that it is fine. A balancer Provisore could not reach records nothing, and the app reports that as a different fact from an unhealthy backend.

Limits

• One balancer per set of domains. No second balancer for the same names, and therefore no highly-available pair. DNS-01 is refused.
• The pool speaks plain HTTP inside a WireGuard tunnel, which is what encrypts and authenticates it. TLS in there would mean issuing certificates for private addresses.
• IPv4 backends only, and one backend port for the whole pool.
• ip_hash is not session storage. The mapping is over the live pool, so one backend failing rehashes every client.
• Upload size at the balancer is fixed at 100 MB and is not configurable per balancer today.
• WebSockets pass through with the upgrade headers set, but a long-lived WebSocket through a balancer has not been exercised, and the balancer hop keeps Nginx's default read timeout where a site's own vhost sets a long one.
• No CLI commands, and weight and enable/disable are API-only. Everything else is on the Load balancers page.