Host a dedicated server

w4-dedicated runs one match of the W4 simulation for other people. The server decides everything that happens; players send orders and get back only what they may see. It lists itself on the Games page through the online service, so people find it in the browser and the engine.

This page is for the person who runs a server. Players read Online play and You can always come back.

w4-dedicated --help (or -h) prints every flag with its default; the table below is the same list, and a test keeps the help in step with the argument parser, so a flag that is not listed does not exist. A flag with a wrong value stops the program with a one-line message.

The smallest server

w4-dedicated --map battle.w4m --listen 0.0.0.0:7777 --address example.org:7777

This listens on every address, accepts any name as a player (development mode) and is not listed anywhere. A development server only accepts connections from the same computer; to let others in you must sign players in through an online service with --online.

A public server looks more like this:

w4-dedicated --map battle.w4m --listen 0.0.0.0:7777 --address example.org:7777 \
  --title "Official: Battle" --online http://127.0.0.1:5179 \
  --tls-cert fullchain.pem --tls-key privkey.pem \
  --checkpoint-dir /var/w4/checkpoints --metrics-listen 127.0.0.1:7790 \
  --drain-file /var/w4/dedicated.drain --allow-guests

The host key comes from the environment variable AQUA4_HOST_KEY here (or --host-key KEY). Prefer the variable on a shared machine: a flag can be read by other users with ps.

Every flag

The map and the address

Who may play

Security of the connection

Keeping the match alive

Watching

Records and observation

Watching in the browser

With --ws-listen 7791 the server also accepts WebSocket connections on /watch. A browser sends one hello and then receives the same feed an engine spectator receives: one JSON message per text frame, five pictures a second, delayed by --spectator-delay-secs. Everything else is the normal spectator code: sign-in or guest rules, the --max-spectators limit, at most 4 viewers per address, the observer view.

The gateway speaks plain ws:// and should stay on loopback. Put your proxy in front for wss://; with Caddy, on the site's domain:

handle /watch* {
	reverse_proxy 127.0.0.1:7791
}

Trying it on your own computer. The Watch a game page takes the gateway's address from the page address: /spectate?ws=ws://localhost:7791/watch. The live website's security policy only lets a page open its own secure socket (wss://w4.nebel.sh), so the override works on a site you run yourself: bun run dev in site/ allows ws://localhost and ws://127.0.0.1, and so does a build made with AQUA4_CSP_LOCAL_WS=1 bun run build (for a local copy only; never deploy it). The normal production build never allows plain ws:// to a local address.

The gateway reads the viewer's address from the last X-Forwarded-For entry, but only when the connection comes from the same computer (the proxy); a stranger cannot choose its address. Without --online only local addresses may watch. A browser may send only the hello and ping; binary frames, fragments and unmasked frames close the connection. Each viewer costs the bandwidth of the feed (about 100 bytes a unit, five times a second), so size --max-spectators for your uplink.

Restarting without ruining a match

Two things protect a running match, and a good server uses both.

Checkpoints (--checkpoint-dir) make a crash survivable: after a restart the match continues from the last checkpoint, up to ten seconds of game time behind, and everybody who was in it joins again with their usual sign-in. The official server does this.

Draining (--drain-file) makes a planned restart polite. While the file exists, the server:

1. takes its game off the list,
2. refuses everybody who has no seat ("the server is restarting"),
3. lets players who hold a seat come back,
4. exits when the match is over and nobody is left, or at once if nobody ever played.

If the match is still going after an hour, the server writes a final checkpoint and exits anyway, so the match continues in the new process.

To update a server: build the new program, touch the drain file, wait for the old process to exit, and let the service manager start the new one (a stale drain file must be deleted at start; the example below does it). The file is plain on purpose: it works on Windows and on FreeBSD the same way, with no signals.

Run it as a service

systemd (/etc/systemd/system/w4-dedicated.service):

[Unit]
Description=W4 dedicated server
After=network-online.target

[Service]
User=w4
EnvironmentFile=/etc/w4/dedicated.env      # AQUA4_HOST_KEY=...
ExecStartPre=/bin/rm -f /var/lib/w4/dedicated.drain
ExecStart=/opt/w4/w4-dedicated --map /var/lib/w4/maps/battle.w4m \
  --listen 0.0.0.0:7777 --address example.org:7777 --online http://127.0.0.1:5179 \
  --checkpoint-dir /var/lib/w4/checkpoints --metrics-listen 127.0.0.1:7790 \
  --drain-file /var/lib/w4/dedicated.drain
Restart=always
RestartSec=2
LimitNOFILE=4096

[Install]
WantedBy=multi-user.target

FreeBSD (what the official server uses): a small run-dedicated.sh that deletes a stale drain file, creates the checkpoint folder and ends with exec w4-dedicated ... (it reads the host key from a file and passes it with --host-key), started by an rc service made with the spawn tool (daemon(8) restarts it if it exits). The shipped example is server/dedicated/run-dedicated.sh; freebsd-drain.sh touches the drain file and waits for the new process.

Stop the service with SIGTERM (not SIGKILL): it writes a last checkpoint and tells the players.

Metrics, health and the time series

With --metrics-listen 127.0.0.1:7790 the server answers plain GET requests on that address only:

With the server's host key in the header X-W4-Host-Key (the key configured with --host-key; without a key configured these paths do not exist, a wrong or missing key is 401) the same address also serves the operator's room commands:

A command is answered 202 (queued; the room applies it on its next tick), 400 for a bad request, 404 for an unknown room and 429 when 16 commands already wait. Every applied command leaves a log line. Only the rooms of a multi-room server (--rooms) take these. Keep the metrics address on loopback.

curl -X POST -H "X-W4-Host-Key: $KEY" "http://127.0.0.1:7790/admin/room/r1/end"

The website's Server status page draws the last day from /series. Run the site on the same computer and set AQUA4_SERIES_URL=http://127.0.0.1:7790/series for it; without the variable the page says the series is not connected. A target worth watching: 99% of steps under 33.3 ms.

The online service has its own GET /v1/metrics, for administrators and for a caller on the same computer, and /v1/healthz and /v1/readyz.

Backups and restore

The server itself keeps nothing that must be backed up except the checkpoint folder (and only while a match is running). Accounts, games and results live in the online service's database, which writes a whole backup every six hours and keeps the newest eight (backups/w4-<time>.db in its data folder).

To restore: stop the service (it drains first), move the live w4.db, w4.db-wal and w4.db-shm aside, copy the backup in as w4.db, start the service, and check /v1/readyz and a sign-in. Never copy a backup over a running database. To rehearse without risk, copy a backup to a scratch folder and start a second service on another port against it.

Crashes

A panic anywhere in the server writes w4-dedicated-crash-<time>.txt (message, thread, backtrace) next to the program. A panic while the game is stepping does not end the process: the game is stopped, its players are told "the game hit a server error and was stopped", and a fresh game starts over (stderr shows PANIC in ...). A panic elsewhere stops the process; the service manager starts it again, and with checkpoints on the match goes on. Delete old crash files by hand.