Restart IB Gateway before each job when it has lost IBKR's servers

On 2026-09-29 Gateway lost its connection, failed the reconnect's SSL
handshake and sat on an error dialog for 9 hours with its API port open:
clients connected, every startup request timed out, and systemd never
restarted it because java kept running.

- Broker connects with raiseSyncErrors, so a failed startup sync exits
  instead of continuing with an empty portfolio(), which rebalance would
  read as holding nothing.
- `ibkr-quant gateway` probes the Gateway under its own health_client_id.
- deploy/ensure-gateway.sh (`make gateway`) probes, waits a minute for a
  self-reconnect, then restarts ibgateway.service and waits for it.
- The trade and fills units run `make gateway` as ExecStartPre; fills
  now retries on failure like trade does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

deploy/ensure-gateway.sh

@@ -0,0 +1,25 @@
+#!/bin/sh
+# Probe IB Gateway and restart it when it stays unable to serve account requests.
+# Gateway can lose IBKR's servers, fail the reconnect and sit on an error dialog
+# with its API port still open (2026-09-29, for 9 hours): every request times
+# out, yet java keeps running, so systemd's Restart= never fires.
+# Runs before each job (ExecStartPre=make gateway) and by hand as `make gateway`.
+cd "$(dirname "$0")/.." || exit 1
+
+probe() { uv run ibkr-quant gateway; }
+
+probe && exit 0
+# Gateway rides out short outages by reconnecting on its own; give it a minute.
+sleep 60
+probe && exit 0
+
+echo "restarting ibgateway.service" >&2
+systemctl --user restart ibgateway.service || exit 1
+# IBC logged in 28 s after a restart on 2026-09-29; leave room for a slow proxy.
+deadline=$(($(date +%s) + 300))
+while [ "$(date +%s)" -lt "$deadline" ]; do
+    sleep 20
+    probe && exit 0
+done
+echo "IB Gateway still not serving requests 5 min after the restart" >&2
+exit 1

deploy/systemd/ibkr-quant-fills.service

@@ -1,9 +1,15 @@
 [Unit]
 Description=ibkr-quant: record today's executions
 After=ibgateway.service
+# Executions are only reported for today, so retry a failed Gateway for an hour.
+StartLimitIntervalSec=3600
+StartLimitBurst=4
 
 [Service]
 Type=oneshot
 WorkingDirectory=%h/projects/ibkr-quant
 Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
+ExecStartPre=/usr/bin/make gateway
 ExecStart=/usr/bin/make fills
+Restart=on-failure
+RestartSec=300

deploy/systemd/ibkr-quant-trade.service

@@ -1,8 +1,9 @@
 [Unit]
 Description=ibkr-quant daily rebalance: today's bars, then MOC orders
 After=ibgateway.service
-# Retries cover today's bar arriving late (delayed data) and Gateway restarts;
-# make submit refuses to double-submit while orders are open.
+# make gateway restarts a Gateway that has lost IBKR. Retries cover today's bar
+# arriving late (delayed data) and Gateway restarts; make submit refuses to
+# double-submit while orders are open.
 StartLimitIntervalSec=7200
 StartLimitBurst=6
 
@@ -10,6 +11,7 @@ StartLimitBurst=6
 Type=oneshot
 WorkingDirectory=%h/projects/ibkr-quant
 Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
+ExecStartPre=/usr/bin/make gateway
 ExecStart=/usr/bin/make bars submit
 Restart=on-failure
 RestartSec=300