Restart IB Gateway before each job when it has lost IBKR's servers

On 2026-09-29 Gateway lost its connection, failed the reconnect's SSL
handshake and sat on an error dialog for 9 hours with its API port open:
clients connected, every startup request timed out, and systemd never
restarted it because java kept running.

- Broker connects with raiseSyncErrors, so a failed startup sync exits
  instead of continuing with an empty portfolio(), which rebalance would
  read as holding nothing.
- `ibkr-quant gateway` probes the Gateway under its own health_client_id.
- deploy/ensure-gateway.sh (`make gateway`) probes, waits a minute for a
  self-reconnect, then restarts ibgateway.service and waits for it.
- The trade and fills units run `make gateway` as ExecStartPre; fills
  now retries on failure like trade does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-29 16:52:23 +08:00
parent 358cbec9f9
commit dd7dbf8a18
8 changed files with 68 additions and 5 deletions

View File

@@ -1,8 +1,9 @@
[Unit]
Description=ibkr-quant daily rebalance: today's bars, then MOC orders
After=ibgateway.service
# Retries cover today's bar arriving late (delayed data) and Gateway restarts;
# make submit refuses to double-submit while orders are open.
# make gateway restarts a Gateway that has lost IBKR. Retries cover today's bar
# arriving late (delayed data) and Gateway restarts; make submit refuses to
# double-submit while orders are open.
StartLimitIntervalSec=7200
StartLimitBurst=6
@@ -10,6 +11,7 @@ StartLimitBurst=6
Type=oneshot
WorkingDirectory=%h/projects/ibkr-quant
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStartPre=/usr/bin/make gateway
ExecStart=/usr/bin/make bars submit
Restart=on-failure
RestartSec=300