Restart IB Gateway before each job when it has lost IBKR's servers

On 2026-09-29 Gateway lost its connection, failed the reconnect's SSL
handshake and sat on an error dialog for 9 hours with its API port open:
clients connected, every startup request timed out, and systemd never
restarted it because java kept running.

- Broker connects with raiseSyncErrors, so a failed startup sync exits
  instead of continuing with an empty portfolio(), which rebalance would
  read as holding nothing.
- `ibkr-quant gateway` probes the Gateway under its own health_client_id.
- deploy/ensure-gateway.sh (`make gateway`) probes, waits a minute for a
  self-reconnect, then restarts ibgateway.service and waits for it.
- The trade and fills units run `make gateway` as ExecStartPre; fills
  now retries on failure like trade does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-29 16:52:23 +08:00
parent 358cbec9f9
commit dd7dbf8a18
8 changed files with 68 additions and 5 deletions

View File

@@ -6,7 +6,7 @@
# open, so start after 09:47 ET.
RUN = uv run ibkr-quant
.PHONY: test universe check bars plan submit fills backtest install-units
.PHONY: test universe check gateway bars plan submit fills backtest install-units
test:
uv run pytest -q
@@ -17,6 +17,10 @@ universe:
check:
$(RUN) check
# On the server: probe the Gateway and restart it if it has lost IBKR's servers.
gateway:
sh deploy/ensure-gateway.sh
bars:
$(RUN) bars