2 Commits

Author SHA1 Message Date
e55c4ffbdb Restart the Gateway when its historical data farm stops serving
On 2026-10-12 the Gateway answered account requests but every bar request
timed out from 09:50 to 10:24 ET: make gateway reported OK, so the Gateway was
never restarted, and six retries 5 minutes apart gave up after 35 minutes. The
data came back only with the Gateway's daily restart.

ibkr-quant gateway --history also requires SPY daily bars within 30 s; the
trade job's ExecStartPre passes it, so ensure-gateway.sh restarts a Gateway
whose data farm is down. The trade job now retries every 10 minutes until
about 15:30 ET (34 starts in 6 h) instead of giving up after 35 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:27:16 -07:00
dd7dbf8a18 Restart IB Gateway before each job when it has lost IBKR's servers
On 2026-09-29 Gateway lost its connection, failed the reconnect's SSL
handshake and sat on an error dialog for 9 hours with its API port open:
clients connected, every startup request timed out, and systemd never
restarted it because java kept running.

- Broker connects with raiseSyncErrors, so a failed startup sync exits
  instead of continuing with an empty portfolio(), which rebalance would
  read as holding nothing.
- `ibkr-quant gateway` probes the Gateway under its own health_client_id.
- deploy/ensure-gateway.sh (`make gateway`) probes, waits a minute for a
  self-reconnect, then restarts ibgateway.service and waits for it.
- The trade and fills units run `make gateway` as ExecStartPre; fills
  now retries on failure like trade does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 16:52:23 +08:00