Restart the Gateway when its historical data farm stops serving

On 2026-10-12 the Gateway answered account requests but every bar request
timed out from 09:50 to 10:24 ET: make gateway reported OK, so the Gateway was
never restarted, and six retries 5 minutes apart gave up after 35 minutes. The
data came back only with the Gateway's daily restart.

ibkr-quant gateway --history also requires SPY daily bars within 30 s; the
trade job's ExecStartPre passes it, so ensure-gateway.sh restarts a Gateway
whose data farm is down. The trade job now retries every 10 minutes until
about 15:30 ET (34 starts in 6 h) instead of giving up after 35 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-12 23:27:16 -07:00
parent 8238651af6
commit e55c4ffbdb
5 changed files with 53 additions and 13 deletions

View File

@@ -1,17 +1,20 @@
[Unit]
Description=ibkr-quant daily run: today's bars, then MOC orders on rebalance sessions
After=ibgateway.service
# make gateway restarts a Gateway that has lost IBKR. Retries cover today's bar
# arriving late (delayed data) and Gateway restarts; make submit refuses to
# double-submit while orders are open.
StartLimitIntervalSec=7200
StartLimitBurst=6
# make gateway restarts a Gateway that has lost IBKR or its historical data
# farm. Retries cover today's bar arriving late (delayed data) and Gateway
# restarts; make submit refuses to double-submit while orders are open.
# Every 10 minutes from 09:50 until about 15:30 ET (34 starts), inside the
# 09:31-15:45 order window: on 2026-10-12 six retries 5 minutes apart gave up
# after 35 minutes while the data farm was down.
StartLimitIntervalSec=21600
StartLimitBurst=34
[Service]
Type=oneshot
WorkingDirectory=%h/projects/ibkr-quant
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStartPre=/usr/bin/make gateway
ExecStartPre=/usr/bin/make gateway GATEWAY_ARGS=--history
ExecStart=/usr/bin/make bars submit
Restart=on-failure
RestartSec=300
RestartSec=600