Compare commits

..

13 Commits

Author SHA1 Message Date
5aca6b0509 gitignore: option-chain snapshots and broker state under data/
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 21:54:30 -07:00
3d69744068 research: lagged futures signals for stock exposure vs matched ETF rules (none passes, retain baseline)
128 fixed runs (32 profiles x daily ledger / overnight-only x 2 costs) plus 12
post-hoc availability diagnostics on overnight_recovery_top10, 2006-10 to 2026-09.
Futures trend/shock/rates rules do not beat the same rules on SPY/QQQ/IEF or a
static 75% exposure; all six default schemes fail the gates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 21:54:27 -07:00
50b4cb7678 research: 10-day momentum in a $10k/$20k IBKR account with whole shares and low-frequency risk controls vs QQQ (none passes)
Preregistered: rb20, SPY de-risk vol target, fundamental re-sort and half QQQ,
simulated with execution.compute_trades sizing, cash-limited buys, IBKR Tiered/
Fixed fees and constant account size. All fail the 2011-16 Sharpe gate;
rb20_fundmom beats QQQ on 2011-26 CAGR and MaxDD but has no 2008 test.
Also found: wrong-security series in the research panel (filtered by dollar
volume) and the lab engine's missing delisting settlement (~2.8 pts/yr).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 00:47:37 -07:00
7e9e49c71f research: no_letf_book notes - levered trend is even with QQQ outside crash years; the overnight sleeve beats QQQ when orders are cheap
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:46:31 -07:00
f2d8b53898 research: books without leveraged ETFs vs QQQ for the $10k IBKR account (none beats QQQ on the gates)
QQQ trend held in QQQ (no TQQQ), with Reg T margin above 1x, and its blends
with the momentum sleeve, at $1 an order on $10k, compared with QQQ raw and
at QQQ's design-period volatility. No construction raises Sharpe by 0.10 in
both periods. The trend's gain is crash avoidance (MaxDD -28% vs -83% since
1990); exploratory: at 2x margin it earns 17.0% vs 14.9% with MaxDD -50%.
The single-phase momentum series behind us_momentum_qqq_trend's 27.9% was 6pp
a year above the phase-averaged, delisting-aware stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:44:35 -07:00
fd44842570 live: IBKR order layer for trend_rider_v10 (guarded MOC orders via IB Gateway)
ibkr_trader.py computes the strategy's target for the close the way trader.py's
morning phase does, sizes whole-share orders on min(NAV, $10.5k) less a 1%
cash buffer against the account's real positions, and sends MOC orders,
sells before buys. Guards: the login must manage IBKR_ACCOUNT; a live account
needs --live and data/ibkr_live/ARMED; the account may hold only the
strategy's universe; MOC whole-share orders only; one submission per session.
After the close it records executions and commissions, scores each order
against the primary exchange's closing cross (SIP condition 6) and compares
NAV with the simulated account, with a preregistered verdict on the first
20 orders.

A what-if run on the paper Gateway (no orders) accepted GLD/SHY MOC orders
at $1.00 commission and rejected TQQQ/UPRO: the account lacks the Complex or
Leveraged ETP permission. The live service is defined but not enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:17:11 -07:00
aa24ed8c31 research: auction audit of the overnight sleeve's prices (PASS: 0.2 bp a side)
The overnight sleeve buys at the closing cross and sells at the opening
cross; the backtest used Yahoo open/close and 1 bp a side. On 800 random
nights in 2016-26 (7,856 name-nights), the backtest night return exceeds the
cross-to-cross return by +0.12 bp (SE 0.24, median 0): Yahoo's prices are
the crosses. With the SEC fee, cost is +0.20 bp a side, so 1 bp was
conservative. At the Reg T cap this lifts three_sleeve_overnight to 39%/yr
(MaxDD -27%) and the 12_1 variant to 44%. Live fills at the cross remain
unverified (paper does not run auctions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:08:32 -07:00
4ab86d47b7 research: options and futures for leverage and S&P 500 put hedges in $10k-$100k accounts (no route reaches 50%; puts FAIL)
Reg T is the binding limit: 3x ETFs need 75% margin and stocks 50%, so the
three-sleeve overnight book caps at L 1.71, about 36%/yr at MaxDD -27%.
MES futures add S&P beta, not the book's alpha. Box financing adds about
+0.9pp if a Reg T account allows it (unverified). 90%-strike calls do worse.
Puts held to expiry, reconstructed from Cboe PPUT3M/PPUT, cost 2.0-2.5%/yr
per unit of notional and buy at most +1.0pp alongside futures (gate +2pp).
At $10k no futures or index-option contract fits the account.

v1 built the hedge as the daily difference PPUT3M - S&P 500 TR. That embeds
a short S&P position equal to the put's value, so v2 holds the put to
expiry. Both are recorded; the A verdict does not depend on the hedge model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:02:45 -07:00
31a684fe70 live: record each session's target and its frictionless return; compare shows Ideal and Gap
A $10k book in whole shares holds SNDK at 17.8% for a 10% target, so the
live record mixes the strategy with rounding and fee noise (2026-09-25..10-12:
overnight_recovery_top10 -9.0% live vs -7.65% at its own weights). Each
daily_log entry now stores the target held from that close and ideal_return,
the previous target's return to this close with fractional shares, no fees,
no resize band and no idle cash. compare prints Ideal and Gap (live minus
ideal) over the recorded days. The recorded returns compound to the
frictionless ledger (new contract test); an in-memory rerun of 10-09/10-12 on
live states reproduces the recorded equity to the cent.

systemd: SuccessExitStatus=143 on the monitor and overnight-paper units, so a
stop or restart no longer logs 'Failed' and the journal shows only crashes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:49:28 -07:00
a605d68ca8 research: option-volume signals on OPRA bars 2024-03..2026-09 (STOP)
Johnson-So O/S has the opposite sign (Q5-Q1 -11.4%/yr, t -1.63) and
correlates -0.65 with the momentum score; unsigned put/call is flat
(t 0.42). Neither gated signal passes; keep collecting, retest at ~5 years.

The data handling was fixed after a coverage check of the panel, before any
return was computed: stock volume on every day from stock_raw, and a day
without bars counts as 0 contracts only inside a name's bar span (outside it
the options traded under an older ticker or the root's history starts later).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:31:54 -07:00
251014d3b2 options bars: split a symbol batch in half on HTTP 504
SNDK's October-2026 expiry month (1000 contracts, 3.5 months of daily
bars) timed out on every retry, so the backfill skipped it twice. A 504 on
a request for 50+ symbols now restarts that chunk as two halves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 12:31:50 -07:00
1b17a9cd1b research: preregister option-volume signals (O/S, put/call) before the bar backfill completes
Gate per gated signal (low_os_5, low_pc_5): h=5 quintile spread with the
predicted sign at NW t >= 3, same sign in both halves, positive top-20
alpha vs EW. Committed before the volume panel exists, so the rule
predates any look at the data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:21:54 -07:00
dc5ab31bb7 options backfill: skip empty month frames before concat (pandas FutureWarning)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:07:44 -07:00
20 changed files with 3285 additions and 14 deletions

7
.gitignore vendored
View File

@@ -69,3 +69,10 @@ data/precomputed/
# Research panels and results — fetched/rebuilt by research/lab
data/research/
# Option-chain snapshots (research.lab.fetch_options, ~11 MB/day)
data/options/
# Broker state and fills: Alpaca paper overnight book, IBKR live account
data/overnight_paper/
data/ibkr_live/

View File

@@ -50,6 +50,14 @@ uv run python overnight_paper.py buy|sell --dry-run # show the orders it wou
uv run python overnight_paper.py report # paper fills vs official open/close prints, nightly returns
uv run python overnight_paper.py daemon # the scheduler (service quant-trading-overnight-paper)
# trend_rider_v10 in an IBKR account: today's target as trader.py morning computes it, whole-share MOC orders
# via IB Gateway, fills scored against the closing cross (IBKR_ACCOUNT/IBKR_PORT/IBKR_CLIENT_ID from .env)
uv run python ibkr_trader.py check # account, positions, open orders (read-only)
uv run python ibkr_trader.py plan [--what-if] # today's orders; --what-if: IBKR validates/prices them, nothing placed
uv run python ibkr_trader.py submit --live # MOC orders, close-30 to close-15 min; live needs data/ibkr_live/ARMED
uv run python ibkr_trader.py report # executions, fills vs closing cross, live vs simulated account
uv run python ibkr_trader.py daemon --live # the scheduler (service quant-trading-ibkr-live, not enabled)
# Precompute the strategy page's backtests (writes data/precomputed/)
uv run python precompute_backtest.py
@@ -94,6 +102,7 @@ PYTHONPATH=. uv run python -m research.lab.union_shield # SPY>MA
PYTHONPATH=. uv run --with numba python -m research.lab.stream_portfolio # risk-budgeted book of momentum/lowrisk/TrendRider/ETF TSMOM/BTC streams, 2006-26 (preregistered; REJECT 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.hedged_momentum # momentum with its beta hedged by index futures (quick check; STOP 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.futures_trend [--no-proxy] # 24-market futures TSMOM on Yahoo =F + data checks (quick check; Sharpe 0.44, DATA 2026-10-08)
uv run python -m research.lab.futures_stock_signals --output data/research/results/futures_stock_signals_new # lagged futures vs ETF stock-exposure rules; 2026-10-17: 128 fixed runs + 12 availability diagnostics, retain baseline; report in futures_stock_signals_20261017/report.md
PYTHONPATH=. uv run python -m research.lab.managed_futures_book [--no-proxy] # three sleeves + managed futures (tsmom24 proxy -> DBMF), 2006-26 (preregistered; REJECT 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.small_cap_reversal # 5-day reversal on S&P 400/600, EW-hedged, spread costs (quick check; STOP 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.turn_of_month # turn-of-month premium 1928-2026, 3x in the window (quick check; STOP 2026-10-08)
@@ -113,6 +122,11 @@ PYTHONPATH=. uv run --with numba python -m research.lab.svr_combo # low_svr2
PYTHONPATH=. uv run python -m research.lab.fetch_options snapshot|status # daily S&P 500 option chains after the close: indicative quotes (forward-only from 2026-10-12) + OPRA daily bars + print-vs-quote quality sample; data/options/us/; timer quant-trading-options
PYTHONPATH=. uv run python -m research.lab.fetch_options backfill # OPRA daily option bars since 2024-02 for every S&P 500 member since then (resumable, ~2-3 h; pauses 16:10-16:55 NY); then builds volume_panel.csv.gz
PYTHONPATH=. uv run python -m research.lab.fetch_options volume-panel # date x underlying option volume (all 0-100 DTE strikes and the 5-100 DTE / 0.7-1.3 band), os_volume, pc_volume
PYTHONPATH=. uv run python -m research.lab.auction_audit [fetch] # overnight sleeve: Yahoo open/close vs the primary exchange's opening/closing cross prints (Alpaca SIP trades), 800 nights 2016-26 (preregistered; PASS 2026-10-12: +0.20 bp a side incl. SEC fee, the 1 bp backtest cost is conservative)
PYTHONPATH=. uv run python -m research.lab.no_letf_book # books without leveraged ETFs for the $10k IBKR account: QQQ trend held in QQQ (+ Reg T margin), momentum blends, vs QQQ at matched risk (preregistered; NONE BEATS QQQ 2026-10-13: Sharpe gain < 0.10; trend's gain is crashes, MaxDD -28% vs -83% since 1990; x2 margin 17.0% vs 14.9%)
PYTHONPATH=. uv run python -m research.lab.small_account_momentum # 10-day momentum in a $10k/$20k IBKR account: whole shares, Tiered/Fixed fees, rb20 / SPY de-risk / fundamental re-sort / half QQQ vs QQQ (preregistered; NONE PASSES 2026-10-13: all fail 2011-16 Sharpe; rb20_fundmom 26.3%/0.96/-32% vs QQQ 19.1%/0.87/-35% 2011-26 but no 2008 test)
PYTHONPATH=. uv run python -m research.lab.option_overlay [fetch] # leverage routes (Reg T margin, box, MES futures, calls) and S&P 500 puts (Cboe PPUT3M/PPUT held to expiry) for $10k-$100k accounts (preregistered; 2026-10-12: no route reaches 50% under Reg T, cap L 1.71 -> 36%/yr; puts B FAIL +1.0pp)
PYTHONPATH=. uv run --with numba python -m research.lab.option_volume # Johnson-So O/S and put/call volume on OPRA bars 2024-03..2026-09 (preregistered; STOP 2026-10-12: O/S sign reversed, t -1.6, corr -0.65 with momentum; put/call t 0.4)
uv run python -m research.lab.fetch_us_fundamentals --universe mid # EDGAR PIT facts for S&P 400 ever-members (name-checked CIKs)
uv run python -m research.lab.fetch_us_fundamentals --universe small # the same for S&P 600 (IJR)
PYTHONPATH=. uv run python -m research.lab.midcap_sleeve [--only h1..h4] [--universe small] # S&P 400 sleeves vs IJH; S&P 600 replication vs IJR (all REJECT 2026-10-08)
@@ -165,7 +179,7 @@ uv run python -m unittest discover tests
`precompute_backtest.py` reruns the ledger for cash-injection scenarios; metrics use unit NAV after actual fees. The API and `web/src/shared/strategyTypes.ts` share this cash-flow contract. `ENGINE_VERSION` in `execution.py` identifies the execution protocol; API cache keys also include its version.
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Orders: whole A-share lots are allocated nearest-lot within cash (`_allocate_lots`); a held name is resized only when ≥ `RESIZE_BAND` (10%) off target. Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`), the robust-momentum bar strategies `overnight_recovery_top10`, `lowrisk_momentum_top15`, `lowbeta_sharpe_mom_top20`, `lowrisk_top20`, the US sleeve portfolios `us_three_sleeve` and `us_momentum_qqq_trend`, and the agent-reviewed `overnight_recovery_agent_top10` (US only, forward only).
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Orders: whole A-share lots are allocated nearest-lot within cash (`_allocate_lots`); a held name is resized only when ≥ `RESIZE_BAND` (10%) off target. Each `daily_log` entry also records the session `target` and `ideal_return` (what the previous target earned to this close with fractional shares, no fees, no resize band and no idle cash; recorded from 2026-10-13); `compare` prints `Ideal` and `Gap` (live minus ideal) over those days, separating the strategy from $10k whole-share and fee noise. Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`), the robust-momentum bar strategies `overnight_recovery_top10`, `lowrisk_momentum_top15`, `lowbeta_sharpe_mom_top20`, `lowrisk_top20`, the US sleeve portfolios `us_three_sleeve` and `us_momentum_qqq_trend`, and the agent-reviewed `overnight_recovery_agent_top10` (US only, forward only).
**Live books** (`trader.LIVE_BOOKS`): a book is a cohort of strategies, each with its own account, started on one date with one capital and a flat fee per order. The original book (every live strategy, `MONITOR_CAPITAL`, `MARKET_FEES`) keeps state in `data/trader_*.json`; a named book keeps it in `data/books/<book>/`. The monitor runs every book; `auto`/`status`/`log`/`compare`/`replay` and `bridge.py live` take `--book`, and `bridge.py books` lists them for the UI's book selector. `selected_2026_09` (30 strategies, US $10,000 / CN ¥100,000, $2 / ¥10 per order) starts on 2026-09-25 (US) and 2026-09-28 (CN); `overnight_recovery_agent_top10` joined it on 2026-09-30 and starts with the book.
@@ -214,7 +228,7 @@ The comparison writes its fixed budget and acceptance rule before running and re
## Server Deployment
Four `systemd --user` services and one timer, defined in `deploy/systemd/`, run from `/home/gahow/projects/quant-trading`:
Four `systemd --user` services (a fifth defined but not enabled) and one timer, defined in `deploy/systemd/`, run from `/home/gahow/projects/quant-trading`:
| Service | Script | Role |
|---|---|---|
@@ -222,6 +236,7 @@ Four `systemd --user` services and one timer, defined in `deploy/systemd/`, run
| `quant-trading-api` | `deploy/start-web-api.sh` | Web API on `127.0.0.1:8787` |
| `quant-trading-vite` | `deploy/start-web-vite.sh` | Vite on `127.0.0.1:5174`, behind nginx on `:5443` |
| `quant-trading-overnight-paper` | `deploy/start-overnight-paper.sh` | `overnight_paper.py daemon`: overnight-only momentum on the Alpaca paper account (from 2026-10-12) |
| `quant-trading-ibkr-live` (not enabled) | `deploy/start-ibkr-live.sh` | `ibkr_trader.py daemon --live`: trend_rider_v10 in the $10k IBKR Pro live account. Needs a live IB Gateway on port 4001 (the paper Gateway on 4002 belongs to ../ibkr-quant, whose strategy owns that paper account), IBKR_ACCOUNT in .env, the Complex or Leveraged ETP permission (paper what-if 2026-10-13 rejected TQQQ/UPRO without it) and data/ibkr_live/ARMED |
| `quant-trading-options` (timer, 16:25 New York Mon-Fri) | `deploy/start-options-snapshot.sh` | `research.lab.fetch_options snapshot`: S&P 500 option chains (indicative quotes), OPRA daily bars, per-name IV/skew/spread/volume summary and a quote-quality sample into `data/options/us/` (~15 MB/day); `status` prints the indicative-vs-OPRA verdict after 20 sessions |
```bash

23
deploy/start-ibkr-live.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# trend_rider_v10 in the IBKR live account (ibkr_trader.py daemon --live).
# Started by the quant-trading-ibkr-live.service systemd --user unit.
# Needs: a live IB Gateway on IBKR_PORT (default 4001), IBKR_ACCOUNT in .env,
# and data/ibkr_live/ARMED; without ARMED the daemon refuses to send orders.
# Yahoo is refreshed through QUANT_MARKET_PROXY as in start-monitor.sh;
# Alpaca and the local Gateway are reached directly.
set -euo pipefail
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
PROJECT_ROOT="$( cd "$SCRIPT_DIR/.." && pwd )"
cd "$PROJECT_ROOT"
export PATH="/home/gahow/.local/bin:$PATH"
if [[ -f .env ]]; then
set -a; source .env; set +a
fi
if [[ -n "${QUANT_MARKET_PROXY:-}" ]]; then
export http_proxy="$QUANT_MARKET_PROXY" https_proxy="$QUANT_MARKET_PROXY" all_proxy="$QUANT_MARKET_PROXY"
export no_proxy="127.0.0.1,localhost,.alpaca.markets"
fi
exec uv run python ibkr_trader.py daemon --live

View File

@@ -0,0 +1,18 @@
[Unit]
Description=trend_rider_v10 in the IBKR live account (MOC orders via IB Gateway)
After=network-online.target
# Not enabled by default. Install only after the live IB Gateway is logged in,
# IBKR_ACCOUNT is set in .env, the account may trade leveraged ETFs, and
# data/ibkr_live/ARMED exists (see ibkr_trader.py).
[Service]
Type=simple
WorkingDirectory=/home/gahow/projects/quant-trading
ExecStart=/home/gahow/projects/quant-trading/deploy/start-ibkr-live.sh
Restart=always
SuccessExitStatus=143
RestartSec=30
Environment=PYTHONUNBUFFERED=1
[Install]
WantedBy=default.target

View File

@@ -7,6 +7,9 @@ Type=simple
WorkingDirectory=/home/gahow/projects/quant-trading
ExecStart=/home/gahow/projects/quant-trading/deploy/start-monitor.sh
Restart=always
# SIGTERM from a stop/restart ends the Python process with 143; count it as a
# clean stop so 'Failed' in the journal means a real crash.
SuccessExitStatus=143
RestartSec=10
Environment=PYTHONUNBUFFERED=1

View File

@@ -7,6 +7,9 @@ Type=simple
WorkingDirectory=/home/gahow/projects/quant-trading
ExecStart=/home/gahow/projects/quant-trading/deploy/start-overnight-paper.sh
Restart=always
# SIGTERM from a stop/restart ends the Python process with 143; count it as a
# clean stop so 'Failed' in the journal means a real crash.
SuccessExitStatus=143
RestartSec=30
Environment=PYTHONUNBUFFERED=1

582
ibkr_trader.py Normal file
View File

@@ -0,0 +1,582 @@
"""
Trade one registered strategy in an Interactive Brokers account with
market-on-close orders (first: trend_rider_v10, $10k, IBKR Pro). Each session
it computes the strategy's target for the close exactly as trader.py's morning
phase does (session_signals on refreshed Yahoo prices, so the target equals
the simulated account in data/trader_us_trend_rider_v10.json), sizes
whole-share orders against the account's actual positions and sends them as
MOC orders through IB Gateway. After the close it records IBKR's executions
and commissions and scores every fill against the primary listing
exchange's closing cross (SIP condition 6, as in research/lab/auction_audit.py).
uv run python ibkr_trader.py check # account, positions, open orders (read-only)
uv run python ibkr_trader.py plan [--what-if] # today's orders; --what-if has IBKR validate and price
# each order without placing it
uv run python ibkr_trader.py submit [--live] # send the MOC orders (close - 30 to close - 15 min)
uv run python ibkr_trader.py report # executions, fills vs the closing cross, live vs simulated
uv run python ibkr_trader.py daemon [--live] # submit and report on Alpaca's market calendar
Connection settings come from the environment or .env: IBKR_ACCOUNT
(required), IBKR_HOST (127.0.0.1), IBKR_PORT (4001 = live Gateway; the paper
Gateway of ../ibkr-quant listens on 4002), IBKR_CLIENT_ID (31; ibkr-quant uses
17 and 18). Live account numbers are masked in output.
Guards, each enforced in code:
account the Gateway login must manage IBKR_ACCOUNT; an account that is not a
paper account (DU/DF) also needs --live and the file data/ibkr_live/ARMED
(delete it to stop sending orders)
holdings submit refuses an account holding anything outside the strategy's
universe (plan only warns, so it can run on a shared paper account)
size orders are sized on min(net liquidation, MAX_EQUITY) less CASH_BUFFER
orders MOC only, whole shares, universe symbols, sells no larger than the
position; sells are sent and accepted before any buy
once one submission per session, none while an order of ours is open
A buy that IBKR rejects (for example buying power while the sell that funds it
is still pending) leaves the cash idle; the next session's plan buys again.
The live record therefore differs from the simulated account by fills,
whole-share sizing, the cash buffer, commissions and any such delay.
Boundary: NAV-based returns in the report assume no deposits or withdrawals.
State and logs: data/ibkr_live/ (state.json, orders.jsonl, runs/<date>.json,
fills.csv, scored.csv, daily.csv, prereg.json, verdict.json).
"""
from __future__ import annotations
import argparse
import asyncio
import json
import math
import os
import time
from datetime import date, datetime, timedelta, timezone
from zoneinfo import ZoneInfo
import pandas as pd
from execution import compute_trades
OUT = os.path.join("data", "ibkr_live")
NY = ZoneInfo("America/New_York")
ARMED = os.path.join(OUT, "ARMED")
MARKET = "us"
STRATEGY = "trend_rider_v10"
SIM_STATE = os.path.join("data", f"trader_{MARKET}_{STRATEGY}.json")
MAX_EQUITY = 10_500.0
CASH_BUFFER = 0.01 # MOC buys are sized at 15:30 prices and paid at the close
PAPER_PREFIXES = ("DU", "DF")
REF = "ibkr_trader"
SUBMIT_BEFORE_CLOSE = timedelta(minutes=30)
SUBMIT_CUTOFF = timedelta(minutes=15) # NYSE takes MOC orders until 15:50 ET
REPORT_AFTER_CLOSE = timedelta(minutes=60) # after the monitor's 16:35 evening phase
ACCEPTED = {"PreSubmitted", "Submitted", "Filled"}
DONE = {"Filled", "Cancelled", "ApiCancelled", "Inactive"}
PREREG = {
"written": "before the first live order",
"sample": "the first 20 filled MOC orders of the live account",
"reference": "primary exchange closing cross, largest SIP print with condition 6 in 15:59:59-16:10 ET",
"cost_per_side_bp": "fill vs cross, adverse positive, value-weighted; commission excluded (tracked separately)",
"rules": {"cost_per_side_bp": {"PASS": 1.0, "MEASURED": 2.5},
"fill_rate": {"PASS": 0.99, "MEASURED": 0.97},
"cross_match": {"PASS": 0.95, "MEASURED": 0.90, "tolerance_usd": 0.01}},
"decision": "the worst of the three; STOP means stop sending orders and ask IBKR about the fills",
}
# ---------------------------------------------------------------------------
# guards (pure, tested in tests/test_ibkr_trader.py)
# ---------------------------------------------------------------------------
def mask(account: str) -> str:
return account if account.startswith(PAPER_PREFIXES) else f"{account[:1]}***{account[-3:]}"
def check_account(managed: list[str], configured: str, live: bool, armed: bool, trading: bool = True) -> bool:
"""True for a live account; SystemExit unless the login (and, to trade, the flags) allow it."""
if not configured:
raise SystemExit("IBKR_ACCOUNT is not set (environment or .env)")
if configured not in managed:
raise SystemExit(f"the Gateway login manages {[mask(a) for a in managed]}, not {mask(configured)}")
is_live = not configured.startswith(PAPER_PREFIXES)
if not trading:
return is_live
if is_live and not live:
raise SystemExit(f"{mask(configured)} is a live account; pass --live to trade it")
if is_live and not armed:
raise SystemExit(f"{mask(configured)} is a live account and {ARMED} does not exist")
return is_live
def size_orders(target: dict[str, float], holdings: dict[str, int], prices: dict[str, float],
nav: float, resize_band: float, cap: float = MAX_EQUITY,
buffer: float = CASH_BUFFER) -> list[dict]:
"""Whole-share trades taking ``holdings`` to ``target`` on min(nav, cap) less the buffer."""
if missing := sorted(s for s in set(target) | set(holdings) if not prices.get(s, 0) > 0):
raise SystemExit(f"no price for {missing}")
equity = min(nav, cap)
cash = equity * (1 - buffer) - sum(q * prices[s] for s, q in holdings.items())
return compute_trades(holdings, cash, target, prices, min_trade_value=max(50.0, 0.001 * equity),
integer_shares=True, lot_size=1, resize_band=resize_band)
def validate_orders(trades: list[dict], universe: set[str], holdings: dict[str, int]) -> None:
for t in trades:
q = t["shares_delta"]
if t["ticker"] not in universe:
raise SystemExit(f"{t['ticker']} is outside the strategy's universe")
if q != int(q) or q == 0:
raise SystemExit(f"{t['ticker']}: {q} is not a whole, nonzero share count")
if q < 0 and -q > holdings.get(t["ticker"], 0):
raise SystemExit(f"{t['ticker']}: selling {-q} of {holdings.get(t['ticker'], 0)} held")
def moc_order(delta: int, account: str, ref: str):
from ib_async import Order
if delta == 0 or delta != int(delta):
raise ValueError(f"MOC order needs a whole, nonzero quantity, got {delta}")
order = Order(action="BUY" if delta > 0 else "SELL", totalQuantity=abs(int(delta)),
orderType="MOC", tif="DAY")
order.account = account
order.orderRef = ref
return order
# ---------------------------------------------------------------------------
# configuration, state, logs
# ---------------------------------------------------------------------------
def settings() -> dict:
from overnight_paper import _env
env = _env()
return {"host": env.get("IBKR_HOST", "127.0.0.1"), "port": int(env.get("IBKR_PORT", "4001")),
"client_id": int(env.get("IBKR_CLIENT_ID", "31")), "account": env.get("IBKR_ACCOUNT", "")}
def _state() -> dict:
path = os.path.join(OUT, "state.json")
return json.load(open(path)) if os.path.exists(path) else {}
def _save_state(st: dict) -> None:
os.makedirs(OUT, exist_ok=True)
tmp = os.path.join(OUT, ".state.json.tmp")
with open(tmp, "w") as fh:
json.dump(st, fh, indent=1)
os.replace(tmp, os.path.join(OUT, "state.json"))
def _append_jsonl(name: str, row: dict) -> None:
os.makedirs(OUT, exist_ok=True)
with open(os.path.join(OUT, name), "a") as fh:
fh.write(json.dumps(row, default=str) + "\n")
def now_utc() -> datetime:
return datetime.now(timezone.utc)
# ---------------------------------------------------------------------------
# target and prices
# ---------------------------------------------------------------------------
def session_target() -> tuple[date, dict[str, float], list[str]]:
"""The strategy's target for the close of the latest session with an open bar, as trader.py morning computes it."""
import trader
tickers, benchmark = trader.strategy_universe(MARKET, STRATEGY)
close, open_data, close_raw = trader.load_strategy_prices(MARKET, STRATEGY, tickers, benchmark, with_open=True)
tickers = trader.filter_tradable_tickers(close, tickers)
day = open_data.index[-1]
row = trader.session_signals(MARKET, STRATEGY, tickers, pd.DatetimeIndex([day]), close_raw, open_data).loc[day]
return day.date(), {t: float(w) for t, w in row.items() if w > 1e-6}, sorted(trader.ETF_STRATEGY_UNIVERSES[STRATEGY])
def monitor_target(day: date) -> dict[str, float] | None:
"""The simulated account's pending target for ``day`` (written by the monitor's 09:45 morning phase)."""
if not os.path.exists(SIM_STATE):
return None
pending = json.load(open(SIM_STATE)).get("pending_trades") or {}
return pending.get("target_weights") if pending.get("date") == day.isoformat() else None
def sizing_prices(symbols: list[str]) -> dict[str, float]:
from overnight_paper import Alpaca, latest_prices
return latest_prices(Alpaca(), symbols)
# ---------------------------------------------------------------------------
# IB Gateway
# ---------------------------------------------------------------------------
class Gateway:
def __init__(self, cfg: dict) -> None:
from ib_async import IB
self.cfg, self.ib = cfg, IB()
async def __aenter__(self) -> "Gateway":
address = f"{self.cfg['host']}:{self.cfg['port']}"
try:
# Without raiseSyncErrors a Gateway that lost IBKR's servers connects with an
# empty portfolio, which would read as no holdings (ibkr-quant, 2026-09-29).
await self.ib.connectAsync(self.cfg["host"], self.cfg["port"], self.cfg["client_id"], timeout=15,
account=self.cfg["account"], raiseSyncErrors=True)
except ConnectionRefusedError:
raise SystemExit(f"nothing listening on {address}; start IB Gateway") from None
except TimeoutError:
raise SystemExit(f"IB Gateway on {address} did not finish the API handshake") from None
except ConnectionError as e:
raise SystemExit(f"IB Gateway on {address} failed {e}; it may have lost IBKR's servers") from None
return self
async def __aexit__(self, *exc) -> None:
self.ib.disconnect()
async def account(self, account: str) -> dict:
tags = {v.tag: v for v in await self.ib.accountSummaryAsync(account)}
positions, foreign = {}, []
for p in self.ib.positions(account):
if not p.position:
continue
c = p.contract
if (c.secType, c.currency) == ("STK", "USD") and p.position == int(p.position):
positions[c.symbol] = int(p.position)
else:
foreign.append(f"{c.symbol} {c.secType} {c.currency} {p.position}")
return {"nav": float(tags["NetLiquidation"].value), "cash": float(tags["TotalCashValue"].value),
"available": float(tags["AvailableFunds"].value), "positions": positions, "foreign": foreign}
async def our_open_orders(self, account: str) -> list:
trades = await self.ib.reqAllOpenOrdersAsync()
return [t for t in trades if t.order.account in ("", account) and t.order.orderRef.startswith(REF)]
async def contracts(self, symbols: list[str]) -> dict:
from ib_async import Stock
qualified = await self.ib.qualifyContractsAsync(*[Stock(s, "SMART", "USD") for s in symbols])
found = {c.symbol: c for c in qualified if c is not None and getattr(c, "conId", 0)}
if missing := sorted(set(symbols) - set(found)):
raise SystemExit(f"IBKR could not qualify {missing}")
return found
async def confirm(self, trades: list, timeout: float = 30) -> list:
"""Wait until each order is accepted or done; return the ones not accepted."""
loop = asyncio.get_running_loop()
deadline = loop.time() + timeout
while any(t.orderStatus.status not in ACCEPTED | DONE for t in trades) and loop.time() < deadline:
await asyncio.sleep(0.1)
return [t for t in trades if t.orderStatus.status not in ACCEPTED]
def _errors(trade) -> str:
return "; ".join(f"{e.errorCode} {e.message}" for e in trade.log if e.errorCode)
def _order_row(kind: str, trade, day: date) -> dict:
return {"logged_at": now_utc().isoformat(), "kind": kind, "session": day.isoformat(),
"symbol": trade.contract.symbol, "action": trade.order.action, "qty": trade.order.totalQuantity,
"order_id": trade.order.orderId, "perm_id": trade.order.permId, "status": trade.orderStatus.status,
"errors": _errors(trade)}
# ---------------------------------------------------------------------------
# commands
# ---------------------------------------------------------------------------
async def check(cfg: dict) -> None:
async with Gateway(cfg) as gw:
managed = gw.ib.managedAccounts()
acct = cfg["account"] or (managed[0] if len(managed) == 1 else "")
if acct not in managed:
raise SystemExit(f"set IBKR_ACCOUNT; the login manages {[mask(a) for a in managed]}")
a = await gw.account(acct)
print(f"account {mask(acct)} NAV ${a['nav']:,.2f} cash ${a['cash']:,.2f} available ${a['available']:,.2f}")
print(f"positions {a['positions'] or 'none'}" + (f"; other {a['foreign']}" if a["foreign"] else ""))
print(f"open orders of ours: {len(await gw.our_open_orders(acct))}")
async def plan(cfg: dict, what_if: bool = False, submit: bool = False, live: bool = False) -> dict:
day, target, universe = session_target()
today = datetime.now(NY).date()
if submit and day != today:
raise SystemExit(f"latest open bar is {day}, not today {today}; Yahoo has not printed today's open")
mon = monitor_target(day)
agree = mon is not None and set(mon) == set(target) and all(abs(mon[s] - target[s]) < 1e-6 for s in target)
print(f"{STRATEGY} target for the close of {day}: "
+ (", ".join(f"{s} {w:.1%}" for s, w in sorted(target.items())) or "cash")
+ ("" if mon is None else f" (monitor's target {'agrees' if agree else 'DIFFERS: ' + str(mon)})"))
async with Gateway(cfg) as gw:
is_live = check_account(gw.ib.managedAccounts(), cfg["account"], live, os.path.exists(ARMED), trading=submit)
acct = cfg["account"]
a = await gw.account(acct)
held = {s: q for s, q in a["positions"].items() if s in universe}
other = a["foreign"] + [f"{s} {q}" for s, q in a["positions"].items() if s not in universe]
if other:
if submit:
raise SystemExit(f"the account holds positions outside {STRATEGY}'s universe: {other}")
print(f" ignoring positions outside the universe (plan only): {len(other)}")
prices = sizing_prices(sorted(set(target) | set(held)))
trades = size_orders(target, held, prices, a["nav"], resize_band=_resize_band())
validate_orders(trades, set(universe), held)
print(f"account {mask(acct)}{' (live)' if is_live else ' (paper)'} NAV ${a['nav']:,.2f} "
f"sized on ${min(a['nav'], MAX_EQUITY) * (1 - CASH_BUFFER):,.2f} held {held or 'nothing'}")
for t in trades:
print(f" {t['direction']:4s} {abs(t['shares_delta']):5d} {t['ticker']:5s} ~${t['est_value']:>9,.2f} "
f"@ {t['price']:.2f} -> {t['target_shares']} shares")
if not trades:
print(" no orders")
run = {"session": day.isoformat(), "computed_at": now_utc().isoformat(), "target": target,
"monitor_target": mon, "nav": a["nav"], "held": held, "prices": prices, "trades": trades}
if what_if and trades:
contracts = await gw.contracts([t["ticker"] for t in trades])
errors: list[str] = []
on_error = lambda req_id, code, message, contract: errors.append(f"{code} {message}") # noqa: E731
gw.ib.errorEvent += on_error
try:
for t in trades:
errors.clear()
st = await gw.ib.whatIfOrderAsync(contracts[t["ticker"]], moc_order(t["shares_delta"], acct, REF))
if not hasattr(st, "initMarginChange"): # a rejected what-if returns no OrderState
t["what_if"] = {"rejected": "; ".join(errors) or "no answer"}
print(f" what-if {t['ticker']:5s}: REJECTED {t['what_if']['rejected'].replace('<br>', ' ')[:300]}")
continue
t["what_if"] = {k: getattr(st, k, None) for k in ("status", "initMarginChange", "maintMarginChange",
"equityWithLoanAfter", "commission",
"minCommission", "maxCommission", "warningText")}
print(f" what-if {t['ticker']:5s}: accepted, commission "
f"{_money(st.commission, st.minCommission, st.maxCommission)}, init margin change "
f"{st.initMarginChange}{', warning: ' + st.warningText if st.warningText else ''}")
finally:
gw.ib.errorEvent -= on_error
if not submit:
return run
st = _state()
if st.get("submitted") == day.isoformat():
raise SystemExit(f"{day}: orders already submitted")
if mine := await gw.our_open_orders(acct):
raise SystemExit(f"{len(mine)} orders of ours are still open")
if is_live and not os.path.exists(os.path.join(OUT, "prereg.json")):
with open(os.path.join(OUT, "prereg.json"), "w") as fh:
json.dump({**PREREG, "written_at": now_utc().isoformat(), "strategy": STRATEGY}, fh, indent=2)
st["submitted"] = day.isoformat() # before sending: a crash must not resend
_save_state(st)
os.makedirs(os.path.join(OUT, "runs"), exist_ok=True)
contracts = await gw.contracts([t["ticker"] for t in trades]) if trades else {}
failed = []
for side in ("SELL", "BUY"):
batch = [gw.ib.placeOrder(contracts[t["ticker"]], moc_order(t["shares_delta"], acct, f"{REF} {day}"))
for t in trades if t["direction"] == side]
bad = await gw.confirm(batch) if batch else []
for tr in batch:
_append_jsonl("orders.jsonl", _order_row("moc", tr, day))
print(f" {tr.order.action:4s} {tr.order.totalQuantity:>5.0f} {tr.contract.symbol:5s} MOC -> "
f"{tr.orderStatus.status}{' (' + _errors(tr) + ')' if _errors(tr) else ''}")
failed += bad
if side == "SELL" and bad:
break # no buys on top of a sell that did not go in
run["orders"] = [o for o in (json.loads(l) for l in open(os.path.join(OUT, "orders.jsonl")))
if o["session"] == day.isoformat()] if trades else []
with open(os.path.join(OUT, "runs", f"{day}.json"), "w") as fh:
json.dump(run, fh, indent=1, default=str)
if failed:
raise SystemExit("not accepted: " + "; ".join(f"{t.contract.symbol} {t.orderStatus.status} {_errors(t)}"
for t in failed))
return run
def _resize_band() -> float:
import trader
return trader.strategy_resize_band(STRATEGY)
def _money(c, lo, hi) -> str:
ok = lambda x: x is not None and math.isfinite(x) and abs(x) < 1e9
if ok(c):
return f"${c:.2f}"
return f"${lo:.2f}-${hi:.2f}" if ok(lo) and ok(hi) else "n/a"
async def report(cfg: dict) -> None:
from ib_async import ExecutionFilter
async with Gateway(cfg) as gw:
acct = cfg["account"]
if acct not in gw.ib.managedAccounts():
raise SystemExit(f"the Gateway login does not manage {mask(acct)}")
fills = await gw.ib.reqExecutionsAsync(ExecutionFilter(acctCode=acct))
a = await gw.account(acct)
rows = []
for f in fills:
e, cr = f.execution, f.commissionReport
if not e.orderRef.startswith(REF):
continue
commission = cr.commission if cr and math.isfinite(cr.commission) and cr.commission < 1e9 else float("nan")
rows.append({"exec_id": e.execId, "time": e.time.astimezone(NY).isoformat(),
"session": e.time.astimezone(NY).date().isoformat(), "symbol": f.contract.symbol,
"side": "BUY" if e.side == "BOT" else "SELL", "shares": e.shares, "price": e.price,
"exchange": e.exchange, "perm_id": e.permId, "order_ref": e.orderRef, "commission": commission})
path = os.path.join(OUT, "fills.csv")
old = pd.read_csv(path) if os.path.exists(path) else pd.DataFrame()
F = pd.concat([old, pd.DataFrame(rows)], ignore_index=True)
if len(F):
F = F.drop_duplicates("exec_id", keep="last").sort_values("time")
F.to_csv(path, index=False)
today = datetime.now(NY).date().isoformat()
_append_daily(today, a)
if F.empty:
print("no fills yet")
F = pd.DataFrame(columns=["session", "symbol", "side", "shares", "price", "commission"])
_score(F)
_compare_with_simulation()
def _append_daily(day: str, a: dict) -> None:
path = os.path.join(OUT, "daily.csv")
D = pd.read_csv(path) if os.path.exists(path) else pd.DataFrame()
row = pd.DataFrame([{"date": day, "nav": a["nav"], "cash": a["cash"], "positions": json.dumps(a["positions"])}])
D = pd.concat([D[D["date"] != day] if len(D) else D, row], ignore_index=True)
D.to_csv(path, index=False)
def _score(F: pd.DataFrame) -> None:
"""Each accepted MOC order against the closing cross; preregistered verdict on the first 20 orders.
Orders and fills are matched on (session, symbol, side): a session sends at most
one order per symbol. An accepted order with no fill counts against the fill rate.
"""
from research.lab.auction_audit import crosses
import research.lab.fetch_options as fo
path = os.path.join(OUT, "orders.jsonl")
if not os.path.exists(path):
return
O = pd.DataFrame([json.loads(l) for l in open(path)])
O = O[O["status"].isin(ACCEPTED)].drop_duplicates(["session", "symbol", "action"], keep="last")
settled = pd.Timestamp.now(tz=NY) - pd.Timedelta(hours=16, minutes=20) # free SIP data lags 15 min
O = O[pd.to_datetime(O["session"]).dt.tz_localize(NY) <= settled]
if O.empty:
return
f = F.assign(value=F["shares"] * F["price"]).groupby(["session", "symbol", "side"]) \
.agg(filled=("shares", "sum"), value=("value", "sum"), commission=("commission", "sum"))
g = O.join(f, on=["session", "symbol", "action"])[["session", "symbol", "action", "qty", "filled", "value",
"commission"]].fillna({"filled": 0.0})
g["avg"] = g["value"] / g["filled"]
api, ref = fo.Api(min_interval=0.4), {}
for day, h in g.groupby("session"):
for sym, v in crosses(api, sorted(h["symbol"].unique()), pd.Timestamp(day), "15:59:59", "16:10:00", "6").items():
ref[(day, sym)] = v["p"]
g["cross"] = [ref.get((d, sym)) for d, sym in zip(g["session"], g["symbol"])]
g["slip_bp"] = g["action"].map({"BUY": 1, "SELL": -1}) * (g["avg"] / g["cross"] - 1) * 1e4
g["match"] = (g["avg"] - g["cross"]).abs() <= PREREG["rules"]["cross_match"]["tolerance_usd"]
g["commission_bp"] = g["commission"] / g["value"] * 1e4
g = g.sort_values(["session", "symbol"]).reset_index(drop=True)
g.to_csv(os.path.join(OUT, "scored.csv"), index=False)
print(g.tail(10)[["session", "symbol", "action", "qty", "filled", "avg", "cross", "slip_bp", "commission_bp"]]
.to_string(index=False, float_format=lambda x: f"{x:.3f}"))
def measure(x: pd.DataFrame) -> dict:
y = x[x["filled"] > 0].dropna(subset=["cross"])
return {"cost_per_side_bp": (y["slip_bp"] * y["value"]).sum() / y["value"].sum() if len(y) else float("nan"),
"fill_rate": x["filled"].sum() / x["qty"].sum(),
"cross_match": y["match"].mean() if len(y) else float("nan"),
"commission_bp": y["commission"].sum() / y["value"].sum() * 1e4 if len(y) else float("nan")}
m = measure(g)
print(f"orders {len(g)}: cost {m['cost_per_side_bp']:+.2f} bp a side (ex commission), fill rate "
f"{m['fill_rate']:.1%}, cross match {m['cross_match']:.0%}, commission {m['commission_bp']:.2f} bp a side")
verdict_path = os.path.join(OUT, "verdict.json")
if len(g) < 20 or not os.path.exists(os.path.join(OUT, "prereg.json")) or os.path.exists(verdict_path):
return
v, r = measure(g.head(20)), PREREG["rules"]
grades = {
"cost_per_side_bp": "PASS" if v["cost_per_side_bp"] <= r["cost_per_side_bp"]["PASS"]
else "MEASURED" if v["cost_per_side_bp"] <= r["cost_per_side_bp"]["MEASURED"] else "STOP",
"fill_rate": "PASS" if v["fill_rate"] >= r["fill_rate"]["PASS"]
else "MEASURED" if v["fill_rate"] >= r["fill_rate"]["MEASURED"] else "STOP",
"cross_match": "PASS" if v["cross_match"] >= r["cross_match"]["PASS"]
else "MEASURED" if v["cross_match"] >= r["cross_match"]["MEASURED"] else "STOP",
}
decision = max(grades.values(), key=["PASS", "MEASURED", "STOP"].index)
with open(verdict_path, "w") as fh:
json.dump({"decided_at": now_utc().isoformat(), "orders": 20, "values": v, "grades": grades,
"decision": decision}, fh, indent=2)
print(f"verdict on the first 20 orders: {decision} {grades}")
def _compare_with_simulation() -> None:
path = os.path.join(OUT, "daily.csv")
if not (os.path.exists(path) and os.path.exists(SIM_STATE)):
return
live = pd.read_csv(path).set_index("date")["nav"]
sim = pd.Series(json.load(open(SIM_STATE))["daily_equity"])
both = pd.concat({"live": live, "sim": sim}, axis=1).dropna()
if len(both) < 2:
return
r = both / both.iloc[0] - 1
print(f"since {both.index[0]}: live {r['live'].iloc[-1]:+.2%}, simulated account {r['sim'].iloc[-1]:+.2%}, "
f"gap {r['live'].iloc[-1] - r['sim'].iloc[-1]:+.2%} over {len(both) - 1} sessions")
# ---------------------------------------------------------------------------
# daemon
# ---------------------------------------------------------------------------
def daemon(cfg: dict, live: bool) -> None:
from overnight_paper import Alpaca, sessions
cl = Alpaca()
print(f"{now_utc():%Y-%m-%d %H:%M} UTC ibkr_trader daemon started ({STRATEGY})", flush=True)
done: set[tuple[str, str]] = set()
while True:
try:
now = now_utc()
cal = sessions(cl, (now - timedelta(days=1)).date(), (now + timedelta(days=7)).date())
for d, row in cal.iterrows():
key = d.isoformat()
if ("submit", key) not in done and row["close"] - SUBMIT_BEFORE_CLOSE <= now < row["close"] - SUBMIT_CUTOFF:
# Retried each minute inside the window until the session is marked submitted,
# which happens just before the first order goes out: nothing is ever sent twice.
try:
if _state().get("submitted") != key:
asyncio.run(plan(cfg, submit=True, live=live))
finally:
if _state().get("submitted") == key:
done.add(("submit", key))
if ("report", key) not in done and row["close"] + REPORT_AFTER_CLOSE <= now < row["close"] + timedelta(hours=7):
done.add(("report", key))
asyncio.run(report(cfg))
except SystemExit as exc:
print(f"{now_utc():%Y-%m-%d %H:%M} UTC stopped: {exc}", flush=True)
except Exception as exc: # noqa: BLE001 - keep the loop alive
print(f"{now_utc():%Y-%m-%d %H:%M} UTC error: {type(exc).__name__}: {str(exc)[:300]}", flush=True)
time.sleep(60)
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("what", choices=["check", "plan", "submit", "report", "daemon"])
ap.add_argument("--what-if", action="store_true", help="plan: have IBKR validate and price each order")
ap.add_argument("--live", action="store_true", help="allow a live (non-DU) account; also needs data/ibkr_live/ARMED")
a = ap.parse_args()
cfg = settings()
if a.what == "check":
asyncio.run(check(cfg))
elif a.what == "plan":
asyncio.run(plan(cfg, what_if=a.what_if))
elif a.what == "submit":
from overnight_paper import Alpaca, sessions
today = datetime.now(NY).date()
cal = sessions(Alpaca(), today, today)
if today not in cal.index:
raise SystemExit(f"{today}: no session")
close = cal.loc[today, "close"]
if not close - SUBMIT_BEFORE_CLOSE <= now_utc() <= close - SUBMIT_CUTOFF:
raise SystemExit(f"outside the MOC window ({(close - SUBMIT_BEFORE_CLOSE).astimezone(NY):%H:%M}-"
f"{(close - SUBMIT_CUTOFF).astimezone(NY):%H:%M} ET)")
asyncio.run(plan(cfg, submit=True, live=a.live))
elif a.what == "report":
asyncio.run(report(cfg))
else:
daemon(cfg, a.live)
if __name__ == "__main__":
main()

View File

@@ -5,6 +5,7 @@ description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"ib-async>=2.1.0",
"lxml>=6.0.2",
"matplotlib>=3.10.8",
"numpy>=2.4.0",

View File

@@ -0,0 +1,336 @@
"""
Do the overnight sleeve's backtest prices match the auctions it would trade
in? The sleeve buys market-on-close and sells market-on-open, so a real
night earns the primary exchange's closing cross to its next opening cross.
The backtest (research/lab/overnight_only.py) earns Yahoo open_t / close_t-1
and charges 1 bp a side. This measures the gap.
PYTHONPATH=. uv run python -m research.lab.auction_audit fetch # sample, SIP trades, daily bars (resumable)
PYTHONPATH=. uv run python -m research.lab.auction_audit # compare (refuses to overwrite results)
Sample: 800 nights drawn at random (seed 20261012) from 2016-01-05..2026-09-24
(Alpaca SIP history starts 2016). On each night it takes every name the sleeve
holds (sleeve_score, the live score, top 10 by score at close t-2) plus the
overnight_12_1 names (reported).
Auction prints (Alpaca SIP trades, feed=sip): the cross is the largest-size
print with condition 'O' (open) or '6' (close). That is the primary listing
exchange's cross: Nasdaq x=Q (x=T in older data), NYSE x=N.
close d-1, 15:59:59-16:10:00 New York; NYSE close prints can arrive after 16:03
open d, 09:29:59-09:30:03; names still missing 09:30:03-09:31:00, then
09:31:00-09:40:00 (NYSE opens can come seconds to minutes late)
Adjustment: F = close(adjustment=all) / close(raw) from Alpaca daily bars, so
real night R_real = open_x(d) F(d) / (close_x(d-1) F(d-1)) - 1
backtest R_bt = Yahoo open(d) / Yahoo close(d-1) - 1 (research panel)
shortfall R_bt - R_real, per name-night; the night mean is the unit of
observation (SE over nights)
split R_bt - R_bar (Yahoo vs Alpaca daily-bar open/close) and
R_bar - R_real (bar open/close vs the crosses)
Fees on top: SEC Section 31 fee about 0.28 bp on sells (0.14 bp a side on
average); FINRA TAF is negligible. Commissions are $0 at Alpaca.
Rule (fixed before any comparison was computed):
cost per side = night-mean shortfall / 2 + 0.14 bp, for sleeve_score names
<= 1.0 bp PASS: the backtest's 1 bp a side holds; the Reg T-cap ranges stand
(research/lab/option_overlay.py: about 36% three_sleeve_overnight, 41% 12_1)
1.0-2.5 bp plan with the measured cost (about 2 bp: 32% / 37% at the cap)
> 2.5 bp the overnight sleeve's edge over the all-day sleeve is gone
(overnight_book: 4 bp a side is the baseline's Sharpe); no live overnight sleeve
Reported: by year, 2016-20 vs 2021-26, Nasdaq vs NYSE listing, overnight_12_1
names, trimmed mean, the share of name-nights without a cross print.
Not measured: whether orders reach the auction (Alpaca's MOC cutoff 15:50,
MOO before 09:28), impact at larger sizes, partial fills (a market order in
a cross fills unless the stock is halted).
Result (2026-10-12): PASS. The cost per side is +0.20 bp for the
sleeve_score names and +0.28 bp for overnight_12_1, including the SEC fee.
sleeve_score: 7,856 name-nights over 795 nights, 2016-01..2026-09
R_bt - R_real +0.12 bp/night (SE 0.24), median 0.00, trimmed +0.23
Yahoo vs bar -0.02 (SE 0.09)
bar vs cross +0.14 (SE 0.24): the closing cross equals the bar close
(0.00); the bar open vs the opening cross is +0.16
by year within +-1 bp except 2020, -1.6 (SE 0.8);
2016-20 -0.24, 2021-26 +0.48
by listing NYSE +0.18, Nasdaq -0.02
coverage 122 / 74 of 8,000 name-nights had no close / open
cross print (halts, symbol changes); 2 outliers over
5% were dropped
- Yahoo's open and close are the primary exchange's crosses (median
difference 0). The backtest's prices are the prices a MOC/MOO order
gets, so the 1 bp a side the books assumed was conservative. About
0.2 bp a side remains, which is the SEC fee.
- Recomputed after the run at 0.2 bp a side, 2006-10..2026-09 with
margin at T-bill + 1.5% (CAGR / MaxDD):
book 1x Reg T cap
three_sleeve_overnight 23.9% 39.2% at 1.7x (-26.7%)
overnight_12_1 26.6% 44.3% at 1.7x (-26.2%)
two_sleeve_overnight_trend 32.0% 50.6% at 1.6x (-28.7%)
two_sleeve_overnight_trend makes 64% a year in 2017-26 and 39% in
2006-16. It leans on TrendRider's last decade, which the 1990-2026
synthetic test does not support.
- Not measured:
- whether a live MOC/MOO order fills at the cross. Alpaca's paper engine
does not run auctions, so only a small live account can check this.
- impact at larger size. Orders here are thousands of dollars against
crosses of millions.
- whole-share sizing in small accounts.
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
import numpy as np
import pandas as pd
from research.lab.factor_scan import RESULTS, load
OUT = os.path.join(RESULTS, "auction_audit")
RAW = os.path.join("data", "research", "auction_audit")
TRADES_URL = "https://data.alpaca.markets/v2/stocks/trades"
BARS_URL = "https://data.alpaca.markets/v2/stocks/bars"
NY = "America/New_York"
SAMPLE_START, SAMPLE_END, N_NIGHTS, SEED = "2016-01-05", "2026-09-24", 800, 20261012
TOP_N = 10
SEC_FEE_SIDE_BP = 0.14
PASS_BP, STOP_BP = 1.0, 2.5
PREREG = {
"question": "How far is the overnight sleeve's backtest night return (Yahoo open/close) from the closing-cross "
"to opening-cross return it would actually earn?",
"sample": {"nights": N_NIGHTS, "from": SAMPLE_START, "to": SAMPLE_END, "seed": SEED,
"names": "sleeve_score top 10 (score at close t-2); overnight_12_1 names reported"},
"cross": "largest-size SIP print with condition 'O' (open) / '6' (close)",
"adjustment": "Alpaca daily bars: F = close(all) / close(raw)",
"metric": "cost per side = night-mean (R_bt - R_real) / 2 + 0.14 bp SEC fee",
"rule": {"PASS": "<= 1.0 bp/side: the 1 bp backtest cost holds",
"MEASURED": "1.0-2.5 bp/side: re-plan returns with the measured cost",
"STOP": "> 2.5 bp/side: no live overnight sleeve"},
"data_status": "no auction prints compared before this rule; the 09:30 5-min bar check (overnight_only N5) "
"was seen: Yahoo vs bar overnight returns corr 0.996 on selected names, mean diff 0.03-0.3 bp",
}
# ---------------------------------------------------------------------------
# sample
# ---------------------------------------------------------------------------
def selections(p) -> dict[str, pd.DataFrame]:
from research.lab.factors import overnight_mom, rec_126
from research.lab.overnight_only import top_mask
from research.lab.us_sweep import _blend
mem = p.member
scores = {"sleeve_score": _blend(*[f.where(mem).rank(axis=1, pct=True)
for f in (overnight_mom(p), rec_126(p))]).where(mem),
"overnight_12_1": overnight_mom(p).where(mem)}
return {k: top_mask(sc.shift(2), TOP_N) for k, sc in scores.items()}
def sample_nights(p, sels: dict) -> pd.DataFrame:
dates = p.dates[(p.dates >= SAMPLE_START) & (p.dates <= SAMPLE_END)]
dates = [d for d in dates if sels["sleeve_score"].loc[d].sum() == TOP_N]
rng = np.random.default_rng(SEED)
nights = sorted(rng.choice(np.array(dates, dtype="datetime64[ns]"), size=N_NIGHTS, replace=False))
rows = []
for d in pd.DatetimeIndex(nights):
prev = p.dates[p.dates.get_loc(d) - 1]
for k, sel in sels.items():
for s in sel.columns[sel.loc[d].values]:
rows.append({"night": d, "prev": prev, "symbol": s, "candidate": k})
return pd.DataFrame(rows)
# ---------------------------------------------------------------------------
# fetch
# ---------------------------------------------------------------------------
def _ts(day: pd.Timestamp, hhmmss: str) -> str:
t = pd.Timestamp(f"{day.date()} {hhmmss}").tz_localize(NY)
return t.isoformat()
def alp(sym: str) -> str:
return sym.replace("-", ".")
def crosses(api, syms: list[str], day: pd.Timestamp, a: str, b: str, cond: str) -> dict[str, dict]:
"""Largest-size print with condition ``cond`` per symbol in [a, b] New York time."""
best: dict[str, dict] = {}
params = {"symbols": ",".join(alp(s) for s in syms), "start": _ts(day, a), "end": _ts(day, b),
"feed": "sip", "limit": 10000}
for page in api.pages(TRADES_URL, params, "trades"):
if not page:
continue
for s, ts in page.items():
for t in ts:
if cond in (t.get("c") or []) and t["s"] > best.get(s, {}).get("s", -1):
best[s] = {"p": t["p"], "s": t["s"], "x": t["x"], "t": t["t"]}
return {s.replace(".", "-"): v for s, v in best.items()}
def fetch() -> None:
import research.lab.fetch_options as fo
os.makedirs(RAW, exist_ok=True)
os.makedirs(OUT, exist_ok=True)
prereg = os.path.join(OUT, "prereg.json")
if not os.path.exists(prereg):
with open(prereg, "w") as fh:
json.dump(PREREG, fh, indent=2)
p, _ = load("us")
sample_path = os.path.join(RAW, "sample.csv")
if os.path.exists(sample_path):
S = pd.read_csv(sample_path, parse_dates=["night", "prev"])
else:
S = sample_nights(p, selections(p))
S.to_csv(sample_path, index=False)
api = fo.Api(min_interval=0.4, yield_to_snapshot=True)
prints_path = os.path.join(RAW, "prints.csv")
done = set()
if os.path.exists(prints_path):
done = set(pd.read_csv(prints_path, usecols=["night"])["night"].astype(str))
nights = sorted(S["night"].unique())
print(f"{len(nights)} nights, {S['symbol'].nunique()} symbols, {len(done)} done", flush=True)
for i, d in enumerate(nights):
d = pd.Timestamp(d)
if str(d.date()) in done:
continue
g = S[S["night"] == d]
syms, prev = sorted(g["symbol"].unique()), pd.Timestamp(g["prev"].iloc[0])
close = crosses(api, syms, prev, "15:59:59", "16:10:00", "6")
opn = crosses(api, syms, d, "09:29:59", "09:30:03", "O")
for a, b in (("09:30:03", "09:31:00"), ("09:31:00", "09:40:00")):
missing = [s for s in syms if s not in opn]
if not missing:
break
opn.update(crosses(api, missing, d, a, b, "O"))
rows = []
for s in syms:
c, o = close.get(s, {}), opn.get(s, {})
rows.append({"night": str(d.date()), "prev": str(prev.date()), "symbol": s,
"close_p": c.get("p"), "close_s": c.get("s"), "close_x": c.get("x"), "close_t": c.get("t"),
"open_p": o.get("p"), "open_s": o.get("s"), "open_x": o.get("x"), "open_t": o.get("t")})
pd.DataFrame(rows).to_csv(prints_path, mode="a", header=not os.path.exists(prints_path), index=False)
if (i + 1) % 25 == 0:
print(f" [{i + 1}/{len(nights)}] {d.date()} {api.n} requests", flush=True)
bars_path = os.path.join(RAW, "bars.pkl")
if not os.path.exists(bars_path):
syms = sorted(S["symbol"].unique())
out = {}
for adj in ("raw", "all"):
rows = []
for k in range(0, len(syms), 100):
params = {"symbols": ",".join(alp(s) for s in syms[k:k + 100]), "timeframe": "1Day",
"start": "2015-12-01", "end": "2026-09-30", "adjustment": adj, "feed": "sip",
"limit": 10000}
for page in api.pages(BARS_URL, params, "bars"):
if not page:
continue
for s, bl in page.items():
rows += [{"symbol": s.replace(".", "-"), "date": b["t"][:10], "o": b["o"], "c": b["c"]}
for b in bl]
out[adj] = pd.DataFrame(rows)
print(f"bars {adj}: {len(rows)} rows", flush=True)
pd.to_pickle(out, bars_path)
print(f"done, {api.n} requests", flush=True)
# ---------------------------------------------------------------------------
# compare
# ---------------------------------------------------------------------------
def night_stats(x: pd.Series, nights: pd.Series) -> dict:
"""Mean over nights of the night-mean, its SE, and name-level median."""
nm = x.groupby(nights).mean().dropna()
return {"nights": int(len(nm)), "name_nights": int(x.notna().sum()),
"mean_bp": float(nm.mean() * 1e4), "se_bp": float(nm.std() / np.sqrt(len(nm)) * 1e4),
"median_bp": float(x.median() * 1e4),
"trimmed_mean_bp": float(x.clip(x.quantile(0.025), x.quantile(0.975)).mean() * 1e4)}
def main() -> None:
res_path = os.path.join(OUT, "results.json")
if os.path.exists(res_path):
raise SystemExit(f"{res_path} exists; this study does not overwrite results")
S = pd.read_csv(os.path.join(RAW, "sample.csv"), parse_dates=["night", "prev"])
P = pd.read_csv(os.path.join(RAW, "prints.csv"), parse_dates=["night", "prev"]).drop_duplicates(["night", "symbol"])
bars = pd.read_pickle(os.path.join(RAW, "bars.pkl"))
p, _ = load("us")
def wide(df, col):
return df.pivot_table(index=pd.to_datetime(df["date"]), columns="symbol", values=col, aggfunc="last")
braw_c, braw_o, ball_c = wide(bars["raw"], "c"), wide(bars["raw"], "o"), wide(bars["all"], "c")
F = ball_c / braw_c
X = S.merge(P, on=["night", "prev", "symbol"], how="left")
def at(df, dates, syms):
return np.array([df.at[d, s] if (d in df.index and s in df.columns) else np.nan for d, s in zip(dates, syms)],
dtype=float)
X["F_prev"], X["F_night"] = at(F, X["prev"], X["symbol"]), at(F, X["night"], X["symbol"])
X["bar_c_prev"], X["bar_o_night"] = at(braw_c, X["prev"], X["symbol"]), at(braw_o, X["night"], X["symbol"])
X["y_c_prev"] = at(p.close, X["prev"], X["symbol"])
X["y_o_night"] = at(p.open, X["night"], X["symbol"])
X["R_bt"] = X["y_o_night"] / X["y_c_prev"] - 1
X["R_real"] = (X["open_p"] * X["F_night"]) / (X["close_p"] * X["F_prev"]) - 1
X["R_bar"] = (X["bar_o_night"] * X["F_night"]) / (X["bar_c_prev"] * X["F_prev"]) - 1
X["short"] = X["R_bt"] - X["R_real"]
X["yahoo_vs_bar"] = X["R_bt"] - X["R_bar"]
X["bar_vs_cross"] = X["R_bar"] - X["R_real"]
X["close_cross_vs_bar"] = X["close_p"] / X["bar_c_prev"] - 1 # same day, raw: + = paid more than the bar close
X["open_bar_vs_cross"] = X["bar_o_night"] / X["open_p"] - 1 # same day, raw: + = sold below the bar open
X["listing"] = np.where(X["open_x"].isin(["N"]) | X["close_x"].isin(["N"]), "NYSE",
np.where(X["open_x"].isin(["Q", "T"]) | X["close_x"].isin(["Q", "T"]), "Nasdaq", "other"))
ok = X[["R_bt", "R_real"]].notna().all(axis=1) & (X["short"].abs() < 0.05)
res = {"coverage": {}, "candidates": {}}
for k in ("sleeve_score", "overnight_12_1"):
g = X[X["candidate"] == k]
res["coverage"][k] = {"name_nights": int(len(g)), "no_close_cross": int(g["close_p"].isna().sum()),
"no_open_cross": int(g["open_p"].isna().sum()),
"no_yahoo_or_factor": int((g[["R_bt", "F_prev", "F_night"]].isna().any(axis=1)).sum()),
"outliers_over_5pct": int((g["short"].abs() >= 0.05).sum())}
print("coverage:", json.dumps(res["coverage"]))
for k in ("sleeve_score", "overnight_12_1"):
g = X[(X["candidate"] == k) & ok]
d = {c: night_stats(g[c], g["night"]) for c in ("short", "yahoo_vs_bar", "bar_vs_cross",
"close_cross_vs_bar", "open_bar_vs_cross")}
d["cost_per_side_bp"] = d["short"]["mean_bp"] / 2 + SEC_FEE_SIDE_BP
d["by_year"] = {int(y): night_stats(h["short"], h["night"]) for y, h in g.groupby(g["night"].dt.year)}
d["by_period"] = {lab: night_stats(h["short"], h["night"]) for lab, h in
(("2016-20", g[g["night"] < "2021-01-01"]), ("2021-26", g[g["night"] >= "2021-01-01"]))}
d["by_listing"] = {lab: night_stats(h["short"], h["night"]) for lab, h in g.groupby("listing")}
res["candidates"][k] = d
s = d["short"]
print(f"\n== {k}: {s['name_nights']} name-nights over {s['nights']} nights")
print(f" shortfall R_bt - R_real: mean {s['mean_bp']:+.2f} bp/night (SE {s['se_bp']:.2f}), median "
f"{s['median_bp']:+.2f}, trimmed {s['trimmed_mean_bp']:+.2f} -> cost per side {d['cost_per_side_bp']:+.2f} bp")
for c in ("yahoo_vs_bar", "bar_vs_cross", "close_cross_vs_bar", "open_bar_vs_cross"):
print(f" {c:20s} mean {d[c]['mean_bp']:+.2f} bp (SE {d[c]['se_bp']:.2f}) median {d[c]['median_bp']:+.2f}")
print(" by year:", " ".join(f"{y} {v['mean_bp']:+.1f}±{v['se_bp']:.1f}" for y, v in d["by_year"].items()))
print(" by period:", {a: round(v["mean_bp"], 2) for a, v in d["by_period"].items()},
" by listing:", {a: (round(v["mean_bp"], 2), v["name_nights"]) for a, v in d["by_listing"].items()})
cost = res["candidates"]["sleeve_score"]["cost_per_side_bp"]
verdict = "PASS" if cost <= PASS_BP else ("MEASURED" if cost <= STOP_BP else "STOP")
res["verdict"] = {"cost_per_side_bp": cost, "decision": verdict}
print(f"\nverdict: cost per side {cost:+.2f} bp -> {verdict}")
with open(res_path, "w") as fh:
json.dump(res, fh, indent=2)
X.to_csv(os.path.join(OUT, "name_nights.csv.gz"), index=False)
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("cmd", nargs="?", default="run", choices=["run", "fetch"])
if ap.parse_args().cmd == "fetch":
fetch()
else:
main()

View File

@@ -106,6 +106,10 @@ OCC = re.compile(r"^[A-Z]{1,5}\d{6,7}[CP]\d{8}$") # what the bars endpoint acc
BAR_COLS = ["date", "underlying", "symbol", "type", "expiry", "strike", "o", "h", "l", "c", "v", "n", "vw"]
class BackendTimeout(RuntimeError):
"""HTTP 504 on a request for many symbols: the caller splits the batch."""
class Api:
"""Paced GET with retries. ``yield_to_snapshot`` sleeps through 16:10-16:55
New York on weekdays so a backfill leaves the rate limit to the daily run."""
@@ -140,6 +144,8 @@ class Api:
time.sleep(2 ** attempt)
continue
self.n += 1
if r.status_code == 504 and params.get("symbols", "").count(",") >= 50:
raise BackendTimeout(r.text[:200]) # the same batch times out again (SNDK, 1000 contracts)
if r.status_code == 429 or r.status_code >= 500:
time.sleep(2 ** attempt)
continue
@@ -238,14 +244,23 @@ def option_bars(api: Api, symbols: list[str], start: str, end: str, timeframe: s
e = (dt.datetime.combine(dt.date.fromisoformat(end), dt.time(0), NY) if len(end) == 10
else dt.datetime.fromisoformat(end))
end = min(e, dt.datetime.now(NY) - dt.timedelta(minutes=16)).isoformat()
def fetch(chunk: list[str]) -> list[dict]:
params = {"symbols": ",".join(chunk), "timeframe": timeframe, "start": start, "end": end, "limit": 10000}
got = []
try:
for bars in api.pages(f"{DATA}/v1beta1/options/bars", params, "bars"):
for sym, bl in bars.items():
got += [{"symbol": sym, "t": b["t"], "o": b["o"], "h": b["h"], "l": b["l"], "c": b["c"],
"v": b["v"], "n": b.get("n"), "vw": b.get("vw")} for b in bl]
except BackendTimeout: # restart the chunk as two halves
h = len(chunk) // 2
return fetch(chunk[:h]) + fetch(chunk[h:])
return got
rows = []
for i in range(0, len(symbols), 1000):
params = {"symbols": ",".join(symbols[i:i + 1000]), "timeframe": timeframe, "start": start, "end": end,
"limit": 10000}
for bars in api.pages(f"{DATA}/v1beta1/options/bars", params, "bars"):
for sym, bl in bars.items():
rows += [{"symbol": sym, "t": b["t"], "o": b["o"], "h": b["h"], "l": b["l"], "c": b["c"],
"v": b["v"], "n": b.get("n"), "vw": b.get("vw")} for b in bl]
rows += fetch(symbols[i:i + 1000])
return rows
@@ -562,6 +577,7 @@ def backfill_one(api: Api, u: str, start: str, end: str, lte_active: str, hist:
+ dt.timedelta(days=1)).isoformat()), g)
lo = (pd.to_datetime(f["expiry"]) - pd.Timedelta(days=DTE_MAX)).dt.strftime("%Y-%m-%d")
frames.append(f[(f["date"] >= lo) & (f["date"] <= f["expiry"]) & (f["date"] <= end)])
frames = [f for f in frames if len(f)]
bars = pd.concat(frames) if frames else pd.DataFrame(columns=BAR_COLS)
_atomic_csv(bars, os.path.join(hist, f"{u}.csv.gz"))
rec = {"underlying": u, "contracts": int(len(meta)), "bars": int(len(bars)), "requests": api.n - n0,

View File

@@ -0,0 +1,442 @@
"""Fixed-budget, lagged futures signals for stock exposure; research only.
uv run python -m research.lab.futures_stock_signals --output data/research/results/futures_stock_signals_20261017
"""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import pickle
import platform
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
import pandas as pd
import metrics
from backtest_runner import ENGINE_VERSION, prepare_strategy, replay_prepared
from research.lab.fetch_long_history import _cached
from research.lab.overnight_only import legs, nightly, top_mask
from research.lab.trials import append_trials
from strategies.robust_momentum import signal_overnight_recovery
START, END = "2006-10-02", "2026-09-24"
BLOCKS = [(START, "2012-12-31"), ("2013-01-01", "2019-12-31"), ("2020-01-01", END)]
POLICIES = ("equity_trend", "equity_shock", "rates_shock")
SOURCES = {"futures": ("ES=F", "NQ=F", "ZN=F"), "etf": ("SPY", "QQQ", "IEF")}
SETTINGS = ("default", "neighbor_low", "neighbor_high", "delay_two", "jump_masked")
PANEL = Path("data/research/us.panel.pkl")
SEED = 20261017
def clean(value):
if isinstance(value, dict):
return {str(k): clean(v) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [clean(v) for v in value]
if isinstance(value, (np.integer, np.floating)):
value = value.item()
if isinstance(value, float) and not math.isfinite(value):
return None
return value
def write_json(path, value):
Path(path).write_text(json.dumps(clean(value), indent=2, allow_nan=False) + "\n")
def configurations():
out = [{"name": "baseline"}, {"name": "static_75"}]
for policy in POLICIES:
for setting in SETTINGS:
for source in SOURCES:
look = ({"neighbor_low": 10, "neighbor_high": 40}.get(setting, 20)
if policy == "equity_trend" else 20)
threshold = {"neighbor_low": 1.0, "neighbor_high": 2.0}.get(setting, 1.5)
out.append({"name": f"{source}_{policy}_{setting}", "source": source,
"policy": policy, "setting": setting, "look": look,
"threshold": threshold, "lag": 2 if setting == "delay_two" else 1,
"mask_jumps": setting == "jump_masked"})
return out
def risk_exposure(prices: pd.DataFrame, config: dict) -> tuple[pd.Series, pd.Series]:
"""Row t trades at close t; the newest observable daily bar is t-lag.
Missing features leave the baseline unchanged and are counted as missing,
rather than interpreting an unavailable observation as a bearish signal.
"""
if config["name"] in ("baseline", "static_75"):
value = 1.0 if config["name"] == "baseline" else .75
return pd.Series(value, index=prices.index), pd.Series(True, index=prices.index)
names = SOURCES[config["source"]]
ret = np.log(prices[list(names)] / prices[list(names)].shift(1))
if config["mask_jumps"]:
scale = ret.rolling(63, min_periods=63).std().shift(1)
ret = ret.mask(ret.abs() > 6 * scale)
equity = ret.iloc[:, :2].mean(axis=1).where(ret.iloc[:, :2].notna().all(axis=1))
trend = equity.rolling(config["look"], min_periods=config["look"]).sum()
if config["policy"] == "equity_trend":
valid = trend.notna()
risk = (trend < 0).where(valid)
else:
x = equity if config["policy"] == "equity_shock" else ret.iloc[:, 2]
scale = x.rolling(63, min_periods=63).std().shift(1) * np.sqrt(5)
shock = x.rolling(5, min_periods=5).sum() / scale.where(scale > 0)
valid = shock.notna()
risk = shock < -config["threshold"]
if config["policy"] == "rates_shock":
valid &= trend.notna()
risk &= trend < 0
risk = risk.where(valid)
# Hold the reduction for five sessions after a shock to limit churn.
risk = risk.astype(float).rolling(5, min_periods=1).max().where(valid)
risk = risk.shift(config["lag"])
known = risk.notna()
return 1.0 - .5 * risk.fillna(0).astype(float), known
def overnight_exposure(entry_exposure: pd.Series) -> pd.Series:
# A return dated t is earned close t-1 -> open t, so the entry decision
# must already exist on t-1. This is separate from the source-data lag.
return entry_exposure.shift(1).fillna(1.0)
def describe(returns: pd.Series) -> dict:
r = returns.dropna()
anchor = r.index[0] - pd.Timedelta(days=1)
equity = pd.concat([pd.Series([1.0], index=[anchor]), (1 + r).cumprod()])
s = metrics.raw_summary(equity)
return {"cagr": s["annualizedReturn"], "sharpe": s["sharpeRatio"],
"vol": s["annualizedVolatility"], "mdd": s["maxDrawdown"],
"worst_day": float(r.min()), "es_5pct": float(r[r <= r.quantile(.05)].mean()),
"sessions": len(r),
"blocks": [describe_basic(r.loc[a:b]) for a, b in BLOCKS],
"later_period": describe_basic(r.loc["2017-01-01":]),
"yearly": {str(y): float((1 + g).prod() - 1) for y, g in r.groupby(r.index.year)}}
def describe_basic(r: pd.Series) -> dict:
anchor = r.index[0] - pd.Timedelta(days=1)
s = metrics.raw_summary(pd.concat([pd.Series([1.0], index=[anchor]), (1 + r).cumprod()]))
return {"cagr": s["annualizedReturn"], "sharpe": s["sharpeRatio"], "mdd": s["maxDrawdown"]}
def paired_sharpe_interval(candidate, control, repetitions=2000):
pair = pd.concat([candidate, control], axis=1).dropna().to_numpy()
n = len(pair)
rng = np.random.default_rng(SEED)
differences = []
for _ in range(repetitions):
starts = rng.integers(0, n - 20 + 1, size=math.ceil(n / 20))
idx = (starts[:, None] + np.arange(20)).ravel()[:n]
sample = pair[idx]
sharpe = sample.mean(axis=0) / sample.std(axis=0, ddof=1) * np.sqrt(252)
differences.append(sharpe[0] - sharpe[1])
return {"lower": float(np.quantile(differences, .025)),
"upper": float(np.quantile(differences, .975)),
"block_sessions": 20, "repetitions": repetitions, "seed": SEED,
"role": "paired descriptive interval; dependent on historical stationarity, not independent OOS"}
def data_inventory(prices, dates):
out = {}
for name, p in prices.items():
x = p.reindex(dates)
r = np.log(x / x.shift(1))
jump = r.abs() > 6 * r.rolling(63, min_periods=63).std().shift(1)
out[name] = {"first": str(p.first_valid_index().date()), "last": str(p.last_valid_index().date()),
"coverage": float(x.loc[START:END].notna().mean()), "jump_days": int(jump.sum())}
return out
def protocol(destination, configs):
source_paths = [Path(__file__), Path("execution.py"), Path("signals.py"), Path("backtest_runner.py"),
Path("metrics.py"), Path("trader.py"), Path("research/lab/overnight_only.py"),
Path("strategies/base.py"), Path("strategies/robust_momentum.py"),
Path("tests/test_futures_stock_signals.py")]
data_paths = [PANEL] + [Path(_cached(t)) for tickers in SOURCES.values() for t in tickers]
missing = [str(p) for p in data_paths if not p.exists()]
if missing:
raise FileNotFoundError(f"Required cached data missing: {missing}")
record = {
"created_at": datetime.now(timezone.utc).isoformat(),
"claim": "Lagged futures information improves stock exposure beyond matched ETF signals and simple de-risking.",
"primary": "futures_equity_shock_default on overnight_recovery_top10",
"hypotheses": {"incremental_information": "Futures beats both baseline and the same ETF rule after costs.",
"redundant_or_stale": "ETF control matches it, or costs/delay remove the benefit."},
"span": [START, END], "blocks": BLOCKS,
"role": "All historical periods previously inspected; fixed-rule development study, not independent OOS.",
"design_verdict": "APPROVED TO RUN: user authorized local research and backtests; one local process, no paid data.",
"baseline": "production RobustMomentumStrategy overnight_recovery_top10, PIT panel, unchanged target stock list",
"secondary": "nightly top10 by the same score through entry-day minus one; MOC -> next open, flat intraday",
"engine": ENGINE_VERSION, "capital": 10000, "integer_shares": True,
"daily_costs": [.001, .0025], "daily_fixed_fee": 2.0, "resize_band": .1,
"overnight_costs": [.0001, .0003], "overnight_fixed_fee": 0.0,
"cash_interest": 0.0, "sharpe": "mean daily return / sample standard deviation * sqrt(252), no rf subtraction",
"controlled": "identical stock selections, data, timing, capital and per-engine fees for all exposure rules",
"configurations": configs,
"budget": {"profiles": len(configs), "engines": 2, "costs_per_engine": 2,
"backtests": len(configs) * 4, "bootstrap_repetitions": 2000},
"rules": {"trend": "mean ES/NQ (or SPY/QQQ) log return over 20 sessions < 0",
"shock": "5-session return / (prior 63-session daily vol * sqrt(5)) < -1.5",
"rates": "ZN (or IEF) shock AND equity trend < 0",
"exposure": "0.5 in risk state, 1 otherwise; shock risk held five sessions",
"neighbors": "trend 10/40 sessions; shock threshold 1.0/2.0",
"timing": "entry close t uses source data <= t-1; overnight exit t additionally shifts exposure one session",
"missing": "no backward fill; missing signal preserves baseline, coverage recorded; require >=99% usable signals",
"jumps": "default untouched; sensitivity masks >6 times trailing63 volatility without using future observations"},
"gates": {"G1": "full-span Sharpe >= baseline +0.10 at both costs",
"G2": "full-span Sharpe >= matched ETF +0.05 AND static75 at both costs",
"G3": "CAGR >= baseline -2pp AND MaxDD improves >=3pp at both costs",
"G4": "beats baseline and ETF Sharpe in >=2/3 blocks AND later2017+ at both costs",
"G5": "base-cost later2017+ paired block-bootstrap 95% lower bound of Sharpe delta >0 vs baseline AND ETF",
"G6": "base-cost Sharpe delta vs ETF positive for both neighbors, lag2 and jump-masked variants",
"G7": "default source-feature availability >=99%"},
"decisions": "All gates pass -> candidate for forward validation only; else retain baseline. No live changes.",
"excluded": {"basis": "continuous contracts lack contract maturity, synchronous spot, rate and dividend inputs",
"positions": "no release-time-verified historical futures OI/COT panel",
"premarket": "daily open/close cache cannot reconstruct premarket decision-time quotes",
"industry_selection": "separate next-stage test; this study only changes total stock exposure"},
"known_limits": ["Yahoo continuous contracts are not roll-adjusted; jump masking is sensitivity, not reconstruction.",
"Futures/ETF timestamps and duration differ; lagged inputs do not isolate pure price discovery.",
"Historical PIT member price coverage is incomplete, especially before 2010.",
"Adjusted-share prices and delisting settlement assumptions follow the existing ledger.",
"Overnight open fallback to same-day close is counted and tested with zero-return sensitivity.",
"No independent forward sample; descriptive bootstrap does not resolve model-selection bias."],
"references": ["https://www.cmegroup.com/education/articles-and-reports/faq-daily-settlement-price-determination-time-change.html",
"https://www.cmegroup.com/trading/equity-index/fairvalue.html",
"https://www.cftc.gov/MarketReports/CommitmentsofTraders/index.htm"],
"environment": {"python": sys.version, "platform": platform.platform(), "numpy": np.__version__,
"pandas": pd.__version__, "command": sys.argv,
"git": subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()},
"source_hashes": {str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in source_paths},
"data_hashes": {str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in data_paths},
}
destination.mkdir(parents=True, exist_ok=False)
write_json(destination / "protocol.json", record)
(destination / "experiment_card.md").write_text(
"# Experiment: Lagged Futures Information for Stock Exposure\n\n"
"> Status: Approved for the user-authorized local research run.\n\n"
"## Claim and Decision\n\n"
+ record["claim"] + "\n\n"
"Futures-information hypothesis predicts an improvement over the unchanged stock strategy, "
"matched ETF rule and static 75% exposure. The redundancy hypothesis predicts that the ETF "
"rule matches it or costs and a further one-session lag remove the benefit. "
"All seven fixed gates must pass before recommending a forward experiment; no production promotion.\n\n"
"## Setup\n\n"
"Only exposure changes. Production stock targets, PIT panel, capital and fees stay fixed. "
"Separate production daily-close ledger and nightly MOC/MOO models have their own matched controls. "
"Three fixed signal families, two neighboring parameters, lag2 and causal jump masking; "
"128 runs total. All parameter values and gates are in protocol.json, written before any evaluation. "
"No fresh OOS label is assigned to previously inspected history.\n\n"
"## Expected Artifacts and Review\n\n"
"Daily returns, exposures, production-ledger trades, costs, subperiods, paired 20-session "
"block-bootstrap intervals and complete trial records. Results are tables; no result figure planned. "
"Local low-cost execution is authorized by the user's instruction to start research and backtests.\n\n"
"## Reproduction Details\n\n"
"protocol.json records the command, runtime versions, Git revision and source/data SHA256. "
"No downloads, paid sources, shared-host jobs or live-account writes.\n\n"
"## Benchmark Audit\n\n"
"| Issue | Verdict | Evidence |\n|---|---|---|\n"
"| Timing | PASS | Source lag1 and extra entry-to-exit shift for overnight returns; causal tests |\n"
"| Fair baselines | PASS | Same stocks, costs and execution; ETF and static-exposure controls |\n"
"| Search budget | PASS | Fixed 32 profiles and both costs before execution; all runs retained |\n"
"| Uncertainty | PASS | Paired block bootstrap and three chronological blocks |\n"
"| Independent OOS | NEEDS EVIDENCE | All historical periods previously inspected |\n"
"| Continuous-contract data | NEEDS EVIDENCE | No contract/roll history; use sensitivity, limit claims |\n"
"| Serving workload | N/A | Financial backtest, no serving system or request trace |\n\n"
"## Results\n\nRecorded after execution in results.json and report.md.\n")
return record
def decisions(rows, curves, known):
out = []
for engine, costs in (("daily", (.001, .0025)), ("overnight", (.0001, .0003))):
indexed = {(r["profile"], r["cost"]): r for r in rows if r["engine"] == engine}
for policy in POLICIES:
name, control = f"futures_{policy}_default", f"etf_{policy}_default"
checks = {f"G{i}": True for i in range(1, 8)}
comparisons = []
for cost in costs:
f, e, b, s = (indexed[(k, cost)] for k in (name, control, "baseline", "static_75"))
checks["G1"] &= f["sharpe"] >= b["sharpe"] + .10
checks["G2"] &= f["sharpe"] >= e["sharpe"] + .05 and f["sharpe"] >= s["sharpe"]
checks["G3"] &= f["cagr"] >= b["cagr"] - .02 and f["mdd"] >= b["mdd"] + .03
blocks = sum(x["sharpe"] > max(y["sharpe"], z["sharpe"])
for x, y, z in zip(f["blocks"], b["blocks"], e["blocks"]))
checks["G4"] &= blocks >= 2 and f["later_period"]["sharpe"] > max(
b["later_period"]["sharpe"], e["later_period"]["sharpe"])
comparisons.append({"cost": cost, "sharpe_delta_baseline": f["sharpe"] - b["sharpe"],
"sharpe_delta_etf": f["sharpe"] - e["sharpe"], "blocks_better_than_both": blocks})
intervals = {}
for other in ("baseline", control):
intervals[other] = paired_sharpe_interval(
curves[f"{engine}|{name}|{costs[0]}"] .loc["2017-01-01":],
curves[f"{engine}|{other}|{costs[0]}"] .loc["2017-01-01":])
checks["G5"] = all(c["lower"] > 0 for c in intervals.values())
sensitivities = {}
for setting in SETTINGS[1:]:
f, e = (indexed[(f"{source}_{policy}_{setting}", costs[0])] for source in SOURCES)
sensitivities[setting] = f["sharpe"] - e["sharpe"]
checks["G6"] = all(v > 0 for v in sensitivities.values())
checks["G7"] = known[name] >= .99
out.append({"engine": engine, "policy": policy, "checks": checks,
"comparisons": comparisons, "later_period_bootstrap": intervals,
"sensitivity_sharpe_delta_etf": sensitivities,
"decision": "forward_candidate_only" if all(checks.values()) else "retain_baseline"})
return out
def report(destination, result):
labels = {"baseline": "原策略", "static_75": "固定75%仓位",
"futures_equity_trend_default": "股指期货20日趋势", "etf_equity_trend_default": "对应ETF20日趋势",
"futures_equity_shock_default": "股指期货5日急跌", "etf_equity_shock_default": "对应ETF5日急跌",
"futures_rates_shock_default": "国债期货急跌+股指走弱", "etf_rates_shock_default": "对应ETF国债急跌+股指走弱"}
passed = sum(d["decision"] == "forward_candidate_only" for d in result["decisions"])
text = ["# 期货信号辅助股票仓位:固定规则回测", "",
f"结论:三类期货信号、两种持有方式共6个默认方案,{passed}个通过预定标准。", "",
"本研究只改变股票总仓位,未交易期货,也未改变股票评分。",
"完成32种配置×2种持有方式×2档成本,共128组预先固定的回测。",
"全部历史区间已在过去研究中查看;结果属于开发证据,不能称为独立样本外或实盘收益。", "",
f"区间:{START} 至 {END}。指标中的回撤为负数;Sharpe 未减无风险利率。", ""]
for engine, cost in (("daily", .001), ("overnight", .0001)):
text += ["## " + ("连续持仓版:原股票策略" if engine == "daily" else "隔夜版:收盘买、次日开盘卖"), "",
"| 版本 | 年化收益 | Sharpe | 最大回撤 | 平均目标仓位 |", "|---|---:|---:|---:|---:|"]
for row in result["rows"]:
if row["engine"] == engine and row["cost"] == cost and row["profile"] in labels:
text.append(f"| {labels[row['profile']]} | {row['cagr']:.2%} | {row['sharpe']:.3f} | "
f"{row['mdd']:.2%} | {row['mean_target_exposure']:.1%} |")
text.append("")
text += ["连续持仓版:生产执行账本、$10,000、整股、单边10/25 bp及每单$2,10%调仓带。",
"隔夜版:每日全额往返、单边1/3 bp、无固定佣金、分数股;两种引擎分别比较,不互相排名。", "",
"## 验收", "", "| 引擎 | 信号 | 未通过门槛 | 结论 |", "|---|---|---|---|"]
for d in result["decisions"]:
failed = ", ".join(k for k, v in d["checks"].items() if not v) or "无"
text.append(f"| {d['engine']} | {d['policy']} | {failed} | {d['decision']} |")
text += ["", "G1:超越原策略;G2:超越ETF及简单降仓;G3:收益和回撤门槛;G4:分段稳定;",
"G5:2017年后配对分块bootstrap区间;G6:邻近参数、延迟及跳变敏感性;G7:数据覆盖。", "",
"bootstrap区间为名义95%区间,未作多重比较校正,不能代替独立前向验证。", "",
"## 数据与边界", "", "| 默认期货特征 | 可用率 |", "|---|---:|"]
for policy in POLICIES:
key = f"futures_{policy}_default"
text.append(f"| {labels[key]} | {result['audit']['signal_coverage'][key]:.2%} |")
text += ["", "原始日线有缺口,滚动窗口会进一步放大特征不可用范围;覆盖情况见data_audit.json。",
f"隔夜模型有{result['audit']['overnight_missing_open_nights']}个夜晚涉及个别选中股票缺开盘价;",
"已将这些股票的该夜收益设为0做敏感性检查,完整结果列于results.json。",
"早期指数成员价格覆盖不足;逐年观测成员数列于data_audit.json。", "",
"未测试真正期现基差、期货持仓/COT、盘前分钟信号和行业重排;对应数据缺口列于protocol.json。",
"下一步优先取得带明确时间戳的期货分钟数据、逐月合约及持仓发布记录,再检验信息增量。", "",
"原始日线、源代码及股票面板的SHA256列于protocol.json;完整参数与成本结果见results.csv、",
"results.json;逐日收益、目标仓位见daily_returns.csv.gz和exposure.csv;生产账本成交见trades.csv.gz。", ""]
ablation = destination / "availability_results.json"
if ablation.exists():
a = json.loads(ablation.read_text())
text += ["## 数据缺口匹配诊断", "",
"主实验结束后另固定12组诊断:在期货特征不可用的日期,让ETF版本也保持原仓位。",
"该诊断只解释数据可用性,不参与方案选择。下表是期货Sharpe减去此ETF对照的Sharpe。", "",
"| 信号 | 连续持仓低成本档 | 隔夜低成本档 |", "|---|---:|---:|"]
for policy in POLICIES:
selected = [r for r in a["rows"] if r["profile"] == f"etf_{policy}_availability_matched"
and r["cost"] == (.001 if r["engine"] == "daily" else .0001)]
delta = {r["engine"]: r["futures_sharpe_minus_matched_etf"] for r in selected}
text.append(f"| {labels[f'futures_{policy}_default']} | {delta['daily']:+.4f} | {delta['overnight']:+.4f} |")
text += ["", "完整两档成本及仅在可观测日期计算的信号一致率见availability_results.json。", ""]
if (destination / "execution_source.py").exists():
text += ["首次执行的脚本保存在execution_source.py,匹配预先登记的源代码SHA256;后续仅改善报告排版。", ""]
text += ["复现:", "", "```bash",
f"uv run python -m research.lab.futures_stock_signals --output {destination}_rerun", "```", ""]
(destination / "report.md").write_text("\n".join(text))
def run(destination: Path):
configs = configurations()
record = protocol(destination, configs)
print(f"Protocol saved before evaluation: {destination}; budget {record['budget']}", flush=True)
with PANEL.open("rb") as fh:
p = pickle.load(fh)
prices = pd.DataFrame({t: pd.read_csv(_cached(t), index_col=0, parse_dates=True)["Close"]
for tickers in SOURCES.values() for t in tickers})
prices = prices.reindex(p.dates)
if p.dates[-1] < pd.Timestamp(END):
raise ValueError("Stock panel ends before the fixed evaluation interval")
panels = {field: getattr(p, field) for field in ("open", "high", "low", "close", "volume")}
panels["close"] = p.close.join(p.extra["etf"][["SPY"]])
plan = prepare_strategy("overnight_recovery_top10", "us", panels, list(p.close.columns),
member=p.member, start=START, end=END)
score = signal_overnight_recovery({"open": p.open, "close": p.close,
"eligible": p.member & p.close.notna()})
selected = top_mask(score.shift(2), 10)
overnight, _ = legs(p)
gross_night = nightly(selected, overnight).loc[START:END]
missing_opens = selected & p.open.isna()
zero_open_sensitivity = nightly(selected, overnight.mask(p.open.isna(), 0.0)).loc[START:END]
entries = {}
known = {}
for config in configs:
exposure, valid = risk_exposure(prices, config)
entries[config["name"]] = exposure
known[config["name"]] = float(valid.loc[START:END].mean())
pd.DataFrame(entries).loc[START:END].to_csv(destination / "exposure.csv")
audit = {"daily_bar_inventory": data_inventory(prices, p.dates), "signal_coverage": known,
"pit_mean_observed_members_by_year": p.member.sum(axis=1).groupby(p.dates.year).mean().to_dict(),
"overnight_selected_missing_opens": int(missing_opens.loc[START:END].sum().sum()),
"overnight_missing_open_nights": int(missing_opens.loc[START:END].any(axis=1).sum()),
"max_gross_return_change_zero_open_sensitivity": float((gross_night - zero_open_sensitivity).abs().max())}
write_json(destination / "data_audit.json", audit)
rows, curves, trades = [], {}, []
for config in configs:
name = config["name"]
entry = entries[name].loc[START:END]
daily_plan = dict(plan, targets=plan["targets"].mul(entry, axis=0))
for cost in record["daily_costs"]:
ledger = replay_prepared(daily_plan, 10000, cost, 2.0, integer_shares=True, resize_band=.1)
r = ledger["equity"].pct_change().dropna()
if ledger["cash"].min() < -1e-7:
raise AssertionError("Unfunded replay")
row = {"engine": "daily", "profile": name, "cost": cost, **describe(r),
"mean_target_exposure": float(entry.mean()),
"mean_actual_exposure": float((1 - ledger["cash"] / ledger["equity"]).dropna().mean()),
"orders": len(ledger["trade_log"]),
"fees": sum(t["commission"] for t in ledger["trade_log"]),
"minimum_cash": float(ledger["cash"].min())}
rows.append(row)
curves[f"daily|{name}|{cost}"] = r
trades.extend({"profile": name, "cost": cost, **t} for t in ledger["trade_log"])
append_trials(destination / "trials.jsonl", [row], {"study": "futures_stock_signals", "config": config})
print(f"daily {name:43s} {cost:.4f}: CAGR {row['cagr']:.2%} Sh {row['sharpe']:.3f} DD {row['mdd']:.2%}", flush=True)
night_exp = overnight_exposure(entries[name]).reindex(gross_night.index)
for cost in record["overnight_costs"]:
r = night_exp * (gross_night - 2 * cost)
sensitivity = describe_basic(night_exp * (zero_open_sensitivity - 2 * cost))
row = {"engine": "overnight", "profile": name, "cost": cost, **describe(r),
"mean_target_exposure": float(night_exp.mean()), "zero_open_sensitivity": sensitivity}
rows.append(row)
curves[f"overnight|{name}|{cost}"] = r
append_trials(destination / "trials.jsonl", [row], {"study": "futures_stock_signals", "config": config})
verdicts = decisions(rows, curves, known)
result = {"protocol": record, "audit": audit, "rows": rows, "decisions": verdicts}
write_json(destination / "results.json", result)
pd.DataFrame([{k: v for k, v in r.items() if k not in ("blocks", "later_period", "yearly", "zero_open_sensitivity")}
for r in rows]).to_csv(destination / "results.csv", index=False)
pd.DataFrame(curves).to_csv(destination / "daily_returns.csv.gz", compression="gzip")
pd.DataFrame(trades).to_csv(destination / "trades.csv.gz", index=False, compression="gzip")
report(destination, result)
append_trials(Path("data/research/results/trials_us.jsonl"),
[{"name": f"futures_stock_signals:{r['engine']}:{r['profile']}", "cost": r["cost"],
"cagr": r["cagr"], "sharpe": r["sharpe"], "mdd": r["mdd"]} for r in rows],
{"study": "futures_stock_signals", "role": record["role"], "path": str(destination)})
print(json.dumps(clean(verdicts), indent=2), flush=True)
print(f"Saved {destination / 'report.md'}", flush=True)
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
run(parser.parse_args().output)

View File

@@ -0,0 +1,280 @@
"""
Can a book that holds no leveraged ETF beat QQQ? The live account (2026-10-13)
is $10k at IBKR Pro without the Complex or Leveraged ETP permission, trades
whole shares only (IBKR's API refuses fractional orders), and may borrow on
Reg T margin (at most 2x). The QQQ trend sleeve of us_momentum_qqq_trend held
TQQQ; here it holds QQQ itself and borrows for exposure above 1x.
PYTHONPATH=. uv run python -m research.lab.fetch_long_history # once
PYTHONPATH=. uv run python -m research.lab.no_letf_book
Constructions (fixed before the run; no parameter is searched):
qqq QQQ total return, buy and hold: the yardstick
qqq_trend_1x E = 1{QQQ > 200-day average} x min(1.0, 30% / sigma) in QQQ, the
rest in SHY; sigma = QQQ's EWMA vol (20-day half-life). The rule of
strategies/qqq_trend.py (QQQVolTrend) with the cap cut to 1x
qqq_trend_m15 the same with QQQVolTrend's own cap 1.5: exposure above 1x is
borrowed at T-bill + 1.5% (IBKR Pro under $100k)
momentum overnight_recovery_top10 held all day (stream_portfolio streams.pkl)
mom_qqq_1x momentum + qqq_trend_1x, equal capital, month-start reset when a
share is more than 25% off (managed_futures_book.band_blend)
mom_qqq_m15 momentum + qqq_trend_m15 (book exposure at most 1.25x)
Trend fills: decided on closes through t-1, traded at the close of t ("lag" in
research/lab/vol_trend.py, the live MOC timing); the book trades back to target
when the signal flips or E is more than 25% off; 10 bp per unit traded. Every
construction also pays $1 an order on a $10k account (IBKR Pro Fixed, the
commission the paper account's what-if returned): 2 orders a rebalance for the
trend (QQQ and SHY legs) and 142 a year for momentum (the web engine's 1,250
orders over 2017-12..2026-10).
Risk-matched reading: each construction is scaled by L* = (QQQ's volatility /
its own) over the design period, capped so that gross exposure stays within
Reg T's 2x; L* above 1 borrows at T-bill + 1.5%, below 1 leaves cash at the
T-bill rate. L* is fixed on the design period and applied unchanged to the
holdout.
Periods: QQQ-only constructions design 1990-2016, holdout 2017-2026-09 (QQQ
from ^NDX + 0.5%/yr dividends before 1999-03, research/lab/letf_history.py);
momentum constructions design 2006-10..2016, holdout 2017..2026-09. The
momentum signal was chosen on this panel, the panel is survivor-biased before
about 2014, and us_momentum_qqq_trend's TQQQ version was seen on 2026-09-29:
the holdout is not independent out-of-sample evidence.
"Beats QQQ" needs all three:
G1 excess Sharpe above QQQ's by at least 0.10 in the design period and the holdout
G2 MaxDD over the construction's span shallower than QQQ's over the same span
G3 risk-matched holdout CAGR above QQQ's holdout CAGR
The rule is written to prereg.json before the run; the script refuses to
overwrite results.json.
Result (2026-10-13): NONE BEATS QQQ. G1 (Sharpe +0.10 in both periods) fails
for every construction; the QQQ trends pass G2 and G3.
construction from CAGR Sharpe design / holdout Sharpe MaxDD L* matched holdout CAGR
qqq (yardstick) 1990 14.9% 0.56 0.47 / 0.86 -83% 1.00 21.6%
qqq_trend_1x 1990 12.1% 0.60 0.51 / 0.85 -28% 1.57 22.1%
qqq_trend_m15 1990 13.8% 0.57 0.47 / 0.85 -36% 1.24 24.0%
momentum 2006 18.6% 0.64 0.54 / 0.77 (QQQ .60/.86) -69% (QQQ -53%) 0.62 16.1%
mom_qqq_1x 2006 14.9% 0.69 0.55 / 0.84 -45% 0.99 19.0%
mom_qqq_m15 2006 16.4% 0.70 0.57 / 0.84 -46% 0.91 19.6%
The momentum stream is phase-averaged and carries Tiingo's delisted names
(20.3% a year before the $1 orders). The single-phase series of beat_qqq.py is
6pp a year higher with corr 0.98, so us_momentum_qqq_trend's 27.9% vs 21.5%
for 2017-26 (2026-09-29) was optimistic.
Reported after the verdict (exploratory, not gated):
- The trend's gain is crash avoidance, not Sharpe. 2000-02: +0.2% vs QQQ
-73%; 2008: -8% vs -42%; 2022: -23% vs -33%. 14 orders a year.
- Margin frontier, qqq_trend_1x x L, 1990-2026, borrowing at T-bill + 1.5%
(charged even on the SHY leg, so conservative):
x1.5 14.9% (= QQQ), holdout 21.4%, MaxDD -40%
x2.0 17.0% vs 14.9%, holdout 26.1% vs 21.6%, MaxDD -50% vs -83%,
worst year -43% (2022)
- mom_qqq_1x x1.25 matches QQQ's 2006+ MaxDD (-53%) at 17.2% vs 16.6%.
- QQQ held overnight only (gross, Yahoo opens): Sharpe 0.59 / 1.08 / 1.26 /
0.80 in 2000-09 / 2010-16 / 2017-20 / 2021-26, intraday -0.35 / 0.35 /
0.53 / 0.40. Two orders a night cost 1.9% (IBKR Tiered) to 5% (Fixed) a
year at $10k.
- IBKR's API refuses fractional orders (error 10243, what-if 2026-10-13). At
$10k the momentum half holds $500 a name; 5 of the 2026-10-12 top 10 cost
more than that.
- Two checks run after that:
- The margin-levered trend does not beat QQQ outside the crash years. From
2010, x2 earns 16.4% at MaxDD -50% vs QQQ's 19.2% at -35%. From 2006 the
two are even: 17.1% / -50% vs 16.6% / -53%.
- What does beat QQQ in every window is the overnight momentum sleeve alone
(overnight_only.py sleeve_score, 1x, no leveraged ETF), provided orders
are cheap. Cost per side = 0.2 bp measured + $0.35 / order size + 0.1 bp:
venue / account 2006-26 CAGR / MaxDD / Sharpe 2010-26 2017-26
QQQ 16.6% / -53% / 0.73 19.2% / -35% / 0.88 21.6% / -35% / 0.86
$0 commission 35.7% / -30% / 1.40 29.9% / -30% / 1.32 34.2% / -30% / 1.31
IBKR Tiered $10k (3.8bp) 13.2% / -36% / 0.59 8.3% / -36% / 0.42 12.0% / -36% / 0.51
IBKR Tiered $25k (1.7bp) 25.9% / -31% / 1.06 20.4% / -31% / 0.95 24.5% / -31% / 0.98
IBKR Tiered $50k (1.0bp) 30.4% / -31% / 1.22 24.7% / -31% / 1.12 28.9% / -31% / 1.13
Same caveats as overnight_only.py: the signal was chosen on this panel,
the panel is survivor-biased before about 2014, and live fills are
untested.
"""
from __future__ import annotations
import json
import os
import pickle
import numpy as np
import pandas as pd
from research.lab.factor_scan import RESULTS
from research.lab.letf_history import SRC, build
from research.lab.levered_book import stats
from research.lab.managed_futures_book import band_blend
from research.lab.overnight_book import STREAMS
OUT = os.path.join(RESULTS, "no_letf_book")
END = "2026-09-24"
QQQ_SPAN, MOM_SPAN = ("1990-01-02", END), ("2006-10-02", END)
HOLDOUT = "2017-01-01"
MA, TARGET, HALF_LIFE = 200, 0.30, 20
COST, BAND = 0.001, 0.25
SPREAD = 0.015 # IBKR Pro margin: benchmark + 1.5% under $100k
ACCOUNT, PER_ORDER = 10_000.0, 1.0
MOMENTUM_ORDERS_A_YEAR = 142
REG_T = 2.0
PREREG = {
"question": "does a book without leveraged ETFs beat QQQ, raw and at QQQ's volatility",
"constructions": ["qqq", "qqq_trend_1x", "qqq_trend_m15", "momentum", "mom_qqq_1x", "mom_qqq_m15"],
"trend_rule": "1{QQQ > MA200} x min(cap, 30% / EWMA vol hl20), lag fills, 25% band, 10 bp a unit",
"costs": "$1 an order on $10k: trend 2 an rebalance, momentum 142 a year",
"financing": "T-bill + 1.5% on exposure above 1x; cash earns SHY (trend) or T-bill (scaling)",
"risk_matched": "L* = design-period vol(QQQ) / vol(construction), gross exposure <= 2x, fixed for the holdout",
"periods": {"qqq_only": [QQQ_SPAN[0], "2016-12-31", HOLDOUT, END],
"momentum": [MOM_SPAN[0], "2016-12-31", HOLDOUT, END]},
"gates": {"G1": "Sharpe >= QQQ + 0.10 in design and holdout",
"G2": "MaxDD shallower than QQQ over the same span",
"G3": "risk-matched holdout CAGR > QQQ holdout CAGR"},
"data_status": "momentum signal and the TQQQ version of the blend were examined before; not independent OOS",
}
def trend(syn: pd.DataFrame, rf: pd.Series, cap: float) -> tuple[pd.Series, pd.Series, pd.Series]:
"""Daily returns, exposure held and orders sent of the QQQ trend held in QQQ (margin above 1x)."""
px = syn["QQQ"]
r = px.pct_change().fillna(0.0)
on = (px > px.rolling(MA).mean()).astype(float)
sigma = np.sqrt((r ** 2).ewm(halflife=HALF_LIFE, min_periods=40).mean() * 252)
E = (on * (TARGET / sigma).clip(upper=cap)).fillna(0.0)
E.iloc[:MA] = 0.0
tgt = E.shift(2).fillna(0.0).to_numpy() # held over t: decided at t-2's close, traded at t-1's
rq, rs = r.to_numpy(), syn["SHY"].pct_change().fillna(0.0).to_numpy()
fin = ((rf.reindex(px.index).ffill().fillna(0.0) + SPREAD) / 252).to_numpy()
w = 0.0
out, held, orders = np.zeros(len(tgt)), np.zeros(len(tgt)), np.zeros(len(tgt))
for t in range(1, len(tgt)):
g = 1 + w * rq[t] + ((1 - w) * rs[t] if w <= 1 else -(w - 1) * fin[t])
out[t] = g - 1
w = w * (1 + rq[t]) / g
want = tgt[t + 1] if t + 1 < len(tgt) else w
if (want == 0) != (w == 0) or abs(want - w) > BAND * max(w, 1e-9):
legs = (want < 1 or w < 1) + 1 # the SHY leg trades too while either side holds cash
out[t] -= abs(want - w) * COST * legs
orders[t] = legs
w = want
held[t] = w
idx = px.index
return pd.Series(out, index=idx), pd.Series(held, index=idx), pd.Series(orders, index=idx)
def with_commissions(r: pd.Series, orders: pd.Series, scale: float = 1.0) -> pd.Series:
"""$1 an order, charged on the day it is sent, per dollar of a sleeve holding ``scale`` of the $10k account."""
return r - orders.reindex(r.index).fillna(0.0) * PER_ORDER / (ACCOUNT * scale)
def scaled(r: pd.Series, rf: pd.Series, L: float) -> pd.Series:
cash = rf.reindex(r.index).ffill().fillna(0.0) / 252
return L * r + (1 - L) * cash - max(L - 1, 0.0) * SPREAD / 252
def describe(r: pd.Series, rf: pd.Series) -> dict:
s = stats(r, rf)
yr = (1 + r).groupby(r.index.year).prod() - 1
return {"cagr": float(s["cagr"]), "sharpe": float(s["sharpe"]), "vol": float(s["vol"]), "mdd": float(s["mdd"]),
"worst_yr": float(yr.min()), "years": {int(y): float(v) for y, v in yr.items()}}
def main() -> None:
os.makedirs(OUT, exist_ok=True)
res_path = os.path.join(OUT, "results.json")
if os.path.exists(res_path):
raise SystemExit(f"{res_path} exists; this study does not overwrite results")
with open(os.path.join(OUT, "prereg.json"), "w") as fh:
json.dump(PREREG, fh, indent=2)
with open(SRC, "rb") as fh:
src = pickle.load(fh)
syn, _ = build(src)
fred = src["fred"].ffill()
rf = (fred["DTB3"].reindex(syn.index.union(fred.index)).ffill().reindex(syn.index) / 100).fillna(0.0)
qqq = syn["QQQ"].pct_change().fillna(0.0)
t1, e1, o1 = trend(syn, rf, 1.0)
t15, e15, o15 = trend(syn, rf, 1.5)
R = pd.DataFrame({"qqq": qqq, "qqq_trend_1x": with_commissions(t1, o1),
"qqq_trend_m15": with_commissions(t15, o15)})
streams = pd.read_pickle(STREAMS)
mom = streams["momentum"].reindex(R.index)
mom_orders = pd.Series(MOMENTUM_ORDERS_A_YEAR / 252, index=R.index)
R["momentum"] = with_commissions(mom, mom_orders)
span = R.loc[MOM_SPAN[0]:MOM_SPAN[1]]
for name, leg, orders in (("mom_qqq_1x", "qqq_trend_1x", o1), ("mom_qqq_m15", "qqq_trend_m15", o15)):
# Each sleeve already pays its own orders at full-account scale; on half the
# capital the same order count costs twice as much per dollar of the sleeve.
X = pd.DataFrame({"momentum": with_commissions(mom, mom_orders, 0.5),
leg: with_commissions(t1 if leg == "qqq_trend_1x" else t15, orders, 0.5)}).loc[span.index]
R.loc[span.index, name], _ = band_blend(X, rf.reindex(span.index))
max_exposure = {"qqq": 1.0, "qqq_trend_1x": 1.0, "qqq_trend_m15": 1.5, "momentum": 1.0,
"mom_qqq_1x": 1.0, "mom_qqq_m15": 1.25}
res = {"constructions": {}, "verdict": {}}
rows = []
for name in PREREG["constructions"]:
start = QQQ_SPAN[0] if name.startswith("qqq") else MOM_SPAN[0]
r = R[name].loc[start:END].dropna()
q = R["qqq"].loc[r.index]
des, hol = r.loc[:"2016-12-31"], r.loc[HOLDOUT:]
qd, qh = q.loc[:"2016-12-31"], q.loc[HOLDOUT:]
L = min(qd.std() / des.std(), REG_T / max_exposure[name])
rl = scaled(r, rf, L)
d = {"span": [str(r.index[0].date()), END], "full": describe(r, rf), "design": describe(des, rf),
"holdout": describe(hol, rf), "qqq_full": describe(q, rf), "qqq_design": describe(qd, rf),
"qqq_holdout": describe(qh, rf), "L_star": float(L),
"matched_full": describe(rl, rf), "matched_holdout": describe(rl.loc[HOLDOUT:], rf)}
if name.startswith("qqq_trend"):
e = e1 if name == "qqq_trend_1x" else e15
o = o1 if name == "qqq_trend_1x" else o15
d["avg_exposure"] = float(e.loc[r.index].mean())
d["orders_a_year"] = float(o.loc[r.index].sum() / (len(r) / 252))
d["crashes"] = {k: float((1 + r.loc[a:b]).prod() - 1) for k, (a, b) in
{"2000-02": ("2000-01-01", "2002-12-31"), "2008": ("2008-01-01", "2008-12-31"),
"2022": ("2022-01-01", "2022-12-31")}.items()}
g = {"G1": d["design"]["sharpe"] >= d["qqq_design"]["sharpe"] + 0.10
and d["holdout"]["sharpe"] >= d["qqq_holdout"]["sharpe"] + 0.10,
"G2": d["full"]["mdd"] > d["qqq_full"]["mdd"],
"G3": d["matched_holdout"]["cagr"] > d["qqq_holdout"]["cagr"]}
d["gates"] = g
res["constructions"][name] = d
rows.append({"construction": name, "span": d["span"][0][:4], "cagr": d["full"]["cagr"],
"sharpe": d["full"]["sharpe"], "mdd": d["full"]["mdd"],
"sh_des": d["design"]["sharpe"], "sh_hold": d["holdout"]["sharpe"],
"cagr_hold": d["holdout"]["cagr"], "L*": L, "matched_cagr_hold": d["matched_holdout"]["cagr"],
"matched_mdd": d["matched_full"]["mdd"], "qqq_sh_des": d["qqq_design"]["sharpe"],
"qqq_sh_hold": d["qqq_holdout"]["sharpe"], "qqq_cagr_hold": d["qqq_holdout"]["cagr"],
"qqq_mdd": d["qqq_full"]["mdd"], **g})
T = pd.DataFrame(rows)
pd.set_option("display.width", 250)
print(T.to_string(index=False, float_format=lambda x: f"{x:.3f}"))
for name in ("qqq_trend_1x", "qqq_trend_m15"):
d = res["constructions"][name]
print(f"{name}: avg exposure {d['avg_exposure']:.2f}, {d['orders_a_year']:.1f} orders a year, "
f"crashes {({k: round(v, 3) for k, v in d['crashes'].items()})}")
passing = [n for n, d in res["constructions"].items() if n != "qqq" and all(d["gates"].values())]
res["verdict"] = {"passing": passing,
"decision": "BEATS QQQ: " + ", ".join(passing) if passing else "NONE BEATS QQQ"}
print(f"\n-> {res['verdict']['decision']}")
with open(res_path, "w") as fh:
json.dump(res, fh, indent=2)
R.to_pickle(os.path.join(OUT, "daily.pkl"))
from research.lab.trials import append_trials
append_trials(os.path.join(RESULTS, "trials_us.jsonl"),
[{"name": f"no_letf_book:{n}", **{k: v for k, v in d["full"].items() if k != "years"}}
for n, d in res["constructions"].items()],
{"study": "no_letf_book", "role": "preregistered"})
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,563 @@
"""
Can a $10k-$100k account lever the overnight book toward 50% a year (A), and
do S&P 500 puts make that leverage safer (B)?
PYTHONPATH=. uv run python -m research.lab.option_overlay fetch # once: Cboe indices + S&P 500 TR
PYTHONPATH=. uv run python -m research.lab.option_overlay # v2: puts held to expiry
PYTHONPATH=. uv run python -m research.lab.option_overlay --hedge diff # v1 (kept for the record)
Book: three_sleeve_overnight (research/lab/overnight_book.py books.pkl, the
paper-tested book: overnight momentum + lowrisk + trend_v7, equal shares),
daily 2006-10-02..2026-09-24, 1x, costs inside. Reported:
three_sleeve_overnight_12_1, two_sleeve_overnight_trend.
Account rules (US retail Reg T margin account; portfolio margin needs $110k+
and is out of scope):
stocks initial margin 50% (Reg T), so 2x
3x ETFs 75% (FINRA Notice 09-53: 3x the 25% maintenance). The trend
sleeve holds them, so levering the whole book at L needs
0.5 x stock share x L + 0.75 x 3x-ETF share x L <= 1;
equal thirds give L <= 1.71
margin loan T-bill + 1.5% (IBKR Pro, first $100k); reported + 2.5% (IBKR Lite)
MES futures $5 x S&P 500 = $38.5k a contract (S&P 7,701, 2026-09-28);
margin 8% of notional, taken from the same equity
SPY/XSP options 100 x $778 = $77.8k a contract; MES options $38.5k
whole contracts today's notional / account size is the exposure step:
MES 3.85 / 1.54 / 0.77 / 0.385 of equity at $10k / $25k / $50k / $100k,
SPY options 7.78 / 3.11 / 1.56 / 0.78
Routes (A), each on a grid of book leverage L (step 0.05) and index exposure
k (whole contracts, k <= 3):
A0 reference L x book at T-bill + 1%, no cap (overnight_book's line; what
portfolio margin would allow)
A1 Reg T L x book up to the margin cap, T-bill + 1.5%
A2 Reg T + box A1 financed with SPX/XSP box spreads at T-bill + 0.5%.
Reported only: sources say a short box belongs in a portfolio
margin account; its Reg T treatment is unverified
A3 Reg T + MES A1 plus k S&P 500 futures: S&P 500 TR minus T-bill minus
0.1%/yr; margin 0.08 k joins the cap
A4 calls A3 with the futures replaced by 90%-strike SPY calls rolled
quarterly (call = forward + put): futures + the put_10q
hedge on k, financing T-bill + 0.5%, 11% of notional paid in
cash, steps of 0.9 x the SPY-option notional
Hedges (B), added to A1 and A3: puts on c = (L - 1) + k of S&P 500 notional
(the borrowed exposure), rounded to whole contracts (c can round to 0):
put_10q (gated) 10% OTM quarterly SPX puts: Cboe PPUT3M minus the S&P 500 TR
(real SPX option prices: VWAP purchase, mid marks)
put_5m 5% OTM monthly: Cboe PPUT minus the S&P 500 TR (neighbour)
vix_tail Cboe VXTH minus the S&P 500 TR (reported; VIX calls are small
contracts, so c is not rounded)
retail cost buying at the ask instead of the VWAP: 3% of the premium on
SPY/XSP, 10% on MES options, on an assumed premium of 1.0%
(put_10q) / 0.8% (put_5m) of notional per roll
Account sizes: $10k, $25k, $50k, $100k.
Metrics per route x account size x hedge:
M35 highest CAGR among configs with MaxDD >= -35%
D50 among configs with CAGR >= 50%, the shallowest MaxDD (or none)
halves (2006-10..2016-12, 2017-01..2026-09) CAGR, MaxDD and Calmar at the M35 config
Gates (fixed before the Cboe hedge series were computed on):
A a route is viable at an account size if D50 exists with MaxDD >= -40%
(A0 is the reference and A2 is flagged, so neither counts)
B PASS if at $100k on A3 with SPY/XSP contracts: put_10q M35 >= unhedged
M35 + 2pp, put_5m M35 >= unhedged M35, and put_10q's M35 config has
Calmar >= the unhedged M35 config's in both halves
Decision:
A viable at a size -> write the live spec for the best viable route there
(paper first); none -> 50%/yr needs portfolio margin or a larger account;
report M35 per size
B PASS -> hedge spec (contract, strike, roll); FAIL -> no option hedge, cap
leverage by drawdown
Data status: the book returns and their uniform-leverage table (overnight_book,
T-bill + 1%) were seen before this rule. The Cboe PPUT, PPUT3M and VXTH files
were downloaded, and only their first and last rows were printed. The Cboe PUT
(put-write), QYLD/XYLD/TAIL/SWAN/JEPI numbers were looked at the same day; put
selling is not a hedge and is not tested here.
Result (2026-10-12, v2): A: no route viable at any account size. B: FAIL
(B1 and B3 fail; B2 passes).
three_sleeve_overnight, 2006-10..2026-09. M35 is the best CAGR with
MaxDD >= -35%; "-" means no contract fits.
route $10k $25k $50k $100k config at $100k
A0 reference (no cap, portfolio margin) 47.0% 47.0% 47.0% 47.0% L 2.20, MaxDD -34.6%; 50% needs L 2.35 (-36.8%)
A1 Reg T margin (T-bill + 1.5%) 36.2% 36.2% 36.2% 36.2% L 1.70 (cap), MaxDD -27.3%
A2 + box financing (Reg T unverified) 37.1% 37.1% 37.1% 37.1% L 1.70, MaxDD -27.0%
A3 + MES futures 36.2% 36.2% 36.2% 36.2% best k = 0
A3 + MES + 10% OTM quarterly puts (SPY) - 35.6% 37.2% 37.2% L 1.60, k 0.77, puts 1.56, MaxDD -31.3%
A3 + MES + 5% OTM monthly puts (SPY) - - 36.3% 36.3% L 1.65, k 0.39, puts 0.78, MaxDD -26.2%
A1 + 10% OTM quarterly puts (SPY) - - - 33.6% L 1.70, puts 0.78, MaxDD -23.5%
A4 90%-strike calls - - - 31.8% L 1.25, k 0.70, MaxDD -35.0%
reported, $100k: three_sleeve_overnight_12_1 A1 40.9% (-25.8%), A3 + 5% puts 43.1%;
two_sleeve_overnight_trend A1 45.8% (-32.7%), A3 reaches 50% only at MaxDD -52.5%
- The ceiling is Reg T, not the drawdown budget. 3x ETFs need 75% and
stocks 50%, so equal thirds cap L at 1.71. At the cap the book makes
36%/yr with MaxDD -27%, and options and futures do not lift that cap.
- Futures add S&P 500 beta, not the book's alpha. Inside -35% they add
nothing to the primary book (best k = 0) and +0.5pp to the 12_1 book.
- The puts cost 2.0%/yr (10% OTM quarterly) and 2.5%/yr (5% OTM monthly)
per unit of notional, against the S&P 500 TR. On A1, puts on the borrowed
0.78 cut MaxDD from -27.3% to -23.5% for -2.6pp of CAGR. Paired with
futures, they buy +1.0pp (37.2%) at MaxDD -31.3%; the gate asked +2pp.
The second half of that config is better and the first half worse (Calmar
fails in 2006-16).
- Small accounts:
- At $10k nothing fits: one MES is 3.85x the account and one SPY option
7.8x.
- At $25k one MES is 1.54x, and the nearest SPY put lot over-hedges by 2x.
- v1 (option_overlay/results.json, B FAIL) built the hedge as the daily
difference PPUT3M - S&P 500 TR. That holds the puts plus a short S&P
position as large as the put's value, so it understated the 2008 payoff
(+4.2% vs the index spread's +11.4pp). v2 holds the puts to expiry,
reconstructed from the Cboe index (yearly rebuild error: median 0.30pp,
2008 +2.9pp in the hedge's favour). The A verdict does not depend on the
hedge model.
- Decision:
- No options or futures in the live books.
- Under Reg T, a $10k-$100k account gets about 36%/yr from
three_sleeve_overnight and 41-46% from the 12_1 and two-sleeve variants.
These are backtest numbers, before any live haircut.
- 50% needs portfolio margin ($110k+ at IBKR, NEEDS EVIDENCE on current
terms) or a different book mix: levering only the stock sleeves is not
tested here.
Boundary: k and c are held as constant fractions of equity (daily
rebalanced); real contracts drift between rolls. The margin cap uses the
book's base shares; the band blend drifts them by up to 25%. The overnight
sleeve's 1 bp a side is not yet measured (overnight_paper.py). FINRA counts
selling an overnight position and rebuying it the same day as no day trade,
but a broker's matching can differ (PDT under $25k).
"""
from __future__ import annotations
import argparse
import json
import os
import numpy as np
import pandas as pd
from research.lab.factor_scan import RESULTS
from research.lab.levered_book import stats, tbill
OUT = os.path.join(RESULTS, "option_overlay")
RAW = os.path.join("data", "research", "option_overlay")
BOOKS = os.path.join(RESULTS, "overnight_book", "books.pkl")
SPAN = ("2006-10-02", "2026-09-24")
HALVES = [("2006-10-02", "2016-12-30"), ("2017-01-03", "2026-09-24")]
PRIMARY = "three_sleeve_overnight"
REPORTED_BOOKS = ["three_sleeve_overnight_12_1", "two_sleeve_overnight_trend"]
LETF_SHARE = {"three_sleeve_overnight": 1 / 3, "three_sleeve_overnight_12_1": 1 / 3,
"two_sleeve_overnight_trend": 1 / 2}
ACCOUNTS = [10_000, 25_000, 50_000, 100_000]
SPX_LEVEL, SPY_PRICE = 7701.08, 778.10
MES_NOTIONAL = 5 * SPX_LEVEL
SPY_OPT_NOTIONAL = 100 * SPY_PRICE
M_STOCK, M_LETF, M_FUT = 0.50, 0.75, 0.08
REF_SPREAD, MARGIN_SPREAD, MARGIN_SPREAD_HI, BOX_SPREAD = 0.01, 0.015, 0.025, 0.005
FUT_COST = 0.001
CALL_DELTA, CALL_CASH = 0.9, 0.11
HEDGES = {"put_10q": ("PPUT3M", 4, 0.010), "put_5m": ("PPUT", 12, 0.008), "vix_tail": ("VXTH", 0, 0.0)}
ASK_COST = {"spy": 0.03, "mes": 0.10}
L_GRID = np.round(np.arange(1.0, 3.0001, 0.05), 2)
K_MAX = 3.0
DD_BUDGET, TARGET, DD_VIABLE = -0.35, 0.50, -0.40
CBOE = "https://cdn-api.cboe.com/api/global/us_indices/daily_prices/{}_History.csv"
LONG = os.path.join("data", "research", "long_history.pkl")
# v2: puts bought at each Cboe roll and held to expiry (source index, strike / S, roll months)
STATIC = {"put_10q": ("PPUT3M", 0.90, (3, 6, 9, 12)), "put_5m": ("PPUT", 0.95, None)}
PREREG = {
"question": "Which leverage route reaches 50%/yr on three_sleeve_overnight in a $10k-$100k Reg T account, "
"and do S&P 500 puts on the borrowed exposure raise the CAGR available at MaxDD >= -35%?",
"span": SPAN, "halves": HALVES, "book": PRIMARY, "reported_books": REPORTED_BOOKS,
"account_sizes": ACCOUNTS,
"rules": {"stock_margin": M_STOCK, "letf_3x_margin": M_LETF, "futures_margin": M_FUT,
"margin_loan": "T-bill + 1.5% (reported + 2.5%)", "box": "T-bill + 0.5% (reported, Reg T unverified)",
"contract_notional": {"MES": MES_NOTIONAL, "SPY_XSP_option": SPY_OPT_NOTIONAL,
"MES_option": MES_NOTIONAL},
"futures_cost": "0.1%/yr", "calls": "90% strike quarterly = futures + put_10q, T-bill + 0.5%, "
"11% cash, steps 0.9 x SPY-option notional"},
"hedges": {"put_10q (gated)": "Cboe PPUT3M - S&P 500 TR", "put_5m": "Cboe PPUT - S&P 500 TR",
"vix_tail (reported)": "Cboe VXTH - S&P 500 TR",
"size": "c = (L - 1) + k, whole contracts", "ask_cost": "3% (SPY/XSP) / 10% (MES opt) of a "
"1.0% / 0.8% premium per roll"},
"metrics": {"M35": "max CAGR with MaxDD >= -35%", "D50": "shallowest MaxDD with CAGR >= 50%"},
"gates": {"A": "route viable at a size if D50 exists with MaxDD >= -40% (A0 reference, A2 flagged: excluded)",
"B": "at $100k on A3 (SPY/XSP): put_10q M35 >= unhedged + 2pp; put_5m M35 >= unhedged; "
"put_10q M35 config Calmar >= unhedged M35 config Calmar in both halves"},
"decision": {"A viable": "live spec for the best viable route at that size, paper first",
"A none": "50%/yr needs portfolio margin or a larger account; report M35 per size",
"B PASS": "hedge spec (contract, strike, roll)",
"B FAIL": "no option hedge; cap leverage by drawdown"},
"data_status": "book returns and their T-bill+1% leverage table seen before; Cboe PPUT/PPUT3M/VXTH downloaded, "
"first/last rows only; PUT and covered-call/tail ETF summaries seen (not tested here)",
}
# ---------------------------------------------------------------------------
# data
# ---------------------------------------------------------------------------
def fetch() -> None:
import requests
import yfinance as yf
from research.lab.fetch_us import _proxy_env
_proxy_env()
os.makedirs(RAW, exist_ok=True)
for name in ("PPUT", "PPUT3M", "VXTH"):
r = requests.get(CBOE.format(name), headers={"User-Agent": "Mozilla/5.0"}, timeout=60)
r.raise_for_status()
with open(os.path.join(RAW, f"{name}.csv"), "wb") as fh:
fh.write(r.content)
print(f"{name}: {len(r.content)} bytes")
h = yf.Ticker("^SP500TR").history(period="max", auto_adjust=True)["Close"]
h.index = h.index.tz_localize(None).normalize()
h.rename("SP500TR").to_csv(os.path.join(RAW, "SP500TR.csv"))
print(f"SP500TR: {len(h)} rows")
def load_levels(index: pd.DatetimeIndex) -> pd.DataFrame:
cols = {}
for name in ("PPUT", "PPUT3M", "VXTH"):
d = pd.read_csv(os.path.join(RAW, f"{name}.csv"))
d.index = pd.to_datetime(d["DATE"], format="%m/%d/%Y")
cols[name] = pd.to_numeric(d[name], errors="coerce")
s = pd.read_csv(os.path.join(RAW, "SP500TR.csv"), index_col=0, parse_dates=True).iloc[:, 0]
cols["SP500TR"] = s
lv = pd.DataFrame(cols).sort_index()
return lv.reindex(lv.index.union(index)).ffill().reindex(index)
def roll_days(index: pd.DatetimeIndex, months) -> pd.DatetimeIndex:
"""Third Friday of each roll month; on a holiday, the preceding session."""
out = []
for p in pd.period_range(index[0], index[-1], freq="M"):
if months and p.month not in months:
continue
third = pd.date_range(p.start_time, p.end_time, freq="W-FRI")[2]
prior = index[(index <= third) & (index >= p.start_time)]
if len(prior):
out.append(prior[-1])
return pd.DatetimeIndex(out)
def static_put(lv: pd.DataFrame, spx_open: pd.Series, spx_close: pd.Series, src: str, mny: float,
months) -> tuple[pd.Series, pd.Series, pd.Series]:
"""The Cboe index's put, held from one roll to the next, per unit of S&P 500 notional.
Between rolls a and b the index holds S + P (Cboe methodology 3.1), so with
y = I/I_a - T/T_a and s = T/T_a (I the Cboe index, T the S&P 500 TR) the
put's value per unit of S_a is x = y (1 + p) + p s, where p = P_a / S_a. p is
solved on the session before b from x = max(0, K - S) / S_a, with K = mny x
S_a rounded up to 5 points; on b the put settles at max(0, K - open) / S_a.
Returns dx (daily change of x inside a period), pid (period number, 0
before the first roll) and the premium p paid at each roll (on the roll day).
"""
idx = lv.index
rolls = roll_days(idx, months)
I, T = lv[src], lv["SP500TR"]
S, O = spx_close.reindex(idx).ffill(), spx_open.reindex(idx).ffill()
dx = pd.Series(0.0, index=idx)
pid = pd.Series(0, index=idx)
prem = pd.Series(0.0, index=idx)
ps = []
bounds = list(zip(rolls[:-1], rolls[1:])) + [(rolls[-1], idx[-1])]
for n, (a, b) in enumerate(bounds, start=1):
seg = idx[(idx >= a) & (idx <= b)]
if len(seg) < 2:
continue
y = I.loc[seg] / I.loc[a] - T.loc[seg] / T.loc[a]
s = T.loc[seg] / T.loc[a]
K = np.ceil(mny * S.loc[a] / 5) * 5
complete = b in rolls and len(seg) >= 3
if complete:
pre = seg[-2]
x_pre = max(0.0, K - S.loc[pre]) / S.loc[a]
p = float(np.clip((x_pre - y.loc[pre]) / (y.loc[pre] + s.loc[pre]), 0.0, 0.5))
else: # the last, unfinished period
p = float(np.median(ps[-4:])) if ps else 0.01
ps.append(p)
x = y * (1 + p) + p * s
x.loc[a] = p
if complete:
x.loc[b] = max(0.0, K - O.loc[b]) / S.loc[a]
dx.loc[seg[1:]] = x.diff().loc[seg[1:]].values
pid.loc[seg[1:]] = n
prem.loc[a] = p
return dx, pid, prem
def with_static(r: pd.Series, dx: pd.Series, pid: pd.Series, prem: pd.Series, c: float, ask: float) -> pd.Series:
"""r plus puts on c x (equity at the last roll) of S&P 500 notional, held to
expiry: E_t = E_t-1 (1 + r_t) + c E_roll dx_t, so E_t / E_roll =
g_t (1 + c sum dx_u / g_u) with g the period's growth of r. The ask cost
(ask x premium x c) is charged on each roll day."""
on = pid > 0
g = (1 + r.where(on, 0.0)).groupby(pid).cumprod()
e = g * (1 + c * (dx / g).groupby(pid).cumsum())
prev = e.groupby(pid).shift(1).fillna(1.0)
return (e / prev - 1).where(on, r) - c * ask * prem
# ---------------------------------------------------------------------------
# routes
# ---------------------------------------------------------------------------
def margin_used(L: float, letf: float, k: float = 0.0, k_margin: float = M_FUT) -> float:
return M_STOCK * (1 - letf) * L + M_LETF * letf * L + k_margin * k
def steps(step: float, cap: float = K_MAX) -> list[float]:
return [round(n * step, 4) for n in range(int(cap / step) + 1)] if step <= cap else [0.0]
def rounded(x: float, step: float) -> float:
return float(np.floor(x / step + 0.5) * step)
def summarize(r: pd.Series, rf: pd.Series) -> dict:
s = stats(r, rf)
d = {"cagr": float(s["cagr"]), "mdd": float(s["mdd"]), "sharpe": float(s["sharpe"]),
"worst_yr": float(s["worst_yr"])}
for i, (a, b) in enumerate(HALVES):
h = stats(r.loc[a:b], rf)
d[f"h{i + 1}_cagr"], d[f"h{i + 1}_mdd"] = float(h["cagr"]), float(h["mdd"])
return d
def configs(book: str, r: pd.Series, rf_d: pd.Series, fut_x: pd.Series, hedge, hedges: dict,
size: int) -> list[dict]:
"""Every feasible config of every route at one account size. ``hedge(r, name,
c, contract)`` returns r with the named hedge on c of S&P 500 notional."""
letf = LETF_SHARE[book]
fut_step = MES_NOTIONAL / size
spy_step, mes_opt_step = SPY_OPT_NOTIONAL / size, MES_NOTIONAL / size
call_step = CALL_DELTA * SPY_OPT_NOTIONAL / size
out = []
def base(L, spread):
return L * r - (L - 1) * (rf_d + spread / 252)
def add(route, L, k, r_cfg, hedge="none", contract="-", c=0.0):
out.append({"route": route, "L": float(L), "k": float(k), "hedge": hedge, "contract": contract,
"c": float(c), "_r": r_cfg})
for L in L_GRID:
add("A0_reference", L, 0.0, base(L, REF_SPREAD))
if margin_used(L, letf) <= 1 + 1e-9:
add("A1_regt", L, 0.0, base(L, MARGIN_SPREAD))
add("A1_regt_lite_rate", L, 0.0, base(L, MARGIN_SPREAD_HI))
add("A2_regt_box", L, 0.0, base(L, BOX_SPREAD))
for k in steps(fut_step):
if margin_used(L, letf, k) > 1 + 1e-9:
continue
r3 = base(L, MARGIN_SPREAD) + k * (fut_x - FUT_COST / 252)
add("A3_regt_mes", L, k, r3)
for hname in hedges:
want = (L - 1) + k
options = (("vix", None),) if hname == "vix_tail" else (("spy", spy_step), ("mes", mes_opt_step))
for contract, step in options:
c = want if step is None else rounded(want, step)
if c <= 0:
continue
for route, rr in (("A3_regt_mes", r3),) + ((("A1_regt", base(L, MARGIN_SPREAD)),) if k == 0 else ()):
add(route, L, k, hedge(rr, hname, c, contract), hname, contract, c)
for k in steps(call_step):
if k == 0 or margin_used(L, letf, k, CALL_CASH) > 1 + 1e-9:
continue
r4 = base(L, MARGIN_SPREAD) + k * (fut_x - (FUT_COST + BOX_SPREAD) / 252)
add("A4_calls", L, k, hedge(r4, "put_10q", k, "spy"), "embedded put_10q", "spy", k)
return out
def best(rows: list[dict]) -> tuple[dict | None, dict | None]:
ok = [x for x in rows if x["mdd"] >= DD_BUDGET]
m35 = max(ok, key=lambda x: x["cagr"]) if ok else None
hit = [x for x in rows if x["cagr"] >= TARGET]
d50 = max(hit, key=lambda x: x["mdd"]) if hit else None
return m35, d50
def calmar(x: dict, h: int) -> float:
return x[f"h{h}_cagr"] / abs(x[f"h{h}_mdd"]) if x[f"h{h}_mdd"] < 0 else float("inf")
PREREG_V2 = {
**PREREG,
"version": 2,
"hedge_model": "puts bought at each Cboe roll and held to expiry, sized on equity at the roll; the put's "
"value per unit of S&P 500 notional is reconstructed from the Cboe index (static_put); the ask "
"cost is ask x the reconstructed premium per roll; vix_tail dropped (no VIX roll model)",
"why_v2": "v1 used the daily difference PPUT3M - S&P 500 TR, which holds puts plus a short S&P position equal "
"to the put's value; after a crash that short is large and loses in the rebound (2008: v1 hedge "
"+4.2% vs index spread +11.4pp). Found after reading v1's results; gates and every other rule unchanged.",
"data_status": PREREG["data_status"] + "; v1 results (option_overlay/results.json) seen before v2",
}
def main(model: str = "static") -> None:
out_dir = OUT if model == "diff" else OUT + "_v2"
os.makedirs(out_dir, exist_ok=True)
res_path = os.path.join(out_dir, "results.json")
if os.path.exists(res_path):
raise SystemExit(f"{res_path} exists; this study does not overwrite results")
with open(os.path.join(out_dir, "prereg.json"), "w") as fh:
json.dump(PREREG if model == "diff" else PREREG_V2, fh, indent=2)
B = pd.read_pickle(BOOKS).loc[SPAN[0]:SPAN[1]]
idx = B.index
rf = tbill(idx).fillna(0.0)
rf_d = rf / 252
lv = load_levels(idx)
ret = lv.pct_change(fill_method=None).fillna(0.0)
fut_x = ret["SP500TR"] - rf_d
res = {"model": model, "hedge_series": {}, "books": {}}
if model == "diff":
hedges = list(HEDGES)
hedge_x = {h: ret[src] - ret["SP500TR"] for h, (src, _, _) in HEDGES.items()}
def hedge(r_, name, c, contract):
_, rolls, prem = HEDGES[name]
return r_ + c * (hedge_x[name] - rolls * ASK_COST[contract] * prem / 252)
else:
hedges = list(STATIC)
L_ = pd.read_pickle(LONG)
spx_o, spx_c = L_["open"]["^GSPC"].dropna(), L_["close"]["^GSPC"].dropna()
paths = {h: static_put(lv, spx_o, spx_c, src, mny, months) for h, (src, mny, months) in STATIC.items()}
def hedge(r_, name, c, contract):
dx, pid, prem = paths[name]
return with_static(r_, dx, pid, prem, c, ASK_COST[contract])
print("=== static puts: reconstruction check (S&P 500 TR + 1 put per unit vs the Cboe index)")
for h, (src, mny, months) in STATIC.items():
dx, pid, prem = paths[h]
p = prem[prem > 0]
rebuilt = with_static(ret["SP500TR"], dx, pid, prem, 1.0, 0.0)
yr = pd.DataFrame({"rebuilt": rebuilt, "cboe": ret[src], "sp500tr": ret["SP500TR"]})
yr = (1 + yr).groupby(yr.index.year).prod() - 1
err = (yr["rebuilt"] - yr["cboe"]).abs()
d = {"rolls": int(len(p)), "premium_median": float(p.median()), "premium_max": float(p.max()),
"premium_max_date": str(p.idxmax().date()), "yearly_abs_err_median": float(err.median()),
"yearly_abs_err_max": float(err.max()), "worst_err_year": int(err.idxmax()),
"cost_pa_vs_sp500tr": float(((1 + rebuilt).prod() ** (252 / len(rebuilt))
- (1 + ret["SP500TR"]).prod() ** (252 / len(rebuilt)))),
"years": {int(y): {k: float(v) for k, v in row.items()} for y, row in yr.loc[[2008, 2009, 2020, 2022]].iterrows()}}
res["hedge_series"][h] = d
print(f" {h}: {d['rolls']} rolls, premium median {d['premium_median']*100:.2f}% max "
f"{d['premium_max']*100:.1f}% ({d['premium_max_date']}); rebuilt vs Cboe yearly |err| median "
f"{d['yearly_abs_err_median']*100:.2f}pp, max {d['yearly_abs_err_max']*100:.2f}pp ({d['worst_err_year']}); "
f"hedged S&P vs S&P {d['cost_pa_vs_sp500tr']*100:+.2f}%/yr")
print(" " + " ".join(f"{y}: rebuilt {v['rebuilt']*100:+.1f}% cboe {v['cboe']*100:+.1f}% "
f"S&P {v['sp500tr']*100:+.1f}%" for y, v in d["years"].items()))
hedge_x = {}
if hedge_x:
print("=== hedge series (per unit of S&P 500 notional, before retail costs)")
for h, x in hedge_x.items():
yr = (1 + x).groupby(x.index.year).prod() - 1
d = {"mean_pa": float(x.mean() * 252), "y2008": float(yr.get(2008, np.nan)),
"y2020": float(yr.get(2020, np.nan)), "y2022": float(yr.get(2022, np.nan)),
"corr_book_monthly": float(((1 + x).resample("ME").prod() - 1).corr(
(1 + B[PRIMARY]).resample("ME").prod() - 1))}
res["hedge_series"][h] = d
print(f" {h:9s} mean {d['mean_pa']*100:+5.2f}%/yr 2008 {d['y2008']*100:+5.1f}% 2020 {d['y2020']*100:+5.1f}% "
f"2022 {d['y2022']*100:+5.1f}% monthly corr with book {d['corr_book_monthly']:+.2f}")
all_rows = []
for book in [PRIMARY] + REPORTED_BOOKS:
r = B[book]
cap = 1 / (M_STOCK * (1 - LETF_SHARE[book]) + M_LETF * LETF_SHARE[book])
res["books"][book] = {"regt_cap_L": cap, "sizes": {}}
print(f"\n=== {book}: Reg T cap L <= {cap:.2f}")
for size in ACCOUNTS:
rows = configs(book, r, rf_d, fut_x, hedge, hedges, size)
for x in rows:
x.update(summarize(x.pop("_r"), rf))
x.update({"book": book, "size": size})
all_rows += rows
groups = {}
for x in rows:
groups.setdefault((x["route"], x["hedge"], x["contract"]), []).append(x)
table = {}
for key, g in sorted(groups.items()):
m35, d50 = best(g)
table["|".join(key)] = {"M35": m35, "D50": d50}
res["books"][book]["sizes"][size] = table
if book != PRIMARY and size != 100_000:
continue
print(f" -- ${size // 1000}k (MES step {MES_NOTIONAL / size:.2f}, SPY-option step "
f"{SPY_OPT_NOTIONAL / size:.2f})")
for key, t in table.items():
m, d = t["M35"], t["D50"]
ms = (f"M35 {m['cagr']*100:5.1f}% (L {m['L']:.2f} k {m['k']:.2f} c {m['c']:.2f}, mdd {m['mdd']*100:5.1f}%,"
f" halves {m['h1_cagr']*100:5.1f}/{m['h2_cagr']*100:5.1f}%)") if m else "M35 none"
ds = (f"D50 mdd {d['mdd']*100:5.1f}% (L {d['L']:.2f} k {d['k']:.2f} c {d['c']:.2f})") if d else "D50 none"
print(f" {key:34s} {ms} {ds}")
# gates on the primary book
sizes = res["books"][PRIMARY]["sizes"]
viable = {}
for size in ACCOUNTS:
cands = []
for key, t in sizes[size].items():
if key.startswith(("A0", "A2")) or "lite_rate" in key:
continue
d = t["D50"]
if d and d["mdd"] >= DD_VIABLE:
cands.append((d["mdd"], key, d))
cands.sort(key=lambda z: -z[0])
viable[size] = [{"route": k, "mdd": d["mdd"], "cagr": d["cagr"], "L": d["L"], "k": d["k"], "c": d["c"]}
for _, k, d in cands]
t100 = sizes[100_000]
un, p10, p5 = (t100.get(k, {}).get("M35") for k in
("A3_regt_mes|none|-", "A3_regt_mes|put_10q|spy", "A3_regt_mes|put_5m|spy"))
b_checks = {
"B1_put10q_plus_2pp": bool(un and p10 and p10["cagr"] >= un["cagr"] + 0.02),
"B2_put5m_not_worse": bool(un and p5 and p5["cagr"] >= un["cagr"]),
"B3_calmar_both_halves": bool(un and p10 and all(calmar(p10, h) >= calmar(un, h) for h in (1, 2))),
}
res["verdict"] = {
"A_viable_routes_by_size": viable,
"A_decision": {s: (v[0]["route"] if v else "none") for s, v in viable.items()},
"B_checks": b_checks, "B_decision": "PASS" if all(b_checks.values()) else "FAIL",
}
print("\n=== verdict")
for s, v in viable.items():
top = v[0] if v else None
print(f" A ${s // 1000}k: " + (f"viable, best {top['route']} mdd {top['mdd']*100:.1f}% "
f"(L {top['L']:.2f} k {top['k']:.2f} c {top['c']:.2f})" if top else "no viable route"))
print(f" B {b_checks} -> {res['verdict']['B_decision']}")
with open(res_path, "w") as fh:
json.dump(res, fh, indent=2, default=float)
pd.DataFrame(all_rows).to_csv(os.path.join(out_dir, "configs.csv.gz"), index=False)
from research.lab.trials import append_trials
trial_rows = []
for size, table in sizes.items():
for key, t in table.items():
if t["M35"]:
trial_rows.append({"name": f"option_overlay{'' if model == 'diff' else '_v2'}:{key}:{size}",
**{x: t["M35"][x] for x in ("cagr", "mdd", "sharpe", "L", "k", "c")}})
append_trials(os.path.join(RESULTS, "trials_us.jsonl"), trial_rows,
{"study": "option_overlay" + ("" if model == "diff" else "_v2"),
"role": "preregistered; M35 config per route/size"})
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("cmd", nargs="?", default="run", choices=["run", "fetch"])
ap.add_argument("--hedge", choices=["static", "diff"], default="static",
help="static: puts held to expiry (v2, default); diff: daily index difference (v1)")
a = ap.parse_args()
if a.cmd == "fetch":
fetch()
else:
main(a.hedge)

View File

@@ -0,0 +1,244 @@
"""
Do option-volume signals predict S&P 500 stock returns? This uses OPRA
daily option bars from 2024-02 (research/lab/fetch_options.py backfill,
volume_panel.csv.gz).
uv run python -m research.lab.fetch_options backfill # once: bars + volume_panel.csv.gz
PYTHONPATH=. uv run --with numba python -m research.lab.option_volume
Signals, known at the close of day t (OPRA bars are final after 16:15 New York):
option volume contracts of every listed option expiring within 0-100 days
(all strikes), calls C and puts P
stock volume unadjusted SIP volume, so a split does not break the ratio
low_os_5 (gated) -(sum_5d (C + P) x 100 / sum_5d stock volume)
Johnson & So (2012): high O/S marks negative private
information that short-sale costs push into options,
and predicts low returns.
low_pc_5 (gated) -(sum_5d P / sum_5d (C + P))
Pan & Poteshman (2006) use signed open-buy put/call
volume; signed data is not available, so this is the
unsigned ratio and a weaker test of the same idea.
reported low_os_21 (the monthly version), low_os_abn (-(os_5 / its
126-day mean)), low_os_band_5 (O/S on the 5-100 day, 0.7-1.3
moneyness band only), call_shift (-(pc_5 - pc_63))
Test (PIT S&P 500 research panel, members only, >= 50 names a day):
- Daily rank IC and top-minus-bottom quintile returns, from open(t+1) to
close(t+h), for h = 5 (gated) and h = 21. Newey-West t with h lags,
annualised x 252/h.
- Long-only top 20 rebalanced every h days, buffer 2, next-open fills,
10 bp a side, phase-averaged: alpha vs the EW member basket.
- Correlation with house_overnight_recovery (overnight_mom + rec_126) and
the spanning t on it, as in research/lab/factor_census.py.
Window: 2024-03-01 .. the panel's last date (2026-09-24), about 2.5 years.
Halves are split at 2025-06-01.
Data handling (set after a coverage check of the panel, before any return was
computed): stock volume comes from stock_raw.csv.gz on every day. A day
without bars counts as 0 contracts inside the name's first..last bar date
(thin names such as NWS, REG, UDR: 0.6% of stock-days). It is missing
outside that span, where the options traded under an older ticker or Alpaca's
history for the root starts later (BNY, XYZ, BRK.B, BF.B ...), and on
2024-02-02, which has no bars at all.
Gate per gated signal (fixed before the backfill finished; the factor census
asks |t| >= 3 of a new factor):
V1 h=5 quintile spread with the predicted sign and NW t >= 3.0
V2 the spread has the same sign in both halves
V3 top-20 (every 5 days) alpha vs EW > 0
Decision:
CONTINUE a gated signal passes all three -> preregister a combination with
overnight_recovery_top10 (score blend and veto) and add the daily
signal to the live data path
STOP none passes -> keep collecting (the daily snapshot stores the bars)
and retest when the history is about 5 years long
This is the first look at these data. Nothing in the window was used before.
The rule is written to prereg.json before the run; the script refuses to
overwrite results.json.
Result (2026-10-12): STOP. low_os_5 fails V1, V2 and V3. low_pc_5 fails V1
and V3.
2024-03..2026-09, h=5 IC (t) Q5-Q1/yr (t) halves %/yr top-20 alpha vs EW (t) corr house
low_os_5 (gated) -0.013 (-1.32) -11.4% (-1.63) -1.1 / -21.3 -5.4% (-1.40) -0.65
low_pc_5 (gated) +0.000 (+0.07) +1.2% (+0.42) +1.0 / +1.4 -5.4% (-1.40) -0.08
low_os_21 -0.018 (-1.70) -11.7% (-1.68) -4.0 / -19.1 -2.7% (-0.71) -0.65
low_os_band_5 -0.013 (-1.36) -10.1% (-1.52) -1.2 / -18.6 -5.0% (-1.24) -0.65
low_os_abn -0.003 (-0.60) -3.7% (-1.05) +2.1 / -7.7 -6.0% (-1.34) -0.46
call_shift -0.009 (-1.73) -2.5% (-0.87) -1.0 / -3.7 -7.0% (-1.95) -0.19
house_overnight_recovery +0.014 (+1.07) +8.0% (+0.91) -2.5 / +18.1 +31.0% (+1.90) 1
- O/S has the opposite sign to Johnson-So in this window. Names with high
option volume relative to stock volume did better, mostly after 2025-06
(-21%/yr in the second half). The O/S score correlates -0.65 with the
momentum score: high O/S marks recent winners with heavy call trading.
Its spanning t on the momentum spread is -1.28, so momentum already holds
what O/S carries. It is not a usable signal in either direction (|t| < 2).
- Unsigned put/call volume is flat (t 0.42). Without open-buy signing this
does not test Pan-Poteshman's result.
- One of the 24 reported rows reaches |t| >= 2: call_shift's top-20 alpha at
h=21 (-7.8%, t -2.34). That is within what chance gives.
- Boundary: 2.5 years in one regime (AI-led momentum). SNDK was fetched after
a batch-splitting fix for HTTP 504. CDAY (renamed DAY) and NVR (no listed
options) have no bars.
- Next: keep collecting the daily bars and retest at ~5 years (2029).
"""
from __future__ import annotations
import json
import os
import numpy as np
import pandas as pd
from research.lab.alpha import alpha_stats
from research.lab.factor_census import cross_section, nw_mean_t, pct_rank, span_t
from research.lab.factor_scan import RESULTS, load
from research.lab.factors import FACTORS
from research.lab.levered_book import stats, tbill
from research.lab.us_sweep import run
OUT = os.path.join(RESULTS, "option_volume")
PANEL = os.path.join("data", "options", "us", "volume_panel.csv.gz")
STOCK = os.path.join("data", "options", "us", "stock_raw.csv.gz")
START = "2024-03-01"
SPLIT = "2025-06-01"
GATED = ["low_os_5", "low_pc_5"]
PREREG = {
"question": "Do option-volume signals (Johnson-So O/S, put/call volume) predict S&P 500 returns, 2024-03..2026-09?",
"signals": {"gated": GATED, "reported": ["low_os_21", "low_os_abn", "low_os_band_5", "call_shift"]},
"gates_per_signal": ["V1 h=5 quintile spread predicted sign, NW t >= 3.0",
"V2 same sign in both halves (split 2025-06-01)",
"V3 top-20 every-5-days alpha vs EW > 0"],
"decision": {"CONTINUE": "preregister a combination with overnight_recovery_top10 and add the live signal",
"STOP": "keep collecting; retest at ~5 years of history"},
"data_status": "first look; option bars backfilled 2026-10-12, not inspected before this rule",
"data_handling": "fixed after a coverage check of the panel, before any return was computed: stock volume "
"from stock_raw on every day; a day without bars counts as 0 contracts inside the name's "
"first..last bar date, and is missing outside it (older tickers, late root history) and on "
"2024-02-02 (no bars)",
}
def wide(P: pd.DataFrame, col: str, dates, cols) -> pd.DataFrame:
w = P.pivot_table(index="date", columns="underlying", values=col, aggfunc="sum")
w.index = pd.to_datetime(w.index)
w.columns = [c.replace(".", "-") for c in w.columns]
return w.reindex(index=dates, columns=cols)
def coverage(P: pd.DataFrame, dates, cols) -> pd.DataFrame:
"""True where a missing bar row means no option traded: inside the name's
first..last bar date and on a date with any bars. Outside it the volume is
unknown (options listed under an older ticker, e.g. XYZ before 2025-01 as SQ;
Alpaca's history for the root starts later, e.g. BRK.B 2024-08; 2024-02-02
has no bars at all)."""
span = P.groupby("underlying")["date"].agg(["min", "max"])
span.index = [c.replace(".", "-") for c in span.index]
lo = pd.to_datetime(span["min"]).reindex(cols).values[None, :]
hi = pd.to_datetime(span["max"]).reindex(cols).values[None, :]
d = dates.values[:, None]
has_day = np.asarray(dates.isin(pd.to_datetime(P["date"].unique())))[:, None]
return pd.DataFrame((d >= lo) & (d <= hi) & has_day, index=dates, columns=cols)
def signals(p, P: pd.DataFrame, stock: pd.DataFrame) -> dict[str, pd.DataFrame]:
dates, cols = p.dates, p.close.columns
sv = wide(stock.rename(columns={"symbol": "underlying"}), "volume", dates, cols)
sv = sv.where(sv > 0) # unadjusted SIP volume on every stock day
cover = coverage(P, dates, cols)
C = wide(P, "v_C", dates, cols).fillna(0).where(cover)
Pv = wide(P, "v_P", dates, cols).fillna(0).where(cover)
Cb = wide(P, "v_band_C", dates, cols).fillna(0).where(cover)
Pb = wide(P, "v_band_P", dates, cols).fillna(0).where(cover)
s = lambda x, n: x.rolling(n, min_periods=n).sum() # noqa: E731
os5 = s(C + Pv, 5) * 100 / s(sv, 5)
os21 = s(C + Pv, 21) * 100 / s(sv, 21)
tot5 = s(C + Pv, 5)
pc5 = s(Pv, 5) / tot5.where(tot5 > 0)
tot63 = s(C + Pv, 63)
pc63 = s(Pv, 63) / tot63.where(tot63 > 0)
return {
"low_os_5": -os5,
"low_pc_5": -pc5,
"low_os_21": -os21,
"low_os_abn": -(os5 / os5.rolling(126, min_periods=100).mean()),
"low_os_band_5": -(s(Cb + Pb, 5) * 100 / s(sv, 5)),
"call_shift": -(pc5 - pc63),
}
def main() -> None:
os.makedirs(OUT, exist_ok=True)
res_path = os.path.join(OUT, "results.json")
if os.path.exists(res_path):
raise SystemExit(f"{res_path} exists; this study does not overwrite results")
if not os.path.exists(PANEL):
raise SystemExit(f"{PANEL} missing; run fetch_options backfill first")
with open(os.path.join(OUT, "prereg.json"), "w") as fh:
json.dump(PREREG, fh, indent=2)
p, _ = load("us")
P = pd.read_csv(PANEL)
S = signals(p, P, pd.read_csv(STOCK))
end = str(p.dates[-1].date())
m = p.member
house = (pct_rank(FACTORS["overnight_mom"](p), m) + pct_rank(FACTORS["rec_126"](p), m)) / 2
o = p.open.where(p.open > 0)
later = p.close.ffill()
fwd = {h: later.shift(-h) / o.shift(-1) - 1 for h in (5, 21)}
ret = p.close.pct_change(fill_method=None)
ew = ret.where(m.shift(1, fill_value=False)).mean(axis=1)
rf = tbill(p.dates).fillna(0.0)
sp = p.slice("2024-01-02", end)
res = {"window": [START, end], "signals": {}}
for name, sc in {**S, "house_overnight_recovery": house}.items():
row = {}
for h in (5, 21):
ic, ls, tx, turn, n = cross_section(sc, fwd[h], m)
ic, ls, tx, n = (x.loc[START:end] for x in (ic, ls, tx, n))
r = {"names": float(n.mean())}
r["ic"], r["ic_t"], _ = nw_mean_t(ic, h)
mu, r["ls_t"], _ = nw_mean_t(ls, h)
r["ls"] = mu * 252 / h
mu, r["top_x_t"], _ = nw_mean_t(tx, h)
r["top_x"] = mu * 252 / h
halves = []
for a, b in ((START, SPLIT), (SPLIT, end)):
v = ls.loc[a:b].dropna()
halves.append(float(v.mean() * 252 / h) if len(v) > 60 else float("nan"))
r["ls_halves"] = halves
hr = cross_section(house, fwd[h], m)[1].loc[START:end]
r["corr_house"] = float(ls.corr(hr))
r["span_t_house"] = span_t(ls, hr)
st = run(sp, sc.where(m), [(START, end)], top_n=20, buffer=2.0, every=h)
rr = st["_rets"].loc[START:end]
s_, al = stats(rr, rf), alpha_stats(rr, ew.reindex(rr.index), rf.reindex(rr.index))
r.update({"top20_cagr": float(s_["cagr"]), "top20_sharpe": float(s_["sharpe"]),
"top20_mdd": float(s_["mdd"]), "alpha": float(al["alpha"]), "alpha_t": float(al["t"])})
row[f"h{h}"] = r
print(f"{name:26s} h{h:2d} names {r['names']:5.0f} ic {r['ic']:+.3f} (t {r['ic_t']:+.2f}) "
f"ls {r['ls']*100:+6.1f}%/yr t {r['ls_t']:+5.2f} halves {halves[0]*100:+6.1f} / {halves[1]*100:+6.1f} "
f"| top20 sh {r['top20_sharpe']:.2f} alpha {r['alpha']*100:+5.1f}% (t {r['alpha_t']:+.2f}) "
f"| corr house {r['corr_house']:+.2f} span t {r['span_t_house']:+.2f}", flush=True)
res["signals"][name] = row
checks = {}
for g in GATED:
r = res["signals"][g]["h5"]
checks[g] = {"V1": r["ls_t"] >= 3.0, "V2": all(np.isfinite(x) and x > 0 for x in r["ls_halves"]),
"V3": r["alpha"] > 0}
passing = [g for g, c in checks.items() if all(c.values())]
res["verdict"] = {"checks": checks, "passing": passing, "decision": "CONTINUE" if passing else "STOP"}
print(f"checks {checks} -> {res['verdict']['decision']} {passing}")
with open(res_path, "w") as fh:
json.dump(res, fh, indent=2)
from research.lab.trials import append_trials
append_trials(os.path.join(RESULTS, "trials_us.jsonl"),
[{"name": f"option_volume:{k}:{h}", **{x: v for x, v in d.items() if x != "ls_halves"}}
for k, row in res["signals"].items() for h, d in row.items()],
{"study": "option_volume", "role": "preregistered first look"})
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,496 @@
"""
10-day momentum in a $10k-$20k IBKR account: whole shares, real commissions
and low-frequency risk controls, against holding QQQ (preregistered).
PYTHONPATH=. uv run python -m research.lab.small_account_momentum
Question (2026-10-13). The $10k IBKR account has no leveraged-ETF permission.
Nothing in no_letf_book beat QQQ, and the overnight sleeve needs near-free
orders. The 10-day momentum book (overnight_recovery_top10, what ../ibkr-quant
runs on IBKR paper) sends about 140 orders a year, so commissions are not its
problem; its problem is risk (MaxDD -69% since 2006, Sharpe below QQQ). This
study asks whether three low-frequency changes, already studied one at a time
on fractional books, make it beat QQQ once whole shares, IBKR commissions and
a small account are modelled:
rb20 resize band 20% instead of 10% (fewer resizing orders)
derisk de-risk-only vol target: exposure = min(1, expanding median / EWMA
vol (hl 20) of SPY), read at the signal date, applied at each
rebalance (us_sweep 2026-09-29: MaxDD halved, Sharpe +0.03)
fundmom momentum top 30 re-sorted by fundamental momentum (mean percentile
of SUE, revenue SUE, gross-margin change, ROA change; EDGAR first
filings), top 10, buffer 1 (momentum_fundamentals 2026-09-30:
holdout Sharpe 0.97 -> 1.06, MaxDD -42% -> -35%)
half_qqq the momentum book on half the account, QQQ on the other half
Simulation (fixed before running)
* PIT S&P 500 research panel with the Tiingo overlay. Signal on bars through
day d-1, MOC fill at day d's close (the live convention). Top 10, every 10
sessions, rank buffer 2 (fundmom: 1), equal weight, 2% cash buffer.
* Whole shares at the price of the day: split-adjusted, dividend-unadjusted
Yahoo closes x later split ratios, else Tiingo's raw close, else the
adjusted close. Orders are sized by execution.compute_trades, the live
ledger's rule (round toward zero, leftover cash buys one more share for
the names furthest below target, $50 minimum order, resize band) and
filled like execution.execute_trades (sells first, then each buy shrunk
to the whole shares the remaining cash pays for: no margin).
* Constant account size: at every rebalance the account is valued at the
stated capital ($10k or $20k) for share counts and fees, so the friction
of a small account does not fade as the backtest compounds.
* IBKR Pro Tiered: max(0.35, 0.0035/share) capped at 1% of value, +0.0002/share
clearing, +0.0012/share closing-auction fee (approximate), SEC 0.0000206 x
sales, CAT 0.000003/share. Fixed: max(1, 0.005/share) capped at 1%, + SEC/CAT.
Plus 2 bp a side against the official close (auction_audit measured 0.2).
* Idle cash earns nothing (IBKR pays no interest on the first $10k).
Dividends are reinvested in the name (adjusted prices). A held name with
no print for 5 sessions that has left the index is settled at its last print.
* Phase averaging: 10 rebalance phases; statistics are the mean over phases.
* Data filter: a member-day whose 63-day median dollar volume is under $2M is
neither selectable nor tradable. Found while building this study: EP, CPWR,
SW, GR, SII, MIL, GENZ, KG, LSI carry another security's (or stale) prices
in the research panel; ~13.8k of 2.6M member-days.
Windows: 2011-01-03..2026-09-24 for every variant (fundamentals cover ~90% of
members from 2011), design 2011-2016, holdout 2017-2026. Price-only variants
are also run from 2006-10-02 to include 2008.
Gates (each at $10k and at $20k, Tiered), against QQQ total return:
G1 CAGR >= QQQ in design and holdout
G2 MaxDD 2011-2026 no deeper than QQQ's; price-only variants also 2006-2026
G3 excess-return Sharpe >= QQQ in design and holdout
PASS = G1 and G2 and G3. Among passing variants the highest full-span Sharpe
is preferred. Data status: the momentum signal, rb20, the de-risk overlay and
fund_momentum were each examined on these years before; the holdout is not
independent out-of-sample data. What is new here is the combination, whole
shares, commissions at $10k/$20k and the gates against QQQ.
Results (2026-10-13): NONE PASSES. Every variant fails G3 in 2011-2016, when
QQQ itself ran at Sharpe 0.92 with a -16% drawdown.
mean of 10 phases, $10k Tiered CAGR des/hold Sharpe des/hold MaxDD 11-26 orders/yr
QQQ 15.2% / 21.5% 0.92 / 0.86 -35%
base (ibkr-quant) 17.2% / 29.0% 0.78 / 0.87 -38% 82
rb20 17.3% / 29.5% 0.79 / 0.88 -37% 67
rb20_derisk 16.3% / 21.7% 0.82 / 0.74 -36% 79
rb20_fundmom 16.5% / 32.7% 0.80 / 1.04 -32% G1 G2 92
rb20_fundmom_derisk 15.1% / 25.7% 0.82 / 0.93 -31% G2 103
rb20_half_qqq 16.2% / 24.2% 0.87 / 0.91 -29% 58
($20k: rb20_fundmom 16.7% / 34.9%, 0.80 / 1.05, -35%; 2006-26 price-only:
base 21.5% / -73%, rb20_derisk 18.2% / -38%, half_qqq 18.5% / -62%, QQQ 16.5% / -53%)
* Friction is small: Tiered fees 0.3-0.5%/yr, Fixed 0.6-1.2%. At $10k the
larger cost is whole shares: 5-8% of picks get no shares (cash left after
in-band winners, or one share above the name's budget) vs ~0.5% at $20k;
rb20_fundmom holdout 32.7% at $10k, 34.9% at $20k, 35.3% frictionless.
* Plain momentum's holdout edge is 2026: 2017-2025 rb20 21.7% / 0.75 vs QQQ
20.9% / 0.84; it beat QQQ in 5 of 16 years. rb20_fundmom 2017-2025 28.6% /
1.02 / -31%, beat QQQ in 10 of 16 years ($20k: 11).
* The de-risk overlay pays only in 2008-type crashes (2006-26 MaxDD -73% ->
-38%); in 2011-26 it costs 8 points of holdout CAGR and lowers Sharpe.
* rb20_fundmom meets the user's two stated criteria (return >= QQQ, drawdown
no deeper) in 2011-26 but is untested in 2008 (EDGAR coverage starts 2011)
and was picked in momentum_fundamentals after seeing these years.
* Simulator check (--smoke): with no resize band, no costs and no delisting
settlement it reproduces research/lab/engine.py (21.7% vs 22.0%, daily
corr 0.99). The lab engine never settles delisted names: capital in
acquired names stays frozen, which costs momentum ~2.8 points a year in
2011-26 (24.5% with settlement).
"""
from __future__ import annotations
import dataclasses
import json
import os
import pickle
import sys
import warnings
from concurrent.futures import ProcessPoolExecutor
import numpy as np
import pandas as pd
from execution import compute_trades
from research.lab import fundamentals as fu
from research.lab import overlays as ov
from research.lab.alpha import alpha_stats, bench_returns
from research.lab.engine import Costs, select_top
from research.lab.factor_scan import RESULTS, load
from research.lab.levered_book import stats, tbill
from research.lab.rotation import factor_ranks
from research.lab.us_sweep import _blend
warnings.filterwarnings("ignore")
OUT = os.path.join(RESULTS, "small_account_momentum")
SPAN = ("2011-01-03", "2026-09-24")
LONG = ("2006-10-02", "2026-09-24")
DESIGN_END, HOLDOUT = "2016-12-31", "2017-01-01"
TOP_N, EVERY = 10, 10
CASH_BUFFER, MIN_ORDER, MARKET_BP = 0.02, 50.0, 2.0
SETTLE_AFTER = 5
MIN_DOLLAR_VOLUME = 2e6 # 63-day median; below it the series is not the member's (data filter)
TIINGO = os.path.join("data", "research", "tiingo")
VARIANTS = {
"base": dict(score="mom", band=0.10, derisk=False, qqq=0.0), # what ibkr-quant runs
"rb20": dict(score="mom", band=0.20, derisk=False, qqq=0.0),
"rb20_derisk": dict(score="mom", band=0.20, derisk=True, qqq=0.0),
"rb20_fundmom": dict(score="fundmom", band=0.20, derisk=False, qqq=0.0),
"rb20_fundmom_derisk": dict(score="fundmom", band=0.20, derisk=True, qqq=0.0),
"rb20_half_qqq": dict(score="mom", band=0.20, derisk=False, qqq=0.5),
}
# (label, capital, pricing, market bp a side, cash earns T-bill)
CONFIGS = {
"10k_tiered": (10_000.0, "tiered", MARKET_BP, False),
"20k_tiered": (20_000.0, "tiered", MARKET_BP, False),
"10k_fixed": (10_000.0, "fixed", MARKET_BP, False),
"20k_fixed": (20_000.0, "fixed", MARKET_BP, False),
"10k_tiered_10bp": (10_000.0, "tiered", 10.0, False),
"10k_tiered_tbill_cash": (10_000.0, "tiered", MARKET_BP, True),
"frictionless": (1e9, "none", 0.0, False),
}
GATE_CONFIGS = ("10k_tiered", "20k_tiered")
LONG_CONFIGS = ("10k_tiered", "20k_tiered", "frictionless")
PREREG = {
"question": "does 10-day momentum with low-frequency risk controls beat QQQ in a $10k/$20k IBKR account",
"variants": {k: v for k, v in VARIANTS.items()},
"simulation": "PIT S&P 500 + Tiingo; signal through d-1, MOC at d's close; top 10 every 10 sessions; "
"whole shares via execution.compute_trades at raw prices; constant account size; "
"10 phases averaged; idle cash earns 0; 2% cash buffer",
"costs": "IBKR Pro Tiered max(0.35, 0.0035/sh) cap 1% + 0.0002/sh clearing + 0.0012/sh auction + SEC/CAT; "
"Fixed max(1, 0.005/sh) cap 1% + SEC/CAT; 2 bp a side market cost",
"configs": {k: list(v) for k, v in CONFIGS.items()},
"windows": {"span": SPAN, "design_end": DESIGN_END, "holdout_start": HOLDOUT, "long_price_only": LONG},
"gates": {"G1": "mean-phase CAGR >= QQQ in design and holdout",
"G2": "mean-phase MaxDD 2011-26 no deeper than QQQ; price-only also 2006-26",
"G3": "mean-phase excess Sharpe >= QQQ in design and holdout"},
"evaluated_at": GATE_CONFIGS,
"decision": "PASS = G1 & G2 & G3; prefer the highest full-span Sharpe among passing variants; "
"none passing -> report the gate margins and keep QQQ as the default",
"data_filter": "member-days with 63-day median dollar volume < $2M excluded (wrong-security series)",
"data_status": "signal, rb20, de-risk overlay and fund_momentum examined on these years before; "
"holdout is not independent OOS",
}
# --- prices ---------------------------------------------------------------
def raw_prices(p) -> tuple[pd.DataFrame, dict]:
"""Close as it printed that day: Yahoo split-adjusted (not dividend-adjusted)
close x the split ratios after the day; Tiingo's raw close for names Yahoo
lacks; the adjusted close as the last resort."""
with open(os.path.join("data", "research", "us_fund_raw.pkl"), "rb") as fh:
raw = pickle.load(fh)
sa = raw["close_sa"].reindex(p.dates)
ratio = raw["splits"].reindex(p.dates).fillna(0.0).replace(0.0, 1.0)
after = ratio[::-1].cumprod()[::-1] / ratio
R = (sa * after.reindex(columns=sa.columns)).reindex(columns=p.close.columns)
src = {"yahoo": 0, "tiingo": 0, "adjusted": 0}
for t in p.close.columns:
if R[t].notna().any():
src["yahoo"] += 1
continue
path = os.path.join(TIINGO, f"{t}.csv")
if os.path.exists(path):
h = pd.read_csv(path, index_col=0, parse_dates=True)
if "close" in h:
h.index = h.index.tz_localize(None) if h.index.tz is not None else h.index
R[t] = h["close"].reindex(p.dates)
src["tiingo"] += 1
continue
src["adjusted"] += 1
C = p.close
# gaps (dates a source lacks) take the adjusted close scaled by the nearest known raw/adjusted ratio
k = (R / C).replace([np.inf, -np.inf], np.nan)
k = k.ffill().bfill().fillna(1.0)
R = R.where(R.notna(), C * k)
return R, src
def commission(shares: float, price: float, sell: bool, plan: str) -> float:
shares = abs(shares)
value = shares * price
if plan == "none" or shares == 0:
return 0.0
reg = (0.0000206 * value if sell else 0.0) + 0.000003 * shares
if plan == "tiered":
return min(max(0.0035 * shares, 0.35), 0.01 * value) + 0.0014 * shares + reg
return min(max(0.005 * shares, 1.0), 0.01 * value) + reg
# --- one account ------------------------------------------------------------
_G: dict = {}
def _init(data: dict) -> None:
_G.update(data)
def run_account(job: tuple) -> dict:
"""One account over [start, end] at one rebalance phase."""
variant, config, phase, start, end = job
v, (capital, plan, market_bp, tbill_cash) = VARIANTS[variant], CONFIGS[config]
G = _G
dates, cols = G["dates"], G["cols"]
C, R, ok, member = G["C"], G["R"], G["tradable"], G["member"]
S = G["score"][v["score"]]
buffer = 1.0 if v["score"] == "fundmom" else 2.0
expo = G["derisk"] if v["derisk"] else None
rf = G["rf"]
q = G["qqq_col"]
i0 = int(dates.searchsorted(pd.Timestamp(start)))
i1 = int(dates.searchsorted(pd.Timestamp(end), side="right")) - 1
first = i0 - EVERY + phase # invested by the window's first day in every phase
a = np.zeros(len(cols)) # positions in adjusted-price units
cash = capital
last_print = np.full(len(cols), -1)
eq = np.empty(i1 - first + 1)
orders = fees = 0.0
misses = picks = 0
cash_after = []
for j, d in enumerate(range(first, i1 + 1)):
last_print[ok[d]] = d
if tbill_cash and cash > 0:
cash *= 1 + rf[d] / 252
held = np.where(a > 0)[0]
stale = held[(d - last_print[held] >= SETTLE_AFTER) & ~member[d, held] & (held != q)]
for i in stale: # delisted: settled at the last print
cash += a[i] * C[d, i]
a[i] = 0.0
if (d - first) % EVERY == 0:
row = np.where(member[d - 1], S[d - 1], np.nan)
row[q] = np.nan
chosen = select_top(row, TOP_N, a > 0, buffer)
if len(chosen) >= TOP_N // 2:
V = cash + float((a * C[d]).sum())
m = V / capital # account-dollar scale
e = 1.0 if expo is None else float(np.clip(expo[d - 1], 0.0, 1.0))
stock_w = (1 - CASH_BUFFER) * (1 - v["qqq"]) * e / len(chosen)
target = {cols[i]: stock_w for i in chosen}
if v["qqq"] > 0:
target[cols[q]] = (1 - CASH_BUFFER) * v["qqq"]
idx = {cols[i]: i for i in set(chosen) | set(np.where(a > 0)[0]) | ({q} if v["qqq"] else set())}
prices = {t: float(R[d, i]) for t, i in idx.items()}
holdings = {t: a[i] * C[d, i] / R[d, i] / m for t, i in idx.items() if a[i] > 0}
frozen = {t for t, i in idx.items() if not ok[d, i] or not np.isfinite(R[d, i])}
trades = compute_trades(holdings, cash / m, target, prices, min_trade_value=MIN_ORDER,
integer_shares=True, resize_band=v["band"], frozen=frozen)
# execution.execute_trades: sells first, then each buy (ticker order)
# shrunk to the whole shares the cash left after its fees can pay for
acct = cash / m
sells = [tr for tr in trades if tr["shares_delta"] < 0]
buys = [tr for tr in trades if tr["shares_delta"] > 0]
for tr in sells + buys:
t = tr["ticker"]
i, P = idx[t], prices[t]
if tr["shares_delta"] < 0:
k = -holdings.get(t, 0.0) if tr["target_shares"] == 0 else \
max(float(tr["shares_delta"]), -holdings.get(t, 0.0))
else:
k = float(tr["shares_delta"])
cost = lambda n: n * P + commission(n, P, False, plan) + n * P * market_bp / 1e4 # noqa: E731
if cost(k) > acct:
k = float(max(0, int(acct / (P * (1 + market_bp / 1e4) + 0.005))))
while k > 0 and cost(k) > acct:
k -= 1
if k <= 0:
continue
fee = commission(k, P, k < 0, plan) + abs(k * P) * market_bp / 1e4
acct -= k * P + fee
a[i] = 0.0 if (k < 0 and tr["target_shares"] == 0) else a[i] + k * P * m / C[d, i]
orders += 1
fees += fee * m / V
cash = acct * m
for i in chosen:
picks += 1
misses += a[i] <= 0 and ok[d, i]
cash_after.append(cash / (cash + float((a * C[d]).sum())))
eq[j] = cash + float((a * C[d]).sum())
E = pd.Series(eq, index=dates[first:i1 + 1])
r = E.pct_change().fillna(0.0).loc[start:end]
return {"variant": variant, "config": config, "phase": phase, "start": start, "returns": r,
"orders_yr": orders / (len(E) / 252),
"fees_yr": fees / (len(E) / 252), "miss_rate": misses / max(picks, 1),
"cash_avg": float(np.mean(cash_after)) if cash_after else 0.0}
# --- study -------------------------------------------------------------------
def build(p) -> dict:
pr = factor_ranks(p, "us")
mom = _blend(pr["overnight_mom"], pr["rec_126"]).where(p.member)
F = fu.load(p)
pct = lambda x: x.where(p.member).rank(axis=1, pct=True) # noqa: E731
fmom = _blend(*(pct(F[k]) for k in ("sue", "sue_rev", "gm_change", "roa_change")))
fundmom = fmom.fillna(0.5).where(mom.rank(axis=1, ascending=False) <= 30)
R, src = raw_prices(p)
cols = list(p.close.columns) + ["QQQ"]
qqq_px = (1 + bench_returns(p.dates, "QQQ").fillna(0.0)).cumprod() * 100.0
C = p.close.ffill().fillna(0.0) # 0 before a name's first print (never held)
C["QQQ"] = qqq_px
R = R.ffill() # last print values a name the day it has none
R["QQQ"] = qqq_px # adjusted QQQ stands in for its raw price
# A few delisted tickers carry another security's prices (EP = Empire Petroleum
# pennies, CPWR, SW, GR, ...): an S&P 500 member never trades < $2M a day.
dv = (p.close * p.volume).where(p.member).rolling(63, min_periods=20).median()
junk = p.member & (dv < MIN_DOLLAR_VOLUME)
ok = (p.close.notna() & (p.volume.fillna(0) > 0)) & ~junk
ok["QQQ"] = True
member = p.member & ~junk
member["QQQ"] = False
pad = lambda x: x.reindex(columns=cols).to_numpy(float) # noqa: E731
return {"dates": p.dates, "cols": cols, "C": C[cols].to_numpy(float), "R": R[cols].to_numpy(float),
"tradable": ok[cols].to_numpy(bool), "member": member[cols].fillna(False).to_numpy(bool),
"score": {"mom": pad(mom), "fundmom": pad(fundmom)},
"derisk": ov.derisk_vol(p.benchmark.ffill()).reindex(p.dates).fillna(1.0).to_numpy(float),
"rf": tbill(p.dates).fillna(0.0).to_numpy(float), "qqq_col": len(cols) - 1,
"_src": {**src, "junk_member_days": int(junk.sum().sum()), "junk_names": int(junk.any().sum())}}
def summarize(runs: list[dict], rf: pd.Series, qqq: pd.Series, a: str, b: str) -> dict:
"""Mean over phases of each window's statistics, worst phase, and the tranche book's alpha."""
out = {}
windows = {"design": (a, DESIGN_END), "holdout": (HOLDOUT, b), "full": (a, b)}
for w, (x, y) in windows.items():
S = [stats(r["returns"].loc[x:y], rf) for r in runs]
out[w] = {k: float(np.mean([s[k] for s in S])) for k in ("cagr", "sharpe", "mdd", "vol", "worst_yr")}
out[w]["worst_phase"] = {k: float(min(s[k] for s in S)) for k in ("cagr", "sharpe", "mdd")}
tranche = pd.concat([r["returns"].loc[x:y] for r in runs], axis=1).mean(axis=1)
al = alpha_stats(tranche, qqq.loc[x:y], rf.loc[x:y])
out[w]["alpha"], out[w]["alpha_t"], out[w]["beta"] = float(al["alpha"]), float(al["t"]), float(al["beta"])
for k in ("orders_yr", "fees_yr", "miss_rate", "cash_avg"):
out[k] = float(np.mean([r[k] for r in runs]))
tranche = pd.concat([r["returns"] for r in runs], axis=1).mean(axis=1)
out["years"] = {int(yr): float(v) for yr, v in ((1 + tranche).groupby(tranche.index.year).prod() - 1).items()}
out["_tranche"] = tranche
return out
def smoke() -> None:
"""Simulator check, no study results: the frictionless account with no resize
band against the lab engine (research/lab/engine.py) on the same phase."""
from research.lab.engine import simulate
p, _ = load("us")
data = build(p)
print(f"raw price sources (columns): {data.pop('_src')}", flush=True)
_init(data)
VARIANTS["_band0"] = dict(score="mom", band=0.0, derisk=False, qqq=0.0)
rf = tbill(p.dates).fillna(0.0)
score = pd.DataFrame(data["score"]["mom"][:, :-1], index=p.dates, columns=p.close.columns)
zero = dataclasses.replace(p, costs=Costs(buy=0.0, sell=0.0))
a, b = SPAN
i0 = p.dates.searchsorted(pd.Timestamp(a))
for ph in (0, 5):
mine = run_account(("_band0", "frictionless", ph, a, b))
# lab: score at s's close, traded at s+1's close; first signal row = this account's first rebalance - 1
sp = zero.slice(p.dates[i0 - EVERY + ph - 1], b)
lab = simulate(sp, score, top_n=TOP_N, every=EVERY, buffer=2.0, exec_at="close", min_trade=0.0).returns.loc[a:b]
s1, s2 = stats(mine["returns"], rf), stats(lab, rf)
corr = float(np.corrcoef(mine["returns"], lab.reindex(mine["returns"].index).fillna(0))[0, 1])
print(f"phase {ph}: account cagr {s1['cagr']:.3f} sh {s1['sharpe']:.2f} mdd {s1['mdd']:.3f} | "
f"lab cagr {s2['cagr']:.3f} sh {s2['sharpe']:.2f} mdd {s2['mdd']:.3f} | daily corr {corr:.4f}")
for cn in ("10k_tiered", "10k_fixed"):
r = run_account(("base", cn, 0, a, b))
print(f"base {cn} phase 0: orders/yr {r['orders_yr']:.0f} fees/yr {r['fees_yr']*100:.2f}% "
f"miss {r['miss_rate']:.3f} cash after rebalance {r['cash_avg']:.3f}")
def main() -> None:
os.makedirs(OUT, exist_ok=True)
res_path = os.path.join(OUT, "results.json")
if "--smoke" in sys.argv:
return smoke()
if os.path.exists(res_path):
raise SystemExit(f"{res_path} exists; this study does not overwrite results")
with open(os.path.join(OUT, "prereg.json"), "w") as fh:
json.dump(PREREG, fh, indent=2, default=list)
p, _ = load("us")
data = build(p)
print(f"raw price sources (columns): {data.pop('_src')}", flush=True)
jobs = [(vn, cn, ph, *SPAN) for vn in VARIANTS for cn in CONFIGS for ph in range(EVERY)]
jobs += [(vn, cn, ph, *LONG) for vn, v in VARIANTS.items() if v["score"] == "mom"
for cn in LONG_CONFIGS for ph in range(EVERY)]
_init(data) # forked workers inherit the arrays
with ProcessPoolExecutor(max_workers=min(16, os.cpu_count() or 4),
mp_context=__import__("multiprocessing").get_context("fork")) as pool:
runs = list(pool.map(run_account, jobs, chunksize=4))
rf = tbill(p.dates).fillna(0.0)
qqq = bench_returns(p.dates, "QQQ").fillna(0.0)
res = {"qqq": {}, "variants": {}, "verdict": {}}
for label, (a, b) in (("span", SPAN), ("long", LONG)):
res["qqq"][label] = {w: {k: float(v) for k, v in stats(qqq.loc[x:y], rf).items()}
for w, (x, y) in {"design": (a, DESIGN_END), "holdout": (HOLDOUT, b),
"full": (a, b)}.items()}
tranches = {}
for vn in VARIANTS:
res["variants"][vn] = {}
for label, (a, b) in (("span", SPAN), ("long", LONG)):
for cn in CONFIGS:
group = [r for r in runs if r["variant"] == vn and r["config"] == cn and r["start"] == a]
if not group:
continue
s = summarize(group, rf, qqq, a, b)
tranches[f"{vn}|{cn}|{label}"] = s.pop("_tranche")
res["variants"][vn][f"{cn}|{label}"] = s
Q, QL = res["qqq"]["span"], res["qqq"]["long"]
rows = []
for vn, v in VARIANTS.items():
for cn in GATE_CONFIGS:
s = res["variants"][vn][f"{cn}|span"]
g1 = s["design"]["cagr"] >= Q["design"]["cagr"] and s["holdout"]["cagr"] >= Q["holdout"]["cagr"]
g2 = s["full"]["mdd"] >= Q["full"]["mdd"]
if v["score"] == "mom":
g2 = g2 and res["variants"][vn][f"{cn}|long"]["full"]["mdd"] >= QL["full"]["mdd"]
g3 = s["design"]["sharpe"] >= Q["design"]["sharpe"] and s["holdout"]["sharpe"] >= Q["holdout"]["sharpe"]
res["variants"][vn][f"{cn}|gates"] = {"G1": g1, "G2": g2, "G3": g3, "pass": g1 and g2 and g3}
pd.set_option("display.width", 250)
for cn in CONFIGS:
print(f"\n== {cn}, {SPAN[0][:4]}-{SPAN[1][:4]} (mean of {EVERY} phases) ==")
T = []
for vn in VARIANTS:
s = res["variants"][vn][f"{cn}|span"]
g = res["variants"][vn].get(f"{cn}|gates", {})
T.append({"variant": vn, "cagr_des": s["design"]["cagr"], "cagr_hold": s["holdout"]["cagr"],
"sh_des": s["design"]["sharpe"], "sh_hold": s["holdout"]["sharpe"],
"mdd": s["full"]["mdd"], "mdd_worst_ph": s["full"]["worst_phase"]["mdd"],
"alpha_hold": s["holdout"]["alpha"], "t_hold": s["holdout"]["alpha_t"],
"orders_yr": s["orders_yr"], "fees_yr": s["fees_yr"], "miss": s["miss_rate"],
"cash": s["cash_avg"], **{k: g.get(k, "") for k in ("G1", "G2", "G3", "pass")}})
T.append({"variant": "QQQ", "cagr_des": Q["design"]["cagr"], "cagr_hold": Q["holdout"]["cagr"],
"sh_des": Q["design"]["sharpe"], "sh_hold": Q["holdout"]["sharpe"], "mdd": Q["full"]["mdd"]})
print(pd.DataFrame(T).to_string(index=False, float_format=lambda x: f"{x:.3f}"))
print(f"\n== price-only variants {LONG[0][:4]}-{LONG[1][:4]} ==")
for vn, v in VARIANTS.items():
if v["score"] != "mom":
continue
for cn in LONG_CONFIGS:
s = res["variants"][vn][f"{cn}|long"]["full"]
print(f"{vn:22s} {cn:14s} cagr {s['cagr']*100:5.1f}% sharpe {s['sharpe']:.2f} mdd {s['mdd']*100:5.1f}% "
f"worst yr {s['worst_yr']*100:5.1f}%")
print(f"{'QQQ':22s} {'':14s} cagr {QL['full']['cagr']*100:5.1f}% sharpe {QL['full']['sharpe']:.2f} "
f"mdd {QL['full']['mdd']*100:5.1f}%")
verdict = {}
for cn in GATE_CONFIGS:
passing = [vn for vn in VARIANTS if res["variants"][vn][f"{cn}|gates"]["pass"]]
best = max(passing, key=lambda vn: res["variants"][vn][f"{cn}|span"]["full"]["sharpe"]) if passing else None
verdict[cn] = {"passing": passing, "preferred": best}
res["verdict"] = verdict
print(f"\n-> {verdict}")
with open(res_path, "w") as fh:
json.dump(res, fh, indent=1, default=float)
pd.DataFrame(tranches).to_pickle(os.path.join(OUT, "daily.pkl"))
from research.lab.trials import append_trials
append_trials(os.path.join(RESULTS, "trials_us.jsonl"),
[{"name": f"small_account_momentum:{vn}:{cn}",
**{k: v for k, v in res["variants"][vn][f"{cn}|span"]["full"].items() if k != "worst_phase"}}
for vn in VARIANTS for cn in GATE_CONFIGS],
{"study": "small_account_momentum", "role": "preregistered"})
if __name__ == "__main__":
main()

View File

@@ -52,6 +52,26 @@ class ExecutionContracts(unittest.TestCase):
self.assertAlmostEqual(state["daily_equity"][str(day.date())], batch["equity"].loc[day], places=2)
self.assertEqual(state["trade_log"], batch["trade_log"])
def test_recorded_ideal_return_compounds_to_the_frictionless_ledger(self):
dates = pd.bdate_range("2026-01-05", periods=6)
prices = pd.DataFrame({"A": [100, 110, 104, 70, 90, 95], "B": [20, 21, 22, 22, 23, 21.5]}, index=dates)
targets = pd.DataFrame([[.5, .5], [.5, .5], [0, .9], [0, 1], [.6, .4], [.6, .4]],
index=dates, columns=prices.columns)
ideal = simulate_book(targets, prices, 10000, transaction_cost=0, fixed_fee=0, integer_shares=False,
resize_band=0, min_trade_value=0)
state = trader.init_state("us", "test", 10000)
for day in dates: # live-like: whole shares, fees, the 10% band
trader._run_session(state, "us", day, "B", prices, prices, {"tx_cost": .001, "fixed_fee": 2.},
True, targets.loc[day])
log = state["daily_log"]
self.assertNotIn("ideal_return", log[0])
self.assertEqual(log[2]["target"], {"B": 0.9})
growth = np.cumprod([1 + e["ideal_return"] for e in log[1:]])
np.testing.assert_allclose(growth, ideal["equity"].iloc[1:] / ideal["equity"].iloc[0], rtol=1e-6)
window = trader._ideal_window(state)
self.assertEqual(window["start"], str(dates[0].date()))
self.assertAlmostEqual(window["ideal"], growth[-1] - 1)
def test_a_delisted_stock_is_settled_at_its_last_print_in_both_ledgers(self):
dates = pd.bdate_range("2026-01-05", periods=9)
# A is acquired: last print on day 2, out of the index from day 4.

View File

@@ -0,0 +1,64 @@
import unittest
import numpy as np
import pandas as pd
from research.lab.futures_stock_signals import configurations, overnight_exposure, risk_exposure
class FuturesTimingTests(unittest.TestCase):
def setUp(self):
dates = pd.bdate_range("2020-01-01", periods=180)
t = np.arange(len(dates))
equity = 100 * np.exp(.001 * t + .002 * np.sin(t))
rates = 100 * np.exp(.0001 * t + .0002 * np.sin(t))
self.prices = pd.DataFrame({"ES=F": equity, "NQ=F": equity, "ZN=F": rates,
"SPY": equity, "QQQ": equity, "IEF": rates}, index=dates)
def test_future_prices_cannot_change_earlier_decisions(self):
changed = self.prices.copy()
changed.iloc[110:, :3] *= .7
for config in configurations():
original, known = risk_exposure(self.prices, config)
modified, modified_known = risk_exposure(changed, config)
last = 110 + config.get("lag", 1)
with self.subTest(config=config["name"]):
pd.testing.assert_series_equal(original.iloc[:last], modified.iloc[:last])
pd.testing.assert_series_equal(known.iloc[:last], modified_known.iloc[:last])
def test_same_day_futures_crash_is_not_usable_for_entry_or_preceding_night(self):
config = next(c for c in configurations() if c["name"] == "futures_equity_shock_default")
changed = self.prices.copy()
changed.iloc[110:, :2] *= .7
entry, _ = risk_exposure(changed, config)
self.assertEqual(entry.iloc[110], 1.0)
self.assertEqual(entry.iloc[111], .5)
night = overnight_exposure(entry)
self.assertEqual(night.iloc[111], 1.0)
self.assertEqual(night.iloc[112], .5)
def test_identical_etf_and_futures_inputs_have_identical_rules(self):
configs = {c["name"]: c for c in configurations()}
for name, config in configs.items():
if not name.startswith("futures_"):
continue
with self.subTest(config=name):
a, av = risk_exposure(self.prices, config)
b, bv = risk_exposure(self.prices, configs[name.replace("futures_", "etf_", 1)])
pd.testing.assert_series_equal(a, b, check_names=False)
pd.testing.assert_series_equal(av, bv, check_names=False)
def test_unavailable_prices_are_not_backward_filled_from_future(self):
config = next(c for c in configurations() if c["name"] == "futures_equity_shock_default")
missing = self.prices.copy()
missing.iloc[110, :2] = np.nan
exposure, known = risk_exposure(missing, config)
self.assertFalse(known.iloc[111])
self.assertEqual(exposure.iloc[111], 1.0)
prefix, prefix_known = risk_exposure(missing.iloc[:112], config)
pd.testing.assert_series_equal(exposure.iloc[:112], prefix)
pd.testing.assert_series_equal(known.iloc[:112], prefix_known)
if __name__ == "__main__":
unittest.main()

74
tests/test_ibkr_trader.py Normal file
View File

@@ -0,0 +1,74 @@
"""Guards of the IBKR order layer (ibkr_trader.py): which accounts it may trade,
how much it sizes, and which orders it will send."""
import unittest
import ibkr_trader as it
class AccountGuard(unittest.TestCase):
def test_paper_account_trades_without_flags(self):
self.assertFalse(it.check_account(["DU123"], "DU123", live=False, armed=False))
def test_account_must_be_managed_by_the_login(self):
with self.assertRaises(SystemExit):
it.check_account(["DU123"], "U999", live=True, armed=True)
with self.assertRaises(SystemExit):
it.check_account(["U999"], "", live=True, armed=True)
def test_live_account_needs_flag_and_armed_file(self):
with self.assertRaises(SystemExit):
it.check_account(["U999"], "U999", live=False, armed=True)
with self.assertRaises(SystemExit):
it.check_account(["U999"], "U999", live=True, armed=False)
self.assertTrue(it.check_account(["U999"], "U999", live=True, armed=True))
def test_read_only_use_skips_the_trading_flags(self):
self.assertTrue(it.check_account(["U999"], "U999", live=False, armed=False, trading=False))
def test_live_account_number_is_masked(self):
self.assertEqual(it.mask("U1234567"), "U***567")
self.assertEqual(it.mask("DU123"), "DU123")
class Sizing(unittest.TestCase):
def test_orders_are_sized_on_the_capped_equity(self):
trades = it.size_orders({"TQQQ": 1.0}, {}, {"TQQQ": 100.0}, nav=972_000.0, resize_band=0.1)
self.assertEqual([t["ticker"] for t in trades], ["TQQQ"])
self.assertLessEqual(trades[0]["shares_delta"] * 100.0, it.MAX_EQUITY * (1 - it.CASH_BUFFER))
self.assertGreater(trades[0]["shares_delta"], 95)
def test_a_switch_sells_the_old_leg_and_buys_the_new(self):
trades = it.size_orders({"GLD": 1.0}, {"TQQQ": 100}, {"TQQQ": 100.0, "GLD": 300.0}, nav=10_000.0,
resize_band=0.1)
by = {t["ticker"]: t["shares_delta"] for t in trades}
self.assertEqual(by["TQQQ"], -100)
self.assertLessEqual(by["GLD"] * 300.0, 10_000.0 * (1 - it.CASH_BUFFER))
self.assertGreaterEqual(by["GLD"], 32)
def test_missing_price_stops_the_run(self):
with self.assertRaises(SystemExit):
it.size_orders({"TQQQ": 1.0}, {"GLD": 3}, {"TQQQ": 100.0}, nav=10_000.0, resize_band=0.1)
class OrderWhitelist(unittest.TestCase):
def test_only_universe_symbols_and_whole_shares(self):
with self.assertRaises(SystemExit):
it.validate_orders([{"ticker": "AAPL", "shares_delta": 1}], {"TQQQ"}, {})
with self.assertRaises(SystemExit):
it.validate_orders([{"ticker": "TQQQ", "shares_delta": 1.5}], {"TQQQ"}, {})
def test_never_sells_more_than_held(self):
with self.assertRaises(SystemExit):
it.validate_orders([{"ticker": "TQQQ", "shares_delta": -5}], {"TQQQ"}, {"TQQQ": 3})
it.validate_orders([{"ticker": "TQQQ", "shares_delta": -3}], {"TQQQ"}, {"TQQQ": 3})
def test_orders_are_market_on_close(self):
o = it.moc_order(-7, "DU123", "ibkr_trader 2026-10-13")
self.assertEqual((o.action, o.totalQuantity, o.orderType, o.tif, o.account), ("SELL", 7, "MOC", "DAY", "DU123"))
with self.assertRaises(ValueError):
it.moc_order(0, "DU123", "x")
if __name__ == "__main__":
unittest.main()

View File

@@ -734,11 +734,15 @@ def bar_session_weights(strategy: BarStrategy, bars: dict, days: pd.DatetimeInde
def record_daily_snapshot(state: dict, date_str: str, prices: dict,
trades: list[dict], prev_equity: float) -> None:
trades: list[dict], prev_equity: float,
target: dict | None = None, ideal_return: float | None = None) -> None:
"""Append a daily snapshot to state['daily_log'].
Each entry captures the full picture: date, holdings with prices/values,
cash, total equity, daily return, and today's operations.
cash, total equity, daily return, and today's operations. ``target`` is the
weight the strategy asked to hold from this close, and ``ideal_return`` what
the previous entry's target earned to this close without whole shares, fees,
the resize band or idle cash (see ``_ideal_return``).
"""
holdings = state["holdings"]
cash = state["cash"]
@@ -778,6 +782,10 @@ def record_daily_snapshot(state: dict, date_str: str, prices: dict,
"holdings": positions,
"operations": operations,
}
if target is not None:
entry["target"] = {t: round(float(w), 6) for t, w in sorted(target.items())}
if ideal_return is not None:
entry["ideal_return"] = round(float(ideal_return), 8)
if "daily_log" not in state:
state["daily_log"] = []
@@ -797,6 +805,33 @@ def get_prices_for_date(tickers: list[str], date_idx, price_df: pd.DataFrame) ->
return {t: float(row[t]) for t in tickers if t in row.index and pd.notna(row[t])}
def _ideal_return(prev_entry: dict | None, day: pd.Timestamp, close_data: pd.DataFrame) -> float | None:
"""Return of ``prev_entry``'s target weights from its close to ``day``'s
close: the strategy's own weights held frictionlessly (fractional shares,
no fees, no resize band; the unallocated part earns nothing). A name
without a price on either date contributes 0. None when the previous entry
recorded no target (state written before targets were recorded)."""
if not prev_entry or "target" not in prev_entry or prev_entry["date"] >= str(day.date()):
return None
w = prev_entry["target"]
if not w:
return 0.0
p0 = get_prices_for_date(list(w), pd.Timestamp(prev_entry["date"]), close_data)
p1 = get_prices_for_date(list(w), day, close_data)
return float(sum(x * (p1[t] / p0[t] - 1) for t, x in w.items() if t in p0 and t in p1 and p0[t] > 0))
def _ideal_window(state: dict) -> dict | None:
"""Live vs ideal over the days whose ``ideal_return`` is recorded."""
log = state.get("daily_log") or []
k = next((i for i, e in enumerate(log) if e.get("ideal_return") is not None), None)
if k is None or k == 0:
return None
ideal = float(np.prod([1 + e.get("ideal_return", 0.0) for e in log[k:]]) - 1)
live = log[-1]["equity"] / log[k - 1]["equity"] - 1
return {"start": log[k - 1]["date"], "live": live, "ideal": ideal, "gap": live - ideal}
def _with_last_known_prices(state: dict, prices: dict) -> dict:
"""Fill prices of held tickers missing from the panel with their last recorded price.
@@ -923,7 +958,9 @@ def _run_session(state: dict, market: str, day: pd.Timestamp, benchmark: str,
previous = [d for d in sorted(state["daily_equity"]) if d < day_str]
prev_eq = state["daily_equity"][previous[-1]] if previous else state["initial_capital"]
state["daily_equity"][day_str] = round(post_value, 2)
record_daily_snapshot(state, day_str, close_prices, trades, prev_eq)
prev_entry = state["daily_log"][-1] if state.get("daily_log") else None
record_daily_snapshot(state, day_str, close_prices, trades, prev_eq, target=target_weights,
ideal_return=_ideal_return(prev_entry, day, close_data))
state["last_morning"] = day_str
state["last_evening"] = day_str
state["pending_trades"] = None
@@ -1317,6 +1354,7 @@ def cmd_compare(args):
"n_days": n_days,
"n_positions": len(state.get("holdings", {})),
"cash": state.get("cash", 0),
"ideal": _ideal_window(state),
})
# Sort by total return descending
@@ -1324,13 +1362,22 @@ def cmd_compare(args):
# Print ranking
print(f"\n {'#':<3} {'Strategy':<25} {'Return':>9} {'Ann.Ret':>9} "
f"{'Sharpe':>8} {'MaxDD':>8} {'Trades':>7} {'Final':>14}")
print(f" {'─'*87}")
f"{'Sharpe':>8} {'MaxDD':>8} {'Trades':>7} {'Final':>14} {'Ideal':>8} {'Gap':>7}")
print(f" {'─'*104}")
for i, r in enumerate(results, 1):
iw = r["ideal"]
ideal = f"{iw['ideal'] * 100:>+7.1f}% {iw['gap'] * 100:>+6.1f}pp" if iw else f"{'—':>8} {'—':>7}"
print(f" {i:<3} {r['name']:<25} {r['total_ret']:>+8.1f}% {r['ann_ret']:>+8.1f}% "
f"{r['sharpe']:>8.2f} {r['max_dd']:>7.1f}% {r['n_trades']:>7} "
f"${r['final']:>12,.2f}")
f"${r['final']:>12,.2f} {ideal}")
starts = sorted({r["ideal"]["start"] for r in results if r["ideal"]})
if starts:
print(f" Ideal: the strategy's own target weights held without whole shares, fees, the resize band "
f"or idle cash, from the close of {starts[0]}" + (f" (latest start {starts[-1]})" if len(starts) > 1 else "")
+ ". Gap: live return minus ideal over the same days.")
else:
print(" Ideal: not recorded yet (state files record session targets from the next session on).")
# Equity curve comparison (last 20 dates)
recent_dates = all_dates[-20:]

37
uv.lock generated
View File

@@ -2,6 +2,18 @@ version = 1
revision = 3
requires-python = ">=3.12"
[[package]]
name = "aeventkit"
version = "2.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "numpy" },
]
sdist = { url = "https://files.pythonhosted.org/packages/5c/8c/c08db1a1910f8d04ec6a524de522edd0bac181bdf94dbb01183f7685cd77/aeventkit-2.1.0.tar.gz", hash = "sha256:4e7d81bb0a67227121da50a23e19e5bbf13eded541a9f4857eeb6b7b857b738a", size = 24703, upload-time = "2025-06-22T15:54:03.961Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/8d/8c/2a4b912b1afa201b25bdd0f5bccf96d5a8b5dccb6131316a8dd2d9cabcc1/aeventkit-2.1.0-py3-none-any.whl", hash = "sha256:962d43f79e731ac43527f2d0defeed118e6dbaa85f1487f5667540ebb8f00729", size = 26678, upload-time = "2025-06-22T15:54:02.141Z" },
]
[[package]]
name = "beautifulsoup4"
version = "4.14.3"
@@ -284,6 +296,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/38/74/f94141b38a51a553efef7f510fc213894161ae49b88bffd037f8d2a7cb2f/frozendict-2.4.7-py3-none-any.whl", hash = "sha256:972af65924ea25cf5b4d9326d549e69a9a4918d8a76a9d3a7cd174d98b237550", size = 16264, upload-time = "2025-11-11T22:40:12.836Z" },
]
[[package]]
name = "ib-async"
version = "2.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aeventkit" },
{ name = "nest-asyncio" },
{ name = "tzdata" },
]
sdist = { url = "https://files.pythonhosted.org/packages/30/4d/dfc1da8224c3ffcdcd668da7283c4e5f14239a07f83ea66af99700296fc3/ib_async-2.1.0.tar.gz", hash = "sha256:6a03a87d6c06acacb0217a5bea60a8a168ecd5b5a7e86e1c73678d5b48cbc796", size = 87678, upload-time = "2025-12-08T01:42:32.004Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/80/e7/8f33801788c66f15e9250957ff7f53a8000843f79af1a3ed7a96def0e96b/ib_async-2.1.0-py3-none-any.whl", hash = "sha256:f6d8b991bdbd6dd38e700c61b3dced06ebe0f14be4e5263e2ef10ba10b88d434", size = 88876, upload-time = "2025-12-08T01:42:30.883Z" },
]
[[package]]
name = "idna"
version = "3.11"
@@ -505,6 +531,15 @@ version = "0.0.12"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/17/0d/74f0293dfd7dcc3837746d0138cbedd60b31701ecc75caec7d3f281feba0/multitasking-0.0.12.tar.gz", hash = "sha256:2fba2fa8ed8c4b85e227c5dd7dc41c7d658de3b6f247927316175a57349b84d1", size = 19984, upload-time = "2025-07-20T21:27:51.636Z" }
[[package]]
name = "nest-asyncio"
version = "1.6.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" },
]
[[package]]
name = "numpy"
version = "2.4.0"
@@ -765,6 +800,7 @@ name = "quant-trading"
version = "0.1.0"
source = { virtual = "." }
dependencies = [
{ name = "ib-async" },
{ name = "lxml" },
{ name = "matplotlib" },
{ name = "numpy" },
@@ -776,6 +812,7 @@ dependencies = [
[package.metadata]
requires-dist = [
{ name = "ib-async", specifier = ">=2.1.0" },
{ name = "lxml", specifier = ">=6.0.2" },
{ name = "matplotlib", specifier = ">=3.10.8" },
{ name = "numpy", specifier = ">=2.4.0" },