Gahow Wang
3d69744068
128 fixed runs (32 profiles x daily ledger / overnight-only x 2 costs) plus 12 post-hoc availability diagnostics on overnight_recovery_top10, 2006-10 to 2026-09. Futures trend/shock/rates rules do not beat the same rules on SPY/QQQ/IEF or a static 75% exposure; all six default schemes fail the gates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
272 lines
37 KiB
Markdown
272 lines
37 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project Overview
|
||
|
||
quant-trading: a quantitative backtesting framework and live forward-testing trader for equity strategies across US (S&P 500) and China A-shares (CSI 300), plus a web UI to browse backtests and live books. Uses Yahoo Finance data with persistent local CSV storage.
|
||
|
||
The repository has two halves that share one data directory and one `.env`:
|
||
|
||
- **Python engine** (repo root, `uv`): strategies, backtests, `trader.py`, research scripts.
|
||
- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`. Both backtest paths run through `backtest_runner.py`.
|
||
|
||
`bridge.py`'s JSON output and `web/src/shared/strategyTypes.ts` are one contract: change both in the same commit.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
# Backtesting
|
||
uv run python main.py # Run US backtest (default)
|
||
uv run python main.py --market cn # Run China A-share backtest
|
||
uv run python main.py --capital 50000 # Custom starting capital
|
||
uv run python main.py --top-n 20 # Override stock selection count
|
||
uv run python main.py --years 5 # Backtest last N years only
|
||
uv run python main.py --no-plot # Skip matplotlib charts
|
||
uv run python main.py --fixed-fee 2.0 # Add $2 per-trade fixed fee
|
||
uv run python main.py --execution open-close # Signal on open, execute at close
|
||
|
||
# Daily trading simulator
|
||
uv run python trader.py auto --market us --strategy recovery_mom_top10 # Single daily run (for cron)
|
||
uv run python trader.py morning --market us --strategy recovery_mom_top10 # Morning: generate orders
|
||
uv run python trader.py evening --market us --strategy recovery_mom_top10 # Evening: record execution
|
||
uv run python trader.py monitor # Monitor ALL markets & strategies (for tmux)
|
||
uv run python trader.py monitor --market us # Monitor US only
|
||
uv run python trader.py status --market us --strategy recovery_mom_top10 # Portfolio status
|
||
uv run python trader.py simulate --market us --strategy recovery_mom_top10 --start 2026-01-01 --end 2026-04-01 # Historical replay
|
||
uv run python trader.py log --market us --strategy sim_recovery_mom_top10 --start 2026-03-01 # View daily log
|
||
uv run python trader.py simulate --integer-shares --fixed-fee 2.0 ... # Integer shares mode
|
||
uv run python trader.py compare --market us --book selected_2026_09 # A named live book
|
||
|
||
# Agent-reviewed strategy overnight_recovery_agent_top10: daily loop (skill .claude/skills/agent-stock-review)
|
||
uv run python agent_review.py candidates # after the US close: top-50 pack -> data/agent/us/candidates_<as_of>.json
|
||
uv run python agent_review.py candidates --analyst-refresh # immutable Yahoo EPS-trend/revision snapshots + analyst context in the pack
|
||
uv run python agent_review.py submit data/agent/us/draft_<as_of>.json # before 16:00 New York of the next session
|
||
uv run python agent_review.py evaluate # score IC, veto outcomes, book vs overnight_recovery_top10
|
||
|
||
# Overnight-only momentum book on Alpaca PAPER (MOC buy top 10 at close-20 min, MOO sell at open-20 min; refuses non-paper endpoints)
|
||
uv run python overnight_paper.py signal # next session's top 10, no orders
|
||
uv run python overnight_paper.py buy|sell --dry-run # show the orders it would send
|
||
uv run python overnight_paper.py report # paper fills vs official open/close prints, nightly returns
|
||
uv run python overnight_paper.py daemon # the scheduler (service quant-trading-overnight-paper)
|
||
|
||
# trend_rider_v10 in an IBKR account: today's target as trader.py morning computes it, whole-share MOC orders
|
||
# via IB Gateway, fills scored against the closing cross (IBKR_ACCOUNT/IBKR_PORT/IBKR_CLIENT_ID from .env)
|
||
uv run python ibkr_trader.py check # account, positions, open orders (read-only)
|
||
uv run python ibkr_trader.py plan [--what-if] # today's orders; --what-if: IBKR validates/prices them, nothing placed
|
||
uv run python ibkr_trader.py submit --live # MOC orders, close-30 to close-15 min; live needs data/ibkr_live/ARMED
|
||
uv run python ibkr_trader.py report # executions, fills vs closing cross, live vs simulated account
|
||
uv run python ibkr_trader.py daemon --live # the scheduler (service quant-trading-ibkr-live, not enabled)
|
||
|
||
# Precompute the strategy page's backtests (writes data/precomputed/)
|
||
uv run python precompute_backtest.py
|
||
|
||
# Factor research harness (PIT universe, next-open fills, phase averaging; data in data/research/)
|
||
uv run python -m research.lab.fetch_us # 20y OHLCV for every S&P 500 ever-member
|
||
uv run python -m research.lab.fetch_cn_yahoo # CSI 300/500 ever-members (membership from baostock snapshot)
|
||
PYTHONPATH=. uv run python -m research.lab.factor_scan --market us # or cn_hs300
|
||
PYTHONPATH=. uv run python -m research.lab.stepwise --market us --objective maximin
|
||
PYTHONPATH=. uv run python -m research.lab.evaluate --market us # candidates vs champions
|
||
uv run python -m research.lab.fetch_us_fundamentals # SEC EDGAR point-in-time facts + Yahoo splits (US)
|
||
PYTHONPATH=. uv run python -m research.lab.rotation --market us # walk-forward factor weighting schemes
|
||
PYTHONPATH=. uv run python -m research.lab.persistence # does last year's best combo persist?
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.us_sweep --part overlays # construction / overlay sweeps
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.us_final # 3-sleeve US portfolio check
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.short_horizon # short-horizon / residual sleeves, concentration
|
||
PYTHONPATH=. uv run python -m research.lab.levered_book # vol-managed leverage, beta-hedged alpha
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_long_history [--no-proxy] # indices from 1927/1985, ETFs, BTC, FRED yields
|
||
PYTHONPATH=. uv run python -m research.lab.letf_history # synthetic 3x ETFs from 1990; TrendRider stress test
|
||
PYTHONPATH=. uv run python -m research.lab.vol_trend # vol-targeted leveraged trend, walk-forward
|
||
PYTHONPATH=. uv run python -m research.lab.diversifiers # multi-asset ETF trend, bitcoin trend
|
||
PYTHONPATH=. uv run python -m research.lab.us_target50 # the book 2006-2026 and the leverage 50%/yr needs
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.qqq_scan # alpha vs QQQ of every sleeve / factor / mega-cap variant
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.beat_qqq # sleeves picked on 2006-16, judged 2017-26; odds of beating QQQ
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.momentum_fundamentals [--sensitivity] # financial-statement vetoes/confirmations on overnight_recovery_top10
|
||
uv run python -m research.lab.analyst_estimates evaluate # EPS-revision overlay vs the same ledger/costs; forward only, no live promotion
|
||
uv run python -m research.lab.fetch_insider && PYTHONPATH=. uv run python -m research.lab.insider # SEC Form 4 insider factors
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_insider --universe mid small && PYTHONPATH=. uv run python -m research.lab.insider_mid # S&P 400/600 Form 4 factors (all REJECT 2026-10-09)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_lazy_prices && PYTHONPATH=. uv run python -m research.lab.lazy_prices # 10-K/10-Q text similarity, 8-K earnings dates, SIC
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.earnings # earnings-announcement return (needs sec_8k_earnings.csv)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.industry # industry momentum on SIC groups
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_us_mid holdings|prices|status # PIT S&P 400/600 from iShares holdings (research/lab/panel_mid.py)
|
||
PYTHONPATH=. uv run python -m research.lab.review_live_us # Apr-Sep 2026 replay review: attribution, AI-chain stages, missed winners
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_supply_links [--universe mid] # 10-K customer/supplier/competitor mentions (sec_10k_links.csv)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.lead_lag --part all # economic-link momentum, lead-lag network, earnings spillover
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.theme_propagation # does a hot industry's linked laggard catch up?
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.lead_lag_small # the same links on S&P 400/600, 2019-26
|
||
PYTHONPATH=. uv run python -m research.lab.sector_cycle # business-cycle rotation, sector momentum/reversal, semis cycle
|
||
uv run python -m research.lab.fetch_wiki_attention && PYTHONPATH=. uv run --with numba python -m research.lab.attention # Wikipedia-pageview attention
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.concentration # industry caps on the momentum sleeve
|
||
PYTHONPATH=. uv run python -m research.lab.universe_midcap # overnight_recovery_top10 on S&P 500 ∪ S&P 400 (preregistered; FAIL 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.union_shield # SPY>MA150 overlay on the union arms (preregistered; FAIL 2026-10-09)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.stream_portfolio # risk-budgeted book of momentum/lowrisk/TrendRider/ETF TSMOM/BTC streams, 2006-26 (preregistered; REJECT 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.hedged_momentum # momentum with its beta hedged by index futures (quick check; STOP 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.futures_trend [--no-proxy] # 24-market futures TSMOM on Yahoo =F + data checks (quick check; Sharpe 0.44, DATA 2026-10-08)
|
||
uv run python -m research.lab.futures_stock_signals --output data/research/results/futures_stock_signals_new # lagged futures vs ETF stock-exposure rules; 2026-10-17: 128 fixed runs + 12 availability diagnostics, retain baseline; report in futures_stock_signals_20261017/report.md
|
||
PYTHONPATH=. uv run python -m research.lab.managed_futures_book [--no-proxy] # three sleeves + managed futures (tsmom24 proxy -> DBMF), 2006-26 (preregistered; REJECT 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.small_cap_reversal # 5-day reversal on S&P 400/600, EW-hedged, spread costs (quick check; STOP 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.turn_of_month # turn-of-month premium 1928-2026, 3x in the window (quick check; STOP 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.short_vol [--no-proxy] # short VIX futures with a VIX/VIX3M filter, 2006-26 (quick check; STOP 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_alpaca etfs|stocks|status # Alpaca free SIP intraday bars 2016+ (1-min ETFs, 5-min liquid S&P 1500; keys from .env, never logged)
|
||
PYTHONPATH=. uv run python -m research.lab.intraday_momentum # SPY noise-area intraday momentum replication + 5 ETFs (preregistered; STOP 2026-10-09)
|
||
PYTHONPATH=. uv run python -m research.lab.orb_stocks_in_play # 5-min opening-range breakout on the top-20 stocks in play (preregistered; STOP 2026-10-09: 5-min bounds -70%/yr vs +71%/yr, optimistic breakeven slippage 3.6 bp/side)
|
||
PYTHONPATH=. uv run python -m research.lab.overnight_only # momentum names held close->open only, MOC/MOO costs (preregistered; STOP 2026-10-09: net Sharpe 1.29-1.42 < 1.5, but the sleeve's whole return is overnight)
|
||
PYTHONPATH=. uv run python -m research.lab.overnight_book # three-sleeve book with the overnight-only momentum sleeve, leverage for 50%/yr (preregistered; CONTINUE 2026-10-09: Sharpe 0.98 -> 1.38, MaxDD -37% -> -17% at 1 bp/side; 4 bp/side erases it)
|
||
PYTHONPATH=. uv run python -m research.lab.overnight_book_mf [--no-proxy] # the overnight books + a DBMF/CTA sleeve, levered to 50%/yr (preregistered; KEEP 2026-10-09: levered MaxDD unchanged, no CTA sleeve)
|
||
PYTHONPATH=. uv run python -m research.lab.daytime_reversal # buy the S&P 500's biggest overnight losers at 09:35, sell at the close (preregistered; STOP 2026-10-09: gross 1-6 bp/day, net Sharpe ~0)
|
||
PYTHONPATH=. uv run python -m research.lab.overnight_midsmall # the overnight-only momentum book on S&P 400/600 (preregistered; STOP 2026-10-09: 0.93 net at 3 bp/side, corr 0.83 with the S&P 500 book)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.dip_buying # buy-the-dip: SPY/QQQ RSI(2), trend-filtered stock RSI(2), drawdown tables (preregistered; STOP 2026-10-12: SPY 46 bp/trade t 3.4 but 14% of days, 3x Sharpe 0.57 = SPY; stocks breakeven 10 bp/side)
|
||
PYTHONPATH=. uv run python -m research.lab.dip_leverage_overlay # +1x SPY only inside SPY RSI(2) dip windows, on the overnight_book books (preregistered; STOP 2026-10-12: Sharpe 0.98->0.99 / 1.38->1.33, placebo rank 77-88%; the books already earn the bounce at ~2x SPY beta)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.factor_census [--part construction] # every lab factor (price, EDGAR, insider, Lazy Prices, EAR, SVR, FTD) measured one way on the current PIT panel: IC, quintile spread, top-20 alpha vs EW (2026-10-12: 1 A low_svr21, 2 B accruals/rev_5, 69 N; no |t|>=3)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.svr_combo # low_svr21 blended into overnight_recovery_top10 / as a fourth book sleeve, 2018-26 (preregistered; STOP 2026-10-12: blend Sharpe 0.81->0.87 but MaxDD -35%->-41%; book 1.49->1.42)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_options snapshot|status # daily S&P 500 option chains after the close: indicative quotes (forward-only from 2026-10-12) + OPRA daily bars + print-vs-quote quality sample; data/options/us/; timer quant-trading-options
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_options backfill # OPRA daily option bars since 2024-02 for every S&P 500 member since then (resumable, ~2-3 h; pauses 16:10-16:55 NY); then builds volume_panel.csv.gz
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_options volume-panel # date x underlying option volume (all 0-100 DTE strikes and the 5-100 DTE / 0.7-1.3 band), os_volume, pc_volume
|
||
PYTHONPATH=. uv run python -m research.lab.auction_audit [fetch] # overnight sleeve: Yahoo open/close vs the primary exchange's opening/closing cross prints (Alpaca SIP trades), 800 nights 2016-26 (preregistered; PASS 2026-10-12: +0.20 bp a side incl. SEC fee, the 1 bp backtest cost is conservative)
|
||
PYTHONPATH=. uv run python -m research.lab.no_letf_book # books without leveraged ETFs for the $10k IBKR account: QQQ trend held in QQQ (+ Reg T margin), momentum blends, vs QQQ at matched risk (preregistered; NONE BEATS QQQ 2026-10-13: Sharpe gain < 0.10; trend's gain is crashes, MaxDD -28% vs -83% since 1990; x2 margin 17.0% vs 14.9%)
|
||
PYTHONPATH=. uv run python -m research.lab.small_account_momentum # 10-day momentum in a $10k/$20k IBKR account: whole shares, Tiered/Fixed fees, rb20 / SPY de-risk / fundamental re-sort / half QQQ vs QQQ (preregistered; NONE PASSES 2026-10-13: all fail 2011-16 Sharpe; rb20_fundmom 26.3%/0.96/-32% vs QQQ 19.1%/0.87/-35% 2011-26 but no 2008 test)
|
||
PYTHONPATH=. uv run python -m research.lab.option_overlay [fetch] # leverage routes (Reg T margin, box, MES futures, calls) and S&P 500 puts (Cboe PPUT3M/PPUT held to expiry) for $10k-$100k accounts (preregistered; 2026-10-12: no route reaches 50% under Reg T, cap L 1.71 -> 36%/yr; puts B FAIL +1.0pp)
|
||
PYTHONPATH=. uv run --with numba python -m research.lab.option_volume # Johnson-So O/S and put/call volume on OPRA bars 2024-03..2026-09 (preregistered; STOP 2026-10-12: O/S sign reversed, t -1.6, corr -0.65 with momentum; put/call t 0.4)
|
||
uv run python -m research.lab.fetch_us_fundamentals --universe mid # EDGAR PIT facts for S&P 400 ever-members (name-checked CIKs)
|
||
uv run python -m research.lab.fetch_us_fundamentals --universe small # the same for S&P 600 (IJR)
|
||
PYTHONPATH=. uv run python -m research.lab.midcap_sleeve [--only h1..h4] [--universe small] # S&P 400 sleeves vs IJH; S&P 600 replication vs IJR (all REJECT 2026-10-08)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_shortvolume # FINRA REG SHO daily short volume (~8y archive, resume-safe)
|
||
PYTHONPATH=. uv run python -m research.lab.short_volume [us|midsmall|all] # SVR (short-volume ratio) IC / quintile scan on PIT panels
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_fails # SEC fails-to-deliver archives (2018+, resume-safe)
|
||
PYTHONPATH=. uv run python -m research.lab.fails [us|midsmall|all] # FTD intensity scans (value/shares per ADV$, PIT +21d publication lag)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_13d [--quarters N] # EDGAR quarterly index: SC 13D/13G rows (activist/passive stakes)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_13d_targets [--limit N] # subject company of each SC 13D (index lists the filer)
|
||
PYTHONPATH=. uv run python -m research.lab.activists [us|midsmall|all] # 13D event study: market-adjusted CARs, momentum/SVR splits
|
||
PYTHONPATH=. uv run python -m research.lab.activist_strategy [midsmall|us] # 13D event sleeve (lab engine; dead in 2021-26 — research only)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_13f # SEC DERA 13F structured data sets (2016q1-2023q4), per-CUSIP aggregates
|
||
PYTHONPATH=. uv run python -m research.lab.inst_flow [us|midsmall] # institutional q/q flow factor scan (IC t 2.0 but Q5-Q1 spread nil)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_gdelt && PYTHONPATH=. uv run python -m research.lab.news_volume # GDELT news volume (API rate-limited to ~1 query/10-15s; sample only)
|
||
PYTHONPATH=. uv run python -m research.svr_overlay # SVR-veto overlay on overnight_recovery_top10 (lab)
|
||
|
||
# Tiingo backfill (delisted names Yahoo purged; keys from TIINGO_API_KEY, never logged)
|
||
PYTHONPATH=. uv run python -m research.lab.fetch_tiingo --file list.txt --min-interval 75 # EOD backfill (free tier: 500 symbols/month, 50/hour; cache = resume)
|
||
PYTHONPATH=. uv run python -m research.lab.panel_gaps # price gaps vs PIT membership (both panels); writes data/research/tiingo_plan.json
|
||
PYTHONPATH=. uv run python -m research.lab.tiingo_merge # report: overlay caches onto the research S&P panel (auto at panel build)
|
||
PYTHONPATH=. uv run python -m research.lab.tiingo_merge --production [--apply] # merge into data/us*.csv (backups .bak-tiingo; --apply writes)
|
||
PYTHONPATH=. uv run python -m research.lab.tiingo_merge --yahoo-mid # convert caches to yahoo_mid format (fetch_us_mid.load_bars falls back to it)
|
||
PYTHONPATH=. uv run python -m research.factor_panel_check # pre-vs-post overlay S&P factor comparison
|
||
PYTHONPATH=. uv run python -m research.midsmall_scan --panel both # mid/small factor scan: Yahoo-only vs Tiingo-extended panel
|
||
PYTHONPATH=. uv run python -m research.midsmall_rebuild # convert every cached Tiingo name + rebuild the mid/small PIT panels (re-run after each monthly batch)
|
||
PYTHONPATH=. uv run python -m research.prod_impact [STRATEGY ...] # A/B registered replays: backfilled store vs .bak-tiingo snapshot
|
||
|
||
# Setup
|
||
uv sync # Install/sync dependencies
|
||
cp .env.example .env # Then fill IPADS_API_KEY and QUANT_MARKET_PROXY
|
||
(cd web && npm ci)
|
||
|
||
# Tests
|
||
uv run python -m unittest discover tests
|
||
(cd web && npm test && npm run build)
|
||
|
||
# Web UI in development (API :8787, Vite :5173)
|
||
(cd web && npm run dev)
|
||
```
|
||
|
||
## Architecture
|
||
|
||
**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia; CSI 300 from China Securities Index's published constituent file, falling back to the latest `data/csi300_history.json` snapshot, then Wikipedia, whose table lagged the index by ~60 names in 2026) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
|
||
|
||
**Point-in-time membership** (`universe_history.py`): S&P 500 intervals rebuilt from Wikipedia's "Historical components of the S&P 500" change log into `data/sp500_history.json`. The log is sparse before 2011, so current members' earlier additions come from the list page's "Date added"; companies removed before 2011 that the log never recorded stay missing, and the research panel holds only 347 / 367 / 438 / 453 / 484 priced members in mid-2004 / 2006 / 2010 / 2012 / 2016 (about 500 from 2020): US research results before ~2014 are survivor-biased. CSI 300 monthly snapshots in `data/csi300_history.json` (seeded from baostock 2008–2026, appended whenever the current list changes). `mask_prices_pit(prices, market)` masks non-member dates.
|
||
|
||
**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_{open,high,low,volume}.csv`. First run downloads 10 years of history; every run re-downloads the last 30 days. New tickers from index rebalances are backfilled automatically; tickers missing from a partial batch are retried alone before being marked unavailable (retried weekly). `update(market, tickers, with_open=True)` returns (close, open); `update(..., fields=[...])` returns `{field: panel}`.
|
||
|
||
**Backtest engine** (`main.py`, `backtest_runner.py`): all replay paths use `signals.session_targets` and the cash-funded order ledger in `execution.py`. Price strategies may use the session open and fill at its close; close-only inputs and BarStrategy inputs use prior completed bars. Historical membership restricts ranks and selection separately from price history. Missing quotes and zero volume cannot fill, except that a held US stock with no print for 5 sessions that has left the index is settled at its last print (`execution.SETTLE_AFTER_SESSIONS`; acquisitions otherwise froze capital forever); the book checks available cash after fees, handles whole shares/lots, and applies conservative CN closing-limit restrictions and sell-side stamp tax. Registry replays retain at least 315 prior sessions and score all compared strategies on a common actual window. `--offline` on main/precompute reads a frozen local snapshot. Data coverage and adjusted-share/corporate-action limitations are exposed in API metadata.
|
||
|
||
`precompute_backtest.py` reruns the ledger for cash-injection scenarios; metrics use unit NAV after actual fees. The API and `web/src/shared/strategyTypes.ts` share this cash-flow contract. `ENGINE_VERSION` in `execution.py` identifies the execution protocol; API cache keys also include its version.
|
||
|
||
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Orders: whole A-share lots are allocated nearest-lot within cash (`_allocate_lots`); a held name is resized only when ≥ `RESIZE_BAND` (10%) off target. Each `daily_log` entry also records the session `target` and `ideal_return` (what the previous target earned to this close with fractional shares, no fees, no resize band and no idle cash; recorded from 2026-10-13); `compare` prints `Ideal` and `Gap` (live minus ideal) over those days, separating the strategy from $10k whole-share and fee noise. Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`), the robust-momentum bar strategies `overnight_recovery_top10`, `lowrisk_momentum_top15`, `lowbeta_sharpe_mom_top20`, `lowrisk_top20`, the US sleeve portfolios `us_three_sleeve` and `us_momentum_qqq_trend`, and the agent-reviewed `overnight_recovery_agent_top10` (US only, forward only).
|
||
|
||
**Live books** (`trader.LIVE_BOOKS`): a book is a cohort of strategies, each with its own account, started on one date with one capital and a flat fee per order. The original book (every live strategy, `MONITOR_CAPITAL`, `MARKET_FEES`) keeps state in `data/trader_*.json`; a named book keeps it in `data/books/<book>/`. The monitor runs every book; `auto`/`status`/`log`/`compare`/`replay` and `bridge.py live` take `--book`, and `bridge.py books` lists them for the UI's book selector. `selected_2026_09` (30 strategies, US $10,000 / CN ¥100,000, $2 / ¥10 per order) starts on 2026-09-25 (US) and 2026-09-28 (CN); `overnight_recovery_agent_top10` joined it on 2026-09-30 and starts with the book.
|
||
|
||
**Strategy protocol** (`strategies/base.py`, `signals.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy returns lagged weights (row t uses bars through t-1). `signals.session_targets` converts this to a feasible execution-day target; the ledger trades at that day’s close. `StockStrategy` accepts a separate `eligible` mask so pre-admission observations remain usable but nonmembers cannot be selected. A `BarStrategy` declares `fields` (close/open/high/low/volume) and receives `{field: DataFrame}`; live it uses completed bars only (row d from bars through d-1, traded at the close of d), while close-price strategies read the session's open as their latest price.
|
||
|
||
**Strategies**:
|
||
- `buy_and_hold.py` — Equal-weight static allocation
|
||
- `momentum.py` — Cross-sectional momentum (12-1 month factor), selects top_n winners
|
||
- `inverse_vol.py` — Inverse-volatility (risk parity) weighting
|
||
- `multi_factor.py` — Combines momentum + value factors with benchmark MA timing filter
|
||
- `mean_reversion.py` — Short-term mean reversion
|
||
- `trend_following.py` — MA crossover + momentum trend filter
|
||
- `dual_momentum.py` — Absolute + relative momentum
|
||
- `momentum_quality.py` — Momentum + return consistency + low drawdown
|
||
- `adaptive_momentum.py` — Momentum weighted by inverse volatility
|
||
- `recovery_momentum.py` — Recovery (price/63d low) + 12-1mo momentum composite. Best US performer.
|
||
- `robust_momentum.py` — Bar strategies from the 2026-09 PIT research (report: https://claude.ai/artifact/Mo9nrzs5SWj5FfyshERyWS): US `overnight_recovery` (overnight-return momentum + distance from 126d low), CN `lowrisk_momentum` (amplitude-cut momentum + low beta + price-volume corr + momentum/vol) and close-only `lowbeta_sharpe_mom`. Top-N with rank hysteresis, rebalance every 10 days, weights drift between rebalances. `low_risk` (US `lowrisk_top20`: idiosyncratic vol + low beta, top 20, monthly) is the defensive sleeve of the 2026-09-29 US research (report: https://claude.ai/artifact/VGhZvKkS9UBYZkLeqGjcNC).
|
||
- `sleeve_portfolio.py` — Several strategies in one account with fixed capital shares. `us_three_sleeve` = 1/3 each of `overnight_recovery_top10`, `lowrisk_top20`, `trend_rider_v7_vt24` (a mixed stock+ETF strategy); sleeve shares drift with their returns and reset at a month start once any is >25% off. 2011-2026 research: Sharpe 1.21 vs 0.88-1.03 for the sleeves alone. Research finding behind it: choosing factors by their recent record (rotation, factor momentum, last year's winner) did not beat a fixed composite; construction knobs (N, weighting, dynamic N) barely matter. On synthetic 3x ETFs from 1990 (`research/lab/letf_history.py`), TrendRider V7's 2016-2026 record does not hold: 1990-99 Sharpe 0.2-0.3, 2003-07 negative, 1990-2026 about 0.7 with MaxDD -58% (vt24) to -69% (vt36).
|
||
- `trend_rider_v9.py` — V7 pipeline (V3 regime + vol target + profit-take) plus an instrument trailing stop: while risk-on, a close 20% below the held leg's 30-day high (PIT, through t-1) switches to the risk-off basket immediately; re-entry still needs the normal regime confirmation. Registered as `trend_rider_v9` (VT34, the default) and `trend_rider_v9_vt24/_vt32/_vt36`. Engine window 2017-12→2026-10, $10k, $2/order: `trend_rider_v9` 52.3%/yr Sharpe 1.55 MaxDD -29.3%, `_vt36` 52.5%/1.53, `_vt32` 51.5%/1.57, `_vt24` 42.4%/1.58. 2010-2026 live-book simulation ($2/order): every tier improves on V7 (e.g. vt34: 34.4%/1.14 vs vt32 32.2%/1.10); 2010-2016 blocks are essentially unchanged.
|
||
- `trend_rider_v10` (same file) — V9 plus a refined profit-take: clear at +28% (not 30%), restore band 8pp (re-enter sooner on dips), 10-day re-clear lockout after a restore (stops the 2022-style clear/restore churn on GLD/DBC). Registered as `trend_rider_v10` (VT34), `_vt32`, `_vt36`. Engine window to 2026-10-07: v10 55.9%/1.66/MaxDD -27.4%, v10_vt32 55.0%/1.68, v10_vt36 56.2%/1.63 — beats every other registered strategy on both CAGR and Sharpe (previous best: v9_vt36 52.5%, v9_vt24 1.58). 2010-2026 live-book simulation confirms robustness: the PT refinement lifts every 3-4y block's Sharpe (2010-13 0.83→0.92, 2016-21 1.74→1.89, 2021-26 1.38→1.43; full period 36.8%/1.22 vs 34.4%/1.14 for plain V9-trail). Research trail: `research/tr9_{lab,run,long,trials,audit}.py`; rejected knobs are logged in `data/research/results/trials_us.jsonl` (macro throttles on VIX term/credit, RSP breadth, instrument MA exits, fast re-entry, SOXL/TECL/TMF/SQQQ baskets).
|
||
- `qqq_trend.py` — `QQQVolTrend`: QQQ exposure = 1{QQQ > 200-day average} × min(1.5, 30% / EWMA vol), held as TQQQ (E/3) + SHY. 1990-2026 on synthetic ETFs: 16.0% vs QQQ 14.9%, MaxDD -37% vs -83%; 2011-2026 on real ETFs it trails QQQ (17.5% vs 19.1%) with a smaller drawdown. `us_momentum_qqq_trend` (in `sleeve_portfolio.py`) holds it half and half with `overnight_recovery_top10`. Research behind it (`research/lab/qqq_scan.py`, `beat_qqq.py`): measured as alpha against QQQ with a 2006-16 design / 2017-26 holdout split, momentum top 10 is the only stock selection whose alpha held (+10%/yr, t 1.4); single factors, walk-forward ML, a mega-cap universe, SEC insider trades, 10-K/10-Q text similarity, earnings-announcement returns, industry momentum and a PIT S&P 400/600 universe (`panel_mid.py`) showed none. In the web engine (2016-09 to 2026-09, $2/order) `us_momentum_qqq_trend` earns 21.0% vs QQQ 20.9%, MaxDD -32%.
|
||
- `svr_momentum.py` — RESEARCH ONLY (not registered): `overnight_recovery_top10` + a FINRA short-volume veto (drop top-70%-quantile SVR names). The 2026-10-07 window looked good (Sharpe 0.87→0.89, MaxDD -47%→-39%) but the quantile neighborhood killed it — 0.60/0.75/0.80 vetoes all score below baseline. A lone spike; registered and reverted the same day. The raw cross-sectional reading (SVR negative predictor, `research/lab/short_volume.py`) stands; the concentrated-portfolio exclusion rule does not.
|
||
- `agent_review.py` — `overnight_recovery_agent_top10`: overnight_recovery's top 50, re-ranked each day by an LLM agent's scores (−2..+2 from filings, fundamentals, demand-chain logic, event risk; `final = quant percentile + 0.25 × score`), vetoes for five event types, top 10 on the base strategy's rebalance rows and hysteresis. Score files in `data/agent/us/` are stamped by `agent_review.py submit` and steer a session only if stamped before its 16:00 New York close; without one the strategy holds exactly what `overnight_recovery_top10` holds, so the two books' difference measures the agent. `AGENT_REVIEW_STRATEGIES` in trader.py keeps it US-only and out of `precompute_backtest.py` (it has no history before its first score file). Research behind the design (2026-09-30): picking 10 of the quant top 50 at random earns 13.4%/yr vs 26.2% for the top 10 (2006-2026), and an LLM cannot be backtested before its training cutoff, so the record starts 2026-09-29.
|
||
- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol`, `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom`, `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap` — these were mined on today's CSI 300 members; on PIT members they earn ~3-7% a year (2016-2026). All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
|
||
|
||
**Metrics** (`metrics.py`): Standalone functions for portfolio analytics (Sharpe, Sortino, Calmar, max drawdown, etc.). `summary()` prints a formatted report and returns a dict.
|
||
|
||
## Key Conventions
|
||
|
||
- Strategies output **weights**, not buy/sell signals. Weights are normalized so each row sums to 1 (or 0 during warm-up).
|
||
- Warm-up periods: each strategy zeros out its first N rows where indicators are undefined.
|
||
- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests), `data/web/` (watchlist, per-stock cache, backtest result cache) and `data/research/` (research panels, ~2 GB). Not checked into git (only `data/.gitkeep` is tracked). `data/sp500_history.json` is rebuilt from Wikipedia on first use; `data/csi300_history.json` is not rebuilt automatically (its 2008-2026 snapshots came from baostock), so copy it to a new machine or CN membership fails to load.
|
||
- Python 3.12+, managed with `uv`.
|
||
- Research code lives in `research/lab/` (PIT panels, next-open fills, preregistered studies with their verdict in the docstring). Pre-lab scripts removed in the 2026-10-09 clean-up are indexed in `research/ARCHIVE.md` with the commit to restore them from. `research/permanent_yearly.py` stays because bridge.py and precompute_backtest.py import `equity_with_cashflows`.
|
||
- Do not show matplotlib figures when running backtests; use `--no-plot`.
|
||
|
||
The `*_rb20` strategies are execution-only candidates with a 20% resize band; baseline strategies keep 10%. They run alongside their baselines only in `validation_2026_10`, starting with the first market session on/after 2026-10-02, with matching capital and fees. Existing forward histories are preserved and future daily snapshots record the engine version. These candidates were selected from previously examined historical data and have no independent forward performance yet.
|
||
|
||
Registered-strategy comparisons use the same ledger:
|
||
|
||
```bash
|
||
uv run python -m research.lab.evaluate --market us --registered overnight_recovery_top10 us_three_sleeve --start 2018-01-01 --end 2026-09-28 --output data/research/results/new_comparison
|
||
```
|
||
|
||
The comparison writes its fixed budget and acceptance rule before running and refuses to overwrite an existing study. Trial producers append versioned JSONL; legacy CSV trial counts remain a lower bound. `stepwise --objective maximin` selects only inside the design period. Previously inspected validation periods must not be described as independent OOS data.
|
||
|
||
## Server Deployment
|
||
|
||
Four `systemd --user` services (a fifth defined but not enabled) and one timer, defined in `deploy/systemd/`, run from `/home/gahow/projects/quant-trading`:
|
||
|
||
| Service | Script | Role |
|
||
|---|---|---|
|
||
| `quant-trading-monitor` | `deploy/start-monitor.sh` | `trader.py monitor`: morning/evening phases for US + CN, all live strategies |
|
||
| `quant-trading-api` | `deploy/start-web-api.sh` | Web API on `127.0.0.1:8787` |
|
||
| `quant-trading-vite` | `deploy/start-web-vite.sh` | Vite on `127.0.0.1:5174`, behind nginx on `:5443` |
|
||
| `quant-trading-overnight-paper` | `deploy/start-overnight-paper.sh` | `overnight_paper.py daemon`: overnight-only momentum on the Alpaca paper account (from 2026-10-12) |
|
||
| `quant-trading-ibkr-live` (not enabled) | `deploy/start-ibkr-live.sh` | `ibkr_trader.py daemon --live`: trend_rider_v10 in the $10k IBKR Pro live account. Needs a live IB Gateway on port 4001 (the paper Gateway on 4002 belongs to ../ibkr-quant, whose strategy owns that paper account), IBKR_ACCOUNT in .env, the Complex or Leveraged ETP permission (paper what-if 2026-10-13 rejected TQQQ/UPRO without it) and data/ibkr_live/ARMED |
|
||
| `quant-trading-options` (timer, 16:25 New York Mon-Fri) | `deploy/start-options-snapshot.sh` | `research.lab.fetch_options snapshot`: S&P 500 option chains (indicative quotes), OPRA daily bars, per-name IV/skew/spread/volume summary and a quote-quality sample into `data/options/us/` (~15 MB/day); `status` prints the indicative-vs-OPRA verdict after 20 sessions |
|
||
|
||
```bash
|
||
cp deploy/systemd/*.service ~/.config/systemd/user/
|
||
systemctl --user daemon-reload
|
||
systemctl --user enable --now quant-trading-monitor quant-trading-api quant-trading-vite
|
||
systemctl --user enable --now quant-trading-options.timer
|
||
# one-off, resumable: OPRA option-bar history since 2024-02 (rerun the same command if it stops)
|
||
systemd-run --user --unit=quant-trading-options-backfill --working-directory=$PWD --setenv=PYTHONPATH=. uv run python -m research.lab.fetch_options backfill
|
||
journalctl --user -u quant-trading-monitor -f
|
||
```
|
||
|
||
The monitor catches up on startup, so restarting it between phases is safe. Its hot-reload re-imports trader.py only, not `strategies/`: after changing strategy code, restart the monitor (a registry whose strategies fail to build is rejected and logged with "restart the monitor"). If it starts mid-day (after open, before close), it falls back to `auto` mode for the evening phase.
|
||
|
||
Compare live books after a while:
|
||
|
||
```bash
|
||
uv run python trader.py compare # US strategies
|
||
uv run python trader.py compare --market cn # CN strategies
|
||
```
|
||
|
||
### Alternative: cron
|
||
|
||
Run daily after market close. The `auto` command is idempotent — safe to re-run:
|
||
|
||
```bash
|
||
# US market: 5pm ET weekdays
|
||
0 17 * * 1-5 cd /home/gahow/projects/quant-trading && uv run python trader.py auto --market us --strategy recovery_mom_top10 >> data/cron.log 2>&1
|
||
|
||
# CN market: 4pm CST weekdays
|
||
0 16 * * 1-5 cd /home/gahow/projects/quant-trading && uv run python trader.py auto --market cn --strategy inverse_vol >> data/cron.log 2>&1
|
||
```
|