Compare commits

...

2 Commits

Author SHA1 Message Date
cc1366b2dc feat: robust-momentum strategies live; fix live universe, lots and UI backtests
Strategies
- BarStrategy protocol (close/open/high/low/volume bars; row d uses bars
  through d-1) and RobustMomentumStrategy: overnight_recovery_top10 (US),
  lowrisk_momentum_top15 and lowbeta_sharpe_mom_top20 (CN), top-N with
  rank hysteresis, 10-day rebalance, weights drift between rebalances.
- trader: strategy_fields/strategy_bars/bar_session_weights; one signal
  path (_signals_for_sessions) replaces generate_target_weights.

Live fixes
- CSI 300 constituents come from China Securities Index's file; the
  Wikipedia table was 122 names off the real index.
- S&P 500 change log moved to "Historical components of the S&P 500".
- data_manager: tickers missing from a partial Yahoo batch are retried
  alone before being marked unavailable (CEG, GEV, SNDK ... were dropped).
  update() takes fields=[...] and stores high/low/volume panels.
- Whole-lot orders go to the nearest lot within cash (CN books held ~62%
  cash); held names are resized only beyond RESIZE_BAND (10%), so books no
  longer pay a ticket fee to trade daily drift.

Web backtests
- backtest_runner.py replaces the duplicated bridge/precompute code: PIT
  membership in both markets (data/csi300_history.json), stock universe
  limited to index members (ETFs and benchmarks in the shared panel were
  pickable), and the book kept like the live trader (simulate_live_book:
  signal timing, drift, order floor, resize band, fees on orders placed).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 19:12:02 +08:00
0e378efa67 research: point-in-time factor research harness (research/lab)
Event-driven long-only simulator with next-open fills, CN tradability
(halts, limit-up/down, stamp duty), PIT S&P 500 / CSI 300 membership,
rebalance-phase averaging, factor IC scans, maximin stepwise composite
search across 5-year blocks, walk-forward LightGBM/ridge combiners,
overlays, and deflated Sharpe. Fetchers pull 20y US OHLCV from Yahoo and
CSI 300/500 ever-members (membership snapshot from baostock, bars from
Yahoo). Research data lives in data/research/ (ignored).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 19:11:59 +08:00
35 changed files with 18228 additions and 606 deletions

3
.gitignore vendored
View File

@@ -59,3 +59,6 @@ memory/
# Web UI state and precomputed backtests (web/src/server/paths.ts)
data/web/
data/precomputed/
# Research panels and results — fetched/rebuilt by research/lab
data/research/

View File

@@ -9,7 +9,7 @@ quant-trading: a quantitative backtesting framework and live forward-testing tra
The repository has two halves that share one data directory and one `.env`:
- **Python engine** (repo root, `uv`): strategies, backtests, `trader.py`, research scripts.
- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`.
- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`. Both backtest paths run through `backtest_runner.py`.
`bridge.py`'s JSON output and `web/src/shared/strategyTypes.ts` are one contract: change both in the same commit.
@@ -40,6 +40,13 @@ uv run python trader.py simulate --integer-shares --fixed-fee 2.0 ... # Integ
# Precompute the strategy page's backtests (writes data/precomputed/)
uv run python precompute_backtest.py
# Factor research harness (PIT universe, next-open fills, phase averaging; data in data/research/)
uv run python -m research.lab.fetch_us # 20y OHLCV for every S&P 500 ever-member
uv run python -m research.lab.fetch_cn_yahoo # CSI 300/500 ever-members (membership from baostock snapshot)
PYTHONPATH=. uv run python -m research.lab.factor_scan --market us # or cn_hs300
PYTHONPATH=. uv run python -m research.lab.stepwise --market us --objective maximin
PYTHONPATH=. uv run python -m research.lab.evaluate --market us # candidates vs champions
# Setup
uv sync # Install/sync dependencies
cp .env.example .env # Then fill IPADS_API_KEY and QUANT_MARKET_PROXY
@@ -55,15 +62,17 @@ uv run python -m unittest discover tests
## Architecture
**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia, CSI 300 from Wikipedia) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia; CSI 300 from China Securities Index's published constituent file, falling back to the latest `data/csi300_history.json` snapshot, then Wikipedia, whose table lagged the index by ~60 names in 2026) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_open.csv` (open). First run downloads 10 years of history; subsequent runs append only new trading days (fast bulk fetch for gaps ≤7 days). New tickers from index rebalances are backfilled automatically. Tickers with >50% missing data are dropped. Pass `with_open=True` to also maintain open price files.
**Point-in-time membership** (`universe_history.py`): S&P 500 intervals rebuilt from Wikipedia's "Historical components of the S&P 500" change log into `data/sp500_history.json`; CSI 300 monthly snapshots in `data/csi300_history.json` (seeded from baostock 2008–2026, appended whenever the current list changes). `mask_prices_pit(prices, market)` masks non-member dates.
**Backtest engine** (`main.py`): Orchestrates data loading, strategy execution, and visualization. The `backtest()` function is vectorized — it takes a strategy and price DataFrame, applies transaction costs (proportional + optional fixed per-trade fee) via turnover, and returns an equity curve. Supports two execution modes: `close` (classic) and `open-close` (signal on open prices, execute at close).
**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_{open,high,low,volume}.csv`. First run downloads 10 years of history; every run re-downloads the last 30 days. New tickers from index rebalances are backfilled automatically; tickers missing from a partial batch are retried alone before being marked unavailable (retried weekly). `update(market, tickers, with_open=True)` returns (close, open); `update(..., fields=[...])` returns `{field: panel}`.
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, plus 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`).
**Backtest engine** (`main.py`): Orchestrates data loading, strategy execution, and visualization. `backtest()` has two modes. Close mode (no `open_data`, used by `main.py` and old research scripts) is vectorized and trades at the close the signal saw. Live mode (`open_data` given, used by the web UI through `backtest_runner.py`) keeps the book like `trader.py`: same signal timing (`strategy_positions`), holdings that drift, the $50/0.1% order floor, `trader.RESIZE_BAND`, fees on the orders placed (`simulate_live_book`).
**Strategy protocol** (`strategies/base.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy is responsible for applying its own 1-day lag via `.shift(1)` to avoid lookahead bias — the backtest engine does not shift.
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Orders: whole A-share lots are allocated nearest-lot within cash (`_allocate_lots`); a held name is resized only when ≥ `RESIZE_BAND` (10%) off target. Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`), and the robust-momentum bar strategies `overnight_recovery_top10`, `lowrisk_momentum_top15`, `lowbeta_sharpe_mom_top20`.
**Strategy protocol** (`strategies/base.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy is responsible for applying its own 1-day lag via `.shift(1)` to avoid lookahead bias — the backtest engine does not shift. A `BarStrategy` declares `fields` (close/open/high/low/volume) and receives `{field: DataFrame}`; live it uses completed bars only (row d from bars through d-1, traded at the close of d), while close-price strategies read the session's open as their latest price.
**Strategies**:
- `buy_and_hold.py` — Equal-weight static allocation
@@ -76,7 +85,8 @@ uv run python -m unittest discover tests
- `momentum_quality.py` — Momentum + return consistency + low drawdown
- `adaptive_momentum.py` — Momentum weighted by inverse volatility
- `recovery_momentum.py` — Recovery (price/63d low) + 12-1mo momentum composite. Best US performer.
- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol` (50.7% CAGR monthly), `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom` (26.1% CAGR monthly), `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap`. All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
- `robust_momentum.py` — Bar strategies from the 2026-09 PIT research (report: https://claude.ai/artifact/Mo9nrzs5SWj5FfyshERyWS): US `overnight_recovery` (overnight-return momentum + distance from 126d low), CN `lowrisk_momentum` (amplitude-cut momentum + low beta + price-volume corr + momentum/vol) and close-only `lowbeta_sharpe_mom`. Top-N with rank hysteresis, rebalance every 10 days, weights drift between rebalances.
- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol`, `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom`, `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap` — these were mined on today's CSI 300 members; on PIT members they earn ~3-7% a year (2016-2026). All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
**Metrics** (`metrics.py`): Standalone functions for portfolio analytics (Sharpe, Sortino, Calmar, max drawdown, etc.). `summary()` prints a formatted report and returns a dict.
@@ -84,7 +94,7 @@ uv run python -m unittest discover tests
- Strategies output **weights**, not buy/sell signals. Weights are normalized so each row sums to 1 (or 0 during warm-up).
- Warm-up periods: each strategy zeros out its first N rows where indicators are undefined.
- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests) and `data/web/` (watchlist, per-stock cache, backtest result cache). Not checked into git (only `data/.gitkeep` and `data/sp500_history.json` are tracked).
- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests), `data/web/` (watchlist, per-stock cache, backtest result cache) and `data/research/` (research panels, ~2 GB). Not checked into git (only `data/.gitkeep`, `data/sp500_history.json` and `data/csi300_history.json` are tracked).
- Python 3.12+, managed with `uv`.
- Do not show matplotlib figures when running backtests; use `--no-plot`.

197
backtest_runner.py Normal file
View File

@@ -0,0 +1,197 @@
"""
Market-level backtests behind the web UI: ``bridge.py backtest`` (on demand)
and ``precompute_backtest.py`` (the strategy page) share this one data path.
* Universe: point-in-time index members in both markets (S&P 500 history from
Wikipedia, CSI 300 history in data/csi300_history.json); prices outside a
stock's membership are masked, so strategies cannot pick it then.
* Book: kept like the live trader keeps it — its signal timing
(``main.strategy_positions`` live mode), drifting holdings, the same order
size floor and resize band (``main.simulate_live_book``).
* Bars: panels a BarStrategy needs (open/high/low/volume) are loaded too.
"""
from __future__ import annotations
import sys
import pandas as pd
import data_manager
import universe_history as uh
from main import simulate_live_book, strategy_positions
from strategies.base import BarStrategy
from trader import (
ETF_STRATEGY_UNIVERSES,
LIVE_EXCLUDED_STRATEGIES,
RESIZE_BAND,
STRATEGY_REGISTRY,
convert_quote_currency,
filter_tradable_tickers,
price_request,
strategy_extra_tickers,
strategy_fields,
)
from universe import UNIVERSES
# Rebalances below this size are not charged the per-trade fee in backtests,
# matching the live trader's minimum order size.
MIN_TRADE_VALUE = 50.0
def _log(msg: str) -> None:
print(msg, file=sys.stderr)
def trade_events(diff: pd.DataFrame, threshold: float = 0.01) -> list[dict]:
"""Trade events from per-ticker traded weights (rows = days)."""
trades = []
for date in diff.index:
row = diff.loc[date]
significant = row[row.abs() > threshold]
if significant.empty:
continue
buys = significant[significant > 0]
sells = significant[significant < 0]
if len(buys) > 0 or len(sells) > 0:
trades.append({
"date": date.strftime("%Y-%m-%d"),
"buys": len(buys),
"sells": len(sells),
"buyTickers": list(buys.nlargest(3).index),
"sellTickers": list(sells.nsmallest(3).index),
})
return trades
def benchmark_curve(prices: pd.Series, dates: pd.DatetimeIndex, capital: float) -> list[dict]:
"""Benchmark value on ``dates``, rebased to ``capital`` on the first date it has a price."""
series = prices.reindex(dates).ffill().dropna()
if series.empty:
return []
values = series / series.iloc[0] * capital
return [{"date": d.strftime("%Y-%m-%d"), "value": round(float(v), 2)} for d, v in values.items()]
def _slice_years(panels: dict[str, pd.DataFrame], years: int | None) -> dict[str, pd.DataFrame]:
if not years:
return panels
close = panels["close"]
cutoff = close.index[-1] - pd.DateOffset(years=years)
return {f: p[p.index >= cutoff] for f, p in panels.items()}
def load_stock_panels(market: str, strategy_names: list[str], years: int | None) -> dict:
"""PIT-masked panels for the market's stock strategies."""
universe = UNIVERSES[market]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
since = None
if years:
since = (pd.Timestamp.today() - pd.DateOffset(years=years + 1)).strftime("%Y-%m-%d")
ever = uh.pit_tickers_ever(market, since=since) or []
extras = {t for n in strategy_names for t in strategy_extra_tickers(n)}
request = sorted(set(tickers) | set(ever) | extras | {benchmark})
fields = sorted({"close", "open"} | {f for n in strategy_names for f in strategy_fields(n)})
_log(f"PIT universe: {len(request)} tickers ({len(ever)} ever index members); fields {fields}")
panels = data_manager.update(market, request, fields=fields)
panels = _slice_years(panels, years)
member = None
if ever:
member = uh.mask_prices_pit(panels["close"], market).notna() | panels["close"].isna()
# Only tickers that were index members: the shared panel also holds
# benchmarks and the ETFs of mixed strategies (TQQQ, SOXL, ...), which
# the membership mask passes through unmasked.
members_ever = set(ever)
stocks = [t for t in panels["close"].columns
if t in members_ever and t not in extras
and panels["close"][t].where(member[t]).notna().any()]
else:
stocks = filter_tradable_tickers(panels["close"], tickers)
return {"panels": panels, "member": member, "stocks": stocks, "benchmark": benchmark,
"benchmark_label": universe["benchmark_label"]}
def _run(strategy, panels: dict[str, pd.DataFrame], cols: list[str], capital: float,
tx_cost: float, fixed_fee: float, fee_base: float, fee_per_share: float,
member: pd.DataFrame | None = None, min_trade_value: float = MIN_TRADE_VALUE):
"""Close-price strategies see prices masked to index membership (they
cannot pick a non-member); a BarStrategy sees full histories plus the
membership as ``bars["eligible"]``. The book is valued on unmasked closes."""
close, opens = panels["close"][cols], panels["open"][cols]
if isinstance(strategy, BarStrategy):
bars = {f: panels[f][cols] for f in strategy.fields}
if member is not None:
bars["eligible"] = member.reindex(columns=cols).fillna(True)
held = strategy_positions(strategy, close, open_data=opens, bars=bars)
else:
m = member.reindex(columns=cols).fillna(True) if member is not None else None
held = strategy_positions(strategy, close.where(m) if m is not None else close,
open_data=opens.where(m) if m is not None else opens)
equity, traded = simulate_live_book(held, close, capital, tx_cost, fixed_fee, fee_base,
fee_per_share, min_trade_value, RESIZE_BAND)
return equity, trade_events(traded)
def run_market(market: str, strategy_names: list[str], years: int | None, capital: float,
tx_cost: float, fixed_fee: float, fee_base: float = 0.0,
fee_per_share: float = 0.0) -> dict | None:
"""Backtest every strategy once; returns raw equity series and metadata."""
stock = [n for n in strategy_names if n.removeprefix("sim_") not in ETF_STRATEGY_UNIVERSES]
etf = [n for n in strategy_names if n.removeprefix("sim_") in ETF_STRATEGY_UNIVERSES]
fees = dict(tx_cost=tx_cost, fixed_fee=fixed_fee, fee_base=fee_base, fee_per_share=fee_per_share)
runs: list[dict] = []
out = {"market": market, "benchmark_label": "SPY", "benchmark_curve": [],
"start_date": None, "end_date": None, "tx_cost": tx_cost, "fixed_fee": fixed_fee}
if stock:
data = load_stock_panels(market, stock, years)
panels, stocks = data["panels"], data["stocks"]
close = panels["close"]
_log(f"Universe: {len(stocks)} stocks, {close.index[0].date()} to {close.index[-1].date()}")
for name in stock:
_log(f" Running: {name}")
try:
strategy = STRATEGY_REGISTRY[name]()
cols = stocks + [t for t in strategy_extra_tickers(name)
if t in close.columns and t not in stocks]
# Strategies the live book cannot hold (e.g. equal weight over
# ~500 names) are shown without the live order-size floor.
floor = 0.0 if name in LIVE_EXCLUDED_STRATEGIES else MIN_TRADE_VALUE
equity, trades = _run(strategy, panels, cols, capital, **fees,
member=data["member"], min_trade_value=floor)
runs.append({"name": name, "equity": equity, "trades": trades})
_log(f" OK: {name} (final: {equity.iloc[-1]:,.0f})")
except Exception as e: # noqa: BLE001 — one broken strategy must not sink the page
_log(f" FAILED: {name}: {e}")
out.update(benchmark_label=data["benchmark_label"],
benchmark_curve=benchmark_curve(close[data["benchmark"]], close.index, capital),
start_date=close.index[0].strftime("%Y-%m-%d"),
end_date=close.index[-1].strftime("%Y-%m-%d"))
for name in etf:
_log(f" Running ETF strategy: {name}")
try:
etf_tickers = ETF_STRATEGY_UNIVERSES[name.removeprefix("sim_")]
close, opens = data_manager.update(
"etfs", price_request(market, sorted(set(etf_tickers + ["SPY"]))), with_open=True)
panels = _slice_years({"close": convert_quote_currency(close),
"open": convert_quote_currency(opens)}, years)
strategy = STRATEGY_REGISTRY[name]()
equity, trades = _run(strategy, panels, filter_tradable_tickers(panels["close"], etf_tickers),
capital, **fees)
runs.append({"name": name, "equity": equity, "trades": trades})
_log(f" OK: {name} (final: {equity.iloc[-1]:,.0f})")
if not stock and not out["benchmark_curve"] and "SPY" in panels["close"].columns:
spy = panels["close"]["SPY"]
out.update(benchmark_label="SPY (Benchmark)",
benchmark_curve=benchmark_curve(spy, spy.index, capital),
start_date=spy.index[0].strftime("%Y-%m-%d"),
end_date=spy.index[-1].strftime("%Y-%m-%d"))
except Exception as e: # noqa: BLE001
_log(f" FAILED: {name}: {e}")
if not runs:
return None
out["strategy_runs"] = runs
return out

229
bridge.py
View File

@@ -20,29 +20,19 @@ import pandas as pd
import data_manager
import metrics
import universe_history as uh
from main import backtest
from backtest_runner import benchmark_curve, run_market
from research.permanent_yearly import equity_with_cashflows
from trader import (
strategy_extra_tickers,
strategy_supports_market,
ETF_STRATEGY_UNIVERSES,
LIVE_EXCLUDED_STRATEGIES,
MARKET_FEE_TIERED,
STRATEGY_REGISTRY,
convert_quote_currency,
price_request,
strategy_supports_market,
strategy_data_market,
filter_tradable_tickers,
strategy_supports_market,
)
MARKET_CURRENCY = {"us": "USD", "cn": "CNY"}
# Rebalances below this size are not charged the per-trade fee in backtests,
# matching the live trader's minimum order size.
MIN_TRADE_VALUE = 50.0
from universe import UNIVERSES
# ---------------------------------------------------------------------------
@@ -440,6 +430,25 @@ STRATEGY_META = {
"params": {},
"markets": ["us"],
},
# --- Robust momentum (2026-09 PIT research): US pick, CN picks ---
"overnight_recovery_top10": {
"label": "Robust: Overnight Mom + Recovery Top 10",
"category": "factor_combo",
"params": {},
"markets": ["us", "cn"],
},
"lowrisk_momentum_top15": {
"label": "Robust: Low-Risk Momentum Top 15",
"category": "factor_combo",
"params": {},
"markets": ["us", "cn"],
},
"lowbeta_sharpe_mom_top20": {
"label": "Robust: Low-Beta + Sharpe Mom Top 20",
"category": "factor_combo",
"params": {},
"markets": ["us", "cn"],
},
}
@@ -515,28 +524,6 @@ def compute_drawdown_periods(equity: pd.Series, threshold: float = -0.05) -> lis
return periods
def compute_rebalance_trades(weights: pd.DataFrame, threshold: float = 0.01) -> list[dict]:
"""Extract significant trade events from weight changes."""
diff = weights.diff().fillna(0.0)
trades = []
for date in diff.index:
row = diff.loc[date]
significant = row[row.abs() > threshold]
if significant.empty:
continue
buys = significant[significant > 0]
sells = significant[significant < 0]
if len(buys) > 0 or len(sells) > 0:
trades.append({
"date": date.strftime("%Y-%m-%d"),
"buys": len(buys),
"sells": len(sells),
"buyTickers": list(buys.nlargest(3).index),
"sellTickers": list(sells.nsmallest(3).index),
})
return trades
def compute_rebalance_win_rate(equity: pd.Series, trade_events: list[dict]) -> float | None:
"""Fraction of inter-rebalance periods with positive cumulative return.
@@ -600,15 +587,6 @@ def compute_periodic_returns(equity: pd.Series) -> tuple[dict, dict]:
return monthly, yearly
def benchmark_curve(prices: pd.Series, dates: pd.DatetimeIndex, capital: float) -> list[dict]:
"""Benchmark value on ``dates``, rebased to ``capital`` on the first date it has a price."""
series = prices.reindex(dates).ffill().dropna()
if series.empty:
return []
values = series / series.iloc[0] * capital
return [{"date": d.strftime("%Y-%m-%d"), "value": round(float(v), 2)} for d, v in values.items()]
def build_injection_schedule(equity: pd.Series, schedule: dict) -> tuple[pd.Series, list[dict]]:
"""Build a contributions Series from injection schedule config.
@@ -725,159 +703,18 @@ def cmd_backtest(args: argparse.Namespace) -> None:
_log(f"Running backtest: market={market}, years={years}, capital={capital}")
_log(f"Strategies: {strategy_names}")
# Determine data needs
# Group strategies by their data source
stock_strategies = []
etf_strategies = []
for name in strategy_names:
if name not in STRATEGY_REGISTRY:
_log(f"WARNING: Unknown strategy '{name}', skipping")
continue
base_name = name.removeprefix("sim_")
if base_name in ETF_STRATEGY_UNIVERSES:
etf_strategies.append(name)
else:
stock_strategies.append(name)
results = []
# Process stock strategies
if stock_strategies:
universe = UNIVERSES[market]
with _redirect_stdout_to_stderr():
tickers = universe["fetch"]()
benchmark_ticker = universe["benchmark"]
benchmark_label = universe["benchmark_label"]
# PIT universe: for US market, include all tickers that were EVER
# S&P 500 members so that removed stocks are present in price data.
pit_intervals = None
if market == "us":
with _redirect_stdout_to_stderr():
pit_intervals = uh.load_sp500_history()
historical_tickers = uh.all_tickers_ever(pit_intervals)
all_tickers = sorted(set(tickers + historical_tickers + [benchmark_ticker])
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
_log(f"PIT universe: {len(all_tickers)} tickers (current + historical members)")
else:
all_tickers = sorted(set(tickers + [benchmark_ticker])
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
_log(f"WARNING: {market.upper()} market lacks PIT membership history; "
"backtest uses current index constituents (survivorship bias possible)")
_log(f"Fetching price data for {len(all_tickers)} tickers...")
with _redirect_stdout_to_stderr():
data = data_manager.update(market, all_tickers, with_open=False)
if isinstance(data, tuple):
data = data[0]
if years:
cutoff = data.index[-1] - pd.DateOffset(years=years)
data = data[data.index >= cutoff]
# Apply PIT mask: set prices to NaN for dates when a stock was NOT
# an index member. Strategies naturally skip NaN entries.
if pit_intervals is not None:
_log("Applying PIT membership mask (survivorship-bias fix)...")
data = uh.mask_prices(data, pit_intervals)
tickers = filter_tradable_tickers(data, tickers)
# For PIT mode, expand tickers to all columns with valid data
# (historical members that have been masked are NaN when not members,
# but have valid data during their membership period).
if pit_intervals is not None:
all_data_tickers = [
t for t in data.columns
if t != benchmark_ticker and data[t].notna().any()
]
tickers = all_data_tickers
_log(f"Universe: {len(tickers)} tradable stocks, period: {data.index[0].date()} to {data.index[-1].date()}")
for name in stock_strategies:
_log(f" Running: {name}")
# Registry defaults are the single source of strategy parameters,
# shared with the live trader.
strategy = STRATEGY_REGISTRY[name]()
# Mixed stock+ETF strategies also read their fixed ETFs.
cols = tickers + [t for t in strategy_extra_tickers(name)
if t in data.columns and t not in tickers]
equity = backtest(
strategy, data[cols],
initial_capital=capital,
transaction_cost=tx_cost,
fixed_fee=fixed_fee,
min_trade_value=MIN_TRADE_VALUE,
**MARKET_FEE_TIERED.get(market, {}),
)
# Get weight signals for trade extraction
weights = strategy.generate_signals(data[cols])
weights = weights.reindex(data[cols].index).fillna(0.0)
trade_events = compute_rebalance_trades(weights)
result_entry = _build_strategy_result(
name, equity, trade_events, injection_schedule, capital
)
results.append(result_entry)
# Benchmark over exactly the strategies' dates
bench_curve = benchmark_curve(data[benchmark_ticker], data.index, capital)
else:
benchmark_label = "SPY"
bench_curve = []
# Process ETF strategies
if etf_strategies:
for name in etf_strategies:
_log(f" Running ETF strategy: {name}")
base_name = name.removeprefix("sim_")
etf_tickers = ETF_STRATEGY_UNIVERSES[base_name]
all_etf = sorted(set(etf_tickers + ["SPY"]))
with _redirect_stdout_to_stderr():
etf_data = data_manager.update("etfs", price_request(market, all_etf), with_open=False)
if isinstance(etf_data, tuple):
etf_data = etf_data[0]
etf_data = convert_quote_currency(etf_data)
if years:
cutoff = etf_data.index[-1] - pd.DateOffset(years=years)
etf_data = etf_data[etf_data.index >= cutoff]
strategy = STRATEGY_REGISTRY[name]()
tradable = filter_tradable_tickers(etf_data, etf_tickers)
equity = backtest(
strategy, etf_data[tradable],
initial_capital=capital,
transaction_cost=tx_cost,
fixed_fee=fixed_fee,
min_trade_value=MIN_TRADE_VALUE,
**MARKET_FEE_TIERED.get(market, {}),
)
weights = strategy.generate_signals(etf_data[tradable])
weights = weights.reindex(etf_data[tradable].index).fillna(0.0)
trade_events = compute_rebalance_trades(weights)
result_entry = _build_strategy_result(
name, equity, trade_events, injection_schedule, capital
)
results.append(result_entry)
# If no stock strategies were run, use SPY as benchmark from ETF data
if not stock_strategies and len(etf_strategies) > 0:
benchmark_label = "SPY (Benchmark)"
with _redirect_stdout_to_stderr():
spy_data = data_manager.update("etfs", ["SPY"], with_open=False)
if isinstance(spy_data, tuple):
spy_data = spy_data[0]
if years:
cutoff = spy_data.index[-1] - pd.DateOffset(years=years)
spy_data = spy_data[spy_data.index >= cutoff]
if "SPY" in spy_data.columns:
bench_curve = benchmark_curve(spy_data["SPY"], spy_data.index, capital)
known = [n for n in strategy_names if n in STRATEGY_REGISTRY]
for name in set(strategy_names) - set(known):
_log(f"WARNING: Unknown strategy '{name}', skipping")
with _redirect_stdout_to_stderr():
raw = run_market(market, known, years, capital, tx_cost, fixed_fee,
**MARKET_FEE_TIERED.get(market, {}))
results = [
_build_strategy_result(run["name"], run["equity"], run["trades"], injection_schedule, capital)
for run in (raw["strategy_runs"] if raw else [])
]
benchmark_label = raw["benchmark_label"] if raw else "SPY"
bench_curve = raw["benchmark_curve"] if raw else []
output = {
"meta": {

14803
data/csi300_history.json Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because one or more lines are too long

View File

@@ -265,94 +265,148 @@ def _fill_open(close: pd.DataFrame, open_raw: pd.DataFrame | None) -> pd.DataFra
return aligned.fillna(close)
def update(market: str, tickers: list[str],
with_open: bool = False) -> pd.DataFrame | tuple[pd.DataFrame, pd.DataFrame]:
YAHOO_FIELDS = {"close": "Close", "open": "Open", "high": "High", "low": "Low", "volume": "Volume"}
def _backfill(tickers: list[str], start: str, yahoo_fields: list[str],
index: pd.DatetimeIndex) -> tuple[dict[str, pd.DataFrame], list[str]]:
"""Download new tickers over ``index``; return (panels, tickers still missing).
yfinance reports a ticker it failed to fetch (rate limit, proxy error) the
same way as a ticker it has no data for, so tickers missing from the first
batch are downloaded once more on their own before being given up on.
"""
Refresh the stored panel for ``tickers`` and return it forward-filled.
def fetch(batch: list[str]) -> dict[str, pd.DataFrame]:
raw = _download(batch, start=start, fields=yahoo_fields)
close = raw["Close"].dropna(axis=1, how="all").reindex(index=index).dropna(axis=1, how="all")
return {f: raw.get(f, pd.DataFrame()).reindex(index=index, columns=close.columns)
for f in yahoo_fields}
got = fetch(tickers)
missing = [t for t in tickers if t not in got["Close"].columns]
if missing and len(missing) < len(tickers):
retry = fetch(missing)
if not retry["Close"].empty:
got = {f: pd.concat([got[f], retry[f]], axis=1) for f in yahoo_fields}
missing = [t for t in missing if t not in retry["Close"].columns]
return got, missing
def update(market: str, tickers: list[str], with_open: bool = False,
fields: list[str] | None = None):
"""
Refresh the stored panels for ``tickers`` and return them forward-filled.
- First run: downloads 10 years of history.
- Every run: re-downloads the last ``REFRESH_DAYS`` calendar days for the
requested tickers already in the panel and overwrites them.
- New tickers are backfilled over the panel's full date range; tickers
Yahoo has no data for are remembered and retried weekly.
- A field requested for the first time (e.g. ``high``) is downloaded for
every ticker in the stored close panel.
Returns the close panel, or ``(close, open)`` when ``with_open`` is True.
Returns the close panel, ``(close, open)`` when ``with_open`` is True, or
``{field: panel}`` when ``fields`` is given. Open/high/low fall back to the
close where a bar is missing; volume is returned raw (NaN = no print).
"""
fields = ["Close", "Open"] if with_open else ["Close"]
names = list(fields) if fields is not None else (["close", "open"] if with_open else ["close"])
if "close" not in names:
names = ["close"] + names
unknown = [f for f in names if f not in YAHOO_FIELDS]
if unknown:
raise ValueError(f"Unsupported market data field(s): {unknown}")
yahoo = [YAHOO_FIELDS[f] for f in names]
with _panel_lock(market):
close_raw = load_raw(market, "close")
open_raw = load_raw(market, "open") if with_open else None
raw = {f: load_raw(market, f) for f in names}
unavailable = _load_unavailable(market)
today = datetime.now()
if close_raw is None:
if raw["close"] is None:
start = (today - timedelta(days=365 * 10)).strftime("%Y-%m-%d")
downloaded = _download(tickers, start=start, fields=fields)
close_raw = _merge_refresh(None, downloaded["Close"], None)
if with_open:
open_raw = _merge_refresh(None, downloaded.get("Open", pd.DataFrame()), None)
downloaded = _download(tickers, start=start, fields=yahoo)
for f, y in zip(names, yahoo):
raw[f] = _merge_refresh(None, downloaded.get(y, pd.DataFrame()), None)
else:
if with_open and open_raw is None:
print("--- Downloading Open prices for the full panel (first time) ---")
start = close_raw.index[0].strftime("%Y-%m-%d")
opens = _download(list(close_raw.columns), start=start, fields=["Open"])
open_raw = _merge_refresh(None, opens["Open"], None)
close_raw = raw["close"]
first = [(f, y) for f, y in zip(names, yahoo) if raw[f] is None]
if first:
print(f"--- Downloading {[y for _, y in first]} for the full panel (first time) ---")
got = _download(list(close_raw.columns), start=close_raw.index[0].strftime("%Y-%m-%d"),
fields=[y for _, y in first])
for f, y in first:
raw[f] = _merge_refresh(None, got[y], None)
known = [t for t in tickers if t in close_raw.columns]
if known:
window_start = close_raw.index[-1] - pd.Timedelta(days=REFRESH_DAYS)
fresh = _download(known, start=window_start.strftime("%Y-%m-%d"), fields=fields)
fresh = _download(known, start=window_start.strftime("%Y-%m-%d"), fields=yahoo)
if not fresh["Close"].empty:
close_raw = _merge_refresh(close_raw, fresh["Close"], window_start)
if with_open:
open_raw = _merge_refresh(open_raw, fresh.get("Open", pd.DataFrame()), window_start)
for f, y in zip(names, yahoo):
raw[f] = _merge_refresh(raw[f], fresh.get(y, pd.DataFrame()), window_start)
else:
print("--- Refresh download returned no data; keeping stored panel ---")
cutoff = (today - timedelta(days=UNAVAILABLE_RETRY_DAYS)).strftime("%Y-%m-%d")
new_tickers = [t for t in tickers
if t not in close_raw.columns and unavailable.get(t, "") < cutoff]
if t not in raw["close"].columns and unavailable.get(t, "") < cutoff]
if new_tickers:
start = close_raw.index[0].strftime("%Y-%m-%d")
backfill = _download(new_tickers, start=start, fields=fields)
downloaded_any = not backfill["Close"].dropna(how="all").empty
got = backfill["Close"].dropna(axis=1, how="all")
got = got.reindex(index=close_raw.index)
got = got.dropna(axis=1, how="all")
for t in new_tickers:
if t in got.columns:
unavailable.pop(t, None)
elif downloaded_any:
# Only a ticker missing from an otherwise successful
# download is unavailable; a failed request is retried.
got, missing = _backfill(new_tickers, raw["close"].index[0].strftime("%Y-%m-%d"),
yahoo, raw["close"].index)
added = list(got["Close"].columns)
for t in added:
unavailable.pop(t, None)
# A batch that returned nothing at all is a failed request, not
# evidence the tickers have no data: retry those next run.
if added:
for t in missing:
unavailable[t] = today.strftime("%Y-%m-%d")
if not got.empty:
close_raw = pd.concat([close_raw, got], axis=1)
if with_open and open_raw is not None:
opens = backfill.get("Open", pd.DataFrame())
opens = opens.reindex(index=open_raw.index, columns=got.columns)
open_raw = pd.concat([open_raw, opens], axis=1)
print(f"--- Backfilled {got.shape[1]} new tickers ---")
missing = len(new_tickers) - got.shape[1]
if added:
for f, y in zip(names, yahoo):
panel = got[y].reindex(index=raw[f].index)
raw[f] = pd.concat([raw[f], panel], axis=1)
print(f"--- Backfilled {len(added)} new tickers ---")
if missing:
print(f"--- {missing} tickers have no Yahoo data; retry after "
print(f"--- {len(missing)} tickers have no Yahoo data; retry after "
f"{UNAVAILABLE_RETRY_DAYS} days ---")
close_raw = close_raw.loc[:, ~close_raw.columns.duplicated()]
_write_atomic(close_raw, _data_path(market, "close"))
print(f"--- Saved {close_raw.shape[0]} days x {close_raw.shape[1]} tickers to "
for f in names:
raw[f] = raw[f].loc[:, ~raw[f].columns.duplicated()]
_write_atomic(raw[f], _data_path(market, f))
print(f"--- Saved {raw['close'].shape[0]} days x {raw['close'].shape[1]} tickers to "
f"{_data_path(market, 'close')} ---")
if with_open and open_raw is not None:
open_raw = open_raw.loc[:, ~open_raw.columns.duplicated()]
_write_atomic(open_raw, _data_path(market, "open"))
_save_unavailable(market, unavailable)
close = close_raw.ffill()
close = raw["close"].ffill()
out = {"close": close}
for f in names:
if f in ("open", "high", "low"):
out[f] = _fill_open(close, raw[f])
elif f == "volume":
out[f] = raw[f].reindex(index=close.index, columns=close.columns)
if fields is not None:
return out
if with_open:
return close, _fill_open(close, open_raw)
return close, out["open"]
return close
def load_fields(market: str, fields: list[str]) -> dict[str, pd.DataFrame]:
"""Stored panels without refreshing, aligned and filled the way ``update`` returns them."""
close = load(market, "close")
out = {"close": close}
for f in fields:
if f == "close":
continue
stored = load_raw(market, f)
if f == "volume":
out[f] = (stored.reindex(index=close.index, columns=close.columns)
if stored is not None else pd.DataFrame(np.nan, index=close.index, columns=close.columns))
else:
out[f] = _fill_open(close, stored)
return out
def rebuild(market: str, with_open: bool = True, years: int = 10) -> None:
"""Re-download the full history of every ticker in a stored panel.

130
main.py
View File

@@ -9,6 +9,7 @@ import factor_attribution
import metrics
import universe_history as uh
from strategies.adaptive_momentum import AdaptiveMomentumStrategy
from strategies.base import BarStrategy
from strategies.buy_and_hold import BuyAndHoldStrategy
from strategies.dual_momentum import DualMomentumStrategy
from strategies.inverse_vol import InverseVolatilityStrategy
@@ -25,6 +26,99 @@ from universe import UNIVERSES
# Backtest engine
# ---------------------------------------------------------------------------
def strategy_positions(strategy, data: pd.DataFrame, open_data: pd.DataFrame | None = None,
bars: dict[str, pd.DataFrame] | None = None) -> pd.DataFrame:
"""Weights actually held over (close t-1, close t] for each row t.
Close mode (``open_data`` is None): row t's weights use closes through t-1
and trade at the close of t-1 — the same bar the signal saw.
Live mode (``open_data`` given), matching trader.py:
* close-price strategies read the session's open as their latest price
and trade at that session's close;
* a BarStrategy reads completed bars through t-1 and trades at the close
of t.
"""
if isinstance(strategy, BarStrategy):
weights = strategy.generate_signals(bars if bars is not None else {"close": data})
weights = weights.reindex(data.index).fillna(0.0)
return weights.shift(1).fillna(0.0) if open_data is not None else weights
if open_data is not None:
# The strategy's shift(1) on opens gives weights[t] = f(open_{t-1});
# undo it so the row-t decision uses open_t, then hold it from close t.
weights = strategy.generate_signals(open_data).shift(-1).fillna(0.0)
return weights.reindex(data.index).fillna(0.0).shift(1).fillna(0.0)
return strategy.generate_signals(data).reindex(data.index).fillna(0.0)
def simulate_live_book(held: pd.DataFrame, data: pd.DataFrame, initial_capital: float,
transaction_cost: float = 0.001, fixed_fee: float = 0.0,
fee_base: float = 0.0, fee_per_share: float = 0.0,
min_trade_value: float = 0.0,
resize_band: float = 0.0) -> tuple[pd.Series, pd.DataFrame]:
"""Day-by-day book kept the way trader.py keeps it.
``held`` row t holds the target weights for (close t-1, close t]. Holdings
drift with prices; at each close the book trades toward the next row's
targets: entries and exits always, a held name only once it is at least
``resize_band`` (relative to its target) away. Orders smaller than
``max(min_trade_value, 0.1% of equity)`` are skipped unless they close a
position. Costs are charged on the orders placed.
Returns (equity, traded weight per ticker on the row whose holdings changed).
"""
targets = held.reindex(index=data.index, columns=data.columns).fillna(0.0).to_numpy()
prices = data.ffill().to_numpy(dtype=float)
rets = np.nan_to_num(data.ffill().pct_change(fill_method=None).to_numpy(dtype=float), nan=0.0)
per_share = np.array([0.0 if str(c).endswith(".HK") else fee_per_share for c in data.columns])
n, m = targets.shape
w = np.zeros(m)
value = float(initial_capital)
equity = np.empty(n)
traded = np.zeros((n, m))
for t in range(n):
if value <= 0:
equity[t:] = 0.0
break
gross = w * (1.0 + rets[t])
growth = gross.sum() + (1.0 - w.sum()) # idle cash earns nothing
value *= growth
w = gross / growth if growth > 0 else np.zeros(m)
if t + 1 < n:
tgt = targets[t + 1]
new = w.copy()
entry_exit = (tgt <= 0) | (w <= 0)
resize = ~entry_exit & (np.abs(tgt - w) >= resize_band * tgt)
change = (entry_exit | resize) & (np.abs(tgt - w) > 0)
new[change] = tgt[change]
if new.sum() > 1.0: # kept winners leave less room
buys = change & (new > w)
room = max(1.0 - new[~buys].sum(), 0.0)
new[buys] *= room / new[buys].sum() if new[buys].sum() > 0 else 0.0
delta = new - w
floor = (max(min_trade_value, 0.001 * value) / value) if min_trade_value > 0 else 1e-12
skip = (np.abs(delta) < floor) & (new > 0)
delta[skip] = 0.0
new[skip] = w[skip]
orders = np.abs(delta) > 1e-12
if orders.any():
dollars = np.abs(delta) * value
cost = dollars.sum() * transaction_cost
if fixed_fee > 0 or fee_base > 0 or fee_per_share > 0:
with np.errstate(divide="ignore", invalid="ignore"):
shares = np.where(prices[t] > 0, dollars / prices[t], 0.0)
fee = np.maximum(fixed_fee, fee_base + per_share * shares)
cost += fee[orders].sum()
if cost >= value: # fees ate the book
equity[t:] = 0.0
break
w = new * value / (value - cost)
value -= cost
traded[t + 1] = delta
equity[t] = value
return (pd.Series(equity, index=data.index),
pd.DataFrame(traded, index=data.index, columns=data.columns))
def backtest(
strategy,
data: pd.DataFrame,
@@ -35,6 +129,8 @@ def backtest(
fee_per_share: float = 0.0,
open_data: pd.DataFrame | None = None,
min_trade_value: float = 0.0,
bars: dict[str, pd.DataFrame] | None = None,
resize_band: float = 0.0,
) -> pd.Series:
"""
Vectorized backtest.
@@ -60,39 +156,33 @@ def backtest(
With fee_base=1.88 + fee_per_share=0.009 + fixed_fee=2.0 you get an
IBKR-style schedule: max(2, 1.88 + 0.009 * shares).
open_data : pd.DataFrame, optional
Open prices. When provided, enables open-to-close execution mode.
Open prices. When provided, the backtest keeps the book the way the
live trader does: its signal timing (``strategy_positions``), drifting
holdings, whole orders and ``resize_band`` (``simulate_live_book``).
min_trade_value : float
Per-ticker rebalances smaller than ``max(min_trade_value, 0.1% of
equity)`` are not charged the per-trade fee, mirroring the live
trader, which never places such orders. 0 (default) charges every
weight change.
bars : dict, optional
``{field: DataFrame}`` panels for a BarStrategy (close/open/high/low/volume).
Returns
-------
pd.Series
Daily equity curve.
"""
positions = strategy_positions(strategy, data, open_data=open_data, bars=bars)
if open_data is not None:
# Open-to-close mode:
# Strategy's shift(1) on open prices gives: weights[t] = f(open_{t-1})
# But open_t is known at morning of day t, so undo shift to get f(open_t)
# Then execute at close_t, earning close_t → close_{t+1}
weights = strategy.generate_signals(open_data)
weights = weights.shift(-1).fillna(0.0)
weights = weights.reindex(data.index).fillna(0.0)
positions = weights
equity, _ = simulate_live_book(
positions, data, initial_capital, transaction_cost, fixed_fee, fee_base,
fee_per_share, min_trade_value, resize_band,
)
return equity
# Returns earned: close[t] → close[t+1], weighted by positions decided at open[t]
close_returns = data.pct_change().fillna(0.0)
portfolio_returns = (close_returns * positions.shift(1).fillna(0.0)).sum(axis=1)
else:
# Classic close-to-close mode
weights = strategy.generate_signals(data)
weights = weights.reindex(data.index).fillna(0.0)
positions = weights
daily_returns = data.pct_change().fillna(0.0)
portfolio_returns = (daily_returns * positions).sum(axis=1)
# A missing close carries the last price (zero return), as before.
daily_returns = data.ffill().pct_change(fill_method=None).fillna(0.0)
portfolio_returns = (daily_returns * positions).sum(axis=1)
# Turnover cost: sum of absolute weight changes each day
turnover = positions.diff().abs().sum(axis=1).fillna(0.0)

View File

@@ -25,32 +25,15 @@ import os
import sys
from datetime import datetime
import numpy as np
import pandas as pd
import data_manager
import metrics
import universe_history as uh
from backtest_runner import run_market
from bridge import (
MIN_TRADE_VALUE,
STRATEGY_META,
_build_strategy_result,
_sanitize_for_json,
benchmark_curve,
compute_rebalance_trades,
)
from main import backtest
from trader import (
strategy_extra_tickers,
strategy_supports_market,
ETF_STRATEGY_UNIVERSES,
MONITOR_CAPITAL,
STRATEGY_REGISTRY,
convert_quote_currency,
filter_tradable_tickers,
price_request,
)
from universe import UNIVERSES
from trader import MONITOR_CAPITAL, STRATEGY_REGISTRY, strategy_supports_market
# Initial capital per market, in local currency — the same amounts the live
@@ -102,171 +85,6 @@ def _equity_series_from_curve(curve: list[dict]) -> pd.Series:
return pd.Series(values, index=dates)
def run_market_raw(
market: str,
strategy_names: list[str],
years: int,
capital: float,
tx_cost: float,
fixed_fee: float,
fee_base: float = 0.0,
fee_per_share: float = 0.0,
) -> dict | None:
"""Run all strategies once; return raw equity series + metadata per strategy."""
stock_strategies = []
etf_strategies = []
for name in strategy_names:
base_name = name.removeprefix("sim_")
if base_name in ETF_STRATEGY_UNIVERSES:
etf_strategies.append(name)
else:
stock_strategies.append(name)
raw_strategies: list[dict] = []
benchmark_label = "SPY"
bench_curve = []
start_date = None
end_date = None
# --- Stock strategies ---
if stock_strategies:
universe = UNIVERSES[market]
tickers = universe["fetch"]()
benchmark_ticker = universe["benchmark"]
benchmark_label = universe["benchmark_label"]
pit_intervals = None
if market == "us":
pit_intervals = uh.load_sp500_history()
historical_tickers = uh.all_tickers_ever(pit_intervals)
all_tickers = sorted(set(tickers + historical_tickers + [benchmark_ticker])
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
log(f"PIT universe: {len(all_tickers)} tickers")
else:
all_tickers = sorted(set(tickers + [benchmark_ticker])
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
log(f"WARNING: {market.upper()} lacks PIT history; using current constituents")
log(f"Fetching price data for {len(all_tickers)} tickers...")
data = data_manager.update(market, all_tickers, with_open=False)
if isinstance(data, tuple):
data = data[0]
if years:
cutoff = data.index[-1] - pd.DateOffset(years=years)
data = data[data.index >= cutoff]
if pit_intervals is not None:
log("Applying PIT membership mask...")
data = uh.mask_prices(data, pit_intervals)
tickers = [
t for t in data.columns
if t != benchmark_ticker and data[t].notna().any()
]
else:
tickers = filter_tradable_tickers(data, tickers)
log(f"Universe: {len(tickers)} stocks, {data.index[0].date()} to {data.index[-1].date()}")
for name in stock_strategies:
log(f" Running: {name}")
try:
strategy = STRATEGY_REGISTRY[name]()
# Mixed stock+ETF strategies also read their fixed ETFs.
cols = tickers + [t for t in strategy_extra_tickers(name)
if t in data.columns and t not in tickers]
equity = backtest(
strategy, data[cols],
initial_capital=capital,
transaction_cost=tx_cost,
fixed_fee=fixed_fee,
fee_base=fee_base,
fee_per_share=fee_per_share,
min_trade_value=MIN_TRADE_VALUE,
)
weights = strategy.generate_signals(data[cols])
weights = weights.reindex(data[cols].index).fillna(0.0)
trade_events = compute_rebalance_trades(weights)
raw_strategies.append({"name": name, "equity": equity, "trades": trade_events})
log(f" OK: {name} (final: ${equity.iloc[-1]:,.0f})")
except Exception as e:
log(f" FAILED: {name}: {e}")
# Benchmark on exactly the strategies' dates
bench_curve = benchmark_curve(data[benchmark_ticker], data.index, capital)
start_date = data.index[0].strftime("%Y-%m-%d")
end_date = data.index[-1].strftime("%Y-%m-%d")
# --- ETF strategies ---
if etf_strategies:
for name in etf_strategies:
log(f" Running ETF strategy: {name}")
try:
base_name = name.removeprefix("sim_")
etf_tickers = ETF_STRATEGY_UNIVERSES[base_name]
all_etf = sorted(set(etf_tickers + ["SPY"]))
etf_data = data_manager.update("etfs", price_request(market, all_etf), with_open=False)
if isinstance(etf_data, tuple):
etf_data = etf_data[0]
etf_data = convert_quote_currency(etf_data)
if years:
cutoff = etf_data.index[-1] - pd.DateOffset(years=years)
etf_data = etf_data[etf_data.index >= cutoff]
strategy = STRATEGY_REGISTRY[name]()
tradable = filter_tradable_tickers(etf_data, etf_tickers)
equity = backtest(
strategy, etf_data[tradable],
initial_capital=capital,
transaction_cost=tx_cost,
fixed_fee=fixed_fee,
fee_base=fee_base,
fee_per_share=fee_per_share,
min_trade_value=MIN_TRADE_VALUE,
)
weights = strategy.generate_signals(etf_data[tradable])
weights = weights.reindex(etf_data[tradable].index).fillna(0.0)
trade_events = compute_rebalance_trades(weights)
raw_strategies.append({"name": name, "equity": equity, "trades": trade_events})
log(f" OK: {name} (final: ${equity.iloc[-1]:,.0f})")
except Exception as e:
log(f" FAILED: {name}: {e}")
if not stock_strategies:
benchmark_label = "SPY (Benchmark)"
spy_data = data_manager.update("etfs", ["SPY"], with_open=False)
if isinstance(spy_data, tuple):
spy_data = spy_data[0]
if years:
cutoff = spy_data.index[-1] - pd.DateOffset(years=years)
spy_data = spy_data[spy_data.index >= cutoff]
if "SPY" in spy_data.columns:
bench_curve = benchmark_curve(spy_data["SPY"], spy_data.index, capital)
start_date = spy_data.index[0].strftime("%Y-%m-%d")
end_date = spy_data.index[-1].strftime("%Y-%m-%d")
if not raw_strategies:
return None
return {
"market": market,
"strategy_runs": raw_strategies,
"benchmark_label": benchmark_label,
"benchmark_curve": bench_curve,
"start_date": start_date,
"end_date": end_date,
"tx_cost": tx_cost,
"fixed_fee": fixed_fee,
}
def build_scenario_for_market(
raw: dict,
scenario: dict,
@@ -358,7 +176,7 @@ def main():
log("=" * 60)
# US fee schedule (IBKR-style tiered):
# fee = max(2.0, 1.88 + 0.009 * shares)
us_raw = run_market_raw(
us_raw = run_market(
"us", us_strategies,
years=args.years, capital=us_capital,
tx_cost=0.001, fixed_fee=2.0,
@@ -368,7 +186,7 @@ def main():
log("=" * 60)
log(f"CN MARKET: {len(cn_strategies)} strategies @ ¥{cn_capital:,.0f}")
log("=" * 60)
cn_raw = run_market_raw(
cn_raw = run_market(
"cn", cn_strategies,
years=args.years, capital=cn_capital,
tx_cost=0.001, fixed_fee=5.0,

View File

@@ -10,5 +10,6 @@ dependencies = [
"numpy>=2.4.0",
"pandas>=2.3.3",
"scipy>=1.17.1",
"xlrd>=2.0.2",
"yfinance>=1.0",
]

0
research/lab/__init__.py Normal file
View File

47
research/lab/baselines.py Normal file
View File

@@ -0,0 +1,47 @@
"""
Re-run the existing factor_combo champions under the research engine, with
the same rules every new candidate gets: next-open execution, PIT universe,
costs, and averaging over rebalance phases.
PYTHONPATH=. uv run python -m research.lab.baselines --market us
"""
import argparse
import warnings
import pandas as pd
from research.lab.factor_scan import load, phase_avg
from research.lab.report import fmt
from strategies.factor_combo import SIGNAL_REGISTRY
warnings.filterwarnings("ignore")
CHAMPIONS = {
"us": ["rec_mfilt+deep_upvol", "ma200+mom7m+rec126", "rec_mfilt+ma200", "mom7m+rec126"],
"cn": ["up_cap+quality_mom", "down_resil+qual_mom", "rec63+mom_gap", "up_cap+mom_gap"],
}
def baseline_scores(p, market: str) -> dict[str, pd.DataFrame]:
mk = "us" if market == "us" else "cn"
# factor_combo signals were built on prices masked to index membership
px = p.close.where(p.member).ffill(limit=5)
return {name: SIGNAL_REGISTRY[name](px) for name in CHAMPIONS[mk]}
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--market", default="us")
args = ap.parse_args()
p, periods = load(args.market)
scores = baseline_scores(p, args.market)
for name, s in scores.items():
for every, label in [(5, "weekly"), (21, "monthly")]:
for per, (a, b) in periods.items():
st = phase_avg(p, s, a, b, 10, every)
print(f"{name:22s} top10 {label:8s} {per:8s} {fmt(st)}", flush=True)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,61 @@
"""
Construction sweep: holdings count, buffer (hysteresis), rebalance interval and
weighting for a fixed score, reported per ~5-year block.
PYTHONPATH=. uv run python -m research.lab.construction --market us --combo overnight_mom+rec_126
"""
import argparse
import itertools
import warnings
import numpy as np
import pandas as pd
from research.lab.baselines import baseline_scores
from research.lab.factor_scan import load, phase_avg
from research.lab.factors import FACTORS, zrank
from research.lab.stepwise import SUBPERIODS
warnings.filterwarnings("ignore")
def combo_score(p, combo: str, market: str) -> pd.DataFrame:
if combo.startswith("baseline:"):
return baseline_scores(p, market)[combo.split(":", 1)[1]]
parts = combo.split("+")
return sum(zrank(FACTORS[n](p), p.member) for n in parts) / len(parts)
def block_stats(p, score, subs, **kw) -> dict:
per = [phase_avg(p, score, a, b, **kw) for a, b in subs]
return {"sharpe": np.mean([x["sharpe"] for x in per]), "min_sh": min(x["sharpe"] for x in per),
"cagr": np.mean([x["cagr"] for x in per]), "mdd": min(x["mdd"] for x in per),
"turn": np.mean([x["turnover"] for x in per]),
"blocks": " ".join(f"{x['sharpe']:.2f}/{x['cagr']*100:.0f}%" for x in per)}
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--market", default="us")
ap.add_argument("--combo", nargs="+", required=True)
ap.add_argument("--tops", nargs="*", type=int, default=[10, 15, 20, 30])
ap.add_argument("--buffers", nargs="*", type=float, default=[0, 0.5, 1.0, 2.0])
ap.add_argument("--everys", nargs="*", type=int, default=[5, 10])
ap.add_argument("--weightings", nargs="*", default=["equal", "invvol"])
args = ap.parse_args()
mk = "us" if args.market == "us" else "cn"
p, _ = load(args.market)
subs = SUBPERIODS[mk]
for combo in args.combo:
s = combo_score(p, combo, args.market)
print(f"=== {combo}")
for top, buf, every, wt in itertools.product(args.tops, args.buffers, args.everys, args.weightings):
st = block_stats(p, s, subs, top_n=top, every=every, buffer=buf, weighting=wt)
print(f" top{top:<3d} buf{buf:<4} every{every:<3d} {wt:7s} sh {st['sharpe']:.2f} min {st['min_sh']:.2f} "
f"cagr {st['cagr']*100:5.1f}% worstMDD {st['mdd']*100:5.1f}% turn {st['turn']:4.1f}x [{st['blocks']}]",
flush=True)
if __name__ == "__main__":
main()

252
research/lab/engine.py Normal file
View File

@@ -0,0 +1,252 @@
"""
Event-driven long-only portfolio simulator for factor research.
Differences from ``main.backtest`` (the engine behind the web UI numbers):
* Execution lag. A score computed from the close of day s is traded at the
OPEN of day s+1 (``exec_at="open"``) or the CLOSE of day s+1
(``exec_at="close"``). ``main.backtest`` fills at the same close the signal
saw, which the live trader cannot do.
* Holdings drift between rebalances; costs are charged on actual trades.
* Tradability. A stock that is suspended, or opens at limit-up (buy) or
limit-down (sell) on the execution day keeps its current position. Stocks
outside the point-in-time index on the signal date are never bought.
* Costs are asymmetric (CN stamp tax on sells) and include an optional fixed
per-order fee expressed against a reference capital.
Conventions: every DataFrame shares one date index and one column set
(``Panel.close.columns``). ``score`` is "higher is better"; NaN means
"not eligible".
"""
from __future__ import annotations
from dataclasses import dataclass, field
import numpy as np
import pandas as pd
@dataclass
class Costs:
buy: float = 0.0010 # fraction of traded value
sell: float = 0.0010
sell_tax: pd.Series | None = None # extra sell-side tax by date (CN stamp duty)
fixed_fee: float = 0.0 # currency per order
capital: float = 10_000.0 # reference capital for the fixed fee
@dataclass
class Panel:
market: str
open: pd.DataFrame
high: pd.DataFrame
low: pd.DataFrame
close: pd.DataFrame
volume: pd.DataFrame
member: pd.DataFrame # bool: in PIT index on that date
can_buy: pd.DataFrame # bool: buyable at that day's open
can_sell: pd.DataFrame # bool: sellable at that day's open
cash_ret: pd.Series # daily return of the idle-cash asset
benchmark: pd.Series # index level (cap-weighted)
costs: Costs = field(default_factory=Costs)
extra: dict = field(default_factory=dict) # market-specific fields (turn, pe, ...)
@property
def dates(self) -> pd.DatetimeIndex:
return self.close.index
def slice(self, start=None, end=None) -> "Panel":
idx = self.dates
m = np.ones(len(idx), bool)
if start is not None:
m &= idx >= pd.Timestamp(start)
if end is not None:
m &= idx <= pd.Timestamp(end)
def sub(x):
if isinstance(x, (pd.Series, pd.DataFrame)) and x.index.equals(idx):
return x.loc[m]
return x
return Panel(self.market, sub(self.open), sub(self.high), sub(self.low), sub(self.close),
sub(self.volume), sub(self.member), sub(self.can_buy), sub(self.can_sell),
sub(self.cash_ret), sub(self.benchmark), self.costs,
{k: sub(v) for k, v in self.extra.items() if not k.startswith("_")})
@dataclass
class Result:
equity: pd.Series
returns: pd.Series
turnover: pd.Series # one-way traded value / equity, per day
holdings: pd.DataFrame # weights after each rebalance (rows = exec dates)
exposure: pd.Series # invested fraction at each close
name: str = ""
def rebalance_dates(dates: pd.DatetimeIndex, every: int | str, offset: int = 0,
start: int = 0) -> np.ndarray:
"""Boolean mask of signal dates.
``every`` is a trading-day interval (5 = weekly) or "W"/"M" for the last
trading day of each calendar week/month.
"""
mask = np.zeros(len(dates), bool)
if isinstance(every, str):
key = dates.to_period(every[0])
last = pd.Series(np.arange(len(dates)), index=dates).groupby(key).max().values
mask[last] = True
mask[:start] = False
return mask
mask[start + offset::every] = True
return mask
def select_top(score_row: np.ndarray, top_n: int, held: np.ndarray | None = None,
buffer: float = 0.0) -> np.ndarray:
"""Indices of the chosen names. With a buffer, current holdings stay while
their rank is within ``top_n * (1 + buffer)``."""
valid = np.where(~np.isnan(score_row))[0]
if len(valid) == 0:
return valid
order = valid[np.argsort(-score_row[valid], kind="stable")]
if buffer <= 0 or held is None or not held.any():
return order[:top_n]
rank = np.empty(len(score_row)); rank[:] = np.inf
rank[order] = np.arange(len(order))
keep = [i for i in np.where(held)[0] if rank[i] < top_n * (1 + buffer)]
keep = sorted(keep, key=lambda i: rank[i])[:top_n]
chosen = list(keep)
for i in order:
if len(chosen) >= top_n:
break
if i not in keep:
chosen.append(i)
return np.array(chosen, int)
def simulate(panel: Panel, score: pd.DataFrame | None = None, top_n: int = 10,
every: int | str = 5, offset: int = 0, warmup: int = 0,
weighting: str = "equal", buffer: float = 0.0,
exposure: pd.Series | None = None, exec_at: str = "open",
target_weights: pd.DataFrame | None = None, min_trade: float = 0.002,
name: str = "") -> Result:
"""Run a long-only top-N strategy.
score : higher = better, evaluated at each signal date's close.
target_weights : alternative to score — explicit weights per signal date
(rows need only exist on signal dates; NaN rows skip).
weighting : "equal" | "invvol" (60-day vol) | "score" (score-proportional, clipped >0)
exposure : fraction of equity to invest, read on the signal date
(e.g. a market-trend or vol-target overlay); rest earns cash_ret.
exec_at : "open" → trade at next day's open; "close" → next day's close.
min_trade : resizing trades smaller than this fraction of equity are
skipped (the live trader skips tiny orders too); full
entries and exits always trade.
"""
dates = panel.dates
n_days, n_assets = panel.close.shape
C = panel.close.ffill().values
O = panel.open.values
O = np.where(np.isnan(O), C, O) # missing open → fall back to close
prevC = np.vstack([C[:1], C[:-1]])
member = panel.member.values
can_buy = panel.can_buy.values
can_sell = panel.can_sell.values
cash_ret = panel.cash_ret.reindex(dates).fillna(0.0).values
sell_tax = (panel.costs.sell_tax.reindex(dates).fillna(0.0).values
if panel.costs.sell_tax is not None else np.zeros(n_days))
S = score.reindex(index=dates, columns=panel.close.columns).values if score is not None else None
TW = (target_weights.reindex(index=dates, columns=panel.close.columns).values
if target_weights is not None else None)
expo = exposure.reindex(dates).ffill().fillna(1.0).values if exposure is not None else np.ones(n_days)
if weighting == "invvol":
rets = panel.close.pct_change(fill_method=None)
vol = rets.rolling(60, min_periods=40).std().values
sig_mask = rebalance_dates(dates, every, offset, start=warmup)
h = np.zeros(n_assets) # value held per asset (at last mark)
cash = 1.0
eq = np.empty(n_days); turn = np.zeros(n_days); inv = np.zeros(n_days)
hold_rows, hold_idx = [], []
pending = None # target weight vector awaiting execution
for d in range(n_days):
# mark overnight / full-day moves
with np.errstate(invalid="ignore", divide="ignore"):
gap = np.where(prevC[d] > 0, O[d] / prevC[d], 1.0)
intraday = np.where(O[d] > 0, C[d] / O[d], 1.0)
gap = np.nan_to_num(gap, nan=1.0, posinf=1.0)
intraday = np.nan_to_num(intraday, nan=1.0, posinf=1.0)
cash *= 1 + cash_ret[d]
if pending is not None:
if exec_at == "open":
h *= gap
else:
h *= gap * intraday
V = h.sum() + cash
tgt = pending * V
# untradable names keep their current value
frozen_buy = (tgt > h) & ~can_buy[d]
frozen_sell = (tgt < h) & ~can_sell[d]
frozen = frozen_buy | frozen_sell
if frozen.any():
tgt[frozen] = h[frozen]
free = ~frozen & (pending > 0)
room = V - tgt[frozen].sum()
if free.any() and pending[free].sum() > 0:
budget = min(room, pending[free].sum() * V)
tgt[free] = pending[free] / pending[free].sum() * max(budget, 0.0)
if min_trade > 0:
tiny = (np.abs(tgt - h) < min_trade * V) & (tgt > 0) & (h > 0)
tgt[tiny] = h[tiny]
trade = tgt - h
buys = trade[trade > 0].sum()
sells = -trade[trade < 0].sum()
n_orders = int((np.abs(trade) > 1e-4 * V).sum())
cost = buys * panel.costs.buy + sells * (panel.costs.sell + sell_tax[d])
# Equity is normalized to 1.0 == costs.capital, so a fixed fee per
# order is fixed_fee / capital in these units at any book size.
if panel.costs.fixed_fee > 0:
cost += n_orders * panel.costs.fixed_fee / panel.costs.capital
cash += h.sum() - tgt.sum() - cost
h = tgt
turn[d] = (buys + sells) / 2 / V if V > 0 else 0.0
hold_rows.append(h / max(h.sum() + cash, 1e-12)); hold_idx.append(dates[d])
if exec_at == "open":
h *= intraday
pending = None
else:
h *= gap * intraday
V = h.sum() + cash
eq[d] = V
inv[d] = h.sum() / V if V > 0 else 0.0
if sig_mask[d] and d + 1 < n_days:
if TW is not None:
w = np.nan_to_num(TW[d], nan=0.0)
if np.isnan(TW[d]).all():
continue
else:
row = np.where(member[d], S[d], np.nan)
held = h > 1e-12
idx = select_top(row, top_n, held, buffer)
w = np.zeros(n_assets)
if len(idx) >= min(max(1, top_n // 2), 20):
if weighting == "invvol":
iv = 1.0 / np.where(vol[d, idx] > 0, vol[d, idx], np.nan)
iv = np.nan_to_num(iv, nan=np.nanmean(iv) if np.isfinite(iv).any() else 1.0)
w[idx] = iv / iv.sum()
elif weighting == "score":
s = np.clip(row[idx] - np.nanmin(row[idx]) + 1e-6, 0, None)
w[idx] = s / s.sum()
else:
w[idx] = 1.0 / len(idx)
pending = w * float(np.clip(expo[d], 0.0, 1.0))
equity = pd.Series(eq, index=dates)
rets = equity.pct_change().fillna(0.0)
holdings = pd.DataFrame(hold_rows, index=hold_idx, columns=panel.close.columns)
return Result(equity, rets, pd.Series(turn, index=dates), holdings,
pd.Series(inv, index=dates), name)

143
research/lab/evaluate.py Normal file
View File

@@ -0,0 +1,143 @@
"""
Final evaluation of candidate strategies against the existing champions,
under identical rules (PIT universe, next-open execution, costs, phase
averaging). Writes data/research/results/final_{market}.json.
PYTHONPATH=. uv run python -m research.lab.evaluate --market us
"""
import argparse
import json
import os
import warnings
import numpy as np
import pandas as pd
from research.lab import overlays as ov
from research.lab.construction import combo_score
from research.lab.engine import Costs, simulate
from research.lab.factor_scan import RESULTS, load
from research.lab.report import deflated_sharpe, stats, yearly
from research.lab.stepwise import SUBPERIODS
warnings.filterwarnings("ignore")
# name -> (combo, construction). "baseline:" combos are the current factor_combo champions.
CANDIDATES = {
"us": {
"champion rec_mfilt+deep_upvol monthly": ("baseline:rec_mfilt+deep_upvol", dict(top_n=10, every=21)),
"champion mom7m+rec126 monthly": ("baseline:mom7m+rec126", dict(top_n=10, every=21)),
"champion mom7m+rec126 weekly": ("baseline:mom7m+rec126", dict(top_n=10, every=5)),
"EW S&P 500 (PIT)": ("ew", dict(top_n=10000, every=5)),
"new overnight+rec": ("overnight_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
"new overnight6+rec": ("overnight_mom_6+rec_126", dict(top_n=10, every=10, buffer=2.0)),
"new smooth-mom4": ("overnight_mom+overnight_mom_6+amp_cut_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
"new overnight+ampcut+rec": ("overnight_mom+amp_cut_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
},
"cn": {
"champion up_cap+quality_mom weekly": ("baseline:up_cap+quality_mom", dict(top_n=10, every=5)),
"champion up_cap+mom_gap weekly": ("baseline:up_cap+mom_gap", dict(top_n=10, every=5)),
"champion down_resil+qual_mom biweekly": ("baseline:down_resil+qual_mom", dict(top_n=10, every=10)),
"EW CSI 300 (PIT)": ("ew", dict(top_n=10000, every=5)),
},
}
LIVE_FEES = {"us": (2.0, 10_000.0), "cn": (5.0, 70_000.0)}
WINDOW = ("2016-09-27", "2026-09-24") # the web UI's 10-year window
def score_for(p, combo, market):
if combo == "ew":
return p.close.notna().astype(float).where(p.member)
return combo_score(p, combo, market)
def phase_runs(p, score, a, b, exposure=None, **kw):
every = kw.get("every", 5)
w0 = p.dates.searchsorted(pd.Timestamp(a))
sp = p.slice(p.dates[max(w0 - 5, 0)], b)
runs = []
for off in range(every if kw.get("top_n", 10) < 1000 else 1):
r = simulate(sp, score, offset=off, warmup=5, exposure=exposure, **kw)
runs.append(r)
return runs
def summarize(runs, a, b) -> dict:
sts = [stats(r.equity.loc[a:b], r.turnover) for r in runs]
df = pd.DataFrame(sts)
rets = pd.concat([r.returns.loc[a:b] for r in runs], axis=1).mean(axis=1)
eq = (1 + rets).cumprod()
out = {k: float(df[k].mean()) for k in df.columns}
out["cagr_phase_min"] = float(df["cagr"].min())
out["cagr_phase_max"] = float(df["cagr"].max())
out["yearly"] = {str(k.year): float(v) for k, v in yearly(eq).items()}
out["_rets"] = rets
return out
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--market", default="us")
ap.add_argument("--add", nargs="*", default=[], help="extra 'name=combo|top|every|buffer'")
ap.add_argument("--n-trials", type=int, default=0, help="override trial count for DSR")
args = ap.parse_args()
mk = "us" if args.market == "us" else "cn"
p, _ = load(args.market)
cands = dict(CANDIDATES[mk])
for spec in args.add:
name, rest = spec.split("=", 1)
combo, top, every, buf = rest.split("|")
cands[name] = (combo, dict(top_n=int(top), every=int(every), buffer=float(buf)))
trials_path = os.path.join(RESULTS, f"trials_{args.market}.csv")
n_trials = args.n_trials or (len(pd.read_csv(trials_path)) if os.path.exists(trials_path) else 100)
full = (SUBPERIODS[mk][0][0], WINDOW[1])
out = {}
fee, cap = LIVE_FEES[mk]
for name, (combo, kw) in cands.items():
s = score_for(p, combo, args.market)
res = {"combo": combo, "construction": kw}
base_runs = phase_runs(p, s, *WINDOW, **kw)
res["window"] = summarize(base_runs, *WINDOW)
res["full"] = summarize(phase_runs(p, s, *full, **kw), *full)
res["blocks"] = [summarize(phase_runs(p, s, a, b, **kw), a, b) for a, b in SUBPERIODS[mk]]
# live-book fixed fee
pf = p.slice(); pf.costs = Costs(p.costs.buy, p.costs.sell, p.costs.sell_tax, fee, cap)
res["window_livefee"] = summarize(phase_runs(pf, s, *WINDOW, **kw), *WINDOW)
# double proportional cost
p2 = p.slice(); p2.costs = Costs(p.costs.buy * 2, p.costs.sell * 2, p.costs.sell_tax, 0.0, cap)
res["window_2xcost"] = summarize(phase_runs(p2, s, *WINDOW, **kw), *WINDOW)
res["window_exec_close"] = summarize(phase_runs(p, s, *WINDOW, exec_at="close", **kw), *WINDOW)
# drawdown-control overlay: slow/fast market trend blend on the EW index
ew = ov.ew_index(p)
slow = (ew / ew.shift(252) > 1).astype(float)
fast = (ew / ew.shift(21) > 1).astype(float)
expo = (0.5 + 0.25 * slow + 0.25 * fast).where(slow + fast < 2, 1.0)
res["window_trend_overlay"] = summarize(phase_runs(p, s, *WINDOW, exposure=expo, **kw), *WINDOW)
dsr = deflated_sharpe(res["full"]["_rets"], n_trials)
res["dsr_full"] = dsr
out[name] = res
w, f = res["window"], res["full"]
print(f"{name:42s} 2016-26: CAGR {w['cagr']*100:5.1f}% [{w['cagr_phase_min']*100:4.1f},{w['cagr_phase_max']*100:4.1f}] "
f"Sh {w['sharpe']:.2f} MDD {w['mdd']*100:5.1f}% | full: CAGR {f['cagr']*100:5.1f}% Sh {f['sharpe']:.2f} "
f"MDD {f['mdd']*100:5.1f}% | blocks Sh {' '.join(f'{b_['sharpe']:.2f}' for b_ in res['blocks'])} "
f"| livefee {res['window_livefee']['cagr']*100:5.1f}% 2x {res['window_2xcost']['cagr']*100:5.1f}% "
f"close {res['window_exec_close']['cagr']*100:5.1f}% trend {res['window_trend_overlay']['cagr']*100:5.1f}%/"
f"{res['window_trend_overlay']['sharpe']:.2f}/{res['window_trend_overlay']['mdd']*100:.0f}% "
f"| DSR {dsr['dsr']:.2f} (N={n_trials})", flush=True)
def clean(o):
if isinstance(o, dict):
return {k: clean(v) for k, v in o.items() if not k.startswith("_")}
if isinstance(o, list):
return [clean(v) for v in o]
if isinstance(o, (np.floating, np.integer)):
return float(o)
return o
os.makedirs(RESULTS, exist_ok=True)
with open(os.path.join(RESULTS, f"final_{args.market}.json"), "w") as fh:
json.dump(clean(out), fh, indent=1)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,78 @@
"""
Export phase-averaged equity curves of the final candidates and baselines for
the report (weekly samples), plus the US/CN correlation of the new strategies.
PYTHONPATH=. uv run python -m research.lab.export_curves
"""
import json
import os
import warnings
import numpy as np
import pandas as pd
from research.lab.evaluate import phase_runs, score_for
from research.lab.factor_scan import RESULTS, load
from research.lab.report import stats
warnings.filterwarnings("ignore")
SETS = {
"us": ("2006-09-27", "2026-09-24", {
"新: overnight+recovery": ("overnight_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
"现: rec_mfilt+deep_upvol 月频": ("baseline:rec_mfilt+deep_upvol", dict(top_n=10, every=21)),
"现: mom7m+rec126 月频": ("baseline:mom7m+rec126", dict(top_n=10, every=21)),
"等权 S&P 500 (PIT)": ("ew", dict(top_n=10000, every=5)),
}),
"cn_hs300": ("2010-01-04", "2026-09-24", {
"新: 低风险动量4因子": ("amp_cut_mom+low_beta+pv_corr+sharpe_mom", dict(top_n=15, every=10, buffer=2.0)),
"新: low_beta+sharpe_mom": ("low_beta+sharpe_mom", dict(top_n=20, every=10, buffer=1.0)),
"现: down_resil+qual_mom 双周": ("baseline:down_resil+qual_mom", dict(top_n=10, every=10)),
"现: up_cap+quality_mom 周频": ("baseline:up_cap+quality_mom", dict(top_n=10, every=5)),
"等权 CSI 300 (PIT)": ("ew", dict(top_n=10000, every=5)),
}),
}
def main() -> None:
out, daily = {}, {}
for market, (a, b, cands) in SETS.items():
p, _ = load(market)
curves = {}
for name, (combo, kw) in cands.items():
s = score_for(p, combo, market)
runs = phase_runs(p, s, a, b, **kw)
rets = pd.concat([r.returns.loc[a:b] for r in runs], axis=1).mean(axis=1)
daily[(market, name)] = rets
eq = (1 + rets).cumprod()
wk = eq.resample("W-FRI").last().dropna()
dd = eq / eq.cummax() - 1
curves[name] = {"dates": [d.strftime("%Y-%m-%d") for d in wk.index],
"equity": [round(float(v), 4) for v in wk.values],
"drawdown": [round(float(v), 4) for v in dd.resample("W-FRI").min().dropna().values],
"stats": {k: float(v) for k, v in stats(eq).items()}}
print(market, name, {k: round(v, 3) for k, v in curves[name]["stats"].items()}, flush=True)
out[market] = curves
us = daily[("us", "新: overnight+recovery")]
cn = daily[("cn_hs300", "新: 低风险动量4因子")]
# compound each market on its own calendar before joining: the exchanges
# have different holidays, and an inner join on days would drop returns
m = pd.concat([us, cn], axis=1, keys=["us", "cn"]).dropna().loc["2016-09-27":]
mon = pd.concat({k: (1 + v.loc["2016-09-27":]).resample("ME").prod() - 1
for k, v in (("us", us), ("cn", cn))}, axis=1).dropna()
cross = {"daily_corr": float(m.corr().iloc[0, 1]), "monthly_corr": float(mon.corr().iloc[0, 1])}
for col in ["us", "cn", "5050"]:
r = mon.mean(axis=1) if col == "5050" else mon[col]
eq = (1 + r).cumprod()
cross[col] = {"cagr": float(eq.iloc[-1] ** (12 / len(r)) - 1),
"sharpe_monthly": float(r.mean() / r.std() * np.sqrt(12)),
"mdd_monthly": float((eq / eq.cummax() - 1).min())}
out["cross"] = cross
print("cross", out["cross"])
with open(os.path.join(RESULTS, "curves.json"), "w") as fh:
json.dump(out, fh, ensure_ascii=False, default=float)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,95 @@
"""
Single-factor scan: rank IC and long-only top-N backtests per period.
PYTHONPATH=. uv run python -m research.lab.factor_scan --market us
PYTHONPATH=. uv run python -m research.lab.factor_scan --market cn_hs300
Writes data/research/results/factor_scan_{market}.csv.
"""
import argparse
import os
import time
import warnings
import pandas as pd
from research.lab import panel as pnl
from research.lab.engine import simulate
from research.lab.factors import FACTORS
from research.lab.report import (forward_returns, ic_series, ic_summary, stats,
top_minus_universe)
warnings.filterwarnings("ignore")
PERIODS = {
"us": {"design": ("2006-01-01", "2016-09-26"), "holdout": ("2016-09-27", "2026-09-24")},
"cn": {"design": ("2010-01-01", "2016-09-26"), "holdout": ("2016-09-27", "2026-09-24")},
}
RESULTS = os.path.join("data", "research", "results")
def load(market: str):
if market == "us":
return pnl.load_us(), PERIODS["us"]
return pnl.load_cn(market.split("_", 1)[1]), PERIODS["cn"]
def phase_avg(p, score, start, end, top_n, every, **kw) -> dict:
"""Average stats over every rebalance phase (offset 0..every-1)."""
w0 = p.dates.searchsorted(pd.Timestamp(start))
sp = p.slice(p.dates[max(w0 - 5, 0)], end)
rows = []
for off in range(every):
r = simulate(sp, score, top_n=top_n, every=every, offset=off, warmup=5, **kw)
rows.append(stats(r.equity.loc[start:end], r.turnover))
return pd.DataFrame(rows).mean().to_dict()
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--market", default="us")
ap.add_argument("--top-n", type=int, default=20)
ap.add_argument("--every", type=int, default=5)
ap.add_argument("--only", nargs="*")
args = ap.parse_args()
mk = "us" if args.market == "us" else "cn"
p, periods = load(args.market)
fwd5 = forward_returns(p, 5)
fwd20 = forward_returns(p, 20)
rows = []
names = args.only or [n for n, f in FACTORS.items() if mk in f.markets]
for name in names:
t = time.time()
try:
s = FACTORS[name](p)
except Exception as exc: # noqa: BLE001
print(f"{name}: failed {exc}")
continue
row = {"factor": name}
for per, (a, b) in periods.items():
sl = slice(a, b)
ic5 = ic_summary(ic_series(s.loc[sl], fwd5.loc[sl], p.member.loc[sl], 5))
ic20 = ic_summary(ic_series(s.loc[sl], fwd20.loc[sl], p.member.loc[sl], 20), 12)
tmu = top_minus_universe(s.loc[sl], fwd5.loc[sl], p.member.loc[sl], args.top_n, 5)
st = phase_avg(p, s, a, b, args.top_n, args.every)
row.update({f"{per}_ic5": ic5["ic"], f"{per}_ic5_t": ic5["t"],
f"{per}_ic20": ic20["ic"], f"{per}_ic20_t": ic20["t"],
f"{per}_top_excess_ann": tmu.mean() * 52,
f"{per}_cagr": st["cagr"], f"{per}_sharpe": st["sharpe"],
f"{per}_mdd": st["mdd"], f"{per}_turn": st["turnover"]})
rows.append(row)
print(f"{name:16s} "
f"D: ic5 {row['design_ic5']:+.3f} (t{row['design_ic5_t']:+.1f}) ic20 {row['design_ic20']:+.3f} "
f"xs {row['design_top_excess_ann']*100:+5.1f}% cagr {row['design_cagr']*100:5.1f}% sh {row['design_sharpe']:.2f} | "
f"H: ic5 {row['holdout_ic5']:+.3f} (t{row['holdout_ic5_t']:+.1f}) ic20 {row['holdout_ic20']:+.3f} "
f"xs {row['holdout_top_excess_ann']*100:+5.1f}% cagr {row['holdout_cagr']*100:5.1f}% sh {row['holdout_sharpe']:.2f} "
f"turn {row['holdout_turn']:.0f}x [{time.time()-t:.0f}s]", flush=True)
# equal-weight universe reference
os.makedirs(RESULTS, exist_ok=True)
out = pd.DataFrame(rows)
out.to_csv(os.path.join(RESULTS, f"factor_scan_{args.market}.csv"), index=False)
if __name__ == "__main__":
main()

529
research/lab/factors.py Normal file
View File

@@ -0,0 +1,529 @@
"""
Factor library. Every factor maps a Panel to a DataFrame (dates x assets)
where HIGHER = MORE ATTRACTIVE and the value on date t uses data up to the
close of t only. Sign conventions follow the literature: e.g. ``ivol`` returns
NEGATIVE idiosyncratic volatility because low-ivol stocks earn more.
Windows are in trading days: 21 ≈ 1 month, 63 ≈ 1 quarter, 252 ≈ 1 year.
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from research.lab.engine import Panel
FACTORS: dict[str, callable] = {}
def factor(name: str, *, markets: tuple[str, ...] = ("us", "cn")):
def deco(fn):
fn.markets = markets
FACTORS[name] = fn
return fn
return deco
# ---------------------------------------------------------------------------
# helpers
# ---------------------------------------------------------------------------
def _ret(p: Panel) -> pd.DataFrame:
key = "_ret"
if key not in p.extra:
p.extra[key] = p.close.pct_change(fill_method=None)
return p.extra[key]
def _mkt(p: Panel) -> pd.Series:
"""Equal-weight return of PIT members (the investable universe)."""
key = "_mkt"
if key not in p.extra:
p.extra[key] = _ret(p).where(p.member).mean(axis=1).fillna(0.0)
return p.extra[key]
def _rolling_beta(r: pd.DataFrame, m: pd.Series, win: int, minp: int) -> pd.DataFrame:
mr = r.mul(m, axis=0)
e_xy = mr.rolling(win, min_periods=minp).mean()
e_x = r.rolling(win, min_periods=minp).mean()
e_y = m.rolling(win, min_periods=minp).mean()
var_y = (m ** 2).rolling(win, min_periods=minp).mean() - e_y ** 2
return (e_xy - e_x.mul(e_y, axis=0)).div(var_y, axis=0)
def resid_returns(p: Panel, win: int = 252) -> pd.DataFrame:
key = f"_resid{win}"
if key not in p.extra:
r = _ret(p)
m = _mkt(p)
beta = _rolling_beta(r, m, win, win // 2).shift(1)
p.extra[key] = r - beta.mul(m, axis=0)
return p.extra[key]
def sector_map(p: Panel) -> pd.Series | None:
return p.extra.get("sector")
def demean_by_sector(df: pd.DataFrame, p: Panel) -> pd.DataFrame:
sec = sector_map(p)
if sec is None:
return df.sub(df.where(p.member).mean(axis=1), axis=0)
out = df.copy()
m = df.where(p.member)
for s in sec.dropna().unique():
cols = [c for c in sec.index[sec == s] if c in df.columns]
if cols:
out[cols] = df[cols].sub(m[cols].mean(axis=1), axis=0)
return out
# ---------------------------------------------------------------------------
# momentum family
# ---------------------------------------------------------------------------
@factor("mom_12_1")
def mom_12_1(p):
c = p.close.ffill()
return c.shift(21) / c.shift(252) - 1
@factor("mom_6_1")
def mom_6_1(p):
c = p.close.ffill()
return c.shift(21) / c.shift(126) - 1
@factor("mom_12_7") # Novy-Marx (2012) intermediate momentum
def mom_12_7(p):
c = p.close.ffill()
return c.shift(147) / c.shift(252) - 1
@factor("mom_3")
def mom_3(p):
c = p.close.ffill()
return c / c.shift(63) - 1
@factor("resid_mom") # Blitz, Huij & Martens (2011): residual 12-1, vol-scaled
def resid_mom(p):
e = resid_returns(p)
s = e.shift(21).rolling(231, min_periods=180)
return s.sum() / s.std()
@factor("resid_mom_6")
def resid_mom_6(p):
e = resid_returns(p)
s = e.shift(21).rolling(105, min_periods=80)
return s.sum() / s.std()
@factor("sharpe_mom") # 12-1 return / 12-month vol
def sharpe_mom(p):
r = _ret(p)
return mom_12_1(p) / r.rolling(252, min_periods=180).std()
@factor("high_52w") # George & Hwang (2004)
def high_52w(p):
c = p.close.ffill()
return c / c.rolling(252, min_periods=200).max()
@factor("fip") # Da, Gurun & Warachka (2014) frog-in-the-pan (continuous info)
def fip(p):
r = _ret(p).shift(21)
pos = (r > 0).astype(float).where(r.notna()).rolling(231, min_periods=180).mean()
neg = (r < 0).astype(float).where(r.notna()).rolling(231, min_periods=180).mean()
m = mom_12_1(p)
return -np.sign(m) * (neg - pos) # high = smooth, continuous gains
@factor("mom_fip") # momentum restricted to continuous-information winners
def mom_fip(p):
return mom_12_1(p).rank(axis=1, pct=True) + fip(p).rank(axis=1, pct=True)
@factor("rec_126") # existing house factor: distance from 126d low
def rec_126(p):
c = p.close.ffill()
return c / c.rolling(126, min_periods=126).min() - 1
@factor("ma200_dist")
def ma200_dist(p):
c = p.close.ffill()
return c / c.rolling(200, min_periods=200).mean() - 1
@factor("ts_trend") # fraction of 10 lookbacks (1..12m) with positive return
def ts_trend(p):
c = p.close.ffill()
looks = [21, 42, 63, 84, 105, 126, 147, 168, 189, 210, 231, 252]
return sum((c / c.shift(k) > 1).astype(float) for k in looks) / len(looks)
@factor("sector_mom") # Moskowitz & Grinblatt (1999) industry momentum, 6-1
def sector_mom(p):
sec = sector_map(p)
m = mom_6_1(p)
if sec is None:
return m * np.nan
out = pd.DataFrame(np.nan, index=m.index, columns=m.columns)
mm = m.where(p.member)
for s in sec.dropna().unique():
cols = [c for c in sec.index[sec == s] if c in m.columns]
if cols:
avg = mm[cols].mean(axis=1)
out[cols] = np.repeat(avg.values[:, None], len(cols), axis=1)
return out
# ---------------------------------------------------------------------------
# reversal family
# ---------------------------------------------------------------------------
@factor("rev_5")
def rev_5(p):
c = p.close.ffill()
return -(c / c.shift(5) - 1)
@factor("rev_21")
def rev_21(p):
c = p.close.ffill()
return -(c / c.shift(21) - 1)
@factor("rev_resid_21") # residual (beta-adjusted) 1-month reversal
def rev_resid_21(p):
return -resid_returns(p).rolling(21, min_periods=15).sum()
@factor("rev_sector_21") # industry-adjusted reversal (Da, Liu & Schaumburg 2014 style)
def rev_sector_21(p):
return demean_by_sector(rev_21(p), p)
@factor("overnight_mom") # Lou, Polk & Skouras (2019): overnight returns persist
def overnight_mom(p):
o = p.open
cprev = p.close.ffill().shift(1)
on = np.log(o / cprev)
return on.rolling(252, min_periods=180).sum().shift(21)
@factor("intraday_rev") # intraday component of last month reverses
def intraday_rev(p):
intr = np.log(p.close / p.open)
return -intr.rolling(21, min_periods=15).sum()
# ---------------------------------------------------------------------------
# risk / lottery family (low-risk anomaly)
# ---------------------------------------------------------------------------
@factor("ivol") # Ang, Hodrick, Xing & Zhang (2006): low ivol
def ivol(p):
return -resid_returns(p).rolling(63, min_periods=40).std()
@factor("low_vol")
def low_vol(p):
return -_ret(p).rolling(252, min_periods=180).std()
@factor("low_beta") # Frazzini & Pedersen (2014) BAB, long leg
def low_beta(p):
return -_rolling_beta(_ret(p), _mkt(p), 252, 180)
@factor("max_ret") # Bali, Cakici & Whitelaw (2011): avoid lottery stocks (MAX(1))
def max_ret(p):
return -_ret(p).rolling(21, min_periods=15).max()
@factor("skew") # low skewness earns more
def skew(p):
return -_ret(p).rolling(252, min_periods=180).skew()
# ---------------------------------------------------------------------------
# volume / liquidity family
# ---------------------------------------------------------------------------
def _dollar_volume(p):
if "amount" in p.extra:
return p.extra["amount"]
return p.close * p.volume
def _turnover(p):
if "turn" in p.extra:
return p.extra["turn"]
# US: no shares outstanding → dollar volume relative to its own 1y mean
return None
@factor("amihud") # Amihud (2002): illiquid earns more
def amihud(p):
dv = _dollar_volume(p).replace(0, np.nan)
return np.log((_ret(p).abs() / dv).rolling(21, min_periods=15).mean())
@factor("abn_volume") # Gervais, Kaniel & Mingelgrin (2001) high-volume return premium
def abn_volume(p):
dv = _dollar_volume(p)
return np.log(dv.rolling(5, min_periods=4).mean() / dv.rolling(252, min_periods=180).mean())
@factor("low_turnover") # CN: low turnover / low attention earns more
def low_turnover(p):
t = _turnover(p)
if t is None:
dv = _dollar_volume(p)
return -np.log(dv.rolling(21, min_periods=15).mean() / dv.rolling(252, min_periods=180).mean())
return -t.rolling(21, min_periods=15).mean()
@factor("abn_turnover") # CN: turnover now vs 1y (negative = attention fading)
def abn_turnover(p):
t = _turnover(p)
if t is None:
return abn_volume(p) * -1
return -np.log(t.rolling(21, min_periods=15).mean() / t.rolling(252, min_periods=180).mean())
@factor("turn_vol") # CN: volatility of turnover (low is better)
def turn_vol(p):
t = _turnover(p)
if t is None:
t = _dollar_volume(p)
s = t.rolling(21, min_periods=15)
return -(s.std() / s.mean())
@factor("pv_corr") # CN: price-volume correlation (negative corr is better)
def pv_corr(p):
v = _dollar_volume(p)
return -p.close.ffill().rolling(21, min_periods=15).corr(v)
@factor("upvol_ratio") # share of volume on up days, 21d
def upvol_ratio(p):
r = _ret(p)
v = _dollar_volume(p)
up = v.where(r > 0, 0.0).rolling(21, min_periods=15).sum()
return up / v.rolling(21, min_periods=15).sum()
# ---------------------------------------------------------------------------
# value / fundamentals (CN: baostock daily ratios)
# ---------------------------------------------------------------------------
def _inv(x: pd.DataFrame) -> pd.DataFrame:
return (1.0 / x).replace([np.inf, -np.inf], np.nan)
@factor("ep", markets=("cn",))
def ep(p):
return _inv(p.extra["pe"])
@factor("bp", markets=("cn",))
def bp(p):
return _inv(p.extra["pb"])
@factor("sp", markets=("cn",))
def sp(p):
return _inv(p.extra["ps"])
@factor("cfp", markets=("cn",))
def cfp(p):
return _inv(p.extra["pcf"])
@factor("roe_proxy", markets=("cn",)) # ROE = PB / PE (TTM earnings / book)
def roe_proxy(p):
roe = p.extra["pb"] / p.extra["pe"]
return roe.where(p.extra["pe"] > 0)
@factor("size", markets=("cn",)) # small float cap earns more (within index)
def size(p):
return -np.log(p.extra["float_cap"].rolling(21, min_periods=5).mean())
# ---------------------------------------------------------------------------
# composite utilities
# ---------------------------------------------------------------------------
def zrank(df: pd.DataFrame, member: pd.DataFrame | None = None) -> pd.DataFrame:
"""Cross-sectional percentile rank in [0, 1] among members (NaN elsewhere)."""
if member is not None:
df = df.where(member)
return df.rank(axis=1, pct=True)
def composite(p: Panel, weights: dict[str, float], cache: dict | None = None) -> pd.DataFrame:
total = None
wsum = 0.0
for name, w in weights.items():
f = cache[name] if cache is not None and name in cache else FACTORS[name](p)
r = zrank(f, p.member)
total = r * w if total is None else total.add(r * w, fill_value=np.nan)
wsum += abs(w)
return total / wsum
# ---------------------------------------------------------------------------
# literature shortlist additions
# ---------------------------------------------------------------------------
@factor("overnight_mom_6") # 6-month overnight return, no skip
def overnight_mom_6(p):
on = np.log(p.open / p.close.ffill().shift(1))
return on.rolling(126, min_periods=90).sum()
@factor("overnight_mom_1") # last month's overnight return
def overnight_mom_1(p):
on = np.log(p.open / p.close.ffill().shift(1))
return on.rolling(21, min_periods=15).sum()
@factor("stmom_hiturn") # Medhat & Schmeling (2022): 1-month momentum among high-turnover
def stmom_hiturn(p): # names, skipping the last 3 days; turnover proxied by
c = p.close.ffill() # abnormal dollar volume when shares outstanding are unknown
r1 = c.shift(3) / c.shift(24) - 1
t = _turnover(p)
if t is None:
dv = _dollar_volume(p)
t = dv.rolling(21, min_periods=15).mean() / dv.rolling(252, min_periods=180).mean()
else:
t = t.rolling(21, min_periods=15).mean()
hi = t.where(p.member).rank(axis=1, pct=True) >= 0.7
return r1.where(hi)
@factor("seasonality") # Heston & Sadka (2008): same-calendar-month return, years 1-10
def seasonality(p):
m = p.close.ffill().resample("ME").last()
mret = m.pct_change(fill_method=None)
lags = [mret.shift(12 * k) for k in range(1, 11)]
avg = sum(l.fillna(0) for l in lags) / sum(l.notna().astype(float) for l in lags).replace(0, np.nan)
# value known at month-end t predicts month t+1: shift so the forecast for
# next month (lag 11, 23, ...) is available at the start of that month
fwd = avg.shift(-1) # uses months t+1-12k, all <= t
daily = fwd.reindex(p.close.index, method="ffill")
return daily
@factor("trend_hzz") # Han, Zhou & Zhu (2016) trend factor
def trend_hzz(p):
c = p.close.ffill()
lags = [3, 5, 10, 20, 50, 100, 200, 400]
feats = {L: (c.rolling(L, min_periods=L).mean() / c) for L in lags}
# monthly cross-sectional regression of next-month return on MA signals
month_end = c.resample("ME").last().index
me = c.index[c.index.searchsorted(month_end, side="right") - 1]
fwd = c.shift(-21) / c - 1
betas = []
for d in me:
y = fwd.loc[d].where(p.member.loc[d])
X = pd.DataFrame({L: feats[L].loc[d] for L in lags})
ok = y.notna() & X.notna().all(axis=1)
if ok.sum() < 50:
betas.append(pd.Series(np.nan, index=lags, name=d)); continue
Xo = np.column_stack([np.ones(ok.sum()), X[ok].values])
b = np.linalg.lstsq(Xo, y[ok].values, rcond=None)[0][1:]
betas.append(pd.Series(b, index=lags, name=d))
B = pd.DataFrame(betas)
# the regression at month d uses returns through d+21, so it is usable at d+21;
# shift by one month, then average the last 12 known betas
Ebeta = B.shift(1).rolling(12, min_periods=6).mean()
Ed = Ebeta.reindex(c.index, method="ffill")
return sum(feats[L].mul(Ed[L], axis=0) for L in lags)
# ---------------------------------------------------------------------------
# A-share additions (CN literature review). Volume ratios stand in for
# turnover ratios: shares outstanding rarely change inside the window.
# ---------------------------------------------------------------------------
def _limit_hit(p) -> pd.DataFrame:
r = _ret(p)
lim = p.extra.get("limit")
if lim is None:
return r > 0.095
return r >= lim.reindex_like(r) - 0.005
@factor("mom_ex_limitup", markets=("cn",)) # 国信: 12-1 momentum without limit-up days
def mom_ex_limitup(p):
lr = np.log1p(_ret(p)).where(~_limit_hit(p), 0.0)
return lr.shift(21).rolling(231, min_periods=180).sum()
@factor("amp_cut_mom", markets=("us", "cn")) # 开源: momentum on the 70% lowest-amplitude days (160d)
def amp_cut_mom(p):
amp = (p.high / p.low - 1).values
r = _ret(p).values
n, keep = 160, 0.7
out = np.full(r.shape, np.nan)
from numpy.lib.stride_tricks import sliding_window_view as swv
A = swv(np.nan_to_num(amp, nan=np.inf), n, axis=0) # (T-n+1, N, n)
R = swv(np.nan_to_num(r, nan=0.0), n, axis=0)
for t0 in range(0, A.shape[0], 256): # chunk to bound memory
a, rr = A[t0:t0 + 256], R[t0:t0 + 256]
thr = np.quantile(np.where(np.isinf(a), np.nan, a), keep, axis=2, keepdims=True)
out[t0 + n - 1:t0 + n - 1 + a.shape[0]] = np.where(a <= thr, rr, 0.0).sum(axis=2)
valid = pd.DataFrame(amp, index=p.close.index).notna().rolling(n).sum() >= 0.8 * n
return pd.DataFrame(out, index=p.close.index, columns=p.close.columns).where(valid.values)
@factor("ideal_amp", markets=("us", "cn")) # 开源 理想振幅: amp on high-price days minus low-price days
def ideal_amp(p):
n, lam = 20, 0.25
amp = (p.high / p.low - 1).values
c = p.close.ffill().values
from numpy.lib.stride_tricks import sliding_window_view as swv
A = swv(amp, n, axis=0)
Cw = swv(c, n, axis=0)
out = np.full(amp.shape, np.nan)
hi_q = np.nanquantile(Cw, 1 - lam, axis=2, keepdims=True)
lo_q = np.nanquantile(Cw, lam, axis=2, keepdims=True)
with np.errstate(invalid="ignore"):
v_hi = np.nanmean(np.where(Cw >= hi_q, A, np.nan), axis=2)
v_lo = np.nanmean(np.where(Cw <= lo_q, A, np.nan), axis=2)
out[n - 1:] = v_hi - v_lo
return -pd.DataFrame(out, index=p.close.index, columns=p.close.columns)
@factor("abn_vol_ratio", markets=("us", "cn")) # LSY PMO: -mean(vol,20)/mean(vol,250)
def abn_vol_ratio(p):
v = p.volume.where(p.volume > 0)
return -(v.rolling(20, min_periods=15).mean() / v.rolling(250, min_periods=180).mean())
@factor("vol_cv", markets=("us", "cn")) # HQZ turnover volatility, scale-free
def vol_cv(p):
v = p.volume.where(p.volume > 0)
s = v.rolling(20, min_periods=15)
return -(s.std() / s.mean())
@factor("overnight_20", markets=("us", "cn")) # 国信 overnight momentum, 20d
def overnight_20(p):
return (p.open / p.close.ffill().shift(1) - 1).rolling(20, min_periods=15).sum()
@factor("ivol_20", markets=("us", "cn"))
def ivol_20(p):
return -resid_returns(p).rolling(20, min_periods=15).std()

157
research/lab/fetch_cn.py Normal file
View File

@@ -0,0 +1,157 @@
"""
Fetch point-in-time CSI 300 / CSI 500 membership and daily bars from baostock.
The live CN panel (data/cn.csv) holds today's CSI 300 members only, so every
CN backtest on it carries survivorship bias. baostock serves historical index
membership, delisted stocks, suspensions (tradestatus), ST flags, turnover and
valuation ratios, which lets research backtests trade the index as it was.
Usage:
uv run --with baostock python -m research.lab.fetch_cn [--start 2007-01-01] [--workers 4]
Writes data/research/cn_ohlcv.pkl:
fields: {"open","high","low","close","preclose","volume","amount","turn",
"pctchg","tradestatus","isst","pe","pb","ps","pcf"} DataFrames
(dates x codes; prices back-adjusted, 后复权)
members: {"hs300": {date: [codes]}, "zz500": {date: [codes]}}
index: DataFrame of CSI 300 / CSI 500 index closes
"""
import argparse
import os
import pickle
from datetime import date
from multiprocessing import Pool
import pandas as pd
OUT = os.path.join("data", "research", "cn_ohlcv.pkl")
K_FIELDS = ("date,code,open,high,low,close,preclose,volume,amount,turn,"
"tradestatus,pctChg,peTTM,pbMRQ,psTTM,pcfNcfTTM,isST")
NUMERIC = ["open", "high", "low", "close", "preclose", "volume", "amount", "turn",
"pctChg", "peTTM", "pbMRQ", "psTTM", "pcfNcfTTM", "tradestatus", "isST"]
RENAME = {"pctChg": "pctchg", "peTTM": "pe", "pbMRQ": "pb", "psTTM": "ps",
"pcfNcfTTM": "pcf", "isST": "isst"}
def _rows(rs) -> list[list[str]]:
out = []
while rs.error_code == "0" and rs.next():
out.append(rs.get_row_data())
return out
def _login() -> None:
import baostock as bs
bs.login()
def fetch_members(start: str) -> dict[str, dict[str, list[str]]]:
path = os.path.join(BAR_DIR, f"members_{start}_{date.today()}.pkl")
if os.path.exists(path):
return pd.read_pickle(path)
members = _fetch_members(start)
pd.to_pickle(members, path)
return members
def _fetch_members(start: str) -> dict[str, dict[str, list[str]]]:
import baostock as bs
bs.login()
months = pd.date_range(start, date.today(), freq="MS")
members: dict[str, dict[str, list[str]]] = {"hs300": {}, "zz500": {}}
for d in months:
ds = d.strftime("%Y-%m-%d")
members["hs300"][ds] = [r[1] for r in _rows(bs.query_hs300_stocks(date=ds))]
members["zz500"][ds] = [r[1] for r in _rows(bs.query_zz500_stocks(date=ds))]
bs.logout()
return members
BAR_DIR = os.path.join("data", "research", "cn_bars")
def fetch_one(args: tuple[str, str, str]) -> pd.DataFrame | None:
"""Fetch one stock's bars, cached per code so an interrupted run resumes."""
code, start, end = args
path = os.path.join(BAR_DIR, f"{code}_{end}.pkl")
if os.path.exists(path):
return pd.read_pickle(path)
df = _fetch_one(code, start, end)
pd.to_pickle(df, path)
return df
def _fetch_one(code: str, start: str, end: str) -> pd.DataFrame | None:
import baostock as bs
for _ in range(3):
rs = bs.query_history_k_data_plus(code, K_FIELDS, start_date=start, end_date=end,
frequency="d", adjustflag="1")
if rs.error_code == "0":
rows = _rows(rs)
if not rows:
return None
df = pd.DataFrame(rows, columns=rs.fields)
df["date"] = pd.to_datetime(df["date"])
for c in NUMERIC:
df[c] = pd.to_numeric(df[c], errors="coerce")
return df
bs.logout()
bs.login()
print(f" failed {code}: {rs.error_msg}", flush=True)
return None
def fetch_index(start: str, end: str) -> pd.DataFrame:
import baostock as bs
bs.login()
out = {}
for code in ("sh.000300", "sh.000905"):
rs = bs.query_history_k_data_plus(code, "date,close", start_date=start, end_date=end,
frequency="d")
df = pd.DataFrame(_rows(rs), columns=rs.fields)
out[code] = pd.Series(pd.to_numeric(df["close"]).values, index=pd.to_datetime(df["date"]))
bs.logout()
return pd.DataFrame(out)
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--start", default="2007-01-01")
ap.add_argument("--member-start", default="2008-01-01")
ap.add_argument("--workers", type=int, default=4)
ap.add_argument("--hs300-only", action="store_true")
args = ap.parse_args()
end = date.today().strftime("%Y-%m-%d")
os.makedirs(BAR_DIR, exist_ok=True)
print("Fetching index membership...", flush=True)
members = fetch_members(args.member_start)
hs300 = sorted({c for lst in members["hs300"].values() for c in lst})
codes = list(hs300)
if not args.hs300_only:
codes += sorted({c for lst in members["zz500"].values() for c in lst} - set(hs300))
print(f" {len(codes)} codes ever in the index set", flush=True)
frames = []
with Pool(args.workers, initializer=_login) as pool:
for i, df in enumerate(pool.imap(fetch_one, [(c, args.start, end) for c in codes])):
if df is not None:
frames.append(df)
if (i + 1) % 100 == 0:
print(f" bars {i + 1}/{len(codes)}", flush=True)
bars = pd.concat(frames, ignore_index=True)
fields = {}
for c in NUMERIC:
fields[RENAME.get(c, c)] = bars.pivot(index="date", columns="code", values=c).sort_index()
panel = {"fields": fields, "members": members, "index": fetch_index(args.start, end)}
os.makedirs(os.path.dirname(OUT), exist_ok=True)
with open(OUT, "wb") as fh:
pickle.dump(panel, fh)
c = fields["close"]
print(f"Saved {OUT}: {c.shape[1]} codes, {c.index[0].date()} → {c.index[-1].date()}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,70 @@
"""
Fetch adjusted daily OHLCV for CSI 300 / CSI 500 ever-members from Yahoo
(through the market-data proxy). Delisted names Yahoo lacks stay missing
(listed in data/research/cn_missing.txt): Eastmoney and Tencent throttle bulk
requests, and Tencent's back-adjusted series damps returns (limit-down days
show -7% instead of -10%), so neither is used.
uv run python -m research.lab.fetch_cn_yahoo [--hs300-only]
Writes data/research/cn_ohlcv.pkl (fields open/high/low/close/volume/preclose/
pctchg/tradestatus + members + index + per-code "source").
"""
import argparse
import glob
import os
import pickle
import pandas as pd
from research.lab.fetch_us import _proxy_env, fetch
OUT = os.path.join("data", "research", "cn_ohlcv.pkl")
def to_yahoo(code: str) -> str:
ex, num = code.split(".")
return f"{num}.{'SS' if ex == 'sh' else 'SZ'}"
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--hs300-only", action="store_true")
ap.add_argument("--start", default="2007-01-01")
args = ap.parse_args()
member_file = sorted(glob.glob(os.path.join("data", "research", "cn_bars", "members_*.pkl")))[-1]
members = pd.read_pickle(member_file)
hs300 = sorted({c for lst in members["hs300"].values() for c in lst})
codes = list(hs300)
if not args.hs300_only:
codes += sorted({c for lst in members["zz500"].values() for c in lst} - set(hs300))
ymap = {to_yahoo(c): c for c in codes}
_proxy_env()
raw = fetch(list(ymap) + ["000300.SS", "000905.SS"], args.start)
fields = {k: raw[k].rename(columns=ymap) for k in ("open", "high", "low", "close", "volume")}
source = {c: "yahoo" for c in fields["close"].columns if c in codes}
idx = pd.DataFrame({"sh.000300": raw["close"].get("000300.SS"),
"sh.000905": raw["close"].get("000905.SS")})
keep = [c for c in fields["close"].columns if c in source]
fields = {k: v[keep].sort_index() for k, v in fields.items()}
# Yahoo prints flat bars (volume 0, O=H=L=C) on suspended days: treat as no trade
close = fields["close"]
traded = (fields["volume"] > 0) & close.notna()
fields["tradestatus"] = traded.astype(float)
fields["preclose"] = close.where(traded).ffill().shift(1)
fields["pctchg"] = (close / fields["preclose"] - 1) * 100
missing = sorted(set(codes) - set(source))
panel = {"fields": fields, "members": members, "index": idx, "source": source,
"missing": missing}
with open(OUT, "wb") as fh:
pickle.dump(panel, fh)
print(f"Saved {OUT}: {len(keep)} codes, {len(missing)} missing, "
f"{close.index[0].date()} → {close.index[-1].date()}")
with open(os.path.join("data", "research", "cn_missing.txt"), "w") as fh:
fh.write("\n".join(missing))
if __name__ == "__main__":
main()

91
research/lab/fetch_us.py Normal file
View File

@@ -0,0 +1,91 @@
"""
Fetch a long OHLCV history for every ticker that was ever an S&P 500 member.
The live panels in data/us.csv hold 10 years of close/open for tickers with
>50% coverage. Factor research needs more: volume and high/low, 20+ years of
history for an out-of-sample design split, and recently listed members (CEG,
GEV, ...) that the coverage rule drops. Delisted names that Yahoo no longer
serves stay missing; ``coverage_report`` in panel.py quantifies that gap.
Usage:
uv run python -m research.lab.fetch_us [--start 2004-01-01]
Writes data/research/us_ohlcv.pkl: {"open","high","low","close","volume"}
DataFrames (dates x tickers), split/dividend adjusted.
"""
import argparse
import os
import pickle
import time
import pandas as pd
import yfinance as yf
import universe_history as uh
OUT = os.path.join("data", "research", "us_ohlcv.pkl")
FIELDS = ["Open", "High", "Low", "Close", "Volume"]
# Benchmarks, cash proxies and sector ETFs used by overlays and attribution.
EXTRA = ["SPY", "RSP", "QQQ", "SHY", "BIL", "IEF", "TLT", "GLD",
"XLB", "XLE", "XLF", "XLI", "XLK", "XLP", "XLU", "XLV", "XLY"]
def _proxy_env() -> None:
proxy = os.environ.get("QUANT_MARKET_PROXY")
if not proxy and os.path.exists(".env"):
for line in open(".env"):
if line.startswith("QUANT_MARKET_PROXY="):
proxy = line.split("=", 1)[1].strip().strip("'\"")
if proxy:
for k in ("http_proxy", "https_proxy", "HTTP_PROXY", "HTTPS_PROXY"):
os.environ[k] = proxy
def fetch(tickers: list[str], start: str, chunk: int = 60) -> dict[str, pd.DataFrame]:
parts: dict[str, list[pd.DataFrame]] = {f: [] for f in FIELDS}
for i in range(0, len(tickers), chunk):
batch = tickers[i:i + chunk]
for attempt in range(4):
try:
raw = yf.download(batch, start=start, auto_adjust=True, progress=False,
group_by="column", threads=True)
break
except Exception as exc: # noqa: BLE001 — retry transient Yahoo errors
print(f" batch {i}: {exc}; retry {attempt + 1}")
time.sleep(10 * (attempt + 1))
else:
continue
for f in FIELDS:
if f in raw.columns.get_level_values(0):
parts[f].append(raw[f])
print(f" fetched {min(i + chunk, len(tickers))}/{len(tickers)}", flush=True)
time.sleep(2)
out = {}
for f in FIELDS:
if not parts[f]:
raise RuntimeError(f"no {f} data downloaded")
df = pd.concat(parts[f], axis=1)
df = df.loc[:, ~df.columns.duplicated()].sort_index()
out[f.lower()] = df.dropna(axis=1, how="all")
return out
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--start", default="2004-01-01")
args = ap.parse_args()
_proxy_env()
intervals = uh.load_sp500_history()
tickers = sorted(set(uh.all_tickers_ever(intervals)) | set(EXTRA))
print(f"Fetching {len(tickers)} tickers from {args.start}")
panel = fetch(tickers, args.start)
os.makedirs(os.path.dirname(OUT), exist_ok=True)
with open(OUT, "wb") as fh:
pickle.dump(panel, fh)
c = panel["close"]
print(f"Saved {OUT}: {c.shape[1]} tickers, {c.index[0].date()} → {c.index[-1].date()}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,43 @@
"""Fetch GICS-like sectors from Yahoo for S&P 500 ever-members missing from
data/us_sectors.csv. Writes data/research/us_sectors_extra.csv.
uv run python -m research.lab.fetch_us_sectors
"""
import os
import pickle
import pandas as pd
import yfinance as yf
from research.lab.fetch_us import _proxy_env
# Yahoo sector names → GICS sector names used in data/us_sectors.csv
YAHOO_TO_GICS = {
"Technology": "Information Technology", "Healthcare": "Health Care",
"Financial Services": "Financials", "Consumer Cyclical": "Consumer Discretionary",
"Consumer Defensive": "Consumer Staples", "Basic Materials": "Materials",
"Communication Services": "Communication Services", "Industrials": "Industrials",
"Energy": "Energy", "Utilities": "Utilities", "Real Estate": "Real Estate",
}
def main() -> None:
_proxy_env()
with open(os.path.join("data", "research", "us_ohlcv.pkl"), "rb") as fh:
tickers = list(pickle.load(fh)["close"].columns)
known = set(pd.read_csv(os.path.join("data", "us_sectors.csv"))["ticker"])
rows = []
for t in sorted(set(tickers) - known):
try:
s = yf.Ticker(t).info.get("sector")
except Exception: # noqa: BLE001
s = None
rows.append({"ticker": t, "sector": YAHOO_TO_GICS.get(s, s)})
out = pd.DataFrame(rows)
out.to_csv(os.path.join("data", "research", "us_sectors_extra.csv"), index=False)
print(out["sector"].value_counts(dropna=False))
if __name__ == "__main__":
main()

95
research/lab/ml.py Normal file
View File

@@ -0,0 +1,95 @@
"""
Walk-forward signal combination.
Features are cross-sectional percentile ranks of every factor among PIT
members on each date; the target is the cross-sectional rank of the forward
open-to-open return over ``horizon`` days. Each calendar year is predicted by
a model trained only on samples whose labels ended before that year starts
(a ``horizon``-day purge), so every score is out-of-sample.
from research.lab import ml
feats = ml.feature_ranks(panel, names)
score = ml.walk_forward(panel, feats, model="lgbm", horizon=20)
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from research.lab.factors import FACTORS
from research.lab.report import forward_returns
def feature_ranks(p, names: list[str]) -> dict[str, pd.DataFrame]:
out = {}
for n in names:
f = FACTORS[n](p).reindex(index=p.dates, columns=p.close.columns)
out[n] = f.where(p.member).rank(axis=1, pct=True)
return out
def _stack(feats: dict[str, pd.DataFrame], rows: pd.DatetimeIndex) -> pd.DataFrame:
cols = {n: f.loc[rows].stack(future_stack=True) for n, f in feats.items()}
return pd.DataFrame(cols)
def walk_forward(p, feats: dict[str, pd.DataFrame], model: str = "lgbm", horizon: int = 20,
sample_every: int = 5, first_test_year: int = 2010, min_train_years: int = 4,
params: dict | None = None, seed: int = 0) -> pd.DataFrame:
fwd = forward_returns(p, horizon).where(p.member)
target = fwd.rank(axis=1, pct=True)
dates = p.dates
sample_rows = dates[::sample_every]
X_all = _stack(feats, sample_rows)
y_all = target.loc[sample_rows].stack(future_stack=True).reindex(X_all.index)
ok = y_all.notna() & (X_all.notna().sum(axis=1) >= len(feats) // 2)
X_all, y_all = X_all[ok], y_all[ok]
sample_dates = X_all.index.get_level_values(0)
score = pd.DataFrame(np.nan, index=dates, columns=p.close.columns)
years = sorted({d.year for d in dates if d.year >= first_test_year})
for yr in years:
test_start = pd.Timestamp(f"{yr}-01-01")
pos = dates.searchsorted(test_start)
if pos >= len(dates):
break
# purge: the label of a sample at d uses opens through d + horizon + 1
cutoff = dates[max(pos - horizon - 2, 0)]
tr = sample_dates <= cutoff
if (sample_dates[tr].max() - sample_dates[tr].min()).days < 365 * min_train_years:
continue
Xtr, ytr = X_all[tr], y_all[tr]
test_rows = dates[(dates >= test_start) & (dates < pd.Timestamp(f"{yr + 1}-01-01"))]
Xte = _stack(feats, test_rows)
pred = _fit_predict(model, Xtr, ytr, Xte, params, seed)
s = pd.Series(pred, index=Xte.index).unstack()
score.loc[s.index, s.columns] = s.values
return score
def _fit_predict(model, Xtr, ytr, Xte, params, seed):
if model == "lgbm":
import lightgbm as lgb
prm = dict(objective="regression", learning_rate=0.03, num_leaves=31,
min_data_in_leaf=1000, bagging_fraction=0.7, bagging_freq=1,
feature_fraction=0.7, lambda_l2=1.0, seed=seed, verbose=-1, num_threads=8)
prm.update(params or {})
rounds = prm.pop("n_estimators", 300)
booster = lgb.train(prm, lgb.Dataset(Xtr.values, ytr.values), num_boost_round=rounds)
return booster.predict(Xte.values)
if model == "ridge":
Xf = Xtr.fillna(0.5).values - 0.5
lam = (params or {}).get("lam", 10.0) * len(Xf)
beta = np.linalg.solve(Xf.T @ Xf + lam * np.eye(Xf.shape[1]), Xf.T @ (ytr.values - 0.5))
return (Xte.fillna(0.5).values - 0.5) @ beta
if model == "ic_weight":
# sign-and-size each feature by its training-period mean rank IC
d = Xtr.index.get_level_values(0)
ics = {}
for c in Xtr.columns:
df = pd.DataFrame({"x": Xtr[c], "y": ytr, "d": d}).dropna()
ics[c] = df.groupby("d").apply(lambda g: g["x"].corr(g["y"])).mean()
w = pd.Series(ics).fillna(0.0)
return (Xte.fillna(0.5) - 0.5).values @ w.values
raise ValueError(model)

55
research/lab/overlays.py Normal file
View File

@@ -0,0 +1,55 @@
"""
Exposure overlays: series in [0, 1] read on signal dates by engine.simulate
(``exposure=``). The un-invested part earns the panel's cash return.
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from research.lab.engine import simulate
def ew_index(p) -> pd.Series:
"""Equal-weight index of PIT members (daily rebalanced)."""
r = p.close.pct_change(fill_method=None).where(p.member.shift(1, fill_value=False))
return (1 + r.mean(axis=1).fillna(0)).cumprod()
def trend(level: pd.Series, ma: int = 200, floor: float = 0.0) -> pd.Series:
"""1 when the index is above its moving average, else ``floor``."""
above = level > level.rolling(ma, min_periods=ma).mean()
return above.astype(float).where(above, floor)
def breadth(p, ma: int = 200, lo: float = 0.4, hi: float = 0.6) -> pd.Series:
"""Share of members above their own MA mapped linearly from lo→0 to hi→1."""
c = p.close.ffill()
above = (c > c.rolling(ma, min_periods=ma).mean()).where(p.member)
share = above.mean(axis=1)
return ((share - lo) / (hi - lo)).clip(0, 1)
def drawdown_brake(level: pd.Series, dd: float = 0.15, ma: int = 50) -> pd.Series:
"""Cut to zero when the index is > dd below its 1y high AND below its MA."""
high = level.rolling(252, min_periods=60).max()
bad = (level / high - 1 < -dd) & (level < level.rolling(ma, min_periods=ma).mean())
return (~bad).astype(float)
def vol_target(p, score, target: float, lookback: int = 60, cap: float = 1.0,
base_kwargs: dict | None = None, exposure: pd.Series | None = None) -> pd.Series:
"""Scale exposure by target / realized vol of the unscaled strategy.
Two passes: run the strategy fully invested, measure its trailing
daily-return vol, then size the next rebalance; any given ``exposure``
(e.g. a trend filter) multiplies the result.
"""
kw = dict(base_kwargs or {})
base = simulate(p, score, **kw)
rv = base.returns.rolling(lookback, min_periods=lookback // 2).std() * np.sqrt(252)
scale = (target / rv).clip(upper=cap).fillna(cap)
if exposure is not None:
scale = scale * exposure.reindex(scale.index).ffill().fillna(1.0)
return scale

196
research/lab/panel.py Normal file
View File

@@ -0,0 +1,196 @@
"""
Build research Panels (see engine.Panel) from the fetched raw data.
from research.lab import panel
us = panel.load_us() # S&P 500 PIT, 2004 →
cn = panel.load_cn("hs300") # CSI 300 PIT, 2007 → ("zz800" = CSI 300 + 500)
Panels are cached in data/research/*.panel.pkl; delete them after refetching.
"""
from __future__ import annotations
import json
import os
import pickle
import numpy as np
import pandas as pd
from research.lab.engine import Costs, Panel
RAW_DIR = os.path.join("data", "research")
US_ETFS = {"SPY", "RSP", "QQQ", "SHY", "BIL", "IEF", "TLT", "GLD",
"XLB", "XLE", "XLF", "XLI", "XLK", "XLP", "XLU", "XLV", "XLY"}
def _cached(name: str, build):
path = os.path.join(RAW_DIR, f"{name}.panel.pkl")
if os.path.exists(path):
with open(path, "rb") as fh:
return pickle.load(fh)
p = build()
with open(path, "wb") as fh:
pickle.dump(p, fh)
return p
# ---------------------------------------------------------------------------
# US
# ---------------------------------------------------------------------------
def _us_membership(dates: pd.DatetimeIndex, tickers: list[str]) -> pd.DataFrame:
with open(os.path.join("data", "sp500_history.json")) as fh:
intervals = json.load(fh)["intervals"]
mask = pd.DataFrame(False, index=dates, columns=tickers)
for t in tickers:
for start, end in intervals.get(t, []):
s = pd.Timestamp(start) if start else dates[0]
e = pd.Timestamp(end) if end else dates[-1] + pd.Timedelta(days=1)
mask.loc[(dates >= s) & (dates < e), t] = True
return mask
def _build_us() -> Panel:
with open(os.path.join(RAW_DIR, "us_ohlcv.pkl"), "rb") as fh:
raw = pickle.load(fh)
close = raw["close"]
# keep rows where most listed tickers printed (drops stray holiday rows)
alive = close.notna().sum(axis=1)
close = close[alive >= 0.5 * alive.rolling(20, min_periods=1).max()]
dates = close.index
etf = {k: raw[k].reindex(dates)[[c for c in raw[k].columns if c in US_ETFS]]
for k in ("close",)}["close"]
stocks = sorted(c for c in close.columns if c not in US_ETFS)
f = {k: raw[k].reindex(index=dates, columns=stocks) for k in ("open", "high", "low", "close", "volume")}
# Yahoo sometimes prints zero/negative adjusted opens for old bars
for k in ("open", "high", "low"):
f[k] = f[k].where(f[k] > 0)
has_px = f["close"].notna()
member = _us_membership(dates, stocks) & has_px
cash_px = etf["BIL"].combine_first(etf["SHY"]) if "BIL" in etf else etf["SHY"]
bil = etf["BIL"].pct_change(fill_method=None)
shy = etf["SHY"].pct_change(fill_method=None)
cash_ret = bil.where(bil.notna(), shy).fillna(0.0)
del cash_px
return Panel(
market="us", open=f["open"], high=f["high"], low=f["low"], close=f["close"],
volume=f["volume"], member=member, can_buy=has_px, can_sell=has_px,
cash_ret=cash_ret, benchmark=etf["SPY"],
costs=Costs(buy=0.001, sell=0.001, fixed_fee=0.0, capital=10_000.0),
extra={"etf": etf, "sector": _us_sectors(stocks)},
)
def _us_sectors(stocks: list[str]) -> pd.Series:
"""GICS sector per ticker; current members from data/us_sectors.csv, the
rest from data/research/us_sectors_extra.csv when it has been fetched."""
sec = pd.read_csv(os.path.join("data", "us_sectors.csv")).set_index("ticker")["GICS Sector"]
extra = os.path.join(RAW_DIR, "us_sectors_extra.csv")
if os.path.exists(extra):
sec = sec.combine_first(pd.read_csv(extra).set_index("ticker")["sector"])
return sec.reindex(stocks)
def load_us() -> Panel:
return _cached("us", _build_us)
# ---------------------------------------------------------------------------
# CN
# ---------------------------------------------------------------------------
def _limit_pct(codes: list[str], dates: pd.DatetimeIndex, isst: pd.DataFrame) -> pd.DataFrame:
"""Daily price-limit fraction per stock (10% main board, 20% ChiNext after
2020-08-24 and STAR, 5% ST)."""
lim = pd.DataFrame(0.10, index=dates, columns=codes)
star = [c for c in codes if c.split(".")[1].startswith("688")]
chinext = [c for c in codes if c.split(".")[1].startswith("30")]
lim[star] = 0.20
lim.loc[dates >= "2020-08-24", chinext] = 0.20
st = isst.reindex(index=dates, columns=codes).fillna(0).astype(bool)
lim = lim.mask(st & (lim < 0.2), 0.05)
return lim
def _cn_membership(members: dict[str, list[str]], dates: pd.DatetimeIndex,
codes: list[str]) -> pd.DataFrame:
snap = sorted(members.items())
mask = pd.DataFrame(False, index=dates, columns=codes)
col = {c: i for i, c in enumerate(codes)}
arr = mask.values
for i, (d, lst) in enumerate(snap):
s = pd.Timestamp(d)
e = pd.Timestamp(snap[i + 1][0]) if i + 1 < len(snap) else dates[-1] + pd.Timedelta(days=1)
rows = (dates >= s) & (dates < e)
cols = [col[c] for c in lst if c in col]
arr[np.ix_(rows, cols)] = True
return pd.DataFrame(arr, index=dates, columns=codes)
def _build_cn(index_set: str) -> Panel:
with open(os.path.join(RAW_DIR, "cn_ohlcv.pkl"), "rb") as fh:
raw = pickle.load(fh)
F = raw["fields"]
# Codes an earlier fetcher filled from Tencent's 后复权 series are dropped:
# it damps returns for long dividend histories (limit-down shows -7%).
bad = {c for c, src in raw.get("source", {}).items() if src == "tencent"}
close = F["close"]
dates = close.index
codes = [c for c in close.columns if c not in bad]
f = {k: v.reindex(index=dates, columns=codes) for k, v in F.items()}
# Yahoo bars lack ST flags and valuation ratios; index members
# are rarely ST (the index drops them at the next review).
if "isst" not in f:
f["isst"] = pd.DataFrame(0.0, index=dates, columns=codes)
traded = (f["tradestatus"] == 1) & (f["volume"] > 0) & f["close"].notna()
if index_set == "hs300":
member = _cn_membership(raw["members"]["hs300"], dates, codes)
elif index_set == "zz500":
member = _cn_membership(raw["members"]["zz500"], dates, codes)
else: # zz800
member = (_cn_membership(raw["members"]["hs300"], dates, codes)
| _cn_membership(raw["members"]["zz500"], dates, codes))
st = f["isst"].fillna(0).astype(bool)
member = member & traded.rolling(5, min_periods=1).max().astype(bool) & ~st
lim = _limit_pct(codes, dates, f["isst"])
open_ret = f["open"] / f["preclose"] - 1
can_buy = traded & ~(open_ret >= lim - 0.002)
can_sell = traded & ~(open_ret <= -lim + 0.002)
# halted days carry the last price, so returns across a halt are zero
for k in ("open", "high", "low", "close"):
f[k] = f[k].where(traded)
stamp = pd.Series(0.001, index=dates)
stamp[dates >= "2023-08-28"] = 0.0005
idx = raw["index"].reindex(dates).ffill()
return Panel(
market=f"cn_{index_set}", open=f["open"], high=f["high"], low=f["low"], close=f["close"],
volume=f["volume"].where(traded), member=member, can_buy=can_buy, can_sell=can_sell,
cash_ret=pd.Series(0.02 / 252, index=dates), benchmark=idx["sh.000300"],
costs=Costs(buy=0.0008, sell=0.0008, sell_tax=stamp, fixed_fee=0.0, capital=70_000.0),
extra=_cn_extra(f, traded, lim, idx),
)
def _cn_extra(f: dict, traded: pd.DataFrame, lim: pd.DataFrame, idx: pd.DataFrame) -> dict:
extra = {"limit": lim, "index": idx, "source": "baostock" if "pe" in f else "yahoo"}
if "turn" in f:
extra["turn"] = f["turn"].where(traded)
if "amount" in f:
extra["amount"] = f["amount"].where(traded)
if "turn" in f:
extra["float_cap"] = (f["amount"] / (f["turn"] / 100)).where(traded & (f["turn"] > 0))
for k in ("pe", "pb", "ps", "pcf"):
if k in f:
extra[k] = f[k]
return extra
def load_cn(index_set: str = "hs300") -> Panel:
return _cached(f"cn_{index_set}", lambda: _build_cn(index_set))
def coverage_report(p: Panel) -> pd.DataFrame:
"""Per year: PIT members with price data vs expected index size."""
m = p.member.resample("YE").last()
return pd.DataFrame({"members_with_data": m.sum(axis=1)})

118
research/lab/report.py Normal file
View File

@@ -0,0 +1,118 @@
"""Performance statistics, factor IC analysis and overfitting controls."""
from __future__ import annotations
import math
import numpy as np
import pandas as pd
from scipy import stats as sps
import metrics
def stats(equity: pd.Series, turnover: pd.Series | None = None) -> dict:
eq = equity.dropna()
r = eq.pct_change().dropna()
out = {
"cagr": metrics.annualized_return(eq),
"vol": metrics.annualized_volatility(r),
"sharpe": metrics.sharpe_ratio(r),
"sortino": metrics.sortino_ratio(r),
"mdd": metrics.max_drawdown(eq),
}
out["calmar"] = out["cagr"] / abs(out["mdd"]) if out["mdd"] < 0 else np.inf
if turnover is not None:
t = turnover.loc[eq.index[0]:eq.index[-1]]
years = max((eq.index[-1] - eq.index[0]).days / 365.25, 1e-9)
out["turnover"] = t.sum() / years # one-way, x per year
return out
def fmt(s: dict) -> str:
base = (f"CAGR {s['cagr']*100:6.1f}% vol {s['vol']*100:5.1f}% Sharpe {s['sharpe']:5.2f} "
f"Sortino {s['sortino']:5.2f} MDD {s['mdd']*100:6.1f}% Calmar {s['calmar']:5.2f}")
if "turnover" in s:
base += f" turn {s['turnover']:5.1f}x"
return base
def yearly(equity: pd.Series) -> pd.Series:
e = equity.dropna()
ye = e.resample("YE").last()
first = e.iloc[0]
return ye.pct_change().fillna(ye.iloc[0] / first - 1)
# ---------------------------------------------------------------------------
# Factor analytics
# ---------------------------------------------------------------------------
def forward_returns(panel, horizon: int = 5) -> pd.DataFrame:
"""Return from the open of t+1 to the open of t+1+horizon — the return a
signal formed at the close of t can actually earn."""
o = panel.open.where(panel.open > 0).ffill()
return o.shift(-(1 + horizon)) / o.shift(-1) - 1
def ic_series(score: pd.DataFrame, fwd: pd.DataFrame, member: pd.DataFrame,
every: int = 5) -> pd.Series:
"""Cross-sectional Spearman rank IC on every ``every``-th date."""
idx = score.index[::every]
out = {}
for d in idx:
s = score.loc[d].where(member.loc[d])
f = fwd.loc[d]
ok = s.notna() & f.notna()
if ok.sum() < 30:
continue
out[d] = sps.spearmanr(s[ok], f[ok]).statistic
return pd.Series(out)
def top_minus_universe(score: pd.DataFrame, fwd: pd.DataFrame, member: pd.DataFrame,
top_n: int = 20, every: int = 5) -> pd.Series:
"""Mean forward return of the top-N names minus the member average."""
idx = score.index[::every]
out = {}
for d in idx:
s = score.loc[d].where(member.loc[d]).dropna()
f = fwd.loc[d]
s = s[f.reindex(s.index).notna()]
if len(s) < top_n * 3:
continue
top = s.nlargest(top_n).index
out[d] = f[top].mean() - f[s.index].mean()
return pd.Series(out)
def ic_summary(ic: pd.Series, periods_per_year: float = 52) -> dict:
ic = ic.dropna()
if len(ic) < 5:
return {"ic": np.nan, "icir": np.nan, "t": np.nan, "hit": np.nan, "n": len(ic)}
return {"ic": ic.mean(), "icir": ic.mean() / ic.std() * math.sqrt(periods_per_year),
"t": ic.mean() / ic.std() * math.sqrt(len(ic)), "hit": (ic > 0).mean(), "n": len(ic)}
# ---------------------------------------------------------------------------
# Overfitting controls (Bailey & Lopez de Prado 2014)
# ---------------------------------------------------------------------------
def deflated_sharpe(returns: pd.Series, n_trials: int, sr_trials_var: float | None = None,
periods: int = 252) -> dict:
"""Probability that the true Sharpe exceeds the best Sharpe expected from
``n_trials`` unskilled strategies. Uses per-period Sharpe internally."""
r = returns.dropna()
T = len(r)
sr = r.mean() / r.std()
g3 = sps.skew(r)
g4 = sps.kurtosis(r, fisher=False)
if sr_trials_var is None:
sr_trials_var = 1.0 / T # variance of SR estimates under H0
emc = 0.5772156649
z1 = sps.norm.ppf(1 - 1.0 / n_trials) if n_trials > 1 else 0.0
z2 = sps.norm.ppf(1 - 1.0 / (n_trials * math.e)) if n_trials > 1 else 0.0
sr0 = math.sqrt(sr_trials_var) * ((1 - emc) * z1 + emc * z2)
denom = math.sqrt(max(1 - g3 * sr + (g4 - 1) / 4 * sr ** 2, 1e-12))
dsr = sps.norm.cdf((sr - sr0) * math.sqrt(T - 1) / denom)
return {"sr_annual": sr * math.sqrt(periods), "sr0_annual": sr0 * math.sqrt(periods), "dsr": dsr}

52
research/lab/run_ml.py Normal file
View File

@@ -0,0 +1,52 @@
"""
Walk-forward ML combination experiment.
PYTHONPATH=. uv run --with lightgbm python -m research.lab.run_ml --market us
"""
import argparse
import os
import pickle
import time
import warnings
from research.lab import ml
from research.lab.factor_scan import load, phase_avg
from research.lab.factors import FACTORS
from research.lab.report import fmt
warnings.filterwarnings("ignore")
OUT = os.path.join("data", "research", "results")
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--market", default="us")
ap.add_argument("--models", nargs="*", default=["lgbm", "ridge", "ic_weight"])
ap.add_argument("--horizons", nargs="*", type=int, default=[5, 20])
ap.add_argument("--first-test-year", type=int, default=2010)
args = ap.parse_args()
mk = "us" if args.market == "us" else "cn"
p, periods = load(args.market)
names = [n for n, f in FACTORS.items() if mk in f.markets]
t = time.time()
feats = ml.feature_ranks(p, names)
print(f"{len(names)} features in {time.time() - t:.0f}s", flush=True)
evalp = {"design": (f"{args.first_test_year}-01-01", periods["design"][1]), "holdout": periods["holdout"]}
os.makedirs(OUT, exist_ok=True)
for model in args.models:
for h in args.horizons:
t = time.time()
score = ml.walk_forward(p, feats, model=model, horizon=h, first_test_year=args.first_test_year)
with open(os.path.join(OUT, f"ml_{args.market}_{model}_h{h}.pkl"), "wb") as fh:
pickle.dump(score, fh)
for top_n in (10, 20):
parts = []
for per, (a, b) in evalp.items():
parts.append(f"{per}: {fmt(phase_avg(p, score, a, b, top_n, 5))}")
print(f"{model:9s} h{h:<3d} top{top_n:<3d}", " | ".join(parts), f"[{time.time() - t:.0f}s]", flush=True)
if __name__ == "__main__":
main()

116
research/lab/stepwise.py Normal file
View File

@@ -0,0 +1,116 @@
"""
Forward stepwise selection of equal-weight rank composites, scored on the
DESIGN period only; the holdout is evaluated once for the final picks.
PYTHONPATH=. uv run python -m research.lab.stepwise --market us --max-k 4
Every evaluated composite is appended to the trial ledger
(data/research/results/trials_{market}.csv) so deflated Sharpe can count them.
"""
import argparse
import os
import time
import warnings
import numpy as np
import pandas as pd
from research.lab.factor_scan import RESULTS, load, phase_avg
from research.lab.factors import FACTORS, zrank
warnings.filterwarnings("ignore")
# Four ~5-year blocks; "maximin" scores a composite by its worst block Sharpe.
SUBPERIODS = {
"us": [("2006-09-27", "2011-09-26"), ("2011-09-27", "2016-09-26"),
("2016-09-27", "2021-09-26"), ("2021-09-27", "2026-09-24")],
"cn": [("2010-01-01", "2013-06-30"), ("2013-07-01", "2016-09-26"),
("2016-09-27", "2021-09-26"), ("2021-09-27", "2026-09-24")],
}
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--market", default="us")
ap.add_argument("--max-k", type=int, default=4)
ap.add_argument("--top-n", type=int, default=20)
ap.add_argument("--every", type=int, default=5)
ap.add_argument("--beam", type=int, default=3)
ap.add_argument("--objective", default="sharpe")
ap.add_argument("--exclude", nargs="*", default=[])
ap.add_argument("--buffer", type=float, default=0.0)
args = ap.parse_args()
mk = "us" if args.market == "us" else "cn"
p, periods = load(args.market)
names = [n for n, f in FACTORS.items() if mk in f.markets and n not in args.exclude]
t = time.time()
ranks = {}
for n in names:
try:
ranks[n] = zrank(FACTORS[n](p), p.member)
except Exception as exc: # noqa: BLE001
print(f"skip {n}: {exc}")
names = list(ranks)
print(f"{len(names)} factor ranks in {time.time() - t:.0f}s", flush=True)
a, b = periods["design"]
ledger = []
subs = SUBPERIODS[mk]
def evaluate(combo: tuple[str, ...]) -> dict:
s = sum(ranks[n] for n in combo) / len(combo)
if args.objective == "maximin":
per = [phase_avg(p, s, sa, sb, args.top_n, args.every, buffer=args.buffer) for sa, sb in subs]
st = {k: float(np.mean([x[k] for x in per])) for k in per[0]}
st["maximin"] = min(x["sharpe"] for x in per)
st["sub_sharpes"] = " ".join(f"{x['sharpe']:.2f}" for x in per)
st["sub_cagrs"] = " ".join(f"{x['cagr']*100:.0f}" for x in per)
else:
st = phase_avg(p, s, a, b, args.top_n, args.every, buffer=args.buffer)
ledger.append({"combo": "+".join(combo), "objective": args.objective, **st})
return st
beam = [()]
best_overall = []
for k in range(1, args.max_k + 1):
cands = {}
for base in beam:
for n in names:
if n in base:
continue
combo = tuple(sorted(base + (n,)))
if combo in cands:
continue
cands[combo] = evaluate(combo)
ranked = sorted(cands.items(), key=lambda kv: -kv[1][args.objective])
beam = [c for c, _ in ranked[:args.beam]]
print(f"--- k={k}: {len(cands)} combos", flush=True)
for c, st in ranked[:6]:
extra = (f" min {st['maximin']:.2f} subSh [{st['sub_sharpes']}] subCAGR [{st['sub_cagrs']}]"
if "maximin" in st else "")
print(f" {'+'.join(c):55s} sh {st['sharpe']:.2f} cagr {st['cagr']*100:5.1f}% "
f"mdd {st['mdd']*100:5.1f}% turn {st['turnover']:.0f}x{extra}", flush=True)
best_overall += ranked[:args.beam]
if args.objective == "maximin":
_save(ledger, args.market)
return
# holdout, once, for the beam winners of each k
print("=== holdout of design winners ===")
ha, hb = periods["holdout"]
for c, st in best_overall:
s = sum(ranks[n] for n in c) / len(c)
hs = phase_avg(p, s, ha, hb, args.top_n, args.every)
print(f" {'+'.join(c):55s} D sh {st['sharpe']:.2f} cagr {st['cagr']*100:5.1f}% | "
f"H sh {hs['sharpe']:.2f} cagr {hs['cagr']*100:5.1f}% mdd {hs['mdd']*100:5.1f}%", flush=True)
_save(ledger, args.market)
def _save(ledger: list[dict], market: str) -> None:
os.makedirs(RESULTS, exist_ok=True)
path = os.path.join(RESULTS, f"trials_{market}.csv")
pd.DataFrame(ledger).to_csv(path, mode="a", header=not os.path.exists(path), index=False)
if __name__ == "__main__":
main()

View File

@@ -1,5 +1,8 @@
from abc import ABC, abstractmethod
import pandas as pd
class Strategy(ABC):
"""
Abstract base class for all strategies.
@@ -15,3 +18,32 @@ class Strategy(ABC):
containing the trading signals (e.g., 'buy', 'hold', 'sell').
"""
pass
class BarStrategy(Strategy):
"""Strategy on completed daily bars with several fields.
``fields`` names the panels it reads among close/open/high/low/volume.
``generate_signals(bars)`` takes ``{field: DataFrame}`` (a bare close
DataFrame is accepted when only "close" is needed) and returns weights
whose row t uses bars through t-1, like every Strategy. Backtests may add
``bars["eligible"]`` (bool, point-in-time index membership): prices stay
unmasked so signals see full histories, and only eligible names are held.
Timing differs from close-price strategies in the live trader: those read
the session's open as their latest price, while a BarStrategy only uses
completed bars, so the weights of row t trade at the close of t.
"""
fields: tuple[str, ...] = ("close",)
def generate_signals(self, data):
bars = data if isinstance(data, dict) else {"close": data}
missing = [f for f in self.fields if f not in bars]
if missing:
raise ValueError(f"{type(self).__name__} needs bar fields {missing}")
return self.generate_weights(bars)
@abstractmethod
def generate_weights(self, bars: dict[str, pd.DataFrame]) -> pd.DataFrame:
"""Weights (dates x assets), row t using bars through t-1."""

View File

@@ -0,0 +1,194 @@
"""
Robust momentum strategies from the 2026-09 point-in-time factor research
(``research/lab``). They were selected by their worst 5-year Sharpe over
2006/2010-2026 on PIT index members with next-open execution, not by their
best window.
Each score averages cross-sectional percentile ranks of its factors. Holdings
are the top ``top_n`` names, re-selected every ``rebal_freq`` rows; a held
name stays while its rank is within ``top_n * (1 + buffer)`` (hysteresis cuts
turnover without giving up the signal). Positions are equal-weighted on
rebalance rows and drift with prices in between, so winners are not trimmed
daily (worth ~3-4 pp a year for overnight_recovery against daily re-weighting).
Signals:
overnight_recovery (US) overnight-return momentum (12m, skip 1m)
+ distance from the 126-day low
lowrisk_momentum (CN) amplitude-cut momentum + low beta
+ negative price-volume correlation + momentum/vol
lowbeta_sharpe_mom (CN) low beta + momentum/vol (close prices only)
"""
import numpy as np
import pandas as pd
from numpy.lib.stride_tricks import sliding_window_view
from strategies.base import BarStrategy
# ---------------------------------------------------------------------------
# Factors: higher = more attractive; row t uses data through t
# ---------------------------------------------------------------------------
def _returns(close: pd.DataFrame) -> pd.DataFrame:
return close.pct_change(fill_method=None)
def overnight_momentum(open_: pd.DataFrame, close: pd.DataFrame,
window: int = 252, skip: int = 21) -> pd.DataFrame:
"""Sum of log overnight returns (open / previous close), Lou-Polk-Skouras."""
overnight = np.log(open_ / close.ffill().shift(1))
return overnight.rolling(window, min_periods=180).sum().shift(skip)
def recovery(close: pd.DataFrame, window: int = 126) -> pd.DataFrame:
c = close.ffill()
return c / c.rolling(window, min_periods=window).min() - 1
def momentum_12_1(close: pd.DataFrame) -> pd.DataFrame:
c = close.ffill()
return c.shift(21) / c.shift(252) - 1
def sharpe_momentum(close: pd.DataFrame) -> pd.DataFrame:
"""12-1 month return divided by 12-month daily volatility."""
return momentum_12_1(close) / _returns(close).rolling(252, min_periods=180).std()
def low_beta(close: pd.DataFrame, window: int = 252) -> pd.DataFrame:
"""Negative rolling beta to the equal-weight average of the panel."""
r = _returns(close)
m = r.mean(axis=1).fillna(0.0)
minp = window * 180 // 252
e_xy = r.mul(m, axis=0).rolling(window, min_periods=minp).mean()
e_x = r.rolling(window, min_periods=minp).mean()
e_y = m.rolling(window, min_periods=minp).mean()
var_y = (m ** 2).rolling(window, min_periods=minp).mean() - e_y ** 2
return -(e_xy - e_x.mul(e_y, axis=0)).div(var_y, axis=0)
def price_volume_corr(close: pd.DataFrame, volume: pd.DataFrame) -> pd.DataFrame:
"""Negative 21-day correlation of price and dollar volume."""
return -close.ffill().rolling(21, min_periods=15).corr(close * volume)
def amplitude_cut_momentum(high: pd.DataFrame, low: pd.DataFrame, close: pd.DataFrame,
window: int = 160, keep: float = 0.7) -> pd.DataFrame:
"""Sum of daily returns over the ``keep`` share of lowest-amplitude
(high/low - 1) days in the window (开源证券, 2020): calm up-moves persist,
noisy ones reverse."""
amp = (high / low - 1).to_numpy(dtype=float)
r = _returns(close).to_numpy(dtype=float)
out = np.full(r.shape, np.nan)
if len(r) >= window:
A = sliding_window_view(np.nan_to_num(amp, nan=np.inf), window, axis=0)
R = sliding_window_view(np.nan_to_num(r, nan=0.0), window, axis=0)
for t0 in range(0, A.shape[0], 256): # chunks bound memory
a, rr = A[t0:t0 + 256], R[t0:t0 + 256]
thr = np.nanquantile(np.where(np.isinf(a), np.nan, a), keep, axis=2, keepdims=True)
out[t0 + window - 1:t0 + window - 1 + a.shape[0]] = np.where(a <= thr, rr, 0.0).sum(axis=2)
valid = pd.DataFrame(amp, index=close.index).notna().rolling(window).sum() >= 0.8 * window
return pd.DataFrame(out, index=close.index, columns=close.columns).where(valid.to_numpy())
def _rank(df: pd.DataFrame) -> pd.DataFrame:
return df.rank(axis=1, pct=True)
def _composite(*factors: pd.DataFrame) -> pd.DataFrame:
return sum(_rank(f) for f in factors) / len(factors)
def signal_overnight_recovery(bars):
return _composite(overnight_momentum(bars["open"], bars["close"]), recovery(bars["close"]))
def signal_lowrisk_momentum(bars):
c = bars["close"]
return _composite(amplitude_cut_momentum(bars["high"], bars["low"], c), low_beta(c),
price_volume_corr(c, bars["volume"]), sharpe_momentum(c))
def signal_lowbeta_sharpe_mom(bars):
c = bars["close"]
return _composite(low_beta(c), sharpe_momentum(c))
# name -> (bar fields, score function)
SIGNALS = {
"overnight_recovery": (("close", "open"), signal_overnight_recovery),
"lowrisk_momentum": (("close", "high", "low", "volume"), signal_lowrisk_momentum),
"lowbeta_sharpe_mom": (("close",), signal_lowbeta_sharpe_mom),
}
# ---------------------------------------------------------------------------
# Selection
# ---------------------------------------------------------------------------
def select_with_buffer(score: pd.DataFrame, top_n: int, rebal_freq: int,
buffer: float, warmup: int,
prices: pd.DataFrame | None = None) -> pd.DataFrame:
"""Equal weights on rebalance rows (every ``rebal_freq`` from ``warmup``);
in between they drift with ``prices`` (row k uses prices through k), or
stay constant when ``prices`` is None. Not yet shifted."""
S = score.to_numpy(dtype=float)
n, m = S.shape
W = np.full((n, m), np.nan)
held = np.zeros(m, dtype=bool)
for r in range(warmup, n, rebal_freq):
row = S[r]
valid = ~np.isnan(row)
w = np.zeros(m)
if valid.sum() >= top_n:
order = np.where(valid)[0][np.argsort(-row[valid], kind="stable")]
rank = np.full(m, np.inf)
rank[order] = np.arange(len(order))
keep = [i for i in np.where(held)[0] if rank[i] < top_n * (1 + buffer)]
chosen = sorted(keep, key=lambda i: rank[i])[:top_n]
taken = set(chosen)
for i in order:
if len(chosen) >= top_n:
break
if i not in taken:
chosen.append(i)
w[chosen] = 1.0 / len(chosen)
W[r] = w
held = w > 0
weights = pd.DataFrame(W, index=score.index, columns=score.columns).ffill().fillna(0.0)
weights.iloc[:warmup] = 0.0
if prices is not None:
px = prices.reindex_like(weights).ffill()
anchor = px.where(pd.Series(~np.isnan(W).all(axis=1), index=weights.index), axis=0).ffill()
grown = weights * (px / anchor).fillna(1.0)
weights = grown.div(grown.sum(axis=1).replace(0.0, np.nan), axis=0).fillna(0.0)
return weights
class RobustMomentumStrategy(BarStrategy):
def __init__(self, signal: str, top_n: int = 10, rebal_freq: int = 10,
buffer: float = 2.0, warmup: int = 252):
if signal not in SIGNALS:
raise ValueError(f"Unknown signal {signal!r}; available: {list(SIGNALS)}")
self.signal = signal
self.fields, self._score = SIGNALS[signal]
self.top_n = top_n
self.rebal_freq = rebal_freq
self.buffer = buffer
self.warmup = warmup
def generate_weights(self, bars: dict[str, pd.DataFrame]) -> pd.DataFrame:
close = bars["close"]
panel = {f: bars[f].reindex(index=close.index, columns=close.columns) for f in self.fields}
score = self._score(panel)
# names without a price are not eligible; a backtest also passes
# "eligible" (point-in-time index membership) so signals can use a
# stock's full price history while only members are picked
eligible = close.notna()
if "eligible" in bars:
eligible &= bars["eligible"].reindex_like(close).fillna(False).astype(bool)
score = score.where(eligible)
weights = select_with_buffer(score, self.top_n, self.rebal_freq, self.buffer, self.warmup,
prices=close)
return weights.shift(1).fillna(0.0)

View File

@@ -0,0 +1,187 @@
"""Contracts added with the robust-momentum strategies: bar strategies are
causal and trade on the live trader's timing, whole-lot orders track targets,
panel backfills retry, and point-in-time CSI 300 membership."""
import unittest
from unittest import mock
import numpy as np
import pandas as pd
import data_manager
import trader
import universe_history as uh
from main import backtest, simulate_live_book, strategy_positions
from strategies.robust_momentum import SIGNALS, RobustMomentumStrategy, select_with_buffer
def _bars(n_days: int = 420, n_assets: int = 30, seed: int = 0) -> dict[str, pd.DataFrame]:
rng = np.random.default_rng(seed)
idx = pd.bdate_range("2024-01-01", periods=n_days)
cols = [f"S{i:02d}" for i in range(n_assets)]
r = rng.normal(0.0004, 0.02, (n_days, n_assets))
close = pd.DataFrame(100 * np.exp(np.cumsum(r, axis=0)), index=idx, columns=cols)
gap = rng.normal(0, 0.005, (n_days, n_assets))
opens = close.shift(1).bfill() * np.exp(gap)
high = np.maximum(opens, close) * (1 + rng.uniform(0, 0.02, (n_days, n_assets)))
low = np.minimum(opens, close) * (1 - rng.uniform(0, 0.02, (n_days, n_assets)))
volume = pd.DataFrame(rng.uniform(1e5, 1e6, (n_days, n_assets)), index=idx, columns=cols)
return {"close": close, "open": opens, "high": high, "low": low, "volume": volume}
class RobustMomentumTests(unittest.TestCase):
def test_weights_on_a_day_ignore_that_day_and_later_bars(self):
bars = _bars()
cut = bars["close"].index[380]
for signal in SIGNALS:
strat = RobustMomentumStrategy(signal, top_n=5, rebal_freq=5, buffer=1.0)
base = strat.generate_signals(bars)
shocked = {f: b.copy() for f, b in bars.items()}
for f in shocked:
shocked[f].loc[cut:] *= 3.0
again = strat.generate_signals(shocked)
pd.testing.assert_frame_equal(base.loc[:cut], again.loc[:cut], obj=signal)
self.assertGreater(base.loc[cut].sum(), 0.99, signal)
def test_held_names_stay_while_inside_the_buffer(self):
idx = pd.bdate_range("2025-01-01", periods=3)
score = pd.DataFrame([[5, 4, 3, 2, 1], [1, 2, 5, 4, 3], [1, 2, 5, 4, 3]],
index=idx, columns=list("ABCDE"), dtype=float)
w = select_with_buffer(score, top_n=2, rebal_freq=1, buffer=1.0, warmup=0)
self.assertEqual(set(w.columns[w.iloc[0] > 0]), {"A", "B"})
# Day 2: A ranks 5th (outside 2 * (1 + 1) = 4) and leaves; B ranks 4th and stays.
self.assertEqual(set(w.columns[w.iloc[1] > 0]), {"B", "C"})
def test_bar_strategy_trades_at_the_close_after_its_signal_day(self):
bars = _bars()
strat = RobustMomentumStrategy("lowbeta_sharpe_mom", top_n=5, rebal_freq=5)
weights = strat.generate_signals(bars)
held = strategy_positions(strat, bars["close"], open_data=bars["open"], bars=bars)
pd.testing.assert_frame_equal(held, weights.shift(1).fillna(0.0))
equity = backtest(strat, bars["close"], initial_capital=1000, open_data=bars["open"], bars=bars)
self.assertEqual(len(equity), len(bars["close"]))
def test_trader_session_signals_use_bars_through_the_previous_day(self):
bars = _bars()
days = bars["close"].index[-3:]
strat = RobustMomentumStrategy("overnight_recovery", top_n=5, rebal_freq=1)
full = strat.generate_signals(bars).loc[days]
got = trader.bar_session_weights(strat, bars, days)
pd.testing.assert_frame_equal(got, full)
# Morning run: today's bar does not exist yet.
today = days[-1] + pd.offsets.BDay()
morning = trader.bar_session_weights(strat, bars, pd.DatetimeIndex([today]))
self.assertAlmostEqual(morning.loc[today].sum(), 1.0)
class LiveBookSimulationTests(unittest.TestCase):
def test_holdings_drift_and_only_resizes_beyond_the_band_trade(self):
idx = pd.bdate_range("2026-01-05", periods=5)
data = pd.DataFrame({"A": [100, 100, 125, 125, 125], "B": [100] * 5, "C": [100] * 5},
index=idx, dtype=float)
third = [0, 1 / 3, 1 / 3, 1 / 3]
held = pd.DataFrame({"A": third + [0.0], "B": third + [0.5], "C": third + [0.5]}, index=idx)
equity, traded = simulate_live_book(held, data, 1000.0, transaction_cost=0.0,
resize_band=0.10)
orders = (traded.abs() > 0).sum(axis=1).tolist()
# Buy all three; A +25% drifts to 38.5% (15% over target): trim A only,
# B and C at 30.8% stay inside the band; then exit A, top up B and C.
self.assertEqual(orders, [0, 3, 0, 1, 3])
self.assertLess(traded.loc[idx[3], "A"], 0)
self.assertAlmostEqual(equity.iloc[2], 1000 * (1 + 0.25 / 3), places=6)
def test_orders_below_the_floor_are_skipped_but_exits_are_not(self):
idx = pd.bdate_range("2026-01-05", periods=3)
data = pd.DataFrame({"A": [10.0, 10.0, 10.0], "B": [10.0, 10.0, 10.0]}, index=idx)
held = pd.DataFrame({"A": [0, 0.004, 0.0], "B": [0, 0.5, 0.5]}, index=idx)
_, traded = simulate_live_book(held, data, 10_000.0, min_trade_value=50.0)
self.assertEqual(traded["A"].abs().sum(), 0.0) # a $40 entry is below $50
self.assertGreater(traded["B"].abs().sum(), 0.0)
class LotAllocationTests(unittest.TestCase):
def test_leftover_cash_buys_the_lot_that_brings_a_name_nearest_its_target(self):
prices = {"A": 45.0, "B": 13.0}
# 70k book, 50/50: A wants 777.8 shares, B 2692.3. Rounding down leaves
# 4.9k idle; it buys A's 8th lot (3.5k short of target vs half a lot
# 2.25k), after which B's extra lot no longer fits.
trades = trader.compute_trades({}, 70_000.0, {"A": 0.5, "B": 0.5}, prices,
integer_shares=True, lot_size=100)
shares = {t["ticker"]: t["shares_delta"] for t in trades}
self.assertEqual(shares, {"A": 800, "B": 2600})
spent = sum(s * prices[t] for t, s in shares.items())
self.assertLessEqual(spent, 70_000.0)
def test_a_lot_far_above_target_is_not_bought(self):
trades = trader.compute_trades({}, 10_000.0, {"A": 0.3, "B": 0.7}, {"A": 70.0, "B": 1.0},
integer_shares=True, lot_size=100)
# A's target is 3000 = 0.43 of a 7000 lot: stay out.
self.assertNotIn("A", {t["ticker"] for t in trades})
def test_small_drift_is_not_traded_but_entries_and_exits_are(self):
holdings = {"A": 10.0, "B": 10.0}
prices = {"A": 105.0, "B": 95.0, "C": 50.0}
trades = trader.compute_trades(holdings, 0.0, {"A": 0.5, "B": 0.5}, prices,
min_trade_value=1.0, resize_band=0.10)
self.assertEqual(trades, [])
trades = trader.compute_trades(holdings, 0.0, {"A": 0.5, "C": 0.5}, prices,
min_trade_value=1.0, resize_band=0.10)
self.assertEqual({t["ticker"]: t["direction"] for t in trades}, {"B": "SELL", "C": "BUY"})
class PanelBackfillTests(unittest.TestCase):
def test_tickers_missing_from_a_partial_batch_are_retried_before_giving_up(self):
idx = pd.bdate_range("2026-01-05", periods=4)
calls = []
def fake_download(tickers, start, end=None, fields=None):
calls.append(list(tickers))
ok = [t for t in tickers if t == "AAA" or (t == "BBB" and len(calls) > 1)]
frame = pd.DataFrame({t: [1.0, 2.0, 3.0, 4.0] for t in ok}, index=idx)
return {f: frame for f in fields}
with mock.patch.object(data_manager, "_download", fake_download):
got, missing = data_manager._backfill(["AAA", "BBB", "CCC"], "2026-01-05",
["Close", "Volume"], idx)
self.assertEqual(sorted(got["Close"].columns), ["AAA", "BBB"])
self.assertEqual(missing, ["CCC"])
self.assertEqual(calls[1], ["BBB", "CCC"])
class BacktestUniverseTests(unittest.TestCase):
def test_stock_universe_excludes_benchmarks_and_etfs_sharing_the_panel(self):
import backtest_runner
idx = pd.bdate_range("2026-01-05", periods=3)
panel = pd.DataFrame(1.0, index=idx, columns=["AAA", "BBB", "SPY", "TQQQ"])
fake_universe = {"us": {"fetch": lambda: ["AAA"], "benchmark": "SPY",
"benchmark_label": "SPY"}}
with mock.patch.object(backtest_runner, "UNIVERSES", fake_universe), \
mock.patch.object(backtest_runner.uh, "pit_tickers_ever", lambda m, since=None: ["AAA", "BBB"]), \
mock.patch.object(backtest_runner.uh, "mask_prices_pit", lambda p, m: p), \
mock.patch.object(backtest_runner.data_manager, "update",
lambda m, t, fields=None: {f: panel for f in fields}):
data = backtest_runner.load_stock_panels("us", ["momentum"], None)
self.assertEqual(data["stocks"], ["AAA", "BBB"])
class MembershipTests(unittest.TestCase):
def test_csi300_snapshot_applies_until_the_next_one(self):
snaps = {"2026-01-01": ["600000.SS", "000001.SZ"], "2026-02-01": ["600000.SS"]}
dates = pd.to_datetime(["2025-12-31", "2026-01-15", "2026-02-02"])
mask = uh.csi300_membership_mask(dates, ["600000.SS", "000001.SZ", "510300.SS"], snaps)
self.assertEqual(mask["000001.SZ"].tolist(), [False, True, False])
self.assertEqual(mask["600000.SS"].tolist(), [False, True, True])
# Never-member columns (benchmarks) are not masked.
self.assertTrue(mask["510300.SS"].all())
def test_changes_table_with_two_level_headers_is_found(self):
cols = pd.MultiIndex.from_tuples([("Effective Date", "Effective Date"), ("Added", "Ticker"),
("Removed", "Ticker"), ("Reason", "Reason")])
table = pd.DataFrame([["September 21, 2026", "BE", "TAP", "cap"]], columns=cols)
other = pd.DataFrame({"x": [1]})
changes = uh._find_changes_table([other, table])
self.assertEqual(list(changes.columns), ["Date", "Added_Ticker", "Removed_Ticker", "Reason_Reason"])
if __name__ == "__main__":
unittest.main()

228
trader.py
View File

@@ -40,6 +40,7 @@ import pandas as pd
import yfinance as yf
import data_manager
from strategies.base import BarStrategy
from strategies.buy_and_hold import BuyAndHoldStrategy
from strategies.dual_momentum import DualMomentumStrategy
from strategies.factor_combo import FactorComboStrategy
@@ -55,6 +56,7 @@ from strategies.permanent import (
TrendRiderV4,
)
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.robust_momentum import RobustMomentumStrategy
from strategies.trend_following import TrendFollowingStrategy
from strategies.trend_rider_v5 import TrendRiderV5
from strategies.trend_rider_voltgt import (
@@ -95,6 +97,12 @@ MARKET_FEE_TIERED = {
# board lots of 100.
MARKET_LOT_SIZE = {"us": 1, "cn": 100}
# A position already held is resized only once it is this far from its target
# (relative to the target value); entries and exits always trade. Without the
# band the books paid a ticket fee almost daily to trade weight drift that
# the backtests never trade.
RESIZE_BAND = 0.10
# Default live capital per market, in the market's own currency.
MONITOR_CAPITAL = {"us": 10_000.0, "cn": 70_000.0}
@@ -226,6 +234,16 @@ STRATEGY_REGISTRY = {
"recovery_quality_blend_top10": lambda **kw: RecoveryQualityBlendStrategy(top_n=10),
"improved_mom_quality_top20": lambda **kw: ImprovedMomentumQualityStrategy(top_n=20),
"improved_mom_quality_top10": lambda **kw: ImprovedMomentumQualityStrategy(top_n=10),
# --- Robust momentum (2026-09 PIT research; bar strategies) ---
"overnight_recovery_top10": lambda **kw: RobustMomentumStrategy(
"overnight_recovery", top_n=10, rebal_freq=10, buffer=2.0,
),
"lowrisk_momentum_top15": lambda **kw: RobustMomentumStrategy(
"lowrisk_momentum", top_n=15, rebal_freq=10, buffer=2.0,
),
"lowbeta_sharpe_mom_top20": lambda **kw: RobustMomentumStrategy(
"lowbeta_sharpe_mom", top_n=20, rebal_freq=10, buffer=1.0,
),
# --- TrendRiderV6: stock-picking + V5 regime engine ---
"trend_rider_v6": lambda **kw: TrendRiderV6(),
"trend_rider_v6_top10": lambda **kw: TrendRiderV6(top_n=10),
@@ -273,6 +291,12 @@ def strategy_extra_tickers(strategy_name: str) -> list[str]:
return list(MIXED_STRATEGY_EXTRA_TICKERS.get(strategy_name.removeprefix("sim_"), []))
def strategy_fields(strategy_name: str) -> tuple[str, ...]:
"""Bar fields a strategy reads; close-price strategies read ("close",)."""
strategy = STRATEGY_REGISTRY[strategy_name.removeprefix("sim_")]()
return tuple(strategy.fields) if isinstance(strategy, BarStrategy) else ("close",)
def strategy_supports_market(strategy_name: str, market: str) -> bool:
"""ETF-based strategies trade USD-listed ETFs, so they only run in the US book."""
base_name = strategy_name.removeprefix("sim_")
@@ -351,14 +375,15 @@ def load_strategy_prices(market: str, strategy_name: str, tickers: list[str],
stale forward-filled price from a real one.
"""
data_market = strategy_data_market(market, strategy_name)
fields = ["close", "open"] if with_open else ["close"]
fields += [f for f in strategy_fields(strategy_name) if f not in fields]
if refresh:
request = price_request(market, list(tickers) + [benchmark])
result = data_manager.update(data_market, request, with_open=with_open)
close, open_data = result if with_open else (result, None)
panels = data_manager.update(data_market, request, fields=fields)
else:
close = data_manager.load(data_market, "close")
open_data = (data_manager._fill_open(close, data_manager.load_raw(data_market, "open"))
if with_open else None)
panels = data_manager.load_fields(data_market, fields)
close = panels["close"]
open_data = panels["open"] if with_open else None
close_raw = data_manager.load_raw(data_market, "close").reindex(
index=close.index, columns=close.columns,
)
@@ -438,85 +463,102 @@ def portfolio_value(holdings: dict, prices: dict, cash: float) -> float:
return cash + stock_value
def generate_target_weights(strategy, open_data: pd.DataFrame, target_date) -> dict:
"""
Run strategy on full open price history, extract target_date's weights.
def strategy_bars(market: str, strategy_name: str, tickers: list[str],
close: pd.DataFrame, open_data: pd.DataFrame | None) -> dict | None:
"""Stored bar panels a BarStrategy reads (refreshed by load_strategy_prices),
aligned to ``close``; None for close-price strategies."""
fields = strategy_fields(strategy_name)
if not isinstance(STRATEGY_REGISTRY[strategy_name.removeprefix("sim_")](), BarStrategy):
return None
stored = data_manager.load_fields(strategy_data_market(market, strategy_name),
[f for f in fields if f not in ("close", "open")])
bars = {"close": close[tickers]}
if "open" in fields:
bars["open"] = open_data[tickers]
for f in fields:
if f not in bars:
bars[f] = stored[f].reindex(index=close.index, columns=tickers)
return bars
Strategies have an internal shift(1) designed for close prices: weights[t]
uses data up to t-1. With open prices (observable same-day), we want the
signal that uses data up to today. To get this, we append a dummy future
row so the strategy computes one extra day of signals, then we read that
last row — which incorporates today's open prices.
"""
# Append a dummy row one day after the last date so the strategy's
# shift(1) produces a valid signal row that uses target_date's open
dummy_date = open_data.index[-1] + pd.Timedelta(days=1)
dummy_row = pd.DataFrame(
[open_data.iloc[-1].values],
index=[dummy_date],
columns=open_data.columns,
)
extended = pd.concat([open_data, dummy_row])
weights = strategy.generate_signals(extended)
def bar_session_weights(strategy: BarStrategy, bars: dict, days: pd.DatetimeIndex) -> pd.DataFrame:
"""Targets traded at the close of each day in ``days``: row d uses bars through d-1."""
hist = {f: b.loc[:days[-1]] for f, b in bars.items()}
close = hist["close"]
if days[-1] not in close.index:
# morning run before today's bar exists: add an empty row for today
hist = {f: pd.concat([b, pd.DataFrame(np.nan, index=[days[-1]], columns=b.columns)])
for f, b in hist.items()}
return strategy.generate_signals(hist).reindex(days).fillna(0.0)
# The dummy_date row in weights now uses data up to target_date (the real last day)
# Read the dummy row — that's today's actual signal
if dummy_date in weights.index:
row = weights.loc[dummy_date]
elif target_date in weights.index:
# Fallback: use target_date row (which uses data up to target_date - 1)
row = weights.loc[target_date]
else:
idx = weights.index.get_indexer([target_date], method="ffill")
if idx[0] >= 0:
row = weights.iloc[idx[0]]
def _allocate_lots(desired: dict, holdings: dict, prices: dict, cash: float,
lot_size: int, reserve: float) -> dict:
"""Whole-lot share targets: round each move toward zero, then spend the
remaining cash one lot at a time on the names furthest below target,
as long as the lot brings them nearer (nearest-lot rounding)."""
final, spend = {}, 0.0
for t, want in desired.items():
cur = holdings.get(t, 0.0)
if want <= 0:
final[t] = 0.0 # full exit, odd lot included
else:
return {}
# Return only non-zero weights
return {t: float(w) for t, w in row.items() if w > 1e-6}
final[t] = cur + int((want - cur) / lot_size) * lot_size
spend += (final[t] - cur) * prices[t]
budget = cash - spend - reserve
short = sorted(((desired[t] - final[t]) * prices[t], t) for t in desired if desired[t] > 0)
for gap, t in reversed(short):
lot_value = lot_size * prices[t]
if gap > 0.5 * lot_value and lot_value <= budget:
final[t] += lot_size
budget -= lot_value
return final
def compute_trades(holdings: dict, cash: float, target_weights: dict,
prices: dict, min_trade_value: float = 50.0,
integer_shares: bool = False, lot_size: int = 1,
frozen: set | None = None) -> list[dict]:
frozen: set | None = None, resize_band: float = 0.0) -> list[dict]:
"""
Compute trades needed to move from current holdings to target weights.
If integer_shares=True, share deltas are rounded toward zero to whole
multiples of ``lot_size`` (e.g. 100-share A-share board lots); a position
whose target is zero is always sold in full, odd lot included.
If integer_shares=True, positions move in whole multiples of ``lot_size``
(e.g. 100-share A-share board lots): each move is rounded toward zero and
leftover cash then buys one more lot for the names furthest below target
when that lot brings them nearer; a position whose target is zero is
always sold in full, odd lot included.
A held position with a nonzero target is not resized while it is within
``resize_band`` (fraction of its target value) of the target.
Tickers in ``frozen`` (no fresh price today) are neither bought nor sold.
Returns list of {ticker, shares_delta, direction, est_value}.
"""
total = portfolio_value(holdings, prices, cash)
raw = []
frozen = frozen or set()
all_tickers = set(list(target_weights.keys()) + list(holdings.keys()))
for ticker in sorted(all_tickers):
desired = {}
for ticker in sorted(set(target_weights) | set(holdings)):
if ticker in frozen:
continue
target_weight = target_weights.get(ticker, 0.0)
price = prices.get(ticker, 0.0)
if price <= 0:
continue
target_weight = target_weights.get(ticker, 0.0)
target_value = total * target_weight
target_shares = target_value / price
current_shares = holdings.get(ticker, 0.0)
if (current_shares > 0 and target_weight > 0
and abs(target_value - current_shares * price) < resize_band * target_value):
continue
desired[ticker] = target_value / price
targets = (_allocate_lots(desired, holdings, prices, cash, lot_size,
reserve=0.002 * total + 10.0 * len(desired))
if integer_shares else desired)
raw = []
for ticker, target_shares in targets.items():
price = prices[ticker]
current_shares = holdings.get(ticker, 0.0)
delta = target_shares - current_shares
if integer_shares:
if target_weight <= 0 and current_shares > 0:
delta = -current_shares # full exit may include an odd lot
else:
# Round toward zero: never overshoot cash or over-sell.
delta = int(delta / lot_size) * lot_size
trade_value = abs(delta * price)
if trade_value < min_trade_value:
continue
@@ -735,25 +777,32 @@ def _strategy_fees(args, market: str) -> dict:
def _signals_for_sessions(strategy_name: str, open_data: pd.DataFrame,
days: pd.DatetimeIndex) -> pd.DataFrame:
"""Target weights for many sessions in one pass: row ``d`` uses opens up to ``d``.
days: pd.DatetimeIndex, bars: dict | None = None) -> pd.DataFrame:
"""Target weights traded at the close of each day in ``days``, in one pass.
Equivalent to ``generate_target_weights`` per day for causal strategies
(weights at t never depend on data after t), which every registered
strategy must be.
A close-price strategy reads opens as its prices: the row for ``d`` uses
opens up to ``d`` (a dummy row after the last open makes the strategy's
internal shift(1) yield that row). A BarStrategy uses completed bars
through d-1. Every registered strategy must be causal (weights at t never
depend on data after t), so one pass equals running each day separately.
"""
strategy = STRATEGY_REGISTRY[strategy_name]()
if isinstance(strategy, BarStrategy):
return bar_session_weights(strategy, bars, days)
hist = open_data.loc[:days[-1]]
dummy = pd.DataFrame([hist.iloc[-1].values], index=[hist.index[-1] + pd.Timedelta(days=1)],
columns=hist.columns)
weights = STRATEGY_REGISTRY[strategy_name]().generate_signals(pd.concat([hist, dummy]))
weights = strategy.generate_signals(pd.concat([hist, dummy]))
return weights.shift(-1).reindex(days).fillna(0.0)
def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp,
tickers: list[str], benchmark: str, close_data: pd.DataFrame,
open_data: pd.DataFrame, close_raw: pd.DataFrame, fees: dict,
integer_shares: bool, signal_row: pd.Series | None = None) -> dict:
"""Signal on ``day``'s open, execute at ``day``'s close, record the day.
integer_shares: bool, signal_row: pd.Series | None = None,
bars: dict | None = None) -> dict:
"""Signal on ``day``'s open (a BarStrategy: bars through the previous
session), execute at ``day``'s close, record the day.
Tickers without a real closing print on ``day`` (forward-filled price) are
valued but not traded. ``signal_row`` supplies precomputed target weights.
@@ -761,10 +810,9 @@ def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp
day_str = str(day.date())
lot_size = MARKET_LOT_SIZE.get(market, 1) if integer_shares else 1
if signal_row is None:
strategy = STRATEGY_REGISTRY[strategy_name]()
target_weights = generate_target_weights(strategy, open_data.loc[:day, tickers], day)
else:
target_weights = {t: float(w) for t, w in signal_row.items() if w > 1e-6}
signal_row = _signals_for_sessions(strategy_name, open_data[tickers],
pd.DatetimeIndex([day]), bars).loc[day]
target_weights = {t: float(w) for t, w in signal_row.items() if w > 1e-6}
all_held = sorted(set(target_weights) | set(state["holdings"]))
close_prices = _with_last_known_prices(
@@ -778,6 +826,7 @@ def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp
state["holdings"], state["cash"], target_weights, close_prices,
min_trade_value=max(50, pre_value * 0.001),
integer_shares=integer_shares, lot_size=lot_size, frozen=frozen,
resize_band=RESIZE_BAND,
)
n_log = len(state["trade_log"])
execute_trades(state, trades, close_prices, trade_date=day_str,
@@ -840,9 +889,11 @@ def cmd_morning(args):
print(f" Strategy: {strategy_name} | Market: {market.upper()}")
print(f"{'='*60}")
# Run strategy on open prices
strategy = STRATEGY_REGISTRY[strategy_name]()
target_weights = generate_target_weights(strategy, open_data[tickers], today)
# Run strategy on open prices (a BarStrategy: on completed bars)
bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
row = _signals_for_sessions(strategy_name, open_data[tickers],
pd.DatetimeIndex([today]), bars).loc[today]
target_weights = {t: float(w) for t, w in row.items() if w > 1e-6}
# Get today's open prices for trade computation
open_prices = get_prices_for_date(
@@ -854,7 +905,8 @@ def cmd_morning(args):
trades = compute_trades(state["holdings"], state["cash"], target_weights,
open_prices, min_trade_value=max(50, total * 0.001),
integer_shares=args.integer_shares,
lot_size=MARKET_LOT_SIZE.get(market, 1))
lot_size=MARKET_LOT_SIZE.get(market, 1),
resize_band=RESIZE_BAND)
# Store pending
state["pending_trades"] = {
@@ -1335,18 +1387,8 @@ def cmd_simulate(args):
# Pre-compute strategy weights for the full period (much faster than per-day)
# Append dummy row so shift(1) inside strategy produces valid last-day signal
strategy = STRATEGY_REGISTRY[strategy_name]()
open_tickers = open_data[tickers]
dummy_date = open_tickers.index[-1] + pd.Timedelta(days=1)
dummy_row = pd.DataFrame(
[open_tickers.iloc[-1].values], index=[dummy_date], columns=open_tickers.columns
)
extended = pd.concat([open_tickers, dummy_row])
raw_weights = strategy.generate_signals(extended)
# Strategy's shift(1): raw_weights[t] uses data up to t-1
# For date t, the signal using open[t] is in raw_weights[t+1]
# So shift(-1) to align: full_weights[t] = signal using open[t]
full_weights = raw_weights.shift(-1).fillna(0.0)
bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
full_weights = _signals_for_sessions(strategy_name, open_data[tickers], trading_days, bars)
print(f"\n{'='*70}")
print(f" SIMULATION: {strategy_name} | {market.upper()}")
@@ -1385,6 +1427,7 @@ def cmd_simulate(args):
state["holdings"], state["cash"], target_weights,
close_prices, min_trade_value=max(50, pre_value * 0.001),
integer_shares=args.integer_shares, lot_size=lot_size,
resize_band=RESIZE_BAND,
)
execute_trades(state, trades, close_prices,
@@ -1710,13 +1753,16 @@ def cmd_monitor(args):
def refresh_panels(market: str, strategy_names: list[str], label: str) -> None:
"""Refresh every price panel the strategies read, once, for the union of their tickers."""
requests: dict[str, set[str]] = {}
fields: dict[str, set[str]] = {}
for name in strategy_names:
tickers, benchmark = strategy_universe(market, name)
requests.setdefault(strategy_data_market(market, name), set()).update(
data_market = strategy_data_market(market, name)
requests.setdefault(data_market, set()).update(
price_request(market, list(tickers) + [benchmark]))
fields.setdefault(data_market, {"close", "open"}).update(strategy_fields(name))
for data_market, request in requests.items():
print(f"[{label}] Refreshing {data_market} panel for {len(request)} tickers")
data_manager.update(data_market, sorted(request), with_open=True)
data_manager.update(data_market, sorted(request), fields=sorted(fields[data_market]))
def _process_sessions(args, state: dict | None, start: pd.Timestamp | None = None,
@@ -1759,12 +1805,12 @@ def _process_sessions(args, state: dict | None, start: pd.Timestamp | None = Non
f"{todo[0].date()} .. {todo[-1].date()}")
fees = _strategy_fees(args, market)
signals = (_signals_for_sessions(strategy_name, open_data[tickers], todo)
if len(todo) > 1 else None)
bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
signals = _signals_for_sessions(strategy_name, open_data[tickers], todo, bars)
for day in todo:
r = _run_session(state, strategy_name, market, day, tickers, benchmark,
close_data, open_data, close_raw, fees, args.integer_shares,
signal_row=signals.loc[day] if signals is not None else None)
signal_row=signals.loc[day], bars=bars)
frozen = f" | no fresh price: {', '.join(r['frozen'])}" if r["frozen"] else ""
print(f"[{label}] {r['date']} {strategy_name:<34} value {r['value']:>12,.2f} "
f"({r['daily_return']:+.2%}) trades {r['trades']:>2} "

View File

@@ -76,45 +76,70 @@ def get_sp500() -> list[str]:
# Yahoo Finance format: Shanghai → XXXXXX.SS, Shenzhen → XXXXXX.SZ
# Wikipedia CSI 300 page has a table with stock codes and exchanges.
CSI300_CONS_URL = ("https://oss-ch.csindex.com.cn/static/html/csindex/public/uploads/"
"file/autofile/cons/000300cons.xls")
def _csi300_official() -> list[str]:
"""Current constituents from China Securities Index's published file."""
req = urllib.request.Request(CSI300_CONS_URL, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=30) as resp:
payload = resp.read()
df = pd.read_excel(io.BytesIO(payload), dtype=str)
code_col = next(c for c in df.columns if "Constituent Code" in c)
exch_col = next(c for c in df.columns if c.startswith("交易所Exchange"))
tickers = [f"{code}.{'SS' if '上海' in exch else 'SZ'}"
for code, exch in zip(df[code_col], df[exch_col])]
if len(tickers) < 250:
raise RuntimeError(f"CSI 300 file has only {len(tickers)} rows")
return sorted(set(tickers))
def _csi300_latest_snapshot() -> list[str]:
with open(os.path.join(CACHE_DIR, "csi300_history.json")) as f:
snaps = json.load(f)["snapshots"]
return snaps[max(snaps)]
def _csi300_wikipedia() -> list[str]:
tables = _fetch_html_tables("https://en.wikipedia.org/wiki/CSI_300_Index")
# The constituent table has ~300 rows and a "Ticker" column like "SSE: 600519"
df = next((t for t in tables if len(t) >= 200 and "Ticker" in t.columns), None)
if df is None:
raise RuntimeError("Could not find CSI 300 constituent table on Wikipedia")
tickers = []
for raw in df["Ticker"].astype(str):
parts = raw.split(":")
if len(parts) == 2 and parts[1].strip().isdigit():
suffix = ".SS" if parts[0].strip().upper() == "SSE" else ".SZ"
tickers.append(parts[1].strip() + suffix)
return sorted(set(tickers))
def get_csi300() -> list[str]:
"""Fetch current CSI 300 constituents from Wikipedia."""
"""Current CSI 300 constituents.
Source order: China Securities Index's published constituent file, then
the latest snapshot in data/csi300_history.json, then Wikipedia (whose
table lagged the real index by ~60 names in 2026).
"""
cached = _read_cache("csi300")
if cached:
print(f"--- Loaded CSI 300 universe from cache ({len(cached)} tickers) ---")
return cached
print("--- Fetching CSI 300 constituents from Wikipedia ---")
url = "https://en.wikipedia.org/wiki/CSI_300_Index"
tables = _fetch_html_tables(url)
# Find the constituent table — it has ~300 rows and a "Ticker" column
# with format like "SSE: 600519" or "SZSE: 300750"
df = None
for t in tables:
if len(t) >= 200 and "Ticker" in t.columns:
df = t
break
if df is None:
raise RuntimeError("Could not find CSI 300 constituent table on Wikipedia")
tickers = []
for _, row in df.iterrows():
raw = str(row["Ticker"]) # e.g. "SSE: 600519" or "SZSE: 300750"
parts = raw.split(":")
if len(parts) != 2:
for label, source in (("csindex.com.cn", _csi300_official),
("stored CSI 300 history", _csi300_latest_snapshot),
("Wikipedia", _csi300_wikipedia)):
try:
tickers = source()
except Exception as exc: # noqa: BLE001 — fall through to the next source
print(f"--- CSI 300 from {label} failed: {exc} ---")
continue
exchange_prefix = parts[0].strip().upper()
code = parts[1].strip()
if not code.isdigit():
continue
suffix = ".SS" if exchange_prefix == "SSE" else ".SZ"
tickers.append(code + suffix)
tickers = sorted(set(tickers))
_write_cache("csi300", tickers)
print(f"--- Got {len(tickers)} CSI 300 tickers ---")
return tickers
_write_cache("csi300", tickers)
print(f"--- Got {len(tickers)} CSI 300 tickers from {label} ---")
return tickers
raise RuntimeError("No CSI 300 constituent source available")
# ---------------------------------------------------------------------------

View File

@@ -20,6 +20,7 @@ import os
import urllib.request
from datetime import date, datetime
import numpy as np
import pandas as pd
CACHE_DIR = "data"
@@ -30,34 +31,51 @@ _HEADERS = {"User-Agent": "Mozilla/5.0 (quant-backtest)"}
# Fetch + parse Wikipedia
# ---------------------------------------------------------------------------
def _fetch_sp500_tables() -> tuple[pd.DataFrame, pd.DataFrame]:
"""Return (current_list, changes_log) from the S&P 500 Wikipedia page."""
url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
SP500_LIST_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
# The change log moved off the list page in 2026; the list page is still tried
# first in case it moves back.
SP500_CHANGES_URLS = [
SP500_LIST_URL,
"https://en.wikipedia.org/wiki/Historical_components_of_the_S%26P_500",
]
def _read_tables(url: str) -> list[pd.DataFrame]:
req = urllib.request.Request(url, headers=_HEADERS)
with urllib.request.urlopen(req) as resp:
html = resp.read().decode("utf-8")
tables = pd.read_html(io.StringIO(html))
current = tables[0]
change_tables = [
t for t in tables[1:]
if any("Added" in "_".join(map(str, c if isinstance(c, tuple) else (c,))) for c in t.columns)
]
if not change_tables:
raise ValueError("S&P 500 change log table not found on the Wikipedia page")
changes = change_tables[0]
changes.columns = [
"_".join(c).strip() if isinstance(c, tuple) else c
for c in changes.columns
]
changes.columns = [
c.replace("Effective Date_Effective Date", "Date") for c in changes.columns
]
return current, changes
return pd.read_html(io.StringIO(html))
def _find_changes_table(tables: list[pd.DataFrame]) -> pd.DataFrame | None:
for t in tables:
if any("Added" in "_".join(map(str, c if isinstance(c, tuple) else (c,))) for c in t.columns):
changes = t.copy()
changes.columns = [
"_".join(c).strip() if isinstance(c, tuple) else c
for c in changes.columns
]
changes.columns = [
c.replace("Effective Date_Effective Date", "Date") for c in changes.columns
]
return changes
return None
def _fetch_sp500_tables() -> tuple[pd.DataFrame, pd.DataFrame]:
"""Return (current_list, changes_log) from Wikipedia."""
current = _read_tables(SP500_LIST_URL)[0]
for url in SP500_CHANGES_URLS:
changes = _find_changes_table(_read_tables(url))
if changes is not None:
return current, changes
raise ValueError("S&P 500 change log table not found on Wikipedia")
def _normalize_ticker(t: str) -> str:
"""Yahoo Finance ticker format: BRK.B → BRK-B."""
return str(t).replace(".", "-").strip()
# Some change-log cells carry a trailing " |" separator.
return str(t).split("|")[0].replace(".", "-").strip()
# ---------------------------------------------------------------------------
@@ -243,3 +261,101 @@ def mask_prices(prices: pd.DataFrame,
"""
mask = membership_mask(prices.index, intervals, tickers=list(prices.columns))
return prices.where(mask)
# ---------------------------------------------------------------------------
# CSI 300 (China A-shares)
# ---------------------------------------------------------------------------
#
# data/csi300_history.json holds monthly constituent snapshots
# {"snapshots": {"YYYY-MM-DD": [yahoo tickers], ...}}
# seeded from baostock's query_hs300_stocks (first trading day of each month,
# 2008 onward). A snapshot applies from its date until the next one, so an
# index change is picked up within a month. load_csi300_history() appends a
# snapshot whenever the current constituent list (universe.get_csi300) differs
# from the latest one, which keeps the file current without baostock.
def _csi300_path() -> str:
return os.path.join(CACHE_DIR, "csi300_history.json")
def baostock_to_yahoo(code: str) -> str:
"""'sh.600000' -> '600000.SS', 'sz.000001' -> '000001.SZ'."""
ex, num = code.split(".")
return f"{num}.{'SS' if ex == 'sh' else 'SZ'}"
def load_csi300_history(current: list[str] | None = None) -> dict[str, list[str]]:
"""Monthly CSI 300 snapshots; appends ``current`` (or today's list) if it changed."""
path = _csi300_path()
with open(path) as f:
data = json.load(f)
snaps = data["snapshots"]
if current is None:
try:
from universe import get_csi300
current = get_csi300()
except Exception as exc: # noqa: BLE001 — offline runs use the stored history
print(f"--- WARNING: CSI 300 list unavailable ({exc}); using stored history ---")
current = None
if current:
last = snaps[max(snaps)]
today = str(date.today())
if sorted(current) != sorted(last) and today > max(snaps):
snaps[today] = sorted(current)
with open(path, "w") as f:
json.dump(data, f, indent=0)
print(f"--- CSI 300 constituents changed; stored snapshot {today} ---")
return snaps
def csi300_membership_mask(dates: pd.DatetimeIndex, tickers: list[str],
snapshots: dict[str, list[str]] | None = None) -> pd.DataFrame:
"""Boolean mask (dates x tickers): True while the ticker was in the CSI 300."""
snaps = snapshots if snapshots is not None else load_csi300_history()
keys = sorted(snaps)
col = {t: i for i, t in enumerate(tickers)}
arr = np.zeros((len(dates), len(tickers)), dtype=bool)
for i, k in enumerate(keys):
start = pd.Timestamp(k)
end = pd.Timestamp(keys[i + 1]) if i + 1 < len(keys) else dates[-1] + pd.Timedelta(days=1)
rows = (dates >= start) & (dates < end)
cols = [col[t] for t in snaps[k] if t in col]
if rows.any() and cols:
arr[np.ix_(rows, cols)] = True
mask = pd.DataFrame(arr, index=dates, columns=tickers)
# Tickers never in the index (benchmarks, ETFs) pass through unmasked.
ever = {t for lst in snaps.values() for t in lst}
for t in tickers:
if t not in ever:
mask[t] = True
return mask
def all_csi300_tickers_ever(snapshots: dict[str, list[str]] | None = None,
since: str | None = None) -> list[str]:
snaps = snapshots if snapshots is not None else load_csi300_history()
return sorted({t for k, lst in snaps.items() if since is None or k >= since for t in lst})
# ---------------------------------------------------------------------------
# Market-generic helpers
# ---------------------------------------------------------------------------
def pit_tickers_ever(market: str, since: str | None = None) -> list[str] | None:
"""Every ticker that was an index member (since ``since``), or None when the
market has no membership history."""
if market == "us":
return all_tickers_ever()
if market == "cn":
return all_csi300_tickers_ever(since=since)
return None
def mask_prices_pit(prices: pd.DataFrame, market: str) -> pd.DataFrame:
"""NaN out (date, ticker) pairs outside point-in-time index membership."""
if market == "us":
return mask_prices(prices)
if market == "cn":
return prices.where(csi300_membership_mask(prices.index, list(prices.columns)))
return prices

11
uv.lock generated
View File

@@ -770,6 +770,7 @@ dependencies = [
{ name = "numpy" },
{ name = "pandas" },
{ name = "scipy" },
{ name = "xlrd" },
{ name = "yfinance" },
]
@@ -780,6 +781,7 @@ requires-dist = [
{ name = "numpy", specifier = ">=2.4.0" },
{ name = "pandas", specifier = ">=2.3.3" },
{ name = "scipy", specifier = ">=1.17.1" },
{ name = "xlrd", specifier = ">=2.0.2" },
{ name = "yfinance", specifier = ">=1.0" },
]
@@ -935,6 +937,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" },
]
[[package]]
name = "xlrd"
version = "2.0.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/07/5a/377161c2d3538d1990d7af382c79f3b2372e880b65de21b01b1a2b78691e/xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9", size = 100167, upload-time = "2025-06-14T08:46:39.039Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/1a/62/c8d562e7766786ba6587d09c5a8ba9f718ed3fa8af7f4553e8f91c36f302/xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9", size = 96555, upload-time = "2025-06-14T08:46:37.766Z" },
]
[[package]]
name = "yfinance"
version = "1.0"