Compare commits
2 Commits
83495bcd23
...
cc1366b2dc
| Author | SHA1 | Date | |
|---|---|---|---|
| cc1366b2dc | |||
| 0e378efa67 |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -59,3 +59,6 @@ memory/
|
||||
# Web UI state and precomputed backtests (web/src/server/paths.ts)
|
||||
data/web/
|
||||
data/precomputed/
|
||||
|
||||
# Research panels and results — fetched/rebuilt by research/lab
|
||||
data/research/
|
||||
|
||||
26
CLAUDE.md
26
CLAUDE.md
@@ -9,7 +9,7 @@ quant-trading: a quantitative backtesting framework and live forward-testing tra
|
||||
The repository has two halves that share one data directory and one `.env`:
|
||||
|
||||
- **Python engine** (repo root, `uv`): strategies, backtests, `trader.py`, research scripts.
|
||||
- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`.
|
||||
- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`. Both backtest paths run through `backtest_runner.py`.
|
||||
|
||||
`bridge.py`'s JSON output and `web/src/shared/strategyTypes.ts` are one contract: change both in the same commit.
|
||||
|
||||
@@ -40,6 +40,13 @@ uv run python trader.py simulate --integer-shares --fixed-fee 2.0 ... # Integ
|
||||
# Precompute the strategy page's backtests (writes data/precomputed/)
|
||||
uv run python precompute_backtest.py
|
||||
|
||||
# Factor research harness (PIT universe, next-open fills, phase averaging; data in data/research/)
|
||||
uv run python -m research.lab.fetch_us # 20y OHLCV for every S&P 500 ever-member
|
||||
uv run python -m research.lab.fetch_cn_yahoo # CSI 300/500 ever-members (membership from baostock snapshot)
|
||||
PYTHONPATH=. uv run python -m research.lab.factor_scan --market us # or cn_hs300
|
||||
PYTHONPATH=. uv run python -m research.lab.stepwise --market us --objective maximin
|
||||
PYTHONPATH=. uv run python -m research.lab.evaluate --market us # candidates vs champions
|
||||
|
||||
# Setup
|
||||
uv sync # Install/sync dependencies
|
||||
cp .env.example .env # Then fill IPADS_API_KEY and QUANT_MARKET_PROXY
|
||||
@@ -55,15 +62,17 @@ uv run python -m unittest discover tests
|
||||
|
||||
## Architecture
|
||||
|
||||
**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia, CSI 300 from Wikipedia) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
|
||||
**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia; CSI 300 from China Securities Index's published constituent file, falling back to the latest `data/csi300_history.json` snapshot, then Wikipedia, whose table lagged the index by ~60 names in 2026) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
|
||||
|
||||
**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_open.csv` (open). First run downloads 10 years of history; subsequent runs append only new trading days (fast bulk fetch for gaps ≤7 days). New tickers from index rebalances are backfilled automatically. Tickers with >50% missing data are dropped. Pass `with_open=True` to also maintain open price files.
|
||||
**Point-in-time membership** (`universe_history.py`): S&P 500 intervals rebuilt from Wikipedia's "Historical components of the S&P 500" change log into `data/sp500_history.json`; CSI 300 monthly snapshots in `data/csi300_history.json` (seeded from baostock 2008–2026, appended whenever the current list changes). `mask_prices_pit(prices, market)` masks non-member dates.
|
||||
|
||||
**Backtest engine** (`main.py`): Orchestrates data loading, strategy execution, and visualization. The `backtest()` function is vectorized — it takes a strategy and price DataFrame, applies transaction costs (proportional + optional fixed per-trade fee) via turnover, and returns an equity curve. Supports two execution modes: `close` (classic) and `open-close` (signal on open prices, execute at close).
|
||||
**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_{open,high,low,volume}.csv`. First run downloads 10 years of history; every run re-downloads the last 30 days. New tickers from index rebalances are backfilled automatically; tickers missing from a partial batch are retried alone before being marked unavailable (retried weekly). `update(market, tickers, with_open=True)` returns (close, open); `update(..., fields=[...])` returns `{field: panel}`.
|
||||
|
||||
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, plus 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`).
|
||||
**Backtest engine** (`main.py`): Orchestrates data loading, strategy execution, and visualization. `backtest()` has two modes. Close mode (no `open_data`, used by `main.py` and old research scripts) is vectorized and trades at the close the signal saw. Live mode (`open_data` given, used by the web UI through `backtest_runner.py`) keeps the book like `trader.py`: same signal timing (`strategy_positions`), holdings that drift, the $50/0.1% order floor, `trader.RESIZE_BAND`, fees on the orders placed (`simulate_live_book`).
|
||||
|
||||
**Strategy protocol** (`strategies/base.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy is responsible for applying its own 1-day lag via `.shift(1)` to avoid lookahead bias — the backtest engine does not shift.
|
||||
**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Orders: whole A-share lots are allocated nearest-lot within cash (`_allocate_lots`); a held name is resized only when ≥ `RESIZE_BAND` (10%) off target. Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`), and the robust-momentum bar strategies `overnight_recovery_top10`, `lowrisk_momentum_top15`, `lowbeta_sharpe_mom_top20`.
|
||||
|
||||
**Strategy protocol** (`strategies/base.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy is responsible for applying its own 1-day lag via `.shift(1)` to avoid lookahead bias — the backtest engine does not shift. A `BarStrategy` declares `fields` (close/open/high/low/volume) and receives `{field: DataFrame}`; live it uses completed bars only (row d from bars through d-1, traded at the close of d), while close-price strategies read the session's open as their latest price.
|
||||
|
||||
**Strategies**:
|
||||
- `buy_and_hold.py` — Equal-weight static allocation
|
||||
@@ -76,7 +85,8 @@ uv run python -m unittest discover tests
|
||||
- `momentum_quality.py` — Momentum + return consistency + low drawdown
|
||||
- `adaptive_momentum.py` — Momentum weighted by inverse volatility
|
||||
- `recovery_momentum.py` — Recovery (price/63d low) + 12-1mo momentum composite. Best US performer.
|
||||
- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol` (50.7% CAGR monthly), `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom` (26.1% CAGR monthly), `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap`. All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
|
||||
- `robust_momentum.py` — Bar strategies from the 2026-09 PIT research (report: https://claude.ai/artifact/Mo9nrzs5SWj5FfyshERyWS): US `overnight_recovery` (overnight-return momentum + distance from 126d low), CN `lowrisk_momentum` (amplitude-cut momentum + low beta + price-volume corr + momentum/vol) and close-only `lowbeta_sharpe_mom`. Top-N with rank hysteresis, rebalance every 10 days, weights drift between rebalances.
|
||||
- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol`, `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom`, `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap` — these were mined on today's CSI 300 members; on PIT members they earn ~3-7% a year (2016-2026). All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
|
||||
|
||||
**Metrics** (`metrics.py`): Standalone functions for portfolio analytics (Sharpe, Sortino, Calmar, max drawdown, etc.). `summary()` prints a formatted report and returns a dict.
|
||||
|
||||
@@ -84,7 +94,7 @@ uv run python -m unittest discover tests
|
||||
|
||||
- Strategies output **weights**, not buy/sell signals. Weights are normalized so each row sums to 1 (or 0 during warm-up).
|
||||
- Warm-up periods: each strategy zeros out its first N rows where indicators are undefined.
|
||||
- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests) and `data/web/` (watchlist, per-stock cache, backtest result cache). Not checked into git (only `data/.gitkeep` and `data/sp500_history.json` are tracked).
|
||||
- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests), `data/web/` (watchlist, per-stock cache, backtest result cache) and `data/research/` (research panels, ~2 GB). Not checked into git (only `data/.gitkeep`, `data/sp500_history.json` and `data/csi300_history.json` are tracked).
|
||||
- Python 3.12+, managed with `uv`.
|
||||
- Do not show matplotlib figures when running backtests; use `--no-plot`.
|
||||
|
||||
|
||||
197
backtest_runner.py
Normal file
197
backtest_runner.py
Normal file
@@ -0,0 +1,197 @@
|
||||
"""
|
||||
Market-level backtests behind the web UI: ``bridge.py backtest`` (on demand)
|
||||
and ``precompute_backtest.py`` (the strategy page) share this one data path.
|
||||
|
||||
* Universe: point-in-time index members in both markets (S&P 500 history from
|
||||
Wikipedia, CSI 300 history in data/csi300_history.json); prices outside a
|
||||
stock's membership are masked, so strategies cannot pick it then.
|
||||
* Book: kept like the live trader keeps it — its signal timing
|
||||
(``main.strategy_positions`` live mode), drifting holdings, the same order
|
||||
size floor and resize band (``main.simulate_live_book``).
|
||||
* Bars: panels a BarStrategy needs (open/high/low/volume) are loaded too.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
|
||||
import pandas as pd
|
||||
|
||||
import data_manager
|
||||
import universe_history as uh
|
||||
from main import simulate_live_book, strategy_positions
|
||||
from strategies.base import BarStrategy
|
||||
from trader import (
|
||||
ETF_STRATEGY_UNIVERSES,
|
||||
LIVE_EXCLUDED_STRATEGIES,
|
||||
RESIZE_BAND,
|
||||
STRATEGY_REGISTRY,
|
||||
convert_quote_currency,
|
||||
filter_tradable_tickers,
|
||||
price_request,
|
||||
strategy_extra_tickers,
|
||||
strategy_fields,
|
||||
)
|
||||
from universe import UNIVERSES
|
||||
|
||||
# Rebalances below this size are not charged the per-trade fee in backtests,
|
||||
# matching the live trader's minimum order size.
|
||||
MIN_TRADE_VALUE = 50.0
|
||||
|
||||
|
||||
def _log(msg: str) -> None:
|
||||
print(msg, file=sys.stderr)
|
||||
|
||||
|
||||
def trade_events(diff: pd.DataFrame, threshold: float = 0.01) -> list[dict]:
|
||||
"""Trade events from per-ticker traded weights (rows = days)."""
|
||||
trades = []
|
||||
for date in diff.index:
|
||||
row = diff.loc[date]
|
||||
significant = row[row.abs() > threshold]
|
||||
if significant.empty:
|
||||
continue
|
||||
buys = significant[significant > 0]
|
||||
sells = significant[significant < 0]
|
||||
if len(buys) > 0 or len(sells) > 0:
|
||||
trades.append({
|
||||
"date": date.strftime("%Y-%m-%d"),
|
||||
"buys": len(buys),
|
||||
"sells": len(sells),
|
||||
"buyTickers": list(buys.nlargest(3).index),
|
||||
"sellTickers": list(sells.nsmallest(3).index),
|
||||
})
|
||||
return trades
|
||||
|
||||
|
||||
def benchmark_curve(prices: pd.Series, dates: pd.DatetimeIndex, capital: float) -> list[dict]:
|
||||
"""Benchmark value on ``dates``, rebased to ``capital`` on the first date it has a price."""
|
||||
series = prices.reindex(dates).ffill().dropna()
|
||||
if series.empty:
|
||||
return []
|
||||
values = series / series.iloc[0] * capital
|
||||
return [{"date": d.strftime("%Y-%m-%d"), "value": round(float(v), 2)} for d, v in values.items()]
|
||||
|
||||
|
||||
def _slice_years(panels: dict[str, pd.DataFrame], years: int | None) -> dict[str, pd.DataFrame]:
|
||||
if not years:
|
||||
return panels
|
||||
close = panels["close"]
|
||||
cutoff = close.index[-1] - pd.DateOffset(years=years)
|
||||
return {f: p[p.index >= cutoff] for f, p in panels.items()}
|
||||
|
||||
|
||||
def load_stock_panels(market: str, strategy_names: list[str], years: int | None) -> dict:
|
||||
"""PIT-masked panels for the market's stock strategies."""
|
||||
universe = UNIVERSES[market]
|
||||
tickers = universe["fetch"]()
|
||||
benchmark = universe["benchmark"]
|
||||
since = None
|
||||
if years:
|
||||
since = (pd.Timestamp.today() - pd.DateOffset(years=years + 1)).strftime("%Y-%m-%d")
|
||||
ever = uh.pit_tickers_ever(market, since=since) or []
|
||||
extras = {t for n in strategy_names for t in strategy_extra_tickers(n)}
|
||||
request = sorted(set(tickers) | set(ever) | extras | {benchmark})
|
||||
fields = sorted({"close", "open"} | {f for n in strategy_names for f in strategy_fields(n)})
|
||||
_log(f"PIT universe: {len(request)} tickers ({len(ever)} ever index members); fields {fields}")
|
||||
panels = data_manager.update(market, request, fields=fields)
|
||||
panels = _slice_years(panels, years)
|
||||
member = None
|
||||
if ever:
|
||||
member = uh.mask_prices_pit(panels["close"], market).notna() | panels["close"].isna()
|
||||
# Only tickers that were index members: the shared panel also holds
|
||||
# benchmarks and the ETFs of mixed strategies (TQQQ, SOXL, ...), which
|
||||
# the membership mask passes through unmasked.
|
||||
members_ever = set(ever)
|
||||
stocks = [t for t in panels["close"].columns
|
||||
if t in members_ever and t not in extras
|
||||
and panels["close"][t].where(member[t]).notna().any()]
|
||||
else:
|
||||
stocks = filter_tradable_tickers(panels["close"], tickers)
|
||||
return {"panels": panels, "member": member, "stocks": stocks, "benchmark": benchmark,
|
||||
"benchmark_label": universe["benchmark_label"]}
|
||||
|
||||
|
||||
def _run(strategy, panels: dict[str, pd.DataFrame], cols: list[str], capital: float,
|
||||
tx_cost: float, fixed_fee: float, fee_base: float, fee_per_share: float,
|
||||
member: pd.DataFrame | None = None, min_trade_value: float = MIN_TRADE_VALUE):
|
||||
"""Close-price strategies see prices masked to index membership (they
|
||||
cannot pick a non-member); a BarStrategy sees full histories plus the
|
||||
membership as ``bars["eligible"]``. The book is valued on unmasked closes."""
|
||||
close, opens = panels["close"][cols], panels["open"][cols]
|
||||
if isinstance(strategy, BarStrategy):
|
||||
bars = {f: panels[f][cols] for f in strategy.fields}
|
||||
if member is not None:
|
||||
bars["eligible"] = member.reindex(columns=cols).fillna(True)
|
||||
held = strategy_positions(strategy, close, open_data=opens, bars=bars)
|
||||
else:
|
||||
m = member.reindex(columns=cols).fillna(True) if member is not None else None
|
||||
held = strategy_positions(strategy, close.where(m) if m is not None else close,
|
||||
open_data=opens.where(m) if m is not None else opens)
|
||||
equity, traded = simulate_live_book(held, close, capital, tx_cost, fixed_fee, fee_base,
|
||||
fee_per_share, min_trade_value, RESIZE_BAND)
|
||||
return equity, trade_events(traded)
|
||||
|
||||
|
||||
def run_market(market: str, strategy_names: list[str], years: int | None, capital: float,
|
||||
tx_cost: float, fixed_fee: float, fee_base: float = 0.0,
|
||||
fee_per_share: float = 0.0) -> dict | None:
|
||||
"""Backtest every strategy once; returns raw equity series and metadata."""
|
||||
stock = [n for n in strategy_names if n.removeprefix("sim_") not in ETF_STRATEGY_UNIVERSES]
|
||||
etf = [n for n in strategy_names if n.removeprefix("sim_") in ETF_STRATEGY_UNIVERSES]
|
||||
fees = dict(tx_cost=tx_cost, fixed_fee=fixed_fee, fee_base=fee_base, fee_per_share=fee_per_share)
|
||||
runs: list[dict] = []
|
||||
out = {"market": market, "benchmark_label": "SPY", "benchmark_curve": [],
|
||||
"start_date": None, "end_date": None, "tx_cost": tx_cost, "fixed_fee": fixed_fee}
|
||||
|
||||
if stock:
|
||||
data = load_stock_panels(market, stock, years)
|
||||
panels, stocks = data["panels"], data["stocks"]
|
||||
close = panels["close"]
|
||||
_log(f"Universe: {len(stocks)} stocks, {close.index[0].date()} to {close.index[-1].date()}")
|
||||
for name in stock:
|
||||
_log(f" Running: {name}")
|
||||
try:
|
||||
strategy = STRATEGY_REGISTRY[name]()
|
||||
cols = stocks + [t for t in strategy_extra_tickers(name)
|
||||
if t in close.columns and t not in stocks]
|
||||
# Strategies the live book cannot hold (e.g. equal weight over
|
||||
# ~500 names) are shown without the live order-size floor.
|
||||
floor = 0.0 if name in LIVE_EXCLUDED_STRATEGIES else MIN_TRADE_VALUE
|
||||
equity, trades = _run(strategy, panels, cols, capital, **fees,
|
||||
member=data["member"], min_trade_value=floor)
|
||||
runs.append({"name": name, "equity": equity, "trades": trades})
|
||||
_log(f" OK: {name} (final: {equity.iloc[-1]:,.0f})")
|
||||
except Exception as e: # noqa: BLE001 — one broken strategy must not sink the page
|
||||
_log(f" FAILED: {name}: {e}")
|
||||
out.update(benchmark_label=data["benchmark_label"],
|
||||
benchmark_curve=benchmark_curve(close[data["benchmark"]], close.index, capital),
|
||||
start_date=close.index[0].strftime("%Y-%m-%d"),
|
||||
end_date=close.index[-1].strftime("%Y-%m-%d"))
|
||||
|
||||
for name in etf:
|
||||
_log(f" Running ETF strategy: {name}")
|
||||
try:
|
||||
etf_tickers = ETF_STRATEGY_UNIVERSES[name.removeprefix("sim_")]
|
||||
close, opens = data_manager.update(
|
||||
"etfs", price_request(market, sorted(set(etf_tickers + ["SPY"]))), with_open=True)
|
||||
panels = _slice_years({"close": convert_quote_currency(close),
|
||||
"open": convert_quote_currency(opens)}, years)
|
||||
strategy = STRATEGY_REGISTRY[name]()
|
||||
equity, trades = _run(strategy, panels, filter_tradable_tickers(panels["close"], etf_tickers),
|
||||
capital, **fees)
|
||||
runs.append({"name": name, "equity": equity, "trades": trades})
|
||||
_log(f" OK: {name} (final: {equity.iloc[-1]:,.0f})")
|
||||
if not stock and not out["benchmark_curve"] and "SPY" in panels["close"].columns:
|
||||
spy = panels["close"]["SPY"]
|
||||
out.update(benchmark_label="SPY (Benchmark)",
|
||||
benchmark_curve=benchmark_curve(spy, spy.index, capital),
|
||||
start_date=spy.index[0].strftime("%Y-%m-%d"),
|
||||
end_date=spy.index[-1].strftime("%Y-%m-%d"))
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log(f" FAILED: {name}: {e}")
|
||||
|
||||
if not runs:
|
||||
return None
|
||||
out["strategy_runs"] = runs
|
||||
return out
|
||||
229
bridge.py
229
bridge.py
@@ -20,29 +20,19 @@ import pandas as pd
|
||||
|
||||
import data_manager
|
||||
import metrics
|
||||
import universe_history as uh
|
||||
from main import backtest
|
||||
from backtest_runner import benchmark_curve, run_market
|
||||
from research.permanent_yearly import equity_with_cashflows
|
||||
from trader import (
|
||||
strategy_extra_tickers,
|
||||
strategy_supports_market,
|
||||
ETF_STRATEGY_UNIVERSES,
|
||||
LIVE_EXCLUDED_STRATEGIES,
|
||||
MARKET_FEE_TIERED,
|
||||
STRATEGY_REGISTRY,
|
||||
convert_quote_currency,
|
||||
price_request,
|
||||
strategy_supports_market,
|
||||
strategy_data_market,
|
||||
filter_tradable_tickers,
|
||||
strategy_supports_market,
|
||||
)
|
||||
|
||||
MARKET_CURRENCY = {"us": "USD", "cn": "CNY"}
|
||||
|
||||
# Rebalances below this size are not charged the per-trade fee in backtests,
|
||||
# matching the live trader's minimum order size.
|
||||
MIN_TRADE_VALUE = 50.0
|
||||
from universe import UNIVERSES
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -440,6 +430,25 @@ STRATEGY_META = {
|
||||
"params": {},
|
||||
"markets": ["us"],
|
||||
},
|
||||
# --- Robust momentum (2026-09 PIT research): US pick, CN picks ---
|
||||
"overnight_recovery_top10": {
|
||||
"label": "Robust: Overnight Mom + Recovery Top 10",
|
||||
"category": "factor_combo",
|
||||
"params": {},
|
||||
"markets": ["us", "cn"],
|
||||
},
|
||||
"lowrisk_momentum_top15": {
|
||||
"label": "Robust: Low-Risk Momentum Top 15",
|
||||
"category": "factor_combo",
|
||||
"params": {},
|
||||
"markets": ["us", "cn"],
|
||||
},
|
||||
"lowbeta_sharpe_mom_top20": {
|
||||
"label": "Robust: Low-Beta + Sharpe Mom Top 20",
|
||||
"category": "factor_combo",
|
||||
"params": {},
|
||||
"markets": ["us", "cn"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -515,28 +524,6 @@ def compute_drawdown_periods(equity: pd.Series, threshold: float = -0.05) -> lis
|
||||
return periods
|
||||
|
||||
|
||||
def compute_rebalance_trades(weights: pd.DataFrame, threshold: float = 0.01) -> list[dict]:
|
||||
"""Extract significant trade events from weight changes."""
|
||||
diff = weights.diff().fillna(0.0)
|
||||
trades = []
|
||||
for date in diff.index:
|
||||
row = diff.loc[date]
|
||||
significant = row[row.abs() > threshold]
|
||||
if significant.empty:
|
||||
continue
|
||||
buys = significant[significant > 0]
|
||||
sells = significant[significant < 0]
|
||||
if len(buys) > 0 or len(sells) > 0:
|
||||
trades.append({
|
||||
"date": date.strftime("%Y-%m-%d"),
|
||||
"buys": len(buys),
|
||||
"sells": len(sells),
|
||||
"buyTickers": list(buys.nlargest(3).index),
|
||||
"sellTickers": list(sells.nsmallest(3).index),
|
||||
})
|
||||
return trades
|
||||
|
||||
|
||||
def compute_rebalance_win_rate(equity: pd.Series, trade_events: list[dict]) -> float | None:
|
||||
"""Fraction of inter-rebalance periods with positive cumulative return.
|
||||
|
||||
@@ -600,15 +587,6 @@ def compute_periodic_returns(equity: pd.Series) -> tuple[dict, dict]:
|
||||
return monthly, yearly
|
||||
|
||||
|
||||
def benchmark_curve(prices: pd.Series, dates: pd.DatetimeIndex, capital: float) -> list[dict]:
|
||||
"""Benchmark value on ``dates``, rebased to ``capital`` on the first date it has a price."""
|
||||
series = prices.reindex(dates).ffill().dropna()
|
||||
if series.empty:
|
||||
return []
|
||||
values = series / series.iloc[0] * capital
|
||||
return [{"date": d.strftime("%Y-%m-%d"), "value": round(float(v), 2)} for d, v in values.items()]
|
||||
|
||||
|
||||
def build_injection_schedule(equity: pd.Series, schedule: dict) -> tuple[pd.Series, list[dict]]:
|
||||
"""Build a contributions Series from injection schedule config.
|
||||
|
||||
@@ -725,159 +703,18 @@ def cmd_backtest(args: argparse.Namespace) -> None:
|
||||
_log(f"Running backtest: market={market}, years={years}, capital={capital}")
|
||||
_log(f"Strategies: {strategy_names}")
|
||||
|
||||
# Determine data needs
|
||||
# Group strategies by their data source
|
||||
stock_strategies = []
|
||||
etf_strategies = []
|
||||
for name in strategy_names:
|
||||
if name not in STRATEGY_REGISTRY:
|
||||
_log(f"WARNING: Unknown strategy '{name}', skipping")
|
||||
continue
|
||||
base_name = name.removeprefix("sim_")
|
||||
if base_name in ETF_STRATEGY_UNIVERSES:
|
||||
etf_strategies.append(name)
|
||||
else:
|
||||
stock_strategies.append(name)
|
||||
|
||||
results = []
|
||||
|
||||
# Process stock strategies
|
||||
if stock_strategies:
|
||||
universe = UNIVERSES[market]
|
||||
with _redirect_stdout_to_stderr():
|
||||
tickers = universe["fetch"]()
|
||||
benchmark_ticker = universe["benchmark"]
|
||||
benchmark_label = universe["benchmark_label"]
|
||||
|
||||
# PIT universe: for US market, include all tickers that were EVER
|
||||
# S&P 500 members so that removed stocks are present in price data.
|
||||
pit_intervals = None
|
||||
if market == "us":
|
||||
with _redirect_stdout_to_stderr():
|
||||
pit_intervals = uh.load_sp500_history()
|
||||
historical_tickers = uh.all_tickers_ever(pit_intervals)
|
||||
all_tickers = sorted(set(tickers + historical_tickers + [benchmark_ticker])
|
||||
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
|
||||
_log(f"PIT universe: {len(all_tickers)} tickers (current + historical members)")
|
||||
else:
|
||||
all_tickers = sorted(set(tickers + [benchmark_ticker])
|
||||
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
|
||||
_log(f"WARNING: {market.upper()} market lacks PIT membership history; "
|
||||
"backtest uses current index constituents (survivorship bias possible)")
|
||||
|
||||
_log(f"Fetching price data for {len(all_tickers)} tickers...")
|
||||
with _redirect_stdout_to_stderr():
|
||||
data = data_manager.update(market, all_tickers, with_open=False)
|
||||
if isinstance(data, tuple):
|
||||
data = data[0]
|
||||
|
||||
if years:
|
||||
cutoff = data.index[-1] - pd.DateOffset(years=years)
|
||||
data = data[data.index >= cutoff]
|
||||
|
||||
# Apply PIT mask: set prices to NaN for dates when a stock was NOT
|
||||
# an index member. Strategies naturally skip NaN entries.
|
||||
if pit_intervals is not None:
|
||||
_log("Applying PIT membership mask (survivorship-bias fix)...")
|
||||
data = uh.mask_prices(data, pit_intervals)
|
||||
|
||||
tickers = filter_tradable_tickers(data, tickers)
|
||||
# For PIT mode, expand tickers to all columns with valid data
|
||||
# (historical members that have been masked are NaN when not members,
|
||||
# but have valid data during their membership period).
|
||||
if pit_intervals is not None:
|
||||
all_data_tickers = [
|
||||
t for t in data.columns
|
||||
if t != benchmark_ticker and data[t].notna().any()
|
||||
]
|
||||
tickers = all_data_tickers
|
||||
_log(f"Universe: {len(tickers)} tradable stocks, period: {data.index[0].date()} to {data.index[-1].date()}")
|
||||
|
||||
for name in stock_strategies:
|
||||
_log(f" Running: {name}")
|
||||
# Registry defaults are the single source of strategy parameters,
|
||||
# shared with the live trader.
|
||||
strategy = STRATEGY_REGISTRY[name]()
|
||||
# Mixed stock+ETF strategies also read their fixed ETFs.
|
||||
cols = tickers + [t for t in strategy_extra_tickers(name)
|
||||
if t in data.columns and t not in tickers]
|
||||
|
||||
equity = backtest(
|
||||
strategy, data[cols],
|
||||
initial_capital=capital,
|
||||
transaction_cost=tx_cost,
|
||||
fixed_fee=fixed_fee,
|
||||
min_trade_value=MIN_TRADE_VALUE,
|
||||
**MARKET_FEE_TIERED.get(market, {}),
|
||||
)
|
||||
|
||||
# Get weight signals for trade extraction
|
||||
weights = strategy.generate_signals(data[cols])
|
||||
weights = weights.reindex(data[cols].index).fillna(0.0)
|
||||
trade_events = compute_rebalance_trades(weights)
|
||||
|
||||
result_entry = _build_strategy_result(
|
||||
name, equity, trade_events, injection_schedule, capital
|
||||
)
|
||||
results.append(result_entry)
|
||||
|
||||
# Benchmark over exactly the strategies' dates
|
||||
bench_curve = benchmark_curve(data[benchmark_ticker], data.index, capital)
|
||||
else:
|
||||
benchmark_label = "SPY"
|
||||
bench_curve = []
|
||||
|
||||
# Process ETF strategies
|
||||
if etf_strategies:
|
||||
for name in etf_strategies:
|
||||
_log(f" Running ETF strategy: {name}")
|
||||
base_name = name.removeprefix("sim_")
|
||||
etf_tickers = ETF_STRATEGY_UNIVERSES[base_name]
|
||||
all_etf = sorted(set(etf_tickers + ["SPY"]))
|
||||
|
||||
with _redirect_stdout_to_stderr():
|
||||
etf_data = data_manager.update("etfs", price_request(market, all_etf), with_open=False)
|
||||
if isinstance(etf_data, tuple):
|
||||
etf_data = etf_data[0]
|
||||
etf_data = convert_quote_currency(etf_data)
|
||||
|
||||
if years:
|
||||
cutoff = etf_data.index[-1] - pd.DateOffset(years=years)
|
||||
etf_data = etf_data[etf_data.index >= cutoff]
|
||||
|
||||
strategy = STRATEGY_REGISTRY[name]()
|
||||
|
||||
tradable = filter_tradable_tickers(etf_data, etf_tickers)
|
||||
equity = backtest(
|
||||
strategy, etf_data[tradable],
|
||||
initial_capital=capital,
|
||||
transaction_cost=tx_cost,
|
||||
fixed_fee=fixed_fee,
|
||||
min_trade_value=MIN_TRADE_VALUE,
|
||||
**MARKET_FEE_TIERED.get(market, {}),
|
||||
)
|
||||
|
||||
weights = strategy.generate_signals(etf_data[tradable])
|
||||
weights = weights.reindex(etf_data[tradable].index).fillna(0.0)
|
||||
trade_events = compute_rebalance_trades(weights)
|
||||
|
||||
result_entry = _build_strategy_result(
|
||||
name, equity, trade_events, injection_schedule, capital
|
||||
)
|
||||
results.append(result_entry)
|
||||
|
||||
# If no stock strategies were run, use SPY as benchmark from ETF data
|
||||
if not stock_strategies and len(etf_strategies) > 0:
|
||||
benchmark_label = "SPY (Benchmark)"
|
||||
with _redirect_stdout_to_stderr():
|
||||
spy_data = data_manager.update("etfs", ["SPY"], with_open=False)
|
||||
if isinstance(spy_data, tuple):
|
||||
spy_data = spy_data[0]
|
||||
if years:
|
||||
cutoff = spy_data.index[-1] - pd.DateOffset(years=years)
|
||||
spy_data = spy_data[spy_data.index >= cutoff]
|
||||
if "SPY" in spy_data.columns:
|
||||
bench_curve = benchmark_curve(spy_data["SPY"], spy_data.index, capital)
|
||||
known = [n for n in strategy_names if n in STRATEGY_REGISTRY]
|
||||
for name in set(strategy_names) - set(known):
|
||||
_log(f"WARNING: Unknown strategy '{name}', skipping")
|
||||
with _redirect_stdout_to_stderr():
|
||||
raw = run_market(market, known, years, capital, tx_cost, fixed_fee,
|
||||
**MARKET_FEE_TIERED.get(market, {}))
|
||||
results = [
|
||||
_build_strategy_result(run["name"], run["equity"], run["trades"], injection_schedule, capital)
|
||||
for run in (raw["strategy_runs"] if raw else [])
|
||||
]
|
||||
benchmark_label = raw["benchmark_label"] if raw else "SPY"
|
||||
bench_curve = raw["benchmark_curve"] if raw else []
|
||||
|
||||
output = {
|
||||
"meta": {
|
||||
|
||||
14803
data/csi300_history.json
Normal file
14803
data/csi300_history.json
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
156
data_manager.py
156
data_manager.py
@@ -265,94 +265,148 @@ def _fill_open(close: pd.DataFrame, open_raw: pd.DataFrame | None) -> pd.DataFra
|
||||
return aligned.fillna(close)
|
||||
|
||||
|
||||
def update(market: str, tickers: list[str],
|
||||
with_open: bool = False) -> pd.DataFrame | tuple[pd.DataFrame, pd.DataFrame]:
|
||||
YAHOO_FIELDS = {"close": "Close", "open": "Open", "high": "High", "low": "Low", "volume": "Volume"}
|
||||
|
||||
|
||||
def _backfill(tickers: list[str], start: str, yahoo_fields: list[str],
|
||||
index: pd.DatetimeIndex) -> tuple[dict[str, pd.DataFrame], list[str]]:
|
||||
"""Download new tickers over ``index``; return (panels, tickers still missing).
|
||||
|
||||
yfinance reports a ticker it failed to fetch (rate limit, proxy error) the
|
||||
same way as a ticker it has no data for, so tickers missing from the first
|
||||
batch are downloaded once more on their own before being given up on.
|
||||
"""
|
||||
Refresh the stored panel for ``tickers`` and return it forward-filled.
|
||||
def fetch(batch: list[str]) -> dict[str, pd.DataFrame]:
|
||||
raw = _download(batch, start=start, fields=yahoo_fields)
|
||||
close = raw["Close"].dropna(axis=1, how="all").reindex(index=index).dropna(axis=1, how="all")
|
||||
return {f: raw.get(f, pd.DataFrame()).reindex(index=index, columns=close.columns)
|
||||
for f in yahoo_fields}
|
||||
|
||||
got = fetch(tickers)
|
||||
missing = [t for t in tickers if t not in got["Close"].columns]
|
||||
if missing and len(missing) < len(tickers):
|
||||
retry = fetch(missing)
|
||||
if not retry["Close"].empty:
|
||||
got = {f: pd.concat([got[f], retry[f]], axis=1) for f in yahoo_fields}
|
||||
missing = [t for t in missing if t not in retry["Close"].columns]
|
||||
return got, missing
|
||||
|
||||
|
||||
def update(market: str, tickers: list[str], with_open: bool = False,
|
||||
fields: list[str] | None = None):
|
||||
"""
|
||||
Refresh the stored panels for ``tickers`` and return them forward-filled.
|
||||
|
||||
- First run: downloads 10 years of history.
|
||||
- Every run: re-downloads the last ``REFRESH_DAYS`` calendar days for the
|
||||
requested tickers already in the panel and overwrites them.
|
||||
- New tickers are backfilled over the panel's full date range; tickers
|
||||
Yahoo has no data for are remembered and retried weekly.
|
||||
- A field requested for the first time (e.g. ``high``) is downloaded for
|
||||
every ticker in the stored close panel.
|
||||
|
||||
Returns the close panel, or ``(close, open)`` when ``with_open`` is True.
|
||||
Returns the close panel, ``(close, open)`` when ``with_open`` is True, or
|
||||
``{field: panel}`` when ``fields`` is given. Open/high/low fall back to the
|
||||
close where a bar is missing; volume is returned raw (NaN = no print).
|
||||
"""
|
||||
fields = ["Close", "Open"] if with_open else ["Close"]
|
||||
names = list(fields) if fields is not None else (["close", "open"] if with_open else ["close"])
|
||||
if "close" not in names:
|
||||
names = ["close"] + names
|
||||
unknown = [f for f in names if f not in YAHOO_FIELDS]
|
||||
if unknown:
|
||||
raise ValueError(f"Unsupported market data field(s): {unknown}")
|
||||
yahoo = [YAHOO_FIELDS[f] for f in names]
|
||||
|
||||
with _panel_lock(market):
|
||||
close_raw = load_raw(market, "close")
|
||||
open_raw = load_raw(market, "open") if with_open else None
|
||||
raw = {f: load_raw(market, f) for f in names}
|
||||
unavailable = _load_unavailable(market)
|
||||
today = datetime.now()
|
||||
|
||||
if close_raw is None:
|
||||
if raw["close"] is None:
|
||||
start = (today - timedelta(days=365 * 10)).strftime("%Y-%m-%d")
|
||||
downloaded = _download(tickers, start=start, fields=fields)
|
||||
close_raw = _merge_refresh(None, downloaded["Close"], None)
|
||||
if with_open:
|
||||
open_raw = _merge_refresh(None, downloaded.get("Open", pd.DataFrame()), None)
|
||||
downloaded = _download(tickers, start=start, fields=yahoo)
|
||||
for f, y in zip(names, yahoo):
|
||||
raw[f] = _merge_refresh(None, downloaded.get(y, pd.DataFrame()), None)
|
||||
else:
|
||||
if with_open and open_raw is None:
|
||||
print("--- Downloading Open prices for the full panel (first time) ---")
|
||||
start = close_raw.index[0].strftime("%Y-%m-%d")
|
||||
opens = _download(list(close_raw.columns), start=start, fields=["Open"])
|
||||
open_raw = _merge_refresh(None, opens["Open"], None)
|
||||
close_raw = raw["close"]
|
||||
first = [(f, y) for f, y in zip(names, yahoo) if raw[f] is None]
|
||||
if first:
|
||||
print(f"--- Downloading {[y for _, y in first]} for the full panel (first time) ---")
|
||||
got = _download(list(close_raw.columns), start=close_raw.index[0].strftime("%Y-%m-%d"),
|
||||
fields=[y for _, y in first])
|
||||
for f, y in first:
|
||||
raw[f] = _merge_refresh(None, got[y], None)
|
||||
|
||||
known = [t for t in tickers if t in close_raw.columns]
|
||||
if known:
|
||||
window_start = close_raw.index[-1] - pd.Timedelta(days=REFRESH_DAYS)
|
||||
fresh = _download(known, start=window_start.strftime("%Y-%m-%d"), fields=fields)
|
||||
fresh = _download(known, start=window_start.strftime("%Y-%m-%d"), fields=yahoo)
|
||||
if not fresh["Close"].empty:
|
||||
close_raw = _merge_refresh(close_raw, fresh["Close"], window_start)
|
||||
if with_open:
|
||||
open_raw = _merge_refresh(open_raw, fresh.get("Open", pd.DataFrame()), window_start)
|
||||
for f, y in zip(names, yahoo):
|
||||
raw[f] = _merge_refresh(raw[f], fresh.get(y, pd.DataFrame()), window_start)
|
||||
else:
|
||||
print("--- Refresh download returned no data; keeping stored panel ---")
|
||||
|
||||
cutoff = (today - timedelta(days=UNAVAILABLE_RETRY_DAYS)).strftime("%Y-%m-%d")
|
||||
new_tickers = [t for t in tickers
|
||||
if t not in close_raw.columns and unavailable.get(t, "") < cutoff]
|
||||
if t not in raw["close"].columns and unavailable.get(t, "") < cutoff]
|
||||
if new_tickers:
|
||||
start = close_raw.index[0].strftime("%Y-%m-%d")
|
||||
backfill = _download(new_tickers, start=start, fields=fields)
|
||||
downloaded_any = not backfill["Close"].dropna(how="all").empty
|
||||
got = backfill["Close"].dropna(axis=1, how="all")
|
||||
got = got.reindex(index=close_raw.index)
|
||||
got = got.dropna(axis=1, how="all")
|
||||
for t in new_tickers:
|
||||
if t in got.columns:
|
||||
unavailable.pop(t, None)
|
||||
elif downloaded_any:
|
||||
# Only a ticker missing from an otherwise successful
|
||||
# download is unavailable; a failed request is retried.
|
||||
got, missing = _backfill(new_tickers, raw["close"].index[0].strftime("%Y-%m-%d"),
|
||||
yahoo, raw["close"].index)
|
||||
added = list(got["Close"].columns)
|
||||
for t in added:
|
||||
unavailable.pop(t, None)
|
||||
# A batch that returned nothing at all is a failed request, not
|
||||
# evidence the tickers have no data: retry those next run.
|
||||
if added:
|
||||
for t in missing:
|
||||
unavailable[t] = today.strftime("%Y-%m-%d")
|
||||
if not got.empty:
|
||||
close_raw = pd.concat([close_raw, got], axis=1)
|
||||
if with_open and open_raw is not None:
|
||||
opens = backfill.get("Open", pd.DataFrame())
|
||||
opens = opens.reindex(index=open_raw.index, columns=got.columns)
|
||||
open_raw = pd.concat([open_raw, opens], axis=1)
|
||||
print(f"--- Backfilled {got.shape[1]} new tickers ---")
|
||||
missing = len(new_tickers) - got.shape[1]
|
||||
if added:
|
||||
for f, y in zip(names, yahoo):
|
||||
panel = got[y].reindex(index=raw[f].index)
|
||||
raw[f] = pd.concat([raw[f], panel], axis=1)
|
||||
print(f"--- Backfilled {len(added)} new tickers ---")
|
||||
if missing:
|
||||
print(f"--- {missing} tickers have no Yahoo data; retry after "
|
||||
print(f"--- {len(missing)} tickers have no Yahoo data; retry after "
|
||||
f"{UNAVAILABLE_RETRY_DAYS} days ---")
|
||||
|
||||
close_raw = close_raw.loc[:, ~close_raw.columns.duplicated()]
|
||||
_write_atomic(close_raw, _data_path(market, "close"))
|
||||
print(f"--- Saved {close_raw.shape[0]} days x {close_raw.shape[1]} tickers to "
|
||||
for f in names:
|
||||
raw[f] = raw[f].loc[:, ~raw[f].columns.duplicated()]
|
||||
_write_atomic(raw[f], _data_path(market, f))
|
||||
print(f"--- Saved {raw['close'].shape[0]} days x {raw['close'].shape[1]} tickers to "
|
||||
f"{_data_path(market, 'close')} ---")
|
||||
if with_open and open_raw is not None:
|
||||
open_raw = open_raw.loc[:, ~open_raw.columns.duplicated()]
|
||||
_write_atomic(open_raw, _data_path(market, "open"))
|
||||
_save_unavailable(market, unavailable)
|
||||
|
||||
close = close_raw.ffill()
|
||||
close = raw["close"].ffill()
|
||||
out = {"close": close}
|
||||
for f in names:
|
||||
if f in ("open", "high", "low"):
|
||||
out[f] = _fill_open(close, raw[f])
|
||||
elif f == "volume":
|
||||
out[f] = raw[f].reindex(index=close.index, columns=close.columns)
|
||||
if fields is not None:
|
||||
return out
|
||||
if with_open:
|
||||
return close, _fill_open(close, open_raw)
|
||||
return close, out["open"]
|
||||
return close
|
||||
|
||||
|
||||
def load_fields(market: str, fields: list[str]) -> dict[str, pd.DataFrame]:
|
||||
"""Stored panels without refreshing, aligned and filled the way ``update`` returns them."""
|
||||
close = load(market, "close")
|
||||
out = {"close": close}
|
||||
for f in fields:
|
||||
if f == "close":
|
||||
continue
|
||||
stored = load_raw(market, f)
|
||||
if f == "volume":
|
||||
out[f] = (stored.reindex(index=close.index, columns=close.columns)
|
||||
if stored is not None else pd.DataFrame(np.nan, index=close.index, columns=close.columns))
|
||||
else:
|
||||
out[f] = _fill_open(close, stored)
|
||||
return out
|
||||
|
||||
|
||||
def rebuild(market: str, with_open: bool = True, years: int = 10) -> None:
|
||||
"""Re-download the full history of every ticker in a stored panel.
|
||||
|
||||
|
||||
130
main.py
130
main.py
@@ -9,6 +9,7 @@ import factor_attribution
|
||||
import metrics
|
||||
import universe_history as uh
|
||||
from strategies.adaptive_momentum import AdaptiveMomentumStrategy
|
||||
from strategies.base import BarStrategy
|
||||
from strategies.buy_and_hold import BuyAndHoldStrategy
|
||||
from strategies.dual_momentum import DualMomentumStrategy
|
||||
from strategies.inverse_vol import InverseVolatilityStrategy
|
||||
@@ -25,6 +26,99 @@ from universe import UNIVERSES
|
||||
# Backtest engine
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def strategy_positions(strategy, data: pd.DataFrame, open_data: pd.DataFrame | None = None,
|
||||
bars: dict[str, pd.DataFrame] | None = None) -> pd.DataFrame:
|
||||
"""Weights actually held over (close t-1, close t] for each row t.
|
||||
|
||||
Close mode (``open_data`` is None): row t's weights use closes through t-1
|
||||
and trade at the close of t-1 — the same bar the signal saw.
|
||||
Live mode (``open_data`` given), matching trader.py:
|
||||
* close-price strategies read the session's open as their latest price
|
||||
and trade at that session's close;
|
||||
* a BarStrategy reads completed bars through t-1 and trades at the close
|
||||
of t.
|
||||
"""
|
||||
if isinstance(strategy, BarStrategy):
|
||||
weights = strategy.generate_signals(bars if bars is not None else {"close": data})
|
||||
weights = weights.reindex(data.index).fillna(0.0)
|
||||
return weights.shift(1).fillna(0.0) if open_data is not None else weights
|
||||
if open_data is not None:
|
||||
# The strategy's shift(1) on opens gives weights[t] = f(open_{t-1});
|
||||
# undo it so the row-t decision uses open_t, then hold it from close t.
|
||||
weights = strategy.generate_signals(open_data).shift(-1).fillna(0.0)
|
||||
return weights.reindex(data.index).fillna(0.0).shift(1).fillna(0.0)
|
||||
return strategy.generate_signals(data).reindex(data.index).fillna(0.0)
|
||||
|
||||
|
||||
def simulate_live_book(held: pd.DataFrame, data: pd.DataFrame, initial_capital: float,
|
||||
transaction_cost: float = 0.001, fixed_fee: float = 0.0,
|
||||
fee_base: float = 0.0, fee_per_share: float = 0.0,
|
||||
min_trade_value: float = 0.0,
|
||||
resize_band: float = 0.0) -> tuple[pd.Series, pd.DataFrame]:
|
||||
"""Day-by-day book kept the way trader.py keeps it.
|
||||
|
||||
``held`` row t holds the target weights for (close t-1, close t]. Holdings
|
||||
drift with prices; at each close the book trades toward the next row's
|
||||
targets: entries and exits always, a held name only once it is at least
|
||||
``resize_band`` (relative to its target) away. Orders smaller than
|
||||
``max(min_trade_value, 0.1% of equity)`` are skipped unless they close a
|
||||
position. Costs are charged on the orders placed.
|
||||
|
||||
Returns (equity, traded weight per ticker on the row whose holdings changed).
|
||||
"""
|
||||
targets = held.reindex(index=data.index, columns=data.columns).fillna(0.0).to_numpy()
|
||||
prices = data.ffill().to_numpy(dtype=float)
|
||||
rets = np.nan_to_num(data.ffill().pct_change(fill_method=None).to_numpy(dtype=float), nan=0.0)
|
||||
per_share = np.array([0.0 if str(c).endswith(".HK") else fee_per_share for c in data.columns])
|
||||
n, m = targets.shape
|
||||
w = np.zeros(m)
|
||||
value = float(initial_capital)
|
||||
equity = np.empty(n)
|
||||
traded = np.zeros((n, m))
|
||||
for t in range(n):
|
||||
if value <= 0:
|
||||
equity[t:] = 0.0
|
||||
break
|
||||
gross = w * (1.0 + rets[t])
|
||||
growth = gross.sum() + (1.0 - w.sum()) # idle cash earns nothing
|
||||
value *= growth
|
||||
w = gross / growth if growth > 0 else np.zeros(m)
|
||||
if t + 1 < n:
|
||||
tgt = targets[t + 1]
|
||||
new = w.copy()
|
||||
entry_exit = (tgt <= 0) | (w <= 0)
|
||||
resize = ~entry_exit & (np.abs(tgt - w) >= resize_band * tgt)
|
||||
change = (entry_exit | resize) & (np.abs(tgt - w) > 0)
|
||||
new[change] = tgt[change]
|
||||
if new.sum() > 1.0: # kept winners leave less room
|
||||
buys = change & (new > w)
|
||||
room = max(1.0 - new[~buys].sum(), 0.0)
|
||||
new[buys] *= room / new[buys].sum() if new[buys].sum() > 0 else 0.0
|
||||
delta = new - w
|
||||
floor = (max(min_trade_value, 0.001 * value) / value) if min_trade_value > 0 else 1e-12
|
||||
skip = (np.abs(delta) < floor) & (new > 0)
|
||||
delta[skip] = 0.0
|
||||
new[skip] = w[skip]
|
||||
orders = np.abs(delta) > 1e-12
|
||||
if orders.any():
|
||||
dollars = np.abs(delta) * value
|
||||
cost = dollars.sum() * transaction_cost
|
||||
if fixed_fee > 0 or fee_base > 0 or fee_per_share > 0:
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
shares = np.where(prices[t] > 0, dollars / prices[t], 0.0)
|
||||
fee = np.maximum(fixed_fee, fee_base + per_share * shares)
|
||||
cost += fee[orders].sum()
|
||||
if cost >= value: # fees ate the book
|
||||
equity[t:] = 0.0
|
||||
break
|
||||
w = new * value / (value - cost)
|
||||
value -= cost
|
||||
traded[t + 1] = delta
|
||||
equity[t] = value
|
||||
return (pd.Series(equity, index=data.index),
|
||||
pd.DataFrame(traded, index=data.index, columns=data.columns))
|
||||
|
||||
|
||||
def backtest(
|
||||
strategy,
|
||||
data: pd.DataFrame,
|
||||
@@ -35,6 +129,8 @@ def backtest(
|
||||
fee_per_share: float = 0.0,
|
||||
open_data: pd.DataFrame | None = None,
|
||||
min_trade_value: float = 0.0,
|
||||
bars: dict[str, pd.DataFrame] | None = None,
|
||||
resize_band: float = 0.0,
|
||||
) -> pd.Series:
|
||||
"""
|
||||
Vectorized backtest.
|
||||
@@ -60,39 +156,33 @@ def backtest(
|
||||
With fee_base=1.88 + fee_per_share=0.009 + fixed_fee=2.0 you get an
|
||||
IBKR-style schedule: max(2, 1.88 + 0.009 * shares).
|
||||
open_data : pd.DataFrame, optional
|
||||
Open prices. When provided, enables open-to-close execution mode.
|
||||
Open prices. When provided, the backtest keeps the book the way the
|
||||
live trader does: its signal timing (``strategy_positions``), drifting
|
||||
holdings, whole orders and ``resize_band`` (``simulate_live_book``).
|
||||
min_trade_value : float
|
||||
Per-ticker rebalances smaller than ``max(min_trade_value, 0.1% of
|
||||
equity)`` are not charged the per-trade fee, mirroring the live
|
||||
trader, which never places such orders. 0 (default) charges every
|
||||
weight change.
|
||||
bars : dict, optional
|
||||
``{field: DataFrame}`` panels for a BarStrategy (close/open/high/low/volume).
|
||||
|
||||
Returns
|
||||
-------
|
||||
pd.Series
|
||||
Daily equity curve.
|
||||
"""
|
||||
positions = strategy_positions(strategy, data, open_data=open_data, bars=bars)
|
||||
if open_data is not None:
|
||||
# Open-to-close mode:
|
||||
# Strategy's shift(1) on open prices gives: weights[t] = f(open_{t-1})
|
||||
# But open_t is known at morning of day t, so undo shift to get f(open_t)
|
||||
# Then execute at close_t, earning close_t → close_{t+1}
|
||||
weights = strategy.generate_signals(open_data)
|
||||
weights = weights.shift(-1).fillna(0.0)
|
||||
weights = weights.reindex(data.index).fillna(0.0)
|
||||
positions = weights
|
||||
equity, _ = simulate_live_book(
|
||||
positions, data, initial_capital, transaction_cost, fixed_fee, fee_base,
|
||||
fee_per_share, min_trade_value, resize_band,
|
||||
)
|
||||
return equity
|
||||
|
||||
# Returns earned: close[t] → close[t+1], weighted by positions decided at open[t]
|
||||
close_returns = data.pct_change().fillna(0.0)
|
||||
portfolio_returns = (close_returns * positions.shift(1).fillna(0.0)).sum(axis=1)
|
||||
else:
|
||||
# Classic close-to-close mode
|
||||
weights = strategy.generate_signals(data)
|
||||
weights = weights.reindex(data.index).fillna(0.0)
|
||||
positions = weights
|
||||
|
||||
daily_returns = data.pct_change().fillna(0.0)
|
||||
portfolio_returns = (daily_returns * positions).sum(axis=1)
|
||||
# A missing close carries the last price (zero return), as before.
|
||||
daily_returns = data.ffill().pct_change(fill_method=None).fillna(0.0)
|
||||
portfolio_returns = (daily_returns * positions).sum(axis=1)
|
||||
|
||||
# Turnover cost: sum of absolute weight changes each day
|
||||
turnover = positions.diff().abs().sum(axis=1).fillna(0.0)
|
||||
|
||||
@@ -25,32 +25,15 @@ import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
import data_manager
|
||||
import metrics
|
||||
import universe_history as uh
|
||||
from backtest_runner import run_market
|
||||
from bridge import (
|
||||
MIN_TRADE_VALUE,
|
||||
STRATEGY_META,
|
||||
_build_strategy_result,
|
||||
_sanitize_for_json,
|
||||
benchmark_curve,
|
||||
compute_rebalance_trades,
|
||||
)
|
||||
from main import backtest
|
||||
from trader import (
|
||||
strategy_extra_tickers,
|
||||
strategy_supports_market,
|
||||
ETF_STRATEGY_UNIVERSES,
|
||||
MONITOR_CAPITAL,
|
||||
STRATEGY_REGISTRY,
|
||||
convert_quote_currency,
|
||||
filter_tradable_tickers,
|
||||
price_request,
|
||||
)
|
||||
from universe import UNIVERSES
|
||||
from trader import MONITOR_CAPITAL, STRATEGY_REGISTRY, strategy_supports_market
|
||||
|
||||
|
||||
# Initial capital per market, in local currency — the same amounts the live
|
||||
@@ -102,171 +85,6 @@ def _equity_series_from_curve(curve: list[dict]) -> pd.Series:
|
||||
return pd.Series(values, index=dates)
|
||||
|
||||
|
||||
def run_market_raw(
|
||||
market: str,
|
||||
strategy_names: list[str],
|
||||
years: int,
|
||||
capital: float,
|
||||
tx_cost: float,
|
||||
fixed_fee: float,
|
||||
fee_base: float = 0.0,
|
||||
fee_per_share: float = 0.0,
|
||||
) -> dict | None:
|
||||
"""Run all strategies once; return raw equity series + metadata per strategy."""
|
||||
stock_strategies = []
|
||||
etf_strategies = []
|
||||
for name in strategy_names:
|
||||
base_name = name.removeprefix("sim_")
|
||||
if base_name in ETF_STRATEGY_UNIVERSES:
|
||||
etf_strategies.append(name)
|
||||
else:
|
||||
stock_strategies.append(name)
|
||||
|
||||
raw_strategies: list[dict] = []
|
||||
benchmark_label = "SPY"
|
||||
bench_curve = []
|
||||
start_date = None
|
||||
end_date = None
|
||||
|
||||
# --- Stock strategies ---
|
||||
if stock_strategies:
|
||||
universe = UNIVERSES[market]
|
||||
tickers = universe["fetch"]()
|
||||
benchmark_ticker = universe["benchmark"]
|
||||
benchmark_label = universe["benchmark_label"]
|
||||
|
||||
pit_intervals = None
|
||||
if market == "us":
|
||||
pit_intervals = uh.load_sp500_history()
|
||||
historical_tickers = uh.all_tickers_ever(pit_intervals)
|
||||
all_tickers = sorted(set(tickers + historical_tickers + [benchmark_ticker])
|
||||
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
|
||||
log(f"PIT universe: {len(all_tickers)} tickers")
|
||||
else:
|
||||
all_tickers = sorted(set(tickers + [benchmark_ticker])
|
||||
| {t for n in stock_strategies for t in strategy_extra_tickers(n)})
|
||||
log(f"WARNING: {market.upper()} lacks PIT history; using current constituents")
|
||||
|
||||
log(f"Fetching price data for {len(all_tickers)} tickers...")
|
||||
data = data_manager.update(market, all_tickers, with_open=False)
|
||||
if isinstance(data, tuple):
|
||||
data = data[0]
|
||||
|
||||
if years:
|
||||
cutoff = data.index[-1] - pd.DateOffset(years=years)
|
||||
data = data[data.index >= cutoff]
|
||||
|
||||
if pit_intervals is not None:
|
||||
log("Applying PIT membership mask...")
|
||||
data = uh.mask_prices(data, pit_intervals)
|
||||
tickers = [
|
||||
t for t in data.columns
|
||||
if t != benchmark_ticker and data[t].notna().any()
|
||||
]
|
||||
else:
|
||||
tickers = filter_tradable_tickers(data, tickers)
|
||||
|
||||
log(f"Universe: {len(tickers)} stocks, {data.index[0].date()} to {data.index[-1].date()}")
|
||||
|
||||
for name in stock_strategies:
|
||||
log(f" Running: {name}")
|
||||
try:
|
||||
strategy = STRATEGY_REGISTRY[name]()
|
||||
# Mixed stock+ETF strategies also read their fixed ETFs.
|
||||
cols = tickers + [t for t in strategy_extra_tickers(name)
|
||||
if t in data.columns and t not in tickers]
|
||||
|
||||
equity = backtest(
|
||||
strategy, data[cols],
|
||||
initial_capital=capital,
|
||||
transaction_cost=tx_cost,
|
||||
fixed_fee=fixed_fee,
|
||||
fee_base=fee_base,
|
||||
fee_per_share=fee_per_share,
|
||||
min_trade_value=MIN_TRADE_VALUE,
|
||||
)
|
||||
|
||||
weights = strategy.generate_signals(data[cols])
|
||||
weights = weights.reindex(data[cols].index).fillna(0.0)
|
||||
trade_events = compute_rebalance_trades(weights)
|
||||
|
||||
raw_strategies.append({"name": name, "equity": equity, "trades": trade_events})
|
||||
log(f" OK: {name} (final: ${equity.iloc[-1]:,.0f})")
|
||||
except Exception as e:
|
||||
log(f" FAILED: {name}: {e}")
|
||||
|
||||
# Benchmark on exactly the strategies' dates
|
||||
bench_curve = benchmark_curve(data[benchmark_ticker], data.index, capital)
|
||||
start_date = data.index[0].strftime("%Y-%m-%d")
|
||||
end_date = data.index[-1].strftime("%Y-%m-%d")
|
||||
|
||||
# --- ETF strategies ---
|
||||
if etf_strategies:
|
||||
for name in etf_strategies:
|
||||
log(f" Running ETF strategy: {name}")
|
||||
try:
|
||||
base_name = name.removeprefix("sim_")
|
||||
etf_tickers = ETF_STRATEGY_UNIVERSES[base_name]
|
||||
all_etf = sorted(set(etf_tickers + ["SPY"]))
|
||||
|
||||
etf_data = data_manager.update("etfs", price_request(market, all_etf), with_open=False)
|
||||
if isinstance(etf_data, tuple):
|
||||
etf_data = etf_data[0]
|
||||
etf_data = convert_quote_currency(etf_data)
|
||||
if years:
|
||||
cutoff = etf_data.index[-1] - pd.DateOffset(years=years)
|
||||
etf_data = etf_data[etf_data.index >= cutoff]
|
||||
|
||||
strategy = STRATEGY_REGISTRY[name]()
|
||||
|
||||
tradable = filter_tradable_tickers(etf_data, etf_tickers)
|
||||
equity = backtest(
|
||||
strategy, etf_data[tradable],
|
||||
initial_capital=capital,
|
||||
transaction_cost=tx_cost,
|
||||
fixed_fee=fixed_fee,
|
||||
fee_base=fee_base,
|
||||
fee_per_share=fee_per_share,
|
||||
min_trade_value=MIN_TRADE_VALUE,
|
||||
)
|
||||
|
||||
weights = strategy.generate_signals(etf_data[tradable])
|
||||
weights = weights.reindex(etf_data[tradable].index).fillna(0.0)
|
||||
trade_events = compute_rebalance_trades(weights)
|
||||
|
||||
raw_strategies.append({"name": name, "equity": equity, "trades": trade_events})
|
||||
log(f" OK: {name} (final: ${equity.iloc[-1]:,.0f})")
|
||||
except Exception as e:
|
||||
log(f" FAILED: {name}: {e}")
|
||||
|
||||
if not stock_strategies:
|
||||
benchmark_label = "SPY (Benchmark)"
|
||||
spy_data = data_manager.update("etfs", ["SPY"], with_open=False)
|
||||
if isinstance(spy_data, tuple):
|
||||
spy_data = spy_data[0]
|
||||
if years:
|
||||
cutoff = spy_data.index[-1] - pd.DateOffset(years=years)
|
||||
spy_data = spy_data[spy_data.index >= cutoff]
|
||||
if "SPY" in spy_data.columns:
|
||||
bench_curve = benchmark_curve(spy_data["SPY"], spy_data.index, capital)
|
||||
start_date = spy_data.index[0].strftime("%Y-%m-%d")
|
||||
end_date = spy_data.index[-1].strftime("%Y-%m-%d")
|
||||
|
||||
if not raw_strategies:
|
||||
return None
|
||||
|
||||
return {
|
||||
"market": market,
|
||||
"strategy_runs": raw_strategies,
|
||||
"benchmark_label": benchmark_label,
|
||||
"benchmark_curve": bench_curve,
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"tx_cost": tx_cost,
|
||||
"fixed_fee": fixed_fee,
|
||||
}
|
||||
|
||||
|
||||
def build_scenario_for_market(
|
||||
raw: dict,
|
||||
scenario: dict,
|
||||
@@ -358,7 +176,7 @@ def main():
|
||||
log("=" * 60)
|
||||
# US fee schedule (IBKR-style tiered):
|
||||
# fee = max(2.0, 1.88 + 0.009 * shares)
|
||||
us_raw = run_market_raw(
|
||||
us_raw = run_market(
|
||||
"us", us_strategies,
|
||||
years=args.years, capital=us_capital,
|
||||
tx_cost=0.001, fixed_fee=2.0,
|
||||
@@ -368,7 +186,7 @@ def main():
|
||||
log("=" * 60)
|
||||
log(f"CN MARKET: {len(cn_strategies)} strategies @ ¥{cn_capital:,.0f}")
|
||||
log("=" * 60)
|
||||
cn_raw = run_market_raw(
|
||||
cn_raw = run_market(
|
||||
"cn", cn_strategies,
|
||||
years=args.years, capital=cn_capital,
|
||||
tx_cost=0.001, fixed_fee=5.0,
|
||||
|
||||
@@ -10,5 +10,6 @@ dependencies = [
|
||||
"numpy>=2.4.0",
|
||||
"pandas>=2.3.3",
|
||||
"scipy>=1.17.1",
|
||||
"xlrd>=2.0.2",
|
||||
"yfinance>=1.0",
|
||||
]
|
||||
|
||||
0
research/lab/__init__.py
Normal file
0
research/lab/__init__.py
Normal file
47
research/lab/baselines.py
Normal file
47
research/lab/baselines.py
Normal file
@@ -0,0 +1,47 @@
|
||||
"""
|
||||
Re-run the existing factor_combo champions under the research engine, with
|
||||
the same rules every new candidate gets: next-open execution, PIT universe,
|
||||
costs, and averaging over rebalance phases.
|
||||
|
||||
PYTHONPATH=. uv run python -m research.lab.baselines --market us
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import warnings
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.factor_scan import load, phase_avg
|
||||
from research.lab.report import fmt
|
||||
from strategies.factor_combo import SIGNAL_REGISTRY
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
CHAMPIONS = {
|
||||
"us": ["rec_mfilt+deep_upvol", "ma200+mom7m+rec126", "rec_mfilt+ma200", "mom7m+rec126"],
|
||||
"cn": ["up_cap+quality_mom", "down_resil+qual_mom", "rec63+mom_gap", "up_cap+mom_gap"],
|
||||
}
|
||||
|
||||
|
||||
def baseline_scores(p, market: str) -> dict[str, pd.DataFrame]:
|
||||
mk = "us" if market == "us" else "cn"
|
||||
# factor_combo signals were built on prices masked to index membership
|
||||
px = p.close.where(p.member).ffill(limit=5)
|
||||
return {name: SIGNAL_REGISTRY[name](px) for name in CHAMPIONS[mk]}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--market", default="us")
|
||||
args = ap.parse_args()
|
||||
p, periods = load(args.market)
|
||||
scores = baseline_scores(p, args.market)
|
||||
for name, s in scores.items():
|
||||
for every, label in [(5, "weekly"), (21, "monthly")]:
|
||||
for per, (a, b) in periods.items():
|
||||
st = phase_avg(p, s, a, b, 10, every)
|
||||
print(f"{name:22s} top10 {label:8s} {per:8s} {fmt(st)}", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
61
research/lab/construction.py
Normal file
61
research/lab/construction.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""
|
||||
Construction sweep: holdings count, buffer (hysteresis), rebalance interval and
|
||||
weighting for a fixed score, reported per ~5-year block.
|
||||
|
||||
PYTHONPATH=. uv run python -m research.lab.construction --market us --combo overnight_mom+rec_126
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import itertools
|
||||
import warnings
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.baselines import baseline_scores
|
||||
from research.lab.factor_scan import load, phase_avg
|
||||
from research.lab.factors import FACTORS, zrank
|
||||
from research.lab.stepwise import SUBPERIODS
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
|
||||
def combo_score(p, combo: str, market: str) -> pd.DataFrame:
|
||||
if combo.startswith("baseline:"):
|
||||
return baseline_scores(p, market)[combo.split(":", 1)[1]]
|
||||
parts = combo.split("+")
|
||||
return sum(zrank(FACTORS[n](p), p.member) for n in parts) / len(parts)
|
||||
|
||||
|
||||
def block_stats(p, score, subs, **kw) -> dict:
|
||||
per = [phase_avg(p, score, a, b, **kw) for a, b in subs]
|
||||
return {"sharpe": np.mean([x["sharpe"] for x in per]), "min_sh": min(x["sharpe"] for x in per),
|
||||
"cagr": np.mean([x["cagr"] for x in per]), "mdd": min(x["mdd"] for x in per),
|
||||
"turn": np.mean([x["turnover"] for x in per]),
|
||||
"blocks": " ".join(f"{x['sharpe']:.2f}/{x['cagr']*100:.0f}%" for x in per)}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--market", default="us")
|
||||
ap.add_argument("--combo", nargs="+", required=True)
|
||||
ap.add_argument("--tops", nargs="*", type=int, default=[10, 15, 20, 30])
|
||||
ap.add_argument("--buffers", nargs="*", type=float, default=[0, 0.5, 1.0, 2.0])
|
||||
ap.add_argument("--everys", nargs="*", type=int, default=[5, 10])
|
||||
ap.add_argument("--weightings", nargs="*", default=["equal", "invvol"])
|
||||
args = ap.parse_args()
|
||||
mk = "us" if args.market == "us" else "cn"
|
||||
p, _ = load(args.market)
|
||||
subs = SUBPERIODS[mk]
|
||||
for combo in args.combo:
|
||||
s = combo_score(p, combo, args.market)
|
||||
print(f"=== {combo}")
|
||||
for top, buf, every, wt in itertools.product(args.tops, args.buffers, args.everys, args.weightings):
|
||||
st = block_stats(p, s, subs, top_n=top, every=every, buffer=buf, weighting=wt)
|
||||
print(f" top{top:<3d} buf{buf:<4} every{every:<3d} {wt:7s} sh {st['sharpe']:.2f} min {st['min_sh']:.2f} "
|
||||
f"cagr {st['cagr']*100:5.1f}% worstMDD {st['mdd']*100:5.1f}% turn {st['turn']:4.1f}x [{st['blocks']}]",
|
||||
flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
252
research/lab/engine.py
Normal file
252
research/lab/engine.py
Normal file
@@ -0,0 +1,252 @@
|
||||
"""
|
||||
Event-driven long-only portfolio simulator for factor research.
|
||||
|
||||
Differences from ``main.backtest`` (the engine behind the web UI numbers):
|
||||
|
||||
* Execution lag. A score computed from the close of day s is traded at the
|
||||
OPEN of day s+1 (``exec_at="open"``) or the CLOSE of day s+1
|
||||
(``exec_at="close"``). ``main.backtest`` fills at the same close the signal
|
||||
saw, which the live trader cannot do.
|
||||
* Holdings drift between rebalances; costs are charged on actual trades.
|
||||
* Tradability. A stock that is suspended, or opens at limit-up (buy) or
|
||||
limit-down (sell) on the execution day keeps its current position. Stocks
|
||||
outside the point-in-time index on the signal date are never bought.
|
||||
* Costs are asymmetric (CN stamp tax on sells) and include an optional fixed
|
||||
per-order fee expressed against a reference capital.
|
||||
|
||||
Conventions: every DataFrame shares one date index and one column set
|
||||
(``Panel.close.columns``). ``score`` is "higher is better"; NaN means
|
||||
"not eligible".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
@dataclass
|
||||
class Costs:
|
||||
buy: float = 0.0010 # fraction of traded value
|
||||
sell: float = 0.0010
|
||||
sell_tax: pd.Series | None = None # extra sell-side tax by date (CN stamp duty)
|
||||
fixed_fee: float = 0.0 # currency per order
|
||||
capital: float = 10_000.0 # reference capital for the fixed fee
|
||||
|
||||
|
||||
@dataclass
|
||||
class Panel:
|
||||
market: str
|
||||
open: pd.DataFrame
|
||||
high: pd.DataFrame
|
||||
low: pd.DataFrame
|
||||
close: pd.DataFrame
|
||||
volume: pd.DataFrame
|
||||
member: pd.DataFrame # bool: in PIT index on that date
|
||||
can_buy: pd.DataFrame # bool: buyable at that day's open
|
||||
can_sell: pd.DataFrame # bool: sellable at that day's open
|
||||
cash_ret: pd.Series # daily return of the idle-cash asset
|
||||
benchmark: pd.Series # index level (cap-weighted)
|
||||
costs: Costs = field(default_factory=Costs)
|
||||
extra: dict = field(default_factory=dict) # market-specific fields (turn, pe, ...)
|
||||
|
||||
@property
|
||||
def dates(self) -> pd.DatetimeIndex:
|
||||
return self.close.index
|
||||
|
||||
def slice(self, start=None, end=None) -> "Panel":
|
||||
idx = self.dates
|
||||
m = np.ones(len(idx), bool)
|
||||
if start is not None:
|
||||
m &= idx >= pd.Timestamp(start)
|
||||
if end is not None:
|
||||
m &= idx <= pd.Timestamp(end)
|
||||
def sub(x):
|
||||
if isinstance(x, (pd.Series, pd.DataFrame)) and x.index.equals(idx):
|
||||
return x.loc[m]
|
||||
return x
|
||||
return Panel(self.market, sub(self.open), sub(self.high), sub(self.low), sub(self.close),
|
||||
sub(self.volume), sub(self.member), sub(self.can_buy), sub(self.can_sell),
|
||||
sub(self.cash_ret), sub(self.benchmark), self.costs,
|
||||
{k: sub(v) for k, v in self.extra.items() if not k.startswith("_")})
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
equity: pd.Series
|
||||
returns: pd.Series
|
||||
turnover: pd.Series # one-way traded value / equity, per day
|
||||
holdings: pd.DataFrame # weights after each rebalance (rows = exec dates)
|
||||
exposure: pd.Series # invested fraction at each close
|
||||
name: str = ""
|
||||
|
||||
|
||||
def rebalance_dates(dates: pd.DatetimeIndex, every: int | str, offset: int = 0,
|
||||
start: int = 0) -> np.ndarray:
|
||||
"""Boolean mask of signal dates.
|
||||
|
||||
``every`` is a trading-day interval (5 = weekly) or "W"/"M" for the last
|
||||
trading day of each calendar week/month.
|
||||
"""
|
||||
mask = np.zeros(len(dates), bool)
|
||||
if isinstance(every, str):
|
||||
key = dates.to_period(every[0])
|
||||
last = pd.Series(np.arange(len(dates)), index=dates).groupby(key).max().values
|
||||
mask[last] = True
|
||||
mask[:start] = False
|
||||
return mask
|
||||
mask[start + offset::every] = True
|
||||
return mask
|
||||
|
||||
|
||||
def select_top(score_row: np.ndarray, top_n: int, held: np.ndarray | None = None,
|
||||
buffer: float = 0.0) -> np.ndarray:
|
||||
"""Indices of the chosen names. With a buffer, current holdings stay while
|
||||
their rank is within ``top_n * (1 + buffer)``."""
|
||||
valid = np.where(~np.isnan(score_row))[0]
|
||||
if len(valid) == 0:
|
||||
return valid
|
||||
order = valid[np.argsort(-score_row[valid], kind="stable")]
|
||||
if buffer <= 0 or held is None or not held.any():
|
||||
return order[:top_n]
|
||||
rank = np.empty(len(score_row)); rank[:] = np.inf
|
||||
rank[order] = np.arange(len(order))
|
||||
keep = [i for i in np.where(held)[0] if rank[i] < top_n * (1 + buffer)]
|
||||
keep = sorted(keep, key=lambda i: rank[i])[:top_n]
|
||||
chosen = list(keep)
|
||||
for i in order:
|
||||
if len(chosen) >= top_n:
|
||||
break
|
||||
if i not in keep:
|
||||
chosen.append(i)
|
||||
return np.array(chosen, int)
|
||||
|
||||
|
||||
def simulate(panel: Panel, score: pd.DataFrame | None = None, top_n: int = 10,
|
||||
every: int | str = 5, offset: int = 0, warmup: int = 0,
|
||||
weighting: str = "equal", buffer: float = 0.0,
|
||||
exposure: pd.Series | None = None, exec_at: str = "open",
|
||||
target_weights: pd.DataFrame | None = None, min_trade: float = 0.002,
|
||||
name: str = "") -> Result:
|
||||
"""Run a long-only top-N strategy.
|
||||
|
||||
score : higher = better, evaluated at each signal date's close.
|
||||
target_weights : alternative to score — explicit weights per signal date
|
||||
(rows need only exist on signal dates; NaN rows skip).
|
||||
weighting : "equal" | "invvol" (60-day vol) | "score" (score-proportional, clipped >0)
|
||||
exposure : fraction of equity to invest, read on the signal date
|
||||
(e.g. a market-trend or vol-target overlay); rest earns cash_ret.
|
||||
exec_at : "open" → trade at next day's open; "close" → next day's close.
|
||||
min_trade : resizing trades smaller than this fraction of equity are
|
||||
skipped (the live trader skips tiny orders too); full
|
||||
entries and exits always trade.
|
||||
"""
|
||||
dates = panel.dates
|
||||
n_days, n_assets = panel.close.shape
|
||||
C = panel.close.ffill().values
|
||||
O = panel.open.values
|
||||
O = np.where(np.isnan(O), C, O) # missing open → fall back to close
|
||||
prevC = np.vstack([C[:1], C[:-1]])
|
||||
member = panel.member.values
|
||||
can_buy = panel.can_buy.values
|
||||
can_sell = panel.can_sell.values
|
||||
cash_ret = panel.cash_ret.reindex(dates).fillna(0.0).values
|
||||
sell_tax = (panel.costs.sell_tax.reindex(dates).fillna(0.0).values
|
||||
if panel.costs.sell_tax is not None else np.zeros(n_days))
|
||||
S = score.reindex(index=dates, columns=panel.close.columns).values if score is not None else None
|
||||
TW = (target_weights.reindex(index=dates, columns=panel.close.columns).values
|
||||
if target_weights is not None else None)
|
||||
expo = exposure.reindex(dates).ffill().fillna(1.0).values if exposure is not None else np.ones(n_days)
|
||||
if weighting == "invvol":
|
||||
rets = panel.close.pct_change(fill_method=None)
|
||||
vol = rets.rolling(60, min_periods=40).std().values
|
||||
sig_mask = rebalance_dates(dates, every, offset, start=warmup)
|
||||
|
||||
h = np.zeros(n_assets) # value held per asset (at last mark)
|
||||
cash = 1.0
|
||||
eq = np.empty(n_days); turn = np.zeros(n_days); inv = np.zeros(n_days)
|
||||
hold_rows, hold_idx = [], []
|
||||
pending = None # target weight vector awaiting execution
|
||||
|
||||
for d in range(n_days):
|
||||
# mark overnight / full-day moves
|
||||
with np.errstate(invalid="ignore", divide="ignore"):
|
||||
gap = np.where(prevC[d] > 0, O[d] / prevC[d], 1.0)
|
||||
intraday = np.where(O[d] > 0, C[d] / O[d], 1.0)
|
||||
gap = np.nan_to_num(gap, nan=1.0, posinf=1.0)
|
||||
intraday = np.nan_to_num(intraday, nan=1.0, posinf=1.0)
|
||||
cash *= 1 + cash_ret[d]
|
||||
|
||||
if pending is not None:
|
||||
if exec_at == "open":
|
||||
h *= gap
|
||||
else:
|
||||
h *= gap * intraday
|
||||
V = h.sum() + cash
|
||||
tgt = pending * V
|
||||
# untradable names keep their current value
|
||||
frozen_buy = (tgt > h) & ~can_buy[d]
|
||||
frozen_sell = (tgt < h) & ~can_sell[d]
|
||||
frozen = frozen_buy | frozen_sell
|
||||
if frozen.any():
|
||||
tgt[frozen] = h[frozen]
|
||||
free = ~frozen & (pending > 0)
|
||||
room = V - tgt[frozen].sum()
|
||||
if free.any() and pending[free].sum() > 0:
|
||||
budget = min(room, pending[free].sum() * V)
|
||||
tgt[free] = pending[free] / pending[free].sum() * max(budget, 0.0)
|
||||
if min_trade > 0:
|
||||
tiny = (np.abs(tgt - h) < min_trade * V) & (tgt > 0) & (h > 0)
|
||||
tgt[tiny] = h[tiny]
|
||||
trade = tgt - h
|
||||
buys = trade[trade > 0].sum()
|
||||
sells = -trade[trade < 0].sum()
|
||||
n_orders = int((np.abs(trade) > 1e-4 * V).sum())
|
||||
cost = buys * panel.costs.buy + sells * (panel.costs.sell + sell_tax[d])
|
||||
# Equity is normalized to 1.0 == costs.capital, so a fixed fee per
|
||||
# order is fixed_fee / capital in these units at any book size.
|
||||
if panel.costs.fixed_fee > 0:
|
||||
cost += n_orders * panel.costs.fixed_fee / panel.costs.capital
|
||||
cash += h.sum() - tgt.sum() - cost
|
||||
h = tgt
|
||||
turn[d] = (buys + sells) / 2 / V if V > 0 else 0.0
|
||||
hold_rows.append(h / max(h.sum() + cash, 1e-12)); hold_idx.append(dates[d])
|
||||
if exec_at == "open":
|
||||
h *= intraday
|
||||
pending = None
|
||||
else:
|
||||
h *= gap * intraday
|
||||
|
||||
V = h.sum() + cash
|
||||
eq[d] = V
|
||||
inv[d] = h.sum() / V if V > 0 else 0.0
|
||||
|
||||
if sig_mask[d] and d + 1 < n_days:
|
||||
if TW is not None:
|
||||
w = np.nan_to_num(TW[d], nan=0.0)
|
||||
if np.isnan(TW[d]).all():
|
||||
continue
|
||||
else:
|
||||
row = np.where(member[d], S[d], np.nan)
|
||||
held = h > 1e-12
|
||||
idx = select_top(row, top_n, held, buffer)
|
||||
w = np.zeros(n_assets)
|
||||
if len(idx) >= min(max(1, top_n // 2), 20):
|
||||
if weighting == "invvol":
|
||||
iv = 1.0 / np.where(vol[d, idx] > 0, vol[d, idx], np.nan)
|
||||
iv = np.nan_to_num(iv, nan=np.nanmean(iv) if np.isfinite(iv).any() else 1.0)
|
||||
w[idx] = iv / iv.sum()
|
||||
elif weighting == "score":
|
||||
s = np.clip(row[idx] - np.nanmin(row[idx]) + 1e-6, 0, None)
|
||||
w[idx] = s / s.sum()
|
||||
else:
|
||||
w[idx] = 1.0 / len(idx)
|
||||
pending = w * float(np.clip(expo[d], 0.0, 1.0))
|
||||
|
||||
equity = pd.Series(eq, index=dates)
|
||||
rets = equity.pct_change().fillna(0.0)
|
||||
holdings = pd.DataFrame(hold_rows, index=hold_idx, columns=panel.close.columns)
|
||||
return Result(equity, rets, pd.Series(turn, index=dates), holdings,
|
||||
pd.Series(inv, index=dates), name)
|
||||
143
research/lab/evaluate.py
Normal file
143
research/lab/evaluate.py
Normal file
@@ -0,0 +1,143 @@
|
||||
"""
|
||||
Final evaluation of candidate strategies against the existing champions,
|
||||
under identical rules (PIT universe, next-open execution, costs, phase
|
||||
averaging). Writes data/research/results/final_{market}.json.
|
||||
|
||||
PYTHONPATH=. uv run python -m research.lab.evaluate --market us
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import warnings
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab import overlays as ov
|
||||
from research.lab.construction import combo_score
|
||||
from research.lab.engine import Costs, simulate
|
||||
from research.lab.factor_scan import RESULTS, load
|
||||
from research.lab.report import deflated_sharpe, stats, yearly
|
||||
from research.lab.stepwise import SUBPERIODS
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
# name -> (combo, construction). "baseline:" combos are the current factor_combo champions.
|
||||
CANDIDATES = {
|
||||
"us": {
|
||||
"champion rec_mfilt+deep_upvol monthly": ("baseline:rec_mfilt+deep_upvol", dict(top_n=10, every=21)),
|
||||
"champion mom7m+rec126 monthly": ("baseline:mom7m+rec126", dict(top_n=10, every=21)),
|
||||
"champion mom7m+rec126 weekly": ("baseline:mom7m+rec126", dict(top_n=10, every=5)),
|
||||
"EW S&P 500 (PIT)": ("ew", dict(top_n=10000, every=5)),
|
||||
"new overnight+rec": ("overnight_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
|
||||
"new overnight6+rec": ("overnight_mom_6+rec_126", dict(top_n=10, every=10, buffer=2.0)),
|
||||
"new smooth-mom4": ("overnight_mom+overnight_mom_6+amp_cut_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
|
||||
"new overnight+ampcut+rec": ("overnight_mom+amp_cut_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
|
||||
},
|
||||
"cn": {
|
||||
"champion up_cap+quality_mom weekly": ("baseline:up_cap+quality_mom", dict(top_n=10, every=5)),
|
||||
"champion up_cap+mom_gap weekly": ("baseline:up_cap+mom_gap", dict(top_n=10, every=5)),
|
||||
"champion down_resil+qual_mom biweekly": ("baseline:down_resil+qual_mom", dict(top_n=10, every=10)),
|
||||
"EW CSI 300 (PIT)": ("ew", dict(top_n=10000, every=5)),
|
||||
},
|
||||
}
|
||||
LIVE_FEES = {"us": (2.0, 10_000.0), "cn": (5.0, 70_000.0)}
|
||||
WINDOW = ("2016-09-27", "2026-09-24") # the web UI's 10-year window
|
||||
|
||||
|
||||
def score_for(p, combo, market):
|
||||
if combo == "ew":
|
||||
return p.close.notna().astype(float).where(p.member)
|
||||
return combo_score(p, combo, market)
|
||||
|
||||
|
||||
def phase_runs(p, score, a, b, exposure=None, **kw):
|
||||
every = kw.get("every", 5)
|
||||
w0 = p.dates.searchsorted(pd.Timestamp(a))
|
||||
sp = p.slice(p.dates[max(w0 - 5, 0)], b)
|
||||
runs = []
|
||||
for off in range(every if kw.get("top_n", 10) < 1000 else 1):
|
||||
r = simulate(sp, score, offset=off, warmup=5, exposure=exposure, **kw)
|
||||
runs.append(r)
|
||||
return runs
|
||||
|
||||
|
||||
def summarize(runs, a, b) -> dict:
|
||||
sts = [stats(r.equity.loc[a:b], r.turnover) for r in runs]
|
||||
df = pd.DataFrame(sts)
|
||||
rets = pd.concat([r.returns.loc[a:b] for r in runs], axis=1).mean(axis=1)
|
||||
eq = (1 + rets).cumprod()
|
||||
out = {k: float(df[k].mean()) for k in df.columns}
|
||||
out["cagr_phase_min"] = float(df["cagr"].min())
|
||||
out["cagr_phase_max"] = float(df["cagr"].max())
|
||||
out["yearly"] = {str(k.year): float(v) for k, v in yearly(eq).items()}
|
||||
out["_rets"] = rets
|
||||
return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--market", default="us")
|
||||
ap.add_argument("--add", nargs="*", default=[], help="extra 'name=combo|top|every|buffer'")
|
||||
ap.add_argument("--n-trials", type=int, default=0, help="override trial count for DSR")
|
||||
args = ap.parse_args()
|
||||
mk = "us" if args.market == "us" else "cn"
|
||||
p, _ = load(args.market)
|
||||
cands = dict(CANDIDATES[mk])
|
||||
for spec in args.add:
|
||||
name, rest = spec.split("=", 1)
|
||||
combo, top, every, buf = rest.split("|")
|
||||
cands[name] = (combo, dict(top_n=int(top), every=int(every), buffer=float(buf)))
|
||||
trials_path = os.path.join(RESULTS, f"trials_{args.market}.csv")
|
||||
n_trials = args.n_trials or (len(pd.read_csv(trials_path)) if os.path.exists(trials_path) else 100)
|
||||
full = (SUBPERIODS[mk][0][0], WINDOW[1])
|
||||
out = {}
|
||||
fee, cap = LIVE_FEES[mk]
|
||||
for name, (combo, kw) in cands.items():
|
||||
s = score_for(p, combo, args.market)
|
||||
res = {"combo": combo, "construction": kw}
|
||||
base_runs = phase_runs(p, s, *WINDOW, **kw)
|
||||
res["window"] = summarize(base_runs, *WINDOW)
|
||||
res["full"] = summarize(phase_runs(p, s, *full, **kw), *full)
|
||||
res["blocks"] = [summarize(phase_runs(p, s, a, b, **kw), a, b) for a, b in SUBPERIODS[mk]]
|
||||
# live-book fixed fee
|
||||
pf = p.slice(); pf.costs = Costs(p.costs.buy, p.costs.sell, p.costs.sell_tax, fee, cap)
|
||||
res["window_livefee"] = summarize(phase_runs(pf, s, *WINDOW, **kw), *WINDOW)
|
||||
# double proportional cost
|
||||
p2 = p.slice(); p2.costs = Costs(p.costs.buy * 2, p.costs.sell * 2, p.costs.sell_tax, 0.0, cap)
|
||||
res["window_2xcost"] = summarize(phase_runs(p2, s, *WINDOW, **kw), *WINDOW)
|
||||
res["window_exec_close"] = summarize(phase_runs(p, s, *WINDOW, exec_at="close", **kw), *WINDOW)
|
||||
# drawdown-control overlay: slow/fast market trend blend on the EW index
|
||||
ew = ov.ew_index(p)
|
||||
slow = (ew / ew.shift(252) > 1).astype(float)
|
||||
fast = (ew / ew.shift(21) > 1).astype(float)
|
||||
expo = (0.5 + 0.25 * slow + 0.25 * fast).where(slow + fast < 2, 1.0)
|
||||
res["window_trend_overlay"] = summarize(phase_runs(p, s, *WINDOW, exposure=expo, **kw), *WINDOW)
|
||||
dsr = deflated_sharpe(res["full"]["_rets"], n_trials)
|
||||
res["dsr_full"] = dsr
|
||||
out[name] = res
|
||||
w, f = res["window"], res["full"]
|
||||
print(f"{name:42s} 2016-26: CAGR {w['cagr']*100:5.1f}% [{w['cagr_phase_min']*100:4.1f},{w['cagr_phase_max']*100:4.1f}] "
|
||||
f"Sh {w['sharpe']:.2f} MDD {w['mdd']*100:5.1f}% | full: CAGR {f['cagr']*100:5.1f}% Sh {f['sharpe']:.2f} "
|
||||
f"MDD {f['mdd']*100:5.1f}% | blocks Sh {' '.join(f'{b_['sharpe']:.2f}' for b_ in res['blocks'])} "
|
||||
f"| livefee {res['window_livefee']['cagr']*100:5.1f}% 2x {res['window_2xcost']['cagr']*100:5.1f}% "
|
||||
f"close {res['window_exec_close']['cagr']*100:5.1f}% trend {res['window_trend_overlay']['cagr']*100:5.1f}%/"
|
||||
f"{res['window_trend_overlay']['sharpe']:.2f}/{res['window_trend_overlay']['mdd']*100:.0f}% "
|
||||
f"| DSR {dsr['dsr']:.2f} (N={n_trials})", flush=True)
|
||||
|
||||
def clean(o):
|
||||
if isinstance(o, dict):
|
||||
return {k: clean(v) for k, v in o.items() if not k.startswith("_")}
|
||||
if isinstance(o, list):
|
||||
return [clean(v) for v in o]
|
||||
if isinstance(o, (np.floating, np.integer)):
|
||||
return float(o)
|
||||
return o
|
||||
os.makedirs(RESULTS, exist_ok=True)
|
||||
with open(os.path.join(RESULTS, f"final_{args.market}.json"), "w") as fh:
|
||||
json.dump(clean(out), fh, indent=1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
78
research/lab/export_curves.py
Normal file
78
research/lab/export_curves.py
Normal file
@@ -0,0 +1,78 @@
|
||||
"""
|
||||
Export phase-averaged equity curves of the final candidates and baselines for
|
||||
the report (weekly samples), plus the US/CN correlation of the new strategies.
|
||||
|
||||
PYTHONPATH=. uv run python -m research.lab.export_curves
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import warnings
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.evaluate import phase_runs, score_for
|
||||
from research.lab.factor_scan import RESULTS, load
|
||||
from research.lab.report import stats
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
SETS = {
|
||||
"us": ("2006-09-27", "2026-09-24", {
|
||||
"新: overnight+recovery": ("overnight_mom+rec_126", dict(top_n=10, every=10, buffer=2.0)),
|
||||
"现: rec_mfilt+deep_upvol 月频": ("baseline:rec_mfilt+deep_upvol", dict(top_n=10, every=21)),
|
||||
"现: mom7m+rec126 月频": ("baseline:mom7m+rec126", dict(top_n=10, every=21)),
|
||||
"等权 S&P 500 (PIT)": ("ew", dict(top_n=10000, every=5)),
|
||||
}),
|
||||
"cn_hs300": ("2010-01-04", "2026-09-24", {
|
||||
"新: 低风险动量4因子": ("amp_cut_mom+low_beta+pv_corr+sharpe_mom", dict(top_n=15, every=10, buffer=2.0)),
|
||||
"新: low_beta+sharpe_mom": ("low_beta+sharpe_mom", dict(top_n=20, every=10, buffer=1.0)),
|
||||
"现: down_resil+qual_mom 双周": ("baseline:down_resil+qual_mom", dict(top_n=10, every=10)),
|
||||
"现: up_cap+quality_mom 周频": ("baseline:up_cap+quality_mom", dict(top_n=10, every=5)),
|
||||
"等权 CSI 300 (PIT)": ("ew", dict(top_n=10000, every=5)),
|
||||
}),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
out, daily = {}, {}
|
||||
for market, (a, b, cands) in SETS.items():
|
||||
p, _ = load(market)
|
||||
curves = {}
|
||||
for name, (combo, kw) in cands.items():
|
||||
s = score_for(p, combo, market)
|
||||
runs = phase_runs(p, s, a, b, **kw)
|
||||
rets = pd.concat([r.returns.loc[a:b] for r in runs], axis=1).mean(axis=1)
|
||||
daily[(market, name)] = rets
|
||||
eq = (1 + rets).cumprod()
|
||||
wk = eq.resample("W-FRI").last().dropna()
|
||||
dd = eq / eq.cummax() - 1
|
||||
curves[name] = {"dates": [d.strftime("%Y-%m-%d") for d in wk.index],
|
||||
"equity": [round(float(v), 4) for v in wk.values],
|
||||
"drawdown": [round(float(v), 4) for v in dd.resample("W-FRI").min().dropna().values],
|
||||
"stats": {k: float(v) for k, v in stats(eq).items()}}
|
||||
print(market, name, {k: round(v, 3) for k, v in curves[name]["stats"].items()}, flush=True)
|
||||
out[market] = curves
|
||||
us = daily[("us", "新: overnight+recovery")]
|
||||
cn = daily[("cn_hs300", "新: 低风险动量4因子")]
|
||||
# compound each market on its own calendar before joining: the exchanges
|
||||
# have different holidays, and an inner join on days would drop returns
|
||||
m = pd.concat([us, cn], axis=1, keys=["us", "cn"]).dropna().loc["2016-09-27":]
|
||||
mon = pd.concat({k: (1 + v.loc["2016-09-27":]).resample("ME").prod() - 1
|
||||
for k, v in (("us", us), ("cn", cn))}, axis=1).dropna()
|
||||
cross = {"daily_corr": float(m.corr().iloc[0, 1]), "monthly_corr": float(mon.corr().iloc[0, 1])}
|
||||
for col in ["us", "cn", "5050"]:
|
||||
r = mon.mean(axis=1) if col == "5050" else mon[col]
|
||||
eq = (1 + r).cumprod()
|
||||
cross[col] = {"cagr": float(eq.iloc[-1] ** (12 / len(r)) - 1),
|
||||
"sharpe_monthly": float(r.mean() / r.std() * np.sqrt(12)),
|
||||
"mdd_monthly": float((eq / eq.cummax() - 1).min())}
|
||||
out["cross"] = cross
|
||||
print("cross", out["cross"])
|
||||
with open(os.path.join(RESULTS, "curves.json"), "w") as fh:
|
||||
json.dump(out, fh, ensure_ascii=False, default=float)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
95
research/lab/factor_scan.py
Normal file
95
research/lab/factor_scan.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""
|
||||
Single-factor scan: rank IC and long-only top-N backtests per period.
|
||||
|
||||
PYTHONPATH=. uv run python -m research.lab.factor_scan --market us
|
||||
PYTHONPATH=. uv run python -m research.lab.factor_scan --market cn_hs300
|
||||
|
||||
Writes data/research/results/factor_scan_{market}.csv.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import time
|
||||
import warnings
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from research.lab import panel as pnl
|
||||
from research.lab.engine import simulate
|
||||
from research.lab.factors import FACTORS
|
||||
from research.lab.report import (forward_returns, ic_series, ic_summary, stats,
|
||||
top_minus_universe)
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
PERIODS = {
|
||||
"us": {"design": ("2006-01-01", "2016-09-26"), "holdout": ("2016-09-27", "2026-09-24")},
|
||||
"cn": {"design": ("2010-01-01", "2016-09-26"), "holdout": ("2016-09-27", "2026-09-24")},
|
||||
}
|
||||
RESULTS = os.path.join("data", "research", "results")
|
||||
|
||||
|
||||
def load(market: str):
|
||||
if market == "us":
|
||||
return pnl.load_us(), PERIODS["us"]
|
||||
return pnl.load_cn(market.split("_", 1)[1]), PERIODS["cn"]
|
||||
|
||||
|
||||
def phase_avg(p, score, start, end, top_n, every, **kw) -> dict:
|
||||
"""Average stats over every rebalance phase (offset 0..every-1)."""
|
||||
w0 = p.dates.searchsorted(pd.Timestamp(start))
|
||||
sp = p.slice(p.dates[max(w0 - 5, 0)], end)
|
||||
rows = []
|
||||
for off in range(every):
|
||||
r = simulate(sp, score, top_n=top_n, every=every, offset=off, warmup=5, **kw)
|
||||
rows.append(stats(r.equity.loc[start:end], r.turnover))
|
||||
return pd.DataFrame(rows).mean().to_dict()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--market", default="us")
|
||||
ap.add_argument("--top-n", type=int, default=20)
|
||||
ap.add_argument("--every", type=int, default=5)
|
||||
ap.add_argument("--only", nargs="*")
|
||||
args = ap.parse_args()
|
||||
mk = "us" if args.market == "us" else "cn"
|
||||
p, periods = load(args.market)
|
||||
fwd5 = forward_returns(p, 5)
|
||||
fwd20 = forward_returns(p, 20)
|
||||
rows = []
|
||||
names = args.only or [n for n, f in FACTORS.items() if mk in f.markets]
|
||||
for name in names:
|
||||
t = time.time()
|
||||
try:
|
||||
s = FACTORS[name](p)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"{name}: failed {exc}")
|
||||
continue
|
||||
row = {"factor": name}
|
||||
for per, (a, b) in periods.items():
|
||||
sl = slice(a, b)
|
||||
ic5 = ic_summary(ic_series(s.loc[sl], fwd5.loc[sl], p.member.loc[sl], 5))
|
||||
ic20 = ic_summary(ic_series(s.loc[sl], fwd20.loc[sl], p.member.loc[sl], 20), 12)
|
||||
tmu = top_minus_universe(s.loc[sl], fwd5.loc[sl], p.member.loc[sl], args.top_n, 5)
|
||||
st = phase_avg(p, s, a, b, args.top_n, args.every)
|
||||
row.update({f"{per}_ic5": ic5["ic"], f"{per}_ic5_t": ic5["t"],
|
||||
f"{per}_ic20": ic20["ic"], f"{per}_ic20_t": ic20["t"],
|
||||
f"{per}_top_excess_ann": tmu.mean() * 52,
|
||||
f"{per}_cagr": st["cagr"], f"{per}_sharpe": st["sharpe"],
|
||||
f"{per}_mdd": st["mdd"], f"{per}_turn": st["turnover"]})
|
||||
rows.append(row)
|
||||
print(f"{name:16s} "
|
||||
f"D: ic5 {row['design_ic5']:+.3f} (t{row['design_ic5_t']:+.1f}) ic20 {row['design_ic20']:+.3f} "
|
||||
f"xs {row['design_top_excess_ann']*100:+5.1f}% cagr {row['design_cagr']*100:5.1f}% sh {row['design_sharpe']:.2f} | "
|
||||
f"H: ic5 {row['holdout_ic5']:+.3f} (t{row['holdout_ic5_t']:+.1f}) ic20 {row['holdout_ic20']:+.3f} "
|
||||
f"xs {row['holdout_top_excess_ann']*100:+5.1f}% cagr {row['holdout_cagr']*100:5.1f}% sh {row['holdout_sharpe']:.2f} "
|
||||
f"turn {row['holdout_turn']:.0f}x [{time.time()-t:.0f}s]", flush=True)
|
||||
# equal-weight universe reference
|
||||
os.makedirs(RESULTS, exist_ok=True)
|
||||
out = pd.DataFrame(rows)
|
||||
out.to_csv(os.path.join(RESULTS, f"factor_scan_{args.market}.csv"), index=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
529
research/lab/factors.py
Normal file
529
research/lab/factors.py
Normal file
@@ -0,0 +1,529 @@
|
||||
"""
|
||||
Factor library. Every factor maps a Panel to a DataFrame (dates x assets)
|
||||
where HIGHER = MORE ATTRACTIVE and the value on date t uses data up to the
|
||||
close of t only. Sign conventions follow the literature: e.g. ``ivol`` returns
|
||||
NEGATIVE idiosyncratic volatility because low-ivol stocks earn more.
|
||||
|
||||
Windows are in trading days: 21 ≈ 1 month, 63 ≈ 1 quarter, 252 ≈ 1 year.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.engine import Panel
|
||||
|
||||
FACTORS: dict[str, callable] = {}
|
||||
|
||||
|
||||
def factor(name: str, *, markets: tuple[str, ...] = ("us", "cn")):
|
||||
def deco(fn):
|
||||
fn.markets = markets
|
||||
FACTORS[name] = fn
|
||||
return fn
|
||||
return deco
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _ret(p: Panel) -> pd.DataFrame:
|
||||
key = "_ret"
|
||||
if key not in p.extra:
|
||||
p.extra[key] = p.close.pct_change(fill_method=None)
|
||||
return p.extra[key]
|
||||
|
||||
|
||||
def _mkt(p: Panel) -> pd.Series:
|
||||
"""Equal-weight return of PIT members (the investable universe)."""
|
||||
key = "_mkt"
|
||||
if key not in p.extra:
|
||||
p.extra[key] = _ret(p).where(p.member).mean(axis=1).fillna(0.0)
|
||||
return p.extra[key]
|
||||
|
||||
|
||||
def _rolling_beta(r: pd.DataFrame, m: pd.Series, win: int, minp: int) -> pd.DataFrame:
|
||||
mr = r.mul(m, axis=0)
|
||||
e_xy = mr.rolling(win, min_periods=minp).mean()
|
||||
e_x = r.rolling(win, min_periods=minp).mean()
|
||||
e_y = m.rolling(win, min_periods=minp).mean()
|
||||
var_y = (m ** 2).rolling(win, min_periods=minp).mean() - e_y ** 2
|
||||
return (e_xy - e_x.mul(e_y, axis=0)).div(var_y, axis=0)
|
||||
|
||||
|
||||
def resid_returns(p: Panel, win: int = 252) -> pd.DataFrame:
|
||||
key = f"_resid{win}"
|
||||
if key not in p.extra:
|
||||
r = _ret(p)
|
||||
m = _mkt(p)
|
||||
beta = _rolling_beta(r, m, win, win // 2).shift(1)
|
||||
p.extra[key] = r - beta.mul(m, axis=0)
|
||||
return p.extra[key]
|
||||
|
||||
|
||||
def sector_map(p: Panel) -> pd.Series | None:
|
||||
return p.extra.get("sector")
|
||||
|
||||
|
||||
def demean_by_sector(df: pd.DataFrame, p: Panel) -> pd.DataFrame:
|
||||
sec = sector_map(p)
|
||||
if sec is None:
|
||||
return df.sub(df.where(p.member).mean(axis=1), axis=0)
|
||||
out = df.copy()
|
||||
m = df.where(p.member)
|
||||
for s in sec.dropna().unique():
|
||||
cols = [c for c in sec.index[sec == s] if c in df.columns]
|
||||
if cols:
|
||||
out[cols] = df[cols].sub(m[cols].mean(axis=1), axis=0)
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# momentum family
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@factor("mom_12_1")
|
||||
def mom_12_1(p):
|
||||
c = p.close.ffill()
|
||||
return c.shift(21) / c.shift(252) - 1
|
||||
|
||||
|
||||
@factor("mom_6_1")
|
||||
def mom_6_1(p):
|
||||
c = p.close.ffill()
|
||||
return c.shift(21) / c.shift(126) - 1
|
||||
|
||||
|
||||
@factor("mom_12_7") # Novy-Marx (2012) intermediate momentum
|
||||
def mom_12_7(p):
|
||||
c = p.close.ffill()
|
||||
return c.shift(147) / c.shift(252) - 1
|
||||
|
||||
|
||||
@factor("mom_3")
|
||||
def mom_3(p):
|
||||
c = p.close.ffill()
|
||||
return c / c.shift(63) - 1
|
||||
|
||||
|
||||
@factor("resid_mom") # Blitz, Huij & Martens (2011): residual 12-1, vol-scaled
|
||||
def resid_mom(p):
|
||||
e = resid_returns(p)
|
||||
s = e.shift(21).rolling(231, min_periods=180)
|
||||
return s.sum() / s.std()
|
||||
|
||||
|
||||
@factor("resid_mom_6")
|
||||
def resid_mom_6(p):
|
||||
e = resid_returns(p)
|
||||
s = e.shift(21).rolling(105, min_periods=80)
|
||||
return s.sum() / s.std()
|
||||
|
||||
|
||||
@factor("sharpe_mom") # 12-1 return / 12-month vol
|
||||
def sharpe_mom(p):
|
||||
r = _ret(p)
|
||||
return mom_12_1(p) / r.rolling(252, min_periods=180).std()
|
||||
|
||||
|
||||
@factor("high_52w") # George & Hwang (2004)
|
||||
def high_52w(p):
|
||||
c = p.close.ffill()
|
||||
return c / c.rolling(252, min_periods=200).max()
|
||||
|
||||
|
||||
@factor("fip") # Da, Gurun & Warachka (2014) frog-in-the-pan (continuous info)
|
||||
def fip(p):
|
||||
r = _ret(p).shift(21)
|
||||
pos = (r > 0).astype(float).where(r.notna()).rolling(231, min_periods=180).mean()
|
||||
neg = (r < 0).astype(float).where(r.notna()).rolling(231, min_periods=180).mean()
|
||||
m = mom_12_1(p)
|
||||
return -np.sign(m) * (neg - pos) # high = smooth, continuous gains
|
||||
|
||||
|
||||
@factor("mom_fip") # momentum restricted to continuous-information winners
|
||||
def mom_fip(p):
|
||||
return mom_12_1(p).rank(axis=1, pct=True) + fip(p).rank(axis=1, pct=True)
|
||||
|
||||
|
||||
@factor("rec_126") # existing house factor: distance from 126d low
|
||||
def rec_126(p):
|
||||
c = p.close.ffill()
|
||||
return c / c.rolling(126, min_periods=126).min() - 1
|
||||
|
||||
|
||||
@factor("ma200_dist")
|
||||
def ma200_dist(p):
|
||||
c = p.close.ffill()
|
||||
return c / c.rolling(200, min_periods=200).mean() - 1
|
||||
|
||||
|
||||
@factor("ts_trend") # fraction of 10 lookbacks (1..12m) with positive return
|
||||
def ts_trend(p):
|
||||
c = p.close.ffill()
|
||||
looks = [21, 42, 63, 84, 105, 126, 147, 168, 189, 210, 231, 252]
|
||||
return sum((c / c.shift(k) > 1).astype(float) for k in looks) / len(looks)
|
||||
|
||||
|
||||
@factor("sector_mom") # Moskowitz & Grinblatt (1999) industry momentum, 6-1
|
||||
def sector_mom(p):
|
||||
sec = sector_map(p)
|
||||
m = mom_6_1(p)
|
||||
if sec is None:
|
||||
return m * np.nan
|
||||
out = pd.DataFrame(np.nan, index=m.index, columns=m.columns)
|
||||
mm = m.where(p.member)
|
||||
for s in sec.dropna().unique():
|
||||
cols = [c for c in sec.index[sec == s] if c in m.columns]
|
||||
if cols:
|
||||
avg = mm[cols].mean(axis=1)
|
||||
out[cols] = np.repeat(avg.values[:, None], len(cols), axis=1)
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# reversal family
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@factor("rev_5")
|
||||
def rev_5(p):
|
||||
c = p.close.ffill()
|
||||
return -(c / c.shift(5) - 1)
|
||||
|
||||
|
||||
@factor("rev_21")
|
||||
def rev_21(p):
|
||||
c = p.close.ffill()
|
||||
return -(c / c.shift(21) - 1)
|
||||
|
||||
|
||||
@factor("rev_resid_21") # residual (beta-adjusted) 1-month reversal
|
||||
def rev_resid_21(p):
|
||||
return -resid_returns(p).rolling(21, min_periods=15).sum()
|
||||
|
||||
|
||||
@factor("rev_sector_21") # industry-adjusted reversal (Da, Liu & Schaumburg 2014 style)
|
||||
def rev_sector_21(p):
|
||||
return demean_by_sector(rev_21(p), p)
|
||||
|
||||
|
||||
@factor("overnight_mom") # Lou, Polk & Skouras (2019): overnight returns persist
|
||||
def overnight_mom(p):
|
||||
o = p.open
|
||||
cprev = p.close.ffill().shift(1)
|
||||
on = np.log(o / cprev)
|
||||
return on.rolling(252, min_periods=180).sum().shift(21)
|
||||
|
||||
|
||||
@factor("intraday_rev") # intraday component of last month reverses
|
||||
def intraday_rev(p):
|
||||
intr = np.log(p.close / p.open)
|
||||
return -intr.rolling(21, min_periods=15).sum()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# risk / lottery family (low-risk anomaly)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@factor("ivol") # Ang, Hodrick, Xing & Zhang (2006): low ivol
|
||||
def ivol(p):
|
||||
return -resid_returns(p).rolling(63, min_periods=40).std()
|
||||
|
||||
|
||||
@factor("low_vol")
|
||||
def low_vol(p):
|
||||
return -_ret(p).rolling(252, min_periods=180).std()
|
||||
|
||||
|
||||
@factor("low_beta") # Frazzini & Pedersen (2014) BAB, long leg
|
||||
def low_beta(p):
|
||||
return -_rolling_beta(_ret(p), _mkt(p), 252, 180)
|
||||
|
||||
|
||||
@factor("max_ret") # Bali, Cakici & Whitelaw (2011): avoid lottery stocks (MAX(1))
|
||||
def max_ret(p):
|
||||
return -_ret(p).rolling(21, min_periods=15).max()
|
||||
|
||||
|
||||
@factor("skew") # low skewness earns more
|
||||
def skew(p):
|
||||
return -_ret(p).rolling(252, min_periods=180).skew()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# volume / liquidity family
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _dollar_volume(p):
|
||||
if "amount" in p.extra:
|
||||
return p.extra["amount"]
|
||||
return p.close * p.volume
|
||||
|
||||
|
||||
def _turnover(p):
|
||||
if "turn" in p.extra:
|
||||
return p.extra["turn"]
|
||||
# US: no shares outstanding → dollar volume relative to its own 1y mean
|
||||
return None
|
||||
|
||||
|
||||
@factor("amihud") # Amihud (2002): illiquid earns more
|
||||
def amihud(p):
|
||||
dv = _dollar_volume(p).replace(0, np.nan)
|
||||
return np.log((_ret(p).abs() / dv).rolling(21, min_periods=15).mean())
|
||||
|
||||
|
||||
@factor("abn_volume") # Gervais, Kaniel & Mingelgrin (2001) high-volume return premium
|
||||
def abn_volume(p):
|
||||
dv = _dollar_volume(p)
|
||||
return np.log(dv.rolling(5, min_periods=4).mean() / dv.rolling(252, min_periods=180).mean())
|
||||
|
||||
|
||||
@factor("low_turnover") # CN: low turnover / low attention earns more
|
||||
def low_turnover(p):
|
||||
t = _turnover(p)
|
||||
if t is None:
|
||||
dv = _dollar_volume(p)
|
||||
return -np.log(dv.rolling(21, min_periods=15).mean() / dv.rolling(252, min_periods=180).mean())
|
||||
return -t.rolling(21, min_periods=15).mean()
|
||||
|
||||
|
||||
@factor("abn_turnover") # CN: turnover now vs 1y (negative = attention fading)
|
||||
def abn_turnover(p):
|
||||
t = _turnover(p)
|
||||
if t is None:
|
||||
return abn_volume(p) * -1
|
||||
return -np.log(t.rolling(21, min_periods=15).mean() / t.rolling(252, min_periods=180).mean())
|
||||
|
||||
|
||||
@factor("turn_vol") # CN: volatility of turnover (low is better)
|
||||
def turn_vol(p):
|
||||
t = _turnover(p)
|
||||
if t is None:
|
||||
t = _dollar_volume(p)
|
||||
s = t.rolling(21, min_periods=15)
|
||||
return -(s.std() / s.mean())
|
||||
|
||||
|
||||
@factor("pv_corr") # CN: price-volume correlation (negative corr is better)
|
||||
def pv_corr(p):
|
||||
v = _dollar_volume(p)
|
||||
return -p.close.ffill().rolling(21, min_periods=15).corr(v)
|
||||
|
||||
|
||||
@factor("upvol_ratio") # share of volume on up days, 21d
|
||||
def upvol_ratio(p):
|
||||
r = _ret(p)
|
||||
v = _dollar_volume(p)
|
||||
up = v.where(r > 0, 0.0).rolling(21, min_periods=15).sum()
|
||||
return up / v.rolling(21, min_periods=15).sum()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# value / fundamentals (CN: baostock daily ratios)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _inv(x: pd.DataFrame) -> pd.DataFrame:
|
||||
return (1.0 / x).replace([np.inf, -np.inf], np.nan)
|
||||
|
||||
|
||||
@factor("ep", markets=("cn",))
|
||||
def ep(p):
|
||||
return _inv(p.extra["pe"])
|
||||
|
||||
|
||||
@factor("bp", markets=("cn",))
|
||||
def bp(p):
|
||||
return _inv(p.extra["pb"])
|
||||
|
||||
|
||||
@factor("sp", markets=("cn",))
|
||||
def sp(p):
|
||||
return _inv(p.extra["ps"])
|
||||
|
||||
|
||||
@factor("cfp", markets=("cn",))
|
||||
def cfp(p):
|
||||
return _inv(p.extra["pcf"])
|
||||
|
||||
|
||||
@factor("roe_proxy", markets=("cn",)) # ROE = PB / PE (TTM earnings / book)
|
||||
def roe_proxy(p):
|
||||
roe = p.extra["pb"] / p.extra["pe"]
|
||||
return roe.where(p.extra["pe"] > 0)
|
||||
|
||||
|
||||
@factor("size", markets=("cn",)) # small float cap earns more (within index)
|
||||
def size(p):
|
||||
return -np.log(p.extra["float_cap"].rolling(21, min_periods=5).mean())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# composite utilities
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def zrank(df: pd.DataFrame, member: pd.DataFrame | None = None) -> pd.DataFrame:
|
||||
"""Cross-sectional percentile rank in [0, 1] among members (NaN elsewhere)."""
|
||||
if member is not None:
|
||||
df = df.where(member)
|
||||
return df.rank(axis=1, pct=True)
|
||||
|
||||
|
||||
def composite(p: Panel, weights: dict[str, float], cache: dict | None = None) -> pd.DataFrame:
|
||||
total = None
|
||||
wsum = 0.0
|
||||
for name, w in weights.items():
|
||||
f = cache[name] if cache is not None and name in cache else FACTORS[name](p)
|
||||
r = zrank(f, p.member)
|
||||
total = r * w if total is None else total.add(r * w, fill_value=np.nan)
|
||||
wsum += abs(w)
|
||||
return total / wsum
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# literature shortlist additions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@factor("overnight_mom_6") # 6-month overnight return, no skip
|
||||
def overnight_mom_6(p):
|
||||
on = np.log(p.open / p.close.ffill().shift(1))
|
||||
return on.rolling(126, min_periods=90).sum()
|
||||
|
||||
|
||||
@factor("overnight_mom_1") # last month's overnight return
|
||||
def overnight_mom_1(p):
|
||||
on = np.log(p.open / p.close.ffill().shift(1))
|
||||
return on.rolling(21, min_periods=15).sum()
|
||||
|
||||
|
||||
@factor("stmom_hiturn") # Medhat & Schmeling (2022): 1-month momentum among high-turnover
|
||||
def stmom_hiturn(p): # names, skipping the last 3 days; turnover proxied by
|
||||
c = p.close.ffill() # abnormal dollar volume when shares outstanding are unknown
|
||||
r1 = c.shift(3) / c.shift(24) - 1
|
||||
t = _turnover(p)
|
||||
if t is None:
|
||||
dv = _dollar_volume(p)
|
||||
t = dv.rolling(21, min_periods=15).mean() / dv.rolling(252, min_periods=180).mean()
|
||||
else:
|
||||
t = t.rolling(21, min_periods=15).mean()
|
||||
hi = t.where(p.member).rank(axis=1, pct=True) >= 0.7
|
||||
return r1.where(hi)
|
||||
|
||||
|
||||
@factor("seasonality") # Heston & Sadka (2008): same-calendar-month return, years 1-10
|
||||
def seasonality(p):
|
||||
m = p.close.ffill().resample("ME").last()
|
||||
mret = m.pct_change(fill_method=None)
|
||||
lags = [mret.shift(12 * k) for k in range(1, 11)]
|
||||
avg = sum(l.fillna(0) for l in lags) / sum(l.notna().astype(float) for l in lags).replace(0, np.nan)
|
||||
# value known at month-end t predicts month t+1: shift so the forecast for
|
||||
# next month (lag 11, 23, ...) is available at the start of that month
|
||||
fwd = avg.shift(-1) # uses months t+1-12k, all <= t
|
||||
daily = fwd.reindex(p.close.index, method="ffill")
|
||||
return daily
|
||||
|
||||
|
||||
@factor("trend_hzz") # Han, Zhou & Zhu (2016) trend factor
|
||||
def trend_hzz(p):
|
||||
c = p.close.ffill()
|
||||
lags = [3, 5, 10, 20, 50, 100, 200, 400]
|
||||
feats = {L: (c.rolling(L, min_periods=L).mean() / c) for L in lags}
|
||||
# monthly cross-sectional regression of next-month return on MA signals
|
||||
month_end = c.resample("ME").last().index
|
||||
me = c.index[c.index.searchsorted(month_end, side="right") - 1]
|
||||
fwd = c.shift(-21) / c - 1
|
||||
betas = []
|
||||
for d in me:
|
||||
y = fwd.loc[d].where(p.member.loc[d])
|
||||
X = pd.DataFrame({L: feats[L].loc[d] for L in lags})
|
||||
ok = y.notna() & X.notna().all(axis=1)
|
||||
if ok.sum() < 50:
|
||||
betas.append(pd.Series(np.nan, index=lags, name=d)); continue
|
||||
Xo = np.column_stack([np.ones(ok.sum()), X[ok].values])
|
||||
b = np.linalg.lstsq(Xo, y[ok].values, rcond=None)[0][1:]
|
||||
betas.append(pd.Series(b, index=lags, name=d))
|
||||
B = pd.DataFrame(betas)
|
||||
# the regression at month d uses returns through d+21, so it is usable at d+21;
|
||||
# shift by one month, then average the last 12 known betas
|
||||
Ebeta = B.shift(1).rolling(12, min_periods=6).mean()
|
||||
Ed = Ebeta.reindex(c.index, method="ffill")
|
||||
return sum(feats[L].mul(Ed[L], axis=0) for L in lags)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# A-share additions (CN literature review). Volume ratios stand in for
|
||||
# turnover ratios: shares outstanding rarely change inside the window.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _limit_hit(p) -> pd.DataFrame:
|
||||
r = _ret(p)
|
||||
lim = p.extra.get("limit")
|
||||
if lim is None:
|
||||
return r > 0.095
|
||||
return r >= lim.reindex_like(r) - 0.005
|
||||
|
||||
|
||||
@factor("mom_ex_limitup", markets=("cn",)) # 国信: 12-1 momentum without limit-up days
|
||||
def mom_ex_limitup(p):
|
||||
lr = np.log1p(_ret(p)).where(~_limit_hit(p), 0.0)
|
||||
return lr.shift(21).rolling(231, min_periods=180).sum()
|
||||
|
||||
|
||||
@factor("amp_cut_mom", markets=("us", "cn")) # 开源: momentum on the 70% lowest-amplitude days (160d)
|
||||
def amp_cut_mom(p):
|
||||
amp = (p.high / p.low - 1).values
|
||||
r = _ret(p).values
|
||||
n, keep = 160, 0.7
|
||||
out = np.full(r.shape, np.nan)
|
||||
from numpy.lib.stride_tricks import sliding_window_view as swv
|
||||
A = swv(np.nan_to_num(amp, nan=np.inf), n, axis=0) # (T-n+1, N, n)
|
||||
R = swv(np.nan_to_num(r, nan=0.0), n, axis=0)
|
||||
for t0 in range(0, A.shape[0], 256): # chunk to bound memory
|
||||
a, rr = A[t0:t0 + 256], R[t0:t0 + 256]
|
||||
thr = np.quantile(np.where(np.isinf(a), np.nan, a), keep, axis=2, keepdims=True)
|
||||
out[t0 + n - 1:t0 + n - 1 + a.shape[0]] = np.where(a <= thr, rr, 0.0).sum(axis=2)
|
||||
valid = pd.DataFrame(amp, index=p.close.index).notna().rolling(n).sum() >= 0.8 * n
|
||||
return pd.DataFrame(out, index=p.close.index, columns=p.close.columns).where(valid.values)
|
||||
|
||||
|
||||
@factor("ideal_amp", markets=("us", "cn")) # 开源 理想振幅: amp on high-price days minus low-price days
|
||||
def ideal_amp(p):
|
||||
n, lam = 20, 0.25
|
||||
amp = (p.high / p.low - 1).values
|
||||
c = p.close.ffill().values
|
||||
from numpy.lib.stride_tricks import sliding_window_view as swv
|
||||
A = swv(amp, n, axis=0)
|
||||
Cw = swv(c, n, axis=0)
|
||||
out = np.full(amp.shape, np.nan)
|
||||
hi_q = np.nanquantile(Cw, 1 - lam, axis=2, keepdims=True)
|
||||
lo_q = np.nanquantile(Cw, lam, axis=2, keepdims=True)
|
||||
with np.errstate(invalid="ignore"):
|
||||
v_hi = np.nanmean(np.where(Cw >= hi_q, A, np.nan), axis=2)
|
||||
v_lo = np.nanmean(np.where(Cw <= lo_q, A, np.nan), axis=2)
|
||||
out[n - 1:] = v_hi - v_lo
|
||||
return -pd.DataFrame(out, index=p.close.index, columns=p.close.columns)
|
||||
|
||||
|
||||
@factor("abn_vol_ratio", markets=("us", "cn")) # LSY PMO: -mean(vol,20)/mean(vol,250)
|
||||
def abn_vol_ratio(p):
|
||||
v = p.volume.where(p.volume > 0)
|
||||
return -(v.rolling(20, min_periods=15).mean() / v.rolling(250, min_periods=180).mean())
|
||||
|
||||
|
||||
@factor("vol_cv", markets=("us", "cn")) # HQZ turnover volatility, scale-free
|
||||
def vol_cv(p):
|
||||
v = p.volume.where(p.volume > 0)
|
||||
s = v.rolling(20, min_periods=15)
|
||||
return -(s.std() / s.mean())
|
||||
|
||||
|
||||
@factor("overnight_20", markets=("us", "cn")) # 国信 overnight momentum, 20d
|
||||
def overnight_20(p):
|
||||
return (p.open / p.close.ffill().shift(1) - 1).rolling(20, min_periods=15).sum()
|
||||
|
||||
|
||||
@factor("ivol_20", markets=("us", "cn"))
|
||||
def ivol_20(p):
|
||||
return -resid_returns(p).rolling(20, min_periods=15).std()
|
||||
157
research/lab/fetch_cn.py
Normal file
157
research/lab/fetch_cn.py
Normal file
@@ -0,0 +1,157 @@
|
||||
"""
|
||||
Fetch point-in-time CSI 300 / CSI 500 membership and daily bars from baostock.
|
||||
|
||||
The live CN panel (data/cn.csv) holds today's CSI 300 members only, so every
|
||||
CN backtest on it carries survivorship bias. baostock serves historical index
|
||||
membership, delisted stocks, suspensions (tradestatus), ST flags, turnover and
|
||||
valuation ratios, which lets research backtests trade the index as it was.
|
||||
|
||||
Usage:
|
||||
uv run --with baostock python -m research.lab.fetch_cn [--start 2007-01-01] [--workers 4]
|
||||
|
||||
Writes data/research/cn_ohlcv.pkl:
|
||||
fields: {"open","high","low","close","preclose","volume","amount","turn",
|
||||
"pctchg","tradestatus","isst","pe","pb","ps","pcf"} DataFrames
|
||||
(dates x codes; prices back-adjusted, 后复权)
|
||||
members: {"hs300": {date: [codes]}, "zz500": {date: [codes]}}
|
||||
index: DataFrame of CSI 300 / CSI 500 index closes
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import pickle
|
||||
from datetime import date
|
||||
from multiprocessing import Pool
|
||||
|
||||
import pandas as pd
|
||||
|
||||
OUT = os.path.join("data", "research", "cn_ohlcv.pkl")
|
||||
K_FIELDS = ("date,code,open,high,low,close,preclose,volume,amount,turn,"
|
||||
"tradestatus,pctChg,peTTM,pbMRQ,psTTM,pcfNcfTTM,isST")
|
||||
NUMERIC = ["open", "high", "low", "close", "preclose", "volume", "amount", "turn",
|
||||
"pctChg", "peTTM", "pbMRQ", "psTTM", "pcfNcfTTM", "tradestatus", "isST"]
|
||||
RENAME = {"pctChg": "pctchg", "peTTM": "pe", "pbMRQ": "pb", "psTTM": "ps",
|
||||
"pcfNcfTTM": "pcf", "isST": "isst"}
|
||||
|
||||
|
||||
def _rows(rs) -> list[list[str]]:
|
||||
out = []
|
||||
while rs.error_code == "0" and rs.next():
|
||||
out.append(rs.get_row_data())
|
||||
return out
|
||||
|
||||
|
||||
def _login() -> None:
|
||||
import baostock as bs
|
||||
bs.login()
|
||||
|
||||
|
||||
def fetch_members(start: str) -> dict[str, dict[str, list[str]]]:
|
||||
path = os.path.join(BAR_DIR, f"members_{start}_{date.today()}.pkl")
|
||||
if os.path.exists(path):
|
||||
return pd.read_pickle(path)
|
||||
members = _fetch_members(start)
|
||||
pd.to_pickle(members, path)
|
||||
return members
|
||||
|
||||
|
||||
def _fetch_members(start: str) -> dict[str, dict[str, list[str]]]:
|
||||
import baostock as bs
|
||||
bs.login()
|
||||
months = pd.date_range(start, date.today(), freq="MS")
|
||||
members: dict[str, dict[str, list[str]]] = {"hs300": {}, "zz500": {}}
|
||||
for d in months:
|
||||
ds = d.strftime("%Y-%m-%d")
|
||||
members["hs300"][ds] = [r[1] for r in _rows(bs.query_hs300_stocks(date=ds))]
|
||||
members["zz500"][ds] = [r[1] for r in _rows(bs.query_zz500_stocks(date=ds))]
|
||||
bs.logout()
|
||||
return members
|
||||
|
||||
|
||||
BAR_DIR = os.path.join("data", "research", "cn_bars")
|
||||
|
||||
|
||||
def fetch_one(args: tuple[str, str, str]) -> pd.DataFrame | None:
|
||||
"""Fetch one stock's bars, cached per code so an interrupted run resumes."""
|
||||
code, start, end = args
|
||||
path = os.path.join(BAR_DIR, f"{code}_{end}.pkl")
|
||||
if os.path.exists(path):
|
||||
return pd.read_pickle(path)
|
||||
df = _fetch_one(code, start, end)
|
||||
pd.to_pickle(df, path)
|
||||
return df
|
||||
|
||||
|
||||
def _fetch_one(code: str, start: str, end: str) -> pd.DataFrame | None:
|
||||
import baostock as bs
|
||||
for _ in range(3):
|
||||
rs = bs.query_history_k_data_plus(code, K_FIELDS, start_date=start, end_date=end,
|
||||
frequency="d", adjustflag="1")
|
||||
if rs.error_code == "0":
|
||||
rows = _rows(rs)
|
||||
if not rows:
|
||||
return None
|
||||
df = pd.DataFrame(rows, columns=rs.fields)
|
||||
df["date"] = pd.to_datetime(df["date"])
|
||||
for c in NUMERIC:
|
||||
df[c] = pd.to_numeric(df[c], errors="coerce")
|
||||
return df
|
||||
bs.logout()
|
||||
bs.login()
|
||||
print(f" failed {code}: {rs.error_msg}", flush=True)
|
||||
return None
|
||||
|
||||
|
||||
def fetch_index(start: str, end: str) -> pd.DataFrame:
|
||||
import baostock as bs
|
||||
bs.login()
|
||||
out = {}
|
||||
for code in ("sh.000300", "sh.000905"):
|
||||
rs = bs.query_history_k_data_plus(code, "date,close", start_date=start, end_date=end,
|
||||
frequency="d")
|
||||
df = pd.DataFrame(_rows(rs), columns=rs.fields)
|
||||
out[code] = pd.Series(pd.to_numeric(df["close"]).values, index=pd.to_datetime(df["date"]))
|
||||
bs.logout()
|
||||
return pd.DataFrame(out)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--start", default="2007-01-01")
|
||||
ap.add_argument("--member-start", default="2008-01-01")
|
||||
ap.add_argument("--workers", type=int, default=4)
|
||||
ap.add_argument("--hs300-only", action="store_true")
|
||||
args = ap.parse_args()
|
||||
end = date.today().strftime("%Y-%m-%d")
|
||||
|
||||
os.makedirs(BAR_DIR, exist_ok=True)
|
||||
print("Fetching index membership...", flush=True)
|
||||
members = fetch_members(args.member_start)
|
||||
hs300 = sorted({c for lst in members["hs300"].values() for c in lst})
|
||||
codes = list(hs300)
|
||||
if not args.hs300_only:
|
||||
codes += sorted({c for lst in members["zz500"].values() for c in lst} - set(hs300))
|
||||
print(f" {len(codes)} codes ever in the index set", flush=True)
|
||||
|
||||
frames = []
|
||||
with Pool(args.workers, initializer=_login) as pool:
|
||||
for i, df in enumerate(pool.imap(fetch_one, [(c, args.start, end) for c in codes])):
|
||||
if df is not None:
|
||||
frames.append(df)
|
||||
if (i + 1) % 100 == 0:
|
||||
print(f" bars {i + 1}/{len(codes)}", flush=True)
|
||||
bars = pd.concat(frames, ignore_index=True)
|
||||
fields = {}
|
||||
for c in NUMERIC:
|
||||
fields[RENAME.get(c, c)] = bars.pivot(index="date", columns="code", values=c).sort_index()
|
||||
|
||||
panel = {"fields": fields, "members": members, "index": fetch_index(args.start, end)}
|
||||
os.makedirs(os.path.dirname(OUT), exist_ok=True)
|
||||
with open(OUT, "wb") as fh:
|
||||
pickle.dump(panel, fh)
|
||||
c = fields["close"]
|
||||
print(f"Saved {OUT}: {c.shape[1]} codes, {c.index[0].date()} → {c.index[-1].date()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
70
research/lab/fetch_cn_yahoo.py
Normal file
70
research/lab/fetch_cn_yahoo.py
Normal file
@@ -0,0 +1,70 @@
|
||||
"""
|
||||
Fetch adjusted daily OHLCV for CSI 300 / CSI 500 ever-members from Yahoo
|
||||
(through the market-data proxy). Delisted names Yahoo lacks stay missing
|
||||
(listed in data/research/cn_missing.txt): Eastmoney and Tencent throttle bulk
|
||||
requests, and Tencent's back-adjusted series damps returns (limit-down days
|
||||
show -7% instead of -10%), so neither is used.
|
||||
|
||||
uv run python -m research.lab.fetch_cn_yahoo [--hs300-only]
|
||||
|
||||
Writes data/research/cn_ohlcv.pkl (fields open/high/low/close/volume/preclose/
|
||||
pctchg/tradestatus + members + index + per-code "source").
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import pickle
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.fetch_us import _proxy_env, fetch
|
||||
|
||||
OUT = os.path.join("data", "research", "cn_ohlcv.pkl")
|
||||
|
||||
|
||||
def to_yahoo(code: str) -> str:
|
||||
ex, num = code.split(".")
|
||||
return f"{num}.{'SS' if ex == 'sh' else 'SZ'}"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--hs300-only", action="store_true")
|
||||
ap.add_argument("--start", default="2007-01-01")
|
||||
args = ap.parse_args()
|
||||
member_file = sorted(glob.glob(os.path.join("data", "research", "cn_bars", "members_*.pkl")))[-1]
|
||||
members = pd.read_pickle(member_file)
|
||||
hs300 = sorted({c for lst in members["hs300"].values() for c in lst})
|
||||
codes = list(hs300)
|
||||
if not args.hs300_only:
|
||||
codes += sorted({c for lst in members["zz500"].values() for c in lst} - set(hs300))
|
||||
ymap = {to_yahoo(c): c for c in codes}
|
||||
_proxy_env()
|
||||
raw = fetch(list(ymap) + ["000300.SS", "000905.SS"], args.start)
|
||||
fields = {k: raw[k].rename(columns=ymap) for k in ("open", "high", "low", "close", "volume")}
|
||||
source = {c: "yahoo" for c in fields["close"].columns if c in codes}
|
||||
|
||||
idx = pd.DataFrame({"sh.000300": raw["close"].get("000300.SS"),
|
||||
"sh.000905": raw["close"].get("000905.SS")})
|
||||
keep = [c for c in fields["close"].columns if c in source]
|
||||
fields = {k: v[keep].sort_index() for k, v in fields.items()}
|
||||
# Yahoo prints flat bars (volume 0, O=H=L=C) on suspended days: treat as no trade
|
||||
close = fields["close"]
|
||||
traded = (fields["volume"] > 0) & close.notna()
|
||||
fields["tradestatus"] = traded.astype(float)
|
||||
fields["preclose"] = close.where(traded).ffill().shift(1)
|
||||
fields["pctchg"] = (close / fields["preclose"] - 1) * 100
|
||||
missing = sorted(set(codes) - set(source))
|
||||
panel = {"fields": fields, "members": members, "index": idx, "source": source,
|
||||
"missing": missing}
|
||||
with open(OUT, "wb") as fh:
|
||||
pickle.dump(panel, fh)
|
||||
print(f"Saved {OUT}: {len(keep)} codes, {len(missing)} missing, "
|
||||
f"{close.index[0].date()} → {close.index[-1].date()}")
|
||||
with open(os.path.join("data", "research", "cn_missing.txt"), "w") as fh:
|
||||
fh.write("\n".join(missing))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
91
research/lab/fetch_us.py
Normal file
91
research/lab/fetch_us.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""
|
||||
Fetch a long OHLCV history for every ticker that was ever an S&P 500 member.
|
||||
|
||||
The live panels in data/us.csv hold 10 years of close/open for tickers with
|
||||
>50% coverage. Factor research needs more: volume and high/low, 20+ years of
|
||||
history for an out-of-sample design split, and recently listed members (CEG,
|
||||
GEV, ...) that the coverage rule drops. Delisted names that Yahoo no longer
|
||||
serves stay missing; ``coverage_report`` in panel.py quantifies that gap.
|
||||
|
||||
Usage:
|
||||
uv run python -m research.lab.fetch_us [--start 2004-01-01]
|
||||
|
||||
Writes data/research/us_ohlcv.pkl: {"open","high","low","close","volume"}
|
||||
DataFrames (dates x tickers), split/dividend adjusted.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import pickle
|
||||
import time
|
||||
|
||||
import pandas as pd
|
||||
import yfinance as yf
|
||||
|
||||
import universe_history as uh
|
||||
|
||||
OUT = os.path.join("data", "research", "us_ohlcv.pkl")
|
||||
FIELDS = ["Open", "High", "Low", "Close", "Volume"]
|
||||
# Benchmarks, cash proxies and sector ETFs used by overlays and attribution.
|
||||
EXTRA = ["SPY", "RSP", "QQQ", "SHY", "BIL", "IEF", "TLT", "GLD",
|
||||
"XLB", "XLE", "XLF", "XLI", "XLK", "XLP", "XLU", "XLV", "XLY"]
|
||||
|
||||
|
||||
def _proxy_env() -> None:
|
||||
proxy = os.environ.get("QUANT_MARKET_PROXY")
|
||||
if not proxy and os.path.exists(".env"):
|
||||
for line in open(".env"):
|
||||
if line.startswith("QUANT_MARKET_PROXY="):
|
||||
proxy = line.split("=", 1)[1].strip().strip("'\"")
|
||||
if proxy:
|
||||
for k in ("http_proxy", "https_proxy", "HTTP_PROXY", "HTTPS_PROXY"):
|
||||
os.environ[k] = proxy
|
||||
|
||||
|
||||
def fetch(tickers: list[str], start: str, chunk: int = 60) -> dict[str, pd.DataFrame]:
|
||||
parts: dict[str, list[pd.DataFrame]] = {f: [] for f in FIELDS}
|
||||
for i in range(0, len(tickers), chunk):
|
||||
batch = tickers[i:i + chunk]
|
||||
for attempt in range(4):
|
||||
try:
|
||||
raw = yf.download(batch, start=start, auto_adjust=True, progress=False,
|
||||
group_by="column", threads=True)
|
||||
break
|
||||
except Exception as exc: # noqa: BLE001 — retry transient Yahoo errors
|
||||
print(f" batch {i}: {exc}; retry {attempt + 1}")
|
||||
time.sleep(10 * (attempt + 1))
|
||||
else:
|
||||
continue
|
||||
for f in FIELDS:
|
||||
if f in raw.columns.get_level_values(0):
|
||||
parts[f].append(raw[f])
|
||||
print(f" fetched {min(i + chunk, len(tickers))}/{len(tickers)}", flush=True)
|
||||
time.sleep(2)
|
||||
out = {}
|
||||
for f in FIELDS:
|
||||
if not parts[f]:
|
||||
raise RuntimeError(f"no {f} data downloaded")
|
||||
df = pd.concat(parts[f], axis=1)
|
||||
df = df.loc[:, ~df.columns.duplicated()].sort_index()
|
||||
out[f.lower()] = df.dropna(axis=1, how="all")
|
||||
return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--start", default="2004-01-01")
|
||||
args = ap.parse_args()
|
||||
_proxy_env()
|
||||
intervals = uh.load_sp500_history()
|
||||
tickers = sorted(set(uh.all_tickers_ever(intervals)) | set(EXTRA))
|
||||
print(f"Fetching {len(tickers)} tickers from {args.start}")
|
||||
panel = fetch(tickers, args.start)
|
||||
os.makedirs(os.path.dirname(OUT), exist_ok=True)
|
||||
with open(OUT, "wb") as fh:
|
||||
pickle.dump(panel, fh)
|
||||
c = panel["close"]
|
||||
print(f"Saved {OUT}: {c.shape[1]} tickers, {c.index[0].date()} → {c.index[-1].date()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
43
research/lab/fetch_us_sectors.py
Normal file
43
research/lab/fetch_us_sectors.py
Normal file
@@ -0,0 +1,43 @@
|
||||
"""Fetch GICS-like sectors from Yahoo for S&P 500 ever-members missing from
|
||||
data/us_sectors.csv. Writes data/research/us_sectors_extra.csv.
|
||||
|
||||
uv run python -m research.lab.fetch_us_sectors
|
||||
"""
|
||||
|
||||
import os
|
||||
import pickle
|
||||
|
||||
import pandas as pd
|
||||
import yfinance as yf
|
||||
|
||||
from research.lab.fetch_us import _proxy_env
|
||||
|
||||
# Yahoo sector names → GICS sector names used in data/us_sectors.csv
|
||||
YAHOO_TO_GICS = {
|
||||
"Technology": "Information Technology", "Healthcare": "Health Care",
|
||||
"Financial Services": "Financials", "Consumer Cyclical": "Consumer Discretionary",
|
||||
"Consumer Defensive": "Consumer Staples", "Basic Materials": "Materials",
|
||||
"Communication Services": "Communication Services", "Industrials": "Industrials",
|
||||
"Energy": "Energy", "Utilities": "Utilities", "Real Estate": "Real Estate",
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
_proxy_env()
|
||||
with open(os.path.join("data", "research", "us_ohlcv.pkl"), "rb") as fh:
|
||||
tickers = list(pickle.load(fh)["close"].columns)
|
||||
known = set(pd.read_csv(os.path.join("data", "us_sectors.csv"))["ticker"])
|
||||
rows = []
|
||||
for t in sorted(set(tickers) - known):
|
||||
try:
|
||||
s = yf.Ticker(t).info.get("sector")
|
||||
except Exception: # noqa: BLE001
|
||||
s = None
|
||||
rows.append({"ticker": t, "sector": YAHOO_TO_GICS.get(s, s)})
|
||||
out = pd.DataFrame(rows)
|
||||
out.to_csv(os.path.join("data", "research", "us_sectors_extra.csv"), index=False)
|
||||
print(out["sector"].value_counts(dropna=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
95
research/lab/ml.py
Normal file
95
research/lab/ml.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""
|
||||
Walk-forward signal combination.
|
||||
|
||||
Features are cross-sectional percentile ranks of every factor among PIT
|
||||
members on each date; the target is the cross-sectional rank of the forward
|
||||
open-to-open return over ``horizon`` days. Each calendar year is predicted by
|
||||
a model trained only on samples whose labels ended before that year starts
|
||||
(a ``horizon``-day purge), so every score is out-of-sample.
|
||||
|
||||
from research.lab import ml
|
||||
feats = ml.feature_ranks(panel, names)
|
||||
score = ml.walk_forward(panel, feats, model="lgbm", horizon=20)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.factors import FACTORS
|
||||
from research.lab.report import forward_returns
|
||||
|
||||
|
||||
def feature_ranks(p, names: list[str]) -> dict[str, pd.DataFrame]:
|
||||
out = {}
|
||||
for n in names:
|
||||
f = FACTORS[n](p).reindex(index=p.dates, columns=p.close.columns)
|
||||
out[n] = f.where(p.member).rank(axis=1, pct=True)
|
||||
return out
|
||||
|
||||
|
||||
def _stack(feats: dict[str, pd.DataFrame], rows: pd.DatetimeIndex) -> pd.DataFrame:
|
||||
cols = {n: f.loc[rows].stack(future_stack=True) for n, f in feats.items()}
|
||||
return pd.DataFrame(cols)
|
||||
|
||||
|
||||
def walk_forward(p, feats: dict[str, pd.DataFrame], model: str = "lgbm", horizon: int = 20,
|
||||
sample_every: int = 5, first_test_year: int = 2010, min_train_years: int = 4,
|
||||
params: dict | None = None, seed: int = 0) -> pd.DataFrame:
|
||||
fwd = forward_returns(p, horizon).where(p.member)
|
||||
target = fwd.rank(axis=1, pct=True)
|
||||
dates = p.dates
|
||||
sample_rows = dates[::sample_every]
|
||||
X_all = _stack(feats, sample_rows)
|
||||
y_all = target.loc[sample_rows].stack(future_stack=True).reindex(X_all.index)
|
||||
ok = y_all.notna() & (X_all.notna().sum(axis=1) >= len(feats) // 2)
|
||||
X_all, y_all = X_all[ok], y_all[ok]
|
||||
sample_dates = X_all.index.get_level_values(0)
|
||||
|
||||
score = pd.DataFrame(np.nan, index=dates, columns=p.close.columns)
|
||||
years = sorted({d.year for d in dates if d.year >= first_test_year})
|
||||
for yr in years:
|
||||
test_start = pd.Timestamp(f"{yr}-01-01")
|
||||
pos = dates.searchsorted(test_start)
|
||||
if pos >= len(dates):
|
||||
break
|
||||
# purge: the label of a sample at d uses opens through d + horizon + 1
|
||||
cutoff = dates[max(pos - horizon - 2, 0)]
|
||||
tr = sample_dates <= cutoff
|
||||
if (sample_dates[tr].max() - sample_dates[tr].min()).days < 365 * min_train_years:
|
||||
continue
|
||||
Xtr, ytr = X_all[tr], y_all[tr]
|
||||
test_rows = dates[(dates >= test_start) & (dates < pd.Timestamp(f"{yr + 1}-01-01"))]
|
||||
Xte = _stack(feats, test_rows)
|
||||
pred = _fit_predict(model, Xtr, ytr, Xte, params, seed)
|
||||
s = pd.Series(pred, index=Xte.index).unstack()
|
||||
score.loc[s.index, s.columns] = s.values
|
||||
return score
|
||||
|
||||
|
||||
def _fit_predict(model, Xtr, ytr, Xte, params, seed):
|
||||
if model == "lgbm":
|
||||
import lightgbm as lgb
|
||||
prm = dict(objective="regression", learning_rate=0.03, num_leaves=31,
|
||||
min_data_in_leaf=1000, bagging_fraction=0.7, bagging_freq=1,
|
||||
feature_fraction=0.7, lambda_l2=1.0, seed=seed, verbose=-1, num_threads=8)
|
||||
prm.update(params or {})
|
||||
rounds = prm.pop("n_estimators", 300)
|
||||
booster = lgb.train(prm, lgb.Dataset(Xtr.values, ytr.values), num_boost_round=rounds)
|
||||
return booster.predict(Xte.values)
|
||||
if model == "ridge":
|
||||
Xf = Xtr.fillna(0.5).values - 0.5
|
||||
lam = (params or {}).get("lam", 10.0) * len(Xf)
|
||||
beta = np.linalg.solve(Xf.T @ Xf + lam * np.eye(Xf.shape[1]), Xf.T @ (ytr.values - 0.5))
|
||||
return (Xte.fillna(0.5).values - 0.5) @ beta
|
||||
if model == "ic_weight":
|
||||
# sign-and-size each feature by its training-period mean rank IC
|
||||
d = Xtr.index.get_level_values(0)
|
||||
ics = {}
|
||||
for c in Xtr.columns:
|
||||
df = pd.DataFrame({"x": Xtr[c], "y": ytr, "d": d}).dropna()
|
||||
ics[c] = df.groupby("d").apply(lambda g: g["x"].corr(g["y"])).mean()
|
||||
w = pd.Series(ics).fillna(0.0)
|
||||
return (Xte.fillna(0.5) - 0.5).values @ w.values
|
||||
raise ValueError(model)
|
||||
55
research/lab/overlays.py
Normal file
55
research/lab/overlays.py
Normal file
@@ -0,0 +1,55 @@
|
||||
"""
|
||||
Exposure overlays: series in [0, 1] read on signal dates by engine.simulate
|
||||
(``exposure=``). The un-invested part earns the panel's cash return.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.engine import simulate
|
||||
|
||||
|
||||
def ew_index(p) -> pd.Series:
|
||||
"""Equal-weight index of PIT members (daily rebalanced)."""
|
||||
r = p.close.pct_change(fill_method=None).where(p.member.shift(1, fill_value=False))
|
||||
return (1 + r.mean(axis=1).fillna(0)).cumprod()
|
||||
|
||||
|
||||
def trend(level: pd.Series, ma: int = 200, floor: float = 0.0) -> pd.Series:
|
||||
"""1 when the index is above its moving average, else ``floor``."""
|
||||
above = level > level.rolling(ma, min_periods=ma).mean()
|
||||
return above.astype(float).where(above, floor)
|
||||
|
||||
|
||||
def breadth(p, ma: int = 200, lo: float = 0.4, hi: float = 0.6) -> pd.Series:
|
||||
"""Share of members above their own MA mapped linearly from lo→0 to hi→1."""
|
||||
c = p.close.ffill()
|
||||
above = (c > c.rolling(ma, min_periods=ma).mean()).where(p.member)
|
||||
share = above.mean(axis=1)
|
||||
return ((share - lo) / (hi - lo)).clip(0, 1)
|
||||
|
||||
|
||||
def drawdown_brake(level: pd.Series, dd: float = 0.15, ma: int = 50) -> pd.Series:
|
||||
"""Cut to zero when the index is > dd below its 1y high AND below its MA."""
|
||||
high = level.rolling(252, min_periods=60).max()
|
||||
bad = (level / high - 1 < -dd) & (level < level.rolling(ma, min_periods=ma).mean())
|
||||
return (~bad).astype(float)
|
||||
|
||||
|
||||
def vol_target(p, score, target: float, lookback: int = 60, cap: float = 1.0,
|
||||
base_kwargs: dict | None = None, exposure: pd.Series | None = None) -> pd.Series:
|
||||
"""Scale exposure by target / realized vol of the unscaled strategy.
|
||||
|
||||
Two passes: run the strategy fully invested, measure its trailing
|
||||
daily-return vol, then size the next rebalance; any given ``exposure``
|
||||
(e.g. a trend filter) multiplies the result.
|
||||
"""
|
||||
kw = dict(base_kwargs or {})
|
||||
base = simulate(p, score, **kw)
|
||||
rv = base.returns.rolling(lookback, min_periods=lookback // 2).std() * np.sqrt(252)
|
||||
scale = (target / rv).clip(upper=cap).fillna(cap)
|
||||
if exposure is not None:
|
||||
scale = scale * exposure.reindex(scale.index).ffill().fillna(1.0)
|
||||
return scale
|
||||
196
research/lab/panel.py
Normal file
196
research/lab/panel.py
Normal file
@@ -0,0 +1,196 @@
|
||||
"""
|
||||
Build research Panels (see engine.Panel) from the fetched raw data.
|
||||
|
||||
from research.lab import panel
|
||||
us = panel.load_us() # S&P 500 PIT, 2004 →
|
||||
cn = panel.load_cn("hs300") # CSI 300 PIT, 2007 → ("zz800" = CSI 300 + 500)
|
||||
|
||||
Panels are cached in data/research/*.panel.pkl; delete them after refetching.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import pickle
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.engine import Costs, Panel
|
||||
|
||||
RAW_DIR = os.path.join("data", "research")
|
||||
US_ETFS = {"SPY", "RSP", "QQQ", "SHY", "BIL", "IEF", "TLT", "GLD",
|
||||
"XLB", "XLE", "XLF", "XLI", "XLK", "XLP", "XLU", "XLV", "XLY"}
|
||||
|
||||
|
||||
def _cached(name: str, build):
|
||||
path = os.path.join(RAW_DIR, f"{name}.panel.pkl")
|
||||
if os.path.exists(path):
|
||||
with open(path, "rb") as fh:
|
||||
return pickle.load(fh)
|
||||
p = build()
|
||||
with open(path, "wb") as fh:
|
||||
pickle.dump(p, fh)
|
||||
return p
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# US
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _us_membership(dates: pd.DatetimeIndex, tickers: list[str]) -> pd.DataFrame:
|
||||
with open(os.path.join("data", "sp500_history.json")) as fh:
|
||||
intervals = json.load(fh)["intervals"]
|
||||
mask = pd.DataFrame(False, index=dates, columns=tickers)
|
||||
for t in tickers:
|
||||
for start, end in intervals.get(t, []):
|
||||
s = pd.Timestamp(start) if start else dates[0]
|
||||
e = pd.Timestamp(end) if end else dates[-1] + pd.Timedelta(days=1)
|
||||
mask.loc[(dates >= s) & (dates < e), t] = True
|
||||
return mask
|
||||
|
||||
|
||||
def _build_us() -> Panel:
|
||||
with open(os.path.join(RAW_DIR, "us_ohlcv.pkl"), "rb") as fh:
|
||||
raw = pickle.load(fh)
|
||||
close = raw["close"]
|
||||
# keep rows where most listed tickers printed (drops stray holiday rows)
|
||||
alive = close.notna().sum(axis=1)
|
||||
close = close[alive >= 0.5 * alive.rolling(20, min_periods=1).max()]
|
||||
dates = close.index
|
||||
etf = {k: raw[k].reindex(dates)[[c for c in raw[k].columns if c in US_ETFS]]
|
||||
for k in ("close",)}["close"]
|
||||
stocks = sorted(c for c in close.columns if c not in US_ETFS)
|
||||
f = {k: raw[k].reindex(index=dates, columns=stocks) for k in ("open", "high", "low", "close", "volume")}
|
||||
# Yahoo sometimes prints zero/negative adjusted opens for old bars
|
||||
for k in ("open", "high", "low"):
|
||||
f[k] = f[k].where(f[k] > 0)
|
||||
has_px = f["close"].notna()
|
||||
member = _us_membership(dates, stocks) & has_px
|
||||
cash_px = etf["BIL"].combine_first(etf["SHY"]) if "BIL" in etf else etf["SHY"]
|
||||
bil = etf["BIL"].pct_change(fill_method=None)
|
||||
shy = etf["SHY"].pct_change(fill_method=None)
|
||||
cash_ret = bil.where(bil.notna(), shy).fillna(0.0)
|
||||
del cash_px
|
||||
return Panel(
|
||||
market="us", open=f["open"], high=f["high"], low=f["low"], close=f["close"],
|
||||
volume=f["volume"], member=member, can_buy=has_px, can_sell=has_px,
|
||||
cash_ret=cash_ret, benchmark=etf["SPY"],
|
||||
costs=Costs(buy=0.001, sell=0.001, fixed_fee=0.0, capital=10_000.0),
|
||||
extra={"etf": etf, "sector": _us_sectors(stocks)},
|
||||
)
|
||||
|
||||
|
||||
def _us_sectors(stocks: list[str]) -> pd.Series:
|
||||
"""GICS sector per ticker; current members from data/us_sectors.csv, the
|
||||
rest from data/research/us_sectors_extra.csv when it has been fetched."""
|
||||
sec = pd.read_csv(os.path.join("data", "us_sectors.csv")).set_index("ticker")["GICS Sector"]
|
||||
extra = os.path.join(RAW_DIR, "us_sectors_extra.csv")
|
||||
if os.path.exists(extra):
|
||||
sec = sec.combine_first(pd.read_csv(extra).set_index("ticker")["sector"])
|
||||
return sec.reindex(stocks)
|
||||
|
||||
|
||||
def load_us() -> Panel:
|
||||
return _cached("us", _build_us)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CN
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _limit_pct(codes: list[str], dates: pd.DatetimeIndex, isst: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Daily price-limit fraction per stock (10% main board, 20% ChiNext after
|
||||
2020-08-24 and STAR, 5% ST)."""
|
||||
lim = pd.DataFrame(0.10, index=dates, columns=codes)
|
||||
star = [c for c in codes if c.split(".")[1].startswith("688")]
|
||||
chinext = [c for c in codes if c.split(".")[1].startswith("30")]
|
||||
lim[star] = 0.20
|
||||
lim.loc[dates >= "2020-08-24", chinext] = 0.20
|
||||
st = isst.reindex(index=dates, columns=codes).fillna(0).astype(bool)
|
||||
lim = lim.mask(st & (lim < 0.2), 0.05)
|
||||
return lim
|
||||
|
||||
|
||||
def _cn_membership(members: dict[str, list[str]], dates: pd.DatetimeIndex,
|
||||
codes: list[str]) -> pd.DataFrame:
|
||||
snap = sorted(members.items())
|
||||
mask = pd.DataFrame(False, index=dates, columns=codes)
|
||||
col = {c: i for i, c in enumerate(codes)}
|
||||
arr = mask.values
|
||||
for i, (d, lst) in enumerate(snap):
|
||||
s = pd.Timestamp(d)
|
||||
e = pd.Timestamp(snap[i + 1][0]) if i + 1 < len(snap) else dates[-1] + pd.Timedelta(days=1)
|
||||
rows = (dates >= s) & (dates < e)
|
||||
cols = [col[c] for c in lst if c in col]
|
||||
arr[np.ix_(rows, cols)] = True
|
||||
return pd.DataFrame(arr, index=dates, columns=codes)
|
||||
|
||||
|
||||
def _build_cn(index_set: str) -> Panel:
|
||||
with open(os.path.join(RAW_DIR, "cn_ohlcv.pkl"), "rb") as fh:
|
||||
raw = pickle.load(fh)
|
||||
F = raw["fields"]
|
||||
# Codes an earlier fetcher filled from Tencent's 后复权 series are dropped:
|
||||
# it damps returns for long dividend histories (limit-down shows -7%).
|
||||
bad = {c for c, src in raw.get("source", {}).items() if src == "tencent"}
|
||||
close = F["close"]
|
||||
dates = close.index
|
||||
codes = [c for c in close.columns if c not in bad]
|
||||
f = {k: v.reindex(index=dates, columns=codes) for k, v in F.items()}
|
||||
# Yahoo bars lack ST flags and valuation ratios; index members
|
||||
# are rarely ST (the index drops them at the next review).
|
||||
if "isst" not in f:
|
||||
f["isst"] = pd.DataFrame(0.0, index=dates, columns=codes)
|
||||
traded = (f["tradestatus"] == 1) & (f["volume"] > 0) & f["close"].notna()
|
||||
if index_set == "hs300":
|
||||
member = _cn_membership(raw["members"]["hs300"], dates, codes)
|
||||
elif index_set == "zz500":
|
||||
member = _cn_membership(raw["members"]["zz500"], dates, codes)
|
||||
else: # zz800
|
||||
member = (_cn_membership(raw["members"]["hs300"], dates, codes)
|
||||
| _cn_membership(raw["members"]["zz500"], dates, codes))
|
||||
st = f["isst"].fillna(0).astype(bool)
|
||||
member = member & traded.rolling(5, min_periods=1).max().astype(bool) & ~st
|
||||
lim = _limit_pct(codes, dates, f["isst"])
|
||||
open_ret = f["open"] / f["preclose"] - 1
|
||||
can_buy = traded & ~(open_ret >= lim - 0.002)
|
||||
can_sell = traded & ~(open_ret <= -lim + 0.002)
|
||||
# halted days carry the last price, so returns across a halt are zero
|
||||
for k in ("open", "high", "low", "close"):
|
||||
f[k] = f[k].where(traded)
|
||||
stamp = pd.Series(0.001, index=dates)
|
||||
stamp[dates >= "2023-08-28"] = 0.0005
|
||||
idx = raw["index"].reindex(dates).ffill()
|
||||
return Panel(
|
||||
market=f"cn_{index_set}", open=f["open"], high=f["high"], low=f["low"], close=f["close"],
|
||||
volume=f["volume"].where(traded), member=member, can_buy=can_buy, can_sell=can_sell,
|
||||
cash_ret=pd.Series(0.02 / 252, index=dates), benchmark=idx["sh.000300"],
|
||||
costs=Costs(buy=0.0008, sell=0.0008, sell_tax=stamp, fixed_fee=0.0, capital=70_000.0),
|
||||
extra=_cn_extra(f, traded, lim, idx),
|
||||
)
|
||||
|
||||
|
||||
def _cn_extra(f: dict, traded: pd.DataFrame, lim: pd.DataFrame, idx: pd.DataFrame) -> dict:
|
||||
extra = {"limit": lim, "index": idx, "source": "baostock" if "pe" in f else "yahoo"}
|
||||
if "turn" in f:
|
||||
extra["turn"] = f["turn"].where(traded)
|
||||
if "amount" in f:
|
||||
extra["amount"] = f["amount"].where(traded)
|
||||
if "turn" in f:
|
||||
extra["float_cap"] = (f["amount"] / (f["turn"] / 100)).where(traded & (f["turn"] > 0))
|
||||
for k in ("pe", "pb", "ps", "pcf"):
|
||||
if k in f:
|
||||
extra[k] = f[k]
|
||||
return extra
|
||||
|
||||
|
||||
def load_cn(index_set: str = "hs300") -> Panel:
|
||||
return _cached(f"cn_{index_set}", lambda: _build_cn(index_set))
|
||||
|
||||
|
||||
def coverage_report(p: Panel) -> pd.DataFrame:
|
||||
"""Per year: PIT members with price data vs expected index size."""
|
||||
m = p.member.resample("YE").last()
|
||||
return pd.DataFrame({"members_with_data": m.sum(axis=1)})
|
||||
118
research/lab/report.py
Normal file
118
research/lab/report.py
Normal file
@@ -0,0 +1,118 @@
|
||||
"""Performance statistics, factor IC analysis and overfitting controls."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from scipy import stats as sps
|
||||
|
||||
import metrics
|
||||
|
||||
|
||||
def stats(equity: pd.Series, turnover: pd.Series | None = None) -> dict:
|
||||
eq = equity.dropna()
|
||||
r = eq.pct_change().dropna()
|
||||
out = {
|
||||
"cagr": metrics.annualized_return(eq),
|
||||
"vol": metrics.annualized_volatility(r),
|
||||
"sharpe": metrics.sharpe_ratio(r),
|
||||
"sortino": metrics.sortino_ratio(r),
|
||||
"mdd": metrics.max_drawdown(eq),
|
||||
}
|
||||
out["calmar"] = out["cagr"] / abs(out["mdd"]) if out["mdd"] < 0 else np.inf
|
||||
if turnover is not None:
|
||||
t = turnover.loc[eq.index[0]:eq.index[-1]]
|
||||
years = max((eq.index[-1] - eq.index[0]).days / 365.25, 1e-9)
|
||||
out["turnover"] = t.sum() / years # one-way, x per year
|
||||
return out
|
||||
|
||||
|
||||
def fmt(s: dict) -> str:
|
||||
base = (f"CAGR {s['cagr']*100:6.1f}% vol {s['vol']*100:5.1f}% Sharpe {s['sharpe']:5.2f} "
|
||||
f"Sortino {s['sortino']:5.2f} MDD {s['mdd']*100:6.1f}% Calmar {s['calmar']:5.2f}")
|
||||
if "turnover" in s:
|
||||
base += f" turn {s['turnover']:5.1f}x"
|
||||
return base
|
||||
|
||||
|
||||
def yearly(equity: pd.Series) -> pd.Series:
|
||||
e = equity.dropna()
|
||||
ye = e.resample("YE").last()
|
||||
first = e.iloc[0]
|
||||
return ye.pct_change().fillna(ye.iloc[0] / first - 1)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Factor analytics
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def forward_returns(panel, horizon: int = 5) -> pd.DataFrame:
|
||||
"""Return from the open of t+1 to the open of t+1+horizon — the return a
|
||||
signal formed at the close of t can actually earn."""
|
||||
o = panel.open.where(panel.open > 0).ffill()
|
||||
return o.shift(-(1 + horizon)) / o.shift(-1) - 1
|
||||
|
||||
|
||||
def ic_series(score: pd.DataFrame, fwd: pd.DataFrame, member: pd.DataFrame,
|
||||
every: int = 5) -> pd.Series:
|
||||
"""Cross-sectional Spearman rank IC on every ``every``-th date."""
|
||||
idx = score.index[::every]
|
||||
out = {}
|
||||
for d in idx:
|
||||
s = score.loc[d].where(member.loc[d])
|
||||
f = fwd.loc[d]
|
||||
ok = s.notna() & f.notna()
|
||||
if ok.sum() < 30:
|
||||
continue
|
||||
out[d] = sps.spearmanr(s[ok], f[ok]).statistic
|
||||
return pd.Series(out)
|
||||
|
||||
|
||||
def top_minus_universe(score: pd.DataFrame, fwd: pd.DataFrame, member: pd.DataFrame,
|
||||
top_n: int = 20, every: int = 5) -> pd.Series:
|
||||
"""Mean forward return of the top-N names minus the member average."""
|
||||
idx = score.index[::every]
|
||||
out = {}
|
||||
for d in idx:
|
||||
s = score.loc[d].where(member.loc[d]).dropna()
|
||||
f = fwd.loc[d]
|
||||
s = s[f.reindex(s.index).notna()]
|
||||
if len(s) < top_n * 3:
|
||||
continue
|
||||
top = s.nlargest(top_n).index
|
||||
out[d] = f[top].mean() - f[s.index].mean()
|
||||
return pd.Series(out)
|
||||
|
||||
|
||||
def ic_summary(ic: pd.Series, periods_per_year: float = 52) -> dict:
|
||||
ic = ic.dropna()
|
||||
if len(ic) < 5:
|
||||
return {"ic": np.nan, "icir": np.nan, "t": np.nan, "hit": np.nan, "n": len(ic)}
|
||||
return {"ic": ic.mean(), "icir": ic.mean() / ic.std() * math.sqrt(periods_per_year),
|
||||
"t": ic.mean() / ic.std() * math.sqrt(len(ic)), "hit": (ic > 0).mean(), "n": len(ic)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Overfitting controls (Bailey & Lopez de Prado 2014)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def deflated_sharpe(returns: pd.Series, n_trials: int, sr_trials_var: float | None = None,
|
||||
periods: int = 252) -> dict:
|
||||
"""Probability that the true Sharpe exceeds the best Sharpe expected from
|
||||
``n_trials`` unskilled strategies. Uses per-period Sharpe internally."""
|
||||
r = returns.dropna()
|
||||
T = len(r)
|
||||
sr = r.mean() / r.std()
|
||||
g3 = sps.skew(r)
|
||||
g4 = sps.kurtosis(r, fisher=False)
|
||||
if sr_trials_var is None:
|
||||
sr_trials_var = 1.0 / T # variance of SR estimates under H0
|
||||
emc = 0.5772156649
|
||||
z1 = sps.norm.ppf(1 - 1.0 / n_trials) if n_trials > 1 else 0.0
|
||||
z2 = sps.norm.ppf(1 - 1.0 / (n_trials * math.e)) if n_trials > 1 else 0.0
|
||||
sr0 = math.sqrt(sr_trials_var) * ((1 - emc) * z1 + emc * z2)
|
||||
denom = math.sqrt(max(1 - g3 * sr + (g4 - 1) / 4 * sr ** 2, 1e-12))
|
||||
dsr = sps.norm.cdf((sr - sr0) * math.sqrt(T - 1) / denom)
|
||||
return {"sr_annual": sr * math.sqrt(periods), "sr0_annual": sr0 * math.sqrt(periods), "dsr": dsr}
|
||||
52
research/lab/run_ml.py
Normal file
52
research/lab/run_ml.py
Normal file
@@ -0,0 +1,52 @@
|
||||
"""
|
||||
Walk-forward ML combination experiment.
|
||||
|
||||
PYTHONPATH=. uv run --with lightgbm python -m research.lab.run_ml --market us
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import pickle
|
||||
import time
|
||||
import warnings
|
||||
|
||||
|
||||
from research.lab import ml
|
||||
from research.lab.factor_scan import load, phase_avg
|
||||
from research.lab.factors import FACTORS
|
||||
from research.lab.report import fmt
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
OUT = os.path.join("data", "research", "results")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--market", default="us")
|
||||
ap.add_argument("--models", nargs="*", default=["lgbm", "ridge", "ic_weight"])
|
||||
ap.add_argument("--horizons", nargs="*", type=int, default=[5, 20])
|
||||
ap.add_argument("--first-test-year", type=int, default=2010)
|
||||
args = ap.parse_args()
|
||||
mk = "us" if args.market == "us" else "cn"
|
||||
p, periods = load(args.market)
|
||||
names = [n for n, f in FACTORS.items() if mk in f.markets]
|
||||
t = time.time()
|
||||
feats = ml.feature_ranks(p, names)
|
||||
print(f"{len(names)} features in {time.time() - t:.0f}s", flush=True)
|
||||
evalp = {"design": (f"{args.first_test_year}-01-01", periods["design"][1]), "holdout": periods["holdout"]}
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
for model in args.models:
|
||||
for h in args.horizons:
|
||||
t = time.time()
|
||||
score = ml.walk_forward(p, feats, model=model, horizon=h, first_test_year=args.first_test_year)
|
||||
with open(os.path.join(OUT, f"ml_{args.market}_{model}_h{h}.pkl"), "wb") as fh:
|
||||
pickle.dump(score, fh)
|
||||
for top_n in (10, 20):
|
||||
parts = []
|
||||
for per, (a, b) in evalp.items():
|
||||
parts.append(f"{per}: {fmt(phase_avg(p, score, a, b, top_n, 5))}")
|
||||
print(f"{model:9s} h{h:<3d} top{top_n:<3d}", " | ".join(parts), f"[{time.time() - t:.0f}s]", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
116
research/lab/stepwise.py
Normal file
116
research/lab/stepwise.py
Normal file
@@ -0,0 +1,116 @@
|
||||
"""
|
||||
Forward stepwise selection of equal-weight rank composites, scored on the
|
||||
DESIGN period only; the holdout is evaluated once for the final picks.
|
||||
|
||||
PYTHONPATH=. uv run python -m research.lab.stepwise --market us --max-k 4
|
||||
|
||||
Every evaluated composite is appended to the trial ledger
|
||||
(data/research/results/trials_{market}.csv) so deflated Sharpe can count them.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import time
|
||||
import warnings
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.factor_scan import RESULTS, load, phase_avg
|
||||
from research.lab.factors import FACTORS, zrank
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
# Four ~5-year blocks; "maximin" scores a composite by its worst block Sharpe.
|
||||
SUBPERIODS = {
|
||||
"us": [("2006-09-27", "2011-09-26"), ("2011-09-27", "2016-09-26"),
|
||||
("2016-09-27", "2021-09-26"), ("2021-09-27", "2026-09-24")],
|
||||
"cn": [("2010-01-01", "2013-06-30"), ("2013-07-01", "2016-09-26"),
|
||||
("2016-09-27", "2021-09-26"), ("2021-09-27", "2026-09-24")],
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--market", default="us")
|
||||
ap.add_argument("--max-k", type=int, default=4)
|
||||
ap.add_argument("--top-n", type=int, default=20)
|
||||
ap.add_argument("--every", type=int, default=5)
|
||||
ap.add_argument("--beam", type=int, default=3)
|
||||
ap.add_argument("--objective", default="sharpe")
|
||||
ap.add_argument("--exclude", nargs="*", default=[])
|
||||
ap.add_argument("--buffer", type=float, default=0.0)
|
||||
args = ap.parse_args()
|
||||
mk = "us" if args.market == "us" else "cn"
|
||||
p, periods = load(args.market)
|
||||
names = [n for n, f in FACTORS.items() if mk in f.markets and n not in args.exclude]
|
||||
t = time.time()
|
||||
ranks = {}
|
||||
for n in names:
|
||||
try:
|
||||
ranks[n] = zrank(FACTORS[n](p), p.member)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"skip {n}: {exc}")
|
||||
names = list(ranks)
|
||||
print(f"{len(names)} factor ranks in {time.time() - t:.0f}s", flush=True)
|
||||
a, b = periods["design"]
|
||||
ledger = []
|
||||
subs = SUBPERIODS[mk]
|
||||
|
||||
def evaluate(combo: tuple[str, ...]) -> dict:
|
||||
s = sum(ranks[n] for n in combo) / len(combo)
|
||||
if args.objective == "maximin":
|
||||
per = [phase_avg(p, s, sa, sb, args.top_n, args.every, buffer=args.buffer) for sa, sb in subs]
|
||||
st = {k: float(np.mean([x[k] for x in per])) for k in per[0]}
|
||||
st["maximin"] = min(x["sharpe"] for x in per)
|
||||
st["sub_sharpes"] = " ".join(f"{x['sharpe']:.2f}" for x in per)
|
||||
st["sub_cagrs"] = " ".join(f"{x['cagr']*100:.0f}" for x in per)
|
||||
else:
|
||||
st = phase_avg(p, s, a, b, args.top_n, args.every, buffer=args.buffer)
|
||||
ledger.append({"combo": "+".join(combo), "objective": args.objective, **st})
|
||||
return st
|
||||
|
||||
beam = [()]
|
||||
best_overall = []
|
||||
for k in range(1, args.max_k + 1):
|
||||
cands = {}
|
||||
for base in beam:
|
||||
for n in names:
|
||||
if n in base:
|
||||
continue
|
||||
combo = tuple(sorted(base + (n,)))
|
||||
if combo in cands:
|
||||
continue
|
||||
cands[combo] = evaluate(combo)
|
||||
ranked = sorted(cands.items(), key=lambda kv: -kv[1][args.objective])
|
||||
beam = [c for c, _ in ranked[:args.beam]]
|
||||
print(f"--- k={k}: {len(cands)} combos", flush=True)
|
||||
for c, st in ranked[:6]:
|
||||
extra = (f" min {st['maximin']:.2f} subSh [{st['sub_sharpes']}] subCAGR [{st['sub_cagrs']}]"
|
||||
if "maximin" in st else "")
|
||||
print(f" {'+'.join(c):55s} sh {st['sharpe']:.2f} cagr {st['cagr']*100:5.1f}% "
|
||||
f"mdd {st['mdd']*100:5.1f}% turn {st['turnover']:.0f}x{extra}", flush=True)
|
||||
best_overall += ranked[:args.beam]
|
||||
|
||||
if args.objective == "maximin":
|
||||
_save(ledger, args.market)
|
||||
return
|
||||
# holdout, once, for the beam winners of each k
|
||||
print("=== holdout of design winners ===")
|
||||
ha, hb = periods["holdout"]
|
||||
for c, st in best_overall:
|
||||
s = sum(ranks[n] for n in c) / len(c)
|
||||
hs = phase_avg(p, s, ha, hb, args.top_n, args.every)
|
||||
print(f" {'+'.join(c):55s} D sh {st['sharpe']:.2f} cagr {st['cagr']*100:5.1f}% | "
|
||||
f"H sh {hs['sharpe']:.2f} cagr {hs['cagr']*100:5.1f}% mdd {hs['mdd']*100:5.1f}%", flush=True)
|
||||
_save(ledger, args.market)
|
||||
|
||||
|
||||
def _save(ledger: list[dict], market: str) -> None:
|
||||
os.makedirs(RESULTS, exist_ok=True)
|
||||
path = os.path.join(RESULTS, f"trials_{market}.csv")
|
||||
pd.DataFrame(ledger).to_csv(path, mode="a", header=not os.path.exists(path), index=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,5 +1,8 @@
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
import pandas as pd
|
||||
|
||||
|
||||
class Strategy(ABC):
|
||||
"""
|
||||
Abstract base class for all strategies.
|
||||
@@ -15,3 +18,32 @@ class Strategy(ABC):
|
||||
containing the trading signals (e.g., 'buy', 'hold', 'sell').
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class BarStrategy(Strategy):
|
||||
"""Strategy on completed daily bars with several fields.
|
||||
|
||||
``fields`` names the panels it reads among close/open/high/low/volume.
|
||||
``generate_signals(bars)`` takes ``{field: DataFrame}`` (a bare close
|
||||
DataFrame is accepted when only "close" is needed) and returns weights
|
||||
whose row t uses bars through t-1, like every Strategy. Backtests may add
|
||||
``bars["eligible"]`` (bool, point-in-time index membership): prices stay
|
||||
unmasked so signals see full histories, and only eligible names are held.
|
||||
|
||||
Timing differs from close-price strategies in the live trader: those read
|
||||
the session's open as their latest price, while a BarStrategy only uses
|
||||
completed bars, so the weights of row t trade at the close of t.
|
||||
"""
|
||||
|
||||
fields: tuple[str, ...] = ("close",)
|
||||
|
||||
def generate_signals(self, data):
|
||||
bars = data if isinstance(data, dict) else {"close": data}
|
||||
missing = [f for f in self.fields if f not in bars]
|
||||
if missing:
|
||||
raise ValueError(f"{type(self).__name__} needs bar fields {missing}")
|
||||
return self.generate_weights(bars)
|
||||
|
||||
@abstractmethod
|
||||
def generate_weights(self, bars: dict[str, pd.DataFrame]) -> pd.DataFrame:
|
||||
"""Weights (dates x assets), row t using bars through t-1."""
|
||||
|
||||
194
strategies/robust_momentum.py
Normal file
194
strategies/robust_momentum.py
Normal file
@@ -0,0 +1,194 @@
|
||||
"""
|
||||
Robust momentum strategies from the 2026-09 point-in-time factor research
|
||||
(``research/lab``). They were selected by their worst 5-year Sharpe over
|
||||
2006/2010-2026 on PIT index members with next-open execution, not by their
|
||||
best window.
|
||||
|
||||
Each score averages cross-sectional percentile ranks of its factors. Holdings
|
||||
are the top ``top_n`` names, re-selected every ``rebal_freq`` rows; a held
|
||||
name stays while its rank is within ``top_n * (1 + buffer)`` (hysteresis cuts
|
||||
turnover without giving up the signal). Positions are equal-weighted on
|
||||
rebalance rows and drift with prices in between, so winners are not trimmed
|
||||
daily (worth ~3-4 pp a year for overnight_recovery against daily re-weighting).
|
||||
|
||||
Signals:
|
||||
overnight_recovery (US) overnight-return momentum (12m, skip 1m)
|
||||
+ distance from the 126-day low
|
||||
lowrisk_momentum (CN) amplitude-cut momentum + low beta
|
||||
+ negative price-volume correlation + momentum/vol
|
||||
lowbeta_sharpe_mom (CN) low beta + momentum/vol (close prices only)
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from numpy.lib.stride_tricks import sliding_window_view
|
||||
|
||||
from strategies.base import BarStrategy
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Factors: higher = more attractive; row t uses data through t
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _returns(close: pd.DataFrame) -> pd.DataFrame:
|
||||
return close.pct_change(fill_method=None)
|
||||
|
||||
|
||||
def overnight_momentum(open_: pd.DataFrame, close: pd.DataFrame,
|
||||
window: int = 252, skip: int = 21) -> pd.DataFrame:
|
||||
"""Sum of log overnight returns (open / previous close), Lou-Polk-Skouras."""
|
||||
overnight = np.log(open_ / close.ffill().shift(1))
|
||||
return overnight.rolling(window, min_periods=180).sum().shift(skip)
|
||||
|
||||
|
||||
def recovery(close: pd.DataFrame, window: int = 126) -> pd.DataFrame:
|
||||
c = close.ffill()
|
||||
return c / c.rolling(window, min_periods=window).min() - 1
|
||||
|
||||
|
||||
def momentum_12_1(close: pd.DataFrame) -> pd.DataFrame:
|
||||
c = close.ffill()
|
||||
return c.shift(21) / c.shift(252) - 1
|
||||
|
||||
|
||||
def sharpe_momentum(close: pd.DataFrame) -> pd.DataFrame:
|
||||
"""12-1 month return divided by 12-month daily volatility."""
|
||||
return momentum_12_1(close) / _returns(close).rolling(252, min_periods=180).std()
|
||||
|
||||
|
||||
def low_beta(close: pd.DataFrame, window: int = 252) -> pd.DataFrame:
|
||||
"""Negative rolling beta to the equal-weight average of the panel."""
|
||||
r = _returns(close)
|
||||
m = r.mean(axis=1).fillna(0.0)
|
||||
minp = window * 180 // 252
|
||||
e_xy = r.mul(m, axis=0).rolling(window, min_periods=minp).mean()
|
||||
e_x = r.rolling(window, min_periods=minp).mean()
|
||||
e_y = m.rolling(window, min_periods=minp).mean()
|
||||
var_y = (m ** 2).rolling(window, min_periods=minp).mean() - e_y ** 2
|
||||
return -(e_xy - e_x.mul(e_y, axis=0)).div(var_y, axis=0)
|
||||
|
||||
|
||||
def price_volume_corr(close: pd.DataFrame, volume: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Negative 21-day correlation of price and dollar volume."""
|
||||
return -close.ffill().rolling(21, min_periods=15).corr(close * volume)
|
||||
|
||||
|
||||
def amplitude_cut_momentum(high: pd.DataFrame, low: pd.DataFrame, close: pd.DataFrame,
|
||||
window: int = 160, keep: float = 0.7) -> pd.DataFrame:
|
||||
"""Sum of daily returns over the ``keep`` share of lowest-amplitude
|
||||
(high/low - 1) days in the window (开源证券, 2020): calm up-moves persist,
|
||||
noisy ones reverse."""
|
||||
amp = (high / low - 1).to_numpy(dtype=float)
|
||||
r = _returns(close).to_numpy(dtype=float)
|
||||
out = np.full(r.shape, np.nan)
|
||||
if len(r) >= window:
|
||||
A = sliding_window_view(np.nan_to_num(amp, nan=np.inf), window, axis=0)
|
||||
R = sliding_window_view(np.nan_to_num(r, nan=0.0), window, axis=0)
|
||||
for t0 in range(0, A.shape[0], 256): # chunks bound memory
|
||||
a, rr = A[t0:t0 + 256], R[t0:t0 + 256]
|
||||
thr = np.nanquantile(np.where(np.isinf(a), np.nan, a), keep, axis=2, keepdims=True)
|
||||
out[t0 + window - 1:t0 + window - 1 + a.shape[0]] = np.where(a <= thr, rr, 0.0).sum(axis=2)
|
||||
valid = pd.DataFrame(amp, index=close.index).notna().rolling(window).sum() >= 0.8 * window
|
||||
return pd.DataFrame(out, index=close.index, columns=close.columns).where(valid.to_numpy())
|
||||
|
||||
|
||||
def _rank(df: pd.DataFrame) -> pd.DataFrame:
|
||||
return df.rank(axis=1, pct=True)
|
||||
|
||||
|
||||
def _composite(*factors: pd.DataFrame) -> pd.DataFrame:
|
||||
return sum(_rank(f) for f in factors) / len(factors)
|
||||
|
||||
|
||||
def signal_overnight_recovery(bars):
|
||||
return _composite(overnight_momentum(bars["open"], bars["close"]), recovery(bars["close"]))
|
||||
|
||||
|
||||
def signal_lowrisk_momentum(bars):
|
||||
c = bars["close"]
|
||||
return _composite(amplitude_cut_momentum(bars["high"], bars["low"], c), low_beta(c),
|
||||
price_volume_corr(c, bars["volume"]), sharpe_momentum(c))
|
||||
|
||||
|
||||
def signal_lowbeta_sharpe_mom(bars):
|
||||
c = bars["close"]
|
||||
return _composite(low_beta(c), sharpe_momentum(c))
|
||||
|
||||
|
||||
# name -> (bar fields, score function)
|
||||
SIGNALS = {
|
||||
"overnight_recovery": (("close", "open"), signal_overnight_recovery),
|
||||
"lowrisk_momentum": (("close", "high", "low", "volume"), signal_lowrisk_momentum),
|
||||
"lowbeta_sharpe_mom": (("close",), signal_lowbeta_sharpe_mom),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Selection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def select_with_buffer(score: pd.DataFrame, top_n: int, rebal_freq: int,
|
||||
buffer: float, warmup: int,
|
||||
prices: pd.DataFrame | None = None) -> pd.DataFrame:
|
||||
"""Equal weights on rebalance rows (every ``rebal_freq`` from ``warmup``);
|
||||
in between they drift with ``prices`` (row k uses prices through k), or
|
||||
stay constant when ``prices`` is None. Not yet shifted."""
|
||||
S = score.to_numpy(dtype=float)
|
||||
n, m = S.shape
|
||||
W = np.full((n, m), np.nan)
|
||||
held = np.zeros(m, dtype=bool)
|
||||
for r in range(warmup, n, rebal_freq):
|
||||
row = S[r]
|
||||
valid = ~np.isnan(row)
|
||||
w = np.zeros(m)
|
||||
if valid.sum() >= top_n:
|
||||
order = np.where(valid)[0][np.argsort(-row[valid], kind="stable")]
|
||||
rank = np.full(m, np.inf)
|
||||
rank[order] = np.arange(len(order))
|
||||
keep = [i for i in np.where(held)[0] if rank[i] < top_n * (1 + buffer)]
|
||||
chosen = sorted(keep, key=lambda i: rank[i])[:top_n]
|
||||
taken = set(chosen)
|
||||
for i in order:
|
||||
if len(chosen) >= top_n:
|
||||
break
|
||||
if i not in taken:
|
||||
chosen.append(i)
|
||||
w[chosen] = 1.0 / len(chosen)
|
||||
W[r] = w
|
||||
held = w > 0
|
||||
weights = pd.DataFrame(W, index=score.index, columns=score.columns).ffill().fillna(0.0)
|
||||
weights.iloc[:warmup] = 0.0
|
||||
if prices is not None:
|
||||
px = prices.reindex_like(weights).ffill()
|
||||
anchor = px.where(pd.Series(~np.isnan(W).all(axis=1), index=weights.index), axis=0).ffill()
|
||||
grown = weights * (px / anchor).fillna(1.0)
|
||||
weights = grown.div(grown.sum(axis=1).replace(0.0, np.nan), axis=0).fillna(0.0)
|
||||
return weights
|
||||
|
||||
|
||||
class RobustMomentumStrategy(BarStrategy):
|
||||
def __init__(self, signal: str, top_n: int = 10, rebal_freq: int = 10,
|
||||
buffer: float = 2.0, warmup: int = 252):
|
||||
if signal not in SIGNALS:
|
||||
raise ValueError(f"Unknown signal {signal!r}; available: {list(SIGNALS)}")
|
||||
self.signal = signal
|
||||
self.fields, self._score = SIGNALS[signal]
|
||||
self.top_n = top_n
|
||||
self.rebal_freq = rebal_freq
|
||||
self.buffer = buffer
|
||||
self.warmup = warmup
|
||||
|
||||
def generate_weights(self, bars: dict[str, pd.DataFrame]) -> pd.DataFrame:
|
||||
close = bars["close"]
|
||||
panel = {f: bars[f].reindex(index=close.index, columns=close.columns) for f in self.fields}
|
||||
score = self._score(panel)
|
||||
# names without a price are not eligible; a backtest also passes
|
||||
# "eligible" (point-in-time index membership) so signals can use a
|
||||
# stock's full price history while only members are picked
|
||||
eligible = close.notna()
|
||||
if "eligible" in bars:
|
||||
eligible &= bars["eligible"].reindex_like(close).fillna(False).astype(bool)
|
||||
score = score.where(eligible)
|
||||
weights = select_with_buffer(score, self.top_n, self.rebal_freq, self.buffer, self.warmup,
|
||||
prices=close)
|
||||
return weights.shift(1).fillna(0.0)
|
||||
187
tests/test_robust_strategies.py
Normal file
187
tests/test_robust_strategies.py
Normal file
@@ -0,0 +1,187 @@
|
||||
"""Contracts added with the robust-momentum strategies: bar strategies are
|
||||
causal and trade on the live trader's timing, whole-lot orders track targets,
|
||||
panel backfills retry, and point-in-time CSI 300 membership."""
|
||||
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
import data_manager
|
||||
import trader
|
||||
import universe_history as uh
|
||||
from main import backtest, simulate_live_book, strategy_positions
|
||||
from strategies.robust_momentum import SIGNALS, RobustMomentumStrategy, select_with_buffer
|
||||
|
||||
|
||||
def _bars(n_days: int = 420, n_assets: int = 30, seed: int = 0) -> dict[str, pd.DataFrame]:
|
||||
rng = np.random.default_rng(seed)
|
||||
idx = pd.bdate_range("2024-01-01", periods=n_days)
|
||||
cols = [f"S{i:02d}" for i in range(n_assets)]
|
||||
r = rng.normal(0.0004, 0.02, (n_days, n_assets))
|
||||
close = pd.DataFrame(100 * np.exp(np.cumsum(r, axis=0)), index=idx, columns=cols)
|
||||
gap = rng.normal(0, 0.005, (n_days, n_assets))
|
||||
opens = close.shift(1).bfill() * np.exp(gap)
|
||||
high = np.maximum(opens, close) * (1 + rng.uniform(0, 0.02, (n_days, n_assets)))
|
||||
low = np.minimum(opens, close) * (1 - rng.uniform(0, 0.02, (n_days, n_assets)))
|
||||
volume = pd.DataFrame(rng.uniform(1e5, 1e6, (n_days, n_assets)), index=idx, columns=cols)
|
||||
return {"close": close, "open": opens, "high": high, "low": low, "volume": volume}
|
||||
|
||||
|
||||
class RobustMomentumTests(unittest.TestCase):
|
||||
def test_weights_on_a_day_ignore_that_day_and_later_bars(self):
|
||||
bars = _bars()
|
||||
cut = bars["close"].index[380]
|
||||
for signal in SIGNALS:
|
||||
strat = RobustMomentumStrategy(signal, top_n=5, rebal_freq=5, buffer=1.0)
|
||||
base = strat.generate_signals(bars)
|
||||
shocked = {f: b.copy() for f, b in bars.items()}
|
||||
for f in shocked:
|
||||
shocked[f].loc[cut:] *= 3.0
|
||||
again = strat.generate_signals(shocked)
|
||||
pd.testing.assert_frame_equal(base.loc[:cut], again.loc[:cut], obj=signal)
|
||||
self.assertGreater(base.loc[cut].sum(), 0.99, signal)
|
||||
|
||||
def test_held_names_stay_while_inside_the_buffer(self):
|
||||
idx = pd.bdate_range("2025-01-01", periods=3)
|
||||
score = pd.DataFrame([[5, 4, 3, 2, 1], [1, 2, 5, 4, 3], [1, 2, 5, 4, 3]],
|
||||
index=idx, columns=list("ABCDE"), dtype=float)
|
||||
w = select_with_buffer(score, top_n=2, rebal_freq=1, buffer=1.0, warmup=0)
|
||||
self.assertEqual(set(w.columns[w.iloc[0] > 0]), {"A", "B"})
|
||||
# Day 2: A ranks 5th (outside 2 * (1 + 1) = 4) and leaves; B ranks 4th and stays.
|
||||
self.assertEqual(set(w.columns[w.iloc[1] > 0]), {"B", "C"})
|
||||
|
||||
def test_bar_strategy_trades_at_the_close_after_its_signal_day(self):
|
||||
bars = _bars()
|
||||
strat = RobustMomentumStrategy("lowbeta_sharpe_mom", top_n=5, rebal_freq=5)
|
||||
weights = strat.generate_signals(bars)
|
||||
held = strategy_positions(strat, bars["close"], open_data=bars["open"], bars=bars)
|
||||
pd.testing.assert_frame_equal(held, weights.shift(1).fillna(0.0))
|
||||
equity = backtest(strat, bars["close"], initial_capital=1000, open_data=bars["open"], bars=bars)
|
||||
self.assertEqual(len(equity), len(bars["close"]))
|
||||
|
||||
def test_trader_session_signals_use_bars_through_the_previous_day(self):
|
||||
bars = _bars()
|
||||
days = bars["close"].index[-3:]
|
||||
strat = RobustMomentumStrategy("overnight_recovery", top_n=5, rebal_freq=1)
|
||||
full = strat.generate_signals(bars).loc[days]
|
||||
got = trader.bar_session_weights(strat, bars, days)
|
||||
pd.testing.assert_frame_equal(got, full)
|
||||
# Morning run: today's bar does not exist yet.
|
||||
today = days[-1] + pd.offsets.BDay()
|
||||
morning = trader.bar_session_weights(strat, bars, pd.DatetimeIndex([today]))
|
||||
self.assertAlmostEqual(morning.loc[today].sum(), 1.0)
|
||||
|
||||
|
||||
class LiveBookSimulationTests(unittest.TestCase):
|
||||
def test_holdings_drift_and_only_resizes_beyond_the_band_trade(self):
|
||||
idx = pd.bdate_range("2026-01-05", periods=5)
|
||||
data = pd.DataFrame({"A": [100, 100, 125, 125, 125], "B": [100] * 5, "C": [100] * 5},
|
||||
index=idx, dtype=float)
|
||||
third = [0, 1 / 3, 1 / 3, 1 / 3]
|
||||
held = pd.DataFrame({"A": third + [0.0], "B": third + [0.5], "C": third + [0.5]}, index=idx)
|
||||
equity, traded = simulate_live_book(held, data, 1000.0, transaction_cost=0.0,
|
||||
resize_band=0.10)
|
||||
orders = (traded.abs() > 0).sum(axis=1).tolist()
|
||||
# Buy all three; A +25% drifts to 38.5% (15% over target): trim A only,
|
||||
# B and C at 30.8% stay inside the band; then exit A, top up B and C.
|
||||
self.assertEqual(orders, [0, 3, 0, 1, 3])
|
||||
self.assertLess(traded.loc[idx[3], "A"], 0)
|
||||
self.assertAlmostEqual(equity.iloc[2], 1000 * (1 + 0.25 / 3), places=6)
|
||||
|
||||
def test_orders_below_the_floor_are_skipped_but_exits_are_not(self):
|
||||
idx = pd.bdate_range("2026-01-05", periods=3)
|
||||
data = pd.DataFrame({"A": [10.0, 10.0, 10.0], "B": [10.0, 10.0, 10.0]}, index=idx)
|
||||
held = pd.DataFrame({"A": [0, 0.004, 0.0], "B": [0, 0.5, 0.5]}, index=idx)
|
||||
_, traded = simulate_live_book(held, data, 10_000.0, min_trade_value=50.0)
|
||||
self.assertEqual(traded["A"].abs().sum(), 0.0) # a $40 entry is below $50
|
||||
self.assertGreater(traded["B"].abs().sum(), 0.0)
|
||||
|
||||
|
||||
class LotAllocationTests(unittest.TestCase):
|
||||
def test_leftover_cash_buys_the_lot_that_brings_a_name_nearest_its_target(self):
|
||||
prices = {"A": 45.0, "B": 13.0}
|
||||
# 70k book, 50/50: A wants 777.8 shares, B 2692.3. Rounding down leaves
|
||||
# 4.9k idle; it buys A's 8th lot (3.5k short of target vs half a lot
|
||||
# 2.25k), after which B's extra lot no longer fits.
|
||||
trades = trader.compute_trades({}, 70_000.0, {"A": 0.5, "B": 0.5}, prices,
|
||||
integer_shares=True, lot_size=100)
|
||||
shares = {t["ticker"]: t["shares_delta"] for t in trades}
|
||||
self.assertEqual(shares, {"A": 800, "B": 2600})
|
||||
spent = sum(s * prices[t] for t, s in shares.items())
|
||||
self.assertLessEqual(spent, 70_000.0)
|
||||
|
||||
def test_a_lot_far_above_target_is_not_bought(self):
|
||||
trades = trader.compute_trades({}, 10_000.0, {"A": 0.3, "B": 0.7}, {"A": 70.0, "B": 1.0},
|
||||
integer_shares=True, lot_size=100)
|
||||
# A's target is 3000 = 0.43 of a 7000 lot: stay out.
|
||||
self.assertNotIn("A", {t["ticker"] for t in trades})
|
||||
|
||||
def test_small_drift_is_not_traded_but_entries_and_exits_are(self):
|
||||
holdings = {"A": 10.0, "B": 10.0}
|
||||
prices = {"A": 105.0, "B": 95.0, "C": 50.0}
|
||||
trades = trader.compute_trades(holdings, 0.0, {"A": 0.5, "B": 0.5}, prices,
|
||||
min_trade_value=1.0, resize_band=0.10)
|
||||
self.assertEqual(trades, [])
|
||||
trades = trader.compute_trades(holdings, 0.0, {"A": 0.5, "C": 0.5}, prices,
|
||||
min_trade_value=1.0, resize_band=0.10)
|
||||
self.assertEqual({t["ticker"]: t["direction"] for t in trades}, {"B": "SELL", "C": "BUY"})
|
||||
|
||||
|
||||
class PanelBackfillTests(unittest.TestCase):
|
||||
def test_tickers_missing_from_a_partial_batch_are_retried_before_giving_up(self):
|
||||
idx = pd.bdate_range("2026-01-05", periods=4)
|
||||
calls = []
|
||||
|
||||
def fake_download(tickers, start, end=None, fields=None):
|
||||
calls.append(list(tickers))
|
||||
ok = [t for t in tickers if t == "AAA" or (t == "BBB" and len(calls) > 1)]
|
||||
frame = pd.DataFrame({t: [1.0, 2.0, 3.0, 4.0] for t in ok}, index=idx)
|
||||
return {f: frame for f in fields}
|
||||
|
||||
with mock.patch.object(data_manager, "_download", fake_download):
|
||||
got, missing = data_manager._backfill(["AAA", "BBB", "CCC"], "2026-01-05",
|
||||
["Close", "Volume"], idx)
|
||||
self.assertEqual(sorted(got["Close"].columns), ["AAA", "BBB"])
|
||||
self.assertEqual(missing, ["CCC"])
|
||||
self.assertEqual(calls[1], ["BBB", "CCC"])
|
||||
|
||||
|
||||
class BacktestUniverseTests(unittest.TestCase):
|
||||
def test_stock_universe_excludes_benchmarks_and_etfs_sharing_the_panel(self):
|
||||
import backtest_runner
|
||||
idx = pd.bdate_range("2026-01-05", periods=3)
|
||||
panel = pd.DataFrame(1.0, index=idx, columns=["AAA", "BBB", "SPY", "TQQQ"])
|
||||
fake_universe = {"us": {"fetch": lambda: ["AAA"], "benchmark": "SPY",
|
||||
"benchmark_label": "SPY"}}
|
||||
with mock.patch.object(backtest_runner, "UNIVERSES", fake_universe), \
|
||||
mock.patch.object(backtest_runner.uh, "pit_tickers_ever", lambda m, since=None: ["AAA", "BBB"]), \
|
||||
mock.patch.object(backtest_runner.uh, "mask_prices_pit", lambda p, m: p), \
|
||||
mock.patch.object(backtest_runner.data_manager, "update",
|
||||
lambda m, t, fields=None: {f: panel for f in fields}):
|
||||
data = backtest_runner.load_stock_panels("us", ["momentum"], None)
|
||||
self.assertEqual(data["stocks"], ["AAA", "BBB"])
|
||||
|
||||
|
||||
class MembershipTests(unittest.TestCase):
|
||||
def test_csi300_snapshot_applies_until_the_next_one(self):
|
||||
snaps = {"2026-01-01": ["600000.SS", "000001.SZ"], "2026-02-01": ["600000.SS"]}
|
||||
dates = pd.to_datetime(["2025-12-31", "2026-01-15", "2026-02-02"])
|
||||
mask = uh.csi300_membership_mask(dates, ["600000.SS", "000001.SZ", "510300.SS"], snaps)
|
||||
self.assertEqual(mask["000001.SZ"].tolist(), [False, True, False])
|
||||
self.assertEqual(mask["600000.SS"].tolist(), [False, True, True])
|
||||
# Never-member columns (benchmarks) are not masked.
|
||||
self.assertTrue(mask["510300.SS"].all())
|
||||
|
||||
def test_changes_table_with_two_level_headers_is_found(self):
|
||||
cols = pd.MultiIndex.from_tuples([("Effective Date", "Effective Date"), ("Added", "Ticker"),
|
||||
("Removed", "Ticker"), ("Reason", "Reason")])
|
||||
table = pd.DataFrame([["September 21, 2026", "BE", "TAP", "cap"]], columns=cols)
|
||||
other = pd.DataFrame({"x": [1]})
|
||||
changes = uh._find_changes_table([other, table])
|
||||
self.assertEqual(list(changes.columns), ["Date", "Added_Ticker", "Removed_Ticker", "Reason_Reason"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
228
trader.py
228
trader.py
@@ -40,6 +40,7 @@ import pandas as pd
|
||||
import yfinance as yf
|
||||
|
||||
import data_manager
|
||||
from strategies.base import BarStrategy
|
||||
from strategies.buy_and_hold import BuyAndHoldStrategy
|
||||
from strategies.dual_momentum import DualMomentumStrategy
|
||||
from strategies.factor_combo import FactorComboStrategy
|
||||
@@ -55,6 +56,7 @@ from strategies.permanent import (
|
||||
TrendRiderV4,
|
||||
)
|
||||
from strategies.recovery_momentum import RecoveryMomentumStrategy
|
||||
from strategies.robust_momentum import RobustMomentumStrategy
|
||||
from strategies.trend_following import TrendFollowingStrategy
|
||||
from strategies.trend_rider_v5 import TrendRiderV5
|
||||
from strategies.trend_rider_voltgt import (
|
||||
@@ -95,6 +97,12 @@ MARKET_FEE_TIERED = {
|
||||
# board lots of 100.
|
||||
MARKET_LOT_SIZE = {"us": 1, "cn": 100}
|
||||
|
||||
# A position already held is resized only once it is this far from its target
|
||||
# (relative to the target value); entries and exits always trade. Without the
|
||||
# band the books paid a ticket fee almost daily to trade weight drift that
|
||||
# the backtests never trade.
|
||||
RESIZE_BAND = 0.10
|
||||
|
||||
# Default live capital per market, in the market's own currency.
|
||||
MONITOR_CAPITAL = {"us": 10_000.0, "cn": 70_000.0}
|
||||
|
||||
@@ -226,6 +234,16 @@ STRATEGY_REGISTRY = {
|
||||
"recovery_quality_blend_top10": lambda **kw: RecoveryQualityBlendStrategy(top_n=10),
|
||||
"improved_mom_quality_top20": lambda **kw: ImprovedMomentumQualityStrategy(top_n=20),
|
||||
"improved_mom_quality_top10": lambda **kw: ImprovedMomentumQualityStrategy(top_n=10),
|
||||
# --- Robust momentum (2026-09 PIT research; bar strategies) ---
|
||||
"overnight_recovery_top10": lambda **kw: RobustMomentumStrategy(
|
||||
"overnight_recovery", top_n=10, rebal_freq=10, buffer=2.0,
|
||||
),
|
||||
"lowrisk_momentum_top15": lambda **kw: RobustMomentumStrategy(
|
||||
"lowrisk_momentum", top_n=15, rebal_freq=10, buffer=2.0,
|
||||
),
|
||||
"lowbeta_sharpe_mom_top20": lambda **kw: RobustMomentumStrategy(
|
||||
"lowbeta_sharpe_mom", top_n=20, rebal_freq=10, buffer=1.0,
|
||||
),
|
||||
# --- TrendRiderV6: stock-picking + V5 regime engine ---
|
||||
"trend_rider_v6": lambda **kw: TrendRiderV6(),
|
||||
"trend_rider_v6_top10": lambda **kw: TrendRiderV6(top_n=10),
|
||||
@@ -273,6 +291,12 @@ def strategy_extra_tickers(strategy_name: str) -> list[str]:
|
||||
return list(MIXED_STRATEGY_EXTRA_TICKERS.get(strategy_name.removeprefix("sim_"), []))
|
||||
|
||||
|
||||
def strategy_fields(strategy_name: str) -> tuple[str, ...]:
|
||||
"""Bar fields a strategy reads; close-price strategies read ("close",)."""
|
||||
strategy = STRATEGY_REGISTRY[strategy_name.removeprefix("sim_")]()
|
||||
return tuple(strategy.fields) if isinstance(strategy, BarStrategy) else ("close",)
|
||||
|
||||
|
||||
def strategy_supports_market(strategy_name: str, market: str) -> bool:
|
||||
"""ETF-based strategies trade USD-listed ETFs, so they only run in the US book."""
|
||||
base_name = strategy_name.removeprefix("sim_")
|
||||
@@ -351,14 +375,15 @@ def load_strategy_prices(market: str, strategy_name: str, tickers: list[str],
|
||||
stale forward-filled price from a real one.
|
||||
"""
|
||||
data_market = strategy_data_market(market, strategy_name)
|
||||
fields = ["close", "open"] if with_open else ["close"]
|
||||
fields += [f for f in strategy_fields(strategy_name) if f not in fields]
|
||||
if refresh:
|
||||
request = price_request(market, list(tickers) + [benchmark])
|
||||
result = data_manager.update(data_market, request, with_open=with_open)
|
||||
close, open_data = result if with_open else (result, None)
|
||||
panels = data_manager.update(data_market, request, fields=fields)
|
||||
else:
|
||||
close = data_manager.load(data_market, "close")
|
||||
open_data = (data_manager._fill_open(close, data_manager.load_raw(data_market, "open"))
|
||||
if with_open else None)
|
||||
panels = data_manager.load_fields(data_market, fields)
|
||||
close = panels["close"]
|
||||
open_data = panels["open"] if with_open else None
|
||||
close_raw = data_manager.load_raw(data_market, "close").reindex(
|
||||
index=close.index, columns=close.columns,
|
||||
)
|
||||
@@ -438,85 +463,102 @@ def portfolio_value(holdings: dict, prices: dict, cash: float) -> float:
|
||||
return cash + stock_value
|
||||
|
||||
|
||||
def generate_target_weights(strategy, open_data: pd.DataFrame, target_date) -> dict:
|
||||
"""
|
||||
Run strategy on full open price history, extract target_date's weights.
|
||||
def strategy_bars(market: str, strategy_name: str, tickers: list[str],
|
||||
close: pd.DataFrame, open_data: pd.DataFrame | None) -> dict | None:
|
||||
"""Stored bar panels a BarStrategy reads (refreshed by load_strategy_prices),
|
||||
aligned to ``close``; None for close-price strategies."""
|
||||
fields = strategy_fields(strategy_name)
|
||||
if not isinstance(STRATEGY_REGISTRY[strategy_name.removeprefix("sim_")](), BarStrategy):
|
||||
return None
|
||||
stored = data_manager.load_fields(strategy_data_market(market, strategy_name),
|
||||
[f for f in fields if f not in ("close", "open")])
|
||||
bars = {"close": close[tickers]}
|
||||
if "open" in fields:
|
||||
bars["open"] = open_data[tickers]
|
||||
for f in fields:
|
||||
if f not in bars:
|
||||
bars[f] = stored[f].reindex(index=close.index, columns=tickers)
|
||||
return bars
|
||||
|
||||
Strategies have an internal shift(1) designed for close prices: weights[t]
|
||||
uses data up to t-1. With open prices (observable same-day), we want the
|
||||
signal that uses data up to today. To get this, we append a dummy future
|
||||
row so the strategy computes one extra day of signals, then we read that
|
||||
last row — which incorporates today's open prices.
|
||||
"""
|
||||
# Append a dummy row one day after the last date so the strategy's
|
||||
# shift(1) produces a valid signal row that uses target_date's open
|
||||
dummy_date = open_data.index[-1] + pd.Timedelta(days=1)
|
||||
dummy_row = pd.DataFrame(
|
||||
[open_data.iloc[-1].values],
|
||||
index=[dummy_date],
|
||||
columns=open_data.columns,
|
||||
)
|
||||
extended = pd.concat([open_data, dummy_row])
|
||||
|
||||
weights = strategy.generate_signals(extended)
|
||||
def bar_session_weights(strategy: BarStrategy, bars: dict, days: pd.DatetimeIndex) -> pd.DataFrame:
|
||||
"""Targets traded at the close of each day in ``days``: row d uses bars through d-1."""
|
||||
hist = {f: b.loc[:days[-1]] for f, b in bars.items()}
|
||||
close = hist["close"]
|
||||
if days[-1] not in close.index:
|
||||
# morning run before today's bar exists: add an empty row for today
|
||||
hist = {f: pd.concat([b, pd.DataFrame(np.nan, index=[days[-1]], columns=b.columns)])
|
||||
for f, b in hist.items()}
|
||||
return strategy.generate_signals(hist).reindex(days).fillna(0.0)
|
||||
|
||||
# The dummy_date row in weights now uses data up to target_date (the real last day)
|
||||
# Read the dummy row — that's today's actual signal
|
||||
if dummy_date in weights.index:
|
||||
row = weights.loc[dummy_date]
|
||||
elif target_date in weights.index:
|
||||
# Fallback: use target_date row (which uses data up to target_date - 1)
|
||||
row = weights.loc[target_date]
|
||||
else:
|
||||
idx = weights.index.get_indexer([target_date], method="ffill")
|
||||
if idx[0] >= 0:
|
||||
row = weights.iloc[idx[0]]
|
||||
|
||||
def _allocate_lots(desired: dict, holdings: dict, prices: dict, cash: float,
|
||||
lot_size: int, reserve: float) -> dict:
|
||||
"""Whole-lot share targets: round each move toward zero, then spend the
|
||||
remaining cash one lot at a time on the names furthest below target,
|
||||
as long as the lot brings them nearer (nearest-lot rounding)."""
|
||||
final, spend = {}, 0.0
|
||||
for t, want in desired.items():
|
||||
cur = holdings.get(t, 0.0)
|
||||
if want <= 0:
|
||||
final[t] = 0.0 # full exit, odd lot included
|
||||
else:
|
||||
return {}
|
||||
|
||||
# Return only non-zero weights
|
||||
return {t: float(w) for t, w in row.items() if w > 1e-6}
|
||||
final[t] = cur + int((want - cur) / lot_size) * lot_size
|
||||
spend += (final[t] - cur) * prices[t]
|
||||
budget = cash - spend - reserve
|
||||
short = sorted(((desired[t] - final[t]) * prices[t], t) for t in desired if desired[t] > 0)
|
||||
for gap, t in reversed(short):
|
||||
lot_value = lot_size * prices[t]
|
||||
if gap > 0.5 * lot_value and lot_value <= budget:
|
||||
final[t] += lot_size
|
||||
budget -= lot_value
|
||||
return final
|
||||
|
||||
|
||||
def compute_trades(holdings: dict, cash: float, target_weights: dict,
|
||||
prices: dict, min_trade_value: float = 50.0,
|
||||
integer_shares: bool = False, lot_size: int = 1,
|
||||
frozen: set | None = None) -> list[dict]:
|
||||
frozen: set | None = None, resize_band: float = 0.0) -> list[dict]:
|
||||
"""
|
||||
Compute trades needed to move from current holdings to target weights.
|
||||
|
||||
If integer_shares=True, share deltas are rounded toward zero to whole
|
||||
multiples of ``lot_size`` (e.g. 100-share A-share board lots); a position
|
||||
whose target is zero is always sold in full, odd lot included.
|
||||
If integer_shares=True, positions move in whole multiples of ``lot_size``
|
||||
(e.g. 100-share A-share board lots): each move is rounded toward zero and
|
||||
leftover cash then buys one more lot for the names furthest below target
|
||||
when that lot brings them nearer; a position whose target is zero is
|
||||
always sold in full, odd lot included.
|
||||
A held position with a nonzero target is not resized while it is within
|
||||
``resize_band`` (fraction of its target value) of the target.
|
||||
Tickers in ``frozen`` (no fresh price today) are neither bought nor sold.
|
||||
|
||||
Returns list of {ticker, shares_delta, direction, est_value}.
|
||||
"""
|
||||
total = portfolio_value(holdings, prices, cash)
|
||||
raw = []
|
||||
frozen = frozen or set()
|
||||
|
||||
all_tickers = set(list(target_weights.keys()) + list(holdings.keys()))
|
||||
for ticker in sorted(all_tickers):
|
||||
desired = {}
|
||||
for ticker in sorted(set(target_weights) | set(holdings)):
|
||||
if ticker in frozen:
|
||||
continue
|
||||
target_weight = target_weights.get(ticker, 0.0)
|
||||
price = prices.get(ticker, 0.0)
|
||||
if price <= 0:
|
||||
continue
|
||||
|
||||
target_weight = target_weights.get(ticker, 0.0)
|
||||
target_value = total * target_weight
|
||||
target_shares = target_value / price
|
||||
current_shares = holdings.get(ticker, 0.0)
|
||||
if (current_shares > 0 and target_weight > 0
|
||||
and abs(target_value - current_shares * price) < resize_band * target_value):
|
||||
continue
|
||||
desired[ticker] = target_value / price
|
||||
|
||||
targets = (_allocate_lots(desired, holdings, prices, cash, lot_size,
|
||||
reserve=0.002 * total + 10.0 * len(desired))
|
||||
if integer_shares else desired)
|
||||
|
||||
raw = []
|
||||
for ticker, target_shares in targets.items():
|
||||
price = prices[ticker]
|
||||
current_shares = holdings.get(ticker, 0.0)
|
||||
delta = target_shares - current_shares
|
||||
|
||||
if integer_shares:
|
||||
if target_weight <= 0 and current_shares > 0:
|
||||
delta = -current_shares # full exit may include an odd lot
|
||||
else:
|
||||
# Round toward zero: never overshoot cash or over-sell.
|
||||
delta = int(delta / lot_size) * lot_size
|
||||
|
||||
trade_value = abs(delta * price)
|
||||
if trade_value < min_trade_value:
|
||||
continue
|
||||
@@ -735,25 +777,32 @@ def _strategy_fees(args, market: str) -> dict:
|
||||
|
||||
|
||||
def _signals_for_sessions(strategy_name: str, open_data: pd.DataFrame,
|
||||
days: pd.DatetimeIndex) -> pd.DataFrame:
|
||||
"""Target weights for many sessions in one pass: row ``d`` uses opens up to ``d``.
|
||||
days: pd.DatetimeIndex, bars: dict | None = None) -> pd.DataFrame:
|
||||
"""Target weights traded at the close of each day in ``days``, in one pass.
|
||||
|
||||
Equivalent to ``generate_target_weights`` per day for causal strategies
|
||||
(weights at t never depend on data after t), which every registered
|
||||
strategy must be.
|
||||
A close-price strategy reads opens as its prices: the row for ``d`` uses
|
||||
opens up to ``d`` (a dummy row after the last open makes the strategy's
|
||||
internal shift(1) yield that row). A BarStrategy uses completed bars
|
||||
through d-1. Every registered strategy must be causal (weights at t never
|
||||
depend on data after t), so one pass equals running each day separately.
|
||||
"""
|
||||
strategy = STRATEGY_REGISTRY[strategy_name]()
|
||||
if isinstance(strategy, BarStrategy):
|
||||
return bar_session_weights(strategy, bars, days)
|
||||
hist = open_data.loc[:days[-1]]
|
||||
dummy = pd.DataFrame([hist.iloc[-1].values], index=[hist.index[-1] + pd.Timedelta(days=1)],
|
||||
columns=hist.columns)
|
||||
weights = STRATEGY_REGISTRY[strategy_name]().generate_signals(pd.concat([hist, dummy]))
|
||||
weights = strategy.generate_signals(pd.concat([hist, dummy]))
|
||||
return weights.shift(-1).reindex(days).fillna(0.0)
|
||||
|
||||
|
||||
def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp,
|
||||
tickers: list[str], benchmark: str, close_data: pd.DataFrame,
|
||||
open_data: pd.DataFrame, close_raw: pd.DataFrame, fees: dict,
|
||||
integer_shares: bool, signal_row: pd.Series | None = None) -> dict:
|
||||
"""Signal on ``day``'s open, execute at ``day``'s close, record the day.
|
||||
integer_shares: bool, signal_row: pd.Series | None = None,
|
||||
bars: dict | None = None) -> dict:
|
||||
"""Signal on ``day``'s open (a BarStrategy: bars through the previous
|
||||
session), execute at ``day``'s close, record the day.
|
||||
|
||||
Tickers without a real closing print on ``day`` (forward-filled price) are
|
||||
valued but not traded. ``signal_row`` supplies precomputed target weights.
|
||||
@@ -761,10 +810,9 @@ def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp
|
||||
day_str = str(day.date())
|
||||
lot_size = MARKET_LOT_SIZE.get(market, 1) if integer_shares else 1
|
||||
if signal_row is None:
|
||||
strategy = STRATEGY_REGISTRY[strategy_name]()
|
||||
target_weights = generate_target_weights(strategy, open_data.loc[:day, tickers], day)
|
||||
else:
|
||||
target_weights = {t: float(w) for t, w in signal_row.items() if w > 1e-6}
|
||||
signal_row = _signals_for_sessions(strategy_name, open_data[tickers],
|
||||
pd.DatetimeIndex([day]), bars).loc[day]
|
||||
target_weights = {t: float(w) for t, w in signal_row.items() if w > 1e-6}
|
||||
|
||||
all_held = sorted(set(target_weights) | set(state["holdings"]))
|
||||
close_prices = _with_last_known_prices(
|
||||
@@ -778,6 +826,7 @@ def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp
|
||||
state["holdings"], state["cash"], target_weights, close_prices,
|
||||
min_trade_value=max(50, pre_value * 0.001),
|
||||
integer_shares=integer_shares, lot_size=lot_size, frozen=frozen,
|
||||
resize_band=RESIZE_BAND,
|
||||
)
|
||||
n_log = len(state["trade_log"])
|
||||
execute_trades(state, trades, close_prices, trade_date=day_str,
|
||||
@@ -840,9 +889,11 @@ def cmd_morning(args):
|
||||
print(f" Strategy: {strategy_name} | Market: {market.upper()}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# Run strategy on open prices
|
||||
strategy = STRATEGY_REGISTRY[strategy_name]()
|
||||
target_weights = generate_target_weights(strategy, open_data[tickers], today)
|
||||
# Run strategy on open prices (a BarStrategy: on completed bars)
|
||||
bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
|
||||
row = _signals_for_sessions(strategy_name, open_data[tickers],
|
||||
pd.DatetimeIndex([today]), bars).loc[today]
|
||||
target_weights = {t: float(w) for t, w in row.items() if w > 1e-6}
|
||||
|
||||
# Get today's open prices for trade computation
|
||||
open_prices = get_prices_for_date(
|
||||
@@ -854,7 +905,8 @@ def cmd_morning(args):
|
||||
trades = compute_trades(state["holdings"], state["cash"], target_weights,
|
||||
open_prices, min_trade_value=max(50, total * 0.001),
|
||||
integer_shares=args.integer_shares,
|
||||
lot_size=MARKET_LOT_SIZE.get(market, 1))
|
||||
lot_size=MARKET_LOT_SIZE.get(market, 1),
|
||||
resize_band=RESIZE_BAND)
|
||||
|
||||
# Store pending
|
||||
state["pending_trades"] = {
|
||||
@@ -1335,18 +1387,8 @@ def cmd_simulate(args):
|
||||
|
||||
# Pre-compute strategy weights for the full period (much faster than per-day)
|
||||
# Append dummy row so shift(1) inside strategy produces valid last-day signal
|
||||
strategy = STRATEGY_REGISTRY[strategy_name]()
|
||||
open_tickers = open_data[tickers]
|
||||
dummy_date = open_tickers.index[-1] + pd.Timedelta(days=1)
|
||||
dummy_row = pd.DataFrame(
|
||||
[open_tickers.iloc[-1].values], index=[dummy_date], columns=open_tickers.columns
|
||||
)
|
||||
extended = pd.concat([open_tickers, dummy_row])
|
||||
raw_weights = strategy.generate_signals(extended)
|
||||
# Strategy's shift(1): raw_weights[t] uses data up to t-1
|
||||
# For date t, the signal using open[t] is in raw_weights[t+1]
|
||||
# So shift(-1) to align: full_weights[t] = signal using open[t]
|
||||
full_weights = raw_weights.shift(-1).fillna(0.0)
|
||||
bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
|
||||
full_weights = _signals_for_sessions(strategy_name, open_data[tickers], trading_days, bars)
|
||||
|
||||
print(f"\n{'='*70}")
|
||||
print(f" SIMULATION: {strategy_name} | {market.upper()}")
|
||||
@@ -1385,6 +1427,7 @@ def cmd_simulate(args):
|
||||
state["holdings"], state["cash"], target_weights,
|
||||
close_prices, min_trade_value=max(50, pre_value * 0.001),
|
||||
integer_shares=args.integer_shares, lot_size=lot_size,
|
||||
resize_band=RESIZE_BAND,
|
||||
)
|
||||
|
||||
execute_trades(state, trades, close_prices,
|
||||
@@ -1710,13 +1753,16 @@ def cmd_monitor(args):
|
||||
def refresh_panels(market: str, strategy_names: list[str], label: str) -> None:
|
||||
"""Refresh every price panel the strategies read, once, for the union of their tickers."""
|
||||
requests: dict[str, set[str]] = {}
|
||||
fields: dict[str, set[str]] = {}
|
||||
for name in strategy_names:
|
||||
tickers, benchmark = strategy_universe(market, name)
|
||||
requests.setdefault(strategy_data_market(market, name), set()).update(
|
||||
data_market = strategy_data_market(market, name)
|
||||
requests.setdefault(data_market, set()).update(
|
||||
price_request(market, list(tickers) + [benchmark]))
|
||||
fields.setdefault(data_market, {"close", "open"}).update(strategy_fields(name))
|
||||
for data_market, request in requests.items():
|
||||
print(f"[{label}] Refreshing {data_market} panel for {len(request)} tickers")
|
||||
data_manager.update(data_market, sorted(request), with_open=True)
|
||||
data_manager.update(data_market, sorted(request), fields=sorted(fields[data_market]))
|
||||
|
||||
|
||||
def _process_sessions(args, state: dict | None, start: pd.Timestamp | None = None,
|
||||
@@ -1759,12 +1805,12 @@ def _process_sessions(args, state: dict | None, start: pd.Timestamp | None = Non
|
||||
f"{todo[0].date()} .. {todo[-1].date()}")
|
||||
|
||||
fees = _strategy_fees(args, market)
|
||||
signals = (_signals_for_sessions(strategy_name, open_data[tickers], todo)
|
||||
if len(todo) > 1 else None)
|
||||
bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
|
||||
signals = _signals_for_sessions(strategy_name, open_data[tickers], todo, bars)
|
||||
for day in todo:
|
||||
r = _run_session(state, strategy_name, market, day, tickers, benchmark,
|
||||
close_data, open_data, close_raw, fees, args.integer_shares,
|
||||
signal_row=signals.loc[day] if signals is not None else None)
|
||||
signal_row=signals.loc[day], bars=bars)
|
||||
frozen = f" | no fresh price: {', '.join(r['frozen'])}" if r["frozen"] else ""
|
||||
print(f"[{label}] {r['date']} {strategy_name:<34} value {r['value']:>12,.2f} "
|
||||
f"({r['daily_return']:+.2%}) trades {r['trades']:>2} "
|
||||
|
||||
89
universe.py
89
universe.py
@@ -76,45 +76,70 @@ def get_sp500() -> list[str]:
|
||||
# Yahoo Finance format: Shanghai → XXXXXX.SS, Shenzhen → XXXXXX.SZ
|
||||
# Wikipedia CSI 300 page has a table with stock codes and exchanges.
|
||||
|
||||
CSI300_CONS_URL = ("https://oss-ch.csindex.com.cn/static/html/csindex/public/uploads/"
|
||||
"file/autofile/cons/000300cons.xls")
|
||||
|
||||
|
||||
def _csi300_official() -> list[str]:
|
||||
"""Current constituents from China Securities Index's published file."""
|
||||
req = urllib.request.Request(CSI300_CONS_URL, headers={"User-Agent": "Mozilla/5.0"})
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
payload = resp.read()
|
||||
df = pd.read_excel(io.BytesIO(payload), dtype=str)
|
||||
code_col = next(c for c in df.columns if "Constituent Code" in c)
|
||||
exch_col = next(c for c in df.columns if c.startswith("交易所Exchange"))
|
||||
tickers = [f"{code}.{'SS' if '上海' in exch else 'SZ'}"
|
||||
for code, exch in zip(df[code_col], df[exch_col])]
|
||||
if len(tickers) < 250:
|
||||
raise RuntimeError(f"CSI 300 file has only {len(tickers)} rows")
|
||||
return sorted(set(tickers))
|
||||
|
||||
|
||||
def _csi300_latest_snapshot() -> list[str]:
|
||||
with open(os.path.join(CACHE_DIR, "csi300_history.json")) as f:
|
||||
snaps = json.load(f)["snapshots"]
|
||||
return snaps[max(snaps)]
|
||||
|
||||
|
||||
def _csi300_wikipedia() -> list[str]:
|
||||
tables = _fetch_html_tables("https://en.wikipedia.org/wiki/CSI_300_Index")
|
||||
# The constituent table has ~300 rows and a "Ticker" column like "SSE: 600519"
|
||||
df = next((t for t in tables if len(t) >= 200 and "Ticker" in t.columns), None)
|
||||
if df is None:
|
||||
raise RuntimeError("Could not find CSI 300 constituent table on Wikipedia")
|
||||
tickers = []
|
||||
for raw in df["Ticker"].astype(str):
|
||||
parts = raw.split(":")
|
||||
if len(parts) == 2 and parts[1].strip().isdigit():
|
||||
suffix = ".SS" if parts[0].strip().upper() == "SSE" else ".SZ"
|
||||
tickers.append(parts[1].strip() + suffix)
|
||||
return sorted(set(tickers))
|
||||
|
||||
|
||||
def get_csi300() -> list[str]:
|
||||
"""Fetch current CSI 300 constituents from Wikipedia."""
|
||||
"""Current CSI 300 constituents.
|
||||
|
||||
Source order: China Securities Index's published constituent file, then
|
||||
the latest snapshot in data/csi300_history.json, then Wikipedia (whose
|
||||
table lagged the real index by ~60 names in 2026).
|
||||
"""
|
||||
cached = _read_cache("csi300")
|
||||
if cached:
|
||||
print(f"--- Loaded CSI 300 universe from cache ({len(cached)} tickers) ---")
|
||||
return cached
|
||||
|
||||
print("--- Fetching CSI 300 constituents from Wikipedia ---")
|
||||
url = "https://en.wikipedia.org/wiki/CSI_300_Index"
|
||||
tables = _fetch_html_tables(url)
|
||||
|
||||
# Find the constituent table — it has ~300 rows and a "Ticker" column
|
||||
# with format like "SSE: 600519" or "SZSE: 300750"
|
||||
df = None
|
||||
for t in tables:
|
||||
if len(t) >= 200 and "Ticker" in t.columns:
|
||||
df = t
|
||||
break
|
||||
|
||||
if df is None:
|
||||
raise RuntimeError("Could not find CSI 300 constituent table on Wikipedia")
|
||||
|
||||
tickers = []
|
||||
for _, row in df.iterrows():
|
||||
raw = str(row["Ticker"]) # e.g. "SSE: 600519" or "SZSE: 300750"
|
||||
parts = raw.split(":")
|
||||
if len(parts) != 2:
|
||||
for label, source in (("csindex.com.cn", _csi300_official),
|
||||
("stored CSI 300 history", _csi300_latest_snapshot),
|
||||
("Wikipedia", _csi300_wikipedia)):
|
||||
try:
|
||||
tickers = source()
|
||||
except Exception as exc: # noqa: BLE001 — fall through to the next source
|
||||
print(f"--- CSI 300 from {label} failed: {exc} ---")
|
||||
continue
|
||||
exchange_prefix = parts[0].strip().upper()
|
||||
code = parts[1].strip()
|
||||
if not code.isdigit():
|
||||
continue
|
||||
suffix = ".SS" if exchange_prefix == "SSE" else ".SZ"
|
||||
tickers.append(code + suffix)
|
||||
|
||||
tickers = sorted(set(tickers))
|
||||
_write_cache("csi300", tickers)
|
||||
print(f"--- Got {len(tickers)} CSI 300 tickers ---")
|
||||
return tickers
|
||||
_write_cache("csi300", tickers)
|
||||
print(f"--- Got {len(tickers)} CSI 300 tickers from {label} ---")
|
||||
return tickers
|
||||
raise RuntimeError("No CSI 300 constituent source available")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -20,6 +20,7 @@ import os
|
||||
import urllib.request
|
||||
from datetime import date, datetime
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
CACHE_DIR = "data"
|
||||
@@ -30,34 +31,51 @@ _HEADERS = {"User-Agent": "Mozilla/5.0 (quant-backtest)"}
|
||||
# Fetch + parse Wikipedia
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _fetch_sp500_tables() -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||
"""Return (current_list, changes_log) from the S&P 500 Wikipedia page."""
|
||||
url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
|
||||
SP500_LIST_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
|
||||
# The change log moved off the list page in 2026; the list page is still tried
|
||||
# first in case it moves back.
|
||||
SP500_CHANGES_URLS = [
|
||||
SP500_LIST_URL,
|
||||
"https://en.wikipedia.org/wiki/Historical_components_of_the_S%26P_500",
|
||||
]
|
||||
|
||||
|
||||
def _read_tables(url: str) -> list[pd.DataFrame]:
|
||||
req = urllib.request.Request(url, headers=_HEADERS)
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
html = resp.read().decode("utf-8")
|
||||
tables = pd.read_html(io.StringIO(html))
|
||||
current = tables[0]
|
||||
change_tables = [
|
||||
t for t in tables[1:]
|
||||
if any("Added" in "_".join(map(str, c if isinstance(c, tuple) else (c,))) for c in t.columns)
|
||||
]
|
||||
if not change_tables:
|
||||
raise ValueError("S&P 500 change log table not found on the Wikipedia page")
|
||||
changes = change_tables[0]
|
||||
changes.columns = [
|
||||
"_".join(c).strip() if isinstance(c, tuple) else c
|
||||
for c in changes.columns
|
||||
]
|
||||
changes.columns = [
|
||||
c.replace("Effective Date_Effective Date", "Date") for c in changes.columns
|
||||
]
|
||||
return current, changes
|
||||
return pd.read_html(io.StringIO(html))
|
||||
|
||||
|
||||
def _find_changes_table(tables: list[pd.DataFrame]) -> pd.DataFrame | None:
|
||||
for t in tables:
|
||||
if any("Added" in "_".join(map(str, c if isinstance(c, tuple) else (c,))) for c in t.columns):
|
||||
changes = t.copy()
|
||||
changes.columns = [
|
||||
"_".join(c).strip() if isinstance(c, tuple) else c
|
||||
for c in changes.columns
|
||||
]
|
||||
changes.columns = [
|
||||
c.replace("Effective Date_Effective Date", "Date") for c in changes.columns
|
||||
]
|
||||
return changes
|
||||
return None
|
||||
|
||||
|
||||
def _fetch_sp500_tables() -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||
"""Return (current_list, changes_log) from Wikipedia."""
|
||||
current = _read_tables(SP500_LIST_URL)[0]
|
||||
for url in SP500_CHANGES_URLS:
|
||||
changes = _find_changes_table(_read_tables(url))
|
||||
if changes is not None:
|
||||
return current, changes
|
||||
raise ValueError("S&P 500 change log table not found on Wikipedia")
|
||||
|
||||
|
||||
def _normalize_ticker(t: str) -> str:
|
||||
"""Yahoo Finance ticker format: BRK.B → BRK-B."""
|
||||
return str(t).replace(".", "-").strip()
|
||||
# Some change-log cells carry a trailing " |" separator.
|
||||
return str(t).split("|")[0].replace(".", "-").strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -243,3 +261,101 @@ def mask_prices(prices: pd.DataFrame,
|
||||
"""
|
||||
mask = membership_mask(prices.index, intervals, tickers=list(prices.columns))
|
||||
return prices.where(mask)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CSI 300 (China A-shares)
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# data/csi300_history.json holds monthly constituent snapshots
|
||||
# {"snapshots": {"YYYY-MM-DD": [yahoo tickers], ...}}
|
||||
# seeded from baostock's query_hs300_stocks (first trading day of each month,
|
||||
# 2008 onward). A snapshot applies from its date until the next one, so an
|
||||
# index change is picked up within a month. load_csi300_history() appends a
|
||||
# snapshot whenever the current constituent list (universe.get_csi300) differs
|
||||
# from the latest one, which keeps the file current without baostock.
|
||||
|
||||
def _csi300_path() -> str:
|
||||
return os.path.join(CACHE_DIR, "csi300_history.json")
|
||||
|
||||
|
||||
def baostock_to_yahoo(code: str) -> str:
|
||||
"""'sh.600000' -> '600000.SS', 'sz.000001' -> '000001.SZ'."""
|
||||
ex, num = code.split(".")
|
||||
return f"{num}.{'SS' if ex == 'sh' else 'SZ'}"
|
||||
|
||||
|
||||
def load_csi300_history(current: list[str] | None = None) -> dict[str, list[str]]:
|
||||
"""Monthly CSI 300 snapshots; appends ``current`` (or today's list) if it changed."""
|
||||
path = _csi300_path()
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
snaps = data["snapshots"]
|
||||
if current is None:
|
||||
try:
|
||||
from universe import get_csi300
|
||||
current = get_csi300()
|
||||
except Exception as exc: # noqa: BLE001 — offline runs use the stored history
|
||||
print(f"--- WARNING: CSI 300 list unavailable ({exc}); using stored history ---")
|
||||
current = None
|
||||
if current:
|
||||
last = snaps[max(snaps)]
|
||||
today = str(date.today())
|
||||
if sorted(current) != sorted(last) and today > max(snaps):
|
||||
snaps[today] = sorted(current)
|
||||
with open(path, "w") as f:
|
||||
json.dump(data, f, indent=0)
|
||||
print(f"--- CSI 300 constituents changed; stored snapshot {today} ---")
|
||||
return snaps
|
||||
|
||||
|
||||
def csi300_membership_mask(dates: pd.DatetimeIndex, tickers: list[str],
|
||||
snapshots: dict[str, list[str]] | None = None) -> pd.DataFrame:
|
||||
"""Boolean mask (dates x tickers): True while the ticker was in the CSI 300."""
|
||||
snaps = snapshots if snapshots is not None else load_csi300_history()
|
||||
keys = sorted(snaps)
|
||||
col = {t: i for i, t in enumerate(tickers)}
|
||||
arr = np.zeros((len(dates), len(tickers)), dtype=bool)
|
||||
for i, k in enumerate(keys):
|
||||
start = pd.Timestamp(k)
|
||||
end = pd.Timestamp(keys[i + 1]) if i + 1 < len(keys) else dates[-1] + pd.Timedelta(days=1)
|
||||
rows = (dates >= start) & (dates < end)
|
||||
cols = [col[t] for t in snaps[k] if t in col]
|
||||
if rows.any() and cols:
|
||||
arr[np.ix_(rows, cols)] = True
|
||||
mask = pd.DataFrame(arr, index=dates, columns=tickers)
|
||||
# Tickers never in the index (benchmarks, ETFs) pass through unmasked.
|
||||
ever = {t for lst in snaps.values() for t in lst}
|
||||
for t in tickers:
|
||||
if t not in ever:
|
||||
mask[t] = True
|
||||
return mask
|
||||
|
||||
|
||||
def all_csi300_tickers_ever(snapshots: dict[str, list[str]] | None = None,
|
||||
since: str | None = None) -> list[str]:
|
||||
snaps = snapshots if snapshots is not None else load_csi300_history()
|
||||
return sorted({t for k, lst in snaps.items() if since is None or k >= since for t in lst})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Market-generic helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def pit_tickers_ever(market: str, since: str | None = None) -> list[str] | None:
|
||||
"""Every ticker that was an index member (since ``since``), or None when the
|
||||
market has no membership history."""
|
||||
if market == "us":
|
||||
return all_tickers_ever()
|
||||
if market == "cn":
|
||||
return all_csi300_tickers_ever(since=since)
|
||||
return None
|
||||
|
||||
|
||||
def mask_prices_pit(prices: pd.DataFrame, market: str) -> pd.DataFrame:
|
||||
"""NaN out (date, ticker) pairs outside point-in-time index membership."""
|
||||
if market == "us":
|
||||
return mask_prices(prices)
|
||||
if market == "cn":
|
||||
return prices.where(csi300_membership_mask(prices.index, list(prices.columns)))
|
||||
return prices
|
||||
|
||||
11
uv.lock
generated
11
uv.lock
generated
@@ -770,6 +770,7 @@ dependencies = [
|
||||
{ name = "numpy" },
|
||||
{ name = "pandas" },
|
||||
{ name = "scipy" },
|
||||
{ name = "xlrd" },
|
||||
{ name = "yfinance" },
|
||||
]
|
||||
|
||||
@@ -780,6 +781,7 @@ requires-dist = [
|
||||
{ name = "numpy", specifier = ">=2.4.0" },
|
||||
{ name = "pandas", specifier = ">=2.3.3" },
|
||||
{ name = "scipy", specifier = ">=1.17.1" },
|
||||
{ name = "xlrd", specifier = ">=2.0.2" },
|
||||
{ name = "yfinance", specifier = ">=1.0" },
|
||||
]
|
||||
|
||||
@@ -935,6 +937,15 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "xlrd"
|
||||
version = "2.0.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/07/5a/377161c2d3538d1990d7af382c79f3b2372e880b65de21b01b1a2b78691e/xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9", size = 100167, upload-time = "2025-06-14T08:46:39.039Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/1a/62/c8d562e7766786ba6587d09c5a8ba9f718ed3fa8af7f4553e8f91c36f302/xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9", size = 96555, upload-time = "2025-06-14T08:46:37.766Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "yfinance"
|
||||
version = "1.0"
|
||||
|
||||
Reference in New Issue
Block a user