feat: robust-momentum strategies live; fix live universe, lots and UI backtests

Strategies
- BarStrategy protocol (close/open/high/low/volume bars; row d uses bars
  through d-1) and RobustMomentumStrategy: overnight_recovery_top10 (US),
  lowrisk_momentum_top15 and lowbeta_sharpe_mom_top20 (CN), top-N with
  rank hysteresis, 10-day rebalance, weights drift between rebalances.
- trader: strategy_fields/strategy_bars/bar_session_weights; one signal
  path (_signals_for_sessions) replaces generate_target_weights.

Live fixes
- CSI 300 constituents come from China Securities Index's file; the
  Wikipedia table was 122 names off the real index.
- S&P 500 change log moved to "Historical components of the S&P 500".
- data_manager: tickers missing from a partial Yahoo batch are retried
  alone before being marked unavailable (CEG, GEV, SNDK ... were dropped).
  update() takes fields=[...] and stores high/low/volume panels.
- Whole-lot orders go to the nearest lot within cash (CN books held ~62%
  cash); held names are resized only beyond RESIZE_BAND (10%), so books no
  longer pay a ticket fee to trade daily drift.

Web backtests
- backtest_runner.py replaces the duplicated bridge/precompute code: PIT
  membership in both markets (data/csi300_history.json), stock universe
  limited to index members (ETFs and benchmarks in the shared panel were
  pickable), and the book kept like the live trader (simulate_live_book:
  signal timing, drift, order floor, resize band, fees on orders placed).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -9,7 +9,7 @@ quant-trading: a quantitative backtesting framework and live forward-testing tra
 The repository has two halves that share one data directory and one `.env`:
 
 - **Python engine** (repo root, `uv`): strategies, backtests, `trader.py`, research scripts.
-- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`.
+- **Web UI and API** (`web/`, Node + React + Vite): the API spawns `uv run python bridge.py {list,backtest,live}` at the repo root and reads `data/precomputed/backtest_results.json`, written by `precompute_backtest.py`. Both backtest paths run through `backtest_runner.py`.
 
 `bridge.py`'s JSON output and `web/src/shared/strategyTypes.ts` are one contract: change both in the same commit.
 
@@ -40,6 +40,13 @@ uv run python trader.py simulate --integer-shares --fixed-fee 2.0 ...    # Integ
 # Precompute the strategy page's backtests (writes data/precomputed/)
 uv run python precompute_backtest.py
 
+# Factor research harness (PIT universe, next-open fills, phase averaging; data in data/research/)
+uv run python -m research.lab.fetch_us                 # 20y OHLCV for every S&P 500 ever-member
+uv run python -m research.lab.fetch_cn_yahoo           # CSI 300/500 ever-members (membership from baostock snapshot)
+PYTHONPATH=. uv run python -m research.lab.factor_scan --market us       # or cn_hs300
+PYTHONPATH=. uv run python -m research.lab.stepwise --market us --objective maximin
+PYTHONPATH=. uv run python -m research.lab.evaluate --market us          # candidates vs champions
+
 # Setup
 uv sync                                  # Install/sync dependencies
 cp .env.example .env                     # Then fill IPADS_API_KEY and QUANT_MARKET_PROXY
@@ -55,15 +62,17 @@ uv run python -m unittest discover tests
 
 ## Architecture
 
-**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia, CSI 300 from Wikipedia) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
+**Universe provider** (`universe.py`): Dynamically fetches index constituents (S&P 500 from Wikipedia; CSI 300 from China Securities Index's published constituent file, falling back to the latest `data/csi300_history.json` snapshot, then Wikipedia, whose table lagged the index by ~60 names in 2026) and converts to Yahoo Finance ticker format. Constituent lists are cached daily in `data/universe_*.json`. New markets are added by registering in `UNIVERSES` dict.
 
-**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_open.csv` (open). First run downloads 10 years of history; subsequent runs append only new trading days (fast bulk fetch for gaps ≤7 days). New tickers from index rebalances are backfilled automatically. Tickers with >50% missing data are dropped. Pass `with_open=True` to also maintain open price files.
+**Point-in-time membership** (`universe_history.py`): S&P 500 intervals rebuilt from Wikipedia's "Historical components of the S&P 500" change log into `data/sp500_history.json`; CSI 300 monthly snapshots in `data/csi300_history.json` (seeded from baostock 2008–2026, appended whenever the current list changes). `mask_prices_pit(prices, market)` masks non-member dates.
 
-**Backtest engine** (`main.py`): Orchestrates data loading, strategy execution, and visualization. The `backtest()` function is vectorized — it takes a strategy and price DataFrame, applies transaction costs (proportional + optional fixed per-trade fee) via turnover, and returns an equity curve. Supports two execution modes: `close` (classic) and `open-close` (signal on open prices, execute at close).
+**Data manager** (`data_manager.py`): Persistent price storage in `data/{market}.csv` (close) and `data/{market}_{open,high,low,volume}.csv`. First run downloads 10 years of history; every run re-downloads the last 30 days. New tickers from index rebalances are backfilled automatically; tickers missing from a partial batch are retried alone before being marked unavailable (retried weekly). `update(market, tickers, with_open=True)` returns (close, open); `update(..., fields=[...])` returns `{field: panel}`.
 
-**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, plus 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`).
+**Backtest engine** (`main.py`): Orchestrates data loading, strategy execution, and visualization. `backtest()` has two modes. Close mode (no `open_data`, used by `main.py` and old research scripts) is vectorized and trades at the close the signal saw. Live mode (`open_data` given, used by the web UI through `backtest_runner.py`) keeps the book like `trader.py`: same signal timing (`strategy_positions`), holdings that drift, the $50/0.1% order floor, `trader.RESIZE_BAND`, fees on the orders placed (`simulate_live_book`).
 
-**Strategy protocol** (`strategies/base.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy is responsible for applying its own 1-day lag via `.shift(1)` to avoid lookahead bias — the backtest engine does not shift.
+**Daily trader** (`trader.py`): Live/forward-testing system with persistent portfolio state in `data/trader_{market}_{strategy}.json`. The `auto` subcommand runs both signal generation and execution in a single invocation — designed for cron. The `simulate` subcommand replays a date range day-by-day with realistic portfolio tracking (fractional shares, cash, commissions). Orders: whole A-share lots are allocated nearest-lot within cash (`_allocate_lots`); a held name is resized only when ≥ `RESIZE_BAND` (10%) off target. Available strategies: `recovery_mom_top10`, `recovery_mom_top20`, `momentum`, `momentum_quality`, `dual_momentum`, `inverse_vol`, `trend_following`, `buy_and_hold`, 32 factor-combo strategies (`fc_{signal}_{freq}` — see `strategies/factor_combo.py`), and the robust-momentum bar strategies `overnight_recovery_top10`, `lowrisk_momentum_top15`, `lowbeta_sharpe_mom_top20`.
+
+**Strategy protocol** (`strategies/base.py`): All strategies inherit from `Strategy` ABC and implement `generate_signals(data) → DataFrame` where the returned DataFrame contains portfolio weights (rows = dates, columns = assets, values sum to ~1.0 per row). Each strategy is responsible for applying its own 1-day lag via `.shift(1)` to avoid lookahead bias — the backtest engine does not shift. A `BarStrategy` declares `fields` (close/open/high/low/volume) and receives `{field: DataFrame}`; live it uses completed bars only (row d from bars through d-1, traded at the close of d), while close-price strategies read the session's open as their latest price.
 
 **Strategies**:
 - `buy_and_hold.py` — Equal-weight static allocation
@@ -76,7 +85,8 @@ uv run python -m unittest discover tests
 - `momentum_quality.py` — Momentum + return consistency + low drawdown
 - `adaptive_momentum.py` — Momentum weighted by inverse volatility
 - `recovery_momentum.py` — Recovery (price/63d low) + 12-1mo momentum composite. Best US performer.
-- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol` (50.7% CAGR monthly), `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom` (26.1% CAGR monthly), `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap`. All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
+- `robust_momentum.py` — Bar strategies from the 2026-09 PIT research (report: https://claude.ai/artifact/Mo9nrzs5SWj5FfyshERyWS): US `overnight_recovery` (overnight-return momentum + distance from 126d low), CN `lowrisk_momentum` (amplitude-cut momentum + low beta + price-volume corr + momentum/vol) and close-only `lowbeta_sharpe_mom`. Top-N with rank hysteresis, rebalance every 10 days, weights drift between rebalances.
+- `factor_combo.py` — Configurable factor-combination strategies with daily/weekly/biweekly/monthly rebalancing. US champions: `rec_mfilt+deep_upvol`, `ma200+mom7m+rec126`, `rec_mfilt+ma200`, `mom7m+rec126`. CN champions: `up_cap+quality_mom`, `down_resil+qual_mom`, `rec63+mom_gap`, `up_cap+mom_gap` — these were mined on today's CSI 300 members; on PIT members they earn ~3-7% a year (2016-2026). All registered in trader.py as `fc_{signal}_{freq}` (e.g., `fc_rec_mfilt_deep_upvol_monthly`). 32 new strategies total.
 
 **Metrics** (`metrics.py`): Standalone functions for portfolio analytics (Sharpe, Sortino, Calmar, max drawdown, etc.). `summary()` prints a formatted report and returns a dict.
 
@@ -84,7 +94,7 @@ uv run python -m unittest discover tests
 
 - Strategies output **weights**, not buy/sell signals. Weights are normalized so each row sums to 1 (or 0 during warm-up).
 - Warm-up periods: each strategy zeros out its first N rows where indicators are undefined.
-- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests) and `data/web/` (watchlist, per-stock cache, backtest result cache). Not checked into git (only `data/.gitkeep` and `data/sp500_history.json` are tracked).
+- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests), `data/web/` (watchlist, per-stock cache, backtest result cache) and `data/research/` (research panels, ~2 GB). Not checked into git (only `data/.gitkeep`, `data/sp500_history.json` and `data/csi300_history.json` are tracked).
 - Python 3.12+, managed with `uv`.
 - Do not show matplotlib figures when running backtests; use `--no-plot`.
 

backtest_runner.py

@@ -0,0 +1,197 @@
+"""
+Market-level backtests behind the web UI: ``bridge.py backtest`` (on demand)
+and ``precompute_backtest.py`` (the strategy page) share this one data path.
+
+* Universe: point-in-time index members in both markets (S&P 500 history from
+  Wikipedia, CSI 300 history in data/csi300_history.json); prices outside a
+  stock's membership are masked, so strategies cannot pick it then.
+* Book: kept like the live trader keeps it — its signal timing
+  (``main.strategy_positions`` live mode), drifting holdings, the same order
+  size floor and resize band (``main.simulate_live_book``).
+* Bars: panels a BarStrategy needs (open/high/low/volume) are loaded too.
+"""
+
+from __future__ import annotations
+
+import sys
+
+import pandas as pd
+
+import data_manager
+import universe_history as uh
+from main import simulate_live_book, strategy_positions
+from strategies.base import BarStrategy
+from trader import (
+    ETF_STRATEGY_UNIVERSES,
+    LIVE_EXCLUDED_STRATEGIES,
+    RESIZE_BAND,
+    STRATEGY_REGISTRY,
+    convert_quote_currency,
+    filter_tradable_tickers,
+    price_request,
+    strategy_extra_tickers,
+    strategy_fields,
+)
+from universe import UNIVERSES
+
+# Rebalances below this size are not charged the per-trade fee in backtests,
+# matching the live trader's minimum order size.
+MIN_TRADE_VALUE = 50.0
+
+
+def _log(msg: str) -> None:
+    print(msg, file=sys.stderr)
+
+
+def trade_events(diff: pd.DataFrame, threshold: float = 0.01) -> list[dict]:
+    """Trade events from per-ticker traded weights (rows = days)."""
+    trades = []
+    for date in diff.index:
+        row = diff.loc[date]
+        significant = row[row.abs() > threshold]
+        if significant.empty:
+            continue
+        buys = significant[significant > 0]
+        sells = significant[significant < 0]
+        if len(buys) > 0 or len(sells) > 0:
+            trades.append({
+                "date": date.strftime("%Y-%m-%d"),
+                "buys": len(buys),
+                "sells": len(sells),
+                "buyTickers": list(buys.nlargest(3).index),
+                "sellTickers": list(sells.nsmallest(3).index),
+            })
+    return trades
+
+
+def benchmark_curve(prices: pd.Series, dates: pd.DatetimeIndex, capital: float) -> list[dict]:
+    """Benchmark value on ``dates``, rebased to ``capital`` on the first date it has a price."""
+    series = prices.reindex(dates).ffill().dropna()
+    if series.empty:
+        return []
+    values = series / series.iloc[0] * capital
+    return [{"date": d.strftime("%Y-%m-%d"), "value": round(float(v), 2)} for d, v in values.items()]
+
+
+def _slice_years(panels: dict[str, pd.DataFrame], years: int | None) -> dict[str, pd.DataFrame]:
+    if not years:
+        return panels
+    close = panels["close"]
+    cutoff = close.index[-1] - pd.DateOffset(years=years)
+    return {f: p[p.index >= cutoff] for f, p in panels.items()}
+
+
+def load_stock_panels(market: str, strategy_names: list[str], years: int | None) -> dict:
+    """PIT-masked panels for the market's stock strategies."""
+    universe = UNIVERSES[market]
+    tickers = universe["fetch"]()
+    benchmark = universe["benchmark"]
+    since = None
+    if years:
+        since = (pd.Timestamp.today() - pd.DateOffset(years=years + 1)).strftime("%Y-%m-%d")
+    ever = uh.pit_tickers_ever(market, since=since) or []
+    extras = {t for n in strategy_names for t in strategy_extra_tickers(n)}
+    request = sorted(set(tickers) | set(ever) | extras | {benchmark})
+    fields = sorted({"close", "open"} | {f for n in strategy_names for f in strategy_fields(n)})
+    _log(f"PIT universe: {len(request)} tickers ({len(ever)} ever index members); fields {fields}")
+    panels = data_manager.update(market, request, fields=fields)
+    panels = _slice_years(panels, years)
+    member = None
+    if ever:
+        member = uh.mask_prices_pit(panels["close"], market).notna() | panels["close"].isna()
+        # Only tickers that were index members: the shared panel also holds
+        # benchmarks and the ETFs of mixed strategies (TQQQ, SOXL, ...), which
+        # the membership mask passes through unmasked.
+        members_ever = set(ever)
+        stocks = [t for t in panels["close"].columns
+                  if t in members_ever and t not in extras
+                  and panels["close"][t].where(member[t]).notna().any()]
+    else:
+        stocks = filter_tradable_tickers(panels["close"], tickers)
+    return {"panels": panels, "member": member, "stocks": stocks, "benchmark": benchmark,
+            "benchmark_label": universe["benchmark_label"]}
+
+
+def _run(strategy, panels: dict[str, pd.DataFrame], cols: list[str], capital: float,
+         tx_cost: float, fixed_fee: float, fee_base: float, fee_per_share: float,
+         member: pd.DataFrame | None = None, min_trade_value: float = MIN_TRADE_VALUE):
+    """Close-price strategies see prices masked to index membership (they
+    cannot pick a non-member); a BarStrategy sees full histories plus the
+    membership as ``bars["eligible"]``. The book is valued on unmasked closes."""
+    close, opens = panels["close"][cols], panels["open"][cols]
+    if isinstance(strategy, BarStrategy):
+        bars = {f: panels[f][cols] for f in strategy.fields}
+        if member is not None:
+            bars["eligible"] = member.reindex(columns=cols).fillna(True)
+        held = strategy_positions(strategy, close, open_data=opens, bars=bars)
+    else:
+        m = member.reindex(columns=cols).fillna(True) if member is not None else None
+        held = strategy_positions(strategy, close.where(m) if m is not None else close,
+                                  open_data=opens.where(m) if m is not None else opens)
+    equity, traded = simulate_live_book(held, close, capital, tx_cost, fixed_fee, fee_base,
+                                        fee_per_share, min_trade_value, RESIZE_BAND)
+    return equity, trade_events(traded)
+
+
+def run_market(market: str, strategy_names: list[str], years: int | None, capital: float,
+               tx_cost: float, fixed_fee: float, fee_base: float = 0.0,
+               fee_per_share: float = 0.0) -> dict | None:
+    """Backtest every strategy once; returns raw equity series and metadata."""
+    stock = [n for n in strategy_names if n.removeprefix("sim_") not in ETF_STRATEGY_UNIVERSES]
+    etf = [n for n in strategy_names if n.removeprefix("sim_") in ETF_STRATEGY_UNIVERSES]
+    fees = dict(tx_cost=tx_cost, fixed_fee=fixed_fee, fee_base=fee_base, fee_per_share=fee_per_share)
+    runs: list[dict] = []
+    out = {"market": market, "benchmark_label": "SPY", "benchmark_curve": [],
+           "start_date": None, "end_date": None, "tx_cost": tx_cost, "fixed_fee": fixed_fee}
+
+    if stock:
+        data = load_stock_panels(market, stock, years)
+        panels, stocks = data["panels"], data["stocks"]
+        close = panels["close"]
+        _log(f"Universe: {len(stocks)} stocks, {close.index[0].date()} to {close.index[-1].date()}")
+        for name in stock:
+            _log(f"  Running: {name}")
+            try:
+                strategy = STRATEGY_REGISTRY[name]()
+                cols = stocks + [t for t in strategy_extra_tickers(name)
+                                 if t in close.columns and t not in stocks]
+                # Strategies the live book cannot hold (e.g. equal weight over
+                # ~500 names) are shown without the live order-size floor.
+                floor = 0.0 if name in LIVE_EXCLUDED_STRATEGIES else MIN_TRADE_VALUE
+                equity, trades = _run(strategy, panels, cols, capital, **fees,
+                                      member=data["member"], min_trade_value=floor)
+                runs.append({"name": name, "equity": equity, "trades": trades})
+                _log(f"    OK: {name} (final: {equity.iloc[-1]:,.0f})")
+            except Exception as e:  # noqa: BLE001 — one broken strategy must not sink the page
+                _log(f"    FAILED: {name}: {e}")
+        out.update(benchmark_label=data["benchmark_label"],
+                   benchmark_curve=benchmark_curve(close[data["benchmark"]], close.index, capital),
+                   start_date=close.index[0].strftime("%Y-%m-%d"),
+                   end_date=close.index[-1].strftime("%Y-%m-%d"))
+
+    for name in etf:
+        _log(f"  Running ETF strategy: {name}")
+        try:
+            etf_tickers = ETF_STRATEGY_UNIVERSES[name.removeprefix("sim_")]
+            close, opens = data_manager.update(
+                "etfs", price_request(market, sorted(set(etf_tickers + ["SPY"]))), with_open=True)
+            panels = _slice_years({"close": convert_quote_currency(close),
+                                   "open": convert_quote_currency(opens)}, years)
+            strategy = STRATEGY_REGISTRY[name]()
+            equity, trades = _run(strategy, panels, filter_tradable_tickers(panels["close"], etf_tickers),
+                                  capital, **fees)
+            runs.append({"name": name, "equity": equity, "trades": trades})
+            _log(f"    OK: {name} (final: {equity.iloc[-1]:,.0f})")
+            if not stock and not out["benchmark_curve"] and "SPY" in panels["close"].columns:
+                spy = panels["close"]["SPY"]
+                out.update(benchmark_label="SPY (Benchmark)",
+                           benchmark_curve=benchmark_curve(spy, spy.index, capital),
+                           start_date=spy.index[0].strftime("%Y-%m-%d"),
+                           end_date=spy.index[-1].strftime("%Y-%m-%d"))
+        except Exception as e:  # noqa: BLE001
+            _log(f"    FAILED: {name}: {e}")
+
+    if not runs:
+        return None
+    out["strategy_runs"] = runs
+    return out

bridge.py

@@ -20,29 +20,19 @@ import pandas as pd
 
 import data_manager
 import metrics
-import universe_history as uh
-from main import backtest
+from backtest_runner import benchmark_curve, run_market
 from research.permanent_yearly import equity_with_cashflows
 from trader import (
-    strategy_extra_tickers,
-    strategy_supports_market,
-    ETF_STRATEGY_UNIVERSES,
     LIVE_EXCLUDED_STRATEGIES,
     MARKET_FEE_TIERED,
     STRATEGY_REGISTRY,
     convert_quote_currency,
-    price_request,
-    strategy_supports_market,
     strategy_data_market,
-    filter_tradable_tickers,
+    strategy_supports_market,
 )
 
 MARKET_CURRENCY = {"us": "USD", "cn": "CNY"}
 
-# Rebalances below this size are not charged the per-trade fee in backtests,
-# matching the live trader's minimum order size.
-MIN_TRADE_VALUE = 50.0
-from universe import UNIVERSES
 
 
 # ---------------------------------------------------------------------------
@@ -440,6 +430,25 @@ STRATEGY_META = {
         "params": {},
         "markets": ["us"],
     },
+    # --- Robust momentum (2026-09 PIT research): US pick, CN picks ---
+    "overnight_recovery_top10": {
+        "label": "Robust: Overnight Mom + Recovery Top 10",
+        "category": "factor_combo",
+        "params": {},
+        "markets": ["us", "cn"],
+    },
+    "lowrisk_momentum_top15": {
+        "label": "Robust: Low-Risk Momentum Top 15",
+        "category": "factor_combo",
+        "params": {},
+        "markets": ["us", "cn"],
+    },
+    "lowbeta_sharpe_mom_top20": {
+        "label": "Robust: Low-Beta + Sharpe Mom Top 20",
+        "category": "factor_combo",
+        "params": {},
+        "markets": ["us", "cn"],
+    },
 }
 
 
@@ -515,28 +524,6 @@ def compute_drawdown_periods(equity: pd.Series, threshold: float = -0.05) -> lis
     return periods
 
 
-def compute_rebalance_trades(weights: pd.DataFrame, threshold: float = 0.01) -> list[dict]:
-    """Extract significant trade events from weight changes."""
-    diff = weights.diff().fillna(0.0)
-    trades = []
-    for date in diff.index:
-        row = diff.loc[date]
-        significant = row[row.abs() > threshold]
-        if significant.empty:
-            continue
-        buys = significant[significant > 0]
-        sells = significant[significant < 0]
-        if len(buys) > 0 or len(sells) > 0:
-            trades.append({
-                "date": date.strftime("%Y-%m-%d"),
-                "buys": len(buys),
-                "sells": len(sells),
-                "buyTickers": list(buys.nlargest(3).index),
-                "sellTickers": list(sells.nsmallest(3).index),
-            })
-    return trades
-
-
 def compute_rebalance_win_rate(equity: pd.Series, trade_events: list[dict]) -> float | None:
     """Fraction of inter-rebalance periods with positive cumulative return.
 
@@ -600,15 +587,6 @@ def compute_periodic_returns(equity: pd.Series) -> tuple[dict, dict]:
     return monthly, yearly
 
 
-def benchmark_curve(prices: pd.Series, dates: pd.DatetimeIndex, capital: float) -> list[dict]:
-    """Benchmark value on ``dates``, rebased to ``capital`` on the first date it has a price."""
-    series = prices.reindex(dates).ffill().dropna()
-    if series.empty:
-        return []
-    values = series / series.iloc[0] * capital
-    return [{"date": d.strftime("%Y-%m-%d"), "value": round(float(v), 2)} for d, v in values.items()]
-
-
 def build_injection_schedule(equity: pd.Series, schedule: dict) -> tuple[pd.Series, list[dict]]:
     """Build a contributions Series from injection schedule config.
 
@@ -725,159 +703,18 @@ def cmd_backtest(args: argparse.Namespace) -> None:
     _log(f"Running backtest: market={market}, years={years}, capital={capital}")
     _log(f"Strategies: {strategy_names}")
 
-    # Determine data needs
-    # Group strategies by their data source
-    stock_strategies = []
-    etf_strategies = []
-    for name in strategy_names:
-        if name not in STRATEGY_REGISTRY:
-            _log(f"WARNING: Unknown strategy '{name}', skipping")
-            continue
-        base_name = name.removeprefix("sim_")
-        if base_name in ETF_STRATEGY_UNIVERSES:
-            etf_strategies.append(name)
-        else:
-            stock_strategies.append(name)
-
-    results = []
-
-    # Process stock strategies
-    if stock_strategies:
-        universe = UNIVERSES[market]
-        with _redirect_stdout_to_stderr():
-            tickers = universe["fetch"]()
-        benchmark_ticker = universe["benchmark"]
-        benchmark_label = universe["benchmark_label"]
-
-        # PIT universe: for US market, include all tickers that were EVER
-        # S&P 500 members so that removed stocks are present in price data.
-        pit_intervals = None
-        if market == "us":
-            with _redirect_stdout_to_stderr():
-                pit_intervals = uh.load_sp500_history()
-            historical_tickers = uh.all_tickers_ever(pit_intervals)
-            all_tickers = sorted(set(tickers + historical_tickers + [benchmark_ticker])
-                                 | {t for n in stock_strategies for t in strategy_extra_tickers(n)})
-            _log(f"PIT universe: {len(all_tickers)} tickers (current + historical members)")
-        else:
-            all_tickers = sorted(set(tickers + [benchmark_ticker])
-                                 | {t for n in stock_strategies for t in strategy_extra_tickers(n)})
-            _log(f"WARNING: {market.upper()} market lacks PIT membership history; "
-                 "backtest uses current index constituents (survivorship bias possible)")
-
-        _log(f"Fetching price data for {len(all_tickers)} tickers...")
-        with _redirect_stdout_to_stderr():
-            data = data_manager.update(market, all_tickers, with_open=False)
-        if isinstance(data, tuple):
-            data = data[0]
-
-        if years:
-            cutoff = data.index[-1] - pd.DateOffset(years=years)
-            data = data[data.index >= cutoff]
-
-        # Apply PIT mask: set prices to NaN for dates when a stock was NOT
-        # an index member. Strategies naturally skip NaN entries.
-        if pit_intervals is not None:
-            _log("Applying PIT membership mask (survivorship-bias fix)...")
-            data = uh.mask_prices(data, pit_intervals)
-
-        tickers = filter_tradable_tickers(data, tickers)
-        # For PIT mode, expand tickers to all columns with valid data
-        # (historical members that have been masked are NaN when not members,
-        # but have valid data during their membership period).
-        if pit_intervals is not None:
-            all_data_tickers = [
-                t for t in data.columns
-                if t != benchmark_ticker and data[t].notna().any()
-            ]
-            tickers = all_data_tickers
-        _log(f"Universe: {len(tickers)} tradable stocks, period: {data.index[0].date()} to {data.index[-1].date()}")
-
-        for name in stock_strategies:
-            _log(f"  Running: {name}")
-            # Registry defaults are the single source of strategy parameters,
-            # shared with the live trader.
-            strategy = STRATEGY_REGISTRY[name]()
-            # Mixed stock+ETF strategies also read their fixed ETFs.
-            cols = tickers + [t for t in strategy_extra_tickers(name)
-                              if t in data.columns and t not in tickers]
-
-            equity = backtest(
-                strategy, data[cols],
-                initial_capital=capital,
-                transaction_cost=tx_cost,
-                fixed_fee=fixed_fee,
-                min_trade_value=MIN_TRADE_VALUE,
-                **MARKET_FEE_TIERED.get(market, {}),
-            )
-
-            # Get weight signals for trade extraction
-            weights = strategy.generate_signals(data[cols])
-            weights = weights.reindex(data[cols].index).fillna(0.0)
-            trade_events = compute_rebalance_trades(weights)
-
-            result_entry = _build_strategy_result(
-                name, equity, trade_events, injection_schedule, capital
-            )
-            results.append(result_entry)
-
-        # Benchmark over exactly the strategies' dates
-        bench_curve = benchmark_curve(data[benchmark_ticker], data.index, capital)
-    else:
-        benchmark_label = "SPY"
-        bench_curve = []
-
-    # Process ETF strategies
-    if etf_strategies:
-        for name in etf_strategies:
-            _log(f"  Running ETF strategy: {name}")
-            base_name = name.removeprefix("sim_")
-            etf_tickers = ETF_STRATEGY_UNIVERSES[base_name]
-            all_etf = sorted(set(etf_tickers + ["SPY"]))
-
-            with _redirect_stdout_to_stderr():
-                etf_data = data_manager.update("etfs", price_request(market, all_etf), with_open=False)
-            if isinstance(etf_data, tuple):
-                etf_data = etf_data[0]
-            etf_data = convert_quote_currency(etf_data)
-
-            if years:
-                cutoff = etf_data.index[-1] - pd.DateOffset(years=years)
-                etf_data = etf_data[etf_data.index >= cutoff]
-
-            strategy = STRATEGY_REGISTRY[name]()
-
-            tradable = filter_tradable_tickers(etf_data, etf_tickers)
-            equity = backtest(
-                strategy, etf_data[tradable],
-                initial_capital=capital,
-                transaction_cost=tx_cost,
-                fixed_fee=fixed_fee,
-                min_trade_value=MIN_TRADE_VALUE,
-                **MARKET_FEE_TIERED.get(market, {}),
-            )
-
-            weights = strategy.generate_signals(etf_data[tradable])
-            weights = weights.reindex(etf_data[tradable].index).fillna(0.0)
-            trade_events = compute_rebalance_trades(weights)
-
-            result_entry = _build_strategy_result(
-                name, equity, trade_events, injection_schedule, capital
-            )
-            results.append(result_entry)
-
-        # If no stock strategies were run, use SPY as benchmark from ETF data
-        if not stock_strategies and len(etf_strategies) > 0:
-            benchmark_label = "SPY (Benchmark)"
-            with _redirect_stdout_to_stderr():
-                spy_data = data_manager.update("etfs", ["SPY"], with_open=False)
-            if isinstance(spy_data, tuple):
-                spy_data = spy_data[0]
-            if years:
-                cutoff = spy_data.index[-1] - pd.DateOffset(years=years)
-                spy_data = spy_data[spy_data.index >= cutoff]
-            if "SPY" in spy_data.columns:
-                bench_curve = benchmark_curve(spy_data["SPY"], spy_data.index, capital)
+    known = [n for n in strategy_names if n in STRATEGY_REGISTRY]
+    for name in set(strategy_names) - set(known):
+        _log(f"WARNING: Unknown strategy '{name}', skipping")
+    with _redirect_stdout_to_stderr():
+        raw = run_market(market, known, years, capital, tx_cost, fixed_fee,
+                         **MARKET_FEE_TIERED.get(market, {}))
+    results = [
+        _build_strategy_result(run["name"], run["equity"], run["trades"], injection_schedule, capital)
+        for run in (raw["strategy_runs"] if raw else [])
+    ]
+    benchmark_label = raw["benchmark_label"] if raw else "SPY"
+    bench_curve = raw["benchmark_curve"] if raw else []
 
     output = {
         "meta": {

data_manager.py

@@ -265,94 +265,148 @@ def _fill_open(close: pd.DataFrame, open_raw: pd.DataFrame | None) -> pd.DataFra
     return aligned.fillna(close)
 
 
-def update(market: str, tickers: list[str],
-           with_open: bool = False) -> pd.DataFrame | tuple[pd.DataFrame, pd.DataFrame]:
+YAHOO_FIELDS = {"close": "Close", "open": "Open", "high": "High", "low": "Low", "volume": "Volume"}
+
+
+def _backfill(tickers: list[str], start: str, yahoo_fields: list[str],
+              index: pd.DatetimeIndex) -> tuple[dict[str, pd.DataFrame], list[str]]:
+    """Download new tickers over ``index``; return (panels, tickers still missing).
+
+    yfinance reports a ticker it failed to fetch (rate limit, proxy error) the
+    same way as a ticker it has no data for, so tickers missing from the first
+    batch are downloaded once more on their own before being given up on.
     """
-    Refresh the stored panel for ``tickers`` and return it forward-filled.
+    def fetch(batch: list[str]) -> dict[str, pd.DataFrame]:
+        raw = _download(batch, start=start, fields=yahoo_fields)
+        close = raw["Close"].dropna(axis=1, how="all").reindex(index=index).dropna(axis=1, how="all")
+        return {f: raw.get(f, pd.DataFrame()).reindex(index=index, columns=close.columns)
+                for f in yahoo_fields}
+
+    got = fetch(tickers)
+    missing = [t for t in tickers if t not in got["Close"].columns]
+    if missing and len(missing) < len(tickers):
+        retry = fetch(missing)
+        if not retry["Close"].empty:
+            got = {f: pd.concat([got[f], retry[f]], axis=1) for f in yahoo_fields}
+            missing = [t for t in missing if t not in retry["Close"].columns]
+    return got, missing
+
+
+def update(market: str, tickers: list[str], with_open: bool = False,
+           fields: list[str] | None = None):
+    """
+    Refresh the stored panels for ``tickers`` and return them forward-filled.
 
     - First run: downloads 10 years of history.
     - Every run: re-downloads the last ``REFRESH_DAYS`` calendar days for the
       requested tickers already in the panel and overwrites them.
     - New tickers are backfilled over the panel's full date range; tickers
       Yahoo has no data for are remembered and retried weekly.
+    - A field requested for the first time (e.g. ``high``) is downloaded for
+      every ticker in the stored close panel.
 
-    Returns the close panel, or ``(close, open)`` when ``with_open`` is True.
+    Returns the close panel, ``(close, open)`` when ``with_open`` is True, or
+    ``{field: panel}`` when ``fields`` is given. Open/high/low fall back to the
+    close where a bar is missing; volume is returned raw (NaN = no print).
     """
-    fields = ["Close", "Open"] if with_open else ["Close"]
+    names = list(fields) if fields is not None else (["close", "open"] if with_open else ["close"])
+    if "close" not in names:
+        names = ["close"] + names
+    unknown = [f for f in names if f not in YAHOO_FIELDS]
+    if unknown:
+        raise ValueError(f"Unsupported market data field(s): {unknown}")
+    yahoo = [YAHOO_FIELDS[f] for f in names]
+
     with _panel_lock(market):
-        close_raw = load_raw(market, "close")
-        open_raw = load_raw(market, "open") if with_open else None
+        raw = {f: load_raw(market, f) for f in names}
         unavailable = _load_unavailable(market)
         today = datetime.now()
 
-        if close_raw is None:
+        if raw["close"] is None:
             start = (today - timedelta(days=365 * 10)).strftime("%Y-%m-%d")
-            downloaded = _download(tickers, start=start, fields=fields)
-            close_raw = _merge_refresh(None, downloaded["Close"], None)
-            if with_open:
-                open_raw = _merge_refresh(None, downloaded.get("Open", pd.DataFrame()), None)
+            downloaded = _download(tickers, start=start, fields=yahoo)
+            for f, y in zip(names, yahoo):
+                raw[f] = _merge_refresh(None, downloaded.get(y, pd.DataFrame()), None)
         else:
-            if with_open and open_raw is None:
-                print("--- Downloading Open prices for the full panel (first time) ---")
-                start = close_raw.index[0].strftime("%Y-%m-%d")
-                opens = _download(list(close_raw.columns), start=start, fields=["Open"])
-                open_raw = _merge_refresh(None, opens["Open"], None)
+            close_raw = raw["close"]
+            first = [(f, y) for f, y in zip(names, yahoo) if raw[f] is None]
+            if first:
+                print(f"--- Downloading {[y for _, y in first]} for the full panel (first time) ---")
+                got = _download(list(close_raw.columns), start=close_raw.index[0].strftime("%Y-%m-%d"),
+                                fields=[y for _, y in first])
+                for f, y in first:
+                    raw[f] = _merge_refresh(None, got[y], None)
 
             known = [t for t in tickers if t in close_raw.columns]
             if known:
                 window_start = close_raw.index[-1] - pd.Timedelta(days=REFRESH_DAYS)
-                fresh = _download(known, start=window_start.strftime("%Y-%m-%d"), fields=fields)
+                fresh = _download(known, start=window_start.strftime("%Y-%m-%d"), fields=yahoo)
                 if not fresh["Close"].empty:
-                    close_raw = _merge_refresh(close_raw, fresh["Close"], window_start)
-                    if with_open:
-                        open_raw = _merge_refresh(open_raw, fresh.get("Open", pd.DataFrame()), window_start)
+                    for f, y in zip(names, yahoo):
+                        raw[f] = _merge_refresh(raw[f], fresh.get(y, pd.DataFrame()), window_start)
                 else:
                     print("--- Refresh download returned no data; keeping stored panel ---")
 
             cutoff = (today - timedelta(days=UNAVAILABLE_RETRY_DAYS)).strftime("%Y-%m-%d")
             new_tickers = [t for t in tickers
-                           if t not in close_raw.columns and unavailable.get(t, "") < cutoff]
+                           if t not in raw["close"].columns and unavailable.get(t, "") < cutoff]
             if new_tickers:
-                start = close_raw.index[0].strftime("%Y-%m-%d")
-                backfill = _download(new_tickers, start=start, fields=fields)
-                downloaded_any = not backfill["Close"].dropna(how="all").empty
-                got = backfill["Close"].dropna(axis=1, how="all")
-                got = got.reindex(index=close_raw.index)
-                got = got.dropna(axis=1, how="all")
-                for t in new_tickers:
-                    if t in got.columns:
-                        unavailable.pop(t, None)
-                    elif downloaded_any:
-                        # Only a ticker missing from an otherwise successful
-                        # download is unavailable; a failed request is retried.
+                got, missing = _backfill(new_tickers, raw["close"].index[0].strftime("%Y-%m-%d"),
+                                         yahoo, raw["close"].index)
+                added = list(got["Close"].columns)
+                for t in added:
+                    unavailable.pop(t, None)
+                # A batch that returned nothing at all is a failed request, not
+                # evidence the tickers have no data: retry those next run.
+                if added:
+                    for t in missing:
                         unavailable[t] = today.strftime("%Y-%m-%d")
-                if not got.empty:
-                    close_raw = pd.concat([close_raw, got], axis=1)
-                    if with_open and open_raw is not None:
-                        opens = backfill.get("Open", pd.DataFrame())
-                        opens = opens.reindex(index=open_raw.index, columns=got.columns)
-                        open_raw = pd.concat([open_raw, opens], axis=1)
-                    print(f"--- Backfilled {got.shape[1]} new tickers ---")
-                missing = len(new_tickers) - got.shape[1]
+                if added:
+                    for f, y in zip(names, yahoo):
+                        panel = got[y].reindex(index=raw[f].index)
+                        raw[f] = pd.concat([raw[f], panel], axis=1)
+                    print(f"--- Backfilled {len(added)} new tickers ---")
                 if missing:
-                    print(f"--- {missing} tickers have no Yahoo data; retry after "
+                    print(f"--- {len(missing)} tickers have no Yahoo data; retry after "
                           f"{UNAVAILABLE_RETRY_DAYS} days ---")
 
-        close_raw = close_raw.loc[:, ~close_raw.columns.duplicated()]
-        _write_atomic(close_raw, _data_path(market, "close"))
-        print(f"--- Saved {close_raw.shape[0]} days x {close_raw.shape[1]} tickers to "
+        for f in names:
+            raw[f] = raw[f].loc[:, ~raw[f].columns.duplicated()]
+            _write_atomic(raw[f], _data_path(market, f))
+        print(f"--- Saved {raw['close'].shape[0]} days x {raw['close'].shape[1]} tickers to "
               f"{_data_path(market, 'close')} ---")
-        if with_open and open_raw is not None:
-            open_raw = open_raw.loc[:, ~open_raw.columns.duplicated()]
-            _write_atomic(open_raw, _data_path(market, "open"))
         _save_unavailable(market, unavailable)
 
-    close = close_raw.ffill()
+    close = raw["close"].ffill()
+    out = {"close": close}
+    for f in names:
+        if f in ("open", "high", "low"):
+            out[f] = _fill_open(close, raw[f])
+        elif f == "volume":
+            out[f] = raw[f].reindex(index=close.index, columns=close.columns)
+    if fields is not None:
+        return out
     if with_open:
-        return close, _fill_open(close, open_raw)
+        return close, out["open"]
     return close
 
 
+def load_fields(market: str, fields: list[str]) -> dict[str, pd.DataFrame]:
+    """Stored panels without refreshing, aligned and filled the way ``update`` returns them."""
+    close = load(market, "close")
+    out = {"close": close}
+    for f in fields:
+        if f == "close":
+            continue
+        stored = load_raw(market, f)
+        if f == "volume":
+            out[f] = (stored.reindex(index=close.index, columns=close.columns)
+                      if stored is not None else pd.DataFrame(np.nan, index=close.index, columns=close.columns))
+        else:
+            out[f] = _fill_open(close, stored)
+    return out
+
+
 def rebuild(market: str, with_open: bool = True, years: int = 10) -> None:
     """Re-download the full history of every ticker in a stored panel.
 

main.py

@@ -9,6 +9,7 @@ import factor_attribution
 import metrics
 import universe_history as uh
 from strategies.adaptive_momentum import AdaptiveMomentumStrategy
+from strategies.base import BarStrategy
 from strategies.buy_and_hold import BuyAndHoldStrategy
 from strategies.dual_momentum import DualMomentumStrategy
 from strategies.inverse_vol import InverseVolatilityStrategy
@@ -25,6 +26,99 @@ from universe import UNIVERSES
 # Backtest engine
 # ---------------------------------------------------------------------------
 
+def strategy_positions(strategy, data: pd.DataFrame, open_data: pd.DataFrame | None = None,
+                       bars: dict[str, pd.DataFrame] | None = None) -> pd.DataFrame:
+    """Weights actually held over (close t-1, close t] for each row t.
+
+    Close mode (``open_data`` is None): row t's weights use closes through t-1
+    and trade at the close of t-1 — the same bar the signal saw.
+    Live mode (``open_data`` given), matching trader.py:
+      * close-price strategies read the session's open as their latest price
+        and trade at that session's close;
+      * a BarStrategy reads completed bars through t-1 and trades at the close
+        of t.
+    """
+    if isinstance(strategy, BarStrategy):
+        weights = strategy.generate_signals(bars if bars is not None else {"close": data})
+        weights = weights.reindex(data.index).fillna(0.0)
+        return weights.shift(1).fillna(0.0) if open_data is not None else weights
+    if open_data is not None:
+        # The strategy's shift(1) on opens gives weights[t] = f(open_{t-1});
+        # undo it so the row-t decision uses open_t, then hold it from close t.
+        weights = strategy.generate_signals(open_data).shift(-1).fillna(0.0)
+        return weights.reindex(data.index).fillna(0.0).shift(1).fillna(0.0)
+    return strategy.generate_signals(data).reindex(data.index).fillna(0.0)
+
+
+def simulate_live_book(held: pd.DataFrame, data: pd.DataFrame, initial_capital: float,
+                       transaction_cost: float = 0.001, fixed_fee: float = 0.0,
+                       fee_base: float = 0.0, fee_per_share: float = 0.0,
+                       min_trade_value: float = 0.0,
+                       resize_band: float = 0.0) -> tuple[pd.Series, pd.DataFrame]:
+    """Day-by-day book kept the way trader.py keeps it.
+
+    ``held`` row t holds the target weights for (close t-1, close t]. Holdings
+    drift with prices; at each close the book trades toward the next row's
+    targets: entries and exits always, a held name only once it is at least
+    ``resize_band`` (relative to its target) away. Orders smaller than
+    ``max(min_trade_value, 0.1% of equity)`` are skipped unless they close a
+    position. Costs are charged on the orders placed.
+
+    Returns (equity, traded weight per ticker on the row whose holdings changed).
+    """
+    targets = held.reindex(index=data.index, columns=data.columns).fillna(0.0).to_numpy()
+    prices = data.ffill().to_numpy(dtype=float)
+    rets = np.nan_to_num(data.ffill().pct_change(fill_method=None).to_numpy(dtype=float), nan=0.0)
+    per_share = np.array([0.0 if str(c).endswith(".HK") else fee_per_share for c in data.columns])
+    n, m = targets.shape
+    w = np.zeros(m)
+    value = float(initial_capital)
+    equity = np.empty(n)
+    traded = np.zeros((n, m))
+    for t in range(n):
+        if value <= 0:
+            equity[t:] = 0.0
+            break
+        gross = w * (1.0 + rets[t])
+        growth = gross.sum() + (1.0 - w.sum())          # idle cash earns nothing
+        value *= growth
+        w = gross / growth if growth > 0 else np.zeros(m)
+        if t + 1 < n:
+            tgt = targets[t + 1]
+            new = w.copy()
+            entry_exit = (tgt <= 0) | (w <= 0)
+            resize = ~entry_exit & (np.abs(tgt - w) >= resize_band * tgt)
+            change = (entry_exit | resize) & (np.abs(tgt - w) > 0)
+            new[change] = tgt[change]
+            if new.sum() > 1.0:                          # kept winners leave less room
+                buys = change & (new > w)
+                room = max(1.0 - new[~buys].sum(), 0.0)
+                new[buys] *= room / new[buys].sum() if new[buys].sum() > 0 else 0.0
+            delta = new - w
+            floor = (max(min_trade_value, 0.001 * value) / value) if min_trade_value > 0 else 1e-12
+            skip = (np.abs(delta) < floor) & (new > 0)
+            delta[skip] = 0.0
+            new[skip] = w[skip]
+            orders = np.abs(delta) > 1e-12
+            if orders.any():
+                dollars = np.abs(delta) * value
+                cost = dollars.sum() * transaction_cost
+                if fixed_fee > 0 or fee_base > 0 or fee_per_share > 0:
+                    with np.errstate(divide="ignore", invalid="ignore"):
+                        shares = np.where(prices[t] > 0, dollars / prices[t], 0.0)
+                    fee = np.maximum(fixed_fee, fee_base + per_share * shares)
+                    cost += fee[orders].sum()
+                if cost >= value:                        # fees ate the book
+                    equity[t:] = 0.0
+                    break
+                w = new * value / (value - cost)
+                value -= cost
+                traded[t + 1] = delta
+        equity[t] = value
+    return (pd.Series(equity, index=data.index),
+            pd.DataFrame(traded, index=data.index, columns=data.columns))
+
+
 def backtest(
     strategy,
     data: pd.DataFrame,
@@ -35,6 +129,8 @@ def backtest(
     fee_per_share: float = 0.0,
     open_data: pd.DataFrame | None = None,
     min_trade_value: float = 0.0,
+    bars: dict[str, pd.DataFrame] | None = None,
+    resize_band: float = 0.0,
 ) -> pd.Series:
     """
     Vectorized backtest.
@@ -60,39 +156,33 @@ def backtest(
         With fee_base=1.88 + fee_per_share=0.009 + fixed_fee=2.0 you get an
         IBKR-style schedule: max(2, 1.88 + 0.009 * shares).
     open_data : pd.DataFrame, optional
-        Open prices. When provided, enables open-to-close execution mode.
+        Open prices. When provided, the backtest keeps the book the way the
+        live trader does: its signal timing (``strategy_positions``), drifting
+        holdings, whole orders and ``resize_band`` (``simulate_live_book``).
     min_trade_value : float
         Per-ticker rebalances smaller than ``max(min_trade_value, 0.1% of
         equity)`` are not charged the per-trade fee, mirroring the live
         trader, which never places such orders. 0 (default) charges every
         weight change.
+    bars : dict, optional
+        ``{field: DataFrame}`` panels for a BarStrategy (close/open/high/low/volume).
 
     Returns
     -------
     pd.Series
         Daily equity curve.
     """
+    positions = strategy_positions(strategy, data, open_data=open_data, bars=bars)
     if open_data is not None:
-        # Open-to-close mode:
-        # Strategy's shift(1) on open prices gives: weights[t] = f(open_{t-1})
-        # But open_t is known at morning of day t, so undo shift to get f(open_t)
-        # Then execute at close_t, earning close_t → close_{t+1}
-        weights = strategy.generate_signals(open_data)
-        weights = weights.shift(-1).fillna(0.0)
-        weights = weights.reindex(data.index).fillna(0.0)
-        positions = weights
+        equity, _ = simulate_live_book(
+            positions, data, initial_capital, transaction_cost, fixed_fee, fee_base,
+            fee_per_share, min_trade_value, resize_band,
+        )
+        return equity
 
-        # Returns earned: close[t] → close[t+1], weighted by positions decided at open[t]
-        close_returns = data.pct_change().fillna(0.0)
-        portfolio_returns = (close_returns * positions.shift(1).fillna(0.0)).sum(axis=1)
-    else:
-        # Classic close-to-close mode
-        weights = strategy.generate_signals(data)
-        weights = weights.reindex(data.index).fillna(0.0)
-        positions = weights
-
-        daily_returns = data.pct_change().fillna(0.0)
-        portfolio_returns = (daily_returns * positions).sum(axis=1)
+    # A missing close carries the last price (zero return), as before.
+    daily_returns = data.ffill().pct_change(fill_method=None).fillna(0.0)
+    portfolio_returns = (daily_returns * positions).sum(axis=1)
 
     # Turnover cost: sum of absolute weight changes each day
     turnover = positions.diff().abs().sum(axis=1).fillna(0.0)

precompute_backtest.py

@@ -25,32 +25,15 @@ import os
 import sys
 from datetime import datetime
 
-import numpy as np
 import pandas as pd
 
-import data_manager
-import metrics
-import universe_history as uh
+from backtest_runner import run_market
 from bridge import (
-    MIN_TRADE_VALUE,
     STRATEGY_META,
     _build_strategy_result,
     _sanitize_for_json,
-    benchmark_curve,
-    compute_rebalance_trades,
 )
-from main import backtest
-from trader import (
-    strategy_extra_tickers,
-    strategy_supports_market,
-    ETF_STRATEGY_UNIVERSES,
-    MONITOR_CAPITAL,
-    STRATEGY_REGISTRY,
-    convert_quote_currency,
-    filter_tradable_tickers,
-    price_request,
-)
-from universe import UNIVERSES
+from trader import MONITOR_CAPITAL, STRATEGY_REGISTRY, strategy_supports_market
 
 
 # Initial capital per market, in local currency — the same amounts the live
@@ -102,171 +85,6 @@ def _equity_series_from_curve(curve: list[dict]) -> pd.Series:
     return pd.Series(values, index=dates)
 
 
-def run_market_raw(
-    market: str,
-    strategy_names: list[str],
-    years: int,
-    capital: float,
-    tx_cost: float,
-    fixed_fee: float,
-    fee_base: float = 0.0,
-    fee_per_share: float = 0.0,
-) -> dict | None:
-    """Run all strategies once; return raw equity series + metadata per strategy."""
-    stock_strategies = []
-    etf_strategies = []
-    for name in strategy_names:
-        base_name = name.removeprefix("sim_")
-        if base_name in ETF_STRATEGY_UNIVERSES:
-            etf_strategies.append(name)
-        else:
-            stock_strategies.append(name)
-
-    raw_strategies: list[dict] = []
-    benchmark_label = "SPY"
-    bench_curve = []
-    start_date = None
-    end_date = None
-
-    # --- Stock strategies ---
-    if stock_strategies:
-        universe = UNIVERSES[market]
-        tickers = universe["fetch"]()
-        benchmark_ticker = universe["benchmark"]
-        benchmark_label = universe["benchmark_label"]
-
-        pit_intervals = None
-        if market == "us":
-            pit_intervals = uh.load_sp500_history()
-            historical_tickers = uh.all_tickers_ever(pit_intervals)
-            all_tickers = sorted(set(tickers + historical_tickers + [benchmark_ticker])
-                                 | {t for n in stock_strategies for t in strategy_extra_tickers(n)})
-            log(f"PIT universe: {len(all_tickers)} tickers")
-        else:
-            all_tickers = sorted(set(tickers + [benchmark_ticker])
-                                 | {t for n in stock_strategies for t in strategy_extra_tickers(n)})
-            log(f"WARNING: {market.upper()} lacks PIT history; using current constituents")
-
-        log(f"Fetching price data for {len(all_tickers)} tickers...")
-        data = data_manager.update(market, all_tickers, with_open=False)
-        if isinstance(data, tuple):
-            data = data[0]
-
-        if years:
-            cutoff = data.index[-1] - pd.DateOffset(years=years)
-            data = data[data.index >= cutoff]
-
-        if pit_intervals is not None:
-            log("Applying PIT membership mask...")
-            data = uh.mask_prices(data, pit_intervals)
-            tickers = [
-                t for t in data.columns
-                if t != benchmark_ticker and data[t].notna().any()
-            ]
-        else:
-            tickers = filter_tradable_tickers(data, tickers)
-
-        log(f"Universe: {len(tickers)} stocks, {data.index[0].date()} to {data.index[-1].date()}")
-
-        for name in stock_strategies:
-            log(f"  Running: {name}")
-            try:
-                strategy = STRATEGY_REGISTRY[name]()
-                # Mixed stock+ETF strategies also read their fixed ETFs.
-                cols = tickers + [t for t in strategy_extra_tickers(name)
-                                  if t in data.columns and t not in tickers]
-
-                equity = backtest(
-                    strategy, data[cols],
-                    initial_capital=capital,
-                    transaction_cost=tx_cost,
-                    fixed_fee=fixed_fee,
-                    fee_base=fee_base,
-                    fee_per_share=fee_per_share,
-                    min_trade_value=MIN_TRADE_VALUE,
-                )
-
-                weights = strategy.generate_signals(data[cols])
-                weights = weights.reindex(data[cols].index).fillna(0.0)
-                trade_events = compute_rebalance_trades(weights)
-
-                raw_strategies.append({"name": name, "equity": equity, "trades": trade_events})
-                log(f"    OK: {name} (final: ${equity.iloc[-1]:,.0f})")
-            except Exception as e:
-                log(f"    FAILED: {name}: {e}")
-
-        # Benchmark on exactly the strategies' dates
-        bench_curve = benchmark_curve(data[benchmark_ticker], data.index, capital)
-        start_date = data.index[0].strftime("%Y-%m-%d")
-        end_date = data.index[-1].strftime("%Y-%m-%d")
-
-    # --- ETF strategies ---
-    if etf_strategies:
-        for name in etf_strategies:
-            log(f"  Running ETF strategy: {name}")
-            try:
-                base_name = name.removeprefix("sim_")
-                etf_tickers = ETF_STRATEGY_UNIVERSES[base_name]
-                all_etf = sorted(set(etf_tickers + ["SPY"]))
-
-                etf_data = data_manager.update("etfs", price_request(market, all_etf), with_open=False)
-                if isinstance(etf_data, tuple):
-                    etf_data = etf_data[0]
-                etf_data = convert_quote_currency(etf_data)
-                if years:
-                    cutoff = etf_data.index[-1] - pd.DateOffset(years=years)
-                    etf_data = etf_data[etf_data.index >= cutoff]
-
-                strategy = STRATEGY_REGISTRY[name]()
-
-                tradable = filter_tradable_tickers(etf_data, etf_tickers)
-                equity = backtest(
-                    strategy, etf_data[tradable],
-                    initial_capital=capital,
-                    transaction_cost=tx_cost,
-                    fixed_fee=fixed_fee,
-                    fee_base=fee_base,
-                    fee_per_share=fee_per_share,
-                    min_trade_value=MIN_TRADE_VALUE,
-                )
-
-                weights = strategy.generate_signals(etf_data[tradable])
-                weights = weights.reindex(etf_data[tradable].index).fillna(0.0)
-                trade_events = compute_rebalance_trades(weights)
-
-                raw_strategies.append({"name": name, "equity": equity, "trades": trade_events})
-                log(f"    OK: {name} (final: ${equity.iloc[-1]:,.0f})")
-            except Exception as e:
-                log(f"    FAILED: {name}: {e}")
-
-        if not stock_strategies:
-            benchmark_label = "SPY (Benchmark)"
-            spy_data = data_manager.update("etfs", ["SPY"], with_open=False)
-            if isinstance(spy_data, tuple):
-                spy_data = spy_data[0]
-            if years:
-                cutoff = spy_data.index[-1] - pd.DateOffset(years=years)
-                spy_data = spy_data[spy_data.index >= cutoff]
-            if "SPY" in spy_data.columns:
-                bench_curve = benchmark_curve(spy_data["SPY"], spy_data.index, capital)
-                start_date = spy_data.index[0].strftime("%Y-%m-%d")
-                end_date = spy_data.index[-1].strftime("%Y-%m-%d")
-
-    if not raw_strategies:
-        return None
-
-    return {
-        "market": market,
-        "strategy_runs": raw_strategies,
-        "benchmark_label": benchmark_label,
-        "benchmark_curve": bench_curve,
-        "start_date": start_date,
-        "end_date": end_date,
-        "tx_cost": tx_cost,
-        "fixed_fee": fixed_fee,
-    }
-
-
 def build_scenario_for_market(
     raw: dict,
     scenario: dict,
@@ -358,7 +176,7 @@ def main():
     log("=" * 60)
     # US fee schedule (IBKR-style tiered):
     #   fee = max(2.0, 1.88 + 0.009 * shares)
-    us_raw = run_market_raw(
+    us_raw = run_market(
         "us", us_strategies,
         years=args.years, capital=us_capital,
         tx_cost=0.001, fixed_fee=2.0,
@@ -368,7 +186,7 @@ def main():
     log("=" * 60)
     log(f"CN MARKET: {len(cn_strategies)} strategies @ ¥{cn_capital:,.0f}")
     log("=" * 60)
-    cn_raw = run_market_raw(
+    cn_raw = run_market(
         "cn", cn_strategies,
         years=args.years, capital=cn_capital,
         tx_cost=0.001, fixed_fee=5.0,

pyproject.toml

@@ -10,5 +10,6 @@ dependencies = [
     "numpy>=2.4.0",
     "pandas>=2.3.3",
     "scipy>=1.17.1",
+    "xlrd>=2.0.2",
     "yfinance>=1.0",
 ]

strategies/base.py

@@ -1,5 +1,8 @@
 from abc import ABC, abstractmethod
 
+import pandas as pd
+
+
 class Strategy(ABC):
     """
     Abstract base class for all strategies.
@@ -15,3 +18,32 @@ class Strategy(ABC):
                  containing the trading signals (e.g., 'buy', 'hold', 'sell').
         """
         pass
+
+
+class BarStrategy(Strategy):
+    """Strategy on completed daily bars with several fields.
+
+    ``fields`` names the panels it reads among close/open/high/low/volume.
+    ``generate_signals(bars)`` takes ``{field: DataFrame}`` (a bare close
+    DataFrame is accepted when only "close" is needed) and returns weights
+    whose row t uses bars through t-1, like every Strategy. Backtests may add
+    ``bars["eligible"]`` (bool, point-in-time index membership): prices stay
+    unmasked so signals see full histories, and only eligible names are held.
+
+    Timing differs from close-price strategies in the live trader: those read
+    the session's open as their latest price, while a BarStrategy only uses
+    completed bars, so the weights of row t trade at the close of t.
+    """
+
+    fields: tuple[str, ...] = ("close",)
+
+    def generate_signals(self, data):
+        bars = data if isinstance(data, dict) else {"close": data}
+        missing = [f for f in self.fields if f not in bars]
+        if missing:
+            raise ValueError(f"{type(self).__name__} needs bar fields {missing}")
+        return self.generate_weights(bars)
+
+    @abstractmethod
+    def generate_weights(self, bars: dict[str, pd.DataFrame]) -> pd.DataFrame:
+        """Weights (dates x assets), row t using bars through t-1."""

strategies/robust_momentum.py

@@ -0,0 +1,194 @@
+"""
+Robust momentum strategies from the 2026-09 point-in-time factor research
+(``research/lab``). They were selected by their worst 5-year Sharpe over
+2006/2010-2026 on PIT index members with next-open execution, not by their
+best window.
+
+Each score averages cross-sectional percentile ranks of its factors. Holdings
+are the top ``top_n`` names, re-selected every ``rebal_freq`` rows; a held
+name stays while its rank is within ``top_n * (1 + buffer)`` (hysteresis cuts
+turnover without giving up the signal). Positions are equal-weighted on
+rebalance rows and drift with prices in between, so winners are not trimmed
+daily (worth ~3-4 pp a year for overnight_recovery against daily re-weighting).
+
+Signals:
+  overnight_recovery  (US)  overnight-return momentum (12m, skip 1m)
+                            + distance from the 126-day low
+  lowrisk_momentum    (CN)  amplitude-cut momentum + low beta
+                            + negative price-volume correlation + momentum/vol
+  lowbeta_sharpe_mom  (CN)  low beta + momentum/vol (close prices only)
+"""
+
+import numpy as np
+import pandas as pd
+from numpy.lib.stride_tricks import sliding_window_view
+
+from strategies.base import BarStrategy
+
+
+# ---------------------------------------------------------------------------
+# Factors: higher = more attractive; row t uses data through t
+# ---------------------------------------------------------------------------
+
+def _returns(close: pd.DataFrame) -> pd.DataFrame:
+    return close.pct_change(fill_method=None)
+
+
+def overnight_momentum(open_: pd.DataFrame, close: pd.DataFrame,
+                       window: int = 252, skip: int = 21) -> pd.DataFrame:
+    """Sum of log overnight returns (open / previous close), Lou-Polk-Skouras."""
+    overnight = np.log(open_ / close.ffill().shift(1))
+    return overnight.rolling(window, min_periods=180).sum().shift(skip)
+
+
+def recovery(close: pd.DataFrame, window: int = 126) -> pd.DataFrame:
+    c = close.ffill()
+    return c / c.rolling(window, min_periods=window).min() - 1
+
+
+def momentum_12_1(close: pd.DataFrame) -> pd.DataFrame:
+    c = close.ffill()
+    return c.shift(21) / c.shift(252) - 1
+
+
+def sharpe_momentum(close: pd.DataFrame) -> pd.DataFrame:
+    """12-1 month return divided by 12-month daily volatility."""
+    return momentum_12_1(close) / _returns(close).rolling(252, min_periods=180).std()
+
+
+def low_beta(close: pd.DataFrame, window: int = 252) -> pd.DataFrame:
+    """Negative rolling beta to the equal-weight average of the panel."""
+    r = _returns(close)
+    m = r.mean(axis=1).fillna(0.0)
+    minp = window * 180 // 252
+    e_xy = r.mul(m, axis=0).rolling(window, min_periods=minp).mean()
+    e_x = r.rolling(window, min_periods=minp).mean()
+    e_y = m.rolling(window, min_periods=minp).mean()
+    var_y = (m ** 2).rolling(window, min_periods=minp).mean() - e_y ** 2
+    return -(e_xy - e_x.mul(e_y, axis=0)).div(var_y, axis=0)
+
+
+def price_volume_corr(close: pd.DataFrame, volume: pd.DataFrame) -> pd.DataFrame:
+    """Negative 21-day correlation of price and dollar volume."""
+    return -close.ffill().rolling(21, min_periods=15).corr(close * volume)
+
+
+def amplitude_cut_momentum(high: pd.DataFrame, low: pd.DataFrame, close: pd.DataFrame,
+                           window: int = 160, keep: float = 0.7) -> pd.DataFrame:
+    """Sum of daily returns over the ``keep`` share of lowest-amplitude
+    (high/low - 1) days in the window (开源证券, 2020): calm up-moves persist,
+    noisy ones reverse."""
+    amp = (high / low - 1).to_numpy(dtype=float)
+    r = _returns(close).to_numpy(dtype=float)
+    out = np.full(r.shape, np.nan)
+    if len(r) >= window:
+        A = sliding_window_view(np.nan_to_num(amp, nan=np.inf), window, axis=0)
+        R = sliding_window_view(np.nan_to_num(r, nan=0.0), window, axis=0)
+        for t0 in range(0, A.shape[0], 256):          # chunks bound memory
+            a, rr = A[t0:t0 + 256], R[t0:t0 + 256]
+            thr = np.nanquantile(np.where(np.isinf(a), np.nan, a), keep, axis=2, keepdims=True)
+            out[t0 + window - 1:t0 + window - 1 + a.shape[0]] = np.where(a <= thr, rr, 0.0).sum(axis=2)
+    valid = pd.DataFrame(amp, index=close.index).notna().rolling(window).sum() >= 0.8 * window
+    return pd.DataFrame(out, index=close.index, columns=close.columns).where(valid.to_numpy())
+
+
+def _rank(df: pd.DataFrame) -> pd.DataFrame:
+    return df.rank(axis=1, pct=True)
+
+
+def _composite(*factors: pd.DataFrame) -> pd.DataFrame:
+    return sum(_rank(f) for f in factors) / len(factors)
+
+
+def signal_overnight_recovery(bars):
+    return _composite(overnight_momentum(bars["open"], bars["close"]), recovery(bars["close"]))
+
+
+def signal_lowrisk_momentum(bars):
+    c = bars["close"]
+    return _composite(amplitude_cut_momentum(bars["high"], bars["low"], c), low_beta(c),
+                      price_volume_corr(c, bars["volume"]), sharpe_momentum(c))
+
+
+def signal_lowbeta_sharpe_mom(bars):
+    c = bars["close"]
+    return _composite(low_beta(c), sharpe_momentum(c))
+
+
+# name -> (bar fields, score function)
+SIGNALS = {
+    "overnight_recovery": (("close", "open"), signal_overnight_recovery),
+    "lowrisk_momentum": (("close", "high", "low", "volume"), signal_lowrisk_momentum),
+    "lowbeta_sharpe_mom": (("close",), signal_lowbeta_sharpe_mom),
+}
+
+
+# ---------------------------------------------------------------------------
+# Selection
+# ---------------------------------------------------------------------------
+
+def select_with_buffer(score: pd.DataFrame, top_n: int, rebal_freq: int,
+                       buffer: float, warmup: int,
+                       prices: pd.DataFrame | None = None) -> pd.DataFrame:
+    """Equal weights on rebalance rows (every ``rebal_freq`` from ``warmup``);
+    in between they drift with ``prices`` (row k uses prices through k), or
+    stay constant when ``prices`` is None. Not yet shifted."""
+    S = score.to_numpy(dtype=float)
+    n, m = S.shape
+    W = np.full((n, m), np.nan)
+    held = np.zeros(m, dtype=bool)
+    for r in range(warmup, n, rebal_freq):
+        row = S[r]
+        valid = ~np.isnan(row)
+        w = np.zeros(m)
+        if valid.sum() >= top_n:
+            order = np.where(valid)[0][np.argsort(-row[valid], kind="stable")]
+            rank = np.full(m, np.inf)
+            rank[order] = np.arange(len(order))
+            keep = [i for i in np.where(held)[0] if rank[i] < top_n * (1 + buffer)]
+            chosen = sorted(keep, key=lambda i: rank[i])[:top_n]
+            taken = set(chosen)
+            for i in order:
+                if len(chosen) >= top_n:
+                    break
+                if i not in taken:
+                    chosen.append(i)
+            w[chosen] = 1.0 / len(chosen)
+        W[r] = w
+        held = w > 0
+    weights = pd.DataFrame(W, index=score.index, columns=score.columns).ffill().fillna(0.0)
+    weights.iloc[:warmup] = 0.0
+    if prices is not None:
+        px = prices.reindex_like(weights).ffill()
+        anchor = px.where(pd.Series(~np.isnan(W).all(axis=1), index=weights.index), axis=0).ffill()
+        grown = weights * (px / anchor).fillna(1.0)
+        weights = grown.div(grown.sum(axis=1).replace(0.0, np.nan), axis=0).fillna(0.0)
+    return weights
+
+
+class RobustMomentumStrategy(BarStrategy):
+    def __init__(self, signal: str, top_n: int = 10, rebal_freq: int = 10,
+                 buffer: float = 2.0, warmup: int = 252):
+        if signal not in SIGNALS:
+            raise ValueError(f"Unknown signal {signal!r}; available: {list(SIGNALS)}")
+        self.signal = signal
+        self.fields, self._score = SIGNALS[signal]
+        self.top_n = top_n
+        self.rebal_freq = rebal_freq
+        self.buffer = buffer
+        self.warmup = warmup
+
+    def generate_weights(self, bars: dict[str, pd.DataFrame]) -> pd.DataFrame:
+        close = bars["close"]
+        panel = {f: bars[f].reindex(index=close.index, columns=close.columns) for f in self.fields}
+        score = self._score(panel)
+        # names without a price are not eligible; a backtest also passes
+        # "eligible" (point-in-time index membership) so signals can use a
+        # stock's full price history while only members are picked
+        eligible = close.notna()
+        if "eligible" in bars:
+            eligible &= bars["eligible"].reindex_like(close).fillna(False).astype(bool)
+        score = score.where(eligible)
+        weights = select_with_buffer(score, self.top_n, self.rebal_freq, self.buffer, self.warmup,
+                                     prices=close)
+        return weights.shift(1).fillna(0.0)

tests/test_robust_strategies.py

@@ -0,0 +1,187 @@
+"""Contracts added with the robust-momentum strategies: bar strategies are
+causal and trade on the live trader's timing, whole-lot orders track targets,
+panel backfills retry, and point-in-time CSI 300 membership."""
+
+import unittest
+from unittest import mock
+
+import numpy as np
+import pandas as pd
+
+import data_manager
+import trader
+import universe_history as uh
+from main import backtest, simulate_live_book, strategy_positions
+from strategies.robust_momentum import SIGNALS, RobustMomentumStrategy, select_with_buffer
+
+
+def _bars(n_days: int = 420, n_assets: int = 30, seed: int = 0) -> dict[str, pd.DataFrame]:
+    rng = np.random.default_rng(seed)
+    idx = pd.bdate_range("2024-01-01", periods=n_days)
+    cols = [f"S{i:02d}" for i in range(n_assets)]
+    r = rng.normal(0.0004, 0.02, (n_days, n_assets))
+    close = pd.DataFrame(100 * np.exp(np.cumsum(r, axis=0)), index=idx, columns=cols)
+    gap = rng.normal(0, 0.005, (n_days, n_assets))
+    opens = close.shift(1).bfill() * np.exp(gap)
+    high = np.maximum(opens, close) * (1 + rng.uniform(0, 0.02, (n_days, n_assets)))
+    low = np.minimum(opens, close) * (1 - rng.uniform(0, 0.02, (n_days, n_assets)))
+    volume = pd.DataFrame(rng.uniform(1e5, 1e6, (n_days, n_assets)), index=idx, columns=cols)
+    return {"close": close, "open": opens, "high": high, "low": low, "volume": volume}
+
+
+class RobustMomentumTests(unittest.TestCase):
+    def test_weights_on_a_day_ignore_that_day_and_later_bars(self):
+        bars = _bars()
+        cut = bars["close"].index[380]
+        for signal in SIGNALS:
+            strat = RobustMomentumStrategy(signal, top_n=5, rebal_freq=5, buffer=1.0)
+            base = strat.generate_signals(bars)
+            shocked = {f: b.copy() for f, b in bars.items()}
+            for f in shocked:
+                shocked[f].loc[cut:] *= 3.0
+            again = strat.generate_signals(shocked)
+            pd.testing.assert_frame_equal(base.loc[:cut], again.loc[:cut], obj=signal)
+            self.assertGreater(base.loc[cut].sum(), 0.99, signal)
+
+    def test_held_names_stay_while_inside_the_buffer(self):
+        idx = pd.bdate_range("2025-01-01", periods=3)
+        score = pd.DataFrame([[5, 4, 3, 2, 1], [1, 2, 5, 4, 3], [1, 2, 5, 4, 3]],
+                             index=idx, columns=list("ABCDE"), dtype=float)
+        w = select_with_buffer(score, top_n=2, rebal_freq=1, buffer=1.0, warmup=0)
+        self.assertEqual(set(w.columns[w.iloc[0] > 0]), {"A", "B"})
+        # Day 2: A ranks 5th (outside 2 * (1 + 1) = 4) and leaves; B ranks 4th and stays.
+        self.assertEqual(set(w.columns[w.iloc[1] > 0]), {"B", "C"})
+
+    def test_bar_strategy_trades_at_the_close_after_its_signal_day(self):
+        bars = _bars()
+        strat = RobustMomentumStrategy("lowbeta_sharpe_mom", top_n=5, rebal_freq=5)
+        weights = strat.generate_signals(bars)
+        held = strategy_positions(strat, bars["close"], open_data=bars["open"], bars=bars)
+        pd.testing.assert_frame_equal(held, weights.shift(1).fillna(0.0))
+        equity = backtest(strat, bars["close"], initial_capital=1000, open_data=bars["open"], bars=bars)
+        self.assertEqual(len(equity), len(bars["close"]))
+
+    def test_trader_session_signals_use_bars_through_the_previous_day(self):
+        bars = _bars()
+        days = bars["close"].index[-3:]
+        strat = RobustMomentumStrategy("overnight_recovery", top_n=5, rebal_freq=1)
+        full = strat.generate_signals(bars).loc[days]
+        got = trader.bar_session_weights(strat, bars, days)
+        pd.testing.assert_frame_equal(got, full)
+        # Morning run: today's bar does not exist yet.
+        today = days[-1] + pd.offsets.BDay()
+        morning = trader.bar_session_weights(strat, bars, pd.DatetimeIndex([today]))
+        self.assertAlmostEqual(morning.loc[today].sum(), 1.0)
+
+
+class LiveBookSimulationTests(unittest.TestCase):
+    def test_holdings_drift_and_only_resizes_beyond_the_band_trade(self):
+        idx = pd.bdate_range("2026-01-05", periods=5)
+        data = pd.DataFrame({"A": [100, 100, 125, 125, 125], "B": [100] * 5, "C": [100] * 5},
+                            index=idx, dtype=float)
+        third = [0, 1 / 3, 1 / 3, 1 / 3]
+        held = pd.DataFrame({"A": third + [0.0], "B": third + [0.5], "C": third + [0.5]}, index=idx)
+        equity, traded = simulate_live_book(held, data, 1000.0, transaction_cost=0.0,
+                                            resize_band=0.10)
+        orders = (traded.abs() > 0).sum(axis=1).tolist()
+        # Buy all three; A +25% drifts to 38.5% (15% over target): trim A only,
+        # B and C at 30.8% stay inside the band; then exit A, top up B and C.
+        self.assertEqual(orders, [0, 3, 0, 1, 3])
+        self.assertLess(traded.loc[idx[3], "A"], 0)
+        self.assertAlmostEqual(equity.iloc[2], 1000 * (1 + 0.25 / 3), places=6)
+
+    def test_orders_below_the_floor_are_skipped_but_exits_are_not(self):
+        idx = pd.bdate_range("2026-01-05", periods=3)
+        data = pd.DataFrame({"A": [10.0, 10.0, 10.0], "B": [10.0, 10.0, 10.0]}, index=idx)
+        held = pd.DataFrame({"A": [0, 0.004, 0.0], "B": [0, 0.5, 0.5]}, index=idx)
+        _, traded = simulate_live_book(held, data, 10_000.0, min_trade_value=50.0)
+        self.assertEqual(traded["A"].abs().sum(), 0.0)      # a $40 entry is below $50
+        self.assertGreater(traded["B"].abs().sum(), 0.0)
+
+
+class LotAllocationTests(unittest.TestCase):
+    def test_leftover_cash_buys_the_lot_that_brings_a_name_nearest_its_target(self):
+        prices = {"A": 45.0, "B": 13.0}
+        # 70k book, 50/50: A wants 777.8 shares, B 2692.3. Rounding down leaves
+        # 4.9k idle; it buys A's 8th lot (3.5k short of target vs half a lot
+        # 2.25k), after which B's extra lot no longer fits.
+        trades = trader.compute_trades({}, 70_000.0, {"A": 0.5, "B": 0.5}, prices,
+                                       integer_shares=True, lot_size=100)
+        shares = {t["ticker"]: t["shares_delta"] for t in trades}
+        self.assertEqual(shares, {"A": 800, "B": 2600})
+        spent = sum(s * prices[t] for t, s in shares.items())
+        self.assertLessEqual(spent, 70_000.0)
+
+    def test_a_lot_far_above_target_is_not_bought(self):
+        trades = trader.compute_trades({}, 10_000.0, {"A": 0.3, "B": 0.7}, {"A": 70.0, "B": 1.0},
+                                       integer_shares=True, lot_size=100)
+        # A's target is 3000 = 0.43 of a 7000 lot: stay out.
+        self.assertNotIn("A", {t["ticker"] for t in trades})
+
+    def test_small_drift_is_not_traded_but_entries_and_exits_are(self):
+        holdings = {"A": 10.0, "B": 10.0}
+        prices = {"A": 105.0, "B": 95.0, "C": 50.0}
+        trades = trader.compute_trades(holdings, 0.0, {"A": 0.5, "B": 0.5}, prices,
+                                       min_trade_value=1.0, resize_band=0.10)
+        self.assertEqual(trades, [])
+        trades = trader.compute_trades(holdings, 0.0, {"A": 0.5, "C": 0.5}, prices,
+                                       min_trade_value=1.0, resize_band=0.10)
+        self.assertEqual({t["ticker"]: t["direction"] for t in trades}, {"B": "SELL", "C": "BUY"})
+
+
+class PanelBackfillTests(unittest.TestCase):
+    def test_tickers_missing_from_a_partial_batch_are_retried_before_giving_up(self):
+        idx = pd.bdate_range("2026-01-05", periods=4)
+        calls = []
+
+        def fake_download(tickers, start, end=None, fields=None):
+            calls.append(list(tickers))
+            ok = [t for t in tickers if t == "AAA" or (t == "BBB" and len(calls) > 1)]
+            frame = pd.DataFrame({t: [1.0, 2.0, 3.0, 4.0] for t in ok}, index=idx)
+            return {f: frame for f in fields}
+
+        with mock.patch.object(data_manager, "_download", fake_download):
+            got, missing = data_manager._backfill(["AAA", "BBB", "CCC"], "2026-01-05",
+                                                  ["Close", "Volume"], idx)
+        self.assertEqual(sorted(got["Close"].columns), ["AAA", "BBB"])
+        self.assertEqual(missing, ["CCC"])
+        self.assertEqual(calls[1], ["BBB", "CCC"])
+
+
+class BacktestUniverseTests(unittest.TestCase):
+    def test_stock_universe_excludes_benchmarks_and_etfs_sharing_the_panel(self):
+        import backtest_runner
+        idx = pd.bdate_range("2026-01-05", periods=3)
+        panel = pd.DataFrame(1.0, index=idx, columns=["AAA", "BBB", "SPY", "TQQQ"])
+        fake_universe = {"us": {"fetch": lambda: ["AAA"], "benchmark": "SPY",
+                                "benchmark_label": "SPY"}}
+        with mock.patch.object(backtest_runner, "UNIVERSES", fake_universe), \
+                mock.patch.object(backtest_runner.uh, "pit_tickers_ever", lambda m, since=None: ["AAA", "BBB"]), \
+                mock.patch.object(backtest_runner.uh, "mask_prices_pit", lambda p, m: p), \
+                mock.patch.object(backtest_runner.data_manager, "update",
+                                  lambda m, t, fields=None: {f: panel for f in fields}):
+            data = backtest_runner.load_stock_panels("us", ["momentum"], None)
+        self.assertEqual(data["stocks"], ["AAA", "BBB"])
+
+
+class MembershipTests(unittest.TestCase):
+    def test_csi300_snapshot_applies_until_the_next_one(self):
+        snaps = {"2026-01-01": ["600000.SS", "000001.SZ"], "2026-02-01": ["600000.SS"]}
+        dates = pd.to_datetime(["2025-12-31", "2026-01-15", "2026-02-02"])
+        mask = uh.csi300_membership_mask(dates, ["600000.SS", "000001.SZ", "510300.SS"], snaps)
+        self.assertEqual(mask["000001.SZ"].tolist(), [False, True, False])
+        self.assertEqual(mask["600000.SS"].tolist(), [False, True, True])
+        # Never-member columns (benchmarks) are not masked.
+        self.assertTrue(mask["510300.SS"].all())
+
+    def test_changes_table_with_two_level_headers_is_found(self):
+        cols = pd.MultiIndex.from_tuples([("Effective Date", "Effective Date"), ("Added", "Ticker"),
+                                          ("Removed", "Ticker"), ("Reason", "Reason")])
+        table = pd.DataFrame([["September 21, 2026", "BE", "TAP", "cap"]], columns=cols)
+        other = pd.DataFrame({"x": [1]})
+        changes = uh._find_changes_table([other, table])
+        self.assertEqual(list(changes.columns), ["Date", "Added_Ticker", "Removed_Ticker", "Reason_Reason"])
+
+
+if __name__ == "__main__":
+    unittest.main()

trader.py

@@ -40,6 +40,7 @@ import pandas as pd
 import yfinance as yf
 
 import data_manager
+from strategies.base import BarStrategy
 from strategies.buy_and_hold import BuyAndHoldStrategy
 from strategies.dual_momentum import DualMomentumStrategy
 from strategies.factor_combo import FactorComboStrategy
@@ -55,6 +56,7 @@ from strategies.permanent import (
     TrendRiderV4,
 )
 from strategies.recovery_momentum import RecoveryMomentumStrategy
+from strategies.robust_momentum import RobustMomentumStrategy
 from strategies.trend_following import TrendFollowingStrategy
 from strategies.trend_rider_v5 import TrendRiderV5
 from strategies.trend_rider_voltgt import (
@@ -95,6 +97,12 @@ MARKET_FEE_TIERED = {
 # board lots of 100.
 MARKET_LOT_SIZE = {"us": 1, "cn": 100}
 
+# A position already held is resized only once it is this far from its target
+# (relative to the target value); entries and exits always trade. Without the
+# band the books paid a ticket fee almost daily to trade weight drift that
+# the backtests never trade.
+RESIZE_BAND = 0.10
+
 # Default live capital per market, in the market's own currency.
 MONITOR_CAPITAL = {"us": 10_000.0, "cn": 70_000.0}
 
@@ -226,6 +234,16 @@ STRATEGY_REGISTRY = {
     "recovery_quality_blend_top10": lambda **kw: RecoveryQualityBlendStrategy(top_n=10),
     "improved_mom_quality_top20": lambda **kw: ImprovedMomentumQualityStrategy(top_n=20),
     "improved_mom_quality_top10": lambda **kw: ImprovedMomentumQualityStrategy(top_n=10),
+    # --- Robust momentum (2026-09 PIT research; bar strategies) ---
+    "overnight_recovery_top10": lambda **kw: RobustMomentumStrategy(
+        "overnight_recovery", top_n=10, rebal_freq=10, buffer=2.0,
+    ),
+    "lowrisk_momentum_top15": lambda **kw: RobustMomentumStrategy(
+        "lowrisk_momentum", top_n=15, rebal_freq=10, buffer=2.0,
+    ),
+    "lowbeta_sharpe_mom_top20": lambda **kw: RobustMomentumStrategy(
+        "lowbeta_sharpe_mom", top_n=20, rebal_freq=10, buffer=1.0,
+    ),
     # --- TrendRiderV6: stock-picking + V5 regime engine ---
     "trend_rider_v6": lambda **kw: TrendRiderV6(),
     "trend_rider_v6_top10": lambda **kw: TrendRiderV6(top_n=10),
@@ -273,6 +291,12 @@ def strategy_extra_tickers(strategy_name: str) -> list[str]:
     return list(MIXED_STRATEGY_EXTRA_TICKERS.get(strategy_name.removeprefix("sim_"), []))
 
 
+def strategy_fields(strategy_name: str) -> tuple[str, ...]:
+    """Bar fields a strategy reads; close-price strategies read ("close",)."""
+    strategy = STRATEGY_REGISTRY[strategy_name.removeprefix("sim_")]()
+    return tuple(strategy.fields) if isinstance(strategy, BarStrategy) else ("close",)
+
+
 def strategy_supports_market(strategy_name: str, market: str) -> bool:
     """ETF-based strategies trade USD-listed ETFs, so they only run in the US book."""
     base_name = strategy_name.removeprefix("sim_")
@@ -351,14 +375,15 @@ def load_strategy_prices(market: str, strategy_name: str, tickers: list[str],
     stale forward-filled price from a real one.
     """
     data_market = strategy_data_market(market, strategy_name)
+    fields = ["close", "open"] if with_open else ["close"]
+    fields += [f for f in strategy_fields(strategy_name) if f not in fields]
     if refresh:
         request = price_request(market, list(tickers) + [benchmark])
-        result = data_manager.update(data_market, request, with_open=with_open)
-        close, open_data = result if with_open else (result, None)
+        panels = data_manager.update(data_market, request, fields=fields)
     else:
-        close = data_manager.load(data_market, "close")
-        open_data = (data_manager._fill_open(close, data_manager.load_raw(data_market, "open"))
-                     if with_open else None)
+        panels = data_manager.load_fields(data_market, fields)
+    close = panels["close"]
+    open_data = panels["open"] if with_open else None
     close_raw = data_manager.load_raw(data_market, "close").reindex(
         index=close.index, columns=close.columns,
     )
@@ -438,85 +463,102 @@ def portfolio_value(holdings: dict, prices: dict, cash: float) -> float:
     return cash + stock_value
 
 
-def generate_target_weights(strategy, open_data: pd.DataFrame, target_date) -> dict:
-    """
-    Run strategy on full open price history, extract target_date's weights.
+def strategy_bars(market: str, strategy_name: str, tickers: list[str],
+                  close: pd.DataFrame, open_data: pd.DataFrame | None) -> dict | None:
+    """Stored bar panels a BarStrategy reads (refreshed by load_strategy_prices),
+    aligned to ``close``; None for close-price strategies."""
+    fields = strategy_fields(strategy_name)
+    if not isinstance(STRATEGY_REGISTRY[strategy_name.removeprefix("sim_")](), BarStrategy):
+        return None
+    stored = data_manager.load_fields(strategy_data_market(market, strategy_name),
+                                      [f for f in fields if f not in ("close", "open")])
+    bars = {"close": close[tickers]}
+    if "open" in fields:
+        bars["open"] = open_data[tickers]
+    for f in fields:
+        if f not in bars:
+            bars[f] = stored[f].reindex(index=close.index, columns=tickers)
+    return bars
 
-    Strategies have an internal shift(1) designed for close prices: weights[t]
-    uses data up to t-1.  With open prices (observable same-day), we want the
-    signal that uses data up to today.  To get this, we append a dummy future
-    row so the strategy computes one extra day of signals, then we read that
-    last row — which incorporates today's open prices.
-    """
-    # Append a dummy row one day after the last date so the strategy's
-    # shift(1) produces a valid signal row that uses target_date's open
-    dummy_date = open_data.index[-1] + pd.Timedelta(days=1)
-    dummy_row = pd.DataFrame(
-        [open_data.iloc[-1].values],
-        index=[dummy_date],
-        columns=open_data.columns,
-    )
-    extended = pd.concat([open_data, dummy_row])
 
-    weights = strategy.generate_signals(extended)
+def bar_session_weights(strategy: BarStrategy, bars: dict, days: pd.DatetimeIndex) -> pd.DataFrame:
+    """Targets traded at the close of each day in ``days``: row d uses bars through d-1."""
+    hist = {f: b.loc[:days[-1]] for f, b in bars.items()}
+    close = hist["close"]
+    if days[-1] not in close.index:
+        # morning run before today's bar exists: add an empty row for today
+        hist = {f: pd.concat([b, pd.DataFrame(np.nan, index=[days[-1]], columns=b.columns)])
+                for f, b in hist.items()}
+    return strategy.generate_signals(hist).reindex(days).fillna(0.0)
 
-    # The dummy_date row in weights now uses data up to target_date (the real last day)
-    # Read the dummy row — that's today's actual signal
-    if dummy_date in weights.index:
-        row = weights.loc[dummy_date]
-    elif target_date in weights.index:
-        # Fallback: use target_date row (which uses data up to target_date - 1)
-        row = weights.loc[target_date]
-    else:
-        idx = weights.index.get_indexer([target_date], method="ffill")
-        if idx[0] >= 0:
-            row = weights.iloc[idx[0]]
+
+def _allocate_lots(desired: dict, holdings: dict, prices: dict, cash: float,
+                   lot_size: int, reserve: float) -> dict:
+    """Whole-lot share targets: round each move toward zero, then spend the
+    remaining cash one lot at a time on the names furthest below target,
+    as long as the lot brings them nearer (nearest-lot rounding)."""
+    final, spend = {}, 0.0
+    for t, want in desired.items():
+        cur = holdings.get(t, 0.0)
+        if want <= 0:
+            final[t] = 0.0                          # full exit, odd lot included
         else:
-            return {}
-
-    # Return only non-zero weights
-    return {t: float(w) for t, w in row.items() if w > 1e-6}
+            final[t] = cur + int((want - cur) / lot_size) * lot_size
+        spend += (final[t] - cur) * prices[t]
+    budget = cash - spend - reserve
+    short = sorted(((desired[t] - final[t]) * prices[t], t) for t in desired if desired[t] > 0)
+    for gap, t in reversed(short):
+        lot_value = lot_size * prices[t]
+        if gap > 0.5 * lot_value and lot_value <= budget:
+            final[t] += lot_size
+            budget -= lot_value
+    return final
 
 
 def compute_trades(holdings: dict, cash: float, target_weights: dict,
                    prices: dict, min_trade_value: float = 50.0,
                    integer_shares: bool = False, lot_size: int = 1,
-                   frozen: set | None = None) -> list[dict]:
+                   frozen: set | None = None, resize_band: float = 0.0) -> list[dict]:
     """
     Compute trades needed to move from current holdings to target weights.
 
-    If integer_shares=True, share deltas are rounded toward zero to whole
-    multiples of ``lot_size`` (e.g. 100-share A-share board lots); a position
-    whose target is zero is always sold in full, odd lot included.
+    If integer_shares=True, positions move in whole multiples of ``lot_size``
+    (e.g. 100-share A-share board lots): each move is rounded toward zero and
+    leftover cash then buys one more lot for the names furthest below target
+    when that lot brings them nearer; a position whose target is zero is
+    always sold in full, odd lot included.
+    A held position with a nonzero target is not resized while it is within
+    ``resize_band`` (fraction of its target value) of the target.
     Tickers in ``frozen`` (no fresh price today) are neither bought nor sold.
 
     Returns list of {ticker, shares_delta, direction, est_value}.
     """
     total = portfolio_value(holdings, prices, cash)
-    raw = []
     frozen = frozen or set()
-
-    all_tickers = set(list(target_weights.keys()) + list(holdings.keys()))
-    for ticker in sorted(all_tickers):
+    desired = {}
+    for ticker in sorted(set(target_weights) | set(holdings)):
         if ticker in frozen:
             continue
-        target_weight = target_weights.get(ticker, 0.0)
         price = prices.get(ticker, 0.0)
         if price <= 0:
             continue
-
+        target_weight = target_weights.get(ticker, 0.0)
         target_value = total * target_weight
-        target_shares = target_value / price
+        current_shares = holdings.get(ticker, 0.0)
+        if (current_shares > 0 and target_weight > 0
+                and abs(target_value - current_shares * price) < resize_band * target_value):
+            continue
+        desired[ticker] = target_value / price
+
+    targets = (_allocate_lots(desired, holdings, prices, cash, lot_size,
+                              reserve=0.002 * total + 10.0 * len(desired))
+               if integer_shares else desired)
+
+    raw = []
+    for ticker, target_shares in targets.items():
+        price = prices[ticker]
         current_shares = holdings.get(ticker, 0.0)
         delta = target_shares - current_shares
-
-        if integer_shares:
-            if target_weight <= 0 and current_shares > 0:
-                delta = -current_shares     # full exit may include an odd lot
-            else:
-                # Round toward zero: never overshoot cash or over-sell.
-                delta = int(delta / lot_size) * lot_size
-
         trade_value = abs(delta * price)
         if trade_value < min_trade_value:
             continue
@@ -735,25 +777,32 @@ def _strategy_fees(args, market: str) -> dict:
 
 
 def _signals_for_sessions(strategy_name: str, open_data: pd.DataFrame,
-                          days: pd.DatetimeIndex) -> pd.DataFrame:
-    """Target weights for many sessions in one pass: row ``d`` uses opens up to ``d``.
+                          days: pd.DatetimeIndex, bars: dict | None = None) -> pd.DataFrame:
+    """Target weights traded at the close of each day in ``days``, in one pass.
 
-    Equivalent to ``generate_target_weights`` per day for causal strategies
-    (weights at t never depend on data after t), which every registered
-    strategy must be.
+    A close-price strategy reads opens as its prices: the row for ``d`` uses
+    opens up to ``d`` (a dummy row after the last open makes the strategy's
+    internal shift(1) yield that row). A BarStrategy uses completed bars
+    through d-1. Every registered strategy must be causal (weights at t never
+    depend on data after t), so one pass equals running each day separately.
     """
+    strategy = STRATEGY_REGISTRY[strategy_name]()
+    if isinstance(strategy, BarStrategy):
+        return bar_session_weights(strategy, bars, days)
     hist = open_data.loc[:days[-1]]
     dummy = pd.DataFrame([hist.iloc[-1].values], index=[hist.index[-1] + pd.Timedelta(days=1)],
                          columns=hist.columns)
-    weights = STRATEGY_REGISTRY[strategy_name]().generate_signals(pd.concat([hist, dummy]))
+    weights = strategy.generate_signals(pd.concat([hist, dummy]))
     return weights.shift(-1).reindex(days).fillna(0.0)
 
 
 def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp,
                  tickers: list[str], benchmark: str, close_data: pd.DataFrame,
                  open_data: pd.DataFrame, close_raw: pd.DataFrame, fees: dict,
-                 integer_shares: bool, signal_row: pd.Series | None = None) -> dict:
-    """Signal on ``day``'s open, execute at ``day``'s close, record the day.
+                 integer_shares: bool, signal_row: pd.Series | None = None,
+                 bars: dict | None = None) -> dict:
+    """Signal on ``day``'s open (a BarStrategy: bars through the previous
+    session), execute at ``day``'s close, record the day.
 
     Tickers without a real closing print on ``day`` (forward-filled price) are
     valued but not traded. ``signal_row`` supplies precomputed target weights.
@@ -761,10 +810,9 @@ def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp
     day_str = str(day.date())
     lot_size = MARKET_LOT_SIZE.get(market, 1) if integer_shares else 1
     if signal_row is None:
-        strategy = STRATEGY_REGISTRY[strategy_name]()
-        target_weights = generate_target_weights(strategy, open_data.loc[:day, tickers], day)
-    else:
-        target_weights = {t: float(w) for t, w in signal_row.items() if w > 1e-6}
+        signal_row = _signals_for_sessions(strategy_name, open_data[tickers],
+                                           pd.DatetimeIndex([day]), bars).loc[day]
+    target_weights = {t: float(w) for t, w in signal_row.items() if w > 1e-6}
 
     all_held = sorted(set(target_weights) | set(state["holdings"]))
     close_prices = _with_last_known_prices(
@@ -778,6 +826,7 @@ def _run_session(state: dict, strategy_name: str, market: str, day: pd.Timestamp
         state["holdings"], state["cash"], target_weights, close_prices,
         min_trade_value=max(50, pre_value * 0.001),
         integer_shares=integer_shares, lot_size=lot_size, frozen=frozen,
+        resize_band=RESIZE_BAND,
     )
     n_log = len(state["trade_log"])
     execute_trades(state, trades, close_prices, trade_date=day_str,
@@ -840,9 +889,11 @@ def cmd_morning(args):
     print(f"  Strategy: {strategy_name} | Market: {market.upper()}")
     print(f"{'='*60}")
 
-    # Run strategy on open prices
-    strategy = STRATEGY_REGISTRY[strategy_name]()
-    target_weights = generate_target_weights(strategy, open_data[tickers], today)
+    # Run strategy on open prices (a BarStrategy: on completed bars)
+    bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
+    row = _signals_for_sessions(strategy_name, open_data[tickers],
+                                pd.DatetimeIndex([today]), bars).loc[today]
+    target_weights = {t: float(w) for t, w in row.items() if w > 1e-6}
 
     # Get today's open prices for trade computation
     open_prices = get_prices_for_date(
@@ -854,7 +905,8 @@ def cmd_morning(args):
     trades = compute_trades(state["holdings"], state["cash"], target_weights,
                            open_prices, min_trade_value=max(50, total * 0.001),
                            integer_shares=args.integer_shares,
-                           lot_size=MARKET_LOT_SIZE.get(market, 1))
+                           lot_size=MARKET_LOT_SIZE.get(market, 1),
+                           resize_band=RESIZE_BAND)
 
     # Store pending
     state["pending_trades"] = {
@@ -1335,18 +1387,8 @@ def cmd_simulate(args):
 
     # Pre-compute strategy weights for the full period (much faster than per-day)
     # Append dummy row so shift(1) inside strategy produces valid last-day signal
-    strategy = STRATEGY_REGISTRY[strategy_name]()
-    open_tickers = open_data[tickers]
-    dummy_date = open_tickers.index[-1] + pd.Timedelta(days=1)
-    dummy_row = pd.DataFrame(
-        [open_tickers.iloc[-1].values], index=[dummy_date], columns=open_tickers.columns
-    )
-    extended = pd.concat([open_tickers, dummy_row])
-    raw_weights = strategy.generate_signals(extended)
-    # Strategy's shift(1): raw_weights[t] uses data up to t-1
-    # For date t, the signal using open[t] is in raw_weights[t+1]
-    # So shift(-1) to align: full_weights[t] = signal using open[t]
-    full_weights = raw_weights.shift(-1).fillna(0.0)
+    bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
+    full_weights = _signals_for_sessions(strategy_name, open_data[tickers], trading_days, bars)
 
     print(f"\n{'='*70}")
     print(f"  SIMULATION: {strategy_name} | {market.upper()}")
@@ -1385,6 +1427,7 @@ def cmd_simulate(args):
             state["holdings"], state["cash"], target_weights,
             close_prices, min_trade_value=max(50, pre_value * 0.001),
             integer_shares=args.integer_shares, lot_size=lot_size,
+            resize_band=RESIZE_BAND,
         )
 
         execute_trades(state, trades, close_prices,
@@ -1710,13 +1753,16 @@ def cmd_monitor(args):
 def refresh_panels(market: str, strategy_names: list[str], label: str) -> None:
     """Refresh every price panel the strategies read, once, for the union of their tickers."""
     requests: dict[str, set[str]] = {}
+    fields: dict[str, set[str]] = {}
     for name in strategy_names:
         tickers, benchmark = strategy_universe(market, name)
-        requests.setdefault(strategy_data_market(market, name), set()).update(
+        data_market = strategy_data_market(market, name)
+        requests.setdefault(data_market, set()).update(
             price_request(market, list(tickers) + [benchmark]))
+        fields.setdefault(data_market, {"close", "open"}).update(strategy_fields(name))
     for data_market, request in requests.items():
         print(f"[{label}] Refreshing {data_market} panel for {len(request)} tickers")
-        data_manager.update(data_market, sorted(request), with_open=True)
+        data_manager.update(data_market, sorted(request), fields=sorted(fields[data_market]))
 
 
 def _process_sessions(args, state: dict | None, start: pd.Timestamp | None = None,
@@ -1759,12 +1805,12 @@ def _process_sessions(args, state: dict | None, start: pd.Timestamp | None = Non
               f"{todo[0].date()} .. {todo[-1].date()}")
 
     fees = _strategy_fees(args, market)
-    signals = (_signals_for_sessions(strategy_name, open_data[tickers], todo)
-               if len(todo) > 1 else None)
+    bars = strategy_bars(market, strategy_name, tickers, close_data, open_data)
+    signals = _signals_for_sessions(strategy_name, open_data[tickers], todo, bars)
     for day in todo:
         r = _run_session(state, strategy_name, market, day, tickers, benchmark,
                          close_data, open_data, close_raw, fees, args.integer_shares,
-                         signal_row=signals.loc[day] if signals is not None else None)
+                         signal_row=signals.loc[day], bars=bars)
         frozen = f" | no fresh price: {', '.join(r['frozen'])}" if r["frozen"] else ""
         print(f"[{label}]   {r['date']} {strategy_name:<34} value {r['value']:>12,.2f} "
               f"({r['daily_return']:+.2%}) trades {r['trades']:>2} "

universe.py

@@ -76,45 +76,70 @@ def get_sp500() -> list[str]:
 # Yahoo Finance format: Shanghai → XXXXXX.SS, Shenzhen → XXXXXX.SZ
 # Wikipedia CSI 300 page has a table with stock codes and exchanges.
 
+CSI300_CONS_URL = ("https://oss-ch.csindex.com.cn/static/html/csindex/public/uploads/"
+                   "file/autofile/cons/000300cons.xls")
+
+
+def _csi300_official() -> list[str]:
+    """Current constituents from China Securities Index's published file."""
+    req = urllib.request.Request(CSI300_CONS_URL, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=30) as resp:
+        payload = resp.read()
+    df = pd.read_excel(io.BytesIO(payload), dtype=str)
+    code_col = next(c for c in df.columns if "Constituent Code" in c)
+    exch_col = next(c for c in df.columns if c.startswith("交易所Exchange"))
+    tickers = [f"{code}.{'SS' if '上海' in exch else 'SZ'}"
+               for code, exch in zip(df[code_col], df[exch_col])]
+    if len(tickers) < 250:
+        raise RuntimeError(f"CSI 300 file has only {len(tickers)} rows")
+    return sorted(set(tickers))
+
+
+def _csi300_latest_snapshot() -> list[str]:
+    with open(os.path.join(CACHE_DIR, "csi300_history.json")) as f:
+        snaps = json.load(f)["snapshots"]
+    return snaps[max(snaps)]
+
+
+def _csi300_wikipedia() -> list[str]:
+    tables = _fetch_html_tables("https://en.wikipedia.org/wiki/CSI_300_Index")
+    # The constituent table has ~300 rows and a "Ticker" column like "SSE: 600519"
+    df = next((t for t in tables if len(t) >= 200 and "Ticker" in t.columns), None)
+    if df is None:
+        raise RuntimeError("Could not find CSI 300 constituent table on Wikipedia")
+    tickers = []
+    for raw in df["Ticker"].astype(str):
+        parts = raw.split(":")
+        if len(parts) == 2 and parts[1].strip().isdigit():
+            suffix = ".SS" if parts[0].strip().upper() == "SSE" else ".SZ"
+            tickers.append(parts[1].strip() + suffix)
+    return sorted(set(tickers))
+
+
 def get_csi300() -> list[str]:
-    """Fetch current CSI 300 constituents from Wikipedia."""
+    """Current CSI 300 constituents.
+
+    Source order: China Securities Index's published constituent file, then
+    the latest snapshot in data/csi300_history.json, then Wikipedia (whose
+    table lagged the real index by ~60 names in 2026).
+    """
     cached = _read_cache("csi300")
     if cached:
         print(f"--- Loaded CSI 300 universe from cache ({len(cached)} tickers) ---")
         return cached
 
-    print("--- Fetching CSI 300 constituents from Wikipedia ---")
-    url = "https://en.wikipedia.org/wiki/CSI_300_Index"
-    tables = _fetch_html_tables(url)
-
-    # Find the constituent table — it has ~300 rows and a "Ticker" column
-    # with format like "SSE: 600519" or "SZSE: 300750"
-    df = None
-    for t in tables:
-        if len(t) >= 200 and "Ticker" in t.columns:
-            df = t
-            break
-
-    if df is None:
-        raise RuntimeError("Could not find CSI 300 constituent table on Wikipedia")
-
-    tickers = []
-    for _, row in df.iterrows():
-        raw = str(row["Ticker"])  # e.g. "SSE: 600519" or "SZSE: 300750"
-        parts = raw.split(":")
-        if len(parts) != 2:
+    for label, source in (("csindex.com.cn", _csi300_official),
+                          ("stored CSI 300 history", _csi300_latest_snapshot),
+                          ("Wikipedia", _csi300_wikipedia)):
+        try:
+            tickers = source()
+        except Exception as exc:  # noqa: BLE001 — fall through to the next source
+            print(f"--- CSI 300 from {label} failed: {exc} ---")
             continue
-        exchange_prefix = parts[0].strip().upper()
-        code = parts[1].strip()
-        if not code.isdigit():
-            continue
-        suffix = ".SS" if exchange_prefix == "SSE" else ".SZ"
-        tickers.append(code + suffix)
-
-    tickers = sorted(set(tickers))
-    _write_cache("csi300", tickers)
-    print(f"--- Got {len(tickers)} CSI 300 tickers ---")
-    return tickers
+        _write_cache("csi300", tickers)
+        print(f"--- Got {len(tickers)} CSI 300 tickers from {label} ---")
+        return tickers
+    raise RuntimeError("No CSI 300 constituent source available")
 
 
 # ---------------------------------------------------------------------------

universe_history.py

@@ -20,6 +20,7 @@ import os
 import urllib.request
 from datetime import date, datetime
 
+import numpy as np
 import pandas as pd
 
 CACHE_DIR = "data"
@@ -30,34 +31,51 @@ _HEADERS = {"User-Agent": "Mozilla/5.0 (quant-backtest)"}
 # Fetch + parse Wikipedia
 # ---------------------------------------------------------------------------
 
-def _fetch_sp500_tables() -> tuple[pd.DataFrame, pd.DataFrame]:
-    """Return (current_list, changes_log) from the S&P 500 Wikipedia page."""
-    url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
+SP500_LIST_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
+# The change log moved off the list page in 2026; the list page is still tried
+# first in case it moves back.
+SP500_CHANGES_URLS = [
+    SP500_LIST_URL,
+    "https://en.wikipedia.org/wiki/Historical_components_of_the_S%26P_500",
+]
+
+
+def _read_tables(url: str) -> list[pd.DataFrame]:
     req = urllib.request.Request(url, headers=_HEADERS)
     with urllib.request.urlopen(req) as resp:
         html = resp.read().decode("utf-8")
-    tables = pd.read_html(io.StringIO(html))
-    current = tables[0]
-    change_tables = [
-        t for t in tables[1:]
-        if any("Added" in "_".join(map(str, c if isinstance(c, tuple) else (c,))) for c in t.columns)
-    ]
-    if not change_tables:
-        raise ValueError("S&P 500 change log table not found on the Wikipedia page")
-    changes = change_tables[0]
-    changes.columns = [
-        "_".join(c).strip() if isinstance(c, tuple) else c
-        for c in changes.columns
-    ]
-    changes.columns = [
-        c.replace("Effective Date_Effective Date", "Date") for c in changes.columns
-    ]
-    return current, changes
+    return pd.read_html(io.StringIO(html))
+
+
+def _find_changes_table(tables: list[pd.DataFrame]) -> pd.DataFrame | None:
+    for t in tables:
+        if any("Added" in "_".join(map(str, c if isinstance(c, tuple) else (c,))) for c in t.columns):
+            changes = t.copy()
+            changes.columns = [
+                "_".join(c).strip() if isinstance(c, tuple) else c
+                for c in changes.columns
+            ]
+            changes.columns = [
+                c.replace("Effective Date_Effective Date", "Date") for c in changes.columns
+            ]
+            return changes
+    return None
+
+
+def _fetch_sp500_tables() -> tuple[pd.DataFrame, pd.DataFrame]:
+    """Return (current_list, changes_log) from Wikipedia."""
+    current = _read_tables(SP500_LIST_URL)[0]
+    for url in SP500_CHANGES_URLS:
+        changes = _find_changes_table(_read_tables(url))
+        if changes is not None:
+            return current, changes
+    raise ValueError("S&P 500 change log table not found on Wikipedia")
 
 
 def _normalize_ticker(t: str) -> str:
     """Yahoo Finance ticker format: BRK.B → BRK-B."""
-    return str(t).replace(".", "-").strip()
+    # Some change-log cells carry a trailing " |" separator.
+    return str(t).split("|")[0].replace(".", "-").strip()
 
 
 # ---------------------------------------------------------------------------
@@ -243,3 +261,101 @@ def mask_prices(prices: pd.DataFrame,
     """
     mask = membership_mask(prices.index, intervals, tickers=list(prices.columns))
     return prices.where(mask)
+
+
+# ---------------------------------------------------------------------------
+# CSI 300 (China A-shares)
+# ---------------------------------------------------------------------------
+#
+# data/csi300_history.json holds monthly constituent snapshots
+#     {"snapshots": {"YYYY-MM-DD": [yahoo tickers], ...}}
+# seeded from baostock's query_hs300_stocks (first trading day of each month,
+# 2008 onward). A snapshot applies from its date until the next one, so an
+# index change is picked up within a month. load_csi300_history() appends a
+# snapshot whenever the current constituent list (universe.get_csi300) differs
+# from the latest one, which keeps the file current without baostock.
+
+def _csi300_path() -> str:
+    return os.path.join(CACHE_DIR, "csi300_history.json")
+
+
+def baostock_to_yahoo(code: str) -> str:
+    """'sh.600000' -> '600000.SS', 'sz.000001' -> '000001.SZ'."""
+    ex, num = code.split(".")
+    return f"{num}.{'SS' if ex == 'sh' else 'SZ'}"
+
+
+def load_csi300_history(current: list[str] | None = None) -> dict[str, list[str]]:
+    """Monthly CSI 300 snapshots; appends ``current`` (or today's list) if it changed."""
+    path = _csi300_path()
+    with open(path) as f:
+        data = json.load(f)
+    snaps = data["snapshots"]
+    if current is None:
+        try:
+            from universe import get_csi300
+            current = get_csi300()
+        except Exception as exc:  # noqa: BLE001 — offline runs use the stored history
+            print(f"--- WARNING: CSI 300 list unavailable ({exc}); using stored history ---")
+            current = None
+    if current:
+        last = snaps[max(snaps)]
+        today = str(date.today())
+        if sorted(current) != sorted(last) and today > max(snaps):
+            snaps[today] = sorted(current)
+            with open(path, "w") as f:
+                json.dump(data, f, indent=0)
+            print(f"--- CSI 300 constituents changed; stored snapshot {today} ---")
+    return snaps
+
+
+def csi300_membership_mask(dates: pd.DatetimeIndex, tickers: list[str],
+                           snapshots: dict[str, list[str]] | None = None) -> pd.DataFrame:
+    """Boolean mask (dates x tickers): True while the ticker was in the CSI 300."""
+    snaps = snapshots if snapshots is not None else load_csi300_history()
+    keys = sorted(snaps)
+    col = {t: i for i, t in enumerate(tickers)}
+    arr = np.zeros((len(dates), len(tickers)), dtype=bool)
+    for i, k in enumerate(keys):
+        start = pd.Timestamp(k)
+        end = pd.Timestamp(keys[i + 1]) if i + 1 < len(keys) else dates[-1] + pd.Timedelta(days=1)
+        rows = (dates >= start) & (dates < end)
+        cols = [col[t] for t in snaps[k] if t in col]
+        if rows.any() and cols:
+            arr[np.ix_(rows, cols)] = True
+    mask = pd.DataFrame(arr, index=dates, columns=tickers)
+    # Tickers never in the index (benchmarks, ETFs) pass through unmasked.
+    ever = {t for lst in snaps.values() for t in lst}
+    for t in tickers:
+        if t not in ever:
+            mask[t] = True
+    return mask
+
+
+def all_csi300_tickers_ever(snapshots: dict[str, list[str]] | None = None,
+                            since: str | None = None) -> list[str]:
+    snaps = snapshots if snapshots is not None else load_csi300_history()
+    return sorted({t for k, lst in snaps.items() if since is None or k >= since for t in lst})
+
+
+# ---------------------------------------------------------------------------
+# Market-generic helpers
+# ---------------------------------------------------------------------------
+
+def pit_tickers_ever(market: str, since: str | None = None) -> list[str] | None:
+    """Every ticker that was an index member (since ``since``), or None when the
+    market has no membership history."""
+    if market == "us":
+        return all_tickers_ever()
+    if market == "cn":
+        return all_csi300_tickers_ever(since=since)
+    return None
+
+
+def mask_prices_pit(prices: pd.DataFrame, market: str) -> pd.DataFrame:
+    """NaN out (date, ticker) pairs outside point-in-time index membership."""
+    if market == "us":
+        return mask_prices(prices)
+    if market == "cn":
+        return prices.where(csi300_membership_mask(prices.index, list(prices.columns)))
+    return prices

uv.lock

@@ -770,6 +770,7 @@ dependencies = [
     { name = "numpy" },
     { name = "pandas" },
     { name = "scipy" },
+    { name = "xlrd" },
     { name = "yfinance" },
 ]
 
@@ -780,6 +781,7 @@ requires-dist = [
     { name = "numpy", specifier = ">=2.4.0" },
     { name = "pandas", specifier = ">=2.3.3" },
     { name = "scipy", specifier = ">=1.17.1" },
+    { name = "xlrd", specifier = ">=2.0.2" },
     { name = "yfinance", specifier = ">=1.0" },
 ]
 
@@ -935,6 +937,15 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" },
 ]
 
+[[package]]
+name = "xlrd"
+version = "2.0.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/07/5a/377161c2d3538d1990d7af382c79f3b2372e880b65de21b01b1a2b78691e/xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9", size = 100167, upload-time = "2025-06-14T08:46:39.039Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/1a/62/c8d562e7766786ba6587d09c5a8ba9f718ed3fa8af7f4553e8f91c36f302/xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9", size = 96555, upload-time = "2025-06-14T08:46:37.766Z" },
+]
+
 [[package]]
 name = "yfinance"
 version = "1.0"