gahow/quant-trading
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: remove pre-lab research and dead strategies; index them in research/ARCHIVE.md

Browse Source 
Removed (79 files, all in history; research/ARCHIVE.md lists each with its
purpose and the commit to restore it from):
- root factor_*.py mining scripts (2026-04) and yearly_sweep.py
- scripts/run_daily.{py,sh} (superseded by the monitor service; no crontab)
- strategies no registry, test or live book uses: trend_rider_v8,
  adaptive_blend, cross_asset_momentum, long_hedged, ls_momentum
- 55 research/*.py scripts from 2026-04/05 written before research/lab,
  the us-alpha design spec, and the 7 tests that only covered them

Kept research/permanent_yearly.py: bridge.py and precompute_backtest.py
import equity_with_cashflows. CLAUDE.md gains the archive note and the
command lines of the studies committed in the previous commit.

Checks: 99 unit tests OK (125 before, minus the 26 removed with their
modules), all 157 remaining modules import, 94 registry strategies build,
bridge.py list OK, web 23 tests OK and build OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-10-09 22:20:59 -07:00
parent 04e2c5c3be
commit a9683b6d1f
81 changed files with 121 additions and 19668 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
CLAUDE.md
View File

@@ -74,6 +74,7 @@ PYTHONPATH=. uv run --with numba python -m research.lab.qqq_scan # alpha
PYTHONPATH=. uv run --with numba python -m research.lab.beat_qqq # sleeves picked on 2006-16, judged 2017-26; odds of beating QQQ
PYTHONPATH=. uv run --with numba python -m research.lab.momentum_fundamentals [--sensitivity] # financial-statement vetoes/confirmations on overnight_recovery_top10
uv run python -m research.lab.fetch_insider && PYTHONPATH=. uv run python -m research.lab.insider # SEC Form 4 insider factors
PYTHONPATH=. uv run python -m research.lab.fetch_insider --universe mid small && PYTHONPATH=. uv run python -m research.lab.insider_mid # S&P 400/600 Form 4 factors (all REJECT 2026-10-09)
PYTHONPATH=. uv run python -m research.lab.fetch_lazy_prices && PYTHONPATH=. uv run python -m research.lab.lazy_prices # 10-K/10-Q text similarity, 8-K earnings dates, SIC
PYTHONPATH=. uv run --with numba python -m research.lab.earnings # earnings-announcement return (needs sec_8k_earnings.csv)
PYTHONPATH=. uv run --with numba python -m research.lab.industry # industry momentum on SIC groups
@@ -87,6 +88,7 @@ PYTHONPATH=. uv run python -m research.lab.sector_cycle # busin
uv run python -m research.lab.fetch_wiki_attention && PYTHONPATH=. uv run --with numba python -m research.lab.attention # Wikipedia-pageview attention
PYTHONPATH=. uv run --with numba python -m research.lab.concentration # industry caps on the momentum sleeve
PYTHONPATH=. uv run python -m research.lab.universe_midcap # overnight_recovery_top10 on S&P 500 ∪ S&P 400 (preregistered; FAIL 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.union_shield # SPY>MA150 overlay on the union arms (preregistered; FAIL 2026-10-09)
PYTHONPATH=. uv run --with numba python -m research.lab.stream_portfolio # risk-budgeted book of momentum/lowrisk/TrendRider/ETF TSMOM/BTC streams, 2006-26 (preregistered; REJECT 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.hedged_momentum # momentum with its beta hedged by index futures (quick check; STOP 2026-10-08)
PYTHONPATH=. uv run python -m research.lab.futures_trend [--no-proxy] # 24-market futures TSMOM on Yahoo =F + data checks (quick check; Sharpe 0.44, DATA 2026-10-08)
@@ -101,6 +103,7 @@ PYTHONPATH=. uv run python -m research.lab.overnight_only # momentum n
PYTHONPATH=. uv run python -m research.lab.overnight_book # three-sleeve book with the overnight-only momentum sleeve, leverage for 50%/yr (preregistered; CONTINUE 2026-10-09: Sharpe 0.98 -> 1.38, MaxDD -37% -> -17% at 1 bp/side; 4 bp/side erases it)
PYTHONPATH=. uv run python -m research.lab.overnight_book_mf [--no-proxy] # the overnight books + a DBMF/CTA sleeve, levered to 50%/yr (preregistered; KEEP 2026-10-09: levered MaxDD unchanged, no CTA sleeve)
PYTHONPATH=. uv run python -m research.lab.daytime_reversal # buy the S&P 500's biggest overnight losers at 09:35, sell at the close (preregistered; STOP 2026-10-09: gross 1-6 bp/day, net Sharpe ~0)
PYTHONPATH=. uv run python -m research.lab.overnight_midsmall # the overnight-only momentum book on S&P 400/600 (preregistered; STOP 2026-10-09: 0.93 net at 3 bp/side, corr 0.83 with the S&P 500 book)
uv run python -m research.lab.fetch_us_fundamentals --universe mid # EDGAR PIT facts for S&P 400 ever-members (name-checked CIKs)
uv run python -m research.lab.fetch_us_fundamentals --universe small # the same for S&P 600 (IJR)
PYTHONPATH=. uv run python -m research.lab.midcap_sleeve [--only h1..h4] [--universe small] # S&P 400 sleeves vs IJH; S&P 600 replication vs IJR (all REJECT 2026-10-08)
@@ -125,6 +128,7 @@ PYTHONPATH=. uv run python -m research.lab.tiingo_merge --production [--apply]
PYTHONPATH=. uv run python -m research.lab.tiingo_merge --yahoo-mid # convert caches to yahoo_mid format (fetch_us_mid.load_bars falls back to it)
PYTHONPATH=. uv run python -m research.factor_panel_check # pre-vs-post overlay S&P factor comparison
PYTHONPATH=. uv run python -m research.midsmall_scan --panel both # mid/small factor scan: Yahoo-only vs Tiingo-extended panel
PYTHONPATH=. uv run python -m research.midsmall_rebuild # convert every cached Tiingo name + rebuild the mid/small PIT panels (re-run after each monthly batch)
PYTHONPATH=. uv run python -m research.prod_impact [STRATEGY ...] # A/B registered replays: backfilled store vs .bak-tiingo snapshot
# Setup
@@ -186,6 +190,7 @@ uv run python -m unittest discover tests
- Warm-up periods: each strategy zeros out its first N rows where indicators are undefined.
- `data/` contains persistent price CSVs, universe JSON caches, and trader state files, plus `data/precomputed/` (precomputed backtests), `data/web/` (watchlist, per-stock cache, backtest result cache) and `data/research/` (research panels, ~2 GB). Not checked into git (only `data/.gitkeep` is tracked). `data/sp500_history.json` is rebuilt from Wikipedia on first use; `data/csi300_history.json` is not rebuilt automatically (its 2008-2026 snapshots came from baostock), so copy it to a new machine or CN membership fails to load.
- Python 3.12+, managed with `uv`.
- Research code lives in `research/lab/` (PIT panels, next-open fills, preregistered studies with their verdict in the docstring). Pre-lab scripts removed in the 2026-10-09 clean-up are indexed in `research/ARCHIVE.md` with the commit to restore them from. `research/permanent_yearly.py` stays because bridge.py and precompute_backtest.py import `equity_with_cashflows`.
- Do not show matplotlib figures when running backtests; use `--no-plot`.
The `*_rb20` strategies are execution-only candidates with a 20% resize band; baseline strategies keep 10%. They run alongside their baselines only in `validation_2026_10`, starting with the first market session on/after 2026-10-02, with matching capital and fees. Existing forward histories are preserved and future daily snapshots record the engine version. These candidates were selected from previously examined historical data and have no independent forward performance yet.

376
docs/superpowers/specs/2026-04-17-us-alpha-research-design.md
View File

@@ -1,376 +0,0 @@
# US High-Alpha Research Design
**Date:** 2026-04-17
## Goal
Build a research framework for US `long-only` equity strategies that uses only free or already-accessible data, avoids lookahead and survivorship traps as much as the available data allows, and can rank candidate strategy families over `1/2/3/5/10y` windows. The objective is not to manufacture the single highest backtest CAGR, but to identify strategy families whose alpha survives realistic liquidity filters, transaction costs, and point-in-time constraints.
## Constraints
- Data sources must be free or already accessible from the current project environment.
- Portfolio construction must be `long-only`.
- The US research universe may extend beyond the S&P 500 into a broader US stock pool, but all conclusions must clearly distinguish between:
- `strict` results from a point-in-time-clean universe.
- `exploratory` results from a wider free-data universe that is not fully point-in-time-clean.
- All signals must use only information available at the time of decision.
- The framework must explicitly guard against:
- survivorship bias
- lookahead bias
- static industry-label leakage
- microcap and illiquidity contamination
## Success Criteria
The framework is successful if it produces:
1. A unified research and backtest pipeline for US strategies.
2. A ranked comparison of `3-5` high-value strategy families across `1/2/3/5/10y`.
3. Metrics that go beyond headline CAGR, including:
- `CAGR`
- `Sharpe`
- `Sortino`
- `MaxDD`
- `Calmar`
- `Turnover`
- `Average positions`
- `Median ADV usage`
- `Subperiod stability`
4. Tiered interpretation of results:
- `Tier A`: realistic and tradable under tighter liquidity assumptions
- `Tier B`: strong alpha but lower capacity
- `Tier C`: attractive only under loose assumptions and not suitable as a production candidate
Any strategy that reports near-`50% CAGR` must also explain:
- which market regime contributed most of the return
- whether performance depends on low-liquidity or small-cap tails
- whether results survive after removing the most extreme tail names
## Research Philosophy
This project should prefer honest, repeatable alpha discovery over spectacular but fragile backtests. Under the current constraints, a `10y 50% CAGR` should be treated as an upper-end outcome that may appear in selective windows, not as a baseline expectation. The more realistic goal is to find strategies that are strong over `3/5y`, still meaningfully outperform over `10y`, and remain robust after tightening assumptions.
## Strategy Families
The research effort will focus on four strategy families.
### 1. Earnings Drift Proxy
Target the post-information-repricing phase after major company-specific events. This is conceptually the highest-alpha family, but also the most dependent on event data quality.
Primary implementation order:
- use free historical earnings date data if it is stable enough
- otherwise fall back to price-and-volume-defined event proxies
Core signal ingredients:
- strong post-event excess return over `1-3` days
- abnormal volume
- gap that does not immediately fill
- price holding near short- and medium-term highs after the event
### 2. Breakout After Compression
Target stocks that transition from low-volatility congestion into sustained trend expansion. This is the cleanest strategy family to implement with free daily OHLCV data and is the best first candidate for a strict production-grade pipeline.
Core signal ingredients:
- proximity to `120d` or `252d` highs
- volatility compression over the prior `20-40` trading days
- rising dollar volume
- positive relative strength versus market and industry proxies
### 3. Gap-and-Go / High-Volume Continuation
Target the second phase of move continuation after abnormal return and volume shocks rather than blindly chasing the first event day.
Core signal ingredients:
- abnormal `1d` or `3d` return
- abnormal volume versus trailing `60d`
- post-event price holding above the event anchor
- subsequent breakout continuation
This family has high potential upside but is more sensitive to cost assumptions and market regime.
### 4. Regime-Gated Cross-Sectional Alpha
Use broad market and industry-state filters to improve the hit rate of the other strategy families and provide a lower-volatility baseline alpha engine.
Core signal ingredients:
- market risk-on versus risk-off state
- industry ETF leadership
- relative strength
- recovery from drawdowns
- trend quality
- near-`52w` high behavior
- price/volume confirmation
This family is not expected to produce the highest standalone CAGR, but it is expected to improve robustness and reduce participation in hostile environments.
## Prioritization
Recommended implementation order:
1. `Breakout After Compression`
2. `Regime-Gated Cross-Sectional Alpha`
3. `Gap-and-Go / High-Volume Continuation`
4. `Earnings Drift Proxy` only after validating free event-data quality
Rationale:
- `Breakout After Compression` is the most implementable and least ambiguous with free data.
- `Regime-Gated Cross-Sectional Alpha` provides a shared control layer for the rest of the framework.
- `Gap-and-Go` has higher upside but also higher sensitivity to assumptions.
- `Earnings Drift Proxy` is theoretically powerful but should not become the project bottleneck if free event history is incomplete.
## Data Layer
The framework needs a richer data layer than the current `close/open` setup.
### Required price fields
Daily US market data should support at least:
- `open`
- `high`
- `low`
- `close`
- `volume`
This is required to define:
- real breakouts
- gap events
- volatility compression
- abnormal dollar volume
### Required ETF layer
Add stable market and industry ETFs for regime and leadership analysis, at minimum:
- `SPY`
- `QQQ`
- `IWM`
- `MDY`
- `XLF`
- `XLK`
- `XLI`
- `XLV`
- `XLY`
- `XLP`
- `XLE`
- `XLU`
- `XLRE`
- `XLB`
- `SOXX`
- `IGV`
- `SMH`
### Universe modes
The framework must support two explicit modes.
#### Strict mode
Use point-in-time-clean universe membership, initially based on the existing PIT S&P 500 machinery in the repository. This is the baseline for formal, defensible results.
#### Exploratory mode
Use a wider free-data US stock pool to search for stronger alpha patterns. These results are useful for idea generation but must be labeled as exploratory unless later promoted into a point-in-time-clean setup.
## Universe Construction Rules
The tradable universe must be computed daily from lagged information.
### Daily eligibility rules
Each stock may enter the candidate set only if all required conditions hold as of `t-1`:
- enough listing history exists to compute the strategy lookbacks
- enough valid volume observations exist
- minimum lagged price threshold is met
- minimum lagged dollar-volume threshold is met
Representative defaults:
- `close[t-1] > 5`
- `median_dollar_volume_60d[t-1] > $20M` in `strict` mode
- `median_dollar_volume_60d[t-1] > $5M` in `exploratory` mode
- `>= 252` valid trading days before eligibility
- `>= 40` valid volume days in the trailing `60d`
Thresholds should be strategy-specific and tunable in robustness sweeps.
### Industry mapping
Do not use today's static sector labels to explain historical behavior. For historical regime and industry alignment, prefer PIT-safe proxies such as rolling correlation or beta to industry ETFs over `63/126d` windows.
## Anti-Lookahead Rules
The framework must enforce the following rules consistently.
1. Signals computed using `t` daily bars may only be traded no earlier than `t+1`.
2. If an event is effectively published after market close, it becomes tradable no earlier than the next trading day after publication.
3. Rolling inputs for liquidity, volatility, and breakout logic must use complete lagged windows with explicit timing semantics.
4. Cross-sectional ranking must happen only within the daily eligible universe.
5. Universe membership, filters, and factor normalization must be applied before portfolio selection, not after.
## Execution Convention
Default execution convention:
- observe data through `t` close
- compute signal after the `t` close
- trade at `t+1`
The framework may compare `t+1 open` and `t+1 close` execution variants if the data path supports both, but the default research baseline should be conservative and consistent.
## Backtest and Evaluation Framework
Every strategy family must run through a single pipeline that:
1. loads required market data
2. constructs the daily eligible universe
3. computes regime filters
4. computes strategy scores or event states
5. builds a `long-only` portfolio
6. applies transaction costs
7. reports `1/2/3/5/10y` windows
8. records robustness diagnostics
### Portfolio defaults
Initial baseline settings:
- `long-only`
- concentrated books such as `top 5`, `top 10`, `top 20`
- start with `equal weight`
- add `inverse-vol` weighting only as a secondary comparison
Equal-weight concentrated portfolios should be the first baseline because they are harder to over-engineer than adaptive weighting schemes.
### Required robustness checks
Any strategy candidate that looks strong must automatically be re-run under:
- tighter liquidity thresholds
- fewer and more positions
- higher trading costs
- different rebalance frequencies
- exclusion of the lowest-liquidity or smallest-cap tail
Only strategies that survive these perturbations should be promoted to `Tier A`.
## Repository Changes
The following repository changes are required.
### New modules
#### `research/us_universe.py`
Responsibilities:
- build daily tradable-universe masks
- support `strict` and `exploratory` modes
- enforce lagged eligibility rules
#### `data_manager.py` extension or new `market_data.py`
Responsibilities:
- support daily US `OHLCV`
- support ETF data updates
- preserve existing price-loading workflows where practical
#### `research/regime_filters.py`
Responsibilities:
- market risk-on/risk-off filters
- ETF leadership signals
- breadth and relative-strength helpers
#### `research/event_factors.py`
Responsibilities:
- breakout-compression scores
- gap-continuation scores
- high-volume continuation logic
- earnings-drift proxy logic
#### `research/us_alpha_pipeline.py`
Responsibilities:
- orchestrate end-to-end research runs
- load data
- build universe masks
- run strategy families
- produce windowed rankings
- label output as `strict` or `exploratory`
#### `research/us_alpha_report.py`
Responsibilities:
- format tables and CSV outputs
- summarize results by family and horizon
- support markdown export if needed
## Research Phasing
The implementation should be split into two phases.
### Phase 1
Build the strict, defensible research backbone:
- PIT S&P 500 universe
- OHLCV data support
- ETF regime filters
- `Breakout After Compression`
- `Regime-Gated Cross-Sectional Alpha`
- `Gap-and-Go / High-Volume Continuation`
- unified backtest and reporting pipeline
This phase should produce a clean research system that is difficult to fool with future information.
### Phase 2
Expand into higher-upside exploratory research:
- wider US stock universe
- broader signal scanning
- stronger CAGR search
- explicit exploratory labeling
This phase is for alpha discovery, not for making final claims about unbiased production performance.
## Recommended Output
The finished framework should produce:
- a repeatable research entrypoint for US alpha studies
- CSV outputs for `1/2/3/5/10y` windows
- a ranked table of strategy families
- tier classification for candidates
- notes on where near-`50% CAGR` outcomes come from and whether they remain credible after tightening assumptions
## Non-Goals
This project does not aim to:
- promise stable `10y 50% CAGR`
- claim a fully point-in-time-clean all-US-stock universe from free data alone
- optimize to a single headline metric at the expense of realism
- treat exploratory full-market scans as production-quality evidence
## Key Decision
The core design choice is to build infrastructure that minimizes self-deception first, and only then search for extreme CAGR outcomes. Any other order is likely to produce attractive but unreliable results.

213
factor_backtest.py
View File

@@ -1,213 +0,0 @@
"""
Backtest best factor combinations with yearly return breakdown.
US best: momentum + recovery + low_downside_beta + short_term_reversal
CN best: momentum + anti_lottery + vol_reversal
"""
from __future__ import annotations
import argparse
import numpy as np
import pandas as pd
import data_manager
import metrics
from universe import UNIVERSES
from factor_research import (
factor_momentum_12_1,
factor_recovery,
factor_short_term_reversal,
factor_downside_beta_proxy,
factor_lottery_demand,
factor_turnover_reversal,
factor_52w_high_distance,
)
def build_strategy_signals(
prices: pd.DataFrame,
factor_funcs: list,
weights: list[float],
top_n: int = 10,
rebal_freq: int = 21,
) -> pd.DataFrame:
"""Build equal-weight top-N strategy from ranked factor combination."""
signals_list = [f(prices) for f in factor_funcs]
ranked = [s.rank(axis=1, pct=True, na_option="keep") for s in signals_list]
composite = sum(w * r for w, r in zip(weights, ranked))
# Warmup: need at least 252 days
warmup = 252
rank = composite.rank(axis=1, ascending=False, na_option="bottom")
n_valid = composite.notna().sum(axis=1)
enough = n_valid >= top_n
top_mask = (rank <= top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# Monthly rebalance
rebal_mask = pd.Series(False, index=prices.index)
rebal_indices = list(range(warmup, len(prices), rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
return signals.shift(1).fillna(0.0)
def backtest_equity(signals: pd.DataFrame, prices: pd.DataFrame, cost: float = 0.001) -> pd.Series:
"""Simple vectorized backtest returning equity curve."""
returns = prices.pct_change().fillna(0.0)
port_ret = (signals * returns).sum(axis=1)
# Transaction costs via turnover
turnover = signals.diff().abs().sum(axis=1)
port_ret -= turnover * cost
equity = (1 + port_ret).cumprod() * 100000
return equity
def yearly_returns(equity: pd.Series) -> pd.DataFrame:
"""Compute calendar year returns from equity curve."""
daily_ret = equity.pct_change().fillna(0)
years = daily_ret.index.year
rows = []
for year in sorted(years.unique()):
mask = years == year
yr_ret = (1 + daily_ret[mask]).prod() - 1
# Also compute max drawdown for the year
eq_yr = equity[mask]
running_max = eq_yr.cummax()
dd = (eq_yr - running_max) / running_max
rows.append({
"year": year,
"return": yr_ret,
"max_dd": dd.min(),
"start_val": float(eq_yr.iloc[0]),
"end_val": float(eq_yr.iloc[-1]),
})
return pd.DataFrame(rows).set_index("year")
def run(market: str, years_list: list[int]):
config = UNIVERSES[market]
benchmark = config["benchmark"]
print(f"Loading {market.upper()} price data...")
prices = data_manager.load(market)
bench_prices = prices[benchmark] if benchmark in prices.columns else None
stocks = prices.drop(columns=[benchmark], errors="ignore")
if market == "us":
label = "Mom+Recovery+LowDBeta+STR"
factor_funcs = [factor_momentum_12_1, factor_recovery, factor_downside_beta_proxy, factor_short_term_reversal]
weights = [0.25, 0.25, 0.25, 0.25]
baseline_label = "Recovery+Mom (baseline)"
baseline_funcs = [factor_momentum_12_1, factor_recovery]
baseline_weights = [0.5, 0.5]
else:
label = "Mom+Near52wHigh+VolReversal"
factor_funcs = [factor_momentum_12_1, factor_52w_high_distance, factor_turnover_reversal]
weights = [0.40, 0.30, 0.30]
baseline_label = "Mom+Recovery (baseline)"
baseline_funcs = [factor_momentum_12_1, factor_recovery]
baseline_weights = [0.5, 0.5]
for top_n in [10]:
print(f"\n{'='*90}")
print(f" {market.upper()} — Top {top_n} — {label}")
print(f"{'='*90}")
# Best combo
sig = build_strategy_signals(stocks, factor_funcs, weights, top_n=top_n)
eq = backtest_equity(sig, stocks)
# Baseline
sig_base = build_strategy_signals(stocks, baseline_funcs, baseline_weights, top_n=top_n)
eq_base = backtest_equity(sig_base, stocks)
# Benchmark
if bench_prices is not None:
bp = bench_prices.dropna()
eq_bench = bp / bp.iloc[0] * 100000
for n_years in years_list:
cutoff = stocks.index[-1] - pd.DateOffset(years=n_years)
eq_slice = eq[eq.index >= cutoff]
eq_base_slice = eq_base[eq_base.index >= cutoff]
if len(eq_slice) < 50:
continue
# Normalize to starting capital
eq_norm = eq_slice / eq_slice.iloc[0] * 100000
eq_base_norm = eq_base_slice / eq_base_slice.iloc[0] * 100000
yr = yearly_returns(eq_norm)
yr_base = yearly_returns(eq_base_norm)
if bench_prices is not None:
eq_bench_slice = eq_bench[eq_bench.index >= cutoff]
eq_bench_norm = eq_bench_slice / eq_bench_slice.iloc[0] * 100000
yr_bench = yearly_returns(eq_bench_norm)
print(f"\n--- Last {n_years} Years (from {eq_slice.index[0].date()}) ---\n")
# Combined table
print(f" {'Year':<6} | {label:>30} | {baseline_label:>25} | {'Benchmark':>12} | {'Alpha vs Bench':>14}")
print(f" {'-'*6}-+-{'-'*30}-+-{'-'*25}-+-{'-'*12}-+-{'-'*14}")
all_years = sorted(yr.index.tolist())
total_new = 1.0
total_base = 1.0
total_bench = 1.0
for y in all_years:
r_new = yr.loc[y, "return"] if y in yr.index else 0
dd_new = yr.loc[y, "max_dd"] if y in yr.index else 0
r_base = yr_base.loc[y, "return"] if y in yr_base.index else 0
r_bench = yr_bench.loc[y, "return"] if bench_prices is not None and y in yr_bench.index else 0
alpha = r_new - r_bench
total_new *= (1 + r_new)
total_base *= (1 + r_base)
total_bench *= (1 + r_bench)
print(f" {y:<6} | {r_new:>+14.2%} (dd {dd_new:>+7.2%}) | {r_base:>+25.2%} | {r_bench:>+12.2%} | {alpha:>+14.2%}")
total_r_new = total_new - 1
total_r_base = total_base - 1
total_r_bench = total_bench - 1
cagr_new = (total_new ** (1 / n_years)) - 1
cagr_base = (total_base ** (1 / n_years)) - 1
cagr_bench = (total_bench ** (1 / n_years)) - 1
print(f" {'-'*6}-+-{'-'*30}-+-{'-'*25}-+-{'-'*12}-+-{'-'*14}")
print(f" {'Total':<6} | {total_r_new:>+14.2%}{' '*16} | {total_r_base:>+25.2%} | {total_r_bench:>+12.2%} |")
print(f" {'CAGR':<6} | {cagr_new:>+14.2%}{' '*16} | {cagr_base:>+25.2%} | {cagr_bench:>+12.2%} |")
# Full period metrics
print(f"\n Full metrics ({label}):")
daily_ret = eq_norm.pct_change().dropna()
sharpe = daily_ret.mean() / daily_ret.std() * np.sqrt(252) if daily_ret.std() > 0 else 0
running_max = eq_norm.cummax()
max_dd = ((eq_norm - running_max) / running_max).min()
print(f" Sharpe: {sharpe:.2f} | Max Drawdown: {max_dd:.2%} | Win Rate: {(daily_ret > 0).mean():.2%}")
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--market", default="us", choices=["us", "cn"])
args = parser.parse_args()
run(args.market, years_list=[3, 5, 10])
if __name__ == "__main__":
main()

324
factor_deep_analysis.py
View File

@@ -1,324 +0,0 @@
"""
Deep factor analysis — orthogonality, proper correlations, residual alpha.
For the top factor candidates identified in factor_research.py, this script:
1. Computes proper daily cross-sectional rank correlations between factors
2. Tests residual IC after neutralizing known factors (momentum, recovery)
3. Runs sub-period breakdown (2-year windows)
4. Tests factor combinations
"""
from __future__ import annotations
import argparse
import warnings
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from factor_research import (
factor_momentum_12_1,
factor_recovery,
factor_inverse_vol,
factor_short_term_reversal,
factor_idio_vol_change,
factor_max_drawdown_recovery,
factor_mean_reversion_residual,
factor_skewness,
factor_high_low_range as factor_range_compression,
factor_52w_high_distance as factor_near_52w_high,
factor_downside_beta_proxy as factor_low_downside_beta,
factor_lottery_demand,
factor_turnover_reversal,
factor_gap_momentum as factor_smooth_momentum,
factor_up_down_vol_ratio,
factor_trend_strength,
factor_consecutive_up_days,
factor_volume_price_divergence,
factor_recovery_acceleration,
factor_relative_volume_momentum,
factor_price_level,
factor_liquidity_premium,
compute_ic,
)
warnings.filterwarnings("ignore", category=FutureWarning)
def daily_cross_sectional_correlation(
sig_a: pd.DataFrame, sig_b: pd.DataFrame
) -> pd.Series:
"""Daily cross-sectional Spearman correlation between two factor signals."""
common_idx = sig_a.index.intersection(sig_b.index)
common_cols = sig_a.columns.intersection(sig_b.columns)
a = sig_a.loc[common_idx, common_cols]
b = sig_b.loc[common_idx, common_cols]
corrs = {}
for date in common_idx:
va = a.loc[date].dropna()
vb = b.loc[date].dropna()
common = va.index.intersection(vb.index)
if len(common) < 30:
continue
c = va[common].corr(vb[common], method="spearman")
if np.isfinite(c):
corrs[date] = c
return pd.Series(corrs)
def proper_factor_correlation_matrix(factors: dict[str, pd.DataFrame]) -> pd.DataFrame:
"""Compute average daily cross-sectional Spearman correlations."""
names = list(factors.keys())
n = len(names)
matrix = pd.DataFrame(1.0, index=names, columns=names)
for i in range(n):
for j in range(i + 1, n):
corr_series = daily_cross_sectional_correlation(factors[names[i]], factors[names[j]])
avg_corr = corr_series.mean() if len(corr_series) > 0 else np.nan
matrix.loc[names[i], names[j]] = avg_corr
matrix.loc[names[j], names[i]] = avg_corr
return matrix
def residual_signal(
target: pd.DataFrame,
controls: list[pd.DataFrame],
) -> pd.DataFrame:
"""Cross-sectionally orthogonalize target signal against control signals.
For each day, regress target ranks on control ranks, return residual."""
ranked_target = target.rank(axis=1, pct=True, na_option="keep")
ranked_controls = [c.rank(axis=1, pct=True, na_option="keep") for c in controls]
residuals = pd.DataFrame(index=target.index, columns=target.columns, dtype=float)
for date in target.index:
y = ranked_target.loc[date].dropna()
xs = [rc.loc[date].reindex(y.index) for rc in ranked_controls if date in rc.index]
if not xs:
residuals.loc[date] = y
continue
x_df = pd.concat(xs, axis=1).dropna()
common = y.index.intersection(x_df.index)
if len(common) < 30:
continue
y_c = y[common].values
x_c = x_df.loc[common].values
x_c = np.column_stack([np.ones(len(common)), x_c])
try:
coef, _, _, _ = np.linalg.lstsq(x_c, y_c, rcond=None)
resid = y_c - x_c @ coef
residuals.loc[date, common] = resid
except np.linalg.LinAlgError:
residuals.loc[date, common] = y[common].values
return residuals
def subperiod_ic(signal: pd.DataFrame, prices: pd.DataFrame, horizon: int = 5, window_years: int = 2):
"""Compute IC for each rolling sub-period."""
fwd_ret = prices.pct_change(horizon).shift(-horizon)
ic_series = compute_ic(signal, fwd_ret)
if len(ic_series) == 0:
return pd.DataFrame()
window = 252 * window_years
results = []
start = ic_series.index[0]
while start < ic_series.index[-1]:
end = start + pd.DateOffset(years=window_years)
subset = ic_series[(ic_series.index >= start) & (ic_series.index < end)]
if len(subset) > 100:
results.append({
"period": f"{start.year}-{end.year}",
"ic_mean": subset.mean(),
"ic_std": subset.std(),
"icir": subset.mean() / subset.std() if subset.std() > 0 else 0,
"pct_positive": (subset > 0).mean(),
"n_days": len(subset),
})
start = end
return pd.DataFrame(results)
def test_factor_combination(
factors: dict[str, pd.DataFrame],
factor_names: list[str],
weights: list[float],
prices: pd.DataFrame,
label: str,
):
"""Test a weighted combination of factors."""
ranked = [factors[n].rank(axis=1, pct=True, na_option="keep") for n in factor_names]
combo = sum(w * r for w, r in zip(weights, ranked))
fwd_5d = prices.pct_change(5).shift(-5)
ic_series = compute_ic(combo, fwd_5d)
if len(ic_series) == 0:
return None
return {
"combo": label,
"ic_5d": ic_series.mean(),
"icir_5d": ic_series.mean() / ic_series.std() if ic_series.std() > 0 else 0,
"ic_stab": (ic_series.rolling(252).mean().dropna() > 0).mean() if len(ic_series) > 252 else np.nan,
}
def run_analysis(market: str):
config = UNIVERSES[market]
benchmark = config["benchmark"]
print(f"Loading {market.upper()} price data...")
prices = data_manager.load(market)
stocks = prices.drop(columns=[benchmark], errors="ignore")
print(f"Universe: {stocks.shape[1]} stocks, {stocks.shape[0]} days")
# Build factors
print("Computing factors...")
factors = {}
factors["momentum_12_1"] = factor_momentum_12_1(stocks)
factors["recovery"] = factor_recovery(stocks)
factors["inverse_vol"] = factor_inverse_vol(stocks)
factors["short_term_reversal"] = factor_short_term_reversal(stocks)
factors["drawdown_recovery"] = factor_max_drawdown_recovery(stocks)
factors["mean_rev_zscore"] = factor_mean_reversion_residual(stocks)
factors["neg_skewness"] = factor_skewness(stocks)
factors["near_52w_high"] = factor_near_52w_high(stocks)
factors["low_downside_beta"] = factor_low_downside_beta(stocks)
factors["smooth_momentum"] = factor_smooth_momentum(stocks)
factors["recovery_accel"] = factor_recovery_acceleration(stocks)
factors["range_compression"] = factor_range_compression(stocks)
if market == "cn":
factors["anti_lottery"] = factor_lottery_demand(stocks)
factors["vol_reversal"] = factor_turnover_reversal(stocks)
factors["low_price"] = factor_price_level(stocks)
factors["illiquidity"] = factor_liquidity_premium(stocks)
# ---- 1. Proper Cross-Sectional Correlation Matrix ----
print("\n" + "=" * 90)
print(f" 1. CROSS-SECTIONAL FACTOR CORRELATIONS — {market.upper()}")
print("=" * 90)
print("(Average daily Spearman correlation between factor ranks)\n")
corr = proper_factor_correlation_matrix(factors)
print(corr.round(3).to_string())
# ---- 2. Residual IC after neutralizing known factors ----
print("\n" + "=" * 90)
print(f" 2. RESIDUAL IC AFTER NEUTRALIZING KNOWN FACTORS — {market.upper()}")
print("=" * 90)
print("(IC of factor after cross-sectionally regressing out momentum + recovery)\n")
known = [factors["momentum_12_1"], factors["recovery"]]
fwd_5d = stocks.pct_change(5).shift(-5)
new_candidates = [k for k in factors if k not in ("momentum_12_1", "recovery", "inverse_vol")]
rows = []
for name in new_candidates:
resid = residual_signal(factors[name], known)
ic_series = compute_ic(resid, fwd_5d)
if len(ic_series) > 0:
rows.append({
"factor": name,
"raw_ic_5d": compute_ic(factors[name], fwd_5d).mean(),
"residual_ic_5d": ic_series.mean(),
"residual_icir_5d": ic_series.mean() / ic_series.std() if ic_series.std() > 0 else 0,
"pct_pos": (ic_series > 0).mean(),
})
resid_df = pd.DataFrame(rows).set_index("factor").sort_values("residual_icir_5d", ascending=False)
print(resid_df.round(4).to_string())
# ---- 3. Sub-Period Stability ----
print("\n" + "=" * 90)
print(f" 3. SUB-PERIOD IC STABILITY (2-year windows, 5-day horizon) — {market.upper()}")
print("=" * 90)
# Test top factors
if market == "us":
top_factors = ["low_downside_beta", "drawdown_recovery", "mean_rev_zscore", "short_term_reversal", "momentum_12_1"]
else:
top_factors = ["momentum_12_1", "anti_lottery", "inverse_vol", "vol_reversal", "near_52w_high"]
for name in top_factors:
if name not in factors:
continue
print(f"\n {name}:")
sp = subperiod_ic(factors[name], stocks, horizon=5)
if not sp.empty:
print(sp.to_string(index=False))
else:
print(" (insufficient data)")
# ---- 4. Factor Combinations ----
print("\n" + "=" * 90)
print(f" 4. FACTOR COMBINATIONS — {market.upper()}")
print("=" * 90)
print("(Testing multi-factor composites)\n")
combos = []
if market == "us":
tests = [
(["momentum_12_1", "low_downside_beta"], [0.5, 0.5], "mom+low_dbeta"),
(["momentum_12_1", "drawdown_recovery"], [0.5, 0.5], "mom+dd_recovery"),
(["momentum_12_1", "mean_rev_zscore"], [0.5, 0.5], "mom+mean_rev"),
(["momentum_12_1", "short_term_reversal"], [0.5, 0.5], "mom+STR"),
(["recovery", "low_downside_beta"], [0.5, 0.5], "recovery+low_dbeta"),
(["momentum_12_1", "recovery", "low_downside_beta"], [0.33, 0.33, 0.34], "mom+rec+low_dbeta"),
(["momentum_12_1", "recovery", "drawdown_recovery"], [0.33, 0.33, 0.34], "mom+rec+dd_rec"),
(["momentum_12_1", "recovery", "short_term_reversal"], [0.33, 0.33, 0.34], "mom+rec+STR"),
(["momentum_12_1", "recovery", "mean_rev_zscore"], [0.33, 0.33, 0.34], "mom+rec+meanrev"),
(["momentum_12_1", "recovery", "low_downside_beta", "short_term_reversal"],
[0.25, 0.25, 0.25, 0.25], "mom+rec+dbeta+STR"),
(["momentum_12_1", "recovery", "drawdown_recovery", "mean_rev_zscore"],
[0.25, 0.25, 0.25, 0.25], "mom+rec+ddrec+meanrev"),
]
else: # cn
tests = [
(["momentum_12_1", "anti_lottery"], [0.5, 0.5], "mom+anti_lottery"),
(["momentum_12_1", "inverse_vol"], [0.5, 0.5], "mom+inv_vol"),
(["momentum_12_1", "vol_reversal"], [0.5, 0.5], "mom+vol_reversal"),
(["momentum_12_1", "near_52w_high"], [0.5, 0.5], "mom+near52wh"),
(["momentum_12_1", "anti_lottery", "inverse_vol"], [0.33, 0.33, 0.34], "mom+alot+invvol"),
(["momentum_12_1", "anti_lottery", "vol_reversal"], [0.33, 0.33, 0.34], "mom+alot+volrev"),
(["momentum_12_1", "anti_lottery", "near_52w_high"], [0.33, 0.33, 0.34], "mom+alot+near52w"),
(["momentum_12_1", "recovery", "anti_lottery"], [0.33, 0.33, 0.34], "mom+rec+alot"),
(["momentum_12_1", "anti_lottery", "inverse_vol", "vol_reversal"],
[0.25, 0.25, 0.25, 0.25], "mom+alot+invvol+volrev"),
(["momentum_12_1", "anti_lottery", "near_52w_high", "vol_reversal"],
[0.25, 0.25, 0.25, 0.25], "mom+alot+52wh+volrev"),
]
# Also test the existing recovery+momentum baseline
baseline = test_factor_combination(factors, ["momentum_12_1", "recovery"], [0.5, 0.5], stocks, "BASELINE: mom+recovery")
if baseline:
combos.append(baseline)
for names, weights, label in tests:
if all(n in factors for n in names):
result = test_factor_combination(factors, names, weights, stocks, label)
if result:
combos.append(result)
combo_df = pd.DataFrame(combos).set_index("combo").sort_values("icir_5d", ascending=False)
print(combo_df.round(4).to_string())
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--market", default="us", choices=["us", "cn"])
args = parser.parse_args()
run_analysis(args.market)
if __name__ == "__main__":
main()

150
factor_final_check.py
View File

@@ -1,150 +0,0 @@
"""Final robustness check on champions from the discovery loop."""
from __future__ import annotations
import warnings
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from factor_loop import (
strat, bt, stats, combo, yearly,
f_rec_mom, f_rec_126, f_rec_63,
f_mom_12_1, f_mom_6_1, f_mom_intermediate,
f_above_ma200, f_golden_cross,
f_up_volume_proxy, f_gap_up_freq,
f_rec_mom_filtered, f_down_resilience,
f_up_capture, f_52w_high, f_str_10d,
f_earnings_drift, f_reversal_vol,
)
warnings.filterwarnings("ignore")
def f_quality_mom(p):
mom = f_mom_12_1(p)
consist_ret = p.pct_change()
consist = (consist_ret > 0).astype(float).rolling(252, min_periods=126).mean()
mom_r = mom.rank(axis=1, pct=True, na_option="keep")
con_r = consist.rank(axis=1, pct=True, na_option="keep")
up_r = f_up_volume_proxy(p).rank(axis=1, pct=True, na_option="keep")
return 0.4 * mom_r + 0.3 * con_r + 0.3 * up_r
def f_mom_x_gap(p):
mom_r = f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep")
gap_r = f_gap_up_freq(p).rank(axis=1, pct=True, na_option="keep")
return mom_r * gap_r
def rolling_2yr(eq):
dr = eq.pct_change().dropna()
results = []
for end_i in range(504, len(dr), 63):
chunk = dr.iloc[end_i - 504:end_i]
tot = (1 + chunk).prod() - 1
ann = (1 + tot) ** (252 / len(chunk)) - 1
sh = chunk.mean() / chunk.std() * np.sqrt(252) if chunk.std() > 0 else 0
results.append({"end": chunk.index[-1].date(), "ann": ann, "sh": sh})
return pd.DataFrame(results)
def run_robustness(name, func, prices, label_prefix):
print(f"\n {name}:")
# Top-N sensitivity
print(f" Top-N: ", end="")
for n in [5, 10, 15, 20]:
w = strat(prices, func, top_n=n)
eq = bt(w, prices)
s = stats(eq)
print(f"N={n}: {s['cagr']:+.1%}/{s['sharpe']:.2f} ", end="")
print()
# Rebal sensitivity
print(f" Rebal: ", end="")
for r in [5, 10, 21, 42]:
w = strat(prices, func, top_n=10, rebal=r)
eq = bt(w, prices)
s = stats(eq)
print(f"{r}d: {s['cagr']:+.1%}/{s['sharpe']:.2f} ", end="")
print()
# Cost sensitivity
print(f" Cost: ", end="")
for c in [0, 0.001, 0.002, 0.005]:
w = strat(prices, func, top_n=10)
eq = bt(w, prices, cost=c)
s = stats(eq)
print(f"{c*1e4:.0f}bp: {s['cagr']:+.1%} ", end="")
print()
# Rolling 2-year
w = strat(prices, func, top_n=10)
eq = bt(w, prices)
roll = rolling_2yr(eq)
if not roll.empty:
pct_pos = (roll["ann"] > 0).mean()
print(f" 2yr rolling: mean={roll['ann'].mean():+.1%} min={roll['ann'].min():+.1%} "
f"max={roll['ann'].max():+.1%} %pos={pct_pos:.0%} mean_sharpe={roll['sh'].mean():.2f}")
def main():
# ============= US =============
prices_us = data_manager.load("us")
stocks_us = prices_us.drop(columns=["SPY"], errors="ignore")
print("=" * 95)
print(" US FINAL ROBUSTNESS — Champions vs Baseline")
print("=" * 95)
us_champs = [
("BASELINE: rec+mom", f_rec_mom),
("rec_mom_filtered+rec_deep×upvol",
combo([(f_rec_mom_filtered, 0.5),
combo([(f_rec_126, 0.5), (f_up_volume_proxy, 0.5)]), (lambda x: x, 0.0)])), # hack
("above_ma200+mom_7m+rec_126d",
combo([(f_above_ma200, 0.33), (f_mom_intermediate, 0.33), (f_rec_126, 0.34)])),
("rec_mom_filtered+above_ma200",
combo([(f_rec_mom_filtered, 0.5), (f_above_ma200, 0.5)])),
("mom_7m+rec_126d",
combo([(f_mom_intermediate, 0.5), (f_rec_126, 0.5)])),
]
# Fix the first champion - need proper 2-factor combo
us_champs[1] = (
"rec_mom_filt + rec_deep×upvol",
combo([
(f_rec_mom_filtered, 0.5),
(combo([(f_rec_126, 0.5), (f_up_volume_proxy, 0.5)]), 0.5),
])
)
for name, func in us_champs:
run_robustness(name, func, stocks_us, "US")
# ============= CN =============
prices_cn = data_manager.load("cn")
stocks_cn = prices_cn.drop(columns=["000300.SS"], errors="ignore")
print(f"\n{'='*95}")
print(" CN FINAL ROBUSTNESS — Champions vs Baseline")
print("=" * 95)
cn_champs = [
("BASELINE: rec+mom", f_rec_mom),
("up_capture+quality_mom",
combo([(f_up_capture, 0.5), (f_quality_mom, 0.5)])),
("recovery_63d+mom×gap",
combo([(f_rec_63, 0.5), (f_mom_x_gap, 0.5)])),
("down_resilience+quality_mom",
combo([(f_down_resilience, 0.5), (f_quality_mom, 0.5)])),
("up_capture+mom×gap",
combo([(f_up_capture, 0.5), (f_mom_x_gap, 0.5)])),
]
for name, func in cn_champs:
run_robustness(name, func, stocks_cn, "CN")
if __name__ == "__main__":
main()

654
factor_loop.py
View File

@@ -1,654 +0,0 @@
"""
Iterative Factor Discovery Loop.
Round 1: Academic & practitioner hypotheses (30+ factors)
Round 2: Data-driven variations on Round 1 winners
Round 3: Interaction and conditional factors
Round 4: Parameter optimization on finalists
Round 5: Best combinations
Each factor is tested immediately as a top-10 equal-weight strategy
with monthly rebalancing and 10bps transaction costs.
"""
from __future__ import annotations
import argparse
import warnings
from typing import Callable
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
warnings.filterwarnings("ignore")
FactorFunc = Callable[[pd.DataFrame], pd.DataFrame]
# ---------------------------------------------------------------------------
# Backtest infrastructure
# ---------------------------------------------------------------------------
def strat(
prices: pd.DataFrame,
signal_func: FactorFunc,
top_n: int = 10,
rebal: int = 21,
warmup: int = 252,
) -> pd.DataFrame:
sig = signal_func(prices)
rank = sig.rank(axis=1, ascending=False, na_option="bottom")
n_valid = sig.notna().sum(axis=1)
enough = n_valid >= top_n
mask = (rank <= top_n) & enough.values.reshape(-1, 1)
raw = mask.astype(float)
w = raw.div(raw.sum(axis=1).replace(0, np.nan), axis=0).fillna(0.0)
rmask = pd.Series(False, index=prices.index)
rmask.iloc[list(range(warmup, len(prices), rebal))] = True
w[~rmask] = np.nan
w = w.ffill().fillna(0.0)
w.iloc[:warmup] = 0.0
return w.shift(1).fillna(0.0)
def bt(weights: pd.DataFrame, prices: pd.DataFrame, cost: float = 0.001) -> pd.Series:
ret = prices.pct_change().fillna(0.0)
pr = (weights * ret).sum(axis=1)
pr -= weights.diff().abs().sum(axis=1) * cost
return (1 + pr).cumprod() * 100000
def stats(eq: pd.Series) -> dict:
dr = eq.pct_change().dropna()
if len(dr) < 200 or dr.std() == 0:
return {"cagr": np.nan, "sharpe": np.nan, "sortino": np.nan,
"maxdd": np.nan, "calmar": np.nan}
ny = len(dr) / 252
tot = eq.iloc[-1] / eq.iloc[0] - 1
cagr = (1 + tot) ** (1 / ny) - 1
sh = dr.mean() / dr.std() * np.sqrt(252)
sd = dr[dr < 0].std()
so = dr.mean() / sd * np.sqrt(252) if sd > 0 else 0
rm = eq.cummax()
dd = ((eq - rm) / rm).min()
cal = cagr / abs(dd) if dd != 0 else 0
return {"cagr": cagr, "sharpe": sh, "sortino": so, "maxdd": dd, "calmar": cal}
def yearly(eq: pd.Series) -> dict[int, float]:
dr = eq.pct_change().fillna(0)
return {y: float((1 + dr[dr.index.year == y]).prod() - 1) for y in sorted(dr.index.year.unique())}
def test_factor(name: str, func: FactorFunc, prices: pd.DataFrame,
top_n: int = 10) -> dict:
w = strat(prices, func, top_n=top_n)
eq = bt(w, prices)
s = stats(eq)
s["name"] = name
s["equity"] = eq
return s
def combo(fws: list[tuple[FactorFunc, float]]) -> FactorFunc:
def _c(p):
return sum(w * f(p).rank(axis=1, pct=True, na_option="keep") for f, w in fws)
return _c
def print_results(results: list[dict], title: str):
df = pd.DataFrame([{k: v for k, v in r.items() if k != "equity"} for r in results])
df = df.set_index("name").sort_values("cagr", ascending=False)
print(f"\n{'='*95}")
print(f" {title}")
print(f"{'='*95}")
print(f" {'Factor':<45} {'CAGR':>7} {'Sharpe':>7} {'Sortino':>8} {'MaxDD':>7} {'Calmar':>7}")
print(f" {'-'*85}")
for name, row in df.iterrows():
flag = " <<<" if "BASELINE" in str(name) else ""
c = row['cagr']
if np.isnan(c):
continue
print(f" {str(name):<45} {c:>+6.1%} {row['sharpe']:>7.2f} {row['sortino']:>8.2f} "
f"{row['maxdd']:>+6.1%} {row['calmar']:>7.2f}{flag}")
return df
# =====================================================================
# ROUND 1 — Academic & Practitioner Hypotheses
# =====================================================================
# --- Momentum family ---
def f_mom_12_1(p): return p.shift(21).pct_change(231)
def f_mom_6_1(p): return p.shift(21).pct_change(105)
def f_mom_3_1(p): return p.shift(21).pct_change(42)
def f_mom_1_0(p): return p.pct_change(21) # 1-month (reversal in US)
# --- Recovery family ---
def f_rec_63(p): return p / p.rolling(63, min_periods=63).min() - 1
def f_rec_126(p): return p / p.rolling(126, min_periods=126).min() - 1
def f_rec_21(p): return p / p.rolling(21, min_periods=21).min() - 1
# Novy-Marx 2012: intermediate momentum (7-12 month)
def f_mom_intermediate(p): return p.shift(21).pct_change(147) # ~7 month
# Asness et al: quality/profitability proxy via return consistency
def f_consistent_returns(p):
ret = p.pct_change()
return (ret > 0).astype(float).rolling(252, min_periods=126).mean()
# Da, Liu, Schaumburg 2014: information discreteness
# Stocks with many small positive days > stocks with few large positive days
def f_info_discrete(p):
ret = p.pct_change()
n_pos = (ret > 0).astype(float).rolling(60, min_periods=40).sum()
sum_pos = ret.where(ret > 0, 0).rolling(60, min_periods=40).sum()
avg_pos = sum_pos / n_pos.replace(0, np.nan)
# High count of positive days + low average positive = smooth accumulation
return n_pos / avg_pos.replace(0, np.nan)
# Accumulation proxy (worked in Round 1)
def f_up_volume_proxy(p):
ret = p.pct_change()
return ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
# George & Hwang 2004: 52-week high ratio
def f_52w_high(p):
return p / p.rolling(252, min_periods=126).max()
# Frequency of large up-moves (worked in Round 1)
def f_gap_up_freq(p):
ret = p.pct_change()
return (ret > 0.01).astype(float).rolling(60, min_periods=40).mean()
# Bali, Cakici, Whitelaw 2011: MAX effect (lottery demand)
def f_anti_max(p):
ret = p.pct_change()
return -ret.rolling(20, min_periods=15).max()
# Ang et al 2006: idiosyncratic volatility (negative)
def f_neg_ivol(p):
ret = p.pct_change()
return -ret.rolling(20, min_periods=15).std()
# Blitz & van Vliet 2007: low volatility anomaly
def f_low_vol_60(p):
ret = p.pct_change()
return -ret.rolling(60, min_periods=40).std()
# Hurst exponent proxy — autocorrelation of returns
# Stocks with positive autocorrelation = trending
def f_autocorrelation(p):
ret = p.pct_change()
def _ac(x):
x = x.dropna()
if len(x) < 20:
return np.nan
return np.corrcoef(x[:-1], x[1:])[0, 1]
return ret.rolling(60, min_periods=40).apply(_ac, raw=False)
# Short-term reversal (Jegadeesh 1990)
def f_str_5d(p): return -p.pct_change(5)
def f_str_10d(p): return -p.pct_change(10)
# Earnings drift proxy (worked in Round 1)
def f_earnings_drift(p):
ret_5d = p.pct_change(5)
vol = p.pct_change().rolling(60, min_periods=40).std() * np.sqrt(5)
z = ret_5d / vol.replace(0, np.nan)
return z.rolling(60, min_periods=20).mean()
# Risk-adjusted momentum (Sharpe-momentum)
def f_sharpe_mom(p):
ret = p.pct_change()
mu = ret.rolling(252, min_periods=126).mean()
sigma = ret.rolling(252, min_periods=126).std()
return mu / sigma.replace(0, np.nan)
# Trend strength: slope of log-price regression
def f_trend_slope(p):
log_p = np.log(p.replace(0, np.nan))
def _slope(x):
x = x.dropna().values
if len(x) < 30:
return np.nan
t = np.arange(len(x), dtype=float)
t -= t.mean()
return (t * (x - x.mean())).sum() / (t * t).sum()
return log_p.rolling(60, min_periods=30).apply(_slope, raw=False)
# Acceleration: recent momentum vs. longer-term momentum
def f_mom_accel(p):
m3 = p.shift(5).pct_change(58) # ~3mo
m12 = p.shift(21).pct_change(231) # ~12mo
return m3 - m12
# Mean reversion z-score
def f_mean_rev_z(p):
ma20 = p.rolling(20, min_periods=20).mean()
vol = p.pct_change().rolling(60, min_periods=40).std() * p
return -(p - ma20) / vol.replace(0, np.nan)
# Price relative to moving averages
def f_above_ma200(p):
return p / p.rolling(200, min_periods=200).mean() - 1
def f_above_ma50(p):
return p / p.rolling(50, min_periods=50).mean() - 1
# Dual MA signal: 50-day MA / 200-day MA
def f_golden_cross(p):
ma50 = p.rolling(50, min_periods=50).mean()
ma200 = p.rolling(200, min_periods=200).mean()
return ma50 / ma200 - 1
# Drawdown recovery rate
def f_dd_recovery_rate(p):
rm = p.rolling(252, min_periods=126).max()
dd = p / rm - 1 # negative when in drawdown
return dd - dd.shift(20) # positive = recovering from drawdown
# A-share specific: short-term reversal x volatility
def f_reversal_vol(p):
return -p.pct_change(5) * p.pct_change().rolling(20, min_periods=15).std()
# Recovery + momentum (baseline)
def f_rec_mom(p):
r1 = f_rec_63(p).rank(axis=1, pct=True, na_option="keep")
r2 = f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep")
return 0.5 * r1 + 0.5 * r2
# =====================================================================
# ROUND 2 — Second-order ideas from Round 1 analysis
# =====================================================================
# The key insight: "quality of returns" matters more than "magnitude of returns"
# Factors that measure HOW a stock goes up, not just that it went up.
# Smoothness-weighted momentum
def f_smooth_momentum(p):
"""Momentum penalized by path volatility. Stocks that go up smoothly."""
mom = p.shift(21).pct_change(231)
ret = p.pct_change()
vol = ret.rolling(252, min_periods=126).std()
return mom / (vol.replace(0, np.nan) ** 0.5) # sqrt to dampen
# Positive return ratio (like Sharpe numerator)
def f_pos_ratio_60(p):
"""Fraction of positive return days in 60 days. Quality signal."""
ret = p.pct_change()
return (ret > 0).astype(float).rolling(60, min_periods=40).mean()
# Cumulative positive returns vs cumulative negative returns
def f_up_down_asymmetry(p):
"""Ratio of cumulative up-move to cumulative down-move."""
ret = p.pct_change()
up = ret.where(ret > 0, 0).rolling(60, min_periods=40).sum()
down = (-ret.where(ret < 0, 0)).rolling(60, min_periods=40).sum()
return up / down.replace(0, np.nan)
# Streak momentum: max consecutive up days in last 40 days
def f_max_streak(p):
ret = p.pct_change()
pos = (ret > 0).astype(float)
def _max_streak(x):
x = x.dropna().values
if len(x) == 0:
return 0
best = cur = 0
for v in x:
cur = cur + 1 if v > 0.5 else 0
best = max(best, cur)
return best
return pos.rolling(40, min_periods=20).apply(_max_streak, raw=False)
# Overnight proxy: gap between yesterday's close and today's pattern
# Since we only have close prices, use close-to-close 1d return decomposition
def f_up_capture(p):
"""Up-market capture ratio over 60 days."""
ret = p.pct_change()
mkt = ret.mean(axis=1)
up_mkt = mkt > 0
arr = ret.values.copy()
arr[~up_mkt.values, :] = np.nan
stock_up = pd.DataFrame(arr, index=ret.index, columns=ret.columns)
mkt_up_vals = mkt.where(up_mkt, np.nan)
stock_avg = stock_up.rolling(60, min_periods=20).mean()
mkt_avg = mkt_up_vals.rolling(60, min_periods=20).mean()
return stock_avg.div(mkt_avg, axis=0)
# Down-market resilience
def f_down_resilience(p):
"""How much LESS a stock falls on down-market days."""
ret = p.pct_change()
mkt = ret.mean(axis=1)
down_mkt = mkt < 0
arr = ret.values.copy()
arr[~down_mkt.values, :] = np.nan
down_ret = pd.DataFrame(arr, index=ret.index, columns=ret.columns)
return -down_ret.rolling(120, min_periods=30).mean()
# Recovery from rolling max with momentum filter
def f_rec_mom_filtered(p):
"""Recovery factor only for stocks with positive 6-month momentum.
Filters out dead-cat bounces."""
rec = p / p.rolling(126, min_periods=126).min() - 1
mom = p.shift(21).pct_change(105)
return rec.where(mom > 0, np.nan)
# Information discreteness v2: using the sign ratio
def f_sign_ratio(p):
"""Ratio of (count of positive days)^2 * avg_size to total return.
High ratio = many small ups = institutional flow."""
ret = p.pct_change()
n_total = 60
n_pos = (ret > 0).astype(float).rolling(n_total, min_periods=40).sum()
total_ret = ret.rolling(n_total, min_periods=40).sum()
sign_vol = n_pos / n_total
# Stocks where most of the return comes from many small positive days
return sign_vol * total_ret.clip(lower=0)
# =====================================================================
# ROUND 3 — Interaction & conditional factors
# =====================================================================
def f_mom_x_recovery(p):
"""Momentum × Recovery interaction. The product, not the sum."""
mom_r = f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep")
rec_r = f_rec_63(p).rank(axis=1, pct=True, na_option="keep")
return mom_r * rec_r
def f_mom_x_upvol(p):
"""Momentum × Up-volume-proxy interaction."""
mom_r = f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep")
up_r = f_up_volume_proxy(p).rank(axis=1, pct=True, na_option="keep")
return mom_r * up_r
def f_rec_deep_x_upvol(p):
"""Deep recovery × Up-volume interaction."""
rec_r = f_rec_126(p).rank(axis=1, pct=True, na_option="keep")
up_r = f_up_volume_proxy(p).rank(axis=1, pct=True, na_option="keep")
return rec_r * up_r
def f_trend_x_mom(p):
"""Trend strength × Momentum. Trending + momentum = double signal."""
tr_r = f_trend_slope(p).rank(axis=1, pct=True, na_option="keep")
mom_r = f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep")
return tr_r * mom_r
def f_quality_mom(p):
"""Momentum filtered by consistency. Only persistent winners."""
mom = f_mom_12_1(p)
consist = f_consistent_returns(p)
mom_r = mom.rank(axis=1, pct=True, na_option="keep")
con_r = consist.rank(axis=1, pct=True, na_option="keep")
return 0.4 * mom_r + 0.3 * con_r + 0.3 * f_up_volume_proxy(p).rank(axis=1, pct=True, na_option="keep")
def f_rec_deep_x_gap(p):
"""Deep recovery × gap-up frequency."""
rec_r = f_rec_126(p).rank(axis=1, pct=True, na_option="keep")
gap_r = f_gap_up_freq(p).rank(axis=1, pct=True, na_option="keep")
return rec_r * gap_r
def f_mom_x_gap(p):
"""Momentum × gap-up frequency."""
mom_r = f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep")
gap_r = f_gap_up_freq(p).rank(axis=1, pct=True, na_option="keep")
return mom_r * gap_r
# Regime-conditional: momentum with volatility filter
def f_mom_low_vol_regime(p):
"""Momentum only when market vol is below median.
Momentum crashes in high-vol regimes."""
mom = f_mom_12_1(p)
mkt_vol = p.pct_change().mean(axis=1).rolling(60).std()
vol_median = mkt_vol.rolling(252, min_periods=126).median()
low_vol = mkt_vol <= vol_median
mask = pd.DataFrame(
np.tile(low_vol.values[:, None], (1, mom.shape[1])),
index=mom.index, columns=mom.columns,
)
return mom.where(mask, 0)
# =====================================================================
# Main loop
# =====================================================================
def run_round(
name: str,
factors: list[tuple[str, FactorFunc]],
prices: pd.DataFrame,
top_n: int = 10,
) -> list[dict]:
results = []
for fname, func in factors:
r = test_factor(fname, func, prices, top_n=top_n)
results.append(r)
print_results(results, name)
return results
def run_market(market: str):
config = UNIVERSES[market]
benchmark = config["benchmark"]
prices = data_manager.load(market)
bench = prices[benchmark].dropna() if benchmark in prices.columns else None
stocks = prices.drop(columns=[benchmark], errors="ignore")
print(f"\n{'#'*95}")
print(f" FACTOR DISCOVERY LOOP — {market.upper()} MARKET")
print(f" {stocks.shape[1]} stocks, {stocks.shape[0]} days, "
f"{stocks.index[0].date()} → {stocks.index[-1].date()}")
print(f"{'#'*95}")
# Benchmark
if bench is not None:
eq_bench = bench / bench.iloc[0] * 100000
bs = stats(eq_bench)
print(f"\n Benchmark: CAGR {bs['cagr']:+.1%}, Sharpe {bs['sharpe']:.2f}")
# ================================================================
# ROUND 1: Academic & practitioner factors
# ================================================================
r1_factors = [
("BASELINE:rec+mom", f_rec_mom),
# Momentum family
("mom_12_1", f_mom_12_1),
("mom_6_1", f_mom_6_1),
("mom_3_1", f_mom_3_1),
("mom_1_0", f_mom_1_0),
("mom_intermediate_7m", f_mom_intermediate),
("sharpe_momentum", f_sharpe_mom),
# Recovery family
("recovery_63d", f_rec_63),
("recovery_126d", f_rec_126),
("recovery_21d", f_rec_21),
# Trend
("trend_slope_60d", f_trend_slope),
("golden_cross", f_golden_cross),
("above_ma200", f_above_ma200),
# Volatility
("low_vol_60d", f_low_vol_60),
("neg_ivol_20d", f_neg_ivol),
# Reversal
("STR_5d", f_str_5d),
("STR_10d", f_str_10d),
# Quality / accumulation
("consistent_returns", f_consistent_returns),
("up_volume_proxy", f_up_volume_proxy),
("gap_up_freq", f_gap_up_freq),
("info_discrete", f_info_discrete),
("earnings_drift", f_earnings_drift),
# Other academic
("52w_high", f_52w_high),
("anti_max_20d", f_anti_max),
("dd_recovery_rate", f_dd_recovery_rate),
("mom_acceleration", f_mom_accel),
]
if market == "cn":
r1_factors.append(("reversal_vol_cn", f_reversal_vol))
r1 = run_round("ROUND 1 — Academic & Practitioner Hypotheses", r1_factors, stocks)
# Identify top-10 from round 1
r1_sorted = sorted(r1, key=lambda x: x.get("cagr", 0) or 0, reverse=True)
r1_top_names = [r["name"] for r in r1_sorted[:10] if r.get("cagr") and r["cagr"] > 0]
baseline_cagr = next((r["cagr"] for r in r1 if "BASELINE" in r["name"]), 0)
print(f"\n Baseline CAGR: {baseline_cagr:+.1%}")
print(f" Top 10: {r1_top_names}")
# ================================================================
# ROUND 2: Second-order ideas based on what worked
# ================================================================
r2_factors = [
("BASELINE:rec+mom", f_rec_mom),
("smooth_momentum", f_smooth_momentum),
("pos_ratio_60d", f_pos_ratio_60),
("up_down_asymmetry", f_up_down_asymmetry),
("max_streak_40d", f_max_streak),
("up_capture_60d", f_up_capture),
("down_resilience_120d", f_down_resilience),
("rec_mom_filtered", f_rec_mom_filtered),
("sign_ratio", f_sign_ratio),
("autocorrelation_60d", f_autocorrelation),
("mean_rev_z", f_mean_rev_z),
]
r2 = run_round("ROUND 2 — Return Quality & Behavioral Factors", r2_factors, stocks)
# ================================================================
# ROUND 3: Interaction & conditional factors
# ================================================================
r3_factors = [
("BASELINE:rec+mom", f_rec_mom),
("mom×recovery", f_mom_x_recovery),
("mom×upvol", f_mom_x_upvol),
("rec_deep×upvol", f_rec_deep_x_upvol),
("trend×mom", f_trend_x_mom),
("quality_mom", f_quality_mom),
("rec_deep×gap", f_rec_deep_x_gap),
("mom×gap", f_mom_x_gap),
("mom_low_vol_regime", f_mom_low_vol_regime),
]
r3 = run_round("ROUND 3 — Interaction & Conditional Factors", r3_factors, stocks)
# ================================================================
# Collect ALL results from all rounds
# ================================================================
all_results = r1 + r2 + r3
# Deduplicate baseline
seen = set()
unique = []
for r in all_results:
if r["name"] not in seen:
seen.add(r["name"])
unique.append(r)
unique_sorted = sorted(unique, key=lambda x: x.get("cagr", 0) or 0, reverse=True)
print(f"\n{'='*95}")
print(f" ALL ROUNDS COMBINED — TOP 15 FACTORS — {market.upper()}")
print(f"{'='*95}")
print(f" {'Factor':<45} {'CAGR':>7} {'Sharpe':>7} {'Sortino':>8} {'MaxDD':>7} {'Calmar':>7}")
print(f" {'-'*85}")
for r in unique_sorted[:15]:
flag = " <<<" if "BASELINE" in r["name"] else ""
print(f" {r['name']:<45} {r['cagr']:>+6.1%} {r['sharpe']:>7.2f} {r['sortino']:>8.2f} "
f"{r['maxdd']:>+6.1%} {r['calmar']:>7.2f}{flag}")
# ================================================================
# ROUND 4: Combine top non-baseline factors
# ================================================================
top_funcs = {}
func_map_all = dict(r1_factors + r2_factors + r3_factors)
non_baseline = [r for r in unique_sorted if "BASELINE" not in r["name"] and r.get("cagr", 0)]
for r in non_baseline[:12]:
if r["name"] in func_map_all:
top_funcs[r["name"]] = func_map_all[r["name"]]
top_names = list(top_funcs.keys())
print(f"\n Building combinations from top {len(top_names)} factors: {top_names}")
combo_factors = [("BASELINE:rec+mom", f_rec_mom)]
# All pairs from top-8
for i in range(min(8, len(top_names))):
for j in range(i + 1, min(8, len(top_names))):
n1, n2 = top_names[i], top_names[j]
combo_factors.append((
f"{n1[:20]}+{n2[:20]}",
combo([(top_funcs[n1], 0.5), (top_funcs[n2], 0.5)])
))
# Triple combos from top-5
for i in range(min(5, len(top_names))):
for j in range(i + 1, min(5, len(top_names))):
for k in range(j + 1, min(5, len(top_names))):
n1, n2, n3 = top_names[i], top_names[j], top_names[k]
combo_factors.append((
f"{n1[:15]}+{n2[:15]}+{n3[:15]}",
combo([(top_funcs[n1], 0.33), (top_funcs[n2], 0.33), (top_funcs[n3], 0.34)])
))
r4 = run_round("ROUND 4 — Factor Combinations", combo_factors, stocks)
# ================================================================
# ROUND 5: Yearly breakdown of top 5 combos
# ================================================================
r4_sorted = sorted(r4, key=lambda x: x.get("cagr", 0) or 0, reverse=True)
top5 = r4_sorted[:5]
# Make sure baseline is included
base = next((r for r in r4 if "BASELINE" in r["name"]), None)
if base and base not in top5:
top5.append(base)
print(f"\n{'='*95}")
print(f" ROUND 5 — YEARLY RETURNS OF BEST STRATEGIES — {market.upper()}")
print(f"{'='*95}")
cols = [(r["name"], r["equity"]) for r in top5]
if bench is not None:
eq_bench = bench / bench.iloc[0] * 100000
cols.append(("BENCHMARK", eq_bench))
# Header
header = f" {'Year':<6}"
for name, _ in cols:
header += f" | {name[:22]:>22}"
print(header)
print(" " + "-" * (6 + 25 * len(cols)))
all_years = sorted(set(y for _, eq in cols for y in eq.index.year.unique()))
for year in all_years:
line = f" {year:<6}"
for _, eq in cols:
dr = eq.pct_change().fillna(0)
yr = dr[dr.index.year == year]
r = float((1 + yr).prod() - 1) if len(yr) > 0 else 0
line += f" | {r:>+21.1%}"
print(line)
# Period CAGRs
for ny in [3, 5, 10]:
cutoff = stocks.index[-1] - pd.DateOffset(years=ny)
print(f"\n --- {ny}-year CAGR ---")
for name, eq in cols:
sl = eq[eq.index >= cutoff]
if len(sl) < 50:
continue
tot = sl.iloc[-1] / sl.iloc[0] - 1
cagr = (1 + tot) ** (1 / ny) - 1
print(f" {name[:50]:<50} {cagr:>+8.1%}")
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--market", default="us", choices=["us", "cn"])
args = parser.parse_args()
run_market(args.market)
if __name__ == "__main__":
main()

449
factor_real_backtest.py
View File

@@ -1,449 +0,0 @@
"""
Factor research v2 — Portfolio-first approach.
Instead of IC → portfolio, we go directly to:
1. Build factor signal
2. Select top-N stocks
3. Run real backtest with transaction costs
4. Measure CAGR, Sharpe, MaxDD, yearly returns
Tests single factors AND combinations. Compares everything against
the baseline recovery+momentum strategy.
"""
from __future__ import annotations
import argparse
import warnings
import numpy as np
import pandas as pd
import data_manager
import metrics
from universe import UNIVERSES
warnings.filterwarnings("ignore")
# ---------------------------------------------------------------------------
# Factor signals — each returns DataFrame (dates x stocks), higher = better
# ---------------------------------------------------------------------------
def f_momentum_12_1(p: pd.DataFrame) -> pd.DataFrame:
return p.shift(21).pct_change(231)
def f_recovery(p: pd.DataFrame) -> pd.DataFrame:
return p / p.rolling(63, min_periods=63).min() - 1
def f_recovery_mom(p: pd.DataFrame) -> pd.DataFrame:
"""The baseline composite: 50/50 recovery + momentum ranks."""
r1 = f_recovery(p).rank(axis=1, pct=True, na_option="keep")
r2 = f_momentum_12_1(p).rank(axis=1, pct=True, na_option="keep")
return 0.5 * r1 + 0.5 * r2
# --- New single factors ---
def f_short_term_reversal(p: pd.DataFrame) -> pd.DataFrame:
"""5-day return reversal."""
return -p.pct_change(5)
def f_vol_adjusted_mom(p: pd.DataFrame) -> pd.DataFrame:
"""Momentum divided by recent volatility. Sharpe-like signal.
Hypothesis: risk-adjusted momentum is more persistent."""
mom = p.shift(21).pct_change(231)
vol = p.pct_change().rolling(60, min_periods=40).std()
return mom / vol.replace(0, np.nan)
def f_acceleration(p: pd.DataFrame) -> pd.DataFrame:
"""3-month momentum minus 12-month momentum.
Hypothesis: accelerating stocks continue accelerating."""
mom_3m = p.shift(5).pct_change(63 - 5)
mom_12m = p.shift(21).pct_change(231)
return mom_3m - mom_12m
def f_breakout(p: pd.DataFrame) -> pd.DataFrame:
"""Price relative to 20-day high. Close to 1 = breaking out.
Hypothesis: breakouts from consolidation continue."""
return p / p.rolling(20, min_periods=20).max()
def f_recovery_deep(p: pd.DataFrame) -> pd.DataFrame:
"""Recovery from 126-day (6 month) low instead of 63-day.
Hypothesis: deeper recovery = stronger signal."""
return p / p.rolling(126, min_periods=126).min() - 1
def f_recovery_rate(p: pd.DataFrame) -> pd.DataFrame:
"""Speed of recovery: 20-day change in recovery factor.
Hypothesis: accelerating recovery predicts continuation."""
recovery = p / p.rolling(63, min_periods=63).min() - 1
return recovery - recovery.shift(20)
def f_drawdown_bounce(p: pd.DataFrame) -> pd.DataFrame:
"""20-day return from drawdown trough, only for stocks in drawdown.
Hypothesis: strong bounces from drawdowns persist."""
rolling_max = p.rolling(252, min_periods=126).max()
in_drawdown = p < rolling_max * 0.9 # at least 10% below peak
bounce_20d = p.pct_change(20)
# Only score stocks that were recently in drawdown
was_in_drawdown = in_drawdown.rolling(20, min_periods=1).max().astype(bool)
return bounce_20d.where(was_in_drawdown, np.nan)
def f_consistent_winner(p: pd.DataFrame) -> pd.DataFrame:
"""Fraction of months with positive returns over past 12 months.
Hypothesis: stocks that win consistently are higher quality momentum."""
monthly_ret = p.pct_change(21)
return (monthly_ret > 0).astype(float).rolling(252, min_periods=126).mean()
def f_gap_up_freq(p: pd.DataFrame) -> pd.DataFrame:
"""Fraction of days with >1% gain in past 60 days.
Hypothesis: frequent large gains = institutional buying."""
ret = p.pct_change()
return (ret > 0.01).astype(float).rolling(60, min_periods=40).mean()
def f_low_vol_mom(p: pd.DataFrame) -> pd.DataFrame:
"""Momentum only among low-volatility stocks. Combined rank.
Hypothesis: low-vol momentum is more persistent."""
mom = f_momentum_12_1(p).rank(axis=1, pct=True, na_option="keep")
vol = (-p.pct_change().rolling(60, min_periods=40).std()).rank(axis=1, pct=True, na_option="keep")
return 0.5 * mom + 0.5 * vol
def f_52w_channel_position(p: pd.DataFrame) -> pd.DataFrame:
"""Position within 252-day high-low channel. 1 = at high, 0 = at low.
Hypothesis: stocks near highs continue (anchoring + trend)."""
h = p.rolling(252, min_periods=126).max()
l = p.rolling(252, min_periods=126).min()
return (p - l) / (h - l).replace(0, np.nan)
def f_up_volume_proxy(p: pd.DataFrame) -> pd.DataFrame:
"""Proxy for accumulation: sum of returns on up days over 20 days.
Without volume data, use magnitude of positive returns as proxy."""
ret = p.pct_change()
up_ret = ret.where(ret > 0, 0)
return up_ret.rolling(20, min_periods=15).sum()
def f_relative_strength_ma(p: pd.DataFrame) -> pd.DataFrame:
"""Price above 50-day MA relative to 200-day MA position.
Dual MA trend strength."""
ma50 = p.rolling(50, min_periods=50).mean()
ma200 = p.rolling(200, min_periods=200).mean()
above_50 = (p / ma50 - 1)
above_200 = (p / ma200 - 1)
return 0.5 * above_50 + 0.5 * above_200
def f_earnings_drift_proxy(p: pd.DataFrame) -> pd.DataFrame:
"""Proxy for post-earnings drift using 5-day return spikes.
Identify large 5-day moves and bet on continuation.
Hypothesis: large moves driven by information continue."""
ret_5d = p.pct_change(5)
vol = p.pct_change().rolling(60, min_periods=40).std() * np.sqrt(5)
z_score = ret_5d / vol.replace(0, np.nan)
# Smooth: average z-score over past 60 days to capture multiple events
return z_score.rolling(60, min_periods=20).mean()
# --- A-share specific ---
def f_reversal_vol_cn(p: pd.DataFrame) -> pd.DataFrame:
"""Short-term reversal amplified by volatility.
High-vol oversold stocks bounce harder in A-shares."""
ret_5d = p.pct_change(5)
vol = p.pct_change().rolling(20, min_periods=15).std()
# Oversold (negative return) + high vol = positive score
return -ret_5d * vol
def f_momentum_6_1(p: pd.DataFrame) -> pd.DataFrame:
"""6-1 month momentum. Shorter lookback may work better in A-shares."""
return p.shift(21).pct_change(105)
def f_recovery_narrow(p: pd.DataFrame) -> pd.DataFrame:
"""Recovery from 21-day low. Faster recovery signal for A-shares."""
return p / p.rolling(21, min_periods=21).min() - 1
def f_range_breakout_cn(p: pd.DataFrame) -> pd.DataFrame:
"""Breakout from 60-day range. Tuned for A-share volatility."""
h60 = p.rolling(60, min_periods=40).max()
l60 = p.rolling(60, min_periods=40).min()
mid = (h60 + l60) / 2
rng = (h60 - l60) / mid.replace(0, np.nan)
position = (p - l60) / (h60 - l60).replace(0, np.nan)
# Reward stocks breaking out of narrow ranges
return position / rng.replace(0, np.nan)
# ---------------------------------------------------------------------------
# Strategy builder and backtester
# ---------------------------------------------------------------------------
def make_strategy(
prices: pd.DataFrame,
signal_func,
top_n: int = 10,
rebal_freq: int = 21,
warmup: int = 252,
) -> pd.DataFrame:
"""Turn a factor signal into a rebalanced top-N equal-weight strategy."""
signal = signal_func(prices)
rank = signal.rank(axis=1, ascending=False, na_option="bottom")
n_valid = signal.notna().sum(axis=1)
enough = n_valid >= top_n
top_mask = (rank <= top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
weights = raw.div(row_sums, axis=0).fillna(0.0)
# Monthly rebalance
rebal_mask = pd.Series(False, index=prices.index)
rebal_indices = list(range(warmup, len(prices), rebal_freq))
rebal_mask.iloc[rebal_indices] = True
weights[~rebal_mask] = np.nan
weights = weights.ffill().fillna(0.0)
weights.iloc[:warmup] = 0.0
return weights.shift(1).fillna(0.0)
def combo_signal(funcs_and_weights: list[tuple]) -> callable:
"""Create a combined signal function from [(func, weight), ...]."""
def _combo(p: pd.DataFrame) -> pd.DataFrame:
ranked = []
for func, w in funcs_and_weights:
sig = func(p)
ranked.append(w * sig.rank(axis=1, pct=True, na_option="keep"))
return sum(ranked)
return _combo
def run_backtest(
weights: pd.DataFrame,
prices: pd.DataFrame,
cost: float = 0.001,
) -> pd.Series:
"""Vectorized backtest returning equity curve."""
returns = prices.pct_change().fillna(0.0)
port_ret = (weights * returns).sum(axis=1)
turnover = weights.diff().abs().sum(axis=1)
port_ret -= turnover * cost
return (1 + port_ret).cumprod() * 100000
def compute_stats(equity: pd.Series, label: str) -> dict:
"""Compute strategy statistics."""
daily_ret = equity.pct_change().dropna()
if len(daily_ret) < 100 or daily_ret.std() == 0:
return {"name": label, "cagr": np.nan, "sharpe": np.nan, "maxdd": np.nan,
"total": np.nan, "win_rate": np.nan}
n_years = len(daily_ret) / 252
total_ret = equity.iloc[-1] / equity.iloc[0] - 1
cagr = (1 + total_ret) ** (1 / n_years) - 1
sharpe = daily_ret.mean() / daily_ret.std() * np.sqrt(252)
sortino_denom = daily_ret[daily_ret < 0].std()
sortino = daily_ret.mean() / sortino_denom * np.sqrt(252) if sortino_denom > 0 else 0
running_max = equity.cummax()
maxdd = ((equity - running_max) / running_max).min()
calmar = cagr / abs(maxdd) if maxdd != 0 else 0
win_rate = (daily_ret > 0).mean()
return {
"name": label, "cagr": cagr, "sharpe": sharpe, "sortino": sortino,
"maxdd": maxdd, "calmar": calmar, "total": total_ret, "win_rate": win_rate,
}
def yearly_returns(equity: pd.Series) -> dict[int, float]:
daily_ret = equity.pct_change().fillna(0)
years = daily_ret.index.year
result = {}
for year in sorted(years.unique()):
mask = years == year
result[year] = float((1 + daily_ret[mask]).prod() - 1)
return result
def run(market: str):
config = UNIVERSES[market]
benchmark = config["benchmark"]
print(f"Loading {market.upper()} price data...")
prices = data_manager.load(market)
bench = prices[benchmark].dropna() if benchmark in prices.columns else None
stocks = prices.drop(columns=[benchmark], errors="ignore")
print(f"Universe: {stocks.shape[1]} stocks, {stocks.shape[0]} days")
print(f"Period: {stocks.index[0].date()} to {stocks.index[-1].date()}\n")
# --- Define all strategies to test ---
strategies: list[tuple[str, callable]] = []
# Baseline
strategies.append(("BASELINE: recovery+mom", f_recovery_mom))
# Single factors
strategies.append(("momentum_12_1", f_momentum_12_1))
strategies.append(("recovery", f_recovery))
strategies.append(("vol_adj_momentum", f_vol_adjusted_mom))
strategies.append(("acceleration", f_acceleration))
strategies.append(("breakout_20d", f_breakout))
strategies.append(("recovery_deep_126d", f_recovery_deep))
strategies.append(("recovery_rate", f_recovery_rate))
strategies.append(("drawdown_bounce", f_drawdown_bounce))
strategies.append(("consistent_winner", f_consistent_winner))
strategies.append(("gap_up_freq", f_gap_up_freq))
strategies.append(("low_vol_momentum", f_low_vol_mom))
strategies.append(("52w_channel_position", f_52w_channel_position))
strategies.append(("up_volume_proxy", f_up_volume_proxy))
strategies.append(("relative_strength_ma", f_relative_strength_ma))
strategies.append(("earnings_drift_proxy", f_earnings_drift_proxy))
if market == "cn":
strategies.append(("reversal_vol_cn", f_reversal_vol_cn))
strategies.append(("momentum_6_1", f_momentum_6_1))
strategies.append(("recovery_narrow_21d", f_recovery_narrow))
strategies.append(("range_breakout_cn", f_range_breakout_cn))
# Run all single-factor backtests
print("=" * 110)
print(f" SINGLE FACTOR BACKTESTS — {market.upper()} (Top 10, monthly rebal, 10bps cost)")
print("=" * 110)
results = []
equities = {}
for name, func in strategies:
print(f" Running: {name}...")
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
equities[name] = eq
results.append(compute_stats(eq, name))
# Benchmark
if bench is not None:
eq_bench = bench / bench.iloc[0] * 100000
equities["BENCHMARK"] = eq_bench
results.append(compute_stats(eq_bench, "BENCHMARK"))
# Print results table
df = pd.DataFrame(results).set_index("name")
df = df.sort_values("cagr", ascending=False)
print(f"\n{'Strategy':<30} {'CAGR':>8} {'Sharpe':>8} {'Sortino':>8} {'MaxDD':>8} {'Calmar':>8} {'Total':>10}")
print("-" * 90)
for name, row in df.iterrows():
flag = " ***" if name == "BASELINE: recovery+mom" else ""
print(f"{name:<30} {row['cagr']:>+7.1%} {row['sharpe']:>8.2f} {row['sortino']:>8.2f} "
f"{row['maxdd']:>+7.1%} {row['calmar']:>8.2f} {row['total']:>+9.0%}{flag}")
# --- Identify factors that beat or match baseline ---
baseline_cagr = df.loc["BASELINE: recovery+mom", "cagr"]
winners = df[df["cagr"] >= baseline_cagr * 0.8].index.tolist()
winners = [w for w in winners if w not in ("BASELINE: recovery+mom", "BENCHMARK")]
print(f"\nFactors within 80% of baseline CAGR ({baseline_cagr:.1%}): {winners}")
# --- Test combinations of top performers ---
print(f"\n{'='*110}")
print(f" FACTOR COMBINATIONS — {market.upper()}")
print(f"{'='*110}")
# Get top single factors
single_only = df.drop(["BASELINE: recovery+mom", "BENCHMARK"], errors="ignore")
top_singles = single_only.nlargest(8, "cagr").index.tolist()
print(f" Top 8 singles: {top_singles}\n")
# Map names back to functions
func_map = dict(strategies)
combos: list[tuple[str, callable]] = []
# Baseline is always included
combos.append(("BASELINE: recovery+mom", f_recovery_mom))
# Top2 combinations
for i in range(min(6, len(top_singles))):
for j in range(i + 1, min(6, len(top_singles))):
n1, n2 = top_singles[i], top_singles[j]
label = f"{n1} + {n2}"
func = combo_signal([(func_map[n1], 0.5), (func_map[n2], 0.5)])
combos.append((label, func))
# Recovery+mom + each top single (3-factor)
for name in top_singles[:6]:
if name in ("momentum_12_1", "recovery"):
continue
label = f"rec+mom + {name}"
func = combo_signal([
(f_recovery, 0.33), (f_momentum_12_1, 0.33), (func_map[name], 0.34)
])
combos.append((label, func))
# Run combo backtests
combo_results = []
for name, func in combos:
print(f" Running: {name}...")
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
equities[name] = eq
combo_results.append(compute_stats(eq, name))
combo_df = pd.DataFrame(combo_results).set_index("name")
combo_df = combo_df.sort_values("cagr", ascending=False)
print(f"\n{'Combo':<55} {'CAGR':>8} {'Sharpe':>8} {'Sortino':>8} {'MaxDD':>8} {'Calmar':>8}")
print("-" * 105)
for name, row in combo_df.iterrows():
flag = " ***" if name == "BASELINE: recovery+mom" else ""
print(f"{name:<55} {row['cagr']:>+7.1%} {row['sharpe']:>8.2f} {row['sortino']:>8.2f} "
f"{row['maxdd']:>+7.1%} {row['calmar']:>8.2f}{flag}")
# --- Yearly breakdown for top 3 combos ---
top3 = combo_df.nlargest(3, "cagr").index.tolist()
if "BASELINE: recovery+mom" not in top3:
top3.append("BASELINE: recovery+mom")
print(f"\n{'='*110}")
print(f" YEARLY RETURNS — TOP STRATEGIES vs BASELINE — {market.upper()}")
print(f"{'='*110}")
yr_data = {}
for name in top3:
yr_data[name] = yearly_returns(equities[name])
if bench is not None:
yr_data["BENCHMARK"] = yearly_returns(equities["BENCHMARK"])
all_years = sorted(set(y for yd in yr_data.values() for y in yd.keys()))
# Print header
col_names = top3 + (["BENCHMARK"] if bench is not None else [])
header = f" {'Year':<6}"
for c in col_names:
header += f" | {c[:25]:>25}"
print(header)
print(" " + "-" * (6 + 28 * len(col_names)))
for year in all_years:
line = f" {year:<6}"
for c in col_names:
r = yr_data.get(c, {}).get(year, 0)
line += f" | {r:>+24.1%}"
print(line)
# Compute period summaries
for n_years in [3, 5, 10]:
cutoff = stocks.index[-1] - pd.DateOffset(years=n_years)
print(f"\n --- {n_years}-year CAGR ---")
for name in col_names:
eq = equities.get(name)
if eq is None:
continue
eq_slice = eq[eq.index >= cutoff]
if len(eq_slice) < 50:
continue
total = eq_slice.iloc[-1] / eq_slice.iloc[0] - 1
cagr = (1 + total) ** (1 / n_years) - 1
print(f" {name[:40]:<40} {cagr:>+8.1%}")
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--market", default="us", choices=["us", "cn"])
args = parser.parse_args()
run(args.market)
if __name__ == "__main__":
main()

547
factor_research.py
View File

@@ -1,547 +0,0 @@
"""
Factor Research Script — Professional QR-style factor mining.
Tests candidate alpha factors using:
- Information Coefficient (IC): rank correlation of signal vs forward returns
- IC Information Ratio (ICIR): mean(IC) / std(IC), measures signal consistency
- Quintile long-short spread: monotonicity of returns across signal buckets
- Turnover: daily rank change, proxy for trading cost
- Decay profile: IC at 1d, 5d, 10d, 20d horizons
- Sub-period stability: IC consistency across rolling windows
- Factor correlation matrix: ensures new factors are orthogonal to known ones
Usage:
uv run python factor_research.py --market us
uv run python factor_research.py --market cn
"""
from __future__ import annotations
import argparse
import warnings
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
warnings.filterwarnings("ignore", category=FutureWarning)
HORIZONS = [1, 5, 10, 20]
# ---------------------------------------------------------------------------
# Factor definitions — each returns a DataFrame (dates x stocks) of scores
# ---------------------------------------------------------------------------
def _safe_rank(df: pd.DataFrame) -> pd.DataFrame:
return df.rank(axis=1, pct=True, na_option="keep")
def _rolling_ret(prices: pd.DataFrame, window: int) -> pd.DataFrame:
return prices.pct_change(window)
# --- Known factors (baselines) ---
def factor_momentum_12_1(prices: pd.DataFrame) -> pd.DataFrame:
"""Classic 12-1 month momentum."""
return prices.shift(21).pct_change(231)
def factor_recovery(prices: pd.DataFrame) -> pd.DataFrame:
"""Price / 63-day low - 1."""
return prices / prices.rolling(63, min_periods=63).min() - 1
def factor_inverse_vol(prices: pd.DataFrame) -> pd.DataFrame:
"""Negative 60-day realized volatility (low vol = high score)."""
return -prices.pct_change().rolling(60, min_periods=60).std()
# --- NEW candidate factors ---
def factor_short_term_reversal(prices: pd.DataFrame) -> pd.DataFrame:
"""5-day return reversal. Hypothesis: short-term mean reversion."""
return -prices.pct_change(5)
def factor_idio_vol_change(prices: pd.DataFrame) -> pd.DataFrame:
"""Change in idiosyncratic volatility (20d vs 60d).
Hypothesis: declining vol = stabilizing, predicts positive returns."""
ret = prices.pct_change()
vol_20 = ret.rolling(20, min_periods=20).std()
vol_60 = ret.rolling(60, min_periods=60).std()
return -(vol_20 / vol_60.replace(0, np.nan) - 1) # negative = vol declining
def factor_volume_price_divergence(prices: pd.DataFrame, volume: pd.DataFrame | None = None) -> pd.DataFrame:
"""Price up but momentum fading — proxy via acceleration.
Without volume data, use return acceleration as proxy."""
ret_5 = prices.pct_change(5)
ret_20 = prices.pct_change(20)
return ret_5 - ret_20 / 4 # recent returns outpacing trend
def factor_max_drawdown_recovery(prices: pd.DataFrame) -> pd.DataFrame:
"""How much of the 60-day max drawdown has been recovered.
Hypothesis: stocks that recover from drawdowns continue recovering."""
rolling_max = prices.rolling(60, min_periods=60).max()
drawdown = prices / rolling_max - 1 # negative
rolling_min_dd = drawdown.rolling(60, min_periods=20).min() # worst drawdown
recovery_pct = drawdown / rolling_min_dd.replace(0, np.nan)
return recovery_pct # closer to 0 = more recovered
def factor_skewness(prices: pd.DataFrame) -> pd.DataFrame:
"""Negative 20-day return skewness.
Hypothesis: negatively skewed stocks are overpriced (lottery preference)."""
ret = prices.pct_change()
return -ret.rolling(20, min_periods=20).skew()
def factor_high_low_range(prices: pd.DataFrame) -> pd.DataFrame:
"""20-day high-low range relative to price.
Hypothesis: narrow range = consolidation, breakout ahead."""
high_20 = prices.rolling(20, min_periods=20).max()
low_20 = prices.rolling(20, min_periods=20).min()
mid = (high_20 + low_20) / 2
return -(high_20 - low_20) / mid.replace(0, np.nan) # negative range = narrow = high score
def factor_mean_reversion_residual(prices: pd.DataFrame) -> pd.DataFrame:
"""Distance from 20-day MA as fraction of 60-day vol.
Hypothesis: stocks far below MA revert. Z-score style."""
ma_20 = prices.rolling(20, min_periods=20).mean()
vol_60 = prices.pct_change().rolling(60, min_periods=60).std() * prices
return -(prices - ma_20) / vol_60.replace(0, np.nan) # below MA = high score
def factor_up_down_vol_ratio(prices: pd.DataFrame) -> pd.DataFrame:
"""Ratio of upside to downside semi-deviation (20d).
Hypothesis: stocks with more upside vol have positive momentum."""
ret = prices.pct_change()
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).std()
down_vol = ret.where(ret < 0, 0).rolling(20, min_periods=15).std()
return up_vol / down_vol.replace(0, np.nan)
def factor_consecutive_up_days(prices: pd.DataFrame) -> pd.DataFrame:
"""Fraction of positive return days in last 10 days.
Hypothesis: persistent winners keep winning (short-term)."""
ret = prices.pct_change()
return (ret > 0).astype(float).rolling(10, min_periods=10).mean()
def factor_gap_momentum(prices: pd.DataFrame) -> pd.DataFrame:
"""Cumulative overnight-like gaps: close-to-close vs intraday proxy.
Using 1-day returns smoothed over 20 days minus 5-day return.
Hypothesis: smooth consistent returns beat volatile ones."""
ret_1d = prices.pct_change()
smoothness = ret_1d.rolling(20, min_periods=20).mean() * 20
raw_20d = prices.pct_change(20)
return smoothness - raw_20d # positive = smoother path
def factor_recovery_acceleration(prices: pd.DataFrame) -> pd.DataFrame:
"""Rate of change of recovery factor.
Hypothesis: accelerating recovery is stronger signal than level."""
recovery = prices / prices.rolling(63, min_periods=63).min() - 1
return recovery.pct_change(5)
def factor_trend_strength(prices: pd.DataFrame) -> pd.DataFrame:
"""R-squared of log-price vs time over 60 days.
Hypothesis: stocks trending linearly (high R2) continue."""
log_p = np.log(prices.replace(0, np.nan))
def _r2(series):
y = series.dropna().values
if len(y) < 30:
return np.nan
x = np.arange(len(y), dtype=float)
x -= x.mean()
y_dm = y - y.mean()
ss_xy = (x * y_dm).sum()
ss_xx = (x * x).sum()
ss_yy = (y_dm * y_dm).sum()
if ss_xx == 0 or ss_yy == 0:
return np.nan
r2 = (ss_xy ** 2) / (ss_xx * ss_yy)
slope = ss_xy / ss_xx
return r2 if slope > 0 else -r2 # sign by direction
return log_p.rolling(60, min_periods=30).apply(_r2, raw=False)
def factor_relative_volume_momentum(prices: pd.DataFrame) -> pd.DataFrame:
"""Price momentum weighted by how 'cheap' a stock is relative to 52-week range.
Hypothesis: momentum in stocks near lows is more likely to persist."""
mom_20 = prices.pct_change(20)
high_252 = prices.rolling(252, min_periods=126).max()
low_252 = prices.rolling(252, min_periods=126).min()
position_in_range = (prices - low_252) / (high_252 - low_252).replace(0, np.nan)
return mom_20 * (1 - position_in_range) # momentum * cheapness
def factor_52w_high_distance(prices: pd.DataFrame) -> pd.DataFrame:
"""Distance from 52-week high.
Hypothesis: stocks near their highs continue (anchoring bias)."""
high_252 = prices.rolling(252, min_periods=126).max()
return prices / high_252 # closer to 1 = near high
def factor_downside_beta_proxy(prices: pd.DataFrame) -> pd.DataFrame:
"""Proxy for downside beta using co-movement on market down days.
Hypothesis: low downside beta outperforms (asymmetric risk)."""
ret = prices.pct_change()
market_ret = ret.mean(axis=1)
down_days = market_ret < 0
# Mask non-down-day returns to NaN, then rolling mean
# Use numpy for correct broadcasting, wider window (120d) so ~54 down
# days are available, well above min_periods=20
arr = ret.values.copy()
arr[~down_days.values, :] = np.nan
down_ret = pd.DataFrame(arr, index=ret.index, columns=ret.columns)
avg_down = down_ret.rolling(120, min_periods=20).mean()
return -avg_down # negative = less downside = good
# --- A-share specific factors ---
def factor_liquidity_premium(prices: pd.DataFrame) -> pd.DataFrame:
"""Amihud illiquidity proxy (using returns only, no volume).
Hypothesis: illiquid stocks earn premium in A-shares (retail driven)."""
ret = prices.pct_change()
# Use absolute return as illiquidity proxy (higher abs ret = less liquid)
illiq = ret.abs().rolling(20, min_periods=15).mean()
return illiq
def factor_lottery_demand(prices: pd.DataFrame) -> pd.DataFrame:
"""Max daily return in past 20 days (negative).
Hypothesis: lottery stocks (high max return) underperform.
Strong in A-shares due to retail speculation."""
ret = prices.pct_change()
return -ret.rolling(20, min_periods=15).max()
def factor_turnover_reversal(prices: pd.DataFrame) -> pd.DataFrame:
"""Interaction of short-term returns with volatility.
High vol + negative return = oversold bounce candidate.
Common A-share alpha source."""
ret_5 = prices.pct_change(5)
vol_20 = prices.pct_change().rolling(20, min_periods=15).std()
return -ret_5 * vol_20 # oversold + high vol = positive
def factor_price_level(prices: pd.DataFrame) -> pd.DataFrame:
"""Negative absolute price level.
Hypothesis: low-priced stocks attract retail in A-shares (penny stock effect)."""
return -prices
# ---------------------------------------------------------------------------
# IC and analytics engine
# ---------------------------------------------------------------------------
def compute_ic(
signal: pd.DataFrame,
forward_ret: pd.DataFrame,
) -> pd.Series:
"""Cross-sectional rank IC (Spearman) per day."""
common_idx = signal.index.intersection(forward_ret.index)
common_cols = signal.columns.intersection(forward_ret.columns)
sig = signal.loc[common_idx, common_cols]
fwd = forward_ret.loc[common_idx, common_cols]
ics = []
for date in common_idx:
s = sig.loc[date].dropna()
f = fwd.loc[date].dropna()
common = s.index.intersection(f.index)
if len(common) < 30:
continue
ic = s[common].corr(f[common], method="spearman")
if np.isfinite(ic):
ics.append((date, ic))
if not ics:
return pd.Series(dtype=float)
return pd.Series(dict(ics))
def compute_quintile_returns(
signal: pd.DataFrame,
forward_ret: pd.DataFrame,
n_quantiles: int = 5,
) -> pd.DataFrame:
"""Average forward return by signal quintile, per day, then time-averaged."""
common_idx = signal.index.intersection(forward_ret.index)
common_cols = signal.columns.intersection(forward_ret.columns)
sig = signal.loc[common_idx, common_cols]
fwd = forward_ret.loc[common_idx, common_cols]
records = []
for date in common_idx:
s = sig.loc[date].dropna()
f = fwd.loc[date].dropna()
common = s.index.intersection(f.index)
if len(common) < 50:
continue
scores = s[common]
rets = f[common]
try:
quintile = pd.qcut(scores, n_quantiles, labels=False, duplicates="drop")
except ValueError:
continue
for q in range(n_quantiles):
mask = quintile == q
if mask.sum() > 0:
records.append({"date": date, "quintile": q + 1, "return": rets[mask].mean()})
if not records:
return pd.DataFrame()
df = pd.DataFrame(records)
return df.groupby("quintile")["return"].mean() * 252 # annualize
def compute_turnover(signal: pd.DataFrame) -> float:
"""Average daily rank change (turnover proxy)."""
ranked = signal.rank(axis=1, pct=True, na_option="keep")
daily_change = ranked.diff().abs().mean(axis=1)
return float(daily_change.mean())
def compute_factor_correlation(factors: dict[str, pd.DataFrame]) -> pd.DataFrame:
"""Cross-sectional IC correlation between all factor pairs."""
names = list(factors.keys())
n = len(names)
corr_matrix = pd.DataFrame(np.nan, index=names, columns=names)
# Use time-series of rank-averaged signals
avg_ranks = {}
for name, sig in factors.items():
ranked = sig.rank(axis=1, pct=True, na_option="keep")
avg_ranks[name] = ranked.mean(axis=1).dropna()
for i in range(n):
for j in range(i, n):
s1 = avg_ranks[names[i]]
s2 = avg_ranks[names[j]]
common = s1.index.intersection(s2.index)
if len(common) > 100:
c = s1[common].corr(s2[common])
corr_matrix.loc[names[i], names[j]] = c
corr_matrix.loc[names[j], names[i]] = c
if i == j:
corr_matrix.loc[names[i], names[j]] = 1.0
return corr_matrix
def analyze_factor(
name: str,
signal: pd.DataFrame,
prices: pd.DataFrame,
horizons: list[int] | None = None,
) -> dict:
"""Full single-factor analysis."""
if horizons is None:
horizons = HORIZONS
results = {"name": name}
# Forward returns at each horizon
for h in horizons:
fwd_ret = prices.pct_change(h).shift(-h)
ic_series = compute_ic(signal, fwd_ret)
if len(ic_series) == 0:
results[f"ic_{h}d"] = np.nan
results[f"icir_{h}d"] = np.nan
continue
ic_mean = ic_series.mean()
ic_std = ic_series.std()
icir = ic_mean / ic_std if ic_std > 0 else 0.0
results[f"ic_{h}d"] = ic_mean
results[f"icir_{h}d"] = icir
if h == 1:
results["ic_1d_series"] = ic_series
# Quintile analysis at 5-day horizon
fwd_5d = prices.pct_change(5).shift(-5)
quintiles = compute_quintile_returns(signal, fwd_5d)
if not quintiles.empty:
results["q5_return"] = float(quintiles.iloc[-1]) # top quintile
results["q1_return"] = float(quintiles.iloc[0]) # bottom quintile
results["long_short_ann"] = float(quintiles.iloc[-1] - quintiles.iloc[0])
results["monotonicity"] = float(quintiles.corr(pd.Series(range(1, len(quintiles) + 1), index=quintiles.index)))
results["quintile_returns"] = quintiles
else:
results["q5_return"] = np.nan
results["q1_return"] = np.nan
results["long_short_ann"] = np.nan
results["monotonicity"] = np.nan
# Turnover
results["turnover"] = compute_turnover(signal)
# Sub-period IC stability (rolling 252-day IC mean)
if "ic_1d_series" in results and len(results["ic_1d_series"]) > 252:
rolling_ic = results["ic_1d_series"].rolling(252).mean().dropna()
results["ic_stability"] = float((rolling_ic > 0).mean()) # fraction of time IC > 0
results["ic_worst_year"] = float(rolling_ic.min())
results["ic_best_year"] = float(rolling_ic.max())
else:
results["ic_stability"] = np.nan
results["ic_worst_year"] = np.nan
results["ic_best_year"] = np.nan
return results
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def get_all_factors(prices: pd.DataFrame, market: str) -> dict[str, pd.DataFrame]:
"""Build all candidate factor signals."""
factors = {}
# Known baselines
factors["momentum_12_1"] = factor_momentum_12_1(prices)
factors["recovery"] = factor_recovery(prices)
factors["inverse_vol"] = factor_inverse_vol(prices)
# New candidates — universal
factors["short_term_reversal"] = factor_short_term_reversal(prices)
factors["idio_vol_change"] = factor_idio_vol_change(prices)
factors["return_acceleration"] = factor_volume_price_divergence(prices)
factors["drawdown_recovery"] = factor_max_drawdown_recovery(prices)
factors["neg_skewness"] = factor_skewness(prices)
factors["range_compression"] = factor_high_low_range(prices)
factors["mean_rev_zscore"] = factor_mean_reversion_residual(prices)
factors["up_down_vol_ratio"] = factor_up_down_vol_ratio(prices)
factors["win_streak"] = factor_consecutive_up_days(prices)
factors["smooth_momentum"] = factor_gap_momentum(prices)
factors["recovery_accel"] = factor_recovery_acceleration(prices)
factors["trend_r2"] = factor_trend_strength(prices)
factors["cheap_momentum"] = factor_relative_volume_momentum(prices)
factors["near_52w_high"] = factor_52w_high_distance(prices)
factors["low_downside_beta"] = factor_downside_beta_proxy(prices)
# A-share specific (also test on US for comparison)
if market == "cn":
factors["illiquidity"] = factor_liquidity_premium(prices)
factors["anti_lottery"] = factor_lottery_demand(prices)
factors["vol_reversal"] = factor_turnover_reversal(prices)
factors["low_price"] = factor_price_level(prices)
return factors
def print_summary_table(results: list[dict], market: str) -> None:
"""Print a ranked summary of all factors."""
rows = []
for r in results:
rows.append({
"Factor": r["name"],
"IC_1d": r.get("ic_1d", np.nan),
"ICIR_1d": r.get("icir_1d", np.nan),
"IC_5d": r.get("ic_5d", np.nan),
"ICIR_5d": r.get("icir_5d", np.nan),
"IC_20d": r.get("ic_20d", np.nan),
"ICIR_20d": r.get("icir_20d", np.nan),
"LS_5d_ann": r.get("long_short_ann", np.nan),
"Mono": r.get("monotonicity", np.nan),
"Turnover": r.get("turnover", np.nan),
"IC_stab": r.get("ic_stability", np.nan),
"IC_worst_yr": r.get("ic_worst_year", np.nan),
})
df = pd.DataFrame(rows).set_index("Factor")
df = df.sort_values("ICIR_5d", ascending=False)
print(f"\n{'='*100}")
print(f" FACTOR RESEARCH RESULTS — {market.upper()} MARKET")
print(f"{'='*100}")
print("\nRanked by 5-day ICIR (most important metric for tradeable alpha):\n")
print(df.round(4).to_string())
# Highlight top factors
print(f"\n{'='*100}")
print(" TOP FACTORS (ICIR_5d > 0.05 and IC_stability > 0.6)")
print(f"{'='*100}")
top = df[(df["ICIR_5d"].abs() > 0.05) & (df["IC_stab"] > 0.6)]
if top.empty:
top = df.head(5)
print(" (No factor met strict threshold; showing top 5 by ICIR_5d)")
print(top.round(4).to_string())
# Quintile details for top factors
print(f"\n{'='*100}")
print(" QUINTILE RETURN PROFILES (annualized, 5-day forward)")
print(f"{'='*100}")
for r in sorted(results, key=lambda x: abs(x.get("icir_5d", 0)), reverse=True)[:8]:
qr = r.get("quintile_returns")
if qr is not None and not qr.empty:
q_str = " ".join(f"Q{int(k)}: {v:+.1%}" for k, v in qr.items())
ls = r.get("long_short_ann", 0)
print(f" {r['name']:25s} | {q_str} | L/S: {ls:+.1%}")
def main():
parser = argparse.ArgumentParser(description="Factor research")
parser.add_argument("--market", default="us", choices=["us", "cn"])
parser.add_argument("--years", type=int, default=None, help="Limit to last N years")
args = parser.parse_args()
market = args.market
config = UNIVERSES[market]
benchmark = config["benchmark"]
print(f"Loading {market.upper()} price data...")
prices = data_manager.load(market)
# Remove benchmark from stock universe
stocks = prices.drop(columns=[benchmark], errors="ignore")
if args.years:
cutoff = stocks.index[-1] - pd.DateOffset(years=args.years)
stocks = stocks[stocks.index >= cutoff]
print(f"Universe: {stocks.shape[1]} stocks, {stocks.shape[0]} trading days")
print(f"Date range: {stocks.index[0].date()} to {stocks.index[-1].date()}")
# Build all factor signals
print("\nComputing factor signals...")
factors = get_all_factors(stocks, market)
# Analyze each factor
print("Running factor analysis (this may take a few minutes)...")
results = []
for name, signal in factors.items():
print(f" Analyzing: {name}...")
r = analyze_factor(name, signal, stocks)
results.append(r)
# Print results
print_summary_table(results, market)
# Factor correlation matrix
print(f"\n{'='*100}")
print(" FACTOR CORRELATION MATRIX (rank-averaged cross-sectional)")
print(f"{'='*100}")
corr = compute_factor_correlation(factors)
# Show only top factors
top_names = [r["name"] for r in sorted(results, key=lambda x: abs(x.get("icir_5d", 0)), reverse=True)[:10]]
top_names_present = [n for n in top_names if n in corr.index]
print(corr.loc[top_names_present, top_names_present].round(2).to_string())
if __name__ == "__main__":
main()

323
factor_robustness.py
View File

@@ -1,323 +0,0 @@
"""
Robustness checks for winning factor strategies.
Tests:
1. Rolling 2-year window performance (stability)
2. Top-N sensitivity (5, 10, 15, 20)
3. Rebalance frequency sensitivity (5d, 10d, 21d, 42d)
4. Transaction cost sensitivity (0, 10bps, 20bps, 50bps)
5. Drawdown analysis
"""
from __future__ import annotations
import argparse
import warnings
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from factor_real_backtest import (
f_recovery_mom,
f_momentum_12_1,
f_recovery,
f_recovery_deep,
f_up_volume_proxy,
f_gap_up_freq,
f_earnings_drift_proxy,
f_reversal_vol_cn,
f_consistent_winner,
combo_signal,
make_strategy,
run_backtest,
compute_stats,
)
warnings.filterwarnings("ignore")
def rolling_window_performance(equity: pd.Series, window_years: int = 2):
"""Compute rolling window returns."""
daily_ret = equity.pct_change().dropna()
window = 252 * window_years
results = []
for end_idx in range(window, len(daily_ret), 63): # step 3 months
start_idx = end_idx - window
chunk = daily_ret.iloc[start_idx:end_idx]
total = (1 + chunk).prod() - 1
ann = (1 + total) ** (252 / len(chunk)) - 1
sharpe = chunk.mean() / chunk.std() * np.sqrt(252) if chunk.std() > 0 else 0
results.append({
"end_date": chunk.index[-1].date(),
"ann_return": ann,
"sharpe": sharpe,
})
return pd.DataFrame(results)
def drawdown_analysis(equity: pd.Series) -> pd.DataFrame:
"""Find top 5 drawdown episodes."""
running_max = equity.cummax()
drawdown = (equity - running_max) / running_max
# Find drawdown episodes
episodes = []
in_dd = False
start = None
for i in range(len(drawdown)):
if drawdown.iloc[i] < -0.05 and not in_dd:
in_dd = True
start = i
elif drawdown.iloc[i] >= 0 and in_dd:
in_dd = False
trough_idx = drawdown.iloc[start:i].idxmin()
episodes.append({
"start": drawdown.index[start].date(),
"trough": trough_idx.date(),
"end": drawdown.index[i].date(),
"depth": drawdown.loc[trough_idx],
"duration_days": i - start,
})
# Handle ongoing drawdown
if in_dd:
trough_idx = drawdown.iloc[start:].idxmin()
episodes.append({
"start": drawdown.index[start].date(),
"trough": trough_idx.date(),
"end": "ongoing",
"depth": drawdown.loc[trough_idx],
"duration_days": len(drawdown) - start,
})
df = pd.DataFrame(episodes)
if df.empty:
return df
return df.nsmallest(5, "depth")
def run_us(stocks: pd.DataFrame):
print("=" * 100)
print(" US ROBUSTNESS — Winner: momentum_12_1 + up_volume_proxy")
print("=" * 100)
winner_func = combo_signal([(f_momentum_12_1, 0.5), (f_up_volume_proxy, 0.5)])
baseline_func = f_recovery_mom
# 1. Rolling 2-year performance
print("\n--- 1. Rolling 2-Year Performance ---\n")
for label, func in [("Winner: mom+upvol", winner_func),
("Baseline: rec+mom", baseline_func)]:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
roll = rolling_window_performance(eq)
if roll.empty:
continue
win_pct = (roll["ann_return"] > 0).mean()
print(f" {label}:")
print(f" Mean 2yr ann return: {roll['ann_return'].mean():+.1%}")
print(f" Min 2yr ann return: {roll['ann_return'].min():+.1%}")
print(f" Max 2yr ann return: {roll['ann_return'].max():+.1%}")
print(f" % positive 2yr: {win_pct:.0%}")
print(f" Mean 2yr Sharpe: {roll['sharpe'].mean():.2f}")
print()
# 2. Top-N sensitivity
print("--- 2. Top-N Sensitivity ---\n")
header = f" {'Top-N':<8}"
for label in ["Winner: mom+upvol", "Baseline: rec+mom"]:
header += f" | {'CAGR':>8} {'Sharpe':>8} {'MaxDD':>8}"
print(header)
print(" " + "-" * 70)
for top_n in [5, 10, 15, 20, 30]:
line = f" {top_n:<8}"
for func in [winner_func, baseline_func]:
w = make_strategy(stocks, func, top_n=top_n)
eq = run_backtest(w, stocks)
s = compute_stats(eq, "")
line += f" | {s['cagr']:>+7.1%} {s['sharpe']:>8.2f} {s['maxdd']:>+7.1%}"
print(line)
# 3. Rebalance frequency sensitivity
print("\n--- 3. Rebalance Frequency Sensitivity ---\n")
header = f" {'Rebal':<8}"
for label in ["Winner: mom+upvol", "Baseline: rec+mom"]:
header += f" | {'CAGR':>8} {'Sharpe':>8} {'MaxDD':>8}"
print(header)
print(" " + "-" * 70)
for rebal in [5, 10, 21, 42, 63]:
line = f" {rebal}d{'':<5}"
for func in [winner_func, baseline_func]:
w = make_strategy(stocks, func, top_n=10, rebal_freq=rebal)
eq = run_backtest(w, stocks)
s = compute_stats(eq, "")
line += f" | {s['cagr']:>+7.1%} {s['sharpe']:>8.2f} {s['maxdd']:>+7.1%}"
print(line)
# 4. Transaction cost sensitivity
print("\n--- 4. Transaction Cost Sensitivity ---\n")
header = f" {'Cost':<8}"
for label in ["Winner: mom+upvol", "Baseline: rec+mom"]:
header += f" | {'CAGR':>8} {'Sharpe':>8}"
print(header)
print(" " + "-" * 50)
for cost in [0, 0.001, 0.002, 0.005]:
line = f" {cost*10000:.0f}bps{'':<4}"
for func in [winner_func, baseline_func]:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks, cost=cost)
s = compute_stats(eq, "")
line += f" | {s['cagr']:>+7.1%} {s['sharpe']:>8.2f}"
print(line)
# 5. Drawdown analysis
print("\n--- 5. Drawdown Episodes ---\n")
for label, func in [("Winner: mom+upvol", winner_func),
("Baseline: rec+mom", baseline_func)]:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
dd = drawdown_analysis(eq)
print(f" {label}:")
if dd.empty:
print(" No significant drawdowns")
else:
for _, row in dd.iterrows():
print(f" {row['start']} → {row['trough']} → {row['end']}: "
f"{row['depth']:+.1%} ({row['duration_days']}d)")
print()
# 6. Also test the runner-up combos
print("--- 6. Other Strong Combos (Top-10, 21d rebal, 10bps) ---\n")
other_combos = [
("rec_deep+upvol", combo_signal([(f_recovery_deep, 0.5), (f_up_volume_proxy, 0.5)])),
("rec_deep+mom", combo_signal([(f_recovery_deep, 0.5), (f_momentum_12_1, 0.5)])),
("mom+gap_up", combo_signal([(f_momentum_12_1, 0.5), (f_gap_up_freq, 0.5)])),
("rec_deep+upvol+mom", combo_signal([(f_recovery_deep, 0.33), (f_up_volume_proxy, 0.33), (f_momentum_12_1, 0.34)])),
("mom+upvol+gap", combo_signal([(f_momentum_12_1, 0.33), (f_up_volume_proxy, 0.33), (f_gap_up_freq, 0.34)])),
]
for label, func in other_combos:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
s = compute_stats(eq, "")
print(f" {label:<25} CAGR: {s['cagr']:>+7.1%} Sharpe: {s['sharpe']:.2f} MaxDD: {s['maxdd']:>+7.1%} Calmar: {s['calmar']:.2f}")
def run_cn(stocks: pd.DataFrame):
print("\n" + "=" * 100)
print(" CN ROBUSTNESS — Winners: reversal_vol + gap_up, earn_drift + reversal_vol")
print("=" * 100)
winner1_func = combo_signal([(f_reversal_vol_cn, 0.5), (f_gap_up_freq, 0.5)])
winner2_func = combo_signal([(f_earnings_drift_proxy, 0.5), (f_reversal_vol_cn, 0.5)])
baseline_func = f_recovery_mom
# 1. Rolling 2-year performance
print("\n--- 1. Rolling 2-Year Performance ---\n")
for label, func in [("W1: rev_vol+gap_up", winner1_func),
("W2: earn_drift+rev_vol", winner2_func),
("Baseline: rec+mom", baseline_func)]:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
roll = rolling_window_performance(eq)
if roll.empty:
continue
win_pct = (roll["ann_return"] > 0).mean()
print(f" {label}:")
print(f" Mean 2yr ann return: {roll['ann_return'].mean():+.1%}")
print(f" Min 2yr ann return: {roll['ann_return'].min():+.1%}")
print(f" Max 2yr ann return: {roll['ann_return'].max():+.1%}")
print(f" % positive 2yr: {win_pct:.0%}")
print(f" Mean 2yr Sharpe: {roll['sharpe'].mean():.2f}")
print()
# 2. Top-N sensitivity
print("--- 2. Top-N Sensitivity ---\n")
header = f" {'Top-N':<8}"
for label in ["W1: rev+gap", "W2: earn+rev", "Baseline"]:
header += f" | {'CAGR':>8} {'Sharpe':>8} {'MaxDD':>8}"
print(header)
print(" " + "-" * 100)
for top_n in [5, 10, 15, 20]:
line = f" {top_n:<8}"
for func in [winner1_func, winner2_func, baseline_func]:
w = make_strategy(stocks, func, top_n=top_n)
eq = run_backtest(w, stocks)
s = compute_stats(eq, "")
line += f" | {s['cagr']:>+7.1%} {s['sharpe']:>8.2f} {s['maxdd']:>+7.1%}"
print(line)
# 3. Rebalance frequency
print("\n--- 3. Rebalance Frequency ---\n")
header = f" {'Rebal':<8}"
for label in ["W1: rev+gap", "W2: earn+rev", "Baseline"]:
header += f" | {'CAGR':>8} {'Sharpe':>8}"
print(header)
print(" " + "-" * 75)
for rebal in [5, 10, 21, 42]:
line = f" {rebal}d{'':<5}"
for func in [winner1_func, winner2_func, baseline_func]:
w = make_strategy(stocks, func, top_n=10, rebal_freq=rebal)
eq = run_backtest(w, stocks)
s = compute_stats(eq, "")
line += f" | {s['cagr']:>+7.1%} {s['sharpe']:>8.2f}"
print(line)
# 4. Transaction cost sensitivity
print("\n--- 4. Transaction Cost Sensitivity ---\n")
header = f" {'Cost':<8}"
for label in ["W1: rev+gap", "W2: earn+rev", "Baseline"]:
header += f" | {'CAGR':>8} {'Sharpe':>8}"
print(header)
print(" " + "-" * 75)
for cost in [0, 0.001, 0.002, 0.005]:
line = f" {cost*10000:.0f}bps{'':<4}"
for func in [winner1_func, winner2_func, baseline_func]:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks, cost=cost)
s = compute_stats(eq, "")
line += f" | {s['cagr']:>+7.1%} {s['sharpe']:>8.2f}"
print(line)
# 5. Drawdown analysis
print("\n--- 5. Drawdown Episodes ---\n")
for label, func in [("W1: rev_vol+gap_up", winner1_func),
("W2: earn_drift+rev_vol", winner2_func),
("Baseline: rec+mom", baseline_func)]:
w = make_strategy(stocks, func, top_n=10)
eq = run_backtest(w, stocks)
dd = drawdown_analysis(eq)
print(f" {label}:")
if dd.empty:
print(" No significant drawdowns")
else:
for _, row in dd.iterrows():
print(f" {row['start']} → {row['trough']} → {row['end']}: "
f"{row['depth']:+.1%} ({row['duration_days']}d)")
print()
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--market", default="both", choices=["us", "cn", "both"])
args = parser.parse_args()
if args.market in ("us", "both"):
prices = data_manager.load("us")
stocks = prices.drop(columns=["SPY"], errors="ignore")
run_us(stocks)
if args.market in ("cn", "both"):
prices = data_manager.load("cn")
stocks = prices.drop(columns=["000300.SS"], errors="ignore")
run_cn(stocks)
if __name__ == "__main__":
main()

259
factor_yearly_fresh.py
View File

@@ -1,259 +0,0 @@
"""
Rebalancing frequency comparison: daily (1d) vs weekly (5d) vs biweekly (10d) vs monthly (21d).
Shows yearly returns and max drawdown for each frequency, for all champion strategies.
"""
from __future__ import annotations
import warnings
import numpy as np
import pandas as pd
import data_manager
from factor_loop import (
strat, bt, stats, combo,
f_rec_mom, f_rec_126, f_rec_63,
f_mom_12_1, f_mom_6_1, f_mom_intermediate,
f_above_ma200, f_golden_cross,
f_up_volume_proxy, f_gap_up_freq,
f_rec_mom_filtered, f_down_resilience,
f_up_capture, f_52w_high, f_str_10d,
f_earnings_drift, f_reversal_vol,
)
warnings.filterwarnings("ignore")
INITIAL = 10_000
REBAL_CONFIGS = [
("daily", 1),
("weekly", 5),
("biweekly", 10),
("monthly", 21),
]
def f_quality_mom(p):
mom = f_mom_12_1(p)
consist = (p.pct_change() > 0).astype(float).rolling(252, min_periods=126).mean()
mom_r = mom.rank(axis=1, pct=True, na_option="keep")
con_r = consist.rank(axis=1, pct=True, na_option="keep")
up_r = f_up_volume_proxy(p).rank(axis=1, pct=True, na_option="keep")
return 0.4 * mom_r + 0.3 * con_r + 0.3 * up_r
def f_mom_x_gap(p):
return (f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep") *
f_gap_up_freq(p).rank(axis=1, pct=True, na_option="keep"))
def run_equity(func, prices, rebal=21, cost=0.001):
w = strat(prices, func, top_n=10, rebal=rebal)
eq = bt(w, prices, cost=cost)
return eq / eq.iloc[0] * INITIAL
def year_returns(eq: pd.Series) -> dict[int, float]:
dr = eq.pct_change().fillna(0)
return {y: float((1 + dr[dr.index.year == y]).prod() - 1)
for y in sorted(dr.index.year.unique())}
def max_drawdown(eq: pd.Series) -> float:
rm = eq.cummax()
dd = (eq - rm) / rm
return float(dd.min())
def max_drawdown_yearly(eq: pd.Series) -> dict[int, float]:
result = {}
for y in sorted(eq.index.year.unique()):
chunk = eq[eq.index.year == y]
if len(chunk) < 5:
continue
rm = chunk.cummax()
dd = (chunk - rm) / rm
result[y] = float(dd.min())
return result
def cagr(eq: pd.Series) -> float:
dr = eq.pct_change().dropna()
if len(dr) < 100:
return np.nan
ny = len(dr) / 252
tot = eq.iloc[-1] / eq.iloc[0] - 1
return (1 + tot) ** (1 / ny) - 1
def sharpe(eq: pd.Series) -> float:
dr = eq.pct_change().dropna()
if len(dr) < 100 or dr.std() == 0:
return np.nan
return float(dr.mean() / dr.std() * np.sqrt(252))
def turnover_annual(func, prices, rebal):
"""Estimate annualised turnover (one-way)."""
w = strat(prices, func, top_n=10, rebal=rebal)
daily_turn = w.diff().abs().sum(axis=1).mean()
return daily_turn * 252
def print_by_year(strat_defs, prices, bench_eq, bench_label, market_label, years):
"""For each year, print a table: strategies as rows, rebal frequencies as columns."""
freq_labels = [r for r, _ in REBAL_CONFIGS]
# Pre-compute all equities and returns
all_eqs = {} # {(sname, freq): equity}
for sname, func in strat_defs.items():
for rlabel, rdays in REBAL_CONFIGS:
all_eqs[(sname, rlabel)] = run_equity(func, prices, rebal=rdays)
all_rets = {} # {(sname, freq): {year: ret}}
for key, eq in all_eqs.items():
all_rets[key] = year_returns(eq)
bench_rets = year_returns(bench_eq)
snames = list(strat_defs.keys())
name_w = max(len(s) for s in snames) + 1
for year in years:
line_w = name_w + 4 + 20 * (len(freq_labels) + 1)
print(f"\n{'=' * line_w}")
print(f" {market_label} — {year} (fresh $10,000)")
print(f"{'=' * line_w}")
# Header
print(f" {'Strategy':<{name_w}}", end="")
for f in freq_labels:
print(f" {f:>18}", end="")
print(f" {bench_label:>18}")
print(f" {'-'*name_w}", end="")
for _ in range(len(freq_labels) + 1):
print(f" {'-'*18}", end="")
print()
for sname in snames:
print(f" {sname:<{name_w}}", end="")
# Find best freq for this strategy this year
freq_vals = {}
for f in freq_labels:
r = all_rets[(sname, f)].get(year)
if r is not None and abs(r) > 0.0005:
freq_vals[f] = r
best_f = max(freq_vals, key=freq_vals.get) if freq_vals else None
for f in freq_labels:
r = all_rets[(sname, f)].get(year)
if r is not None and abs(r) > 0.0005:
v = INITIAL * (1 + r)
marker = " ★" if f == best_f else " "
print(f" ${v:>9,.0f} {r:>+5.0%}{marker}", end="")
else:
print(f" {'—':>18}", end="")
# Benchmark (same for all strategies)
br = bench_rets.get(year)
if br is not None and abs(br) > 0.0005:
print(f" ${INITIAL*(1+br):>9,.0f} {br:>+5.0%} ", end="")
else:
print(f" {'—':>18}", end="")
print()
# Best strategy per freq
print(f" {'-'*name_w}", end="")
for _ in range(len(freq_labels) + 1):
print(f" {'-'*18}", end="")
print()
print(f" {'BEST':<{name_w}}", end="")
for f in freq_labels:
best_r = -999
best_s = ""
for sname in snames:
r = all_rets[(sname, f)].get(year)
if r is not None and abs(r) > 0.0005 and r > best_r:
best_r = r
best_s = sname
if best_r > -999:
print(f" ${INITIAL*(1+best_r):>9,.0f} {best_r:>+5.0%} ", end="")
else:
print(f" {'—':>18}", end="")
# bench
br = bench_rets.get(year)
if br is not None and abs(br) > 0.0005:
print(f" ${INITIAL*(1+br):>9,.0f} {br:>+5.0%} ", end="")
else:
print(f" {'—':>18}", end="")
print()
def main():
years = list(range(2015, 2027))
# ===== US =====
print(f"\n{'#'*130}")
print(f"{'#'*50} US MARKET {'#'*50}")
print(f"{'#'*130}")
prices_us = data_manager.load("us")
bench_us = prices_us["SPY"].dropna()
stocks_us = prices_us.drop(columns=["SPY"], errors="ignore")
eq_spy = bench_us / bench_us.iloc[0] * INITIAL
us_strats = {
"rec_mfilt+deep×upvol": combo([
(f_rec_mom_filtered, 0.5),
(combo([(f_rec_126, 0.5), (f_up_volume_proxy, 0.5)]), 0.5),
]),
"ma200+mom7m+rec126": combo([
(f_above_ma200, 0.33), (f_mom_intermediate, 0.33), (f_rec_126, 0.34)
]),
"rec_mfilt+ma200": combo([
(f_rec_mom_filtered, 0.5), (f_above_ma200, 0.5)
]),
"mom7m+rec126": combo([
(f_mom_intermediate, 0.5), (f_rec_126, 0.5)
]),
"BASELINE:rec+mom": f_rec_mom,
}
print_by_year(us_strats, stocks_us, eq_spy, "SPY", "US", years)
# ===== CN =====
print(f"\n\n{'#'*130}")
print(f"{'#'*50} CN MARKET {'#'*50}")
print(f"{'#'*130}")
prices_cn = data_manager.load("cn")
bench_cn = prices_cn["000300.SS"].dropna() if "000300.SS" in prices_cn.columns else None
stocks_cn = prices_cn.drop(columns=["000300.SS"], errors="ignore")
cn_strats = {
"up_cap+quality_mom": combo([
(f_up_capture, 0.5), (f_quality_mom, 0.5)
]),
"down_resil+qual_mom": combo([
(f_down_resilience, 0.5), (f_quality_mom, 0.5)
]),
"rec63+mom×gap": combo([
(f_rec_63, 0.5), (f_mom_x_gap, 0.5)
]),
"up_cap+mom×gap": combo([
(f_up_capture, 0.5), (f_mom_x_gap, 0.5)
]),
"BASELINE:rec+mom": f_rec_mom,
}
if bench_cn is not None:
eq_csi = bench_cn / bench_cn.iloc[0] * INITIAL
else:
eq_csi = pd.Series(dtype=float)
print_by_year(cn_strats, stocks_cn, eq_csi, "CSI300", "CN", years)
if __name__ == "__main__":
main()

219
factor_yearly_report.py
View File

@@ -1,219 +0,0 @@
"""
Yearly ROI report for champion strategies vs SPY, starting from $10,000.
"""
from __future__ import annotations
import warnings
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from factor_loop import (
strat, bt, stats, combo,
f_rec_mom, f_rec_126, f_rec_63,
f_mom_12_1, f_mom_6_1, f_mom_intermediate,
f_above_ma200, f_golden_cross,
f_up_volume_proxy, f_gap_up_freq,
f_rec_mom_filtered, f_down_resilience,
f_up_capture, f_52w_high, f_str_10d,
f_earnings_drift, f_reversal_vol,
)
warnings.filterwarnings("ignore")
INITIAL = 10_000
def f_quality_mom(p):
mom = f_mom_12_1(p)
consist = (p.pct_change() > 0).astype(float).rolling(252, min_periods=126).mean()
mom_r = mom.rank(axis=1, pct=True, na_option="keep")
con_r = consist.rank(axis=1, pct=True, na_option="keep")
up_r = f_up_volume_proxy(p).rank(axis=1, pct=True, na_option="keep")
return 0.4 * mom_r + 0.3 * con_r + 0.3 * up_r
def f_mom_x_gap(p):
return (f_mom_12_1(p).rank(axis=1, pct=True, na_option="keep") *
f_gap_up_freq(p).rank(axis=1, pct=True, na_option="keep"))
def run_equity(func, prices, cost=0.001):
w = strat(prices, func, top_n=10)
eq = bt(w, prices, cost=cost)
return eq / eq.iloc[0] * INITIAL
def yearly_table(equities: dict[str, pd.Series], title: str):
print(f"\n{'='*130}")
print(f" {title}")
print(f" Starting capital: ${INITIAL:,.0f}")
print(f"{'='*130}")
names = list(equities.keys())
all_years = sorted(set(y for eq in equities.values() for y in eq.index.year.unique()))
# Header
print(f"\n {'Year':<6}", end="")
for n in names:
print(f" | {n[:24]:>24}", end="")
print()
print(f" {'-'*6}", end="")
for _ in names:
print(f"-+-{'-'*24}", end="")
print()
# Track portfolio values
year_end_vals = {n: INITIAL for n in names}
for year in all_years:
print(f" {year:<6}", end="")
for n in names:
eq = equities[n]
yr_data = eq[eq.index.year == year]
if len(yr_data) < 2:
print(f" | {'—':>24}", end="")
continue
start_val = yr_data.iloc[0]
end_val = yr_data.iloc[-1]
ret = end_val / start_val - 1
year_end_vals[n] = end_val
# Show both return % and portfolio value
print(f" | {ret:>+7.1%} ${end_val:>12,.0f}", end="")
print()
# Summary rows
print(f" {'-'*6}", end="")
for _ in names:
print(f"-+-{'-'*24}", end="")
print()
# Total return
print(f" {'Total':<6}", end="")
for n in names:
eq = equities[n]
total = eq.iloc[-1] / INITIAL - 1
print(f" | {total:>+7.0%} ${eq.iloc[-1]:>12,.0f}", end="")
print()
# CAGR
print(f" {'CAGR':<6}", end="")
for n in names:
eq = equities[n]
ny = len(eq) / 252
total = eq.iloc[-1] / INITIAL - 1
cagr = (1 + total) ** (1 / ny) - 1
print(f" | {cagr:>+7.1%} {'':>12}", end="")
print()
# Sharpe
print(f" {'Sharpe':<6}", end="")
for n in names:
eq = equities[n]
dr = eq.pct_change().dropna()
sh = dr.mean() / dr.std() * np.sqrt(252) if dr.std() > 0 else 0
print(f" | {sh:>7.2f} {'':>12}", end="")
print()
# Max DD
print(f" {'MaxDD':<6}", end="")
for n in names:
eq = equities[n]
rm = eq.cummax()
dd = ((eq - rm) / rm).min()
print(f" | {dd:>+7.1%} {'':>12}", end="")
print()
# Best/Worst year
print(f" {'Best':<6}", end="")
for n in names:
eq = equities[n]
dr = eq.pct_change().fillna(0)
yr_rets = {y: float((1 + dr[dr.index.year == y]).prod() - 1) for y in all_years}
# skip warmup year
active = {y: r for y, r in yr_rets.items() if abs(r) > 0.001}
if active:
best_y = max(active, key=active.get)
print(f" | {active[best_y]:>+7.1%} ({best_y}) ", end="")
else:
print(f" | {'—':>24}", end="")
print()
print(f" {'Worst':<6}", end="")
for n in names:
eq = equities[n]
dr = eq.pct_change().fillna(0)
yr_rets = {y: float((1 + dr[dr.index.year == y]).prod() - 1) for y in all_years}
active = {y: r for y, r in yr_rets.items() if abs(r) > 0.001}
if active:
worst_y = min(active, key=active.get)
print(f" | {active[worst_y]:>+7.1%} ({worst_y}) ", end="")
else:
print(f" | {'—':>24}", end="")
print()
def main():
# ===== US =====
prices_us = data_manager.load("us")
bench_us = prices_us["SPY"].dropna()
stocks_us = prices_us.drop(columns=["SPY"], errors="ignore")
eq_spy = bench_us / bench_us.iloc[0] * INITIAL
us_strats = {
"rec_mfilt+deep×upvol": combo([
(f_rec_mom_filtered, 0.5),
(combo([(f_rec_126, 0.5), (f_up_volume_proxy, 0.5)]), 0.5),
]),
"ma200+mom7m+rec126": combo([
(f_above_ma200, 0.33), (f_mom_intermediate, 0.33), (f_rec_126, 0.34)
]),
"rec_mfilt+ma200": combo([
(f_rec_mom_filtered, 0.5), (f_above_ma200, 0.5)
]),
"mom7m+rec126": combo([
(f_mom_intermediate, 0.5), (f_rec_126, 0.5)
]),
"BASELINE:rec+mom": f_rec_mom,
}
us_equities = {}
for name, func in us_strats.items():
us_equities[name] = run_equity(func, stocks_us)
us_equities["SPY (Benchmark)"] = eq_spy
yearly_table(us_equities, "US MARKET — Champion Strategies vs SPY — $10,000 Starting Capital")
# ===== CN =====
prices_cn = data_manager.load("cn")
bench_cn = prices_cn["000300.SS"].dropna() if "000300.SS" in prices_cn.columns else None
stocks_cn = prices_cn.drop(columns=["000300.SS"], errors="ignore")
cn_strats = {
"up_cap+quality_mom": combo([
(f_up_capture, 0.5), (f_quality_mom, 0.5)
]),
"down_resil+qual_mom": combo([
(f_down_resilience, 0.5), (f_quality_mom, 0.5)
]),
"rec63+mom×gap": combo([
(f_rec_63, 0.5), (f_mom_x_gap, 0.5)
]),
"up_cap+mom×gap": combo([
(f_up_capture, 0.5), (f_mom_x_gap, 0.5)
]),
"BASELINE:rec+mom": f_rec_mom,
}
cn_equities = {}
for name, func in cn_strats.items():
cn_equities[name] = run_equity(func, stocks_cn)
if bench_cn is not None:
cn_equities["CSI300 (Benchmark)"] = bench_cn / bench_cn.iloc[0] * INITIAL
yearly_table(cn_equities, "CN MARKET — Champion Strategies vs CSI 300 — $10,000 Starting Capital")
if __name__ == "__main__":
main()

116
research/ARCHIVE.md Normal file
View File

@@ -0,0 +1,116 @@
# Removed research and legacy code
Removed on 2026-10-09 in the repo clean-up. Every file is in git history; restore one with
```bash
git show 92a2a58:<path> > <path>
```
Nothing live (trader.py, bridge.py, backtest_runner.py, precompute_backtest.py, tests, research/lab)
imported any of them. Their conclusions that still matter are in CLAUDE.md and the research/lab docstrings.
## Root factor-mining scripts (2026-04, before the research/lab harness; superseded by research/lab/factor_scan.py on the PIT panels)
| file | purpose |
|---|---|
| `factor_backtest.py` | Backtest best factor combinations with yearly return breakdown. |
| `factor_deep_analysis.py` | Deep factor analysis — orthogonality, proper correlations, residual alpha. |
| `factor_final_check.py` | Final robustness check on champions from the discovery loop. |
| `factor_loop.py` | Iterative Factor Discovery Loop. |
| `factor_real_backtest.py` | Factor research v2 — Portfolio-first approach. |
| `factor_research.py` | Factor Research Script — Professional QR-style factor mining. |
| `factor_robustness.py` | Robustness checks for winning factor strategies. |
| `factor_yearly_fresh.py` | Rebalancing frequency comparison: daily (1d) vs weekly (5d) vs biweekly (10d) vs monthly (21d). |
| `factor_yearly_report.py` | Yearly ROI report for champion strategies vs SPY, starting from $10,000. |
## Old runners superseded by the monitor service, precompute_backtest.py and research/lab/evaluate.py
| file | purpose |
|---|---|
| `yearly_sweep.py` | Run all US strategies and report yearly performance vs SPY. |
| `scripts/run_daily.py` | Daily live-trading update for all registered strategies. |
| `scripts/run_daily.sh` | cron wrapper for scripts/run_daily.py |
## Strategies no registry, test or live book used
| file | purpose |
|---|---|
| `strategies/trend_rider_v8.py` | TrendRider V8 — V7 + TMF risk-off upgrade. |
| `strategies/adaptive_blend.py` | PIT-Compliant Adaptive Multi-Strategy Blend — Sharpe 1.5+ target. |
| `strategies/cross_asset_momentum.py` | Cross-asset time-series momentum strategy (ETF-only, inherently PIT). |
| `strategies/long_hedged.py` | Market-hedged long-only stock portfolio. |
| `strategies/ls_momentum.py` | Industry-neutral long/short momentum on the S&P 500. |
## research/*.py from 2026-04/05 (written before the research/lab harness of PIT panels, next-open fills and preregistration; the questions that still matter were redone there)
| file | purpose |
|---|---|
| `docs/superpowers/specs/2026-04-17-us-alpha-research-design.md` | design spec for research/us_alpha_* |
| `research/alpha_factors.py` | Alpha factor library — price-only, academically motivated, with a rolling-IC |
| `research/alpha_research.py` | Professional QR-style factor research on the PIT S&P 500 universe. |
| `research/dca_simulation.py` | DCA simulation: $10,000 initial + $5,000 every Feb & Aug from 2017. |
| `research/event_factors.py` | (no docstring) |
| `research/factor_optimize.py` | Factor-level optimization on the point-in-time S&P 500 universe. |
| `research/fetch_historical.py` | Fetch price history for all tickers that were ever S&P 500 members — including |
| `research/interaction_alpha.py` | Interaction / multiplicative factor strategy. |
| `research/ls_momentum_eval.py` | Evaluate the industry-neutral L/S momentum strategy with realistic costs. |
| `research/new_frameworks_eval.py` | New frameworks: Vol trading (#2) + Event-driven simplified (#4). |
| `research/optimize.py` | End-to-end optimization study for the US recovery+momentum strategy family, |
| `research/pit_backtest.py` | Point-in-time backtest runner. |
| `research/pit_comparison.py` | PIT-compliant backtest: mask prices to historical S&P 500 membership. |
| `research/pit_optimization.py` | PIT-compliant strategy optimization. |
| `research/regime_filters.py` | (no docstring) |
| `research/regime_stock_picker_eval.py` | Does V3 regime timing + S&P 500 stock picking improve over either alone? |
| `research/run_interaction.py` | Evaluate interaction/ensemble strategies on 1/3/5/10y PIT windows with a |
| `research/sharpe_blend.py` | PIT-compliant Sharpe 1.5+ blend: V5 ETF timing + PIT stock-picking + cross-asset momentum. |
| `research/single_stock_swing.py` | Single-stock swing trading: adapt V3/V7 concepts to 20 famous stocks. |
| `research/smart_dca_eval.py` | Smart DCA Strategy Evaluation — comprehensive comparison of DCA approaches. |
| `research/sota_ranking.py` | Rank all top strategies head-to-head on the same 10-year PIT-safe data. |
| `research/strategies_plus.py` | Optimization variants of RecoveryMomentumStrategy. |
| `research/strategy_final_report.py` | FINAL REPORT: Strategy improvement results — 10-year yearly backtest. |
| `research/strategy_improvement_eval.py` | Comprehensive strategy improvement evaluation. |
| `research/strategy_improvement_r2.py` | Round 2: Strategy improvement iteration. |
| `research/strategy_improvement_r3.py` | Round 3: Signal-level ensemble and enhanced factor combo. |
| `research/strategy_improvement_r4.py` | Round 4 - Final iteration: Optimize the winning EnhFC strategy. |
| `research/strategy_risk_managed_eval.py` | Risk-Managed Ensemble Strategy Evaluation. |
| `research/strategy_risk_managed_r2.py` | Round 2: Risk-Managed Ensemble with DD-reactive approach. |
| `research/strategy_sharpe_boost.py` | Sharpe boost research: blend pure momentum into the Ensemble signal. |
| `research/strategy_sharpe_boost_v2.py` | Sharpe boost v2: Dispersion-adaptive exposure + momentum blend. |
| `research/strategy_sharpe_boost_v3.py` | Sharpe boost v3: Concentration + rebalance frequency + trailing alpha. |
| `research/strategy_sharpe_boost_v4.py` | Sharpe boost v4: Long holding period (42d rebal) is the key lever. |
| `research/strategy_sharpe_boost_v5.py` | Sharpe boost v5: Fine-tune DD dampener on top of the Sharpe 1.52 config. |
| `research/strategy_sweep.py` | Unified 3/5/10-year PIT backtest for every production strategy. |
| `research/trade_analysis.py` | Trade-level analysis of SharpeBoostedEnsembleStrategy. |
| `research/trend_rider_p0.py` | P0 robustness validation for TrendRiderV3. |
| `research/trend_rider_robustness.py` | Robustness analysis for TrendRiderV3. |
| `research/trend_rider_v5_eval.py` | Evaluate TrendRiderV5 vs V3 baseline and benchmarks. |
| `research/trend_rider_v6_eval.py` | Evaluate TrendRiderV6 vs V5 baseline. |
| `research/us_alpha_pipeline.py` | (no docstring) |
| `research/us_alpha_report.py` | (no docstring) |
| `research/us_combo_sweep.py` | (no docstring) |
| `research/us_fundamentals.py` | (no docstring) |
| `research/us_universe.py` | (no docstring) |
| `research/v5_drawdown_trace.py` | Trace where V3/V5 maximum drawdowns occur and what holdings they had. |
| `research/v5_p0_validate.py` | P0 validation for TrendRiderV5 — walk-forward + bootstrap. |
| `research/v6_voltarget.py` | Vol-targeting overlay on V5/V6 blends — tests if dynamic exposure scaling |
| `research/v7_breakthrough_eval.py` | Three directions to break V7+VT36's ceiling (61% Ann, Sharpe 1.89). |
| `research/v7_breakthrough_fixed.py` | Fixed re-run for Directions B and C based on review feedback. |
| `research/v7_literature_alpha.py` | Literature-informed alpha research: can we beat V7+VT36? |
| `research/v7_parameter_sweep.py` | V7 parameter sweep: vol-target range + adaptive profit-take variants. |
| `research/v7_synthetic_leverage_eval.py` | Direction 2: V7 regime + synthetic 2x/3x leveraged individual stocks. |
| `research/v7_three_ideas_eval.py` | Test three structural improvements to V7+VT36 identified by independent review. |
| `research/v7_trade_audit.py` | V7+VT36 full trade log: regime changes, instrument switches, PT events, |
## Tests of the removed research modules
| file | tested |
|---|---|
| `tests/test_alpha_signals.py` | research/event_factors.py, research/regime_filters.py |
| `tests/test_fetch_historical.py` | research/fetch_historical.py |
| `tests/test_trend_rider_robustness.py` | research/trend_rider_robustness.py |
| `tests/test_us_alpha_pipeline.py` | research/us_alpha_pipeline.py, research/us_alpha_report.py |
| `tests/test_us_combo_sweep.py` | research/us_combo_sweep.py |
| `tests/test_us_fundamentals.py` | research/us_fundamentals.py |
| `tests/test_us_universe.py` | research/us_universe.py |
Kept on purpose: `research/permanent_yearly.py` (bridge.py and precompute_backtest.py import
its `equity_with_cashflows`).

358
research/alpha_factors.py
View File

@@ -1,358 +0,0 @@
"""
Alpha factor library — price-only, academically motivated, with a rolling-IC
combiner, inverse-vol portfolio weighting, and volatility targeting.
Factors (each returns a cross-sectional DataFrame aligned to prices.index):
mom_12_1 12-1 month momentum (Jegadeesh & Titman 1993).
mom_7_1 Intermediate 7-1m momentum (Novy-Marx 2012).
mom_residual Market-residualized 12-1m (Blitz-Huij-Martens 2011).
rev_1m 1-month reversal × -1 (Jegadeesh 1990 / short-term reversal).
w52_high Price / 52-week high, proximity factor (George & Hwang 2004).
max5_neg -avg(top-5 daily returns past 21d) — lottery/MAX (Bali-Cakici-Whitelaw 2011).
idio_vol_neg -residual-vol from 60d market regression (Ang-Hodrick-Xing-Zhang 2006).
low_beta -60d market beta (Betting Against Beta, Frazzini-Pedersen 2014 variant).
trend_strength Slope / RMSE from 63d log-price regression.
recovery_63 Price / 63d low - 1 (project-native, V-rebound proxy).
Combiner:
- Cross-sectional percentile-rank each factor (NaN = keep).
- For each day, blend factors with weights proportional to the rolling
252-day Information Coefficient (Spearman rank corr vs forward 21d return).
- Weights are lagged by 21 days to avoid lookahead; negative-IC factors are
sign-flipped before weighting (so all contribute positively when confident).
Portfolio:
- Rank composite score, pick top_n (default 15) on a rebalance_freq schedule.
- Inverse-vol weight within top_n (60d realized vol).
- Volatility-target the whole portfolio to target_vol (default 18%) using a
trailing 60-day portfolio-vol estimate; exposure clipped to [0.3, 1.5].
- Shift(1) at the end for T-1 signal delivery, matching the project convention.
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from strategies.base import Strategy
# ---------------------------------------------------------------------------
# Factor primitives
# ---------------------------------------------------------------------------
def _pct(p, n):
return p.pct_change(n, fill_method=None)
def f_mom_12_1(p):
return p.shift(21).pct_change(231, fill_method=None)
def f_mom_7_1(p):
return p.shift(21).pct_change(126, fill_method=None)
def f_rev_1m(p):
return -p.pct_change(21, fill_method=None)
def f_w52_high(p):
roll_max = p.rolling(252, min_periods=200).max()
return p / roll_max - 1 # ≤0, closer to 0 = near 52w high
def f_max5_neg(p):
ret = p.pct_change(fill_method=None)
# Mean of top-5 returns over the last 21 trading days; negate.
top5 = ret.rolling(21, min_periods=15).apply(
lambda x: np.mean(np.sort(x)[-5:]) if np.isfinite(x).sum() >= 5 else np.nan,
raw=True,
)
return -top5
def f_recovery_63(p):
return p / p.rolling(63, min_periods=60).min() - 1
def f_trend_strength(p):
"""
Vectorized log-price trend strength: rolling OLS slope ÷ residual RMSE on a
63-day window. t-stat-like measure of directional trend quality.
"""
logp = np.log(p.replace(0, np.nan))
n = 63
idx = np.arange(n, dtype=float)
idx_c = idx - idx.mean()
idx_var = (idx_c ** 2).sum()
# E[x·y] over the window: rolling sum of (idx·y) simplified via decomposition:
# Σ (i - ī)(y - ȳ) = Σ i·y - n·ī·ȳ (but ī is constant so just: Σ (i-ī)·y)
# We compute Σ (i-ī)·y as a rolling window-weighted sum.
weights = idx_c # shape (n,)
def rolling_weighted(series_df, w):
"""Σ_{k=0..n-1} w[k] * y[t-(n-1)+k] for each column, vectorized."""
arr = series_df.values
T, K = arr.shape
out = np.full_like(arr, np.nan, dtype=float)
# Convolution across time axis per column:
for k in range(K):
col = arr[:, k]
# Use np.convolve with reversed weights (equivalent to correlate)
conv = np.convolve(col, w[::-1], mode="valid")
out[n - 1:, k] = conv
return pd.DataFrame(out, index=series_df.index, columns=series_df.columns)
# rolling mean and var for log-price
roll_mean = logp.rolling(n, min_periods=n).mean()
# numerator: Σ (i-ī)(y - ȳ) = Σ (i-ī)·y (since Σ(i-ī) = 0)
num = rolling_weighted(logp.fillna(0.0), weights)
slope = num / idx_var
# Residual variance: Σ(y - ȳ)² / n - slope² * idx_var / n
var_y = logp.rolling(n, min_periods=n).var(ddof=0)
resid_var = (var_y - (slope ** 2) * idx_var / n).clip(lower=1e-18)
rmse = np.sqrt(resid_var)
ts = slope / rmse
# mask rows where the window contained any NaN
valid = logp.rolling(n, min_periods=n).count() == n
return ts.where(valid)
def _rolling_beta_and_residvol(p, mkt_ret, window=60):
"""Return (beta, residual_vol) DataFrames aligned to prices.index."""
ret = p.pct_change(fill_method=None)
mkt = mkt_ret.reindex(p.index)
def pair(stock_ret):
cov = stock_ret.rolling(window, min_periods=window).cov(mkt)
var = mkt.rolling(window, min_periods=window).var()
beta = cov / var
# Residual vol via: var(stock) - beta^2 * var(mkt) (simplification)
var_stock = stock_ret.rolling(window, min_periods=window).var()
resid_var = (var_stock - beta ** 2 * var) .clip(lower=0)
resid_vol = np.sqrt(resid_var)
return beta, resid_vol
betas = {}
resid_vols = {}
for col in ret.columns:
b, rv = pair(ret[col])
betas[col] = b
resid_vols[col] = rv
return pd.DataFrame(betas), pd.DataFrame(resid_vols)
def f_mom_residual(p, mkt_ret, betas=None, window=60):
if betas is None:
betas, _ = _rolling_beta_and_residvol(p, mkt_ret, window=window)
# 12-1m cumulative residual return = cum stock ret - beta * cum mkt ret.
# Reindex mkt_ret to p.index so arithmetic below does not produce a union
# index (which would corrupt downstream shape assumptions).
mkt_aligned = mkt_ret.reindex(p.index)
stock_cum = p.shift(21).pct_change(231, fill_method=None)
mkt_cum_ret = (1 + mkt_aligned).rolling(231).apply(lambda x: np.prod(x) - 1, raw=True)
mkt_cum = mkt_cum_ret.shift(21)
out = stock_cum.sub(betas.mul(mkt_cum, axis=0), fill_value=np.nan)
return out.reindex(p.index)
# ---------------------------------------------------------------------------
# Cross-sectional rank helper
# ---------------------------------------------------------------------------
def xsec_rank(df: pd.DataFrame) -> pd.DataFrame:
return df.rank(axis=1, pct=True, na_option="keep")
# ---------------------------------------------------------------------------
# Rolling IC computation
# ---------------------------------------------------------------------------
def rolling_ic(factor_rank: pd.DataFrame, fwd_ret: pd.DataFrame,
window: int = 252) -> pd.Series:
"""Daily Spearman IC = rank(factor) vs rank(fwd_ret); rolling mean."""
fr = fwd_ret.rank(axis=1, pct=True, na_option="keep")
# Per-day pearson corr of rank-transformed ≡ Spearman.
per_day_ic = factor_rank.corrwith(fr, axis=1)
return per_day_ic.rolling(window, min_periods=window // 2).mean()
def _rolling_ls_sharpe(factor_rank: pd.DataFrame,
prices: pd.DataFrame,
window: int = 252,
rebal: int = 21,
tcost: float = 0.001) -> pd.Series:
"""
Rolling realized Sharpe of a long-top-decile / short-bottom-decile portfolio
constructed on `factor_rank`, rebalanced every `rebal` trading days, with
proportional turnover cost `tcost`. Used as a factor-quality weight.
Returned series is aligned to `prices.index` and the Sharpe at day t is
computed from returns over [t-window, t].
"""
long_mask = factor_rank >= 0.9
short_mask = factor_rank <= 0.1
# Rebalance: hold the mask constant between rebal dates
rebal_mask = pd.Series(False, index=factor_rank.index)
rebal_mask.iloc[::rebal] = True
long_w = long_mask.astype(float).div(long_mask.sum(axis=1).replace(0, np.nan), axis=0)
short_w = short_mask.astype(float).div(short_mask.sum(axis=1).replace(0, np.nan), axis=0)
long_w[~rebal_mask] = np.nan
short_w[~rebal_mask] = np.nan
long_w = long_w.ffill().fillna(0.0)
short_w = short_w.ffill().fillna(0.0)
rets = prices.pct_change(fill_method=None)
long_ret = (long_w.shift(1) * rets).sum(axis=1)
short_ret = (short_w.shift(1) * rets).sum(axis=1)
long_turn = long_w.diff().abs().sum(axis=1).fillna(0.0)
short_turn = short_w.diff().abs().sum(axis=1).fillna(0.0)
ls_ret = (long_ret - short_ret) - (long_turn + short_turn) * tcost
ls_ret = ls_ret.fillna(0.0)
mean = ls_ret.rolling(window, min_periods=window // 2).mean()
std = ls_ret.rolling(window, min_periods=window // 2).std()
sharpe = (mean / std) * np.sqrt(252)
return sharpe
# ---------------------------------------------------------------------------
# Strategy
# ---------------------------------------------------------------------------
class AlphaFactorStrategy(Strategy):
"""
Multi-factor long-only with rolling LS-Sharpe-weighted signal blend,
inverse-vol weighting, and portfolio-level volatility targeting.
Why LS-Sharpe and not IC?
IC (rank-forward correlation) measures directional accuracy but ignores
the magnitude of cross-sectional dispersion. Two factors with identical
IC can have very different P&L. Empirically on this sample rev_1m has
IC t-stat +5 but LS Sharpe -12 — its top decile are freshly crashed
names that keep crashing. We weight by a lagged 252d rolling LS-Sharpe
(top-decile minus bottom-decile, monthly rebalance, 10bps t-cost) and
floor weights at zero so demoted factors simply drop out.
The strategy requires a market return series (e.g. SPY pct_change) passed
at construction time — it is NOT derived from data inside generate_signals,
because the cross-sectional universe contains only selected tickers while
we want a stable market benchmark for beta/residual computations.
"""
def __init__(
self,
mkt_returns: pd.Series,
top_n: int = 15,
rebal_freq: int = 10,
vol_window: int = 60,
vol_target_annual: float | None = 0.18,
ic_window: int = 252,
exposure_clip: tuple[float, float] = (0.30, 1.50),
fwd_window: int = 21,
weight_scheme: str = "ls_sharpe", # {"ls_sharpe", "ic", "equal"}
min_weight: float = 0.0, # floor per-factor weight (0 = drop losers)
):
self.mkt_returns = mkt_returns
self.top_n = top_n
self.rebal_freq = rebal_freq
self.vol_window = vol_window
self.vol_target_annual = vol_target_annual
self.ic_window = ic_window
self.exposure_clip = exposure_clip
self.fwd_window = fwd_window
self.weight_scheme = weight_scheme
self.min_weight = min_weight
# ---- Factor matrix ----
def compute_factors(self, data: pd.DataFrame) -> dict[str, pd.DataFrame]:
betas, resid_vol = _rolling_beta_and_residvol(
data, self.mkt_returns, window=self.vol_window)
factors = {
"mom_12_1": f_mom_12_1(data),
"mom_7_1": f_mom_7_1(data),
"mom_residual": f_mom_residual(data, self.mkt_returns, betas=betas),
"rev_1m": f_rev_1m(data),
"w52_high": f_w52_high(data),
"max5_neg": f_max5_neg(data),
"recovery_63": f_recovery_63(data),
"trend_strength": f_trend_strength(data),
"idio_vol_neg": -resid_vol,
"low_beta": -betas,
}
return factors
# ---- Full pipeline ----
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
factors = self.compute_factors(data)
ranks = {k: xsec_rank(v) for k, v in factors.items()}
if self.weight_scheme == "ic":
fwd_ret = data.shift(-self.fwd_window) / data - 1
weight_series = {
k: rolling_ic(ranks[k], fwd_ret, window=self.ic_window).shift(self.fwd_window)
for k in ranks
}
elif self.weight_scheme == "ls_sharpe":
weight_series = {
k: _rolling_ls_sharpe(ranks[k], data,
window=self.ic_window,
rebal=21, tcost=0.001).shift(self.fwd_window)
for k in ranks
}
elif self.weight_scheme == "equal":
weight_series = {k: pd.Series(1.0, index=ranks[k].index) for k in ranks}
else:
raise ValueError(f"unknown weight_scheme {self.weight_scheme!r}")
composite = None
weight_norm = None
for k, rk in ranks.items():
w = weight_series[k].reindex(rk.index).fillna(0.0)
if self.min_weight is not None:
w = w.where(w > self.min_weight, 0.0)
contrib = rk.mul(w, axis=0)
composite = contrib if composite is None else composite.add(contrib, fill_value=0.0)
abs_w = w.abs()
weight_norm = abs_w if weight_norm is None else weight_norm.add(abs_w, fill_value=0)
weight_norm = weight_norm.replace(0, np.nan)
composite = composite.div(weight_norm, axis=0)
# Top-N selection.
sel_rank = composite.rank(axis=1, ascending=False, na_option="bottom")
n_valid = composite.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (sel_rank <= self.top_n) & enough.values.reshape(-1, 1)
# Inverse-vol weighting within top_n.
rets = data.pct_change(fill_method=None)
vol = rets.rolling(self.vol_window, min_periods=self.vol_window).std()
inv_vol = (1.0 / vol.replace(0, np.nan)).where(top_mask, 0.0).fillna(0.0)
row_sums = inv_vol.sum(axis=1).replace(0, np.nan)
weights = inv_vol.div(row_sums, axis=0).fillna(0.0)
# Rebalance schedule.
warmup = max(252, self.vol_window + 21, self.ic_window + self.fwd_window)
rebal_mask = pd.Series(False, index=data.index)
rebal_idx = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_idx] = True
weights[~rebal_mask] = np.nan
weights = weights.ffill().fillna(0.0)
weights.iloc[:warmup] = 0.0
# Volatility targeting at the portfolio level.
if self.vol_target_annual is not None:
# Use returns of the *current* weight vector; vol is trailing realized
# on the applied weights so no lookahead. Compute after ffill.
port_rets = (weights.shift(1) * rets).sum(axis=1)
port_vol = port_rets.rolling(self.vol_window,
min_periods=self.vol_window).std() * np.sqrt(252)
scale = (self.vol_target_annual / port_vol).clip(*self.exposure_clip)
scale = scale.fillna(method="ffill").fillna(1.0)
weights = weights.mul(scale, axis=0)
return weights.shift(1).fillna(0.0)

279
research/alpha_research.py
View File

@@ -1,279 +0,0 @@
"""
Professional QR-style factor research on the PIT S&P 500 universe.
Stage 1 — Factor diagnostics.
IC (Spearman, 21d fwd), t-stat, realistic long-short decile backtest
(monthly rebalance, 10 bps t-cost).
Stage 2 — Composite backtest 1/3/5/10y vs champions.
For 1y window we pre-pend 2y of warmup then score returns on the last 1y
only, so strategies with 252d+ warmup are actually active in-window.
Stage 3 — Config sweep across weight_scheme × top_n × rebal × vol_target.
Outputs CSVs to data/alpha_research_*.csv.
"""
from __future__ import annotations
import os
import warnings
import numpy as np
import pandas as pd
import research.pit_backtest as pit
from research.alpha_factors import (AlphaFactorStrategy, _rolling_beta_and_residvol,
f_mom_12_1, f_mom_7_1, f_rev_1m, f_w52_high,
f_max5_neg, f_recovery_63, f_trend_strength,
xsec_rank, _rolling_ls_sharpe)
from strategies.factor_combo import FactorComboStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=RuntimeWarning)
DATA_DIR = "data"
BENCHMARK = "SPY"
def load():
raw = pit.load_pit_prices()
masked = pit.pit_universe(raw)
if BENCHMARK in raw.columns:
masked[BENCHMARK] = raw[BENCHMARK]
return masked
def warmup_slice(df: pd.DataFrame, years: int, warmup_days: int = 500) -> tuple[pd.DataFrame, pd.Timestamp]:
"""Return (prices_with_warmup, measurement_start). Strategies are fed the
longer series, but metrics must be computed only from measurement_start."""
measurement_start = df.index[-1] - pd.DateOffset(years=years)
first_day = df.index[0]
# Keep all rows between measurement_start - warmup_days and end.
cutoff = max(first_day, measurement_start - pd.Timedelta(days=warmup_days * 1.5))
sliced = df[df.index >= cutoff]
return sliced, measurement_start
def measure(eq: pd.Series, start: pd.Timestamp, name: str = "") -> dict:
eq = eq[eq.index >= start]
# Re-base to 10_000 at start
eq = eq / eq.iloc[0] * 10_000
return pit.summarize(eq, name=name)
# ---------------------------------------------------------------------------
# Stage 1 — Factor diagnostics
# ---------------------------------------------------------------------------
def factor_diagnostics(masked: pd.DataFrame):
print("\n" + "=" * 110)
print("Stage 1 — Factor diagnostics (full 10y PIT, monthly rebal, 10bps t-cost)")
print("=" * 110)
tickers = [c for c in masked.columns if c != BENCHMARK]
prices = masked[tickers]
mkt_ret = masked[BENCHMARK].pct_change(fill_method=None)
betas, resid_vol = _rolling_beta_and_residvol(prices, mkt_ret, window=60)
from research.alpha_factors import f_mom_residual
factor_builders = {
"mom_12_1": lambda: f_mom_12_1(prices),
"mom_7_1": lambda: f_mom_7_1(prices),
"mom_residual": lambda: f_mom_residual(prices, mkt_ret, betas=betas),
"rev_1m": lambda: f_rev_1m(prices),
"w52_high": lambda: f_w52_high(prices),
"max5_neg": lambda: f_max5_neg(prices),
"recovery_63": lambda: f_recovery_63(prices),
"trend_strength": lambda: f_trend_strength(prices),
"idio_vol_neg": lambda: -resid_vol,
"low_beta": lambda: -betas,
}
fwd_21 = prices.shift(-21) / prices - 1
fwd_rank = fwd_21.rank(axis=1, pct=True, na_option="keep")
rows = []
for name, build in factor_builders.items():
fac = build()
fr = xsec_rank(fac)
ic_daily = fr.corrwith(fwd_rank, axis=1).dropna()
ic_mean = ic_daily.mean()
ic_t = ic_mean / (ic_daily.std() / np.sqrt(len(ic_daily))) if len(ic_daily) > 1 else 0.0
ls = realistic_decile_spread(fr, prices, rebal=21, tcost=0.001)
long_only = realistic_top_decile(fr, prices, rebal=21, tcost=0.001)
rows.append({
"factor": name,
"IC_mean": ic_mean, "IC_t": ic_t,
"LS_CAGR": ls["CAGR"], "LS_Sharpe": ls["Sharpe"],
"LO_CAGR": long_only["CAGR"], "LO_Sharpe": long_only["Sharpe"],
"LO_MaxDD": long_only["MaxDD"],
})
df = pd.DataFrame(rows).sort_values("LO_Sharpe", ascending=False)
df.to_csv(os.path.join(DATA_DIR, "alpha_research_factors.csv"), index=False)
print(df.to_string(index=False, formatters={
"IC_mean": "{:+.4f}".format, "IC_t": "{:+.2f}".format,
"LS_CAGR": "{:+.1%}".format, "LS_Sharpe": "{:+.2f}".format,
"LO_CAGR": "{:+.1%}".format, "LO_Sharpe": "{:+.2f}".format,
"LO_MaxDD": "{:.1%}".format,
}))
return df
def realistic_decile_spread(factor_rank, prices, rebal=21, tcost=0.001):
"""Long top-decile minus short bottom-decile, monthly rebal, 10bps t-cost."""
long_mask = factor_rank >= 0.9
short_mask = factor_rank <= 0.1
long_w = long_mask.astype(float).div(long_mask.sum(axis=1).replace(0, np.nan), axis=0)
short_w = short_mask.astype(float).div(short_mask.sum(axis=1).replace(0, np.nan), axis=0)
rebal_mask = pd.Series(False, index=factor_rank.index)
rebal_mask.iloc[::rebal] = True
long_w[~rebal_mask] = np.nan
short_w[~rebal_mask] = np.nan
long_w = long_w.ffill().fillna(0.0)
short_w = short_w.ffill().fillna(0.0)
rets = prices.pct_change(fill_method=None)
ls = ((long_w.shift(1) * rets).sum(axis=1)
- (short_w.shift(1) * rets).sum(axis=1)) \
- (long_w.diff().abs().sum(axis=1).fillna(0.0)
+ short_w.diff().abs().sum(axis=1).fillna(0.0)) * tcost
ls = ls.fillna(0.0).iloc[252:]
eq = (1 + ls).cumprod() * 10_000
return pit.summarize(eq, name="ls")
def realistic_top_decile(factor_rank, prices, rebal=21, tcost=0.001):
"""Long-only top-decile equal-weight portfolio with t-cost."""
long_mask = factor_rank >= 0.9
long_w = long_mask.astype(float).div(long_mask.sum(axis=1).replace(0, np.nan), axis=0)
rebal_mask = pd.Series(False, index=factor_rank.index)
rebal_mask.iloc[::rebal] = True
long_w[~rebal_mask] = np.nan
long_w = long_w.ffill().fillna(0.0)
rets = prices.pct_change(fill_method=None)
port_ret = (long_w.shift(1) * rets).sum(axis=1) \
- long_w.diff().abs().sum(axis=1).fillna(0.0) * tcost
port_ret = port_ret.fillna(0.0).iloc[252:]
eq = (1 + port_ret).cumprod() * 10_000
return pit.summarize(eq, name="lo")
# ---------------------------------------------------------------------------
# Stage 2 — Composite backtest
# ---------------------------------------------------------------------------
def composite_backtest(masked: pd.DataFrame):
print("\n" + "=" * 110)
print("Stage 2 — IC / LS-Sharpe-weighted composite vs champions (1/3/5/10y)")
print("=" * 110)
tickers = [c for c in masked.columns if c != BENCHMARK]
mkt_ret_full = masked[BENCHMARK].pct_change(fill_method=None)
configs = {
"Alpha(LS-Sharpe, tn=15, rebal=10)":
lambda: AlphaFactorStrategy(mkt_ret_full, top_n=15, rebal_freq=10,
vol_target_annual=None, weight_scheme="ls_sharpe"),
"Alpha(LS-Sharpe, tn=15, rebal=21)":
lambda: AlphaFactorStrategy(mkt_ret_full, top_n=15, rebal_freq=21,
vol_target_annual=None, weight_scheme="ls_sharpe"),
"Alpha(LS-Sharpe+VT18, tn=15, rebal=21)":
lambda: AlphaFactorStrategy(mkt_ret_full, top_n=15, rebal_freq=21,
vol_target_annual=0.18, weight_scheme="ls_sharpe"),
"Alpha(IC, tn=15, rebal=21)":
lambda: AlphaFactorStrategy(mkt_ret_full, top_n=15, rebal_freq=21,
vol_target_annual=None, weight_scheme="ic"),
"Recovery+Mom Top10": lambda: RecoveryMomentumStrategy(top_n=10),
"fc_up_cap+mom_gap": lambda: FactorComboStrategy("up_cap+mom_gap",
rebal_freq=21, top_n=10),
}
all_rows = []
for years in (10, 5, 3, 1):
sliced, measurement_start = warmup_slice(masked, years, warmup_days=500)
prices = sliced[tickers]
print(f"\n --- Window: last {years}y "
f"(measure {measurement_start.date()} → {sliced.index[-1].date()}, "
f"warmup from {sliced.index[0].date()}) ---")
spy = sliced[BENCHMARK].dropna()
spy_eq = (spy / spy.iloc[0]) * 10_000
rows = [{"years": years, "strategy": "SPY buy-and-hold",
**{k: v for k, v in measure(spy_eq, measurement_start, "").items()
if k != "name"}}]
for name, factory in configs.items():
strat = factory()
eq = pit.backtest(strategy=strat, prices=prices,
initial_capital=10_000, transaction_cost=0.001)
m = measure(eq, measurement_start, "")
rows.append({"years": years, "strategy": name,
**{k: v for k, v in m.items() if k != "name"}})
for r in rows:
print(f" {r['strategy']:<42s} "
f"CAGR={r['CAGR']*100:>6.1f}% "
f"Sharpe={r['Sharpe']:>5.2f} "
f"Sortino={r['Sortino']:>5.2f} "
f"MaxDD={r['MaxDD']*100:>6.1f}% "
f"Calmar={r['Calmar']:>5.2f}")
all_rows.extend(rows)
df = pd.DataFrame(all_rows)
df.to_csv(os.path.join(DATA_DIR, "alpha_research_composite.csv"), index=False)
return df
# ---------------------------------------------------------------------------
# Stage 3 — Config sweep
# ---------------------------------------------------------------------------
def config_sweep(masked: pd.DataFrame):
print("\n" + "=" * 110)
print("Stage 3 — AlphaFactor config sweep (10y)")
print("=" * 110)
tickers = [c for c in masked.columns if c != BENCHMARK]
prices = masked[tickers]
mkt_ret = masked[BENCHMARK].pct_change(fill_method=None)
rows = []
for scheme in ("ls_sharpe", "ic", "equal"):
for top_n in (10, 15, 20):
for rebal in (10, 21):
for vt in (None, 0.18):
strat = AlphaFactorStrategy(mkt_ret, top_n=top_n, rebal_freq=rebal,
vol_target_annual=vt,
weight_scheme=scheme)
eq = pit.backtest(strat, prices, initial_capital=10_000,
transaction_cost=0.001)
s = pit.summarize(eq, "")
rows.append({"scheme": scheme, "top_n": top_n, "rebal": rebal,
"vt": vt if vt is not None else "none",
"CAGR": s["CAGR"], "Sharpe": s["Sharpe"],
"MaxDD": s["MaxDD"], "Calmar": s["Calmar"]})
df = pd.DataFrame(rows).sort_values("Sharpe", ascending=False)
df.to_csv(os.path.join(DATA_DIR, "alpha_research_sweep.csv"), index=False)
print(df.head(15).to_string(index=False, formatters={
"CAGR": "{:.1%}".format, "Sharpe": "{:.2f}".format,
"MaxDD": "{:.1%}".format, "Calmar": "{:.2f}".format,
}))
return df
def main():
print("Loading PIT data…")
masked = load()
print(f" shape={masked.shape} range={masked.index[0].date()} → {masked.index[-1].date()}")
factor_diagnostics(masked)
composite_backtest(masked)
sweep = config_sweep(masked)
print("\n" + "=" * 110)
print("Top 5 configs:")
print("=" * 110)
print(sweep.head(5).to_string(index=False, formatters={
"CAGR": "{:.1%}".format, "Sharpe": "{:.2f}".format,
"MaxDD": "{:.1%}".format, "Calmar": "{:.2f}".format,
}))
if __name__ == "__main__":
main()

114
research/dca_simulation.py
View File

@@ -1,114 +0,0 @@
"""
DCA simulation: $10,000 initial + $5,000 every Feb & Aug from 2017.
Uses SharpeBoostedEnsembleStrategy daily returns.
"""
from __future__ import annotations
import os, sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.ensemble_alpha import SharpeBoostedEnsembleStrategy
import data_manager
from universe import get_sp500
def main():
# Load data and generate daily returns
tickers = get_sp500()
data_manager.update("us", tickers)
data = data_manager.load("us")
strat = SharpeBoostedEnsembleStrategy()
weights = strat.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
# Also compute SPY buy-and-hold for comparison
spy_rets = data["SPY"].pct_change().fillna(0.0)
# Trim to evaluation period
start = "2016-04-01"
end = "2026-05-13"
daily_rets = daily_rets.loc[start:end]
spy_rets = spy_rets.loc[start:end]
# --- DCA simulation ---
# Initial: $10,000 at start
# Contributions: $5,000 on first trading day of Feb and Aug, starting 2017
# Find contribution dates (first trading day of each Feb and Aug from 2017)
contrib_dates = []
for year in range(2017, 2027):
for month in [2, 8]:
target = pd.Timestamp(f"{year}-{month:02d}-01")
# Find first trading day on or after target
mask = daily_rets.index >= target
if mask.any():
contrib_dates.append(daily_rets.index[mask][0])
# Filter to only dates within our data range
contrib_dates = [d for d in contrib_dates if d <= daily_rets.index[-1]]
print("=" * 70)
print("DCA SIMULATION: SharpeBoostedEnsembleStrategy")
print("=" * 70)
print(f"Initial investment: $10,000 on {daily_rets.index[0].strftime('%Y-%m-%d')}")
print(f"Contributions: $5,000 on first trading day of Feb & Aug (from 2017)")
print(f"End date: {daily_rets.index[-1].strftime('%Y-%m-%d')}")
print(f"Total contribution dates: {len(contrib_dates)}")
print()
# Simulate for both strategy and SPY
for label, rets in [("Strategy", daily_rets), ("SPY (Buy & Hold)", spy_rets)]:
portfolio_value = 10000.0
total_contributed = 10000.0
contrib_idx = 0
# Track milestones
yearly_values = {}
for i, date in enumerate(rets.index):
# Apply daily return
portfolio_value *= (1 + rets.iloc[i])
# Check if today is a contribution date
if contrib_idx < len(contrib_dates) and date >= contrib_dates[contrib_idx]:
portfolio_value += 5000.0
total_contributed += 5000.0
contrib_idx += 1
# Record year-end values
if i == len(rets.index) - 1 or rets.index[i].year != rets.index[i + 1].year if i < len(rets.index) - 1 else True:
yearly_values[date.year] = portfolio_value
profit = portfolio_value - total_contributed
roi = profit / total_contributed * 100
print(f"--- {label} ---")
print(f" Total contributed: ${total_contributed:,.0f}")
print(f" Final portfolio: ${portfolio_value:,.0f}")
print(f" Total profit: ${profit:,.0f}")
print(f" ROI on contributions: {roi:.1f}%")
print(f" Multiple on capital: {portfolio_value/total_contributed:.2f}x")
print()
# Year-end snapshots
print(f" Year-end portfolio values:")
for year, val in sorted(yearly_values.items()):
# How much contributed by that year
contribs_by_year = 10000 + 5000 * len([d for d in contrib_dates if d.year <= year])
print(f" {year}: ${val:>12,.0f} (contributed: ${contribs_by_year:>8,.0f}, "
f"gain: ${val - contribs_by_year:>+10,.0f})")
print()
# --- Monthly detail of contributions ---
print("--- Contribution schedule ---")
for i, d in enumerate(contrib_dates):
print(f" {i+1:2d}. {d.strftime('%Y-%m-%d')} (${5000:,})")
print(f" Total contributions (excl. initial): ${5000 * len(contrib_dates):,}")
print(f" Total capital deployed: ${10000 + 5000 * len(contrib_dates):,}")
if __name__ == "__main__":
main()

34
research/event_factors.py
View File

@@ -1,34 +0,0 @@
import numpy as np
import pandas as pd
TRAILING_HIGH_WINDOW = 60
COMPRESSION_WINDOW = 20
VOLUME_WINDOW = 20
def breakout_after_compression_score(
close: pd.DataFrame,
high: pd.DataFrame,
low: pd.DataFrame,
volume: pd.DataFrame,
) -> pd.DataFrame:
"""Score breakout setups and shift the result so it is tradable next day."""
close = close.sort_index()
high = high.reindex(index=close.index, columns=close.columns).sort_index()
low = low.reindex(index=close.index, columns=close.columns).sort_index()
volume = volume.reindex(index=close.index, columns=close.columns).sort_index()
trailing_high = close.rolling(TRAILING_HIGH_WINDOW, min_periods=TRAILING_HIGH_WINDOW).max()
proximity_to_high = close / trailing_high.replace(0, np.nan)
recent_high = high.rolling(COMPRESSION_WINDOW, min_periods=COMPRESSION_WINDOW).max()
recent_low = low.rolling(COMPRESSION_WINDOW, min_periods=COMPRESSION_WINDOW).min()
recent_mid = (recent_high + recent_low) / 2
compressed_range = -((recent_high - recent_low) / recent_mid.replace(0, np.nan))
median_volume = volume.rolling(VOLUME_WINDOW, min_periods=VOLUME_WINDOW).median()
abnormal_volume = volume / median_volume.replace(0, np.nan)
score = proximity_to_high + compressed_range + abnormal_volume
return score.shift(1)

294
research/factor_optimize.py
View File

@@ -1,294 +0,0 @@
"""
Factor-level optimization on the point-in-time S&P 500 universe.
Builds on the sweep results in data/sweep_*.csv. Runs four experiments:
O1 — RecoveryMomentumPlus hyperparameter grid (top_n × rec_window × rec_weight × rebal),
with 2016-2022 train / 2023-2026 test split. Picks best by Sharpe-on-test.
O2 — SPY>MA regime filter applied to the 3 highest-Sharpe strategies (10y window).
O3 — Top-3 uncorrelated ensemble: greedy corr<0.85 selection → equal-weight blend.
O4 — Factor-mix parameter sweep on the FactorCombo "up_cap+mom_gap" signal
(top_n × rebal_freq).
All experiments run on PIT-masked data. Results printed + written to
data/factor_optimize_<exp>.csv.
Usage:
uv run python -m research.factor_optimize
"""
from __future__ import annotations
import os
import warnings
import numpy as np
import pandas as pd
import research.pit_backtest as pit
from research.strategies_plus import (EnsembleStrategy, RecoveryMomentumPlus,
spy_ma200_filter)
from strategies.factor_combo import FactorComboStrategy, SIGNAL_REGISTRY
from strategies.momentum_quality import MomentumQualityStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
warnings.filterwarnings("ignore", category=FutureWarning)
DATA_DIR = "data"
BENCHMARK = "SPY"
def load_masked_prices():
raw = pit.load_pit_prices()
masked = pit.pit_universe(raw)
if BENCHMARK in raw.columns:
masked[BENCHMARK] = raw[BENCHMARK]
return masked
def slice_period(df, start=None, end=None):
out = df
if start:
out = out[out.index >= start]
if end:
out = out[out.index <= end]
return out
def run(strat, prices, *, regime_filter=None):
return pit.backtest(
strategy=strat, prices=prices, initial_capital=10_000,
transaction_cost=0.001, regime_filter=regime_filter,
)
# ---------------------------------------------------------------------------
# O1 — RecoveryMomentumPlus hyperparameter grid
# ---------------------------------------------------------------------------
def o1_hyperparam_sweep(masked):
print("\n" + "=" * 100)
print("O1 — RecoveryMomentumPlus sweep (train 2016-2022 / test 2023-2026)")
print("=" * 100)
tickers = [c for c in masked.columns if c != BENCHMARK]
prices = masked[tickers]
train = slice_period(prices, "2016-04-19", "2022-12-31")
test = slice_period(prices, "2023-01-01", None)
grid = []
for top_n in (5, 10, 15, 20):
for rec_win in (42, 63, 126):
for rec_w in (0.3, 0.5, 0.7):
for rebal in (5, 10, 21):
grid.append((top_n, rec_win, rec_w, rebal))
rows = []
for i, (top_n, rec_win, rec_w, rebal) in enumerate(grid, 1):
cfg = dict(top_n=top_n, recovery_window=rec_win,
rec_weight=rec_w, rebal_freq=rebal)
tr = pit.summarize(run(RecoveryMomentumPlus(**cfg), train), "")
te = pit.summarize(run(RecoveryMomentumPlus(**cfg), test), "")
rows.append({**cfg,
"train_CAGR": tr["CAGR"], "train_Sharpe": tr["Sharpe"],
"test_CAGR": te["CAGR"], "test_Sharpe": te["Sharpe"],
"test_MaxDD": te["MaxDD"], "test_Calmar": te["Calmar"]})
if i % 12 == 0 or i == len(grid):
print(f" … {i}/{len(grid)} configs evaluated")
df = pd.DataFrame(rows).sort_values("test_Sharpe", ascending=False)
out = os.path.join(DATA_DIR, "factor_optimize_O1.csv")
df.to_csv(out, index=False)
print("\n --- Top 10 by out-of-sample Sharpe (2023-2026) ---")
disp = ["top_n", "recovery_window", "rec_weight", "rebal_freq",
"train_Sharpe", "test_Sharpe", "train_CAGR", "test_CAGR",
"test_MaxDD", "test_Calmar"]
print(df.head(10)[disp].to_string(index=False, formatters={
"train_Sharpe": "{:.2f}".format, "test_Sharpe": "{:.2f}".format,
"train_CAGR": "{:.1%}".format, "test_CAGR": "{:.1%}".format,
"test_MaxDD": "{:.1%}".format, "test_Calmar": "{:.2f}".format,
}))
return df
# ---------------------------------------------------------------------------
# O2 — Regime filter on the top strategies
# ---------------------------------------------------------------------------
def o2_regime(masked):
print("\n" + "=" * 100)
print("O2 — SPY > MA regime filter on top strategies (full 10y PIT)")
print("=" * 100)
tickers = [c for c in masked.columns if c != BENCHMARK]
prices = masked[tickers]
spy_full = masked[BENCHMARK].dropna()
contenders = {
"Recovery+Mom Top10": RecoveryMomentumStrategy(top_n=10),
"fc_up_cap_mom_gap_monthly": FactorComboStrategy("up_cap+mom_gap",
rebal_freq=21, top_n=10),
"fc_rec63_mom_gap_monthly": FactorComboStrategy("rec63+mom_gap",
rebal_freq=21, top_n=10),
}
rows = []
for name, strat in contenders.items():
base = run(strat, prices)
rows.append({"strategy": name, "filter": "none",
**{k: v for k, v in pit.summarize(base, "").items() if k != "name"}})
for ma in (200, 150, 100):
filt = spy_ma200_filter(spy_full, ma_window=ma).reindex(prices.index).fillna(False)
strat_fresh = _fresh_copy(strat)
eq = run(strat_fresh, prices, regime_filter=filt)
rows.append({"strategy": name, "filter": f"SPY>MA{ma}",
**{k: v for k, v in pit.summarize(eq, "").items() if k != "name"}})
df = pd.DataFrame(rows)
df.to_csv(os.path.join(DATA_DIR, "factor_optimize_O2.csv"), index=False)
print(f" {'strategy':<32s} {'filter':<12s} {'CAGR':>7s} {'Sharpe':>7s} "
f"{'MaxDD':>7s} {'Calmar':>7s}")
for _, r in df.iterrows():
print(f" {r['strategy']:<32s} {r['filter']:<12s} "
f"{r['CAGR']*100:>6.1f}% {r['Sharpe']:>7.2f} "
f"{r['MaxDD']*100:>6.1f}% {r['Calmar']:>7.2f}")
return df
def _fresh_copy(strat):
"""Re-instantiate a strategy so state (if any) is reset between backtests."""
if isinstance(strat, RecoveryMomentumStrategy):
return RecoveryMomentumStrategy(
recovery_window=strat.recovery_window, mom_lookback=strat.mom_lookback,
mom_skip=strat.mom_skip, rebal_freq=strat.rebal_freq, top_n=strat.top_n)
if isinstance(strat, FactorComboStrategy):
return FactorComboStrategy(strat.signal_name, rebal_freq=strat.rebal_freq,
top_n=strat.top_n)
if isinstance(strat, MomentumQualityStrategy):
return MomentumQualityStrategy(
momentum_period=strat.momentum_period, skip=strat.skip,
quality_window=strat.quality_window, top_n=strat.top_n)
return strat # already stateless for our uses
# ---------------------------------------------------------------------------
# O3 — Uncorrelated ensemble
# ---------------------------------------------------------------------------
def o3_ensemble(masked):
print("\n" + "=" * 100)
print("O3 — Greedy uncorrelated ensemble (full 10y PIT)")
print("=" * 100)
tickers = [c for c in masked.columns if c != BENCHMARK]
prices = masked[tickers]
spy_full = masked[BENCHMARK].dropna()
# Candidate pool: the production strategies that cleared 0.75 Sharpe in 10y sweep.
candidates: list[tuple[str, object]] = [
("Recovery+Mom Top10", RecoveryMomentumStrategy(top_n=10)),
("fc_up_cap_mom_gap_monthly", FactorComboStrategy("up_cap+mom_gap", 21, 10)),
("fc_rec63_mom_gap_monthly", FactorComboStrategy("rec63+mom_gap", 21, 10)),
("fc_up_cap_quality_mom_monthly", FactorComboStrategy("up_cap+quality_mom", 21, 10)),
("fc_rec_mfilt_deep_upvol_monthly", FactorComboStrategy("rec_mfilt+deep_upvol", 21, 10)),
("fc_mom7m_rec126_monthly", FactorComboStrategy("mom7m+rec126", 21, 10)),
("Recovery+Mom Top20", RecoveryMomentumStrategy(top_n=20)),
("fc_down_resil_qual_mom_monthly", FactorComboStrategy("down_resil+qual_mom", 21, 10)),
]
equities: dict[str, pd.Series] = {name: run(s, prices) for name, s in candidates}
returns = pd.DataFrame({n: eq.pct_change().fillna(0) for n, eq in equities.items()})
sharpes = {n: pit.summarize(eq, n)["Sharpe"] for n, eq in equities.items()}
order = sorted(candidates, key=lambda t: sharpes[t[0]], reverse=True)
picked_names: list[str] = []
picked: list[tuple[object, float]] = []
for name, strat in order:
if any(returns[name].corr(returns[p]) > 0.85 for p in picked_names):
continue
picked_names.append(name)
picked.append((strat, 1.0))
if len(picked) >= 3:
break
print(f" Selected {len(picked)} uncorrelated components:")
for name in picked_names:
print(f" - {name} (Sharpe={sharpes[name]:.2f})")
ens = EnsembleStrategy(picked)
eq_ens = run(ens, prices)
filt = spy_ma200_filter(spy_full).reindex(prices.index).fillna(False)
eq_ens_reg = run(EnsembleStrategy(picked), prices, regime_filter=filt)
spy_bh = (masked[BENCHMARK].dropna().pipe(lambda s: s / s.iloc[0] * 10_000))
rows = [pit.summarize(spy_bh, "SPY buy-and-hold")]
for name in picked_names:
rows.append(pit.summarize(equities[name], f" component: {name}"))
rows.append(pit.summarize(eq_ens, "ENSEMBLE (equal-weight, no filter)"))
rows.append(pit.summarize(eq_ens_reg, "ENSEMBLE + SPY>MA200 filter"))
for r in rows:
print(pit.fmt_row(r))
df = pd.DataFrame(rows)
df.to_csv(os.path.join(DATA_DIR, "factor_optimize_O3.csv"), index=False)
return df, picked_names
# ---------------------------------------------------------------------------
# O4 — FactorCombo up_cap+mom_gap: top_n × rebal sweep
# ---------------------------------------------------------------------------
def o4_factorcombo_sweep(masked):
print("\n" + "=" * 100)
print("O4 — FactorCombo up_cap+mom_gap: top_n × rebal (full 10y PIT)")
print("=" * 100)
tickers = [c for c in masked.columns if c != BENCHMARK]
prices = masked[tickers]
rows = []
for top_n in (5, 8, 10, 15, 20, 30):
for rebal in (5, 10, 21, 42):
strat = FactorComboStrategy("up_cap+mom_gap", rebal_freq=rebal, top_n=top_n)
eq = run(strat, prices)
s = pit.summarize(eq, f"top_n={top_n} rebal={rebal}")
rows.append({"top_n": top_n, "rebal": rebal,
"CAGR": s["CAGR"], "Sharpe": s["Sharpe"],
"MaxDD": s["MaxDD"], "Calmar": s["Calmar"]})
df = pd.DataFrame(rows).sort_values("Sharpe", ascending=False)
df.to_csv(os.path.join(DATA_DIR, "factor_optimize_O4.csv"), index=False)
print(f" {'top_n':<8s}{'rebal':<8s}{'CAGR':>8s}{'Sharpe':>9s}"
f"{'MaxDD':>9s}{'Calmar':>9s}")
for _, r in df.iterrows():
print(f" {int(r['top_n']):<8d}{int(r['rebal']):<8d}"
f"{r['CAGR']*100:>7.1f}%{r['Sharpe']:>9.2f}"
f"{r['MaxDD']*100:>8.1f}%{r['Calmar']:>9.2f}")
return df
def main():
print("Loading PIT-masked price data…")
masked = load_masked_prices()
print(f" shape={masked.shape} range={masked.index[0].date()} → {masked.index[-1].date()}")
o1 = o1_hyperparam_sweep(masked)
o2 = o2_regime(masked)
o3, picks = o3_ensemble(masked)
o4 = o4_factorcombo_sweep(masked)
print("\n" + "=" * 100)
print("Summary: best config from each experiment")
print("=" * 100)
best_o1 = o1.iloc[0]
print(f" O1 best OOS Sharpe: top_n={int(best_o1['top_n'])} rec_win={int(best_o1['recovery_window'])} "
f"rec_w={best_o1['rec_weight']} rebal={int(best_o1['rebal_freq'])} "
f"→ test Sharpe={best_o1['test_Sharpe']:.2f} test CAGR={best_o1['test_CAGR']*100:.1f}%")
best_o4 = o4.iloc[0]
print(f" O4 best overall: top_n={int(best_o4['top_n'])} rebal={int(best_o4['rebal'])} "
f"Sharpe={best_o4['Sharpe']:.2f} CAGR={best_o4['CAGR']*100:.1f}% "
f"Calmar={best_o4['Calmar']:.2f}")
if __name__ == "__main__":
main()

152
research/fetch_historical.py
View File

@@ -1,152 +0,0 @@
"""
Fetch price history for all tickers that were ever S&P 500 members — including
delisted ones — and save to data/us_pit.csv. This is the foundation for a
survivorship-bias-free backtest.
NOTE: Yahoo Finance no longer serves price data for many fully-delisted tickers
(bankruptcies, old mergers). Those are silently skipped. The result is still
a major improvement over "today's S&P 500 extrapolated 10 years back", but it
is NOT a perfect point-in-time dataset — only a dataset where the universe
mask is correct at each date. A subset of worst-outcome tickers (e.g., ABK,
ACAS) will be missing entirely. This caveat is documented in the run summary.
"""
import os
from datetime import datetime, timedelta
import pandas as pd
import yfinance as yf
import universe_history as uh
DATA_DIR = "data"
OUT_PATH = os.path.join(DATA_DIR, "us_pit.csv")
YEARS = 10
BATCH_SIZE = 50
def _field_out_paths() -> dict[str, str]:
return {
"Close": os.path.join(DATA_DIR, "us_pit_close.csv"),
"High": os.path.join(DATA_DIR, "us_pit_high.csv"),
"Low": os.path.join(DATA_DIR, "us_pit_low.csv"),
"Volume": os.path.join(DATA_DIR, "us_pit_volume.csv"),
}
def fetch_all_historical(force: bool = False) -> pd.DataFrame:
os.makedirs(DATA_DIR, exist_ok=True)
intervals = uh.load_sp500_history()
tickers = uh.all_tickers_ever(intervals) + ["SPY"]
tickers = sorted(set(tickers))
existing = None
if os.path.exists(OUT_PATH) and not force:
existing = pd.read_csv(OUT_PATH, index_col=0, parse_dates=True)
missing = [t for t in tickers if t not in existing.columns]
if not missing:
# Just append latest dates
last_date = existing.index[-1]
if (datetime.now() - last_date.to_pydatetime()).days < 2:
print(f"--- us_pit.csv already up to date: {existing.shape} ---")
return existing
tickers = list(existing.columns)
start = (last_date + timedelta(days=1)).strftime("%Y-%m-%d")
print(f"--- Appending new dates from {start} for {len(tickers)} tickers ---")
new = _download_batched(tickers, start=start)
if new is not None and not new.empty:
combined = pd.concat([existing, new]).sort_index()
combined = combined[~combined.index.duplicated(keep="last")]
combined.to_csv(OUT_PATH)
print(f"--- Saved {combined.shape} to {OUT_PATH} ---")
return combined
return existing
else:
print(f"--- Have {existing.shape[1]} cols; need {len(missing)} more ---")
tickers = missing
start = (datetime.now() - timedelta(days=365 * YEARS)).strftime("%Y-%m-%d")
new = _download_batched(tickers, start=start)
if existing is not None and new is not None and not new.empty:
combined = pd.concat([existing, new.reindex(existing.index)], axis=1)
# Add any new rows from `new` not in existing
new_only_idx = new.index.difference(existing.index)
if len(new_only_idx) > 0:
combined_new = new.loc[new_only_idx].reindex(columns=combined.columns)
combined = pd.concat([combined, combined_new]).sort_index()
else:
combined = new
combined.to_csv(OUT_PATH)
print(f"--- Saved {combined.shape} to {OUT_PATH} ---")
return combined
def fetch_all_historical_ohlcv(force: bool = False) -> dict[str, pd.DataFrame]:
os.makedirs(DATA_DIR, exist_ok=True)
intervals = uh.load_sp500_history()
tickers = uh.all_tickers_ever(intervals) + ["SPY"]
tickers = sorted(set(tickers))
start = (datetime.now() - timedelta(days=365 * YEARS)).strftime("%Y-%m-%d")
panels = _download_batched_fields(tickers, start=start, fields=["Close", "High", "Low", "Volume"])
if not panels:
raise RuntimeError("No PIT OHLCV data downloaded")
close = panels["Close"]
close.to_csv(OUT_PATH)
print(f"--- Saved {close.shape} to {OUT_PATH} ---")
result: dict[str, pd.DataFrame] = {"close": close}
for field, path in _field_out_paths().items():
panel = panels[field]
panel.to_csv(path)
print(f"--- Saved {panel.shape} to {path} ---")
result[field.lower()] = panel
return result
def _download_batched(tickers: list[str], start: str) -> pd.DataFrame | None:
panels = _download_batched_fields(tickers, start=start, fields=["Close"])
if not panels:
return None
return panels["Close"]
def _download_batched_fields(
tickers: list[str],
start: str,
fields: list[str],
) -> dict[str, pd.DataFrame]:
frames = {field: [] for field in fields}
n = len(tickers)
for i in range(0, n, BATCH_SIZE):
batch = tickers[i:i + BATCH_SIZE]
print(f" [{i}/{n}] fetching {len(batch)} tickers...", flush=True)
try:
raw = yf.download(batch, start=start, auto_adjust=True,
progress=False, threads=True)
if raw.empty:
continue
for field in fields:
if isinstance(raw.columns, pd.MultiIndex):
panel = raw[field]
else:
panel = raw[[field]].rename(columns={field: batch[0]})
panel = panel.dropna(axis=1, how="all")
if not panel.empty:
frames[field].append(panel)
except Exception as e:
print(f" batch failed: {e}")
result = {}
for field, field_frames in frames.items():
if field_frames:
panel = pd.concat(field_frames, axis=1).sort_index()
panel = panel.loc[:, ~panel.columns.duplicated()]
result[field] = panel
else:
result[field] = pd.DataFrame()
return result
if __name__ == "__main__":
fetch_all_historical()

233
research/interaction_alpha.py
View File

@@ -1,233 +0,0 @@
"""
Interaction / multiplicative factor strategy.
Rationale: in the 10y PIT diagnostics, each single-factor top decile clocks
~0.5–0.8 Sharpe, yet the production Recovery+Mom Top10 delivers 0.92. The
extra alpha comes from an AND-style interaction — stocks that rank high on
BOTH factors simultaneously. Linear rank-blending loses this because a stock
can make top_n by being middling on many factors instead of extreme on a few.
This module provides:
* `MultiplicativeFactorStrategy` — picks top_n stocks by the geometric mean
(equivalently the product) of cross-sectional factor ranks. Concentrates
on consensus winners.
* `VotingFactorStrategy` — counts how many factors place a stock in its
top `vote_pct`; selects stocks clearing a minimum vote threshold. Breaks
ties by the sum of ranks. Robust when factor ICs drift.
* `SubStrategyEnsemble` — equal-weight blend of Recovery+Mom Top10,
fc_up_cap+mom_gap monthly, and a new Multiplicative("mom × recovery ×
idio_vol_neg") sleeve. Diversifies across independent alpha sources
rather than across factor primitives.
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from research.alpha_factors import (_rolling_beta_and_residvol, f_mom_12_1,
f_mom_7_1, f_rev_1m, f_w52_high, f_max5_neg,
f_recovery_63, f_trend_strength, xsec_rank,
f_mom_residual)
from strategies.base import Strategy
from strategies.factor_combo import FactorComboStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
# ---------------------------------------------------------------------------
# Multiplicative top-N
# ---------------------------------------------------------------------------
class MultiplicativeFactorStrategy(Strategy):
"""
Top-N by product of selected factor ranks (equivalent to rank-geometric-mean).
Parameters
----------
factor_names : list[str]
Keys into the factor library. Supported:
mom_12_1, mom_7_1, mom_residual, recovery_63, w52_high,
idio_vol_neg, mom_x_recovery (shortcut pair).
top_n : int
Number of stocks.
rebal_freq : int
Rebal interval in trading days.
mkt_returns : pd.Series | None
Required for mom_residual / idio_vol_neg.
"""
def __init__(self, factor_names: list[str], top_n: int = 10,
rebal_freq: int = 21, mkt_returns: pd.Series | None = None,
weighting: str = "equal", signal_concentration: float = 0.0,
dispersion_scale: bool = False):
"""
Parameters
----------
signal_concentration : float
Exponent applied to composite score when weighting=='signal'.
0 → equal weight within top_n; higher → more weight on top ranks.
dispersion_scale : bool
Scale total exposure by z-scored cross-sectional rank dispersion,
clipped to [0.5, 1.3]. Expands in high-dispersion regimes.
"""
self.factor_names = factor_names
self.top_n = top_n
self.rebal_freq = rebal_freq
self.mkt_returns = mkt_returns
self.weighting = weighting
self.signal_concentration = signal_concentration
self.dispersion_scale = dispersion_scale
def _build(self, data: pd.DataFrame) -> dict[str, pd.DataFrame]:
betas, resid_vol = (None, None)
if any(f in ("mom_residual", "idio_vol_neg", "low_beta") for f in self.factor_names):
if self.mkt_returns is None:
raise ValueError("mkt_returns required for beta-based factors")
betas, resid_vol = _rolling_beta_and_residvol(data, self.mkt_returns, 60)
lib = {
"mom_12_1": lambda: f_mom_12_1(data),
"mom_7_1": lambda: f_mom_7_1(data),
"mom_residual": lambda: f_mom_residual(data, self.mkt_returns, betas=betas),
"recovery_63": lambda: f_recovery_63(data),
"w52_high": lambda: f_w52_high(data),
"idio_vol_neg": lambda: -resid_vol,
"low_beta": lambda: -betas,
"trend": lambda: f_trend_strength(data),
}
return {n: lib[n]() for n in self.factor_names}
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
factors = self._build(data)
ranks = {n: xsec_rank(v) for n, v in factors.items()}
# Product of ranks. If any rank is NaN, product is NaN → row excluded.
composite = None
for rk in ranks.values():
composite = rk if composite is None else composite.mul(rk, fill_value=np.nan)
composite = composite.where(~rk.isna(), np.nan)
sel_rank = composite.rank(axis=1, ascending=False, na_option="bottom")
n_valid = composite.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (sel_rank <= self.top_n) & enough.values.reshape(-1, 1)
if self.weighting == "equal":
raw = top_mask.astype(float)
elif self.weighting == "inv_vol":
vol = data.pct_change(fill_method=None).rolling(60).std()
raw = (1.0 / vol.replace(0, np.nan)).where(top_mask, 0.0).fillna(0.0)
elif self.weighting == "signal":
# Weight ∝ composite^concentration, only among top_mask picks.
score = composite.where(top_mask, 0.0).fillna(0.0)
raw = score ** max(self.signal_concentration, 1.0)
else:
raise ValueError(f"bad weighting {self.weighting!r}")
row_sums = raw.sum(axis=1).replace(0, np.nan)
weights = raw.div(row_sums, axis=0).fillna(0.0)
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_mask.iloc[list(range(warmup, len(data), self.rebal_freq))] = True
weights[~rebal_mask] = np.nan
weights = weights.ffill().fillna(0.0)
weights.iloc[:warmup] = 0.0
if self.dispersion_scale:
# Cross-sectional rank dispersion = daily std of composite. Scale
# exposure up in high-dispersion regimes (alpha opportunity richer).
disp = composite.std(axis=1)
z = (disp - disp.rolling(252, min_periods=126).mean()) \
/ disp.rolling(252, min_periods=126).std()
scale = (1.0 + 0.3 * z.clip(-1, 1)).clip(0.5, 1.3)
scale = scale.reindex(weights.index).fillna(1.0)
weights = weights.mul(scale, axis=0)
return weights.shift(1).fillna(0.0)
# ---------------------------------------------------------------------------
# Voting top-N
# ---------------------------------------------------------------------------
class VotingFactorStrategy(Strategy):
"""
Top-N by vote count: each factor contributes 1 vote if a stock is in its
top `vote_pct` percentile. Select stocks with vote_count ≥ min_votes,
break ties by sum of ranks.
"""
def __init__(self, factor_names: list[str], top_n: int = 10,
rebal_freq: int = 21, vote_pct: float = 0.25,
min_votes: int = 3, mkt_returns: pd.Series | None = None):
self.factor_names = factor_names
self.top_n = top_n
self.rebal_freq = rebal_freq
self.vote_pct = vote_pct
self.min_votes = min_votes
self.mkt_returns = mkt_returns
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
builder = MultiplicativeFactorStrategy(
factor_names=self.factor_names, top_n=self.top_n,
rebal_freq=self.rebal_freq, mkt_returns=self.mkt_returns)
factors = builder._build(data)
ranks = {n: xsec_rank(v) for n, v in factors.items()}
thresh = 1 - self.vote_pct
votes = sum((rk >= thresh).astype(float) for rk in ranks.values())
rank_sum = sum(rk.fillna(0) for rk in ranks.values())
# Primary sort: vote count; tiebreaker: rank_sum. Build a composite.
composite = votes + rank_sum / (len(ranks) * 10)
composite = composite.where(votes >= self.min_votes, np.nan)
sel_rank = composite.rank(axis=1, ascending=False, na_option="bottom")
n_valid = composite.notna().sum(axis=1)
enough = n_valid >= 1
effective_n = n_valid.clip(upper=self.top_n)
top_mask = (sel_rank <= effective_n.values.reshape(-1, 1)) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
weights = raw.div(row_sums, axis=0).fillna(0.0)
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_mask.iloc[list(range(warmup, len(data), self.rebal_freq))] = True
weights[~rebal_mask] = np.nan
weights = weights.ffill().fillna(0.0)
weights.iloc[:warmup] = 0.0
return weights.shift(1).fillna(0.0)
# ---------------------------------------------------------------------------
# Sub-strategy ensemble
# ---------------------------------------------------------------------------
class SubStrategyEnsemble(Strategy):
"""Equal-weight blend of several long-only sub-strategies."""
def __init__(self, sub_strats: list[Strategy]):
self.sub_strats = sub_strats
self.w = 1.0 / len(sub_strats)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
out = None
for strat in self.sub_strats:
sig = strat.generate_signals(data) * self.w
out = sig if out is None else out.add(sig, fill_value=0.0)
return out
def default_ensemble(mkt_returns: pd.Series) -> SubStrategyEnsemble:
return SubStrategyEnsemble([
RecoveryMomentumStrategy(top_n=10),
FactorComboStrategy("up_cap+mom_gap", rebal_freq=21, top_n=10),
MultiplicativeFactorStrategy(
factor_names=["mom_12_1", "recovery_63", "idio_vol_neg"],
top_n=10, rebal_freq=21, mkt_returns=mkt_returns,
),
])

282
research/ls_momentum_eval.py
View File

@@ -1,282 +0,0 @@
"""Evaluate the industry-neutral L/S momentum strategy with realistic costs.
Costs applied:
* gross slippage : 30 bps × turnover (long+short rebalances)
* borrow fee : 50 bps annualized × |short weight|, daily
* Optional dividend on short leg: 1.5% annualized × |short weight|, daily
Outputs metrics for the L/S strategy alone and blended with TrendRiderV5.
"""
from __future__ import annotations
import argparse
import os
import sys
from dataclasses import asdict
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.permanent_yearly import load_etfs, ETF_CACHE
from research.trend_rider_v6_eval import load_combined_panel
from research.trend_rider_robustness import (
buy_hold_weights,
evaluate_weights,
portfolio_returns,
)
from strategies.permanent import ETF_UNIVERSE
from strategies.trend_rider_v5 import TrendRiderV5
from strategies.ls_momentum import IndustryNeutralLSMomentum, fetch_sp500_sectors
from strategies.long_hedged import LongHedgedStock
IS_START = "2015-01-02"
IS_END = "2020-12-31"
OOS_START = "2021-01-01"
OOS_END = "2026-05-07"
def _fmt(x):
return f"{x*100:7.2f}%"
def ls_returns(weights: pd.DataFrame, prices: pd.DataFrame,
slippage_bps: float = 30.0,
borrow_bps_annual: float = 50.0,
div_short_bps_annual: float = 150.0) -> pd.Series:
"""Daily P&L net of slippage, borrow fee, and short-dividend pass-through.
weights : positive = long, negative = short.
"""
aligned = weights.reindex(index=prices.index, columns=prices.columns).fillna(0.0)
rets = prices.pct_change(fill_method=None).fillna(0.0)
gross = (rets * aligned).sum(axis=1)
turnover = aligned.diff().abs().sum(axis=1).fillna(0.0)
slip_cost = turnover * (slippage_bps / 10_000)
# Daily borrow cost on short leg (negative weights → positive |w|)
short_w = aligned.clip(upper=0.0).abs().sum(axis=1)
borrow_daily = (borrow_bps_annual + div_short_bps_annual) / 10_000 / 252
short_cost = short_w * borrow_daily
return gross - slip_cost - short_cost
def evaluate_ls(label: str, weights: pd.DataFrame, prices: pd.DataFrame,
start: str, end: str,
slippage_bps: float = 30.0,
borrow_bps_annual: float = 50.0,
div_short_bps_annual: float = 150.0):
"""Custom evaluator that handles negative weights and L/S costs."""
rets = ls_returns(weights, prices, slippage_bps, borrow_bps_annual,
div_short_bps_annual)
rets = rets[(rets.index >= start) & (rets.index <= end)]
if rets.empty:
return None
eq = (1 + rets).cumprod()
span = max((rets.index[-1] - rets.index[0]).days / 365.25, 1 / 252)
cagr = float(eq.iloc[-1] ** (1 / span) - 1)
vol = float(rets.std(ddof=1) * np.sqrt(252))
sharpe = float(rets.mean() / rets.std(ddof=1) * np.sqrt(252)) if rets.std(ddof=1) > 0 else 0.0
dd = eq / eq.cummax() - 1
mdd = float(dd.min())
aligned = weights.reindex(index=prices.index, columns=prices.columns).fillna(0.0)
aligned = aligned.loc[(aligned.index >= start) & (aligned.index <= end)]
turn = aligned.diff().abs().sum(axis=1).fillna(0.0)
long_w = aligned.clip(lower=0.0).sum(axis=1)
short_w = aligned.clip(upper=0.0).abs().sum(axis=1)
# Construct an Evaluation-like dict
return {
"label": label,
"start": str(rets.index[0].date()),
"end": str(rets.index[-1].date()),
"days": int(len(rets)),
"cagr": cagr,
"volatility": vol,
"sharpe": sharpe,
"max_drawdown": mdd,
"calmar": float(cagr / abs(mdd)) if mdd < 0 else 0.0,
"final_multiple": float(eq.iloc[-1]),
"switches": int((turn > 0.01).sum()),
"avg_daily_turnover": float(turn.mean()),
"avg_long": float(long_w.mean()),
"avg_short": float(short_w.mean()),
"rets": rets,
}
def print_eval(d: dict, prefix: str = "") -> None:
print(
f" {prefix}{d['label']:<32s} "
f"CAGR {_fmt(d['cagr'])} Vol {_fmt(d['volatility'])} "
f"Sharpe {d['sharpe']:5.2f} MDD {_fmt(d['max_drawdown'])} "
f"Calmar {d['calmar']:5.2f} X {d['final_multiple']:6.2f} "
f"L {d['avg_long']*100:5.1f}% S {d['avg_short']*100:5.1f}%"
)
def annual_returns(rets: pd.Series) -> pd.Series:
return (1.0 + rets).groupby(rets.index.year).prod() - 1.0
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--slippage-bps", type=float, default=30.0)
parser.add_argument("--borrow-bps", type=float, default=15.0)
# auto_adjust=True yfinance already includes dividends; do not double-count
parser.add_argument("--div-short-bps", type=float, default=0.0)
parser.add_argument("--out-dir", default="data")
args = parser.parse_args()
panel = load_combined_panel()
etf_set = (set(ETF_UNIVERSE)
| {"QQQ", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "SPY",
"YINN", "CHAU", "7200.HK", "7500.HK"})
stock_universe = [c for c in panel.columns if c not in etf_set]
print(f"Stock universe: {len(stock_universe)} names")
sector_df = fetch_sp500_sectors()
sector_map = sector_df["GICS Sector"]
coverage = sector_map.reindex(stock_universe).notna().sum()
print(f"Sector coverage: {coverage} / {len(stock_universe)}")
# ---------- #1 + #2: smaller top_n + regime gate ----------
candidates = {
# Baseline from prior run
"Hedged top10 hr1.0 (baseline)": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=10,
hedge_ratio=1.0, stock_universe=stock_universe),
# #1 — concentrated long leg
"Hedged top5 hr1.0": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=5,
hedge_ratio=1.0, stock_universe=stock_universe),
"Hedged top7 hr1.0": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=7,
hedge_ratio=1.0, stock_universe=stock_universe),
# #2 — regime gate (only on when SPY > MA200)
"Hedged top10 hr1.0 +regime": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=10,
hedge_ratio=1.0, regime_gate=True,
stock_universe=stock_universe),
# #1 + #2 combined
"Hedged top5 hr1.0 +regime": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=5,
hedge_ratio=1.0, regime_gate=True,
stock_universe=stock_universe),
"Hedged top7 hr1.0 +regime": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=7,
hedge_ratio=1.0, regime_gate=True,
stock_universe=stock_universe),
# Smaller top_n with partial hedge
"Hedged top5 hr0.7 +regime": LongHedgedStock(
signal_name="rec_mfilt+deep_upvol", top_n=5,
hedge_ratio=0.7, regime_gate=True,
stock_universe=stock_universe),
}
weights_map = {}
print("\n=== Generating signals ===")
for name, strat in candidates.items():
print(f" ... {name}")
# LongHedgedStock needs the full panel (stocks + SPY); IndustryNeutral
# only needs stocks. Generate on appropriate slice.
if isinstance(strat, LongHedgedStock):
weights_map[name] = strat.generate_signals(panel)
else:
weights_map[name] = strat.generate_signals(panel[stock_universe])
print(f"\n=== L/S alone (slippage={args.slippage_bps}bps, "
f"borrow={args.borrow_bps}bps, div_short={args.div_short_bps}bps) ===")
print(f"\n --- FULL (2015 → 2026-05) ---")
rets_map = {}
for name, w in weights_map.items():
# Re-attach to full panel
w_full = w.reindex(columns=panel.columns).fillna(0.0)
d = evaluate_ls(name, w_full, panel, IS_START, OOS_END,
args.slippage_bps, args.borrow_bps, args.div_short_bps)
rets_map[name] = d["rets"]
print_eval(d)
print(f"\n --- IS (2015 → 2020) ---")
for name, w in weights_map.items():
w_full = w.reindex(columns=panel.columns).fillna(0.0)
d = evaluate_ls(name, w_full, panel, IS_START, IS_END,
args.slippage_bps, args.borrow_bps, args.div_short_bps)
print_eval(d)
print(f"\n --- OOS (2021 → 2026-05) ---")
for name, w in weights_map.items():
w_full = w.reindex(columns=panel.columns).fillna(0.0)
d = evaluate_ls(name, w_full, panel, OOS_START, OOS_END,
args.slippage_bps, args.borrow_bps, args.div_short_bps)
print_eval(d)
# ---------- V5 baseline returns ----------
print("\n=== V5 baseline (for blending) ===")
v5 = TrendRiderV5()
v5_w = v5.generate_signals(panel)
v5_rets = portfolio_returns(v5_w, panel[v5_w.columns], 0.001)
# Pick best L/S by full-period Sharpe
best_ls = max(rets_map.keys(),
key=lambda k: rets_map[k][(rets_map[k].index >= IS_START)
& (rets_map[k].index <= OOS_END)]
.pipe(lambda r: r.mean() / r.std(ddof=1) * np.sqrt(252)
if r.std(ddof=1) > 0 else 0))
print(f"\n Best L/S by full-period Sharpe : {best_ls}")
best_ls_rets = rets_map[best_ls]
# ---------- Correlation ----------
common = v5_rets.index.intersection(best_ls_rets.index)
common = common[(common >= pd.Timestamp(IS_START)) & (common <= pd.Timestamp(OOS_END))]
v5r, lsr = v5_rets.loc[common], best_ls_rets.loc[common]
corr_full = v5r.corr(lsr)
is_mask = (common >= pd.Timestamp(IS_START)) & (common <= pd.Timestamp(IS_END))
oos_mask = (common >= pd.Timestamp(OOS_START)) & (common <= pd.Timestamp(OOS_END))
corr_is = v5r[is_mask].corr(lsr[is_mask])
corr_oos = v5r[oos_mask].corr(lsr[oos_mask])
print(f" V5 vs {best_ls} correlations:")
print(f" FULL : {corr_full:6.3f}")
print(f" IS : {corr_is:6.3f}")
print(f" OOS : {corr_oos:6.3f}")
# ---------- Blends ----------
print(f"\n=== V5 + L/S blends (rets-level) ===")
print(f" Window Mix CAGR Vol Sharpe MDD Calmar")
for w5, wls in [(0.50, 0.50), (0.70, 0.30), (0.80, 0.20),
(0.60, 0.40), (0.40, 0.60)]:
for window_name, (s, e) in {"FULL": (IS_START, OOS_END),
"IS": (IS_START, IS_END),
"OOS": (OOS_START, OOS_END)}.items():
mask = (common >= pd.Timestamp(s)) & (common <= pd.Timestamp(e))
r = w5 * v5r[mask] + wls * lsr[mask]
if r.empty:
continue
eq = (1 + r).cumprod()
span = max((r.index[-1] - r.index[0]).days / 365.25, 1 / 252)
cagr = eq.iloc[-1] ** (1 / span) - 1
vol = r.std(ddof=1) * np.sqrt(252)
sharpe = r.mean() / r.std(ddof=1) * np.sqrt(252) if r.std(ddof=1) > 0 else 0
mdd = float((eq / eq.cummax() - 1).min())
calmar = cagr / abs(mdd) if mdd < 0 else 0
print(f" [{window_name:<4s}] V5={w5:.0%}+LS={wls:.0%} "
f"{cagr*100:6.2f}% {vol*100:5.2f}% {sharpe:5.2f} "
f"{mdd*100:6.2f}% {calmar:5.2f}")
print()
# ---------- Annual returns ----------
print("\n=== Annual returns (best L/S vs V5) ===")
a_v5 = annual_returns(v5r).rename("V5")
a_ls = annual_returns(lsr).rename(best_ls)
a_blend50 = annual_returns(0.5 * v5r + 0.5 * lsr).rename("Blend 50/50")
a_blend70 = annual_returns(0.7 * v5r + 0.3 * lsr).rename("Blend 70/30 V5/LS")
annuals = pd.concat([a_v5, a_ls, a_blend50, a_blend70], axis=1)
annuals = annuals.map(lambda x: f"{x*100:7.1f}%" if pd.notna(x) else "")
print(annuals.to_string())
if __name__ == "__main__":
main()

433
research/new_frameworks_eval.py
View File

@@ -1,433 +0,0 @@
"""New frameworks: Vol trading (#2) + Event-driven simplified (#4).
Framework 2: Volatility Trading
A. V7 risk-off → SVXY (harvest vol premium during normal risk-off)
B. Standalone SVXY mean-reversion (buy SVXY when VIX spikes + reverts)
C. V7 + SVXY risk-off with VIX gate (no SVXY when VIX > 30)
Framework 4: Event-Driven Simplified
D. Earnings calendar effect: avoid holding around earnings (high idio risk)
→ Not applicable to ETFs. Instead: monthly seasonality on leveraged ETFs
E. Turn-of-month effect (known anomaly: last 3 + first 3 days of month)
F. Holiday effect (market tends to rise before holidays)
G. Options expiration week effect (OpEx week has different dynamics)
Also: pure standalone strategies (not V7 modifications)
H. Pure SVXY with VIX regime gating
I. TQQQ buy-the-dip: buy TQQQ when RSI < 30, sell when RSI > 70
J. VIX term structure carry: long SVXY when VIX contango, cash when backwardation
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from main import backtest
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
from strategies.trend_rider_v7 import TrendRiderV7
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
# =========================================================================
# Framework 2: Volatility Trading Strategies
# =========================================================================
class V7SvxyRiskOff(Strategy):
"""V7 with SVXY as risk-off instrument (gated by VIX level).
During risk-off:
- If VIX < vix_gate: hold SVXY (harvest vol premium)
- If VIX >= vix_gate: hold GLD/DBC (traditional safe haven)
"""
def __init__(self, vix_gate=30, svxy="SVXY", **v7_kw):
self.vix_gate = vix_gate
self.svxy = svxy
self.v7 = TrendRiderV7(**v7_kw)
def generate_signals(self, data):
w = self.v7.generate_signals(data)
svxy = self.svxy
if svxy not in data.columns or "^VIX" not in data.columns:
return w
if svxy not in w.columns:
w[svxy] = 0.0
vix = data["^VIX"].shift(2).fillna(20) # PIT: 2-day lag matching V3
roff_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
for i in range(len(w)):
roff_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in roff_cols)
if roff_w < 1e-8:
continue
v = vix.iloc[i]
if np.isnan(v) or v >= self.vix_gate:
continue
# Replace risk-off with SVXY
for c in roff_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
w.iat[i, w.columns.get_loc(svxy)] = roff_w
return w
class PureSvxyStrategy(Strategy):
"""Standalone SVXY with VIX regime gating.
Hold SVXY when VIX < upper_gate AND VIX > VIX_MA (mean-reverting down).
Cash otherwise. Vol-target overlay for sizing.
"""
def __init__(self, upper_gate=25, ma_window=20, target_vol=0.20,
min_lev=0.3, svxy="SVXY"):
self.upper_gate = upper_gate
self.ma_window = ma_window
self.target_vol = target_vol
self.min_lev = min_lev
self.svxy = svxy
def generate_signals(self, data):
cols = [c for c in [self.svxy, "^VIX", "SHY"] if c in data.columns]
w = pd.DataFrame(0.0, index=data.index, columns=cols + [self.svxy])
if self.svxy not in data.columns or "^VIX" not in data.columns:
return w
vix = data["^VIX"]
vix_ma = vix.rolling(self.ma_window).mean()
# Hold SVXY when: VIX < gate AND VIX is falling (below its MA)
hold_signal = ((vix < self.upper_gate) & (vix < vix_ma)).shift(2).fillna(False)
w[self.svxy] = hold_signal.astype(float)
w = w.fillna(0.0)
# Vol-target
if self.svxy in data.columns:
rets = data[self.svxy].pct_change(fill_method=None).fillna(0.0)
rv = rets.rolling(60, min_periods=21).std() * np.sqrt(252)
scale = (self.target_vol / rv).clip(lower=self.min_lev, upper=1.0).shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
return w
class TqqqRsiStrategy(Strategy):
"""TQQQ buy-the-dip: buy when RSI < oversold, sell when RSI > overbought."""
def __init__(self, rsi_window=14, buy_level=30, sell_level=70,
target_vol=0.30, min_lev=0.5):
self.rsi_window = rsi_window
self.buy_level = buy_level
self.sell_level = sell_level
self.target_vol = target_vol
self.min_lev = min_lev
def generate_signals(self, data):
if "SPY" not in data.columns or "TQQQ" not in data.columns:
return pd.DataFrame(0.0, index=data.index, columns=data.columns)
# RSI on SPY (not TQQQ — SPY is less noisy)
spy = data["SPY"]
delta = spy.diff()
gain = delta.clip(lower=0)
loss = (-delta).clip(lower=0)
avg_gain = gain.rolling(self.rsi_window).mean()
avg_loss = loss.rolling(self.rsi_window).mean()
rs = avg_gain / avg_loss.clip(lower=1e-10)
rsi = 100 - (100 / (1 + rs))
rsi = rsi.shift(1) # PIT
cols = [c for c in data.columns]
w = pd.DataFrame(0.0, index=data.index, columns=cols)
in_position = False
for i in range(len(data)):
r = rsi.iloc[i]
if np.isnan(r):
continue
if not in_position and r < self.buy_level:
in_position = True
elif in_position and r > self.sell_level:
in_position = False
if in_position and "TQQQ" in w.columns:
w.iat[i, w.columns.get_loc("TQQQ")] = 1.0
# Vol-target
rets = data["TQQQ"].pct_change(fill_method=None).fillna(0.0) if "TQQQ" in data.columns else pd.Series(0.0, index=data.index)
rv = rets.rolling(60, min_periods=21).std() * np.sqrt(252)
scale = (self.target_vol / rv).clip(lower=self.min_lev, upper=1.0).shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
return w
# =========================================================================
# Framework 4: Calendar/Seasonality Strategies
# =========================================================================
class TurnOfMonthStrategy(Strategy):
"""Exploit turn-of-month anomaly: hold TQQQ last N + first M trading days.
Lakonishok & Smidt (1988): stocks earn disproportionate returns
at month boundaries. Payroll flows, pension rebalancing.
"""
def __init__(self, days_before=3, days_after=3, target_vol=0.36, min_lev=0.75):
self.days_before = days_before
self.days_after = days_after
self.target_vol = target_vol
self.min_lev = min_lev
def generate_signals(self, data):
cols = [c for c in data.columns]
w = pd.DataFrame(0.0, index=data.index, columns=cols)
if "TQQQ" not in data.columns:
return w
# Identify turn-of-month windows
dates = data.index
months = dates.to_period("M")
for period in months.unique():
month_mask = months == period
month_dates = dates[month_mask]
if len(month_dates) < 5:
continue
# Last N days of month
for d in month_dates[-self.days_before:]:
w.at[d, "TQQQ"] = 1.0
# First M days of month
for d in month_dates[:self.days_after]:
w.at[d, "TQQQ"] = 1.0
# Shift for PIT
w = w.shift(1).fillna(0.0)
# Vol-target
if "TQQQ" in data.columns:
rets = data["TQQQ"].pct_change(fill_method=None).fillna(0.0)
port_rets = (w["TQQQ"] * rets)
rv = port_rets.rolling(60, min_periods=21).std() * np.sqrt(252)
scale = (self.target_vol / rv).clip(lower=self.min_lev, upper=1.0).shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
return w
class V7TurnOfMonth(Strategy):
"""V7 but only enters risk-on during turn-of-month windows.
Combines V7 regime timing with monthly seasonality.
Outside the window: force to risk-off even if V7 says risk-on.
"""
def __init__(self, days_before=4, days_after=4, **v7_kw):
self.days_before = days_before
self.days_after = days_after
self.v7 = TrendRiderV7(**v7_kw)
def generate_signals(self, data):
w = self.v7.generate_signals(data)
dates = data.index
months = dates.to_period("M")
in_window = pd.Series(False, index=dates)
for period in months.unique():
month_dates = dates[months == period]
if len(month_dates) < 5:
continue
for d in month_dates[-self.days_before:]:
in_window[d] = True
for d in month_dates[:self.days_after]:
in_window[d] = True
risk_on_cols = [c for c in ["TQQQ", "UPRO"] if c in w.columns]
risk_off_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
park = "SHY" if "SHY" in w.columns else ""
for i in range(len(w)):
if in_window.iloc[i]:
continue
ron_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in risk_on_cols)
if ron_w > 0.01:
for c in risk_on_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if park and park in w.columns:
w.iat[i, w.columns.get_loc(park)] = ron_w
return w
# =========================================================================
# Hybrid: V7 risk-on + Vol premium risk-off
# =========================================================================
class V7VolHybrid(Strategy):
"""Best of both worlds: V7 regime for risk-on, SVXY for risk-off.
Risk-on: TQQQ/UPRO (V7 momentum pick) — trend alpha
Risk-off: SVXY when VIX < gate, GLD when VIX >= gate — vol premium alpha
Vol-target + PT on risk-on only.
Two independent alpha sources:
1. Equity trend momentum (V7's regime timing)
2. Volatility risk premium (SVXY during calm periods)
"""
def __init__(self, vix_gate=28, target_vol=0.36, min_lev=0.75,
pt_threshold=0.30, pt_band=0.10):
self.vix_gate = vix_gate
self.target_vol = target_vol
self.min_lev = min_lev
self.pt_threshold = pt_threshold
self.pt_band = pt_band
self.v3 = TrendRiderV3(signal="SPY", risk_on=("TQQQ", "UPRO"),
risk_off=("GLD", "DBC"), ma_long=150)
def generate_signals(self, data):
w = self.v3.generate_signals(data)
for col in ["SHY", "SVXY"]:
if col in data.columns and col not in w.columns:
w[col] = 0.0
has_vix = "^VIX" in data.columns
has_svxy = "SVXY" in data.columns
if has_vix and has_svxy:
vix = data["^VIX"].shift(2).fillna(20)
roff_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
for i in range(len(w)):
roff_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in roff_cols)
if roff_w < 1e-8: continue
v = vix.iloc[i]
if np.isnan(v) or v >= self.vix_gate: continue
for c in roff_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
w.iat[i, w.columns.get_loc("SVXY")] = roff_w
# Vol-target
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
common = w.columns.intersection(daily_ret.columns)
port_rets = (w[common] * daily_ret[common]).sum(axis=1)
rv = port_rets.rolling(60, min_periods=21).std() * np.sqrt(252)
scale = (self.target_vol / rv).clip(lower=self.min_lev, upper=1.0).shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
# PT on risk-on only
if self.pt_threshold <= 0: return w
risk_on_set = {"TQQQ", "UPRO"}
held = w.idxmax(axis=1); mx = w.max(axis=1); held[mx < 1e-8] = ""
park = "SHY" if "SHY" in w.columns else ""
ep, cs, stopped = None, None, False
rl = self.pt_threshold - self.pt_band
for i in range(len(w)):
sym = held.iloc[i]
if not sym or mx.iloc[i] < 1e-8: cs, ep, stopped = None, None, False; continue
if sym != cs:
cs = sym; ep = float(data[sym].iloc[i-1]) if i>0 and sym in data.columns else None; stopped = False; continue
if sym not in risk_on_set: continue
if ep is None or ep <= 0 or sym not in data.columns: continue
y = float(data[sym].iloc[i-1]) if i>0 else float(data[sym].iloc[i])
g = y/ep - 1.0
if stopped:
if g < rl: stopped = False
else: w.iloc[i] = 0.0; (w.at.__setitem__((w.index[i], park), scale.iloc[i]) if park else None)
elif g >= self.pt_threshold:
stopped = True; w.iloc[i] = 0.0; (w.at.__setitem__((w.index[i], park), scale.iloc[i]) if park else None)
return w
# =========================================================================
# Main
# =========================================================================
def main():
print("=" * 110)
print(" NEW FRAMEWORKS: VOL TRADING + CALENDAR EFFECTS")
print("=" * 110)
all_etfs = sorted(set([
"SPY", "QQQ", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "TLT",
"^VIX", "SVXY",
]))
data = data_manager.update("etfs", all_etfs, with_open=False)
if isinstance(data, tuple): data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
data = data[data.index >= cutoff]
print(f"Period: {data.index[0].date()} → {data.index[-1].date()}")
print(f"SVXY: {'yes' if 'SVXY' in data.columns else 'no'}, "
f"VIX: {'yes' if '^VIX' in data.columns else 'no'}")
results = []
def run(label, strategy):
try:
eq = backtest(strategy, data, initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
results.append((label, m))
print(f" {label:<55} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Calmar={m['calmarRatio']:.2f}")
except Exception as e:
print(f" {label:<55} FAILED: {e}")
# === Baseline ===
print("\n--- Baseline ---")
run("V7+VT36 baseline", TrendRiderV7(target_vol=0.36, min_lev=0.75))
# === Framework 2: Vol Trading ===
print("\n--- Framework 2A: V7 + SVXY risk-off ---")
for gate in (25, 28, 30, 35):
run(f"V7+SVXY risk-off (VIX gate={gate})",
V7SvxyRiskOff(vix_gate=gate, target_vol=0.36, min_lev=0.75))
print("\n--- Framework 2B: Pure SVXY strategies ---")
for gate in (20, 25, 30):
run(f"Pure SVXY (VIX<{gate} + falling)", PureSvxyStrategy(upper_gate=gate))
print("\n--- Framework 2C: TQQQ RSI buy-the-dip ---")
for buy, sell in [(25, 70), (30, 70), (30, 65), (20, 75)]:
run(f"TQQQ RSI buy<{buy}/sell>{sell}", TqqqRsiStrategy(buy_level=buy, sell_level=sell))
print("\n--- Framework 2D: V7+Vol Hybrid (best of both) ---")
for gate in (25, 28, 30):
run(f"V7VolHybrid (VIX gate={gate})",
V7VolHybrid(vix_gate=gate))
# === Framework 4: Calendar Effects ===
print("\n--- Framework 4A: Turn-of-month TQQQ ---")
for before, after in [(3, 3), (4, 4), (2, 5)]:
run(f"TQQQ turn-of-month ({before}d before + {after}d after)",
TurnOfMonthStrategy(days_before=before, days_after=after))
print("\n--- Framework 4B: V7 + turn-of-month filter ---")
for before, after in [(4, 4), (5, 5), (3, 6)]:
run(f"V7 risk-on only in TOM window ({before}+{after}d)",
V7TurnOfMonth(days_before=before, days_after=after,
target_vol=0.36, min_lev=0.75))
# === Final ranking ===
results.sort(key=lambda x: x[1]["sharpeRatio"], reverse=True)
print(f"\n{'=' * 115}")
print(" FINAL RANKING (by Sharpe)")
print(f"{'=' * 115}")
print(f"{'#':<4} {'Strategy':<55} {'Ann%':>6} {'Vol%':>6} {'Sharpe':>7} "
f"{'Sortino':>8} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 115)
for i, (label, m) in enumerate(results, 1):
marker = " ★" if i <= 3 else ""
print(f"{i:<4} {label:<55} "
f"{m['annualizedReturn']*100:>5.1f}% "
f"{m['annualizedVolatility']*100:>5.1f}% "
f"{m['sharpeRatio']:>7.2f} {m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>6.1f}% {m['calmarRatio']:>7.2f}{marker}")
print(f"{'=' * 115}")
results.sort(key=lambda x: x[1]["annualizedReturn"], reverse=True)
print(f"\n Top 5 by Ann Return:")
for i, (label, m) in enumerate(results[:5], 1):
print(f" {i}. {label:<50} Ann={m['annualizedReturn']*100:.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}%")
if __name__ == "__main__":
main()

299
research/optimize.py
View File

@@ -1,299 +0,0 @@
"""
End-to-end optimization study for the US recovery+momentum strategy family,
run on a point-in-time (survivorship-bias-mitigated) S&P 500 universe.
Experiments:
E1 — Baseline drift: biased vs point-in-time universe, current top10 params.
E2 — Hyperparameter sweep with 2016-2022 train / 2023-2026 test split.
E3 — SPY MA200 regime filter (compare base vs filtered).
E4 — Weighting schemes: equal vs inverse-vol vs rank.
E5 — Ensemble of top-3 uncorrelated configs.
Usage: uv run python -m research.optimize
"""
import os
import numpy as np
import pandas as pd
import data_manager
import research.pit_backtest as pit
from research.strategies_plus import (EnsembleStrategy, RecoveryMomentumPlus,
spy_ma200_filter)
from strategies.recovery_momentum import RecoveryMomentumStrategy
DATA_DIR = "data"
BIASED_CSV = os.path.join(DATA_DIR, "us.csv")
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def slice_period(df: pd.DataFrame, start: str | None, end: str | None) -> pd.DataFrame:
out = df
if start:
out = out[out.index >= start]
if end:
out = out[out.index <= end]
return out
def run_strategy(strategy, prices, benchmark=None, regime_filter=None,
fixed_fee: float = 0.0) -> pd.Series:
return pit.backtest(
strategy=strategy, prices=prices, initial_capital=10_000,
transaction_cost=0.001, fixed_fee=fixed_fee,
benchmark=benchmark, regime_filter=regime_filter,
)
# ---------------------------------------------------------------------------
# Experiment 1: bias drift
# ---------------------------------------------------------------------------
def exp1_bias_drift(pit_prices_masked: pd.DataFrame) -> pd.DataFrame:
print("\n" + "=" * 90)
print("E1 — Biased universe vs Point-in-time universe (recovery_mom_top10)")
print("=" * 90)
rows = []
# Biased: current 503 tickers extrapolated backward
biased = pd.read_csv(BIASED_CSV, index_col=0, parse_dates=True)
# Use same date range as PIT for a fair comparison
common_start = max(biased.index[0], pit_prices_masked.index[0])
common_end = min(biased.index[-1], pit_prices_masked.index[-1])
biased_window = slice_period(biased, str(common_start.date()), str(common_end.date()))
pit_window = slice_period(pit_prices_masked, str(common_start.date()), str(common_end.date()))
# Drop non-ticker columns (SPY is in PIT but not in the masked tickers)
biased_tickers = [c for c in biased_window.columns if c != "SPY"]
pit_tickers = [c for c in pit_window.columns if c != "SPY"]
# Use RecoveryMomentumPlus with identical defaults to recovery_mom_top10.
# The original strategy uses na_option="bottom" which misranks NaN-masked
# data (non-members appear "top"); the Plus variant uses na_option="keep".
strat = RecoveryMomentumPlus(top_n=10) # defaults match RecoveryMomentumStrategy
eq_biased = run_strategy(strat, biased_window[biased_tickers])
eq_pit = run_strategy(RecoveryMomentumPlus(top_n=10), pit_window[pit_tickers])
rows.append(pit.summarize(eq_biased, name="recovery_mom_top10 (BIASED)"))
rows.append(pit.summarize(eq_pit, name="recovery_mom_top10 (POINT-IN-TIME)"))
# Benchmark: SPY buy-and-hold in same window
if "SPY" in biased_window.columns:
spy_bh = (biased_window["SPY"] / biased_window["SPY"].iloc[0]) * 10_000
rows.append(pit.summarize(spy_bh, name="SPY buy-and-hold"))
for r in rows:
print(pit.fmt_row(r))
return pd.DataFrame(rows)
# ---------------------------------------------------------------------------
# Experiment 2: hyperparameter sweep with train/test split
# ---------------------------------------------------------------------------
def exp2_sweep(pit_masked: pd.DataFrame) -> pd.DataFrame:
print("\n" + "=" * 90)
print("E2 — Hyperparameter sweep (train: 2016-2022, test: 2023-2026)")
print("=" * 90)
tickers = [c for c in pit_masked.columns if c != "SPY"]
prices = pit_masked[tickers]
train = slice_period(prices, "2016-04-01", "2022-12-31")
test = slice_period(prices, "2023-01-01", None)
grid = []
for top_n in [5, 8, 10, 15]:
for rec_win in [42, 63, 126]:
for rec_w in [0.3, 0.5, 0.7]:
for rebal in [10, 21]:
grid.append(dict(top_n=top_n, recovery_window=rec_win,
rec_weight=rec_w, rebal_freq=rebal))
results = []
for i, cfg in enumerate(grid):
strat_train = RecoveryMomentumPlus(**cfg)
eq_tr = run_strategy(strat_train, train)
sum_tr = pit.summarize(eq_tr, name="train")
strat_test = RecoveryMomentumPlus(**cfg)
eq_te = run_strategy(strat_test, test)
sum_te = pit.summarize(eq_te, name="test")
results.append({
**cfg,
"train_CAGR": sum_tr["CAGR"],
"train_Sharpe": sum_tr["Sharpe"],
"train_MaxDD": sum_tr["MaxDD"],
"test_CAGR": sum_te["CAGR"],
"test_Sharpe": sum_te["Sharpe"],
"test_MaxDD": sum_te["MaxDD"],
"test_Calmar": sum_te["Calmar"],
})
if (i + 1) % 10 == 0 or i == len(grid) - 1:
print(f" ... {i+1}/{len(grid)} configs evaluated")
df = pd.DataFrame(results)
df = df.sort_values("test_Sharpe", ascending=False)
# Print top 10 by TEST Sharpe, then top 10 by TRAIN Sharpe to see overfit gap
print("\n --- Top 10 by TEST Sharpe (out-of-sample, 2023-2026) ---")
disp_cols = ["top_n", "recovery_window", "rec_weight", "rebal_freq",
"train_Sharpe", "test_Sharpe", "train_CAGR", "test_CAGR",
"test_MaxDD", "test_Calmar"]
print(df.head(10)[disp_cols].to_string(index=False,
formatters={"train_Sharpe": "{:.2f}".format, "test_Sharpe": "{:.2f}".format,
"train_CAGR": "{:.1%}".format, "test_CAGR": "{:.1%}".format,
"test_MaxDD": "{:.1%}".format, "test_Calmar": "{:.2f}".format}))
print("\n --- Top 10 by TRAIN Sharpe (for comparison / overfit check) ---")
df_tr = df.sort_values("train_Sharpe", ascending=False)
print(df_tr.head(10)[disp_cols].to_string(index=False,
formatters={"train_Sharpe": "{:.2f}".format, "test_Sharpe": "{:.2f}".format,
"train_CAGR": "{:.1%}".format, "test_CAGR": "{:.1%}".format,
"test_MaxDD": "{:.1%}".format, "test_Calmar": "{:.2f}".format}))
return df
# ---------------------------------------------------------------------------
# Experiment 3: regime filter
# ---------------------------------------------------------------------------
def exp3_regime(pit_masked: pd.DataFrame) -> pd.DataFrame:
print("\n" + "=" * 90)
print("E3 — SPY MA200 regime filter (out-of-sample 2023-2026)")
print("=" * 90)
tickers = [c for c in pit_masked.columns if c != "SPY"]
# Compute MA from FULL history so the filter is warmed up before 2023.
spy_full = pit_masked["SPY"].dropna() if "SPY" in pit_masked.columns else None
filt_full_200 = spy_ma200_filter(spy_full, ma_window=200) if spy_full is not None else None
filt_full_150 = spy_ma200_filter(spy_full, ma_window=150) if spy_full is not None else None
test = slice_period(pit_masked, "2023-01-01", None)
prices = test[tickers]
filt = filt_full_200.reindex(test.index).fillna(False).astype(bool) if filt_full_200 is not None else None
filt_150 = filt_full_150.reindex(test.index).fillna(False).astype(bool) if filt_full_150 is not None else None
rows = []
base = RecoveryMomentumPlus(top_n=10)
rows.append(pit.summarize(run_strategy(base, prices), name="top10 (no filter)"))
rows.append(pit.summarize(run_strategy(RecoveryMomentumPlus(top_n=10), prices,
regime_filter=filt),
name="top10 + SPY>MA200 filter"))
rows.append(pit.summarize(run_strategy(RecoveryMomentumPlus(top_n=10), prices,
regime_filter=filt_150),
name="top10 + SPY>MA150 filter"))
for r in rows:
print(pit.fmt_row(r))
return pd.DataFrame(rows)
# ---------------------------------------------------------------------------
# Experiment 4: weighting schemes
# ---------------------------------------------------------------------------
def exp4_weighting(pit_masked: pd.DataFrame) -> pd.DataFrame:
print("\n" + "=" * 90)
print("E4 — Weighting schemes (out-of-sample 2023-2026, top_n=10)")
print("=" * 90)
tickers = [c for c in pit_masked.columns if c != "SPY"]
test = slice_period(pit_masked[tickers], "2023-01-01", None)
rows = []
for w in ["equal", "inv_vol", "rank"]:
strat = RecoveryMomentumPlus(top_n=10, weighting=w)
eq = run_strategy(strat, test)
rows.append(pit.summarize(eq, name=f"top10 weighting={w}"))
for r in rows:
print(pit.fmt_row(r))
return pd.DataFrame(rows)
# ---------------------------------------------------------------------------
# Experiment 5: ensemble
# ---------------------------------------------------------------------------
def exp5_ensemble(pit_masked: pd.DataFrame, sweep_df: pd.DataFrame) -> pd.DataFrame:
print("\n" + "=" * 90)
print("E5 — Ensemble of 3 uncorrelated top configs (out-of-sample 2023-2026)")
print("=" * 90)
tickers = [c for c in pit_masked.columns if c != "SPY"]
test = slice_period(pit_masked[tickers], "2023-01-01", None)
# Pick top-20 by test_Sharpe, then greedily keep picks whose equity curves
# correlate < 0.9 with already-kept picks.
top20 = sweep_df.sort_values("test_Sharpe", ascending=False).head(20)
curves = []
components = []
for _, row in top20.iterrows():
cfg = dict(top_n=int(row["top_n"]),
recovery_window=int(row["recovery_window"]),
rec_weight=float(row["rec_weight"]),
rebal_freq=int(row["rebal_freq"]))
strat = RecoveryMomentumPlus(**cfg)
eq = run_strategy(strat, test)
if any(eq.pct_change().corr(c.pct_change()) > 0.9 for c in curves):
continue
curves.append(eq)
components.append((RecoveryMomentumPlus(**cfg), 1.0))
if len(components) >= 3:
break
print(f" Selected {len(components)} uncorrelated configs for ensemble:")
for strat, _ in components:
print(f" top_n={strat.top_n}, rec_win={strat.recovery_window}, "
f"rec_w={strat.rec_weight}, rebal={strat.rebal_freq}")
ens = EnsembleStrategy(components)
eq_ens = run_strategy(ens, test)
rows = [
pit.summarize(curves[i], name=f" component {i+1}") for i in range(len(curves))
]
rows.append(pit.summarize(eq_ens, name="ENSEMBLE (equal-weight)"))
# Also ensemble + regime filter (compute MA from full history)
if "SPY" in pit_masked.columns:
spy_full = pit_masked["SPY"].dropna()
filt = spy_ma200_filter(spy_full).reindex(test.index).fillna(False).astype(bool)
eq_ens_reg = run_strategy(EnsembleStrategy(components), test, regime_filter=filt)
rows.append(pit.summarize(eq_ens_reg, name="ENSEMBLE + SPY>MA200 filter"))
for r in rows:
print(pit.fmt_row(r))
return pd.DataFrame(rows)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
print("Loading point-in-time price data...")
raw = pit.load_pit_prices()
print(f" Raw (union) shape: {raw.shape}, {raw.index[0].date()} → {raw.index[-1].date()}")
masked = pit.pit_universe(raw)
# Sanity: how many ticker-days are masked out?
total = masked.size
valid = masked.notna().sum().sum()
print(f" Point-in-time valid ticker-days: {valid:,} / {total:,} ({valid/total*100:.1f}%)")
daily_universe = masked.notna().sum(axis=1)
print(f" Universe size per day: min={daily_universe.min()}, median={int(daily_universe.median())}, max={daily_universe.max()}")
e1 = exp1_bias_drift(masked)
sweep = exp2_sweep(masked)
e3 = exp3_regime(masked)
e4 = exp4_weighting(masked)
e5 = exp5_ensemble(masked, sweep)
# Save sweep for inspection
out = os.path.join(DATA_DIR, "research_sweep.csv")
sweep.to_csv(out, index=False)
print(f"\n Full sweep saved to {out}")
if __name__ == "__main__":
main()

125
research/pit_backtest.py
View File

@@ -1,125 +0,0 @@
"""
Point-in-time backtest runner.
Key idea: mask price data to NaN outside S&P 500 membership windows before
passing to the strategy. The strategy's signal computations then naturally
exclude non-members — no refactoring of strategies required.
Caveat: a stock joining the index has no signal for ~252 days after joining
(rolling windows need non-NaN warm-up). This is conservative but unbiased.
"""
import os
import numpy as np
import pandas as pd
import metrics
import universe_history as uh
DATA_DIR = "data"
PIT_CSV = os.path.join(DATA_DIR, "us_pit.csv")
# ---------------------------------------------------------------------------
# Data loading
# ---------------------------------------------------------------------------
def load_pit_prices() -> pd.DataFrame:
"""Load the full historical S&P 500 price matrix (delisted included)."""
if not os.path.exists(PIT_CSV):
raise FileNotFoundError(
f"{PIT_CSV} not found. Run `uv run python -m research.fetch_historical` first."
)
df = pd.read_csv(PIT_CSV, index_col=0, parse_dates=True)
return df.sort_index()
def pit_universe(prices: pd.DataFrame) -> pd.DataFrame:
"""Return prices masked to S&P 500 membership at each date (NaN outside)."""
intervals = uh.load_sp500_history()
return uh.mask_prices(prices, intervals)
# ---------------------------------------------------------------------------
# Backtest engine (mirrors main.backtest but accepts masked prices)
# ---------------------------------------------------------------------------
def backtest(
strategy,
prices: pd.DataFrame,
initial_capital: float = 10_000,
transaction_cost: float = 0.001,
fixed_fee: float = 0.0,
benchmark: pd.Series | None = None,
regime_filter: pd.Series | None = None,
) -> pd.Series:
"""
Vectorized backtest with optional regime filter.
`regime_filter`: boolean series aligned to prices.index. True → be in the
market (use strategy weights). False → go to cash. When None, always invested.
"""
weights = strategy.generate_signals(prices)
weights = weights.reindex(prices.index).fillna(0.0)
if regime_filter is not None:
rf = regime_filter.reindex(prices.index).fillna(False).astype(float)
weights = weights.mul(rf, axis=0)
daily_returns = prices.pct_change().fillna(0.0)
portfolio_returns = (daily_returns * weights).sum(axis=1)
turnover = weights.diff().abs().sum(axis=1).fillna(0.0)
portfolio_returns -= turnover * transaction_cost
if fixed_fee > 0:
weight_changes = weights.diff().fillna(0.0)
n_trades = (weight_changes.abs() > 1e-8).sum(axis=1)
equity_running = (1 + portfolio_returns).cumprod() * initial_capital
fee_impact = (n_trades * fixed_fee) / equity_running.shift(1).fillna(initial_capital)
portfolio_returns -= fee_impact
equity = (1 + portfolio_returns).cumprod() * initial_capital
return equity
# ---------------------------------------------------------------------------
# Metrics helper
# ---------------------------------------------------------------------------
def summarize(equity: pd.Series, name: str = "") -> dict:
"""Return a dict of key performance metrics (no printing)."""
eq = equity.dropna()
if len(eq) < 2:
return {"name": name, "error": "insufficient data"}
daily = eq.pct_change().dropna()
total_return = eq.iloc[-1] / eq.iloc[0] - 1
years = (eq.index[-1] - eq.index[0]).days / 365.25
cagr = (eq.iloc[-1] / eq.iloc[0]) ** (1 / years) - 1 if years > 0 else 0.0
vol = daily.std() * np.sqrt(252)
sharpe = (daily.mean() * 252) / vol if vol > 0 else 0.0
downside = daily[daily < 0].std() * np.sqrt(252)
sortino = (daily.mean() * 252) / downside if downside > 0 else 0.0
dd = (eq / eq.cummax() - 1).min()
calmar = cagr / abs(dd) if dd < 0 else 0.0
return {
"name": name,
"CAGR": cagr,
"Sharpe": sharpe,
"Sortino": sortino,
"MaxDD": dd,
"Calmar": calmar,
"TotalRet": total_return,
"Vol": vol,
}
def fmt_row(r: dict) -> str:
return (f" {r['name']:<38s} "
f"CAGR={r['CAGR']*100:>6.1f}% "
f"Sharpe={r['Sharpe']:>5.2f} "
f"Sortino={r['Sortino']:>5.2f} "
f"MaxDD={r['MaxDD']*100:>6.1f}% "
f"Calmar={r['Calmar']:>5.2f} "
f"Total={r['TotalRet']*100:>7.1f}%")

234
research/pit_comparison.py
View File

@@ -1,234 +0,0 @@
"""
PIT-compliant backtest: mask prices to historical S&P 500 membership.
Compares:
1. BIASED: current S&P 500 constituents applied back to 2016 (what we had before)
2. PIT: historical membership mask — each date only sees stocks that were
actually S&P 500 members on that date
This isolates the survivorship bias in our previous results.
"""
from __future__ import annotations
import os, sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.ensemble_alpha import SharpeBoostedEnsembleStrategy
import universe_history as uh
from research.pit_backtest import load_pit_prices, pit_universe
def compute_metrics(daily_rets: pd.Series) -> dict:
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {"cagr": cagr, "vol": vol, "sharpe": sharpe, "max_dd": max_dd, "calmar": calmar}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
def run_strategy(data: pd.DataFrame, start="2016-10-01", end="2026-05-13"):
"""Run SharpeBoostedEnsembleStrategy on given price data."""
strat = SharpeBoostedEnsembleStrategy()
weights = strat.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:end]
def main():
print("=" * 90)
print("SURVIVORSHIP BIAS TEST: PIT Membership vs Current Constituents")
print("=" * 90)
# --- Load PIT prices (includes delisted stocks) ---
print("\n--- Loading PIT price data ---")
pit_prices_raw = load_pit_prices()
print(f" Raw PIT prices: {pit_prices_raw.shape}")
# --- Apply PIT membership mask ---
print("\n--- Applying PIT membership mask ---")
intervals = uh.load_sp500_history()
pit_prices = pit_universe(pit_prices_raw)
print(f" PIT-masked prices: {pit_prices.shape}")
# Show how many stocks are available at various dates
for d in ["2016-12-30", "2018-12-31", "2020-12-31", "2022-12-30", "2024-12-31"]:
if d in pit_prices.index.strftime("%Y-%m-%d").tolist():
n_avail = pit_prices.loc[d].notna().sum()
print(f" {d}: {n_avail} stocks available")
else:
# Find nearest date
idx = pit_prices.index.get_indexer([pd.Timestamp(d)], method="nearest")
actual = pit_prices.index[idx[0]]
n_avail = pit_prices.loc[actual].notna().sum()
print(f" {actual.strftime('%Y-%m-%d')}: {n_avail} stocks available")
# --- Create biased version: use all stocks in us_pit (no mask) ---
# This simulates "using today's S&P 500 back in 2016"
biased_prices = pit_prices_raw.copy()
print(f"\n Biased (no mask) prices: {biased_prices.shape}")
# --- Run strategy on both ---
# Use start=2016-10-01 because PIT data starts 2016-04-19 and we need
# 252 days of warmup
start = "2017-06-01" # ~252 trading days after 2016-04-19
end = "2026-05-13"
print(f"\n--- Running strategy ({start} to {end}) ---")
print(" Running on PIT-masked data...")
pit_rets = run_strategy(pit_prices, start=start, end=end)
pit_m = compute_metrics(pit_rets)
print(" Running on biased data (no mask)...")
biased_rets = run_strategy(biased_prices, start=start, end=end)
biased_m = compute_metrics(biased_rets)
# --- Also compare with SPY ---
spy_rets = pit_prices_raw["SPY"].pct_change().fillna(0.0).loc[start:end]
spy_m = compute_metrics(spy_rets)
# --- Results ---
print(f"\n{'=' * 90}")
print("RESULTS COMPARISON")
print(f"{'=' * 90}")
print(f"{'Metric':<12s} {'PIT (correct)':>16s} {'Biased (no mask)':>18s} {'SPY':>12s}")
print("-" * 60)
for metric, fmt in [("cagr", "{:.1f}%"), ("vol", "{:.1f}%"), ("sharpe", "{:.2f}"),
("max_dd", "{:.1f}%"), ("calmar", "{:.2f}")]:
scale = 100 if "%" in fmt else 1
pit_val = pit_m[metric] * scale
biased_val = biased_m[metric] * scale
spy_val = spy_m[metric] * scale
print(f" {metric:<12s} {fmt.format(pit_val):>16s} {fmt.format(biased_val):>18s} {fmt.format(spy_val):>12s}")
# --- Yearly comparison ---
print(f"\n{'=' * 90}")
print("YEARLY RETURNS")
print(f"{'=' * 90}")
pit_yr = yearly_returns(pit_rets)
biased_yr = yearly_returns(biased_rets)
spy_yr = yearly_returns(spy_rets)
print(f" {'Year':>4s} {'PIT':>10s} {'Biased':>10s} {'Delta':>10s} {'SPY':>10s}")
print(f" {'-'*50}")
for year in sorted(set(pit_yr.index) | set(biased_yr.index)):
p = pit_yr.get(year, float("nan"))
b = biased_yr.get(year, float("nan"))
s = spy_yr.get(year, float("nan"))
delta = p - b if not (np.isnan(p) or np.isnan(b)) else float("nan")
print(f" {year:>4d} {p*100:>+9.1f}% {b*100:>+9.1f}% {delta*100:>+9.1f}pp {s*100:>+9.1f}%")
# --- Analyze which stocks are affected ---
print(f"\n{'=' * 90}")
print("SURVIVORSHIP BIAS ANALYSIS")
print(f"{'=' * 90}")
# Find stocks that are NOT in current S&P 500 but WERE members historically
from universe import get_sp500
current_sp500 = set(get_sp500())
# Stocks removed from S&P 500 during our backtest period (2016-2026)
removed_during = []
added_during = []
for ticker, ivs in intervals.items():
for start_d, end_d in ivs:
if end_d and "2016" <= end_d <= "2026":
removed_during.append((ticker, end_d))
if start_d and "2016" <= start_d <= "2026":
added_during.append((ticker, start_d))
removed_during.sort(key=lambda x: x[1])
added_during.sort(key=lambda x: x[1])
print(f"\n Stocks REMOVED from S&P 500 during 2016-2026: {len(removed_during)}")
print(f" Stocks ADDED to S&P 500 during 2016-2026: {len(added_during)}")
print(f"\n Most impactful removals (stocks that biased backtest would wrongly exclude):")
# Check which removed stocks had price data and what happened to them
removed_with_prices = []
for ticker, remove_date in removed_during:
if ticker in pit_prices_raw.columns:
# What was their return from when they were removed?
try:
remove_ts = pd.Timestamp(remove_date)
pre = pit_prices_raw.loc[:remove_ts, ticker].dropna()
if len(pre) > 63:
# Get 3-month return before removal
ret_3m = pre.iloc[-1] / pre.iloc[-63] - 1 if len(pre) > 63 else np.nan
removed_with_prices.append((ticker, remove_date, ret_3m))
except Exception:
pass
removed_with_prices.sort(key=lambda x: x[2] if not np.isnan(x[2]) else 0)
print(f" {'Ticker':<8s} {'Removed':>12s} {'3m ret before':>14s} {'Impact'}")
for ticker, rd, ret in removed_with_prices[:15]:
impact = "Would have been selected (recovery signal)" if ret < -0.20 else "Neutral"
print(f" {ticker:<8s} {rd:>12s} {ret*100:>+13.1f}% {impact}")
print(f"\n Notable ADDITIONS (stocks biased backtest wrongly includes early):")
# Key stocks that were added during our period
notable_adds = [(t, d) for t, d in added_during
if t in ["TSLA", "MRNA", "CVNA", "PLTR", "APP", "SMCI", "AXON", "SATS"]]
for ticker, add_date in notable_adds:
print(f" {ticker:<8s} added {add_date} — biased backtest selects it BEFORE this date!")
# --- Check: did we select any non-member stocks in PIT backtest? ---
print(f"\n{'=' * 90}")
print("PIT AUDIT: Verify no look-ahead in PIT backtest")
print(f"{'=' * 90}")
strat = SharpeBoostedEnsembleStrategy()
pit_weights = strat.generate_signals(pit_prices)
# For each date, check that all non-zero weight stocks are S&P 500 members
mask = uh.membership_mask(pit_prices.index, intervals, list(pit_prices.columns))
violations = 0
for date in pit_weights.index:
active = pit_weights.loc[date]
active_tickers = active[active > 0.001].index.tolist()
for t in active_tickers:
if t in mask.columns and not mask.loc[date, t]:
violations += 1
if violations <= 5:
print(f" VIOLATION: {t} selected on {date.strftime('%Y-%m-%d')} but NOT a member!")
if violations == 0:
print(" NO VIOLATIONS: All selected stocks were S&P 500 members on their selection date.")
else:
print(f" Total violations: {violations}")
# --- Bootstrap on PIT returns ---
print(f"\n{'=' * 90}")
print("BOOTSTRAP: PIT-corrected returns")
print(f"{'=' * 90}")
from research.trend_rider_p0 import block_bootstrap
boot = block_bootstrap(pit_rets, n_boot=5000, block_len=42)
print(f" Sharpe: median={boot['sharpe'].median():.2f} "
f"5th={boot['sharpe'].quantile(0.05):.2f} "
f"95th={boot['sharpe'].quantile(0.95):.2f}")
print(f" CAGR: median={boot['cagr'].median()*100:.1f}% "
f"5th={boot['cagr'].quantile(0.05)*100:.1f}% "
f"95th={boot['cagr'].quantile(0.95)*100:.1f}%")
print(f" MaxDD: median={boot['max_drawdown'].median()*100:.1f}% "
f"5th={boot['max_drawdown'].quantile(0.05)*100:.1f}% "
f"95th={boot['max_drawdown'].quantile(0.95)*100:.1f}%")
print(f" P(Sharpe > 1.5): {(boot['sharpe'] > 1.5).mean()*100:.1f}%")
print(f" P(Sharpe > 1.0): {(boot['sharpe'] > 1.0).mean()*100:.1f}%")
if __name__ == "__main__":
main()

285
research/pit_optimization.py
View File

@@ -1,285 +0,0 @@
"""
PIT-compliant strategy optimization.
After fixing survivorship bias, CAGR dropped from 44.7% to 18.1% and Sharpe
from 1.52 to 0.84. The strategy barely beats SPY. Root causes:
1. Many top performers (CVNA, TSLA, MRNA, PLTR, APP) weren't in S&P 500
when the biased backtest selected them
2. "Bad" stocks removed from S&P 500 (PCG, M) WOULD have been selected by
recovery signals → losses not captured in biased backtest
Need to re-sweep parameters on PIT-corrected data:
- Maybe top_n needs to be different
- Rebalance frequency might need adjustment
- DD dampener parameters may need recalibration
- The signal itself might need modification
"""
from __future__ import annotations
import os, sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.base import Strategy
import universe_history as uh
from research.pit_backtest import load_pit_prices, pit_universe
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
def compute_metrics(daily_rets: pd.Series) -> dict:
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {"cagr": cagr, "vol": vol, "sharpe": sharpe, "max_dd": max_dd, "calmar": calmar}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
class PITEnsemble(Strategy):
"""Ensemble strategy with configurable params for PIT optimization."""
def __init__(self, top_n=12, rebal_freq=42, mom_blend=0.0,
asym_vol=True, asym_vol_floor=0.50,
dd_dampen=True, dd_floor=0.70, dd_denom=0.35,
mom_filter_on=True):
self.top_n = top_n
self.rebal_freq = rebal_freq
self.mom_blend = mom_blend
self.asym_vol = asym_vol
self.asym_vol_floor = asym_vol_floor
self.dd_dampen = dd_dampen
self.dd_floor = dd_floor
self.dd_denom = dd_denom
self.mom_filter_on = mom_filter_on
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
p = data
ret = p.pct_change()
# === Signal A: rec_mfilt + deep_upvol ===
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
if self.mom_filter_on:
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
else:
rec_mfilt = rec_126
rec_mfilt_r = _rank(rec_mfilt)
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r
# === Signal B: Recovery 63d + 12-1 momentum ===
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r
# === Signal C: Pure momentum ===
signal_c = mom_r
# === Ensemble ===
α = self.mom_blend
if α > 0:
ensemble = (1 - α) / 2 * signal_a + (1 - α) / 2 * signal_b + α * signal_c
else:
ensemble = 0.5 * signal_a + 0.5 * signal_b
# === Select top_n ===
rank = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# === Rebalance ===
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
signals = signals.shift(1).fillna(0.0)
# === Asymmetric vol ===
if self.asym_vol:
daily_rets = data.pct_change().fillna(0.0)
port_rets = (signals * daily_rets).sum(axis=1)
short_vol = port_rets.rolling(20, min_periods=10).std() * np.sqrt(252)
vol_median = short_vol.rolling(252, min_periods=126).median()
recent_ret = port_rets.rolling(20, min_periods=10).sum()
high_vol_neg = (short_vol > vol_median * 1.5) & (recent_ret < 0)
asym_scale = pd.Series(1.0, index=data.index)
asym_scale[high_vol_neg] = self.asym_vol_floor
signals = signals.mul(asym_scale.shift(1).fillna(1.0), axis=0)
# === DD dampener ===
if self.dd_dampen:
daily_rets = data.pct_change().fillna(0.0)
mkt_rets = daily_rets.mean(axis=1)
mkt_eq = (1 + mkt_rets).cumprod()
mkt_dd = mkt_eq / mkt_eq.cummax() - 1
dd_scale = (1.0 + mkt_dd / self.dd_denom).clip(lower=self.dd_floor, upper=1.0)
signals = signals.mul(dd_scale.shift(1).fillna(1.0), axis=0)
return signals
def run_strategy(strat, data, start="2017-06-01", end="2026-05-13"):
weights = strat.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:end]
def fmt_row(label, m):
return (f"{label:<50s} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>6.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>6.2f}")
def main():
print("=" * 90)
print("PIT-COMPLIANT STRATEGY OPTIMIZATION")
print("=" * 90)
# Load PIT data
pit_raw = load_pit_prices()
intervals = uh.load_sp500_history()
pit_data = uh.mask_prices(pit_raw, intervals)
print(f"PIT data: {pit_data.shape}")
# SPY benchmark
spy_rets = pit_raw["SPY"].pct_change().fillna(0.0).loc["2017-06-01":"2026-05-13"]
spy_m = compute_metrics(spy_rets)
print(f"\nSPY benchmark: CAGR {spy_m['cagr']*100:.1f}% Sharpe {spy_m['sharpe']:.2f}")
header = f"{'Config':<50s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>6s} {'MaxDD':>7s} {'Calmar':>6s}"
# --- Sweep 1: top_n ---
print(f"\n--- top_n sweep (rebal=42, no risk mgmt) ---")
print(header)
print("-" * 90)
for n in [8, 10, 12, 15, 20, 25, 30]:
strat = PITEnsemble(top_n=n, rebal_freq=42, asym_vol=False, dd_dampen=False)
rets = run_strategy(strat, pit_data)
m = compute_metrics(rets)
print(fmt_row(f"top_n={n}", m))
# --- Sweep 2: rebal frequency ---
print(f"\n--- rebal sweep (top_n=20, no risk mgmt) ---")
print(header)
print("-" * 90)
for freq in [21, 42, 63]:
strat = PITEnsemble(top_n=20, rebal_freq=freq, asym_vol=False, dd_dampen=False)
rets = run_strategy(strat, pit_data)
m = compute_metrics(rets)
print(fmt_row(f"rebal={freq}d, top20", m))
# --- Sweep 3: momentum blend ---
print(f"\n--- momentum blend (top_n=20, rebal=42, no risk mgmt) ---")
print(header)
print("-" * 90)
for α in [0.0, 0.20, 0.30, 0.50, 0.70, 1.0]:
strat = PITEnsemble(top_n=20, rebal_freq=42, mom_blend=α, asym_vol=False, dd_dampen=False)
rets = run_strategy(strat, pit_data)
m = compute_metrics(rets)
label = "pure recovery" if α == 0 else "pure momentum" if α == 1.0 else f"mom_blend={α:.0%}"
print(fmt_row(label, m))
# --- Sweep 4: without mom_filter (recovery signal catches more stocks) ---
print(f"\n--- mom_filter ON vs OFF (top_n=20, rebal=42) ---")
print(header)
print("-" * 90)
for mf in [True, False]:
strat = PITEnsemble(top_n=20, rebal_freq=42, mom_filter_on=mf, asym_vol=False, dd_dampen=False)
rets = run_strategy(strat, pit_data)
m = compute_metrics(rets)
print(fmt_row(f"mom_filter={'ON' if mf else 'OFF'}", m))
# --- Sweep 5: risk overlays on best raw config ---
print(f"\n--- Risk overlays (best raw config) ---")
print(header)
print("-" * 90)
configs = [
("raw (no risk)", dict(asym_vol=False, dd_dampen=False)),
("+ asym_vol", dict(asym_vol=True, dd_dampen=False)),
("+ DD dampener", dict(asym_vol=False, dd_dampen=True)),
("+ both", dict(asym_vol=True, dd_dampen=True)),
]
for label, kwargs in configs:
for n in [12, 20]:
strat = PITEnsemble(top_n=n, rebal_freq=42, **kwargs)
rets = run_strategy(strat, pit_data)
m = compute_metrics(rets)
print(fmt_row(f"top{n}, {label}", m))
# --- Best PIT config: yearly breakdown ---
print(f"\n{'=' * 90}")
print("BEST PIT CONFIG — yearly analysis")
print(f"{'=' * 90}")
# Run a broad sweep to find the best
best_sharpe = 0
best_label = ""
best_rets = None
for n in [12, 15, 20, 25]:
for freq in [21, 42, 63]:
for α in [0.0, 0.30, 0.50, 1.0]:
for asym in [False, True]:
for dd in [False, True]:
strat = PITEnsemble(top_n=n, rebal_freq=freq, mom_blend=α,
asym_vol=asym, dd_dampen=dd)
rets = run_strategy(strat, pit_data)
m = compute_metrics(rets)
if m["sharpe"] > best_sharpe:
best_sharpe = m["sharpe"]
best_label = f"top{n}_rebal{freq}_mom{α:.0%}_asym{asym}_dd{dd}"
best_rets = rets
best_m = m
print(f"Best config: {best_label}")
print(fmt_row("BEST", best_m))
print(f"\n--- Yearly ---")
yr = yearly_returns(best_rets)
spy_yr = yearly_returns(spy_rets)
print(f" {'Year':>4s} {'Strategy':>10s} {'SPY':>10s} {'Alpha':>10s}")
for year in sorted(yr.index):
s = spy_yr.get(year, float("nan"))
alpha = yr[year] - s
print(f" {year:>4d} {yr[year]*100:>+9.1f}% {s*100:>+9.1f}% {alpha*100:>+9.1f}pp")
# Bootstrap
print(f"\n--- Bootstrap ---")
from research.trend_rider_p0 import block_bootstrap
boot = block_bootstrap(best_rets, n_boot=5000, block_len=42)
print(f" Sharpe: median={boot['sharpe'].median():.2f} "
f"5th={boot['sharpe'].quantile(0.05):.2f} "
f"95th={boot['sharpe'].quantile(0.95):.2f}")
print(f" P(Sharpe > 1.0): {(boot['sharpe'] > 1.0).mean()*100:.1f}%")
print(f" P(Sharpe > SPY's {spy_m['sharpe']:.2f}): {(boot['sharpe'] > spy_m['sharpe']).mean()*100:.1f}%")
if __name__ == "__main__":
main()

26
research/regime_filters.py
View File

@@ -1,26 +0,0 @@
import pandas as pd
LONG_MA_WINDOW = 200
RS_WINDOW = 63
def build_regime_filter(etf_close: pd.DataFrame, market_col: str = "SPY") -> pd.Series:
"""Return a next-day tradable regime flag based on market trend and ETF leadership."""
prices = etf_close.sort_index()
if market_col not in prices.columns:
raise KeyError(f"{market_col} not found in etf_close")
market = prices[market_col]
market_ma = market.rolling(LONG_MA_WINDOW, min_periods=LONG_MA_WINDOW).mean()
market_ok = market.gt(market_ma)
rs = prices.pct_change(RS_WINDOW, fill_method=None)
non_market_rs = rs.drop(columns=[market_col], errors="ignore")
if non_market_rs.shape[1] == 0:
leader_ok = pd.Series(True, index=prices.index)
else:
leader_ok = non_market_rs.gt(rs[market_col], axis=0).any(axis=1)
regime = (market_ok & leader_ok).astype(bool)
return regime.shift(1, fill_value=False)

366
research/regime_stock_picker_eval.py
View File

@@ -1,366 +0,0 @@
"""Does V3 regime timing + S&P 500 stock picking improve over either alone?
Variants tested:
1. RegimeStockPicker top-10 — V3 regime, risk-on = top-10 momentum stocks
2. RegimeStockPicker top-20 — V3 regime, risk-on = top-20 momentum stocks
3. RegimeRecovery top-10 — V3 regime, risk-on = recovery+momentum top-10
4. RecoveryMomentum top-10 — pure stock picker, no regime filter (baseline)
5. TrendRider V7 — leveraged ETFs (current SOTA)
6. SPY buy-and-hold — benchmark
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
import universe_history as uh
from main import backtest
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.trend_rider_v7 import TrendRiderV7
from universe import UNIVERSES
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
# ---------------------------------------------------------------------------
# Strategy: V3 regime gate + cross-sectional momentum on S&P 500
# ---------------------------------------------------------------------------
class RegimeStockPicker(Strategy):
"""V3 macro regime + S&P 500 momentum stock selection.
Risk-on: equal-weight top-N stocks by ``mom_lookback``-day momentum.
Risk-off: momentum leader of (GLD, DBC).
"""
def __init__(
self,
stock_tickers: list[str],
top_n: int = 10,
signal: str = "SPY",
defensive: tuple[str, ...] = ("GLD", "DBC"),
ma_long: int = 150,
mom_lookback: int = 63,
rebal_every: int = 21,
):
self.stock_tickers = stock_tickers
self.top_n = top_n
self.signal = signal
self.defensive = defensive
self.ma_long = ma_long
self.mom_lookback = mom_lookback
self.rebal_every = rebal_every
self._v3 = TrendRiderV3(
signal=signal, risk_on=("TQQQ", "UPRO"), risk_off=defensive,
ma_long=ma_long,
)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
w = pd.DataFrame(np.nan, index=data.index, columns=data.columns)
if self.signal not in data.columns:
return w.fillna(0.0)
sig_arr = data[self.signal].to_numpy()
mom = data.pct_change(self.mom_lookback, fill_method=None)
avail_stocks = [t for t in self.stock_tickers if t in data.columns]
avail_def = [t for t in self.defensive if t in data.columns]
need = max(self.ma_long, self.mom_lookback + 1,
self._v3.vol_window + 1, self._v3.dd_window,
self._v3.peak_window) + 1
regime: str | None = None
bars = 0
for i in range(len(data)):
if i < need:
continue
closes = sig_arr[:i]
if np.isnan(closes[-1]):
continue
desired = self._v3._desired_regime(closes, regime)
changed = False
if regime is None:
regime, bars, changed = desired, 0, True
else:
bars += 1
if desired != regime and bars >= 15:
regime, bars, changed = desired, 0, True
if not changed and (i - need) % self.rebal_every != 0:
continue
row = {c: 0.0 for c in data.columns}
dt = data.index[i]
if regime == "risk_on":
m = mom.iloc[i][avail_stocks].dropna()
valid = m.index[data.loc[dt, m.index].notna()]
m = m[valid]
m = m[m > 0]
top = m.nlargest(min(self.top_n, len(m)))
if len(top) > 0:
wt = 1.0 / len(top)
for t in top.index:
row[t] = wt
elif avail_def:
row[avail_def[0]] = 1.0
else:
if avail_def:
dm = mom.iloc[i][avail_def].dropna()
best = dm.idxmax() if len(dm) > 0 else avail_def[0]
row[best] = 1.0
for c, v in row.items():
w.at[dt, c] = v
w = w.ffill().fillna(0.0)
return w.shift(1).fillna(0.0)
# ---------------------------------------------------------------------------
# Strategy: V3 regime gate + recovery-momentum composite on S&P 500
# ---------------------------------------------------------------------------
class RegimeRecoveryPicker(Strategy):
"""V3 regime + recovery-momentum composite for stock selection.
Uses the same factor as RecoveryMomentumStrategy but only during risk-on.
"""
def __init__(
self,
stock_tickers: list[str],
top_n: int = 10,
signal: str = "SPY",
defensive: tuple[str, ...] = ("GLD", "DBC"),
ma_long: int = 150,
recovery_window: int = 63,
mom_lookback: int = 252,
mom_skip: int = 21,
rebal_every: int = 21,
):
self.stock_tickers = stock_tickers
self.top_n = top_n
self.signal = signal
self.defensive = defensive
self.ma_long = ma_long
self.recovery_window = recovery_window
self.mom_lookback = mom_lookback
self.mom_skip = mom_skip
self.rebal_every = rebal_every
self._v3 = TrendRiderV3(
signal=signal, risk_on=("TQQQ", "UPRO"), risk_off=defensive,
ma_long=ma_long,
)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
w = pd.DataFrame(np.nan, index=data.index, columns=data.columns)
if self.signal not in data.columns:
return w.fillna(0.0)
sig_arr = data[self.signal].to_numpy()
stock_data = data[[t for t in self.stock_tickers if t in data.columns]]
recovery = stock_data / stock_data.rolling(self.recovery_window).min() - 1
momentum = stock_data.shift(self.mom_skip).pct_change(
self.mom_lookback - self.mom_skip, fill_method=None,
)
rec_rank = recovery.rank(axis=1, pct=True, na_option="keep")
mom_rank = momentum.rank(axis=1, pct=True, na_option="keep")
composite = 0.5 * rec_rank + 0.5 * mom_rank
stock_rank = composite.rank(axis=1, ascending=False, na_option="bottom")
def_mom = data[[t for t in self.defensive if t in data.columns]].pct_change(63, fill_method=None)
avail_def = [t for t in self.defensive if t in data.columns]
need = max(self.ma_long, self.mom_lookback + 1,
self._v3.vol_window + 1, self._v3.dd_window,
self._v3.peak_window, self.recovery_window) + 1
regime: str | None = None
bars = 0
for i in range(len(data)):
if i < need:
continue
closes = sig_arr[:i]
if np.isnan(closes[-1]):
continue
desired = self._v3._desired_regime(closes, regime)
changed = False
if regime is None:
regime, bars, changed = desired, 0, True
else:
bars += 1
if desired != regime and bars >= 15:
regime, bars, changed = desired, 0, True
if not changed and (i - need) % self.rebal_every != 0:
continue
row = {c: 0.0 for c in data.columns}
dt = data.index[i]
if regime == "risk_on":
ranks_i = stock_rank.iloc[i]
n_valid = composite.iloc[i].notna().sum()
if n_valid >= self.top_n:
top = ranks_i[ranks_i <= self.top_n].index
if len(top) > 0:
wt = 1.0 / len(top)
for t in top:
row[t] = wt
if sum(row.values()) < 0.01 and avail_def:
row[avail_def[0]] = 1.0
else:
if avail_def:
dm = def_mom.iloc[i][avail_def].dropna()
best = dm.idxmax() if len(dm) > 0 else avail_def[0]
row[best] = 1.0
for c, v in row.items():
w.at[dt, c] = v
w = w.ffill().fillna(0.0)
return w.shift(1).fillna(0.0)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
print("=" * 80)
print(" REGIME + STOCK PICKER EVALUATION")
print("=" * 80)
# --- Load S&P 500 data (PIT-safe) ---
print("\n[1/3] Loading S&P 500 universe (PIT)...")
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
pit_intervals = uh.load_sp500_history()
hist_tickers = uh.all_tickers_ever(pit_intervals)
etf_extra = ["SPY", "GLD", "DBC", "SHY", "TQQQ", "UPRO", "TLT"]
all_tickers = sorted(set(tickers + hist_tickers + etf_extra))
print(f" Downloading {len(all_tickers)} tickers...")
data = data_manager.update("us", all_tickers, with_open=False)
if isinstance(data, tuple):
data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
data = data[data.index >= cutoff]
data = uh.mask_prices(data, pit_intervals)
stock_tickers = [
t for t in data.columns
if t not in etf_extra and data[t].notna().any()
]
print(f" Period: {data.index[0].date()} → {data.index[-1].date()}")
print(f" Tradable stocks: {len(stock_tickers)}")
# --- Run all strategies ---
print("\n[2/3] Running strategies...")
results: dict[str, pd.Series] = {}
def run(name, strategy, price_data):
print(f" {name}...")
eq = backtest(strategy, price_data, initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
results[name] = eq
# 1-2. Regime + momentum top-N
for n in (10, 20):
run(f"Regime+Mom Top{n}",
RegimeStockPicker(stock_tickers=stock_tickers, top_n=n),
data)
# 3. Regime + recovery-momentum top-10
run("Regime+RecMom Top10",
RegimeRecoveryPicker(stock_tickers=stock_tickers, top_n=10),
data)
# 4. Regime + recovery-momentum top-20
run("Regime+RecMom Top20",
RegimeRecoveryPicker(stock_tickers=stock_tickers, top_n=20),
data)
# 5. Pure recovery momentum (no regime) — baseline
run("RecoveryMom Top10 (pure)",
RecoveryMomentumStrategy(top_n=10),
data[stock_tickers])
run("RecoveryMom Top20 (pure)",
RecoveryMomentumStrategy(top_n=20),
data[stock_tickers])
# 6. TrendRider V7 (leveraged ETFs)
etf_cols = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"]
if t in data.columns]
run("TrendRider V7 (3x ETFs)", TrendRiderV7(), data[etf_cols])
# 7. SPY benchmark
spy = data["SPY"].dropna()
results["SPY (benchmark)"] = (spy / spy.iloc[0]) * CAPITAL
# --- Report ---
print(f"\n[3/3] Results ({YEARS}y, ${CAPITAL:,.0f}, tx={TX_COST*100:.1f}bps + ${FIXED_FEE:.0f}/trade)")
print("=" * 100)
hdr = (f"{'Strategy':<30} {'Ann%':>8} {'Vol%':>8} {'Sharpe':>8} "
f"{'Sortino':>8} {'MaxDD%':>8} {'Calmar':>8} {'WinRate':>8}")
print(hdr)
print("-" * 100)
for name, eq in results.items():
m = metrics.raw_summary(eq)
print(f"{name:<30} {m['annualizedReturn']*100:>7.1f}% "
f"{m['annualizedVolatility']*100:>7.1f}% "
f"{m['sharpeRatio']:>8.2f} {m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>7.1f}% {m['calmarRatio']:>8.2f} "
f"{m['winRate']*100:>7.1f}%")
print("=" * 100)
# Yearly breakdown for top strategies
print("\n--- Yearly Returns ---")
yearly: dict[str, dict[str, float]] = {}
for name, eq in results.items():
yr = {}
for year, grp in eq.groupby(eq.index.year):
if len(grp) >= 2:
yr[str(year)] = grp.iloc[-1] / grp.iloc[0] - 1
yearly[name] = yr
all_years = sorted(set(y for d in yearly.values() for y in d))
header = f"{'Year':<6}" + "".join(f"{name[:20]:>22}" for name in results)
print(header)
print("-" * len(header))
for year in all_years:
cols = []
for name in results:
r = yearly[name].get(year)
cols.append(f"{r*100:>20.1f}%" if r is not None else f"{'—':>21}")
print(f"{year:<6}" + "".join(cols))
if __name__ == "__main__":
main()

136
research/run_interaction.py
View File

@@ -1,136 +0,0 @@
"""
Evaluate interaction/ensemble strategies on 1/3/5/10y PIT windows with a
proper 500-day warmup preload, so 252d-warmup strategies are active from the
measurement start.
"""
from __future__ import annotations
import os
import warnings
import pandas as pd
import research.pit_backtest as pit
from research.alpha_factors import AlphaFactorStrategy
from research.interaction_alpha import (MultiplicativeFactorStrategy,
SubStrategyEnsemble, VotingFactorStrategy,
default_ensemble)
from strategies.factor_combo import FactorComboStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=RuntimeWarning)
DATA_DIR = "data"
BENCHMARK = "SPY"
def load():
raw = pit.load_pit_prices()
masked = pit.pit_universe(raw)
masked[BENCHMARK] = raw[BENCHMARK]
return masked
def warmup_slice(df, years, warmup_days=500):
measurement_start = df.index[-1] - pd.DateOffset(years=years)
cutoff = max(df.index[0], measurement_start - pd.Timedelta(days=warmup_days * 1.5))
return df[df.index >= cutoff], measurement_start
def measure(eq, start, name=""):
eq = eq[eq.index >= start]
eq = eq / eq.iloc[0] * 10_000
return pit.summarize(eq, name=name)
def make_configs(mkt_ret):
pair = ["mom_12_1", "recovery_63"]
return {
"Mult(mom12×rec63) eq, tn=10":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=10, rebal_freq=21, mkt_returns=mkt_ret),
"Mult(mom12×rec63) eq, tn=15":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=15, rebal_freq=21, mkt_returns=mkt_ret),
"Mult(mom12×rec63) eq, rebal=10":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=10, rebal_freq=10, mkt_returns=mkt_ret),
"Mult(mom12×rec63) sig^2, tn=15":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=15, rebal_freq=21, mkt_returns=mkt_ret,
weighting="signal", signal_concentration=2.0),
"Mult(mom12×rec63) sig^4, tn=15":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=15, rebal_freq=21, mkt_returns=mkt_ret,
weighting="signal", signal_concentration=4.0),
"Mult(mom12×rec63) disp-scale":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=10, rebal_freq=21, mkt_returns=mkt_ret,
dispersion_scale=True),
"Mult(mom12×rec63) inv_vol":
lambda: MultiplicativeFactorStrategy(
factor_names=pair, top_n=10, rebal_freq=21, mkt_returns=mkt_ret,
weighting="inv_vol"),
"Ensemble3 (RM/upcap/mult)":
lambda: default_ensemble(mkt_ret),
"Recovery+Mom Top10":
lambda: RecoveryMomentumStrategy(top_n=10),
"fc_up_cap+mom_gap":
lambda: FactorComboStrategy("up_cap+mom_gap", rebal_freq=21, top_n=10),
}
def main():
print("Loading PIT data…")
masked = load()
tickers = [c for c in masked.columns if c != BENCHMARK]
mkt_ret = masked[BENCHMARK].pct_change(fill_method=None)
print(f" shape={masked.shape} range={masked.index[0].date()} → {masked.index[-1].date()}")
rows = []
for years in (10, 5, 3, 1):
sliced, start = warmup_slice(masked, years, warmup_days=500)
prices = sliced[tickers]
print(f"\n--- {years}y window "
f"(measure {start.date()} → {sliced.index[-1].date()}, "
f"warmup from {sliced.index[0].date()}) ---")
spy = sliced[BENCHMARK].dropna()
spy_eq = (spy / spy.iloc[0]) * 10_000
m = measure(spy_eq, start, "")
rows.append({"years": years, "strategy": "SPY buy-and-hold",
**{k: v for k, v in m.items() if k != "name"}})
configs = make_configs(mkt_ret)
for name, factory in configs.items():
strat = factory()
eq = pit.backtest(strategy=strat, prices=prices,
initial_capital=10_000, transaction_cost=0.001)
m = measure(eq, start, "")
rows.append({"years": years, "strategy": name,
**{k: v for k, v in m.items() if k != "name"}})
tail = [r for r in rows if r["years"] == years]
tail.sort(key=lambda r: r["Sharpe"], reverse=True)
for r in tail:
print(f" {r['strategy']:<34s} CAGR={r['CAGR']*100:>6.1f}% "
f"Sharpe={r['Sharpe']:>5.2f} Sortino={r['Sortino']:>5.2f} "
f"MaxDD={r['MaxDD']*100:>6.1f}% Calmar={r['Calmar']:>5.2f}")
df = pd.DataFrame(rows)
df.to_csv(os.path.join(DATA_DIR, "interaction_results.csv"), index=False)
print("\n=== Cross-window CAGR summary (sorted by 10y Sharpe) ===")
pv = df.pivot(index="strategy", columns="years", values="CAGR")
pv.columns = [f"CAGR_{y}y" for y in pv.columns]
sh10 = df[df["years"] == 10].set_index("strategy")["Sharpe"]
pv["Sharpe_10y"] = sh10
pv = pv.sort_values("Sharpe_10y", ascending=False)
print(pv.to_string(formatters={
"CAGR_10y": "{:.1%}".format, "CAGR_5y": "{:.1%}".format,
"CAGR_3y": "{:.1%}".format, "CAGR_1y": "{:.1%}".format,
"Sharpe_10y": "{:.2f}".format,
}))
if __name__ == "__main__":
main()

321
research/sharpe_blend.py
View File

@@ -1,321 +0,0 @@
"""
PIT-compliant Sharpe 1.5+ blend: V5 ETF timing + PIT stock-picking + cross-asset momentum.
Combines three uncorrelated alpha sources with a vol-target overlay.
All components are PIT-safe (ETF-only or membership-masked).
Run:
uv run python -m research.sharpe_blend
"""
from __future__ import annotations
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.permanent_yearly import load_etfs
from research.pit_backtest import load_pit_prices, pit_universe
from research.pit_optimization import PITEnsemble, compute_metrics
from research.trend_rider_robustness import portfolio_returns, evaluate_weights
from research.trend_rider_v6_eval import load_combined_panel
from strategies.cross_asset_momentum import CrossAssetMomentum
from strategies.trend_rider_v5 import TrendRiderV5
# ---------------------------------------------------------------------------
# Data loading
# ---------------------------------------------------------------------------
def load_all_data() -> tuple[pd.DataFrame, pd.DataFrame]:
"""Return (etf_panel, pit_stock_prices) aligned to common dates."""
# ETF panel for V5 and cross-asset
etf_panel = load_combined_panel()
# Ensure cross-asset ETFs are present (TLT, IEF)
extra_etfs = ["TLT", "IEF"]
missing = [t for t in extra_etfs if t not in etf_panel.columns]
if missing:
extra = load_etfs(missing, start="2013-06-01")
extra = extra.reindex(etf_panel.index).ffill()
etf_panel = etf_panel.join(extra, how="left")
# PIT-masked stock prices
pit_prices = load_pit_prices()
pit_masked = pit_universe(pit_prices)
return etf_panel, pit_masked
# ---------------------------------------------------------------------------
# Strategy runners — produce daily returns series
# ---------------------------------------------------------------------------
def run_v5(panel: pd.DataFrame, start: str = "2017-06-01") -> pd.Series:
"""TrendRiderV5 daily returns."""
v5 = TrendRiderV5()
weights = v5.generate_signals(panel)
rets = portfolio_returns(weights, panel, transaction_cost=0.001)
return rets.loc[start:]
def run_pit_stock(pit_prices: pd.DataFrame, start: str = "2017-06-01") -> pd.Series:
"""PIT stock-picking (cross-sectional momentum) daily returns."""
strat = PITEnsemble(
top_n=12, rebal_freq=42, mom_blend=1.0,
asym_vol=True, asym_vol_floor=0.50,
dd_dampen=False,
)
weights = strat.generate_signals(pit_prices)
daily_rets = (weights * pit_prices.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:]
def run_cross_asset(panel: pd.DataFrame, start: str = "2017-06-01") -> pd.Series:
"""Cross-asset time-series momentum daily returns."""
strat = CrossAssetMomentum(lookback=252, top_k=3, rebal_freq=21, vol_scale=True)
weights = strat.generate_signals(panel)
rets = portfolio_returns(weights, panel, transaction_cost=0.001)
return rets.loc[start:]
# ---------------------------------------------------------------------------
# Vol-target overlay (standalone, operates on combined returns)
# ---------------------------------------------------------------------------
def vol_target_returns(
combined_rets: pd.Series,
target_vol: float = 0.18,
vol_window: int = 20,
) -> pd.Series:
"""Scale combined returns by min(1, target_vol / realized_vol)."""
realized = combined_rets.rolling(vol_window).std(ddof=1) * np.sqrt(252)
realized = realized.shift(1).fillna(target_vol)
scale = (target_vol / realized.replace(0.0, np.nan)).clip(upper=1.0).fillna(1.0)
return combined_rets * scale
# ---------------------------------------------------------------------------
# Blend engine
# ---------------------------------------------------------------------------
def blend_returns(
rets_v5: pd.Series,
rets_stock: pd.Series,
rets_xasset: pd.Series,
w_v5: float = 0.50,
w_stock: float = 0.30,
w_xasset: float = 0.20,
) -> pd.Series:
"""Weighted blend of three strategy return streams."""
# Align to common dates
idx = rets_v5.index.intersection(rets_stock.index).intersection(rets_xasset.index)
return (w_v5 * rets_v5.loc[idx]
+ w_stock * rets_stock.loc[idx]
+ w_xasset * rets_xasset.loc[idx])
def inverse_vol_weights(
rets_v5: pd.Series,
rets_stock: pd.Series,
rets_xasset: pd.Series,
window: int = 63,
) -> tuple[float, float, float]:
"""Compute inverse-vol weights from trailing realized vol."""
vols = pd.DataFrame({
"v5": rets_v5.rolling(window).std() * np.sqrt(252),
"stock": rets_stock.rolling(window).std() * np.sqrt(252),
"xasset": rets_xasset.rolling(window).std() * np.sqrt(252),
}).iloc[-1]
inv = 1.0 / vols.replace(0, np.nan)
w = inv / inv.sum()
return w["v5"], w["stock"], w["xasset"]
# ---------------------------------------------------------------------------
# Sweep
# ---------------------------------------------------------------------------
BLEND_CONFIGS = [
("V5=50/Stock=30/XA=20", 0.50, 0.30, 0.20),
("V5=40/Stock=40/XA=20", 0.40, 0.40, 0.20),
("V5=60/Stock=20/XA=20", 0.60, 0.20, 0.20),
("V5=50/Stock=25/XA=25", 0.50, 0.25, 0.25),
("V5=45/Stock=35/XA=20", 0.45, 0.35, 0.20),
("V5=55/Stock=25/XA=20", 0.55, 0.25, 0.20),
]
VOL_TARGETS = [None, 0.15, 0.18, 0.20, 0.22, 0.25]
def run_sweep(rets_v5, rets_stock, rets_xasset) -> pd.DataFrame:
"""Sweep blend configs × vol targets, return summary DataFrame."""
rows = []
# Add inverse-vol config
iv_w = inverse_vol_weights(rets_v5, rets_stock, rets_xasset)
configs = list(BLEND_CONFIGS) + [
(f"InvVol({iv_w[0]:.0%}/{iv_w[1]:.0%}/{iv_w[2]:.0%})", *iv_w)
]
for name, wv, ws, wx in configs:
combined = blend_returns(rets_v5, rets_stock, rets_xasset, wv, ws, wx)
for tgt in VOL_TARGETS:
if tgt is not None:
final = vol_target_returns(combined, target_vol=tgt)
label = f"{name} | VT={tgt}"
else:
final = combined
label = f"{name} | no-VT"
m = compute_metrics(final)
m["label"] = label
m["w_v5"] = wv
m["w_stock"] = ws
m["w_xasset"] = wx
m["vol_target"] = tgt
rows.append(m)
df = pd.DataFrame(rows)
df = df.sort_values("sharpe", ascending=False).reset_index(drop=True)
return df
# ---------------------------------------------------------------------------
# Validation helpers
# ---------------------------------------------------------------------------
def is_oos_split(rets: pd.Series, split_date="2023-01-01"):
"""Split returns into IS and OOS."""
is_rets = rets[rets.index < split_date]
oos_rets = rets[rets.index >= split_date]
return is_rets, oos_rets
def block_bootstrap(rets: pd.Series, n_boot: int = 5000, block_size: int = 63) -> np.ndarray:
"""Block bootstrap of annualized Sharpe ratio."""
n = len(rets)
arr = rets.values
sharpes = np.empty(n_boot)
rng = np.random.default_rng(42)
n_blocks = int(np.ceil(n / block_size))
for i in range(n_boot):
starts = rng.integers(0, n - block_size, size=n_blocks)
sample = np.concatenate([arr[s:s + block_size] for s in starts])[:n]
mu = sample.mean()
sigma = sample.std(ddof=1)
sharpes[i] = mu / sigma * np.sqrt(252) if sigma > 0 else 0.0
return sharpes
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
print("=" * 80)
print("PIT-Compliant Multi-Strategy Blend — Sharpe 1.5+ Target")
print("=" * 80)
# Load data
print("\n[1] Loading data...")
etf_panel, pit_masked = load_all_data()
# Run individual strategies
print("\n[2] Running individual strategies...")
rets_v5 = run_v5(etf_panel)
rets_stock = run_pit_stock(pit_masked)
rets_xasset = run_cross_asset(etf_panel)
# Individual metrics
print("\n--- Individual Strategy Metrics ---")
for name, r in [("V5 ETF Timing", rets_v5),
("PIT Stock Momentum", rets_stock),
("Cross-Asset Momentum", rets_xasset)]:
m = compute_metrics(r)
print(f" {name:<25s} Sharpe={m['sharpe']:5.2f} CAGR={m['cagr']*100:5.1f}% "
f"Vol={m['vol']*100:5.1f}% MaxDD={m['max_dd']*100:5.1f}%")
# Correlation diagnostic
print("\n--- Correlation Matrix (daily returns) ---")
corr_df = pd.DataFrame({
"V5": rets_v5, "Stock": rets_stock, "XAsset": rets_xasset
}).dropna()
corr = corr_df.corr()
print(corr.to_string(float_format=lambda x: f"{x:.3f}"))
# Rolling correlation
print("\n--- Rolling 63d Correlations (mean / max) ---")
for pair in [("V5", "Stock"), ("V5", "XAsset"), ("Stock", "XAsset")]:
roll = corr_df[pair[0]].rolling(63).corr(corr_df[pair[1]])
print(f" {pair[0]:>8s} vs {pair[1]:<8s}: mean={roll.mean():.3f} max={roll.max():.3f}")
# Sweep
print("\n[3] Running blend sweep...")
results = run_sweep(rets_v5, rets_stock, rets_xasset)
print("\n--- Top 15 Configurations ---")
print(f" {'Label':<50s} {'Sharpe':>7s} {'CAGR':>7s} {'Vol':>7s} {'MaxDD':>7s} {'Calmar':>7s}")
for _, row in results.head(15).iterrows():
print(f" {row['label']:<50s} {row['sharpe']:7.2f} "
f"{row['cagr']*100:6.1f}% {row['vol']*100:6.1f}% "
f"{row['max_dd']*100:6.1f}% {row['calmar']:6.2f}")
# Best config validation
best = results.iloc[0]
print(f"\n--- Best Config: {best['label']} ---")
best_rets = blend_returns(rets_v5, rets_stock, rets_xasset,
best["w_v5"], best["w_stock"], best["w_xasset"])
if best["vol_target"] is not None:
best_rets = vol_target_returns(best_rets, target_vol=best["vol_target"])
# IS/OOS
print("\n[4] IS/OOS Validation (split: 2023-01-01)...")
is_rets, oos_rets = is_oos_split(best_rets)
is_m = compute_metrics(is_rets)
oos_m = compute_metrics(oos_rets)
print(f" IS (2017-2022): Sharpe={is_m['sharpe']:5.2f} CAGR={is_m['cagr']*100:5.1f}% MaxDD={is_m['max_dd']*100:5.1f}%")
print(f" OOS (2023-2026): Sharpe={oos_m['sharpe']:5.2f} CAGR={oos_m['cagr']*100:5.1f}% MaxDD={oos_m['max_dd']*100:5.1f}%")
print(f" OOS/IS ratio: {oos_m['sharpe']/is_m['sharpe']:.2f}" if is_m['sharpe'] > 0 else "")
# Bootstrap
print("\n[5] Block Bootstrap (5000 resamples, block=63d)...")
boot = block_bootstrap(best_rets, n_boot=5000)
print(f" Median Sharpe: {np.median(boot):.2f}")
print(f" 5th pctile: {np.percentile(boot, 5):.2f}")
print(f" 95th pctile: {np.percentile(boot, 95):.2f}")
print(f" P(Sharpe>1.0): {(boot > 1.0).mean()*100:.1f}%")
print(f" P(Sharpe>1.3): {(boot > 1.3).mean()*100:.1f}%")
print(f" P(Sharpe>1.5): {(boot > 1.5).mean()*100:.1f}%")
# Parameter sensitivity
print("\n[6] Parameter Sensitivity (±perturbation on blend weights)...")
base_w = (best["w_v5"], best["w_stock"], best["w_xasset"])
perturbations = [
("base", 0, 0, 0),
("+10% V5", 0.10, -0.05, -0.05),
("-10% V5", -0.10, 0.05, 0.05),
("+10% Stock", -0.05, 0.10, -0.05),
("-10% Stock", 0.05, -0.10, 0.05),
]
for pname, dv, ds, dx in perturbations:
wv = max(0.05, base_w[0] + dv)
ws = max(0.05, base_w[1] + ds)
wx = max(0.05, base_w[2] + dx)
total = wv + ws + wx
wv, ws, wx = wv/total, ws/total, wx/total
r = blend_returns(rets_v5, rets_stock, rets_xasset, wv, ws, wx)
if best["vol_target"] is not None:
r = vol_target_returns(r, target_vol=best["vol_target"])
m = compute_metrics(r)
print(f" {pname:<15s}: Sharpe={m['sharpe']:5.2f} CAGR={m['cagr']*100:5.1f}%")
print("\n" + "=" * 80)
print("Done.")
if __name__ == "__main__":
main()

276
research/single_stock_swing.py
View File

@@ -1,276 +0,0 @@
"""Single-stock swing trading: adapt V3/V7 concepts to 20 famous stocks.
Strategy: long the stock when trending, cash when not.
- Trend: stock > MA + momentum > 0 + vol < cap + no dd breach
- Position sizing: vol-target overlay
- Risk mgmt: stop-loss + profit-take
- When flat: 100% cash (0% return)
Tests per-stock optimized parameters + a universal parameter set.
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
from itertools import product
import data_manager
import metrics
YEARS = 5
CAPITAL = 100_000
TX_COST = 0.002 # 2bp for individual stocks (wider spreads)
STOCKS = [
"AAPL", "MSFT", "GOOGL", "AMZN", "NVDA", "META", "TSLA", # Mag 7
"JPM", "V", "MA", # Financials
"JNJ", "UNH", "HD", # Healthcare / Consumer
"PG", "KO", "DIS", # Consumer staples / media
"NFLX", "AMD", "CRM", # Tech / growth
"COST", # Retail
]
def swing_backtest(
prices: pd.Series,
ma_window: int = 50,
mom_window: int = 21,
vol_window: int = 20,
vol_cap: float = 0.40,
dd_window: int = 20,
dd_stop: float = 0.08,
confirm_days: int = 2,
min_hold: int = 5,
stop_loss: float = 0.08,
profit_take: float = 0.0,
target_vol: float = 0.25,
min_scale: float = 0.3,
tx_cost: float = TX_COST,
) -> tuple[pd.Series, dict]:
"""Backtest a single-stock swing strategy. Returns (equity, stats)."""
arr = prices.to_numpy().astype(float)
n = len(arr)
need = max(ma_window, mom_window, vol_window, dd_window) + 1
# Precompute indicators
ma = pd.Series(arr, index=prices.index).rolling(ma_window).mean().to_numpy()
rets = np.diff(arr, prepend=arr[0]) / np.maximum(np.roll(arr, 1), 1e-12)
rets[0] = 0.0
vol = pd.Series(rets).rolling(vol_window, min_periods=10).std().to_numpy() * np.sqrt(252)
# State machine
in_position = False
entry_price = 0.0
bars_held = 0
pending_entry = 0
equity = np.ones(n) * CAPITAL
n_trades = 0
for t in range(1, n):
equity[t] = equity[t - 1]
if t < need:
continue
p = arr[t - 1] # yesterday's close (PIT-safe)
p_ma = ma[t - 1]
p_vol = vol[t - 1]
p_mom = arr[t - 1] / arr[t - 1 - mom_window] - 1 if arr[t - 1 - mom_window] > 0 else 0
p_dd = arr[t - 1] / np.max(arr[max(0, t - 1 - dd_window):t]) - 1
if np.isnan(p) or np.isnan(p_ma):
continue
# --- Trend signal ---
trend_bull = (p > p_ma and p_mom > 0 and
(np.isnan(p_vol) or p_vol < vol_cap) and
p_dd > -dd_stop)
if in_position:
bars_held += 1
# Apply daily return
daily_r = arr[t] / arr[t - 1] - 1 if arr[t - 1] > 0 else 0
# Vol-target scaling
scale = target_vol / p_vol if p_vol > 0.01 else 1.0
scale = np.clip(scale, min_scale, 1.0)
equity[t] = equity[t - 1] * (1 + daily_r * scale)
# Check exit conditions (using yesterday's close, PIT-safe)
gain = p / entry_price - 1 if entry_price > 0 else 0
exit_signal = False
# Stop-loss
if gain <= -stop_loss:
exit_signal = True
# Profit-take
if profit_take > 0 and gain >= profit_take:
exit_signal = True
# Trend reversal (with min_hold)
if not trend_bull and bars_held >= min_hold:
exit_signal = True
if exit_signal:
equity[t] -= equity[t] * tx_cost # exit cost
in_position = False
pending_entry = 0
n_trades += 1
else:
# Check entry
if trend_bull:
pending_entry += 1
if pending_entry >= confirm_days:
in_position = True
entry_price = arr[t] # enter at today's close
bars_held = 0
equity[t] -= equity[t] * tx_cost # entry cost
n_trades += 1
else:
pending_entry = 0
eq = pd.Series(equity, index=prices.index)
total_ret = eq.iloc[-1] / eq.iloc[0] - 1
days_in = sum(1 for t in range(need, n) if equity[t] != equity[t - 1])
pct_in = days_in / (n - need) if n > need else 0
return eq, {
"total_return": total_ret,
"n_trades": n_trades,
"pct_time_in": pct_in,
}
def optimize_stock(prices: pd.Series, stock: str) -> tuple[dict, dict, pd.Series]:
"""Grid search for best parameters on a single stock."""
param_grid = {
"ma_window": [20, 50, 100, 150],
"mom_window": [10, 21, 42],
"vol_cap": [0.30, 0.45, 0.60, 999],
"dd_stop": [0.05, 0.08, 0.12],
"stop_loss": [0.05, 0.08, 0.12],
"profit_take": [0.0, 0.15, 0.25],
"target_vol": [0.20, 0.30, 0.40],
"min_hold": [3, 5, 10],
"confirm_days": [1, 2, 3],
}
# Fixed params
fixed = {"vol_window": 20, "dd_window": 20, "min_scale": 0.3}
best_sharpe = -np.inf
best_params = {}
best_eq = None
keys = list(param_grid.keys())
values = list(param_grid.values())
for combo in product(*values):
params = dict(zip(keys, combo))
params.update(fixed)
try:
eq, stats = swing_backtest(prices, **params)
m = metrics.raw_summary(eq)
if m["sharpeRatio"] > best_sharpe and stats["n_trades"] >= 5:
best_sharpe = m["sharpeRatio"]
best_params = params.copy()
best_eq = eq
except Exception:
continue
return best_params, metrics.raw_summary(best_eq) if best_eq is not None else {}, best_eq
def main():
print("=" * 110)
print(" SINGLE-STOCK SWING TRADING: 20 FAMOUS STOCKS")
print("=" * 110)
# Download data
print(f"\nDownloading {len(STOCKS)} stocks...")
data = data_manager.update("swing", STOCKS, with_open=False)
if isinstance(data, tuple):
data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
data = data[data.index >= cutoff]
print(f"Period: {data.index[0].date()} → {data.index[-1].date()}")
# Buy-and-hold benchmarks
print(f"\n--- Buy & Hold Returns ({YEARS}y) ---")
bh_returns = {}
for stock in STOCKS:
if stock not in data.columns:
continue
s = data[stock].dropna()
if len(s) < 100:
continue
r = s.iloc[-1] / s.iloc[0] - 1
ann = (1 + r) ** (252 / len(s)) - 1
bh_returns[stock] = ann
print(f" {stock:<6}: {ann*100:>+6.1f}% ann")
# Universal parameter set (sensible defaults)
print(f"\n--- Universal Parameters (no per-stock optimization) ---")
print(f"{'Stock':<7} {'Ann%':>7} {'Vol%':>7} {'Sharpe':>7} {'MaxDD%':>7} {'Trades':>7} {'%In':>6} {'B&H Ann%':>9}")
print("-" * 75)
universal = dict(ma_window=50, mom_window=21, vol_window=20, vol_cap=0.45,
dd_window=20, dd_stop=0.08, confirm_days=2, min_hold=5,
stop_loss=0.08, profit_take=0.0, target_vol=0.25, min_scale=0.3)
for stock in STOCKS:
if stock not in data.columns:
continue
prices = data[stock].dropna()
if len(prices) < 200:
continue
eq, stats = swing_backtest(prices, **universal)
m = metrics.raw_summary(eq)
bh = bh_returns.get(stock, 0)
print(f" {stock:<6} {m['annualizedReturn']*100:>+6.1f}% {m['annualizedVolatility']*100:>6.1f}% "
f"{m['sharpeRatio']:>7.2f} {m['maxDrawdown']*100:>6.1f}% "
f"{stats['n_trades']:>7} {stats['pct_time_in']*100:>5.1f}% {bh*100:>+8.1f}%")
# Per-stock optimized
print(f"\n--- Per-Stock Optimized (grid search on Sharpe, min 5 trades) ---")
print(f"{'Stock':<7} {'Ann%':>7} {'Vol%':>7} {'Sharpe':>7} {'MaxDD%':>7} {'Trades':>7} {'%In':>6} {'B&H Ann%':>9} Best params")
print("-" * 130)
opt_results = []
for stock in STOCKS:
if stock not in data.columns:
continue
prices = data[stock].dropna()
if len(prices) < 200:
continue
print(f" Optimizing {stock}...", end=" ", flush=True)
params, m, eq = optimize_stock(prices, stock)
if not m:
print("FAILED")
continue
_, stats = swing_backtest(prices, **params)
bh = bh_returns.get(stock, 0)
key_params = f"MA{params.get('ma_window')}/mom{params.get('mom_window')}/SL{int(params.get('stop_loss',0)*100)}%/PT{int(params.get('profit_take',0)*100)}%/VT{int(params.get('target_vol',0)*100)}%"
print(f"{m['annualizedReturn']*100:>+5.1f}% Sharpe={m['sharpeRatio']:.2f} [{key_params}]")
opt_results.append((stock, m, stats, params, bh))
print(f"\n{'Stock':<7} {'Ann%':>7} {'Vol%':>7} {'Sharpe':>7} {'MaxDD%':>7} {'Trades':>7} {'%In':>6} {'B&H Ann%':>9} Params")
print("-" * 130)
opt_results.sort(key=lambda x: x[1]["sharpeRatio"], reverse=True)
for stock, m, stats, params, bh in opt_results:
key_params = f"MA{params.get('ma_window')}/mom{params.get('mom_window')}/SL{int(params.get('stop_loss',0)*100)}%/PT{int(params.get('profit_take',0)*100)}%/VT{int(params.get('target_vol',0)*100)}%"
beat = "✓" if m["annualizedReturn"] > bh else "✗"
print(f" {stock:<6} {m['annualizedReturn']*100:>+6.1f}% {m['annualizedVolatility']*100:>6.1f}% "
f"{m['sharpeRatio']:>7.2f} {m['maxDrawdown']*100:>6.1f}% "
f"{stats['n_trades']:>7} {stats['pct_time_in']*100:>5.1f}% {bh*100:>+8.1f}% {beat} {key_params}")
winners = sum(1 for _, m, _, _, bh in opt_results if m["annualizedReturn"] > bh)
print(f"\n Beat buy-and-hold: {winners}/{len(opt_results)} stocks")
avg_sharpe = np.mean([m["sharpeRatio"] for _, m, _, _, _ in opt_results])
print(f" Average Sharpe: {avg_sharpe:.2f}")
if __name__ == "__main__":
main()

493
research/smart_dca_eval.py
View File

@@ -1,493 +0,0 @@
"""
Smart DCA Strategy Evaluation — comprehensive comparison of DCA approaches.
Tests 6 DCA strategies across 4 ETFs (SPY, QQQ, TQQQ, UPRO) over 10 years.
Also tests a hybrid V7+DCA approach combining trend-following with smart DCA.
Usage: cd /home/gahow/projects/quant && uv run python research/smart_dca_eval.py
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from main import backtest
from strategies.trend_rider_v7 import TrendRiderV7
# ── Configuration ────────────────────────────────────────────────────────────
INITIAL_CAPITAL = 100_000
MONTHLY_BASE = 1_000
START_DATE = "2016-01-01"
END_DATE = "2026-05-23"
DCA_TICKERS = ["SPY", "QQQ", "TQQQ", "UPRO"]
# Tickers needed for V7 strategy + VIX for smart DCA
EXTRA_TICKERS = ["^VIX", "GLD", "DBC", "SHY"]
# ── Data Loading ─────────────────────────────────────────────────────────────
def load_data() -> pd.DataFrame:
"""Download/update ETF price data and return close prices."""
all_tickers = DCA_TICKERS + EXTRA_TICKERS
data = data_manager.update("etfs", all_tickers, with_open=False)
# Trim to date range
data = data.loc[START_DATE:END_DATE]
# Rename ^VIX to VIX for convenience
if "^VIX" in data.columns:
data = data.rename(columns={"^VIX": "VIX"})
return data
# ── Helper: find first trading day of each month ─────────────────────────────
def monthly_schedule(dates: pd.DatetimeIndex) -> list[pd.Timestamp]:
"""Return the first trading day of each month within the date range."""
schedule = []
seen = set()
for d in dates:
key = (d.year, d.month)
if key not in seen:
seen.add(key)
schedule.append(d)
return schedule
# ── Technical indicators ─────────────────────────────────────────────────────
def compute_rsi(prices: pd.Series, window: int = 14) -> pd.Series:
delta = prices.diff()
gain = delta.clip(lower=0)
loss = (-delta).clip(lower=0)
avg_gain = gain.ewm(alpha=1 / window, min_periods=window).mean()
avg_loss = loss.ewm(alpha=1 / window, min_periods=window).mean()
rs = avg_gain / avg_loss
return 100 - 100 / (1 + rs)
def compute_ma(prices: pd.Series, window: int = 200) -> pd.Series:
return prices.rolling(window, min_periods=window).mean()
# ── DCA Strategy implementations ─────────────────────────────────────────────
def dca_fixed(date, price, vix, rsi, ma200, portfolio_value, target_value):
"""Strategy 1: Fixed $1,000/month."""
return MONTHLY_BASE
def dca_vix_scaled(date, price, vix, rsi, ma200, portfolio_value, target_value):
"""Strategy 2: VIX-scaled DCA."""
if vix is None or np.isnan(vix):
return MONTHLY_BASE
if vix < 15:
return 500
elif vix <= 20:
return 1000
elif vix <= 30:
return 1500
else:
return 2000
def dca_ma_deviation(date, price, vix, rsi, ma200, portfolio_value, target_value):
"""Strategy 3: MA-deviation DCA. Scale by distance below 200-day MA."""
if ma200 is None or np.isnan(ma200) or ma200 == 0:
return MONTHLY_BASE
deviation = (price - ma200) / ma200 # negative when below MA
if deviation >= 0:
return 500
elif deviation >= -0.10:
return 1000
elif deviation >= -0.20:
return 2000
else:
return 3000
def dca_value_averaging(date, price, vix, rsi, ma200, portfolio_value, target_value):
"""Strategy 4: Value Averaging. Target portfolio growth of ~1% per month.
Invest the difference between target and current value, floored at $0."""
diff = target_value - portfolio_value
# Invest at least $0, cap at 3x base to avoid huge lump sums
return max(0, min(diff, MONTHLY_BASE * 3))
def dca_rsi_based(date, price, vix, rsi, ma200, portfolio_value, target_value):
"""Strategy 5: RSI-based DCA. More when oversold, less when overbought."""
if rsi is None or np.isnan(rsi):
return MONTHLY_BASE
if rsi < 30:
return 2000
elif rsi <= 70:
return 1000
else:
return 500
DCA_STRATEGIES = {
"Fixed DCA": dca_fixed,
"VIX-scaled DCA": dca_vix_scaled,
"MA-deviation DCA": dca_ma_deviation,
"Value Averaging": dca_value_averaging,
"RSI-based DCA": dca_rsi_based,
}
# ── Core DCA backtest engine ─────────────────────────────────────────────────
def run_dca_backtest(
prices: pd.Series,
strategy_fn,
vix: pd.Series | None = None,
initial_capital: float = INITIAL_CAPITAL,
) -> dict:
"""
Simulate a DCA strategy on a single ETF.
Returns dict with equity curve, total invested, final value, etc.
"""
dates = prices.index
schedule = monthly_schedule(dates)
# Precompute indicators
rsi_series = compute_rsi(prices)
ma200_series = compute_ma(prices)
# State
cash = initial_capital
shares = 0.0
total_invested = initial_capital
# For value averaging: target grows by 1% per month from initial
va_month_count = 0
equity_curve = pd.Series(index=dates, dtype=float)
schedule_set = set(schedule)
invested_tracker = pd.Series(index=dates, dtype=float)
# Buy initial position on day 1
price_0 = prices.iloc[0]
shares = cash / price_0
cash = 0.0
for i, date in enumerate(dates):
price = prices.iloc[i]
# DCA contribution on scheduled dates (skip the first date — already invested)
if date in schedule_set and date != dates[0]:
va_month_count += 1
portfolio_value = shares * price + cash
# Value averaging target: initial * (1.01)^months
target_value = initial_capital * (1.01 ** va_month_count)
# Add cumulative expected contributions
target_value += MONTHLY_BASE * va_month_count
v = vix.loc[date] if vix is not None and date in vix.index else np.nan
r = rsi_series.loc[date] if date in rsi_series.index else np.nan
m = ma200_series.loc[date] if date in ma200_series.index else np.nan
amount = strategy_fn(date, price, v, r, m, portfolio_value, target_value)
amount = max(0, amount)
# Buy shares with the DCA amount
if amount > 0 and price > 0:
new_shares = amount / price
shares += new_shares
total_invested += amount
equity_curve.iloc[i] = shares * price
invested_tracker.iloc[i] = total_invested
equity_curve = equity_curve.astype(float)
return {
"equity": equity_curve,
"total_invested": total_invested,
"final_value": equity_curve.iloc[-1],
"shares": shares,
"invested_tracker": invested_tracker,
}
# ── Lump-sum benchmark ───────────────────────────────────────────────────────
def run_lump_sum(prices: pd.Series, initial_capital: float = INITIAL_CAPITAL) -> dict:
"""Invest all capital (initial + PV of monthly contributions) at day 1."""
dates = prices.index
schedule = monthly_schedule(dates)
# Total that DCA would invest: initial + $1,000 * (num_months - 1)
n_months = len(schedule) - 1 # skip first month (already counted in initial)
total_capital = initial_capital + MONTHLY_BASE * n_months
shares = total_capital / prices.iloc[0]
equity = shares * prices
return {
"equity": equity,
"total_invested": total_capital,
"final_value": equity.iloc[-1],
"shares": shares,
}
# ── V7+VT36 baseline equity curve ────────────────────────────────────────────
def run_v7_baseline(data: pd.DataFrame) -> pd.Series:
"""Run V7+VT36 strategy and return equity curve."""
v7_tickers = ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"]
available = [t for t in v7_tickers if t in data.columns]
v7_data = data[available]
strategy = TrendRiderV7(target_vol=0.36, min_lev=0.75)
eq = backtest(strategy, v7_data, initial_capital=INITIAL_CAPITAL,
transaction_cost=0.001, fixed_fee=2.0)
return eq
# ── Hybrid V7 + DCA ──────────────────────────────────────────────────────────
def run_hybrid_v7_dca(
data: pd.DataFrame,
dca_ticker: str,
strategy_fn,
v7_pct: float = 0.70,
) -> dict:
"""
Strategy 6: Hybrid — v7_pct of capital in V7+VT36, rest in smart DCA.
The V7 portion gets v7_pct of initial capital and v7_pct of monthly contributions.
The DCA portion gets the rest.
"""
dca_pct = 1.0 - v7_pct
# V7 equity curve (normalized to its portion of capital)
v7_eq = run_v7_baseline(data)
# Scale V7 equity to its capital allocation
v7_eq_scaled = v7_eq * (v7_pct * INITIAL_CAPITAL / INITIAL_CAPITAL)
# DCA portion
prices = data[dca_ticker].dropna()
vix = data["VIX"] if "VIX" in data.columns else None
dca_result = run_dca_backtest(
prices, strategy_fn, vix=vix,
initial_capital=dca_pct * INITIAL_CAPITAL,
)
# Scale monthly contributions for DCA portion (base * dca_pct)
# Already handled since dca_backtest uses MONTHLY_BASE; we need to adjust.
# For simplicity, we just combine the two equity curves.
# Combine: align dates
common = v7_eq_scaled.index.intersection(dca_result["equity"].index)
combined = v7_eq_scaled.loc[common] + dca_result["equity"].loc[common]
# Total invested: V7 gets initial*v7_pct (lump sum, no DCA additions modeled in backtest())
# DCA gets initial*dca_pct + monthly contributions
total_invested = INITIAL_CAPITAL + dca_result["total_invested"] - dca_pct * INITIAL_CAPITAL
return {
"equity": combined,
"total_invested": total_invested,
"final_value": combined.iloc[-1],
}
# ── Reporting ─────────────────────────────────────────────────────────────────
def compute_metrics(result: dict, label: str) -> dict:
"""Compute all metrics for a DCA result."""
eq = result["equity"].dropna()
if len(eq) < 2:
return {"label": label, "error": "insufficient data"}
m = metrics.raw_summary(eq)
m["label"] = label
m["totalInvested"] = result["total_invested"]
m["finalValue"] = result["final_value"]
m["profit"] = result["final_value"] - result["total_invested"]
m["roiOnCapital"] = (result["final_value"] / result["total_invested"] - 1)
return m
def print_comparison_table(rows: list[dict], title: str):
"""Print a formatted comparison table."""
print(f"\n{'=' * 130}")
print(f" {title}")
print(f"{'=' * 130}")
header = (
f"{'Strategy':<35} {'Invested':>12} {'Final':>14} {'Profit':>14} "
f"{'ROI%':>8} {'Ann%':>8} {'Sharpe':>7} {'Sortino':>8} {'MaxDD%':>8} {'Calmar':>7}"
)
print(header)
print("-" * 130)
for r in rows:
if "error" in r:
print(f" {r['label']:<35} ERROR: {r['error']}")
continue
print(
f"{r['label']:<35} "
f"${r['totalInvested']:>11,.0f} "
f"${r['finalValue']:>13,.0f} "
f"${r['profit']:>13,.0f} "
f"{r['roiOnCapital']*100:>7.1f}% "
f"{r['annualizedReturn']*100:>7.1f}% "
f"{r['sharpeRatio']:>7.2f} "
f"{r['sortinoRatio']:>8.2f} "
f"{r['maxDrawdown']*100:>7.1f}% "
f"{r['calmarRatio']:>7.2f}"
)
# ── Main ──────────────────────────────────────────────────────────────────────
def main():
print("=" * 80)
print(" SMART DCA STRATEGY EVALUATION")
print(f" Period: {START_DATE} to {END_DATE}")
print(f" Initial capital: ${INITIAL_CAPITAL:,.0f}")
print(f" Monthly base DCA: ${MONTHLY_BASE:,.0f}")
print("=" * 80)
data = load_data()
vix = data["VIX"] if "VIX" in data.columns else None
print(f"\nData loaded: {data.shape[0]} trading days, {data.shape[1]} tickers")
print(f"Date range: {data.index[0].strftime('%Y-%m-%d')} to {data.index[-1].strftime('%Y-%m-%d')}")
# ── Part 1: DCA strategies across ETFs ────────────────────────────────
for ticker in DCA_TICKERS:
if ticker not in data.columns:
print(f"\nWARNING: {ticker} not in data, skipping.")
continue
prices = data[ticker].dropna()
if len(prices) < 252:
print(f"\nWARNING: {ticker} has <1 year of data, skipping.")
continue
results = []
# Lump-sum benchmark
ls = run_lump_sum(prices)
results.append(compute_metrics(ls, "Lump-sum (all day 1)"))
# Each DCA strategy
for name, fn in DCA_STRATEGIES.items():
r = run_dca_backtest(prices, fn, vix=vix)
results.append(compute_metrics(r, name))
print_comparison_table(results, f"DCA Strategies — {ticker}")
# Print DCA investment summary
print(f"\n Note: Fixed DCA total invested = ${results[1]['totalInvested']:,.0f} "
f"over {len(monthly_schedule(prices.index))-1} months + "
f"${INITIAL_CAPITAL:,.0f} initial")
# ── Part 2: V7+VT36 baseline ─────────────────────────────────────────
print(f"\n{'=' * 130}")
print(" V7+VT36 TREND-FOLLOWING BASELINE (lump-sum $100K, no DCA)")
print(f"{'=' * 130}")
v7_eq = run_v7_baseline(data)
v7_m = metrics.raw_summary(v7_eq)
print(
f" Ann: {v7_m['annualizedReturn']*100:.1f}%, "
f"Vol: {v7_m['annualizedVolatility']*100:.1f}%, "
f"Sharpe: {v7_m['sharpeRatio']:.2f}, "
f"Sortino: {v7_m['sortinoRatio']:.2f}, "
f"MaxDD: {v7_m['maxDrawdown']*100:.1f}%, "
f"Calmar: {v7_m['calmarRatio']:.2f}, "
f"Final: ${v7_eq.iloc[-1]:,.0f}"
)
# ── Part 3: Hybrid V7 + Smart DCA ────────────────────────────────────
hybrid_results = []
# 100% V7 baseline for comparison
hybrid_results.append({
"label": "100% V7+VT36 (no DCA)",
"totalInvested": INITIAL_CAPITAL,
"finalValue": v7_eq.iloc[-1],
"profit": v7_eq.iloc[-1] - INITIAL_CAPITAL,
"roiOnCapital": v7_eq.iloc[-1] / INITIAL_CAPITAL - 1,
**v7_m,
})
# Hybrid: 70% V7 + 30% VIX-scaled DCA into each leveraged ETF
for dca_ticker in ["TQQQ", "UPRO"]:
if dca_ticker not in data.columns:
continue
for strat_name, strat_fn in [("VIX-scaled", dca_vix_scaled),
("MA-deviation", dca_ma_deviation),
("RSI-based", dca_rsi_based)]:
r = run_hybrid_v7_dca(data, dca_ticker, strat_fn, v7_pct=0.70)
label = f"70%V7 + 30%{strat_name}->{dca_ticker}"
hybrid_results.append(compute_metrics(r, label))
print_comparison_table(hybrid_results, "Hybrid V7+VT36 + Smart DCA Combinations")
# ── Part 4: Best of each category summary ─────────────────────────────
print(f"\n{'=' * 130}")
print(" SUMMARY: Best strategy per ETF (by final portfolio value)")
print(f"{'=' * 130}")
for ticker in DCA_TICKERS:
if ticker not in data.columns:
continue
prices = data[ticker].dropna()
if len(prices) < 252:
continue
best_name = None
best_final = 0
for name, fn in DCA_STRATEGIES.items():
r = run_dca_backtest(prices, fn, vix=vix)
if r["final_value"] > best_final:
best_final = r["final_value"]
best_name = name
best_invested = r["total_invested"]
ls = run_lump_sum(prices)
ls_label = "Lump-sum"
if ls["final_value"] > best_final:
best_final = ls["final_value"]
best_name = ls_label
best_invested = ls["total_invested"]
roi = (best_final / best_invested - 1) * 100
print(f" {ticker:<6} => {best_name:<25} Final: ${best_final:>14,.0f} "
f"Invested: ${best_invested:>10,.0f} ROI: {roi:.1f}%")
# ── Part 5: Year-by-year breakdown for top strategies ─────────────────
print(f"\n{'=' * 130}")
print(" YEAR-BY-YEAR: VIX-scaled DCA into TQQQ vs SPY vs Lump-sum SPY")
print(f"{'=' * 130}")
for ticker in ["SPY", "TQQQ"]:
if ticker not in data.columns:
continue
prices = data[ticker].dropna()
vix_dca = run_dca_backtest(prices, dca_vix_scaled, vix=vix)
eq = vix_dca["equity"].dropna()
print(f"\n {ticker} — VIX-scaled DCA:")
print(f" {'Year':<8} {'Year-end Value':>16} {'YTD Return':>12}")
print(f" {'-'*40}")
years = sorted(set(eq.index.year))
for y in years:
year_data = eq[eq.index.year == y]
if len(year_data) < 2:
continue
ytd = year_data.iloc[-1] / year_data.iloc[0] - 1
print(f" {y:<8} ${year_data.iloc[-1]:>15,.0f} {ytd:>11.1%}")
print(f"\n{'=' * 80}")
print(" EVALUATION COMPLETE")
print(f"{'=' * 80}")
if __name__ == "__main__":
main()

166
research/sota_ranking.py
View File

@@ -1,166 +0,0 @@
"""Rank all top strategies head-to-head on the same 10-year PIT-safe data."""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
import universe_history as uh
from main import backtest
from trader import STRATEGY_REGISTRY, ETF_STRATEGY_UNIVERSES, MIXED_STRATEGY_EXTRA_TICKERS, filter_tradable_tickers
from universe import UNIVERSES
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
# Only the most promising strategies — skip redundant freq variants
CANDIDATES = [
# ETF tactical allocation
"trend_rider_v7",
"trend_rider_v7_vt24",
"trend_rider_v7_vt32",
"trend_rider_v3_vt28",
"trend_rider_v3_vt32",
"trend_rider_v5_us",
"trend_rider_v5_panic",
"trend_rider_v3_us",
# V6 hybrids (stock + regime)
"trend_rider_v6",
"trend_rider_v6_top10",
# Stock pickers
"recovery_mom_top10",
"recovery_mom_top20",
"trend_following",
"fc_rec_mfilt_deep_upvol_monthly",
"fc_rec_mfilt_deep_upvol_daily",
# Ensembles
"ensemble_alpha_top10",
"sharpe_boosted_ensemble_top8",
"risk_managed_ensemble_top10",
"enhanced_factor_combo_top10",
]
def main():
print("=" * 95)
print(" COMPREHENSIVE STRATEGY RANKING (10y PIT-safe)")
print("=" * 95)
# Load S&P 500 + PIT
print("\n[1] Loading data...")
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
pit_intervals = uh.load_sp500_history()
hist_tickers = uh.all_tickers_ever(pit_intervals)
# Collect all ETF tickers needed
all_etf = set()
for name in CANDIDATES:
base = name.removeprefix("sim_")
if base in ETF_STRATEGY_UNIVERSES:
all_etf.update(ETF_STRATEGY_UNIVERSES[base])
if base in MIXED_STRATEGY_EXTRA_TICKERS:
all_etf.update(MIXED_STRATEGY_EXTRA_TICKERS[base])
all_etf.update(["SPY", "GLD", "DBC", "SHY", "TQQQ", "UPRO", "TLT", "IEF"])
all_tickers = sorted(set(tickers + hist_tickers + list(all_etf)))
print(f" {len(all_tickers)} tickers to download...")
stock_data = data_manager.update("us", all_tickers, with_open=False)
if isinstance(stock_data, tuple):
stock_data = stock_data[0]
cutoff = stock_data.index[-1] - pd.DateOffset(years=YEARS)
stock_data = stock_data[stock_data.index >= cutoff]
stock_data = uh.mask_prices(stock_data, pit_intervals)
stock_tickers = [t for t in stock_data.columns
if t not in all_etf and stock_data[t].notna().any()]
# Also load pure ETF panel (for pure-ETF strategies that use separate data)
etf_list = sorted(all_etf)
etf_data = data_manager.update("etfs", etf_list, with_open=False)
if isinstance(etf_data, tuple):
etf_data = etf_data[0]
etf_cutoff = etf_data.index[-1] - pd.DateOffset(years=YEARS)
etf_data = etf_data[etf_data.index >= etf_cutoff]
print(f" Stocks: {len(stock_tickers)}, ETFs: {len(etf_list)}")
print(f" Period: {stock_data.index[0].date()} → {stock_data.index[-1].date()}")
# Run strategies
print("\n[2] Running strategies...")
results: list[tuple[str, dict]] = []
for name in CANDIDATES:
if name not in STRATEGY_REGISTRY:
print(f" SKIP {name} (not in registry)")
continue
base = name.removeprefix("sim_")
print(f" {name}...", end=" ", flush=True)
try:
if base in ETF_STRATEGY_UNIVERSES:
# Pure ETF strategy
etf_tickers = ETF_STRATEGY_UNIVERSES[base]
tradable = [t for t in etf_tickers if t in etf_data.columns]
strategy = STRATEGY_REGISTRY[name]()
eq = backtest(strategy, etf_data[tradable],
initial_capital=CAPITAL, transaction_cost=TX_COST,
fixed_fee=FIXED_FEE)
elif base in MIXED_STRATEGY_EXTRA_TICKERS:
# Mixed: stocks + ETFs in one panel
extra = MIXED_STRATEGY_EXTRA_TICKERS[base]
panel_cols = stock_tickers + [t for t in extra if t in stock_data.columns]
panel = stock_data[[c for c in panel_cols if c in stock_data.columns]]
strategy = STRATEGY_REGISTRY[name]()
eq = backtest(strategy, panel,
initial_capital=CAPITAL, transaction_cost=TX_COST,
fixed_fee=FIXED_FEE)
else:
# Pure stock strategy
strategy = STRATEGY_REGISTRY[name](top_n=10)
eq = backtest(strategy, stock_data[stock_tickers],
initial_capital=CAPITAL, transaction_cost=TX_COST,
fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
results.append((name, m))
print(f"Ann={m['annualizedReturn']*100:.1f}%")
except Exception as e:
print(f"FAILED: {e}")
# SPY benchmark
spy = stock_data["SPY"].dropna()
spy_eq = (spy / spy.iloc[0]) * CAPITAL
spy_m = metrics.raw_summary(spy_eq)
results.append(("SPY (benchmark)", spy_m))
# Sort by annualized return
results.sort(key=lambda x: x[1]["annualizedReturn"], reverse=True)
print(f"\n[3] Ranking ({YEARS}y, ${CAPITAL:,.0f}, tx={TX_COST*100:.1f}bps + ${FIXED_FEE:.0f}/trade)")
print("=" * 110)
print(f"{'#':<4} {'Strategy':<40} {'Ann%':>8} {'Vol%':>8} {'Sharpe':>8} {'Sortino':>8} {'MaxDD%':>8} {'Calmar':>8}")
print("-" * 110)
for i, (name, m) in enumerate(results, 1):
print(f"{i:<4} {name:<40} "
f"{m['annualizedReturn']*100:>7.1f}% "
f"{m['annualizedVolatility']*100:>7.1f}% "
f"{m['sharpeRatio']:>8.2f} "
f"{m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>7.1f}% "
f"{m['calmarRatio']:>8.2f}")
print("=" * 110)
if __name__ == "__main__":
main()

150
research/strategies_plus.py
View File

@@ -1,150 +0,0 @@
"""
Optimization variants of RecoveryMomentumStrategy.
Four dimensions explored:
1. Hyperparameters (top_n, recovery_window, mom_lookback, rebal_freq, weights)
2. Regime filter: zero-out weights when SPY < MA200
3. Weighting scheme: equal / inverse-vol / rank-weighted
4. Ensemble: weighted blend of multiple strategies
All strategies follow the same Strategy protocol (generate_signals → weights DF).
"""
import numpy as np
import pandas as pd
from strategies.base import Strategy
# ---------------------------------------------------------------------------
# Generalized Recovery+Momentum strategy
# ---------------------------------------------------------------------------
class RecoveryMomentumPlus(Strategy):
"""
Recovery + momentum composite with configurable blend, weighting, and
regime filter hooks.
Parameters
----------
recovery_window : int
Lookback for the recovery factor (price / rolling min - 1).
mom_lookback : int
Long-horizon momentum window total length.
mom_skip : int
Short-term reversal skip for momentum.
rebal_freq : int
Trading-day rebalance interval.
top_n : int
Number of stocks selected each rebalance.
rec_weight : float in [0, 1]
Weight of recovery factor in composite rank blend (mom_weight = 1 - rec_weight).
weighting : {"equal", "inv_vol", "rank"}
Portfolio weighting scheme for the selected top_n.
vol_window : int
Volatility lookback when weighting="inv_vol".
"""
def __init__(self,
recovery_window: int = 63,
mom_lookback: int = 252,
mom_skip: int = 21,
rebal_freq: int = 21,
top_n: int = 10,
rec_weight: float = 0.5,
weighting: str = "equal",
vol_window: int = 60):
if weighting not in ("equal", "inv_vol", "rank"):
raise ValueError(f"weighting must be equal|inv_vol|rank, got {weighting!r}")
self.recovery_window = recovery_window
self.mom_lookback = mom_lookback
self.mom_skip = mom_skip
self.rebal_freq = rebal_freq
self.top_n = top_n
self.rec_weight = rec_weight
self.weighting = weighting
self.vol_window = vol_window
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
# Factors
recovery = data / data.rolling(self.recovery_window).min() - 1
momentum = data.shift(self.mom_skip).pct_change(self.mom_lookback - self.mom_skip)
rec_rank = recovery.rank(axis=1, pct=True, na_option="keep")
mom_rank = momentum.rank(axis=1, pct=True, na_option="keep")
composite = self.rec_weight * rec_rank + (1 - self.rec_weight) * mom_rank
# Top-N selection
rank = composite.rank(axis=1, ascending=False, na_option="bottom")
n_valid = composite.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
# Weighting within top-N
if self.weighting == "equal":
raw = top_mask.astype(float)
elif self.weighting == "rank":
# Higher composite → higher weight within top-N
ranked_score = composite.where(top_mask, 0.0)
raw = ranked_score
elif self.weighting == "inv_vol":
# Use inverse realized-volatility as weights within top-N
rets = data.pct_change()
vol = rets.rolling(self.vol_window).std()
inv_vol = 1.0 / vol.replace(0, np.nan)
raw = inv_vol.where(top_mask, 0.0).fillna(0.0)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# Rebalance
warmup = max(self.mom_lookback, self.recovery_window, self.vol_window)
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
return signals.shift(1).fillna(0.0)
# ---------------------------------------------------------------------------
# Ensemble
# ---------------------------------------------------------------------------
class EnsembleStrategy(Strategy):
"""
Weighted blend of several sub-strategies. Each sub-strategy produces a
weight matrix; we linearly combine them. The result still sums to (at
most) 1 per row since each sub-strategy does.
"""
def __init__(self, components: list[tuple[Strategy, float]]):
total = sum(w for _, w in components)
self.components = [(s, w / total) for s, w in components]
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
out = None
for strat, w in self.components:
sig = strat.generate_signals(data).mul(w)
if out is None:
out = sig
else:
# Align columns (should be identical since same data passed)
out = out.add(sig, fill_value=0.0)
return out
# ---------------------------------------------------------------------------
# Regime filter helper
# ---------------------------------------------------------------------------
def spy_ma200_filter(spy: pd.Series, ma_window: int = 200) -> pd.Series:
"""
Boolean Series: True when SPY close > SPY MA(ma_window), shifted by 1 to
avoid lookahead. Use as `regime_filter=...` in pit_backtest.backtest().
"""
ma = spy.rolling(ma_window, min_periods=ma_window).mean()
signal = (spy > ma).fillna(False)
return signal.shift(1).fillna(False)

250
research/strategy_final_report.py
View File

@@ -1,250 +0,0 @@
"""
FINAL REPORT: Strategy improvement results — 10-year yearly backtest.
Produces the definitive comparison of:
- Original best strategies
- Improved strategies (winners from 4 rounds of iteration)
- SPY benchmark
With full PIT compliance audit and production readiness notes.
"""
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from main import backtest
from strategies.factor_combo import FactorComboStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.momentum_quality import MomentumQualityStrategy
from strategies.adaptive_momentum import AdaptiveMomentumStrategy
from strategies.improved_momentum_quality import ImprovedMomentumQualityStrategy
from strategies.ensemble_alpha import EnsembleAlphaStrategy, EnhancedFactorComboStrategy
def annual_return(eq): return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq): return ((eq / eq.cummax()) - 1).min()
def sharpe(eq):
d = eq.pct_change().dropna()
return (d.mean() * 252) / (d.std() * np.sqrt(252)) if d.std() > 0 else 0
def sortino(eq):
d = eq.pct_change().dropna()
ds = d[d < 0].std() * np.sqrt(252)
return (d.mean() * 252) / ds if ds > 0 else 0
def cagr(eq):
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1 if yrs > 0 else 0
def calmar(eq):
dd = max_dd(eq)
return cagr(eq) / abs(dd) if dd < 0 else 0
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
print(f"Universe: {len(tickers)} S&P 500 stocks")
print(f"Data range: {data.index[0].date()} to {data.index[-1].date()}")
print(f"Transaction cost: 10 bps per unit turnover")
print()
# Final strategy selection
strategies = {
# --- ORIGINAL BEST ---
"FactorCombo (orig top20)": (
FactorComboStrategy(signal_name="rec_mfilt+deep_upvol", rebal_freq=21, top_n=20),
data[tickers]
),
"Recovery+Mom (orig top20)": (
RecoveryMomentumStrategy(top_n=20),
data[tickers]
),
"Mom+Quality (orig top49)": (
MomentumQualityStrategy(momentum_period=252, skip=21, top_n=49),
data[tickers]
),
"Mom+InvVol (orig top49)": (
AdaptiveMomentumStrategy(top_n=49),
data[tickers]
),
# --- IMPROVED (from iteration) ---
"Improved MomQuality top20": (
ImprovedMomentumQualityStrategy(top_n=20),
data[tickers]
),
"Ensemble Top10 [BEST CAGR]": (
EnsembleAlphaStrategy(top_n=10, tail_protection=False),
data[tickers]
),
"Ensemble Top12 [BEST SHARPE]": (
EnsembleAlphaStrategy(top_n=12, tail_protection=False),
data[tickers]
),
"EnhFC Top10 mom20%": (
EnhancedFactorComboStrategy(top_n=10, mom_boost=0.2, tail_protection=False),
data[tickers]
),
"EnhFC Top12 mom20%": (
EnhancedFactorComboStrategy(top_n=12, mom_boost=0.2, tail_protection=False),
data[tickers]
),
"Ensemble Top15 +TailProt": (
EnsembleAlphaStrategy(top_n=15, tail_protection=True, tail_threshold=-0.12, tail_scale=0.4),
data[tickers]
),
}
# Run backtests
equity = {}
for name, (strat, strat_data) in strategies.items():
print(f" Running: {name}")
equity[name] = backtest(strat, strat_data, initial_capital=10_000)
bench = data[benchmark].dropna()
equity["SPY (Benchmark)"] = (bench / bench.iloc[0]) * 10_000
eq_df = pd.DataFrame(equity).sort_index()
# ===== YEARLY RETURNS TABLE =====
years = sorted(eq_df.index.year.unique())
rows = []
for yr in years:
window = eq_df.loc[eq_df.index.year == yr].dropna(how="all")
if window.empty:
continue
row = {"Year": yr}
for col in eq_df.columns:
s = window[col].dropna()
row[col] = annual_return(s) if len(s) >= 2 else np.nan
rows.append(row)
yr_df = pd.DataFrame(rows).set_index("Year")
# Choose display columns: improved strategies + SPY
display_cols = [
"SPY (Benchmark)",
"FactorCombo (orig top20)",
"Recovery+Mom (orig top20)",
"Improved MomQuality top20",
"EnhFC Top10 mom20%",
"Ensemble Top10 [BEST CAGR]",
"Ensemble Top12 [BEST SHARPE]",
"Ensemble Top15 +TailProt",
]
display_cols = [c for c in display_cols if c in yr_df.columns]
print("\n")
print("=" * 120)
print(" FINAL RESULTS: 10-YEAR YEARLY BACKTEST (% return)")
print("=" * 120)
# Shortened column names for display
short_names = {
"SPY (Benchmark)": "SPY",
"FactorCombo (orig top20)": "FC orig",
"Recovery+Mom (orig top20)": "RecMom orig",
"Improved MomQuality top20": "ImpMQ",
"EnhFC Top10 mom20%": "EnhFC10",
"Ensemble Top10 [BEST CAGR]": "Ens10*",
"Ensemble Top12 [BEST SHARPE]": "Ens12*",
"Ensemble Top15 +TailProt": "Ens15T",
}
display_df = (yr_df[display_cols] * 100).round(1)
display_df.columns = [short_names.get(c, c) for c in display_df.columns]
print(display_df.to_string())
# Excess vs SPY
excess = yr_df[display_cols].sub(yr_df["SPY (Benchmark)"], axis=0)
excess = excess.drop(columns=["SPY (Benchmark)"])
excess_display = (excess * 100).round(1)
excess_display.columns = [short_names.get(c, c) for c in excess_display.columns]
print("\n")
print("=" * 120)
print(" EXCESS RETURN vs SPY (percentage points)")
print("=" * 120)
print(excess_display.to_string())
# Average annual excess
print("\n Average annual excess vs SPY:")
for col in excess.columns:
avg = excess[col].mean() * 100
print(f" {short_names.get(col, col):<15s}: {avg:+.1f} pp/year")
# ===== FULL-PERIOD SUMMARY =====
print("\n")
print("=" * 120)
print(" FULL-PERIOD PERFORMANCE METRICS")
print("=" * 120)
print(f" {'Strategy':<30s} {'CAGR':>7s} {'Sharpe':>7s} {'Sortino':>8s} {'MaxDD':>8s} {'Calmar':>7s} {'Win/Total':>10s} {'$10K→':>10s}")
print(" " + "-" * 93)
for col in display_cols:
eq = eq_df[col].dropna()
if len(eq) < 252:
continue
wins = (excess[col] > 0).sum() if col in excess.columns else "-"
total = len([r for r in rows if not np.isnan(yr_df.loc[r["Year"], col])]) if col in yr_df.columns else 0
final_val = eq.iloc[-1]
label = short_names.get(col, col)
win_str = f"{wins}/{total}" if col in excess.columns else "-"
print(f" {label:<30s} {cagr(eq)*100:>6.1f}% {sharpe(eq):>7.2f} {sortino(eq):>8.2f} {max_dd(eq)*100:>7.1f}% {calmar(eq):>7.2f} {win_str:>10s} ${final_val:>9,.0f}")
# ===== PRODUCTION READINESS AUDIT =====
print("\n")
print("=" * 120)
print(" STRATEGY AUDIT: PIT COMPLIANCE & PRODUCTION READINESS")
print("=" * 120)
print("""
[✓] Point-in-Time (PIT) Compliance:
- All strategies apply .shift(1) to final signals → trade on T+1 close
- Momentum signals use .shift(21) → skip most recent month
- Recovery signals use trailing rolling windows only (no future data)
- Tail protection uses cumulative market returns up to current day
- No survivorship bias: uses current S&P 500 membership (not delisted)
[✓] Transaction Cost Model:
- 10 bps one-way cost per unit turnover applied to all strategies
- Monthly rebalancing (21 trading days) keeps turnover manageable
- Avg daily turnover: ~0.04 (monthly effective: ~0.8 → ~8 bps/month)
[✓] Strategy Logic Review:
- Ensemble Top10/12: Averages two proven alpha signals (recovery×momentum_filtered
+ deep_recovery×up_volume) with (recovery_63d + 12-1_momentum). Top N by composite
rank, equal-weighted, monthly rebalance.
- EnhFC Top10/12: FactorCombo's best signal (rec_mfilt+deep_upvol) boosted with
20% weight on 12-1 month momentum rank as tiebreaker. Concentrated portfolio.
- Both use only price data (no fundamental/accounting data needed)
- All signals are cross-sectional (relative ranking) → robust to market level
[!] Risk Considerations:
- Top10 concentration: single stock = 10% weight → vulnerable to gap risk
- MaxDD -36% to -40% during market crashes (2020, 2022)
- Ensemble Top15 +TailProt reduces MaxDD to -33% with lower CAGR trade-off
- All strategies underperform in strong bull markets where low-quality stocks lead (2021)
[!] Limitations / Out-of-sample concerns:
- Universe is CURRENT S&P 500 (survivorship bias present for pre-2016 analysis)
- 2016-2026 is mostly bullish → recovery signals naturally favor momentum
- Should validate with PIT universe (us_pit.csv) for true out-of-sample
""")
# Save final results
yr_df.to_csv("data/final_improvement_yearly.csv")
print(" Saved: data/final_improvement_yearly.csv")
# Also save equity curves
eq_df.to_csv("data/final_improvement_equity.csv")
print(" Saved: data/final_improvement_equity.csv")
if __name__ == "__main__":
main()

288
research/strategy_improvement_eval.py
View File

@@ -1,288 +0,0 @@
"""
Comprehensive strategy improvement evaluation.
Compares original strategies against improved versions, showing:
- Yearly returns (2016-2025)
- Key metrics (CAGR, Sharpe, MaxDD, Calmar)
- Excess over SPY
- Turnover analysis
"""
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from main import backtest
# Original strategies
from strategies.momentum import MomentumStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.momentum_quality import MomentumQualityStrategy
from strategies.adaptive_momentum import AdaptiveMomentumStrategy
from strategies.dual_momentum import DualMomentumStrategy
from strategies.trend_following import TrendFollowingStrategy
from strategies.multi_factor import MultiFactorStrategy
from strategies.factor_combo import FactorComboStrategy
# Improved strategies
from strategies.enhanced_recovery_momentum import EnhancedRecoveryMomentumStrategy
from strategies.improved_momentum_quality import ImprovedMomentumQualityStrategy
from strategies.composite_alpha import CompositeAlphaStrategy
def annual_return(eq: pd.Series) -> float:
return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq: pd.Series) -> float:
return ((eq / eq.cummax()) - 1).min()
def sharpe(eq: pd.Series) -> float:
daily = eq.pct_change().dropna()
if daily.std() == 0:
return 0.0
return (daily.mean() * 252) / (daily.std() * np.sqrt(252))
def sortino(eq: pd.Series) -> float:
daily = eq.pct_change().dropna()
downside = daily[daily < 0].std() * np.sqrt(252)
if downside == 0:
return 0.0
return (daily.mean() * 252) / downside
def cagr(eq: pd.Series) -> float:
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
if yrs <= 0:
return 0.0
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1
def turnover(weights: pd.DataFrame) -> float:
"""Average daily turnover."""
return weights.diff().abs().sum(axis=1).mean()
def main():
# --- Load data ---
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
top_n = max(5, len(tickers) // 10)
print(f"Universe: {len(tickers)} stocks + {benchmark}. top_n={top_n}")
print(f"Data range: {data.index[0].date()} to {data.index[-1].date()}")
# --- Build strategies ---
strategies = {
# === ORIGINALS ===
"Momentum (orig)": (
MomentumStrategy(lookback=252, skip=21, top_n=top_n),
data[tickers]
),
"Recovery+Mom Top20 (orig)": (
RecoveryMomentumStrategy(top_n=20),
data[tickers]
),
"Mom+Quality (orig)": (
MomentumQualityStrategy(momentum_period=252, skip=21, top_n=top_n),
data[tickers]
),
"Mom+InvVol (orig)": (
AdaptiveMomentumStrategy(top_n=top_n),
data[tickers]
),
"Dual Momentum (orig)": (
DualMomentumStrategy(top_n=top_n),
data[tickers]
),
"Trend Following (orig)": (
TrendFollowingStrategy(ma_window=150, momentum_period=126, top_n=top_n),
data[tickers]
),
"Multi-Factor (orig)": (
MultiFactorStrategy(tickers=tickers, benchmark=benchmark, top_n=top_n),
data
),
"FactorCombo rec+deep (orig)": (
FactorComboStrategy(signal_name="rec_mfilt+deep_upvol", rebal_freq=21, top_n=20),
data[tickers]
),
# === IMPROVED ===
"Enhanced RecMom Top20": (
EnhancedRecoveryMomentumStrategy(
recovery_window=63, mom_lookback=252, mom_skip=21,
intermediate_mom=126, vol_window=60,
rebal_freq=21, top_n=20, regime_scale=True
),
data[tickers]
),
"Enhanced RecMom Top30": (
EnhancedRecoveryMomentumStrategy(
recovery_window=63, mom_lookback=252, mom_skip=21,
intermediate_mom=126, vol_window=60,
rebal_freq=21, top_n=30, regime_scale=True
),
data[tickers]
),
"Improved MomQuality": (
ImprovedMomentumQualityStrategy(
momentum_period=252, skip=21, quality_window=252,
recovery_window=63, vol_window=60, rebal_freq=21, top_n=20
),
data[tickers]
),
"Improved MomQuality Top30": (
ImprovedMomentumQualityStrategy(
momentum_period=252, skip=21, quality_window=252,
recovery_window=63, vol_window=60, rebal_freq=21, top_n=30
),
data[tickers]
),
"Composite Alpha": (
CompositeAlphaStrategy(
tickers=tickers, benchmark=benchmark,
recovery_window=63, intermediate_period=147, skip=21,
quality_window=252, vol_window=60,
rebal_freq=10, top_n=20, regime_gate=True
),
data
),
"Composite Alpha Top30": (
CompositeAlphaStrategy(
tickers=tickers, benchmark=benchmark,
recovery_window=63, intermediate_period=147, skip=21,
quality_window=252, vol_window=60,
rebal_freq=10, top_n=30, regime_gate=True
),
data
),
"Composite Alpha NoRegime": (
CompositeAlphaStrategy(
tickers=tickers, benchmark=benchmark,
recovery_window=63, intermediate_period=147, skip=21,
quality_window=252, vol_window=60,
rebal_freq=10, top_n=20, regime_gate=False
),
data
),
}
# --- Run backtests ---
equity = {}
for name, (strat, strat_data) in strategies.items():
print(f"Running {name}...")
equity[name] = backtest(strat, strat_data, initial_capital=10_000)
# SPY benchmark
bench = data[benchmark].dropna()
equity["SPY"] = (bench / bench.iloc[0]) * 10_000
eq_df = pd.DataFrame(equity).sort_index()
# --- Yearly returns table ---
years = list(range(2016, 2027))
rows = []
for yr in years:
start = pd.Timestamp(f"{yr}-01-01")
end = pd.Timestamp(f"{yr}-12-31")
window = eq_df.loc[(eq_df.index >= start) & (eq_df.index <= end)].dropna(how="all")
if window.empty:
continue
row = {"Year": yr}
for col in eq_df.columns:
s = window[col].dropna()
if len(s) < 2:
row[col] = np.nan
else:
row[col] = annual_return(s)
rows.append(row)
yr_df = pd.DataFrame(rows).set_index("Year")
# --- Print results ---
print("\n" + "=" * 80)
print("YEARLY TOTAL RETURN (%)")
print("=" * 80)
print((yr_df * 100).round(2).to_string())
# Excess over SPY
excess = yr_df.sub(yr_df["SPY"], axis=0).drop(columns=["SPY"])
print("\n" + "=" * 80)
print("EXCESS vs SPY (percentage points)")
print("=" * 80)
print((excess * 100).round(2).to_string())
# --- Full-period summary ---
print("\n" + "=" * 80)
print("FULL-PERIOD METRICS")
print("=" * 80)
summary_rows = []
for col in eq_df.columns:
eq = eq_df[col].dropna()
if len(eq) < 252:
continue
summary_rows.append({
"Strategy": col,
"CAGR %": cagr(eq) * 100,
"Sharpe": sharpe(eq),
"Sortino": sortino(eq),
"Max DD %": max_dd(eq) * 100,
"Calmar": cagr(eq) / abs(max_dd(eq)) if max_dd(eq) < 0 else 0,
"Avg Ann Ret %": yr_df[col].mean() * 100 if col in yr_df.columns else np.nan,
"Win Rate vs SPY": (excess[col] > 0).mean() * 100 if col in excess.columns else np.nan,
})
summary = pd.DataFrame(summary_rows).sort_values("CAGR %", ascending=False)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', 200)
print(summary.round(2).to_string(index=False))
# --- Comparison: Improved vs Original ---
print("\n" + "=" * 80)
print("IMPROVEMENT ANALYSIS (best improved vs best original)")
print("=" * 80)
orig_cols = [c for c in eq_df.columns if "(orig)" in c]
improved_cols = [c for c in eq_df.columns if c not in orig_cols and c != "SPY"]
if orig_cols and improved_cols:
best_orig = max(orig_cols, key=lambda c: cagr(eq_df[c].dropna()))
best_improved = max(improved_cols, key=lambda c: cagr(eq_df[c].dropna()))
orig_eq = eq_df[best_orig].dropna()
imp_eq = eq_df[best_improved].dropna()
print(f"\nBest original: {best_orig}")
print(f" CAGR={cagr(orig_eq)*100:.2f}% Sharpe={sharpe(orig_eq):.2f} "
f"MaxDD={max_dd(orig_eq)*100:.2f}% Calmar={cagr(orig_eq)/abs(max_dd(orig_eq)):.2f}")
print(f"\nBest improved: {best_improved}")
print(f" CAGR={cagr(imp_eq)*100:.2f}% Sharpe={sharpe(imp_eq):.2f} "
f"MaxDD={max_dd(imp_eq)*100:.2f}% Calmar={cagr(imp_eq)/abs(max_dd(imp_eq)):.2f}")
cagr_diff = (cagr(imp_eq) - cagr(orig_eq)) * 100
sharpe_diff = sharpe(imp_eq) - sharpe(orig_eq)
dd_diff = (max_dd(imp_eq) - max_dd(orig_eq)) * 100
print(f"\nDelta: CAGR {cagr_diff:+.2f}pp Sharpe {sharpe_diff:+.2f} MaxDD {dd_diff:+.2f}pp")
# --- Save results ---
out_path = "data/strategy_improvement_results.csv"
yr_df.to_csv(out_path)
print(f"\nSaved yearly returns to {out_path}")
summary_path = "data/strategy_improvement_summary.csv"
summary.to_csv(summary_path, index=False)
print(f"Saved summary to {summary_path}")
if __name__ == "__main__":
main()

201
research/strategy_improvement_r2.py
View File

@@ -1,201 +0,0 @@
"""
Round 2: Strategy improvement iteration.
Tests Hybrid Alpha variants that combine FactorCombo signal with inv-vol weighting,
and RecoveryQualityBlend that uses all strong factors without restrictive gates.
"""
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from main import backtest
# Top performers from round 1
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.factor_combo import FactorComboStrategy
from strategies.improved_momentum_quality import ImprovedMomentumQualityStrategy
# Round 2 strategies
from strategies.hybrid_alpha import HybridAlphaStrategy, RecoveryQualityBlendStrategy
def annual_return(eq: pd.Series) -> float:
return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq: pd.Series) -> float:
return ((eq / eq.cummax()) - 1).min()
def sharpe(eq: pd.Series) -> float:
daily = eq.pct_change().dropna()
if daily.std() == 0:
return 0.0
return (daily.mean() * 252) / (daily.std() * np.sqrt(252))
def sortino(eq: pd.Series) -> float:
daily = eq.pct_change().dropna()
downside = daily[daily < 0].std() * np.sqrt(252)
if downside == 0:
return 0.0
return (daily.mean() * 252) / downside
def cagr(eq: pd.Series) -> float:
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
if yrs <= 0:
return 0.0
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1
def calmar(eq: pd.Series) -> float:
dd = max_dd(eq)
if dd >= 0:
return 0.0
return cagr(eq) / abs(dd)
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
top_n = max(5, len(tickers) // 10)
print(f"Universe: {len(tickers)} stocks + {benchmark}. top_n={top_n}")
print(f"Data range: {data.index[0].date()} to {data.index[-1].date()}")
strategies = {
# === BASELINES (top 3 from round 1) ===
"Recovery+Mom Top20 (base)": (
RecoveryMomentumStrategy(top_n=20),
data[tickers]
),
"FactorCombo rec+deep (base)": (
FactorComboStrategy(signal_name="rec_mfilt+deep_upvol", rebal_freq=21, top_n=20),
data[tickers]
),
"Improved MomQuality (base)": (
ImprovedMomentumQualityStrategy(top_n=20),
data[tickers]
),
# === ROUND 2: HYBRID ALPHA ===
"Hybrid InvVol Top20": (
HybridAlphaStrategy(rebal_freq=21, top_n=20, use_invvol=True, regime_dampen=1.0),
data[tickers]
),
"Hybrid InvVol Top30": (
HybridAlphaStrategy(rebal_freq=21, top_n=30, use_invvol=True, regime_dampen=1.0),
data[tickers]
),
"Hybrid EW Top20": (
HybridAlphaStrategy(rebal_freq=21, top_n=20, use_invvol=False, regime_dampen=1.0),
data[tickers]
),
"Hybrid InvVol Dampen": (
HybridAlphaStrategy(rebal_freq=21, top_n=20, use_invvol=True, regime_dampen=0.5),
data[tickers]
),
"Hybrid Biweekly": (
HybridAlphaStrategy(rebal_freq=10, top_n=20, use_invvol=True, regime_dampen=1.0),
data[tickers]
),
# === ROUND 2: RECOVERY QUALITY BLEND ===
"RecQuality Blend Top20": (
RecoveryQualityBlendStrategy(top_n=20, rebal_freq=21),
data[tickers]
),
"RecQuality Blend Top30": (
RecoveryQualityBlendStrategy(top_n=30, rebal_freq=21),
data[tickers]
),
"RecQuality Blend Biweekly": (
RecoveryQualityBlendStrategy(top_n=20, rebal_freq=10),
data[tickers]
),
}
# Run backtests
equity = {}
for name, (strat, strat_data) in strategies.items():
print(f"Running {name}...")
equity[name] = backtest(strat, strat_data, initial_capital=10_000)
# SPY benchmark
bench = data[benchmark].dropna()
equity["SPY"] = (bench / bench.iloc[0]) * 10_000
eq_df = pd.DataFrame(equity).sort_index()
# Yearly returns
years = list(range(2016, 2027))
rows = []
for yr in years:
start = pd.Timestamp(f"{yr}-01-01")
end = pd.Timestamp(f"{yr}-12-31")
window = eq_df.loc[(eq_df.index >= start) & (eq_df.index <= end)].dropna(how="all")
if window.empty:
continue
row = {"Year": yr}
for col in eq_df.columns:
s = window[col].dropna()
if len(s) < 2:
row[col] = np.nan
else:
row[col] = annual_return(s)
rows.append(row)
yr_df = pd.DataFrame(rows).set_index("Year")
print("\n" + "=" * 80)
print("YEARLY TOTAL RETURN (%)")
print("=" * 80)
print((yr_df * 100).round(2).to_string())
# Excess over SPY
excess = yr_df.sub(yr_df["SPY"], axis=0).drop(columns=["SPY"])
print("\n" + "=" * 80)
print("EXCESS vs SPY (pp)")
print("=" * 80)
print((excess * 100).round(2).to_string())
# Full-period summary
print("\n" + "=" * 80)
print("FULL-PERIOD METRICS (sorted by Calmar)")
print("=" * 80)
summary_rows = []
for col in eq_df.columns:
eq = eq_df[col].dropna()
if len(eq) < 252:
continue
summary_rows.append({
"Strategy": col,
"CAGR %": cagr(eq) * 100,
"Sharpe": sharpe(eq),
"Sortino": sortino(eq),
"Max DD %": max_dd(eq) * 100,
"Calmar": calmar(eq),
"Win vs SPY": f"{(excess[col] > 0).sum()}/{len(excess)}" if col in excess.columns else "-",
})
summary = pd.DataFrame(summary_rows).sort_values("Calmar", ascending=False)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', 200)
print(summary.to_string(index=False))
# Turnover analysis
print("\n" + "=" * 80)
print("TURNOVER ANALYSIS")
print("=" * 80)
for name, (strat, strat_data) in strategies.items():
w = strat.generate_signals(strat_data)
avg_turn = w.diff().abs().sum(axis=1).mean()
print(f" {name:<35s} avg daily turnover: {avg_turn:.4f}")
if __name__ == "__main__":
main()

160
research/strategy_improvement_r3.py
View File

@@ -1,160 +0,0 @@
"""
Round 3: Signal-level ensemble and enhanced factor combo.
Focus: improve on FactorCombo's 34.6% CAGR / 1.02 Calmar by:
1. Ensembling two best signals for pick diversification
2. Adding momentum as a tiebreaker signal
3. Concentrating in fewer high-conviction names
4. Tail-risk protection only in extreme drawdowns
"""
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from main import backtest
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.factor_combo import FactorComboStrategy
from strategies.improved_momentum_quality import ImprovedMomentumQualityStrategy
from strategies.ensemble_alpha import EnsembleAlphaStrategy, EnhancedFactorComboStrategy
def annual_return(eq): return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq): return ((eq / eq.cummax()) - 1).min()
def sharpe(eq):
d = eq.pct_change().dropna()
return (d.mean() * 252) / (d.std() * np.sqrt(252)) if d.std() > 0 else 0
def sortino(eq):
d = eq.pct_change().dropna()
ds = d[d < 0].std() * np.sqrt(252)
return (d.mean() * 252) / ds if ds > 0 else 0
def cagr(eq):
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1 if yrs > 0 else 0
def calmar(eq):
dd = max_dd(eq)
return cagr(eq) / abs(dd) if dd < 0 else 0
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
print(f"Universe: {len(tickers)} stocks, data: {data.index[0].date()} to {data.index[-1].date()}")
strategies = {
# Baselines
"FactorCombo rec+deep": (
FactorComboStrategy(signal_name="rec_mfilt+deep_upvol", rebal_freq=21, top_n=20),
data[tickers]
),
"Recovery+Mom Top20": (
RecoveryMomentumStrategy(top_n=20),
data[tickers]
),
"Improved MomQuality": (
ImprovedMomentumQualityStrategy(top_n=20),
data[tickers]
),
# Round 3: Ensemble
"Ensemble Top20": (
EnsembleAlphaStrategy(top_n=20, tail_protection=False),
data[tickers]
),
"Ensemble Top15": (
EnsembleAlphaStrategy(top_n=15, tail_protection=False),
data[tickers]
),
"Ensemble Top20 +Tail": (
EnsembleAlphaStrategy(top_n=20, tail_protection=True, tail_threshold=-0.15, tail_scale=0.5),
data[tickers]
),
"Ensemble Top20 +Tail10": (
EnsembleAlphaStrategy(top_n=20, tail_protection=True, tail_threshold=-0.10, tail_scale=0.5),
data[tickers]
),
# Round 3: Enhanced FactorCombo
"EnhFC Top15 mom20%": (
EnhancedFactorComboStrategy(top_n=15, mom_boost=0.2, tail_protection=False),
data[tickers]
),
"EnhFC Top20 mom20%": (
EnhancedFactorComboStrategy(top_n=20, mom_boost=0.2, tail_protection=False),
data[tickers]
),
"EnhFC Top15 mom30%": (
EnhancedFactorComboStrategy(top_n=15, mom_boost=0.3, tail_protection=False),
data[tickers]
),
"EnhFC Top20 +Tail": (
EnhancedFactorComboStrategy(top_n=20, mom_boost=0.2, tail_protection=True),
data[tickers]
),
"EnhFC Top10 mom20%": (
EnhancedFactorComboStrategy(top_n=10, mom_boost=0.2, tail_protection=False),
data[tickers]
),
}
# Run backtests
equity = {}
for name, (strat, strat_data) in strategies.items():
print(f" {name}...")
equity[name] = backtest(strat, strat_data, initial_capital=10_000)
bench = data[benchmark].dropna()
equity["SPY"] = (bench / bench.iloc[0]) * 10_000
eq_df = pd.DataFrame(equity).sort_index()
# Yearly returns
years = list(range(2016, 2027))
rows = []
for yr in years:
window = eq_df.loc[f"{yr}"].dropna(how="all") if f"{yr}" in eq_df.index.strftime("%Y").unique() else pd.DataFrame()
if window.empty:
continue
row = {"Year": yr}
for col in eq_df.columns:
s = window[col].dropna()
row[col] = annual_return(s) if len(s) >= 2 else np.nan
rows.append(row)
yr_df = pd.DataFrame(rows).set_index("Year")
excess = yr_df.sub(yr_df["SPY"], axis=0).drop(columns=["SPY"])
print("\n" + "=" * 100)
print("YEARLY RETURNS (%)")
print("=" * 100)
print((yr_df * 100).round(1).to_string())
print("\n" + "=" * 100)
print("FULL-PERIOD METRICS")
print("=" * 100)
print(f"{'Strategy':<30s} {'CAGR%':>7s} {'Sharpe':>7s} {'Sortino':>8s} {'MaxDD%':>8s} {'Calmar':>7s} {'WinSPY':>7s}")
print("-" * 78)
results = []
for col in eq_df.columns:
eq = eq_df[col].dropna()
if len(eq) < 252:
continue
wins = (excess[col] > 0).sum() if col in excess.columns else 0
total = len(excess) if col in excess.columns else 0
results.append((col, cagr(eq)*100, sharpe(eq), sortino(eq), max_dd(eq)*100, calmar(eq), f"{wins}/{total}"))
results.sort(key=lambda x: -x[5]) # sort by Calmar
for r in results:
print(f"{r[0]:<30s} {r[1]:>7.1f} {r[2]:>7.2f} {r[3]:>8.2f} {r[4]:>8.1f} {r[5]:>7.2f} {r[6]:>7s}")
if __name__ == "__main__":
main()

174
research/strategy_improvement_r4.py
View File

@@ -1,174 +0,0 @@
"""
Round 4 - Final iteration: Optimize the winning EnhFC strategy.
Findings so far:
- EnhFC Top10 mom20%: 45.8% CAGR, 1.27 Sharpe, -39.8% MaxDD, 1.15 Calmar
- EnhFC Top15 mom20%: 40.6% CAGR, 1.25 Sharpe, -38.1% MaxDD, 1.07 Calmar
Goal: Reduce MaxDD while preserving CAGR. Test:
1. Tail protection variants (threshold / scale combinations)
2. Top10 with tail protection
3. Top12 as middle ground
4. Different momentum weights
"""
import numpy as np
import pandas as pd
import data_manager
from universe import UNIVERSES
from main import backtest
from strategies.factor_combo import FactorComboStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.ensemble_alpha import EnhancedFactorComboStrategy, EnsembleAlphaStrategy
def annual_return(eq): return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq): return ((eq / eq.cummax()) - 1).min()
def sharpe(eq):
d = eq.pct_change().dropna()
return (d.mean() * 252) / (d.std() * np.sqrt(252)) if d.std() > 0 else 0
def sortino(eq):
d = eq.pct_change().dropna()
ds = d[d < 0].std() * np.sqrt(252)
return (d.mean() * 252) / ds if ds > 0 else 0
def cagr(eq):
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1 if yrs > 0 else 0
def calmar(eq):
dd = max_dd(eq)
return cagr(eq) / abs(dd) if dd < 0 else 0
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
print(f"Universe: {len(tickers)} stocks, data: {data.index[0].date()} to {data.index[-1].date()}")
strategies = {
# Baselines
"FactorCombo (orig)": (
FactorComboStrategy(signal_name="rec_mfilt+deep_upvol", rebal_freq=21, top_n=20),
data[tickers]
),
"Recovery+Mom Top20": (
RecoveryMomentumStrategy(top_n=20),
data[tickers]
),
# Winners from R3
"EnhFC Top10": (
EnhancedFactorComboStrategy(top_n=10, mom_boost=0.2, tail_protection=False),
data[tickers]
),
"EnhFC Top15": (
EnhancedFactorComboStrategy(top_n=15, mom_boost=0.2, tail_protection=False),
data[tickers]
),
# Top10 + tail protection variants
"EnhFC Top10 +Tail15/50": (
EnhancedFactorComboStrategy(top_n=10, mom_boost=0.2, tail_protection=True),
data[tickers]
),
# Top12 as middle ground
"EnhFC Top12": (
EnhancedFactorComboStrategy(top_n=12, mom_boost=0.2, tail_protection=False),
data[tickers]
),
"EnhFC Top12 mom15%": (
EnhancedFactorComboStrategy(top_n=12, mom_boost=0.15, tail_protection=False),
data[tickers]
),
"EnhFC Top12 mom25%": (
EnhancedFactorComboStrategy(top_n=12, mom_boost=0.25, tail_protection=False),
data[tickers]
),
# Ensemble variants
"Ensemble Top12": (
EnsembleAlphaStrategy(top_n=12, tail_protection=False),
data[tickers]
),
"Ensemble Top10": (
EnsembleAlphaStrategy(top_n=10, tail_protection=False),
data[tickers]
),
"Ensemble Top15 +Tail": (
EnsembleAlphaStrategy(top_n=15, tail_protection=True, tail_threshold=-0.12, tail_scale=0.4),
data[tickers]
),
}
# Run
equity = {}
for name, (strat, strat_data) in strategies.items():
print(f" {name}...")
equity[name] = backtest(strat, strat_data, initial_capital=10_000)
bench = data[benchmark].dropna()
equity["SPY"] = (bench / bench.iloc[0]) * 10_000
eq_df = pd.DataFrame(equity).sort_index()
# Yearly returns
years = sorted(eq_df.index.year.unique())
rows = []
for yr in years:
window = eq_df.loc[eq_df.index.year == yr].dropna(how="all")
if window.empty:
continue
row = {"Year": yr}
for col in eq_df.columns:
s = window[col].dropna()
row[col] = annual_return(s) if len(s) >= 2 else np.nan
rows.append(row)
yr_df = pd.DataFrame(rows).set_index("Year")
excess = yr_df.sub(yr_df["SPY"], axis=0).drop(columns=["SPY"])
print("\n" + "=" * 100)
print("YEARLY RETURNS (%)")
print("=" * 100)
print((yr_df * 100).round(1).to_string())
print("\n" + "=" * 100)
print("FULL-PERIOD METRICS (sorted by Calmar)")
print("=" * 100)
print(f"{'Strategy':<28s} {'CAGR%':>7s} {'Sharpe':>7s} {'Sortino':>8s} {'MaxDD%':>8s} {'Calmar':>7s} {'WinSPY':>7s}")
print("-" * 76)
results = []
for col in eq_df.columns:
eq = eq_df[col].dropna()
if len(eq) < 252:
continue
wins = (excess[col] > 0).sum() if col in excess.columns else 0
total = len(excess) if col in excess.columns else 0
results.append((col, cagr(eq)*100, sharpe(eq), sortino(eq), max_dd(eq)*100, calmar(eq), f"{wins}/{total}"))
results.sort(key=lambda x: -x[5])
for r in results:
print(f"{r[0]:<28s} {r[1]:>7.1f} {r[2]:>7.2f} {r[3]:>8.2f} {r[4]:>8.1f} {r[5]:>7.2f} {r[6]:>7s}")
# Highlight the best by different criteria
print("\n--- BEST BY CRITERIA ---")
best_cagr = max(results, key=lambda x: x[1])
best_sharpe = max(results, key=lambda x: x[2])
best_calmar = max(results, key=lambda x: x[5])
best_dd = min(results, key=lambda x: abs(x[4]))
print(f" Best CAGR: {best_cagr[0]} ({best_cagr[1]:.1f}%)")
print(f" Best Sharpe: {best_sharpe[0]} ({best_sharpe[2]:.2f})")
print(f" Best Calmar: {best_calmar[0]} ({best_calmar[5]:.2f})")
print(f" Best MaxDD: {best_dd[0]} ({best_dd[4]:.1f}%)")
if __name__ == "__main__":
main()

370
research/strategy_risk_managed_eval.py
View File

@@ -1,370 +0,0 @@
"""
Risk-Managed Ensemble Strategy Evaluation.
Validation protocol:
1. Parameter sensitivity sweep: target_vol × dd_dampen combinations
2. IS/OOS split: IS=2016-04 to 2022-12, OOS=2023-01 to 2026-05
3. Block bootstrap: CIs for CAGR/Sharpe/MaxDD
4. Yearly returns table
5. Overfitting checks (IS→OOS decay, parameter sensitivity)
"""
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import data_manager
from universe import UNIVERSES
from main import backtest
from strategies.ensemble_alpha import (
EnsembleAlphaStrategy,
RiskManagedEnsembleStrategy,
)
# ---------------------------------------------------------------------------
# Metrics
# ---------------------------------------------------------------------------
def annual_return(eq): return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq): return ((eq / eq.cummax()) - 1).min()
def sharpe(eq):
d = eq.pct_change().dropna()
return (d.mean() * 252) / (d.std() * np.sqrt(252)) if d.std() > 0 else 0
def sortino(eq):
d = eq.pct_change().dropna()
ds = d[d < 0].std() * np.sqrt(252)
return (d.mean() * 252) / ds if ds > 0 else 0
def cagr(eq):
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1 if yrs > 0 else 0
def calmar(eq):
dd = max_dd(eq)
return cagr(eq) / abs(dd) if dd < 0 else 0
def realized_vol(eq):
return eq.pct_change().dropna().std() * np.sqrt(252)
# ---------------------------------------------------------------------------
# Block Bootstrap (from research/trend_rider_p0.py pattern)
# ---------------------------------------------------------------------------
def block_bootstrap(returns: pd.Series, n_boot: int = 5000,
block_len: int = 21, seed: int = 42) -> pd.DataFrame:
"""Stationary block bootstrap preserving autocorrelation."""
r = returns.values
n = len(r)
rng = np.random.default_rng(seed)
n_blocks = int(np.ceil(n / block_len))
span_years = n / 252.0
cagrs = np.empty(n_boot)
sharpes = np.empty(n_boot)
mdds = np.empty(n_boot)
for b in range(n_boot):
starts = rng.integers(0, n - block_len + 1, size=n_blocks)
idx = (starts[:, None] + np.arange(block_len)[None, :]).ravel()[:n]
sample = r[idx]
equity = np.cumprod(1.0 + sample)
cagrs[b] = equity[-1] ** (1.0 / span_years) - 1.0
std = sample.std(ddof=1)
sharpes[b] = (sample.mean() / std * np.sqrt(252)) if std > 0 else 0.0
running_max = np.maximum.accumulate(equity)
mdds[b] = float(np.min(equity / running_max - 1.0))
return pd.DataFrame({"cagr": cagrs, "sharpe": sharpes, "max_drawdown": mdds})
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
IS_END = "2022-12-31"
OOS_START = "2023-01-01"
def run_backtest_window(strat, data, start=None, end=None):
"""Run backtest on a time window."""
d = data.copy()
if start:
d = d[d.index >= start]
if end:
d = d[d.index <= end]
return backtest(strat, d, initial_capital=10_000)
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
stock_data = data[tickers]
print(f"Universe: {len(tickers)} stocks")
print(f"Data range: {data.index[0].date()} to {data.index[-1].date()}")
print(f"IS period: {data.index[0].date()} to {IS_END}")
print(f"OOS period: {OOS_START} to {data.index[-1].date()}")
# =========================================================================
# PART 1: Parameter Sensitivity Sweep (full period)
# =========================================================================
print("\n" + "=" * 100)
print(" PART 1: PARAMETER SENSITIVITY (full period)")
print("=" * 100)
print(f" {'Config':<40s} {'CAGR%':>7s} {'Sharpe':>7s} {'Sortino':>8s} {'MaxDD%':>8s} {'Calmar':>7s} {'Vol%':>6s}")
print(" " + "-" * 83)
# Baseline (no risk management)
base = EnsembleAlphaStrategy(top_n=10, tail_protection=False)
eq_base = backtest(base, stock_data, initial_capital=10_000)
print(f" {'Ensemble Top10 (NO risk mgmt)':<40s} {cagr(eq_base)*100:>7.1f} {sharpe(eq_base):>7.2f} {sortino(eq_base):>8.2f} {max_dd(eq_base)*100:>8.1f} {calmar(eq_base):>7.2f} {realized_vol(eq_base)*100:>6.1f}")
configs = []
# Sweep target_vol × dd_dampen
for tv in [0.15, 0.18, 0.20, 0.22, 0.25]:
for dd_on in [True, False]:
for dd_fl in [0.20, 0.30] if dd_on else [0.30]:
for dd_dn in [0.25, 0.30] if dd_on else [0.30]:
strat = RiskManagedEnsembleStrategy(
top_n=10, target_vol=tv, vol_window=20,
dd_dampen=dd_on, dd_floor=dd_fl, dd_denom=dd_dn,
)
eq = backtest(strat, stock_data, initial_capital=10_000)
label = f"vt={tv:.2f} dd={'Y' if dd_on else 'N'} fl={dd_fl:.2f} dn={dd_dn:.2f}"
c = cagr(eq)
s = sharpe(eq)
so = sortino(eq)
mdd = max_dd(eq)
cal = calmar(eq)
rv = realized_vol(eq)
configs.append({
"label": label, "target_vol": tv, "dd_on": dd_on,
"dd_floor": dd_fl, "dd_denom": dd_dn,
"CAGR": c, "Sharpe": s, "Sortino": so,
"MaxDD": mdd, "Calmar": cal, "Vol": rv,
"equity": eq,
})
print(f" {label:<40s} {c*100:>7.1f} {s:>7.2f} {so:>8.2f} {mdd*100:>8.1f} {cal:>7.2f} {rv*100:>6.1f}")
# Find configs meeting target (CAGR>40%, Sharpe>1.5, MaxDD>-25%)
print("\n --- Configs meeting CAGR>40%, Sharpe>1.5, MaxDD>-25% ---")
meeting = [c for c in configs if c["CAGR"] > 0.40 and c["Sharpe"] > 1.5 and c["MaxDD"] > -0.25]
if meeting:
for c in sorted(meeting, key=lambda x: -x["Calmar"]):
print(f" ✓ {c['label']:<40s} CAGR={c['CAGR']*100:.1f}% Sharpe={c['Sharpe']:.2f} MaxDD={c['MaxDD']*100:.1f}% Calmar={c['Calmar']:.2f}")
else:
print(" (None meet all three criteria simultaneously)")
# Find best Calmar among those with CAGR>35%
print("\n --- Best Calmar with CAGR>35% ---")
high_cagr = [c for c in configs if c["CAGR"] > 0.35]
for c in sorted(high_cagr, key=lambda x: -x["Calmar"])[:5]:
print(f" → {c['label']:<40s} CAGR={c['CAGR']*100:.1f}% Sharpe={c['Sharpe']:.2f} MaxDD={c['MaxDD']*100:.1f}% Calmar={c['Calmar']:.2f}")
# Select recommended config (best Calmar with CAGR>40% OR highest Sharpe with MaxDD>-28%)
candidates = [c for c in configs if c["CAGR"] > 0.38]
if not candidates:
candidates = sorted(configs, key=lambda x: -x["Calmar"])
best = max(candidates, key=lambda x: x["Calmar"])
print(f"\n >>> RECOMMENDED: {best['label']}")
print(f" CAGR={best['CAGR']*100:.1f}% Sharpe={best['Sharpe']:.2f} MaxDD={best['MaxDD']*100:.1f}% Calmar={best['Calmar']:.2f}")
# =========================================================================
# PART 2: IS/OOS Validation
# =========================================================================
print("\n" + "=" * 100)
print(" PART 2: IN-SAMPLE vs OUT-OF-SAMPLE")
print("=" * 100)
rec_strat = RiskManagedEnsembleStrategy(
top_n=10, target_vol=best["target_vol"], vol_window=20,
dd_dampen=best["dd_on"], dd_floor=best["dd_floor"], dd_denom=best["dd_denom"],
)
# IS window
is_data = stock_data[stock_data.index <= IS_END]
eq_is = backtest(rec_strat, is_data, initial_capital=10_000)
# OOS window
oos_data = stock_data[stock_data.index >= OOS_START]
eq_oos = backtest(rec_strat, oos_data, initial_capital=10_000)
# Baseline IS/OOS
eq_base_is = backtest(base, is_data, initial_capital=10_000)
eq_base_oos = backtest(base, oos_data, initial_capital=10_000)
print(f"\n {'Metric':<20s} {'IS (→2022)':<20s} {'OOS (2023→)':<20s} {'Decay':>10s}")
print(" " + "-" * 73)
for name, eq_i, eq_o in [
("RiskManaged", eq_is, eq_oos),
("Base (no RM)", eq_base_is, eq_base_oos),
]:
c_is, c_oos = cagr(eq_i), cagr(eq_o)
s_is, s_oos = sharpe(eq_i), sharpe(eq_o)
d_is, d_oos = max_dd(eq_i), max_dd(eq_o)
decay = (c_oos - c_is) / abs(c_is) * 100 if c_is != 0 else 0
print(f" {name} CAGR {c_is*100:>8.1f}% {c_oos*100:>8.1f}% {decay:>+6.1f}%")
print(f" {name} Sharpe {s_is:>8.2f} {s_oos:>8.2f} {(s_oos/s_is-1)*100 if s_is else 0:>+6.1f}%")
print(f" {name} MaxDD {d_is*100:>8.1f}% {d_oos*100:>8.1f}%")
print()
# =========================================================================
# PART 3: Block Bootstrap
# =========================================================================
print("=" * 100)
print(" PART 3: BLOCK BOOTSTRAP (5000 resamples, block=21 days)")
print("=" * 100)
eq_full = best["equity"]
rets = eq_full.pct_change().dropna()
boot = block_bootstrap(rets, n_boot=5000, block_len=21)
qs = [0.025, 0.05, 0.25, 0.50, 0.75, 0.95, 0.975]
summary = boot.quantile(qs).T
summary.columns = [f"p{q:.1%}" for q in qs]
summary["mean"] = boot.mean()
print(f"\n {summary.to_string()}")
print(f"\n Key probabilities:")
print(f" P(CAGR > 40%) = {(boot['cagr'] > 0.40).mean()*100:.1f}%")
print(f" P(CAGR > 30%) = {(boot['cagr'] > 0.30).mean()*100:.1f}%")
print(f" P(Sharpe > 1.5) = {(boot['sharpe'] > 1.5).mean()*100:.1f}%")
print(f" P(Sharpe > 1.0) = {(boot['sharpe'] > 1.0).mean()*100:.1f}%")
print(f" P(MaxDD > -25%) = {(boot['max_drawdown'] > -0.25).mean()*100:.1f}%")
print(f" P(MaxDD > -30%) = {(boot['max_drawdown'] > -0.30).mean()*100:.1f}%")
print(f" P(MaxDD < -40%) = {(boot['max_drawdown'] < -0.40).mean()*100:.1f}%")
# =========================================================================
# PART 4: Yearly Returns
# =========================================================================
print("\n" + "=" * 100)
print(" PART 4: YEARLY RETURNS")
print("=" * 100)
# SPY benchmark
bench = data[benchmark].dropna()
eq_spy = (bench / bench.iloc[0]) * 10_000
strategies_yearly = {
"Ensemble Top10 (raw)": eq_base,
f"RiskManaged ({best['label']})": eq_full,
"SPY": eq_spy,
}
eq_df = pd.DataFrame(strategies_yearly).sort_index()
years = sorted(eq_df.index.year.unique())
print(f"\n {'Year':<6s} {'Ens Raw%':>10s} {'RiskMgd%':>10s} {'SPY%':>10s} {'RM excess':>10s}")
print(" " + "-" * 50)
for yr in years:
window = eq_df.loc[eq_df.index.year == yr].dropna(how="all")
if window.empty or len(window) < 2:
continue
rets_yr = {}
for col in eq_df.columns:
s = window[col].dropna()
rets_yr[col] = annual_return(s) if len(s) >= 2 else np.nan
spy_r = rets_yr.get("SPY", 0)
rm_r = rets_yr.get(f"RiskManaged ({best['label']})", 0)
raw_r = rets_yr.get("Ensemble Top10 (raw)", 0)
print(f" {yr:<6d} {raw_r*100:>10.1f} {rm_r*100:>10.1f} {spy_r*100:>10.1f} {(rm_r-spy_r)*100:>+10.1f}")
# =========================================================================
# PART 5: Overfitting Assessment
# =========================================================================
print("\n" + "=" * 100)
print(" PART 5: OVERFITTING ASSESSMENT")
print("=" * 100)
checks = []
c_is_rm, c_oos_rm = cagr(eq_is), cagr(eq_oos)
s_is_rm, s_oos_rm = sharpe(eq_is), sharpe(eq_oos)
# Check 1: OOS CAGR >= 80% of IS
ratio = c_oos_rm / c_is_rm if c_is_rm > 0 else 0
checks.append(("OOS CAGR ≥ 80% of IS CAGR", ratio >= 0.8,
f"{ratio:.1%} (IS={c_is_rm*100:.1f}%, OOS={c_oos_rm*100:.1f}%)"))
# Check 2: OOS Sharpe >= IS × 0.8
s_ratio = s_oos_rm / s_is_rm if s_is_rm > 0 else 0
checks.append(("OOS Sharpe ≥ IS × 0.8", s_ratio >= 0.8,
f"{s_ratio:.1%} (IS={s_is_rm:.2f}, OOS={s_oos_rm:.2f})"))
# Check 3: P(MaxDD > -30%) > 90%
p_mdd30 = (boot["max_drawdown"] > -0.30).mean()
checks.append(("Bootstrap P(MaxDD > -30%) > 90%", p_mdd30 > 0.90,
f"{p_mdd30:.1%}"))
# Check 4: P(Sharpe < 1.0) < 10%
p_sharpe1 = (boot["sharpe"] < 1.0).mean()
checks.append(("Bootstrap P(Sharpe < 1.0) < 10%", p_sharpe1 < 0.10,
f"{p_sharpe1:.1%}"))
# Check 5: Parameter sensitivity (check adjacent configs)
adj_configs = [c for c in configs
if abs(c["target_vol"] - best["target_vol"]) <= 0.03
and c["dd_on"] == best["dd_on"]]
if adj_configs:
cagrs_adj = [c["CAGR"] for c in adj_configs]
spread = (max(cagrs_adj) - min(cagrs_adj)) / np.mean(cagrs_adj)
checks.append(("Adjacent params within 20% CAGR spread", spread < 0.20,
f"spread={spread:.1%}, range=[{min(cagrs_adj)*100:.1f}%, {max(cagrs_adj)*100:.1f}%]"))
# Check 6: PIT compliance
checks.append(("PIT compliance (all signals use T-1 data)", True,
"shift(1) in ensemble + shift(1) in vol/dd overlay"))
print()
all_pass = True
for name, passed, detail in checks:
status = "✓ PASS" if passed else "✗ FAIL"
all_pass = all_pass and passed
print(f" [{status}] {name}")
print(f" {detail}")
print(f"\n {'='*40}")
if all_pass:
print(f" ALL CHECKS PASSED — strategy is NOT overfitted")
else:
print(f" SOME CHECKS FAILED — review before production use")
# =========================================================================
# SUMMARY
# =========================================================================
print("\n" + "=" * 100)
print(" FINAL SUMMARY")
print("=" * 100)
print(f"""
Strategy: RiskManagedEnsembleStrategy
Config: top_n=10, target_vol={best['target_vol']:.2f}, vol_window=20,
dd_dampen={best['dd_on']}, dd_floor={best['dd_floor']:.2f}, dd_denom={best['dd_denom']:.2f}
Full-period performance:
CAGR = {best['CAGR']*100:.1f}%
Sharpe = {best['Sharpe']:.2f}
Sortino = {best['Sortino']:.2f}
MaxDD = {best['MaxDD']*100:.1f}%
Calmar = {best['Calmar']:.2f}
Vol = {best['Vol']*100:.1f}%
vs Baseline (no risk mgmt):
CAGR = {cagr(eq_base)*100:.1f}% → {best['CAGR']*100:.1f}% ({(best['CAGR']-cagr(eq_base))*100:+.1f}pp)
Sharpe = {sharpe(eq_base):.2f} → {best['Sharpe']:.2f} ({best['Sharpe']-sharpe(eq_base):+.2f})
MaxDD = {max_dd(eq_base)*100:.1f}% → {best['MaxDD']*100:.1f}% ({(best['MaxDD']-max_dd(eq_base))*100:+.1f}pp)
""")
if __name__ == "__main__":
main()

240
research/strategy_risk_managed_r2.py
View File

@@ -1,240 +0,0 @@
"""
Round 2: Risk-Managed Ensemble with DD-reactive approach.
Key insight from R1: vol-target uniformly compresses returns (including uptrends),
losing too much CAGR. New approach: only cut exposure DURING drawdowns, not globally.
"""
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import data_manager
from universe import UNIVERSES
from main import backtest
from strategies.ensemble_alpha import (
EnsembleAlphaStrategy,
RiskManagedEnsembleStrategy,
)
def annual_return(eq): return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq): return ((eq / eq.cummax()) - 1).min()
def sharpe(eq):
d = eq.pct_change().dropna()
return (d.mean() * 252) / (d.std() * np.sqrt(252)) if d.std() > 0 else 0
def sortino(eq):
d = eq.pct_change().dropna()
ds = d[d < 0].std() * np.sqrt(252)
return (d.mean() * 252) / ds if ds > 0 else 0
def cagr(eq):
yrs = (eq.index[-1] - eq.index[0]).days / 365.25
return (eq.iloc[-1] / eq.iloc[0]) ** (1 / yrs) - 1 if yrs > 0 else 0
def calmar(eq):
dd = max_dd(eq)
return cagr(eq) / abs(dd) if dd < 0 else 0
def realized_vol(eq):
return eq.pct_change().dropna().std() * np.sqrt(252)
def block_bootstrap(returns, n_boot=5000, block_len=21, seed=42):
r = returns.values
n = len(r)
rng = np.random.default_rng(seed)
n_blocks = int(np.ceil(n / block_len))
span_years = n / 252.0
cagrs = np.empty(n_boot)
sharpes = np.empty(n_boot)
mdds = np.empty(n_boot)
for b in range(n_boot):
starts = rng.integers(0, n - block_len + 1, size=n_blocks)
idx = (starts[:, None] + np.arange(block_len)[None, :]).ravel()[:n]
sample = r[idx]
equity = np.cumprod(1.0 + sample)
cagrs[b] = equity[-1] ** (1.0 / span_years) - 1.0
std = sample.std(ddof=1)
sharpes[b] = (sample.mean() / std * np.sqrt(252)) if std > 0 else 0.0
running_max = np.maximum.accumulate(equity)
mdds[b] = float(np.min(equity / running_max - 1.0))
return pd.DataFrame({"cagr": cagrs, "sharpe": sharpes, "max_drawdown": mdds})
IS_END = "2022-12-31"
OOS_START = "2023-01-01"
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
stock_data = data[tickers]
print(f"Universe: {len(tickers)} stocks, {data.index[0].date()} to {data.index[-1].date()}")
# =========================================================================
# Baseline
# =========================================================================
base = EnsembleAlphaStrategy(top_n=10, tail_protection=False)
eq_base = backtest(base, stock_data, initial_capital=10_000)
print(f"\nBaseline (no RM): CAGR={cagr(eq_base)*100:.1f}% Sharpe={sharpe(eq_base):.2f} MaxDD={max_dd(eq_base)*100:.1f}% Vol={realized_vol(eq_base)*100:.1f}%")
# =========================================================================
# Parameter sweep: DD-reactive approach
# =========================================================================
print("\n" + "=" * 110)
print(" DD-REACTIVE RISK MANAGEMENT SWEEP")
print("=" * 110)
print(f" {'Config':<55s} {'CAGR%':>7s} {'Sharpe':>7s} {'Sortino':>8s} {'MaxDD%':>8s} {'Calmar':>7s} {'Vol%':>6s}")
print(" " + "-" * 98)
configs = []
for dd_fl in [0.15, 0.20, 0.25, 0.30, 0.40]:
for dd_dn in [0.15, 0.20, 0.25, 0.30]:
for vsg in [True, False]:
for vsf in [0.40, 0.50, 0.60] if vsg else [0.50]:
strat = RiskManagedEnsembleStrategy(
top_n=10,
dd_floor=dd_fl, dd_denom=dd_dn,
vol_spike_guard=vsg, vol_spike_floor=vsf,
)
eq = backtest(strat, stock_data, initial_capital=10_000)
label = f"fl={dd_fl:.2f} dn={dd_dn:.2f} vsg={'Y' if vsg else 'N'} vsf={vsf:.2f}"
c = cagr(eq); s = sharpe(eq); so = sortino(eq)
mdd = max_dd(eq); cal = calmar(eq); rv = realized_vol(eq)
configs.append({
"label": label, "dd_floor": dd_fl, "dd_denom": dd_dn,
"vsg": vsg, "vsf": vsf,
"CAGR": c, "Sharpe": s, "Sortino": so,
"MaxDD": mdd, "Calmar": cal, "Vol": rv, "equity": eq,
})
# Only print selected configs to keep output manageable
if dd_dn in [0.20, 0.25] and dd_fl in [0.20, 0.25, 0.30] and vsf in [0.50]:
print(f" {label:<55s} {c*100:>7.1f} {s:>7.2f} {so:>8.2f} {mdd*100:>8.1f} {cal:>7.2f} {rv*100:>6.1f}")
# =========================================================================
# Find configs meeting targets
# =========================================================================
print("\n --- MEETING CAGR>40%, Sharpe>1.5, MaxDD>-25% ---")
meeting = [c for c in configs if c["CAGR"] > 0.40 and c["Sharpe"] > 1.5 and c["MaxDD"] > -0.25]
if meeting:
for c in sorted(meeting, key=lambda x: -x["Calmar"])[:8]:
print(f" ✓ {c['label']:<50s} CAGR={c['CAGR']*100:.1f}% Sharpe={c['Sharpe']:.2f} MaxDD={c['MaxDD']*100:.1f}% Calmar={c['Calmar']:.2f}")
else:
print(" (None)")
# Relax criteria
print("\n --- MEETING CAGR>38%, Sharpe>1.4, MaxDD>-25% ---")
meeting2 = [c for c in configs if c["CAGR"] > 0.38 and c["Sharpe"] > 1.4 and c["MaxDD"] > -0.25]
if meeting2:
for c in sorted(meeting2, key=lambda x: -x["Calmar"])[:8]:
print(f" → {c['label']:<50s} CAGR={c['CAGR']*100:.1f}% Sharpe={c['Sharpe']:.2f} MaxDD={c['MaxDD']*100:.1f}% Calmar={c['Calmar']:.2f}")
print("\n --- BEST CALMAR with CAGR>35% ---")
hi = [c for c in configs if c["CAGR"] > 0.35]
for c in sorted(hi, key=lambda x: -x["Calmar"])[:5]:
print(f" → {c['label']:<50s} CAGR={c['CAGR']*100:.1f}% Sharpe={c['Sharpe']:.2f} MaxDD={c['MaxDD']*100:.1f}% Calmar={c['Calmar']:.2f}")
print("\n --- BEST with MaxDD > -25% ---")
lo_dd = [c for c in configs if c["MaxDD"] > -0.25]
for c in sorted(lo_dd, key=lambda x: -x["CAGR"])[:5]:
print(f" → {c['label']:<50s} CAGR={c['CAGR']*100:.1f}% Sharpe={c['Sharpe']:.2f} MaxDD={c['MaxDD']*100:.1f}% Calmar={c['Calmar']:.2f}")
# Pick best overall by Calmar with CAGR > 38%
candidates = [c for c in configs if c["CAGR"] > 0.38]
if not candidates:
candidates = sorted(configs, key=lambda x: -x["Calmar"])
best = max(candidates, key=lambda x: x["Calmar"])
print(f"\n >>> RECOMMENDED: {best['label']}")
print(f" CAGR={best['CAGR']*100:.1f}% Sharpe={best['Sharpe']:.2f} Sortino={best['Sortino']:.2f} MaxDD={best['MaxDD']*100:.1f}% Calmar={best['Calmar']:.2f} Vol={best['Vol']*100:.1f}%")
# =========================================================================
# IS/OOS for recommended
# =========================================================================
print("\n" + "=" * 110)
print(" IS/OOS VALIDATION")
print("=" * 110)
rec_strat = RiskManagedEnsembleStrategy(
top_n=10, dd_floor=best["dd_floor"], dd_denom=best["dd_denom"],
vol_spike_guard=best["vsg"], vol_spike_floor=best["vsf"],
)
is_data = stock_data[stock_data.index <= IS_END]
oos_data = stock_data[stock_data.index >= OOS_START]
eq_is = backtest(rec_strat, is_data, initial_capital=10_000)
eq_oos = backtest(rec_strat, oos_data, initial_capital=10_000)
eq_base_is = backtest(base, is_data, initial_capital=10_000)
eq_base_oos = backtest(base, oos_data, initial_capital=10_000)
print(f"\n {'Strategy':<25s} {'Window':<10s} {'CAGR%':>7s} {'Sharpe':>7s} {'MaxDD%':>8s} {'Calmar':>7s}")
print(" " + "-" * 68)
for nm, ei, eo in [("RiskManaged", eq_is, eq_oos), ("Base (no RM)", eq_base_is, eq_base_oos)]:
print(f" {nm:<25s} {'IS':<10s} {cagr(ei)*100:>7.1f} {sharpe(ei):>7.2f} {max_dd(ei)*100:>8.1f} {calmar(ei):>7.2f}")
print(f" {nm:<25s} {'OOS':<10s} {cagr(eo)*100:>7.1f} {sharpe(eo):>7.2f} {max_dd(eo)*100:>8.1f} {calmar(eo):>7.2f}")
# =========================================================================
# Bootstrap on recommended
# =========================================================================
print("\n" + "=" * 110)
print(" BLOCK BOOTSTRAP (5000 resamples)")
print("=" * 110)
rets = best["equity"].pct_change().dropna()
boot = block_bootstrap(rets)
print(f"\n P(CAGR > 40%) = {(boot['cagr'] > 0.40).mean()*100:.1f}%")
print(f" P(CAGR > 30%) = {(boot['cagr'] > 0.30).mean()*100:.1f}%")
print(f" P(Sharpe > 1.5) = {(boot['sharpe'] > 1.5).mean()*100:.1f}%")
print(f" P(Sharpe > 1.0) = {(boot['sharpe'] > 1.0).mean()*100:.1f}%")
print(f" P(MaxDD > -25%) = {(boot['max_drawdown'] > -0.25).mean()*100:.1f}%")
print(f" P(MaxDD > -30%) = {(boot['max_drawdown'] > -0.30).mean()*100:.1f}%")
# =========================================================================
# Yearly returns
# =========================================================================
print("\n" + "=" * 110)
print(" YEARLY RETURNS")
print("=" * 110)
bench_eq = data[benchmark].dropna()
bench_eq = (bench_eq / bench_eq.iloc[0]) * 10_000
eq_df = pd.DataFrame({
"Raw Ens10": eq_base,
"RiskManaged": best["equity"],
"SPY": bench_eq,
}).sort_index()
years = sorted(eq_df.index.year.unique())
print(f"\n {'Year':<6s} {'Raw%':>8s} {'RM%':>8s} {'SPY%':>8s} {'RM-SPY':>8s}")
print(" " + "-" * 42)
for yr in years:
w = eq_df.loc[eq_df.index.year == yr].dropna(how="all")
if w.empty or len(w) < 2:
continue
r_raw = annual_return(w["Raw Ens10"].dropna()) if len(w["Raw Ens10"].dropna()) >= 2 else 0
r_rm = annual_return(w["RiskManaged"].dropna()) if len(w["RiskManaged"].dropna()) >= 2 else 0
r_spy = annual_return(w["SPY"].dropna()) if len(w["SPY"].dropna()) >= 2 else 0
print(f" {yr:<6d} {r_raw*100:>8.1f} {r_rm*100:>8.1f} {r_spy*100:>8.1f} {(r_rm-r_spy)*100:>+8.1f}")
# =========================================================================
# Summary
# =========================================================================
print(f"\n{'='*110}")
print(f" FINAL: RiskManagedEnsembleStrategy")
print(f" Config: top_n=10, dd_floor={best['dd_floor']}, dd_denom={best['dd_denom']}, vsg={best['vsg']}, vsf={best['vsf']}")
print(f" CAGR={best['CAGR']*100:.1f}% Sharpe={best['Sharpe']:.2f} Sortino={best['Sortino']:.2f} MaxDD={best['MaxDD']*100:.1f}% Calmar={best['Calmar']:.2f}")
print(f" vs Raw: CAGR {(best['CAGR']-cagr(eq_base))*100:+.1f}pp Sharpe {best['Sharpe']-sharpe(eq_base):+.2f} MaxDD {(best['MaxDD']-max_dd(eq_base))*100:+.1f}pp")
if __name__ == "__main__":
main()

291
research/strategy_sharpe_boost.py
View File

@@ -1,291 +0,0 @@
"""
Sharpe boost research: blend pure momentum into the Ensemble signal.
Root cause of Sharpe=1.32 (not 1.5+):
- 2021: recovery signals returned +3% vs SPY +30.5%
- In low-vol steady uptrends, "bouncing from bottom" stocks don't exist
- Pure 12-1 momentum captures "steady grinders" that do well in these regimes
Approach: Add a 3rd signal (pure momentum rank) to the ensemble with weight α,
reducing existing signals to (1-α)/2 each.
Test α ∈ {0.20, 0.25, 0.30, 0.35, 0.40} and pick the one that maximizes Sharpe
without materially hurting CAGR.
Also test: market-DD dampener ON TOP of the blended signal (risk-managed version).
"""
from __future__ import annotations
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.base import Strategy
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
class MomentumBlendEnsembleStrategy(Strategy):
"""
Ensemble of 3 signals: rec_mfilt+deep_upvol, recovery63+mom, pure momentum.
The pure momentum signal provides diversification in low-vol steady trends.
"""
def __init__(
self,
rebal_freq: int = 21,
top_n: int = 10,
mom_blend: float = 0.30, # weight on pure momentum signal
dd_floor: float = 0.40,
dd_denom: float = 0.20,
risk_managed: bool = True,
):
self.rebal_freq = rebal_freq
self.top_n = top_n
self.mom_blend = mom_blend
self.dd_floor = dd_floor
self.dd_denom = dd_denom
self.risk_managed = risk_managed
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
p = data
# === Signal A: rec_mfilt + deep_upvol ===
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
rec_mfilt_r = _rank(rec_mfilt)
ret = p.pct_change()
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r
# === Signal B: Recovery 63d + 12-1 momentum ===
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r
# === Signal C: Pure 12-1 momentum (diversification in melt-ups) ===
signal_c = mom_r # already computed above
# === Ensemble: weighted average ===
α = self.mom_blend
ensemble = (1 - α) / 2.0 * signal_a + (1 - α) / 2.0 * signal_b + α * signal_c
# === Select top_n ===
rank = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
# Equal weight
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# === Monthly rebalance ===
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
signals = signals.shift(1).fillna(0.0) # PIT
# === Risk management: market-DD dampener ===
if self.risk_managed:
daily_rets = data.pct_change().fillna(0.0)
mkt_rets = daily_rets.mean(axis=1)
mkt_eq = (1 + mkt_rets).cumprod()
mkt_dd = mkt_eq / mkt_eq.cummax() - 1
dd_scale = (1.0 + mkt_dd / self.dd_denom).clip(
lower=self.dd_floor, upper=1.0
)
dd_scale_lagged = dd_scale.shift(1).fillna(1.0)
signals = signals.mul(dd_scale_lagged, axis=0)
return signals
# ---------------------------------------------------------------------------
# Evaluation
# ---------------------------------------------------------------------------
def compute_metrics(daily_rets: pd.Series) -> dict:
"""Compute standard performance metrics from daily returns."""
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {
"cagr": cagr,
"vol": vol,
"sharpe": sharpe,
"max_dd": max_dd,
"calmar": calmar,
}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
"""Compute annual returns."""
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
_DATA_CACHE = {}
def backtest_strategy(strategy, start="2016-04-01", end="2026-05-13"):
"""Run backtest and return daily portfolio returns."""
import data_manager
if "data" not in _DATA_CACHE:
from universe import get_sp500
tickers = get_sp500()
data_manager.update("us", tickers)
_DATA_CACHE["data"] = data_manager.load("us")
data = _DATA_CACHE["data"]
if data is None:
raise RuntimeError("No data loaded")
weights = strategy.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
# Trim to evaluation period
daily_rets = daily_rets.loc[start:end]
return daily_rets
def main():
print("=" * 80)
print("SHARPE BOOST: Momentum blend into Ensemble signal")
print("=" * 80)
# --- Parameter sweep: mom_blend ---
blends = [0.0, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40]
print("\n--- Sweep: mom_blend (risk_managed=False) ---")
print(f"{'blend':>6s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>7s} {'MaxDD':>7s} {'Calmar':>7s}")
print("-" * 50)
results_no_rm = {}
for α in blends:
strat = MomentumBlendEnsembleStrategy(
top_n=10, mom_blend=α, risk_managed=False
)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
results_no_rm[α] = {"rets": rets, "metrics": m}
print(
f"{α:>6.2f} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>7.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>7.2f}"
)
print("\n--- Sweep: mom_blend (risk_managed=True, dd_floor=0.40, dd_denom=0.20) ---")
print(f"{'blend':>6s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>7s} {'MaxDD':>7s} {'Calmar':>7s}")
print("-" * 50)
results_rm = {}
for α in blends:
strat = MomentumBlendEnsembleStrategy(
top_n=10, mom_blend=α, risk_managed=True
)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
results_rm[α] = {"rets": rets, "metrics": m}
print(
f"{α:>6.2f} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>7.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>7.2f}"
)
# --- Best config: yearly breakdown ---
best_α = max(results_rm, key=lambda k: results_rm[k]["metrics"]["sharpe"])
print(f"\n{'=' * 80}")
print(f"BEST CONFIG: mom_blend={best_α:.2f} + risk_managed=True")
print(f"{'=' * 80}")
best_rets = results_rm[best_α]["rets"]
best_m = results_rm[best_α]["metrics"]
print(f"CAGR: {best_m['cagr']*100:.1f}% Vol: {best_m['vol']*100:.1f}% "
f"Sharpe: {best_m['sharpe']:.2f} MaxDD: {best_m['max_dd']*100:.1f}% "
f"Calmar: {best_m['calmar']:.2f}")
print("\n--- Yearly returns ---")
yr = yearly_returns(best_rets)
for year, ret in yr.items():
print(f" {year}: {ret*100:>+7.1f}%")
# --- IS/OOS validation ---
print(f"\n{'=' * 80}")
print("IS/OOS VALIDATION")
print(f"{'=' * 80}")
strat_best = MomentumBlendEnsembleStrategy(
top_n=10, mom_blend=best_α, risk_managed=True
)
is_rets = backtest_strategy(strat_best, start="2016-04-01", end="2022-12-31")
oos_rets = backtest_strategy(strat_best, start="2023-01-01", end="2026-05-13")
is_m = compute_metrics(is_rets)
oos_m = compute_metrics(oos_rets)
print(f" IS (2016-2022): CAGR {is_m['cagr']*100:.1f}% Sharpe {is_m['sharpe']:.2f} MaxDD {is_m['max_dd']*100:.1f}%")
print(f" OOS (2023-2026): CAGR {oos_m['cagr']*100:.1f}% Sharpe {oos_m['sharpe']:.2f} MaxDD {oos_m['max_dd']*100:.1f}%")
print(f" OOS/IS CAGR ratio: {oos_m['cagr']/is_m['cagr']:.2f}")
print(f" OOS/IS Sharpe ratio: {oos_m['sharpe']/is_m['sharpe']:.2f}")
# --- Bootstrap confidence intervals ---
print(f"\n{'=' * 80}")
print("BLOCK BOOTSTRAP (5000 resamples, block=21 days)")
print(f"{'=' * 80}")
from research.trend_rider_p0 import block_bootstrap, bootstrap_summary
boot = block_bootstrap(best_rets, n_boot=5000, block_len=21)
summary = bootstrap_summary(boot)
print(summary[["p0250", "p0500", "mean", "p0500", "p0750", "p0950"]].to_string())
print(f"\n P(Sharpe < 1.0): {(boot['sharpe'] < 1.0).mean()*100:.1f}%")
print(f" P(Sharpe < 1.5): {(boot['sharpe'] < 1.5).mean()*100:.1f}%")
print(f" P(MaxDD > 30%): {(boot['max_drawdown'].abs() > 0.30).mean()*100:.1f}%")
print(f" P(MaxDD > 25%): {(boot['max_drawdown'].abs() > 0.25).mean()*100:.1f}%")
# --- Compare with baseline (no momentum blend) ---
print(f"\n{'=' * 80}")
print("COMPARISON: Baseline (α=0) vs Best (α={best_α:.2f})")
print(f"{'=' * 80}")
base_m = results_rm[0.0]["metrics"]
print(f" Baseline: CAGR {base_m['cagr']*100:.1f}% Sharpe {base_m['sharpe']:.2f} MaxDD {base_m['max_dd']*100:.1f}%")
print(f" Best: CAGR {best_m['cagr']*100:.1f}% Sharpe {best_m['sharpe']:.2f} MaxDD {best_m['max_dd']*100:.1f}%")
print(f" Δ Sharpe: {best_m['sharpe'] - base_m['sharpe']:+.2f}")
print(f" Δ CAGR: {(best_m['cagr'] - base_m['cagr'])*100:+.1f}pp")
if __name__ == "__main__":
main()

292
research/strategy_sharpe_boost_v2.py
View File

@@ -1,292 +0,0 @@
"""
Sharpe boost v2: Dispersion-adaptive exposure + momentum blend.
Key insight: Cross-sectional stock-picking signals (recovery, momentum) only
add value when there IS meaningful cross-sectional dispersion. In low-dispersion
regimes (2021: everything moves together), the signal is noise → reduce exposure.
Approach:
1. Compute rolling cross-sectional return dispersion (std of stock returns)
2. When dispersion < historical median → scale down to partial exposure
3. Combine with momentum blend + DD dampener
This is economically justified (not curve-fitting):
- Stock-picking alpha ∝ dispersion (proven in academic literature)
- Low dispersion = herd behavior = stock selection adds no value
- High dispersion = stock differentiation = signal is informative
"""
from __future__ import annotations
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.base import Strategy
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
class DispersionAdaptiveEnsemble(Strategy):
"""
Ensemble with dispersion-adaptive exposure.
Reduces exposure when cross-sectional dispersion is low (signal uninformative).
"""
def __init__(
self,
rebal_freq: int = 21,
top_n: int = 10,
mom_blend: float = 0.25,
# Dispersion filter
disp_window: int = 21,
disp_lookback: int = 252,
disp_percentile: float = 0.40, # below this percentile → reduce
disp_floor: float = 0.50, # minimum exposure in low-disp regime
# DD dampener
dd_floor: float = 0.40,
dd_denom: float = 0.20,
risk_managed: bool = True,
):
self.rebal_freq = rebal_freq
self.top_n = top_n
self.mom_blend = mom_blend
self.disp_window = disp_window
self.disp_lookback = disp_lookback
self.disp_percentile = disp_percentile
self.disp_floor = disp_floor
self.dd_floor = dd_floor
self.dd_denom = dd_denom
self.risk_managed = risk_managed
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
p = data
ret = p.pct_change()
# === Signal A: rec_mfilt + deep_upvol ===
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
rec_mfilt_r = _rank(rec_mfilt)
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r
# === Signal B: Recovery 63d + 12-1 momentum ===
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r
# === Signal C: Pure momentum ===
signal_c = mom_r
# === Ensemble ===
α = self.mom_blend
ensemble = (1 - α) / 2 * signal_a + (1 - α) / 2 * signal_b + α * signal_c
# === Select top_n ===
rank = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# === Monthly rebalance ===
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
signals = signals.shift(1).fillna(0.0) # PIT
# === Dispersion-adaptive exposure ===
# Cross-sectional dispersion: std of stock returns each day
cs_disp = ret.std(axis=1)
# Rolling mean of dispersion
disp_smooth = cs_disp.rolling(self.disp_window, min_periods=10).mean()
# Historical percentile rank
disp_pctile = disp_smooth.rolling(
self.disp_lookback, min_periods=126
).rank(pct=True)
# Scale: 1.0 when dispersion is high, floor when low
# Linear interpolation between floor and 1.0
disp_scale = self.disp_floor + (1.0 - self.disp_floor) * (
(disp_pctile - 0.0) / (self.disp_percentile)
).clip(0.0, 1.0)
# PIT: use yesterday's dispersion estimate
disp_scale_lagged = disp_scale.shift(1).fillna(1.0)
signals = signals.mul(disp_scale_lagged, axis=0)
# === Market DD dampener ===
if self.risk_managed:
daily_rets = data.pct_change().fillna(0.0)
mkt_rets = daily_rets.mean(axis=1)
mkt_eq = (1 + mkt_rets).cumprod()
mkt_dd = mkt_eq / mkt_eq.cummax() - 1
dd_scale = (1.0 + mkt_dd / self.dd_denom).clip(
lower=self.dd_floor, upper=1.0
)
dd_scale_lagged = dd_scale.shift(1).fillna(1.0)
signals = signals.mul(dd_scale_lagged, axis=0)
return signals
# ---------------------------------------------------------------------------
# Evaluation
# ---------------------------------------------------------------------------
def compute_metrics(daily_rets: pd.Series) -> dict:
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {"cagr": cagr, "vol": vol, "sharpe": sharpe, "max_dd": max_dd, "calmar": calmar}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
_DATA_CACHE = {}
def backtest_strategy(strategy, start="2016-04-01", end="2026-05-13"):
import data_manager
if "data" not in _DATA_CACHE:
from universe import get_sp500
tickers = get_sp500()
data_manager.update("us", tickers)
_DATA_CACHE["data"] = data_manager.load("us")
data = _DATA_CACHE["data"]
if data is None:
raise RuntimeError("No data loaded")
weights = strategy.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:end]
def main():
print("=" * 80)
print("SHARPE BOOST v2: Dispersion-Adaptive Exposure")
print("=" * 80)
# --- Test 1: Dispersion filter only (no DD dampener) ---
print("\n--- Dispersion filter sweep (risk_managed=False) ---")
print(f"{'disp_pct':>8s} {'floor':>6s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>7s} {'MaxDD':>7s} {'Calmar':>7s}")
print("-" * 60)
configs = [
(0.30, 0.40),
(0.30, 0.50),
(0.40, 0.40),
(0.40, 0.50),
(0.40, 0.60),
(0.50, 0.40),
(0.50, 0.50),
(0.50, 0.60),
]
for dp, df in configs:
strat = DispersionAdaptiveEnsemble(
top_n=10, mom_blend=0.25, disp_percentile=dp,
disp_floor=df, risk_managed=False
)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(f"{dp:>8.2f} {df:>6.2f} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>7.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>7.2f}")
# --- Test 2: Dispersion filter + DD dampener ---
print("\n--- Dispersion filter + DD dampener (risk_managed=True) ---")
print(f"{'disp_pct':>8s} {'floor':>6s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>7s} {'MaxDD':>7s} {'Calmar':>7s}")
print("-" * 60)
for dp, df in configs:
strat = DispersionAdaptiveEnsemble(
top_n=10, mom_blend=0.25, disp_percentile=dp,
disp_floor=df, risk_managed=True
)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(f"{dp:>8.2f} {df:>6.2f} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>7.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>7.2f}")
# --- Test 3: Best dispersion config — yearly breakdown ---
print(f"\n{'=' * 80}")
print("BEST CONFIG: disp_pct=0.40, floor=0.50, risk_managed=True")
print(f"{'=' * 80}")
best_strat = DispersionAdaptiveEnsemble(
top_n=10, mom_blend=0.25, disp_percentile=0.40,
disp_floor=0.50, risk_managed=True
)
best_rets = backtest_strategy(best_strat)
best_m = compute_metrics(best_rets)
print(f"CAGR: {best_m['cagr']*100:.1f}% Vol: {best_m['vol']*100:.1f}% "
f"Sharpe: {best_m['sharpe']:.2f} MaxDD: {best_m['max_dd']*100:.1f}% "
f"Calmar: {best_m['calmar']:.2f}")
print("\n--- Yearly returns ---")
yr = yearly_returns(best_rets)
for year, ret in yr.items():
print(f" {year}: {ret*100:>+7.1f}%")
# --- Test 4: No filter baseline for comparison ---
print(f"\n--- Baseline (no dispersion filter, no DD) ---")
baseline = DispersionAdaptiveEnsemble(
top_n=10, mom_blend=0.25, disp_percentile=0.0,
disp_floor=1.0, risk_managed=False
)
base_rets = backtest_strategy(baseline)
base_m = compute_metrics(base_rets)
print(f"CAGR: {base_m['cagr']*100:.1f}% Vol: {base_m['vol']*100:.1f}% "
f"Sharpe: {base_m['sharpe']:.2f} MaxDD: {base_m['max_dd']*100:.1f}%")
# --- Test 5: Dispersion diagnostics for 2021 ---
print(f"\n{'=' * 80}")
print("DISPERSION DIAGNOSTIC: Is 2021 actually low dispersion?")
print(f"{'=' * 80}")
import data_manager
data = _DATA_CACHE["data"]
ret = data.pct_change()
cs_disp = ret.std(axis=1)
disp_smooth = cs_disp.rolling(21, min_periods=10).mean()
for year in range(2017, 2027):
yr_disp = disp_smooth.loc[f"{year}"]
if len(yr_disp) > 0:
print(f" {year}: avg disp = {yr_disp.mean()*100:.2f}% "
f"median = {yr_disp.median()*100:.2f}%")
if __name__ == "__main__":
main()

276
research/strategy_sharpe_boost_v3.py
View File

@@ -1,276 +0,0 @@
"""
Sharpe boost v3: Concentration + rebalance frequency + trailing alpha.
Previous findings:
- Momentum blend: Sharpe 1.34 → 1.37 (marginal)
- Dispersion filter: Sharpe 1.34 → 1.31 (worse)
- 2021 problem is NOT about dispersion or vol — it's narrow mega-cap rally
New ideas to test:
1. Higher concentration (top_n=8) → more alpha per stock if signal is good
2. Shorter rebalance (14 days) → capture alpha faster, reduce stale positions
3. Trailing alpha gate: if strategy's 63-day return < market's 63-day return
by >20pp, reduce exposure (signal currently uninformative)
4. Asymmetric vol scaling: only scale down when vol is high AND returns negative
(high vol + positive = good! don't cut that)
"""
from __future__ import annotations
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.base import Strategy
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
def compute_metrics(daily_rets: pd.Series) -> dict:
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {"cagr": cagr, "vol": vol, "sharpe": sharpe, "max_dd": max_dd, "calmar": calmar}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
class EnsembleV2(Strategy):
"""Parameterized ensemble for testing concentration / rebalance / alpha gate."""
def __init__(self, top_n=10, rebal_freq=21, mom_blend=0.0,
alpha_gate=False, alpha_gate_threshold=-0.20,
alpha_gate_window=63, alpha_gate_floor=0.50,
asym_vol=False, asym_vol_window=20, asym_vol_floor=0.50):
self.top_n = top_n
self.rebal_freq = rebal_freq
self.mom_blend = mom_blend
self.alpha_gate = alpha_gate
self.alpha_gate_threshold = alpha_gate_threshold
self.alpha_gate_window = alpha_gate_window
self.alpha_gate_floor = alpha_gate_floor
self.asym_vol = asym_vol
self.asym_vol_window = asym_vol_window
self.asym_vol_floor = asym_vol_floor
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
p = data
ret = p.pct_change()
# === Signal A: rec_mfilt + deep_upvol ===
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
rec_mfilt_r = _rank(rec_mfilt)
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r
# === Signal B: Recovery 63d + 12-1 momentum ===
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r
# === Signal C: Pure momentum ===
signal_c = mom_r
# === Ensemble ===
α = self.mom_blend
if α > 0:
ensemble = (1 - α) / 2 * signal_a + (1 - α) / 2 * signal_b + α * signal_c
else:
ensemble = 0.5 * signal_a + 0.5 * signal_b
# === Select top_n ===
rank = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# === Rebalance ===
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
signals = signals.shift(1).fillna(0.0) # PIT
# === Alpha gate: reduce when trailing alpha is very negative ===
if self.alpha_gate:
daily_rets = data.pct_change().fillna(0.0)
port_rets = (signals * daily_rets).sum(axis=1)
mkt_rets = daily_rets.mean(axis=1)
# Trailing excess return over market
trail_port = port_rets.rolling(self.alpha_gate_window, min_periods=21).sum()
trail_mkt = mkt_rets.rolling(self.alpha_gate_window, min_periods=21).sum()
excess = trail_port - trail_mkt
# When deeply underperforming → scale down
gate_active = excess < self.alpha_gate_threshold
gate_scale = pd.Series(1.0, index=data.index)
gate_scale[gate_active] = self.alpha_gate_floor
gate_scale_lagged = gate_scale.shift(1).fillna(1.0) # PIT
signals = signals.mul(gate_scale_lagged, axis=0)
# === Asymmetric vol scaling ===
if self.asym_vol:
daily_rets = data.pct_change().fillna(0.0)
port_rets = (signals * daily_rets).sum(axis=1)
short_vol = port_rets.rolling(self.asym_vol_window, min_periods=10).std() * np.sqrt(252)
vol_median = short_vol.rolling(252, min_periods=126).median()
# Only scale down when vol is high AND recent returns are negative
recent_ret = port_rets.rolling(self.asym_vol_window, min_periods=10).sum()
high_vol_neg_ret = (short_vol > vol_median * 1.5) & (recent_ret < 0)
asym_scale = pd.Series(1.0, index=data.index)
asym_scale[high_vol_neg_ret] = self.asym_vol_floor
asym_scale_lagged = asym_scale.shift(1).fillna(1.0)
signals = signals.mul(asym_scale_lagged, axis=0)
return signals
_DATA_CACHE = {}
def backtest_strategy(strategy, start="2016-04-01", end="2026-05-13"):
import data_manager
if "data" not in _DATA_CACHE:
from universe import get_sp500
tickers = get_sp500()
data_manager.update("us", tickers)
_DATA_CACHE["data"] = data_manager.load("us")
data = _DATA_CACHE["data"]
weights = strategy.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:end]
def fmt_row(label, m):
return (f"{label:<40s} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>6.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>6.2f}")
def main():
print("=" * 80)
print("SHARPE BOOST v3: Concentration / Rebalance / Alpha Gate / Asym Vol")
print("=" * 80)
header = f"{'Config':<40s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>6s} {'MaxDD':>7s} {'Calmar':>6s}"
# --- Sweep 1: Concentration (top_n) ---
print(f"\n--- Concentration sweep (rebal=21, no risk mgmt) ---")
print(header)
print("-" * 80)
for n in [6, 8, 10, 12, 15]:
strat = EnsembleV2(top_n=n, rebal_freq=21)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"top_n={n}", m))
# --- Sweep 2: Rebalance frequency ---
print(f"\n--- Rebalance frequency sweep (top_n=10) ---")
print(header)
print("-" * 80)
for freq in [5, 10, 14, 21, 42]:
strat = EnsembleV2(top_n=10, rebal_freq=freq)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"rebal={freq}d", m))
# --- Sweep 3: Momentum blend + concentration ---
print(f"\n--- Momentum blend + concentration (rebal=14) ---")
print(header)
print("-" * 80)
for n in [8, 10]:
for α in [0.0, 0.20, 0.30]:
strat = EnsembleV2(top_n=n, rebal_freq=14, mom_blend=α)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"top_n={n}, mom={α:.0%}, rebal=14", m))
# --- Sweep 4: Alpha gate ---
print(f"\n--- Alpha gate (top_n=10, rebal=21) ---")
print(header)
print("-" * 80)
for thresh in [-0.10, -0.15, -0.20]:
for floor in [0.30, 0.50]:
strat = EnsembleV2(top_n=10, rebal_freq=21, alpha_gate=True,
alpha_gate_threshold=thresh, alpha_gate_floor=floor)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"alpha_gate thresh={thresh}, floor={floor}", m))
# --- Sweep 5: Asymmetric vol ---
print(f"\n--- Asymmetric vol (top_n=10, rebal=21) ---")
print(header)
print("-" * 80)
for floor in [0.30, 0.50, 0.70]:
strat = EnsembleV2(top_n=10, rebal_freq=21, asym_vol=True, asym_vol_floor=floor)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"asym_vol floor={floor}", m))
# --- Best combo: everything together ---
print(f"\n{'=' * 80}")
print("COMBO: Best of each mechanism together")
print(f"{'=' * 80}")
print(header)
print("-" * 80)
combos = [
("top8 + rebal14 + mom20%", dict(top_n=8, rebal_freq=14, mom_blend=0.20)),
("top8 + rebal14 + mom20% + alpha_gate", dict(top_n=8, rebal_freq=14, mom_blend=0.20, alpha_gate=True, alpha_gate_threshold=-0.15, alpha_gate_floor=0.50)),
("top8 + rebal14 + mom20% + asym_vol", dict(top_n=8, rebal_freq=14, mom_blend=0.20, asym_vol=True, asym_vol_floor=0.50)),
("top8 + rebal14 + mom20% + both", dict(top_n=8, rebal_freq=14, mom_blend=0.20, alpha_gate=True, alpha_gate_threshold=-0.15, alpha_gate_floor=0.50, asym_vol=True, asym_vol_floor=0.50)),
("top10 + rebal14 + mom30%", dict(top_n=10, rebal_freq=14, mom_blend=0.30)),
("top10 + rebal14 + mom30% + alpha_gate", dict(top_n=10, rebal_freq=14, mom_blend=0.30, alpha_gate=True, alpha_gate_threshold=-0.15, alpha_gate_floor=0.50)),
]
best_sharpe = 0
best_label = ""
best_rets = None
for label, kwargs in combos:
strat = EnsembleV2(**kwargs)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(label, m))
if m["sharpe"] > best_sharpe:
best_sharpe = m["sharpe"]
best_label = label
best_rets = rets
# --- Yearly for best combo ---
print(f"\n--- Best combo: {best_label} (Sharpe={best_sharpe:.2f}) ---")
yr = yearly_returns(best_rets)
for year, ret in yr.items():
print(f" {year}: {ret*100:>+7.1f}%")
if __name__ == "__main__":
main()

278
research/strategy_sharpe_boost_v4.py
View File

@@ -1,278 +0,0 @@
"""
Sharpe boost v4: Long holding period (42d rebal) is the key lever.
Key finding from v3: rebal=42d → Sharpe 1.42 (vs 1.34 for 21d)
Why: Monthly rebal causes turnover-induced noise. Recovery/momentum signals
are slow-moving (126d lookback) so weekly/biweekly rebal is too fast.
42d rebal lets winners run.
Now test: rebal=42d + concentration + mom_blend + asym_vol + DD dampener
"""
from __future__ import annotations
import os, sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.base import Strategy
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
def compute_metrics(daily_rets: pd.Series) -> dict:
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {"cagr": cagr, "vol": vol, "sharpe": sharpe, "max_dd": max_dd, "calmar": calmar}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
class EnsembleV3(Strategy):
"""Ensemble with all levers: rebal, concentration, mom, risk mgmt."""
def __init__(self, top_n=10, rebal_freq=42, mom_blend=0.0,
asym_vol=False, asym_vol_floor=0.50,
dd_dampen=False, dd_floor=0.40, dd_denom=0.20):
self.top_n = top_n
self.rebal_freq = rebal_freq
self.mom_blend = mom_blend
self.asym_vol = asym_vol
self.asym_vol_floor = asym_vol_floor
self.dd_dampen = dd_dampen
self.dd_floor = dd_floor
self.dd_denom = dd_denom
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
p = data
ret = p.pct_change()
# === Signal A: rec_mfilt + deep_upvol ===
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
rec_mfilt_r = _rank(rec_mfilt)
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r
# === Signal B: Recovery 63d + 12-1 momentum ===
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r
# === Signal C: Pure momentum ===
signal_c = mom_r
# === Ensemble ===
α = self.mom_blend
if α > 0:
ensemble = (1 - α) / 2 * signal_a + (1 - α) / 2 * signal_b + α * signal_c
else:
ensemble = 0.5 * signal_a + 0.5 * signal_b
# === Select top_n ===
rank = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
# === Rebalance ===
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
signals = signals.shift(1).fillna(0.0) # PIT
# === Asymmetric vol: only cut in high-vol + negative return ===
if self.asym_vol:
daily_rets = data.pct_change().fillna(0.0)
port_rets = (signals * daily_rets).sum(axis=1)
short_vol = port_rets.rolling(20, min_periods=10).std() * np.sqrt(252)
vol_median = short_vol.rolling(252, min_periods=126).median()
recent_ret = port_rets.rolling(20, min_periods=10).sum()
high_vol_neg = (short_vol > vol_median * 1.5) & (recent_ret < 0)
asym_scale = pd.Series(1.0, index=data.index)
asym_scale[high_vol_neg] = self.asym_vol_floor
signals = signals.mul(asym_scale.shift(1).fillna(1.0), axis=0)
# === Market DD dampener ===
if self.dd_dampen:
daily_rets = data.pct_change().fillna(0.0)
mkt_rets = daily_rets.mean(axis=1)
mkt_eq = (1 + mkt_rets).cumprod()
mkt_dd = mkt_eq / mkt_eq.cummax() - 1
dd_scale = (1.0 + mkt_dd / self.dd_denom).clip(lower=self.dd_floor, upper=1.0)
signals = signals.mul(dd_scale.shift(1).fillna(1.0), axis=0)
return signals
_DATA_CACHE = {}
def backtest_strategy(strategy, start="2016-04-01", end="2026-05-13"):
import data_manager
if "data" not in _DATA_CACHE:
from universe import get_sp500
tickers = get_sp500()
data_manager.update("us", tickers)
_DATA_CACHE["data"] = data_manager.load("us")
data = _DATA_CACHE["data"]
weights = strategy.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:end]
def fmt_row(label, m):
return (f"{label:<50s} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>6.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>6.2f}")
def main():
print("=" * 90)
print("SHARPE BOOST v4: rebal=42d as key lever + combos")
print("=" * 90)
header = f"{'Config':<50s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>6s} {'MaxDD':>7s} {'Calmar':>6s}"
# --- rebal=42d sweep ---
print(f"\n--- rebal=42d + concentration sweep ---")
print(header)
print("-" * 90)
for n in [6, 8, 10, 12]:
strat = EnsembleV3(top_n=n, rebal_freq=42)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"rebal=42, top_n={n}", m))
# --- rebal=42d + momentum blend ---
print(f"\n--- rebal=42d + momentum blend ---")
print(header)
print("-" * 90)
for α in [0.0, 0.15, 0.20, 0.25, 0.30]:
strat = EnsembleV3(top_n=10, rebal_freq=42, mom_blend=α)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"rebal=42, top10, mom={α:.0%}", m))
# --- rebal sweep around 42d ---
print(f"\n--- rebal frequency fine-tuning (top_n=10) ---")
print(header)
print("-" * 90)
for freq in [30, 35, 42, 50, 63]:
strat = EnsembleV3(top_n=10, rebal_freq=freq)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"rebal={freq}d, top10", m))
# --- Best rebal + DD dampener ---
print(f"\n--- rebal=42d + DD dampener ---")
print(header)
print("-" * 90)
for n in [10, 12]:
for α in [0.0, 0.20]:
strat = EnsembleV3(top_n=n, rebal_freq=42, mom_blend=α, dd_dampen=True)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"rebal=42, top{n}, mom={α:.0%}, DD", m))
# --- Best rebal + asym vol ---
print(f"\n--- rebal=42d + asym_vol ---")
print(header)
print("-" * 90)
for n in [10, 12]:
strat = EnsembleV3(top_n=n, rebal_freq=42, asym_vol=True, asym_vol_floor=0.50)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"rebal=42, top{n}, asym_vol", m))
# --- Full combo ---
print(f"\n--- FULL COMBOS ---")
print(header)
print("-" * 90)
combos = [
("rebal42 + top10 + asym_vol + DD", dict(top_n=10, rebal_freq=42, asym_vol=True, dd_dampen=True)),
("rebal42 + top10 + mom20% + asym_vol + DD", dict(top_n=10, rebal_freq=42, mom_blend=0.20, asym_vol=True, dd_dampen=True)),
("rebal42 + top12 + asym_vol + DD", dict(top_n=12, rebal_freq=42, asym_vol=True, dd_dampen=True)),
("rebal42 + top12 + mom20% + asym_vol + DD", dict(top_n=12, rebal_freq=42, mom_blend=0.20, asym_vol=True, dd_dampen=True)),
("rebal63 + top10 + asym_vol + DD", dict(top_n=10, rebal_freq=63, asym_vol=True, dd_dampen=True)),
("rebal63 + top12 + asym_vol + DD", dict(top_n=12, rebal_freq=63, asym_vol=True, dd_dampen=True)),
]
best_sharpe = 0
best_label = ""
best_rets = None
for label, kwargs in combos:
strat = EnsembleV3(**kwargs)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(label, m))
if m["sharpe"] > best_sharpe:
best_sharpe = m["sharpe"]
best_label = label
best_rets = rets
# --- Best: yearly breakdown ---
print(f"\n{'=' * 90}")
print(f"BEST: {best_label} (Sharpe={best_sharpe:.2f})")
best_m = compute_metrics(best_rets)
print(f"CAGR: {best_m['cagr']*100:.1f}% Vol: {best_m['vol']*100:.1f}% "
f"Sharpe: {best_m['sharpe']:.2f} MaxDD: {best_m['max_dd']*100:.1f}% "
f"Calmar: {best_m['calmar']:.2f}")
print(f"{'=' * 90}")
yr = yearly_returns(best_rets)
for year, ret in yr.items():
print(f" {year}: {ret*100:>+7.1f}%")
# --- IS/OOS ---
print(f"\n--- IS/OOS Validation ---")
# Re-run best on IS/OOS splits
is_rets = best_rets.loc["2016-04-01":"2022-12-31"]
oos_rets = best_rets.loc["2023-01-01":"2026-05-13"]
is_m = compute_metrics(is_rets)
oos_m = compute_metrics(oos_rets)
print(f" IS (2016-2022): CAGR {is_m['cagr']*100:.1f}% Sharpe {is_m['sharpe']:.2f} MaxDD {is_m['max_dd']*100:.1f}%")
print(f" OOS (2023-2026): CAGR {oos_m['cagr']*100:.1f}% Sharpe {oos_m['sharpe']:.2f} MaxDD {oos_m['max_dd']*100:.1f}%")
# --- Bootstrap ---
print(f"\n--- Block Bootstrap (5000 samples, block=42d) ---")
from research.trend_rider_p0 import block_bootstrap
boot = block_bootstrap(best_rets, n_boot=5000, block_len=42)
print(f" Sharpe: median={boot['sharpe'].median():.2f} "
f"5th={boot['sharpe'].quantile(0.05):.2f} "
f"95th={boot['sharpe'].quantile(0.95):.2f}")
print(f" MaxDD: median={boot['max_drawdown'].median()*100:.1f}% "
f"5th={boot['max_drawdown'].quantile(0.05)*100:.1f}% "
f"95th={boot['max_drawdown'].quantile(0.95)*100:.1f}%")
print(f" P(Sharpe > 1.5): {(boot['sharpe'] > 1.5).mean()*100:.1f}%")
print(f" P(Sharpe > 1.0): {(boot['sharpe'] > 1.0).mean()*100:.1f}%")
if __name__ == "__main__":
main()

265
research/strategy_sharpe_boost_v5.py
View File

@@ -1,265 +0,0 @@
"""
Sharpe boost v5: Fine-tune DD dampener on top of the Sharpe 1.52 config.
Best raw config: rebal=42, top_n=12, asym_vol (Sharpe 1.52, MaxDD -31.2%)
Now: add a LIGHTER DD dampener to bring MaxDD under 30% without killing Sharpe.
Key: dd_denom controls how aggressively we cut. Larger denom = lighter touch.
"""
from __future__ import annotations
import os, sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from strategies.base import Strategy
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
def compute_metrics(daily_rets: pd.Series) -> dict:
eq = (1 + daily_rets).cumprod()
n_years = len(daily_rets) / 252.0
cagr = eq.iloc[-1] ** (1.0 / n_years) - 1.0
vol = daily_rets.std() * np.sqrt(252)
sharpe = daily_rets.mean() / daily_rets.std() * np.sqrt(252) if daily_rets.std() > 0 else 0
running_max = eq.cummax()
dd = eq / running_max - 1
max_dd = dd.min()
calmar = cagr / abs(max_dd) if max_dd != 0 else 0
return {"cagr": cagr, "vol": vol, "sharpe": sharpe, "max_dd": max_dd, "calmar": calmar}
def yearly_returns(daily_rets: pd.Series) -> pd.Series:
eq = (1 + daily_rets).cumprod()
yearly = eq.resample("YE").last().pct_change()
yearly.iloc[0] = eq.resample("YE").last().iloc[0] - 1
yearly.index = yearly.index.year
return yearly
class EnsembleV3(Strategy):
def __init__(self, top_n=12, rebal_freq=42, mom_blend=0.0,
asym_vol=True, asym_vol_floor=0.50,
dd_dampen=False, dd_floor=0.40, dd_denom=0.20):
self.top_n = top_n
self.rebal_freq = rebal_freq
self.mom_blend = mom_blend
self.asym_vol = asym_vol
self.asym_vol_floor = asym_vol_floor
self.dd_dampen = dd_dampen
self.dd_floor = dd_floor
self.dd_denom = dd_denom
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
p = data
ret = p.pct_change()
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
rec_mfilt_r = _rank(rec_mfilt)
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r
signal_c = mom_r
α = self.mom_blend
if α > 0:
ensemble = (1 - α) / 2 * signal_a + (1 - α) / 2 * signal_b + α * signal_c
else:
ensemble = 0.5 * signal_a + 0.5 * signal_b
rank = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_indices] = True
signals[~rebal_mask] = np.nan
signals = signals.ffill().fillna(0.0)
signals.iloc[:warmup] = 0.0
signals = signals.shift(1).fillna(0.0)
if self.asym_vol:
daily_rets = data.pct_change().fillna(0.0)
port_rets = (signals * daily_rets).sum(axis=1)
short_vol = port_rets.rolling(20, min_periods=10).std() * np.sqrt(252)
vol_median = short_vol.rolling(252, min_periods=126).median()
recent_ret = port_rets.rolling(20, min_periods=10).sum()
high_vol_neg = (short_vol > vol_median * 1.5) & (recent_ret < 0)
asym_scale = pd.Series(1.0, index=data.index)
asym_scale[high_vol_neg] = self.asym_vol_floor
signals = signals.mul(asym_scale.shift(1).fillna(1.0), axis=0)
if self.dd_dampen:
daily_rets = data.pct_change().fillna(0.0)
mkt_rets = daily_rets.mean(axis=1)
mkt_eq = (1 + mkt_rets).cumprod()
mkt_dd = mkt_eq / mkt_eq.cummax() - 1
dd_scale = (1.0 + mkt_dd / self.dd_denom).clip(lower=self.dd_floor, upper=1.0)
signals = signals.mul(dd_scale.shift(1).fillna(1.0), axis=0)
return signals
_DATA_CACHE = {}
def backtest_strategy(strategy, start="2016-04-01", end="2026-05-13"):
import data_manager
if "data" not in _DATA_CACHE:
from universe import get_sp500
tickers = get_sp500()
data_manager.update("us", tickers)
_DATA_CACHE["data"] = data_manager.load("us")
data = _DATA_CACHE["data"]
weights = strategy.generate_signals(data)
daily_rets = (weights * data.pct_change().fillna(0.0)).sum(axis=1)
return daily_rets.loc[start:end]
def fmt_row(label, m):
return (f"{label:<55s} {m['cagr']*100:>6.1f}% {m['vol']*100:>6.1f}% "
f"{m['sharpe']:>6.2f} {m['max_dd']*100:>6.1f}% {m['calmar']:>6.2f}")
def main():
print("=" * 95)
print("SHARPE BOOST v5: Fine-tune DD dampener on Sharpe 1.52 base")
print("=" * 95)
header = f"{'Config':<55s} {'CAGR':>7s} {'Vol':>7s} {'Sharpe':>6s} {'MaxDD':>7s} {'Calmar':>6s}"
# --- Baseline (no DD) ---
print(f"\n--- Baseline: rebal42 + top12 + asym_vol (no DD) ---")
print(header)
print("-" * 95)
strat = EnsembleV3(top_n=12, rebal_freq=42, asym_vol=True, dd_dampen=False)
base_rets = backtest_strategy(strat)
base_m = compute_metrics(base_rets)
print(fmt_row("NO DD (baseline)", base_m))
# --- Light DD: larger dd_denom (gentler), higher floor ---
print(f"\n--- DD dampener tuning (lighter touch) ---")
print(header)
print("-" * 95)
configs = [
# (dd_floor, dd_denom) — larger denom = need bigger crash to trigger
(0.60, 0.25),
(0.60, 0.30),
(0.60, 0.35),
(0.70, 0.25),
(0.70, 0.30),
(0.70, 0.35),
(0.50, 0.25),
(0.50, 0.30),
(0.50, 0.35),
(0.40, 0.20), # original (aggressive)
]
results = {}
for dd_floor, dd_denom in configs:
strat = EnsembleV3(top_n=12, rebal_freq=42, asym_vol=True,
dd_dampen=True, dd_floor=dd_floor, dd_denom=dd_denom)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
results[(dd_floor, dd_denom)] = {"rets": rets, "m": m}
print(fmt_row(f"DD floor={dd_floor:.2f} denom={dd_denom:.2f}", m))
# --- Also test: top_n=10 vs 12 with lighter DD ---
print(f"\n--- top_n comparison with light DD (floor=0.60, denom=0.30) ---")
print(header)
print("-" * 95)
for n in [8, 10, 12]:
strat = EnsembleV3(top_n=n, rebal_freq=42, asym_vol=True,
dd_dampen=True, dd_floor=0.60, dd_denom=0.30)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
print(fmt_row(f"top_n={n}, light DD", m))
# --- Also try: mom_blend with the good configs ---
print(f"\n--- Add momentum blend to best configs ---")
print(header)
print("-" * 95)
for α in [0.0, 0.15, 0.20]:
for dd_floor, dd_denom in [(0.60, 0.30), (0.70, 0.30)]:
strat = EnsembleV3(top_n=12, rebal_freq=42, mom_blend=α, asym_vol=True,
dd_dampen=True, dd_floor=dd_floor, dd_denom=dd_denom)
rets = backtest_strategy(strat)
m = compute_metrics(rets)
results[(dd_floor, dd_denom, α)] = {"rets": rets, "m": m}
print(fmt_row(f"top12, mom={α:.0%}, DD f={dd_floor} d={dd_denom}", m))
# --- Pick best Sharpe >= 1.5 config ---
print(f"\n{'=' * 95}")
print("SELECTING BEST CONFIG WITH Sharpe >= 1.50")
print(f"{'=' * 95}")
# Find best among all tested
best_key = None
best_sharpe = 0
for key, v in results.items():
if v["m"]["sharpe"] >= best_sharpe:
best_sharpe = v["m"]["sharpe"]
best_key = key
if best_key:
best = results[best_key]
print(f"Config: {best_key}")
print(fmt_row("BEST", best["m"]))
print(f"\n--- Yearly returns ---")
yr = yearly_returns(best["rets"])
for year, ret in yr.items():
print(f" {year}: {ret*100:>+7.1f}%")
# IS/OOS
print(f"\n--- IS/OOS ---")
is_rets = best["rets"].loc["2016-04-01":"2022-12-31"]
oos_rets = best["rets"].loc["2023-01-01":"2026-05-13"]
is_m = compute_metrics(is_rets)
oos_m = compute_metrics(oos_rets)
print(f" IS (2016-2022): CAGR {is_m['cagr']*100:.1f}% Sharpe {is_m['sharpe']:.2f} MaxDD {is_m['max_dd']*100:.1f}%")
print(f" OOS (2023-2026): CAGR {oos_m['cagr']*100:.1f}% Sharpe {oos_m['sharpe']:.2f} MaxDD {oos_m['max_dd']*100:.1f}%")
# Bootstrap
print(f"\n--- Bootstrap ---")
from research.trend_rider_p0 import block_bootstrap
boot = block_bootstrap(best["rets"], n_boot=5000, block_len=42)
print(f" Sharpe: median={boot['sharpe'].median():.2f} "
f"5th={boot['sharpe'].quantile(0.05):.2f} "
f"95th={boot['sharpe'].quantile(0.95):.2f}")
print(f" MaxDD: median={boot['max_drawdown'].median()*100:.1f}% "
f"5th={boot['max_drawdown'].quantile(0.05)*100:.1f}% "
f"95th={boot['max_drawdown'].quantile(0.95)*100:.1f}%")
print(f" P(Sharpe > 1.5): {(boot['sharpe'] > 1.5).mean()*100:.1f}%")
print(f" P(Sharpe > 1.0): {(boot['sharpe'] > 1.0).mean()*100:.1f}%")
print(f" P(MaxDD > 30%): {(boot['max_drawdown'].abs() > 0.30).mean()*100:.1f}%")
else:
print("No config achieved Sharpe >= 1.50")
# Show best anyway
best_key = max(results, key=lambda k: results[k]["m"]["sharpe"])
print(f"Closest: {best_key} → Sharpe {results[best_key]['m']['sharpe']:.2f}")
if __name__ == "__main__":
main()

145
research/strategy_sweep.py
View File

@@ -1,145 +0,0 @@
"""
Unified 3/5/10-year PIT backtest for every production strategy.
Runs the full strategy roster against the point-in-time S&P 500 price matrix
from research/pit_backtest and reports CAGR / Sharpe / Sortino / MaxDD / Calmar
for three trailing windows. Results are written to data/sweep_<years>y.csv and
printed to stdout.
Usage:
uv run python -m research.strategy_sweep
"""
import os
import pandas as pd
import research.pit_backtest as pit
from strategies.adaptive_momentum import AdaptiveMomentumStrategy
from strategies.dual_momentum import DualMomentumStrategy
from strategies.factor_combo import SIGNAL_REGISTRY, FactorComboStrategy
from strategies.inverse_vol import InverseVolatilityStrategy
from strategies.mean_reversion import MeanReversionStrategy
from strategies.momentum import MomentumStrategy
from strategies.momentum_quality import MomentumQualityStrategy
from strategies.multi_factor import MultiFactorStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.trend_following import TrendFollowingStrategy
DATA_DIR = "data"
BENCHMARK = "SPY"
def build_strategies(tickers: list[str]) -> dict:
"""Instantiate every production strategy; returns {name: strategy}."""
top_n = max(5, len(tickers) // 10)
strategies: dict = {
# --- Baselines ---
"SPY buy-and-hold": None, # handled separately
"Momentum": MomentumStrategy(lookback=252, skip=21, top_n=top_n),
"Inverse Volatility": InverseVolatilityStrategy(vol_window=20),
"Multi-Factor": MultiFactorStrategy(tickers=tickers, benchmark=BENCHMARK,
top_n=top_n),
"Mean Reversion": MeanReversionStrategy(top_n=top_n),
"Trend Following": TrendFollowingStrategy(ma_window=150, momentum_period=126,
top_n=top_n),
"Dual Momentum": DualMomentumStrategy(top_n=top_n),
"Momentum+Quality": MomentumQualityStrategy(momentum_period=252, skip=21,
top_n=top_n),
"Mom+InvVol": AdaptiveMomentumStrategy(top_n=top_n),
"Recovery+Mom Top20": RecoveryMomentumStrategy(top_n=min(20, top_n)),
"Recovery+Mom Top10": RecoveryMomentumStrategy(top_n=10),
}
# Factor-combo (monthly rebalance; biweekly is the other interesting one,
# but monthly aligns with how the RecoveryMomentum defaults are set).
for name in SIGNAL_REGISTRY:
key = f"fc_{name.replace('+', '_').replace('×', 'x')}_monthly"
strategies[key] = FactorComboStrategy(name, rebal_freq=21, top_n=10)
return strategies
def slice_years(prices: pd.DataFrame, years: int) -> pd.DataFrame:
cutoff = prices.index[-1] - pd.DateOffset(years=years)
return prices[prices.index >= cutoff]
def run_one(name: str, strat, prices: pd.DataFrame,
tickers: list[str]) -> dict:
if strat is None:
# SPY buy-and-hold
spy = prices[BENCHMARK].dropna()
eq = (spy / spy.iloc[0]) * 10_000
return {"strategy": name, **{k: v for k, v in pit.summarize(eq, name=name).items()
if k != "name"}}
# MultiFactor needs the benchmark column → pass full `prices`; others only tickers.
if isinstance(strat, MultiFactorStrategy):
strat_prices = prices # keep SPY column
else:
strat_prices = prices[tickers]
eq = pit.backtest(strategy=strat, prices=strat_prices, initial_capital=10_000,
transaction_cost=0.001)
return {"strategy": name, **{k: v for k, v in pit.summarize(eq, name=name).items()
if k != "name"}}
def fmt(row: dict) -> str:
return (f" {row['strategy']:<44s} "
f"CAGR={row['CAGR']*100:>6.1f}% "
f"Sharpe={row['Sharpe']:>5.2f} "
f"Sortino={row['Sortino']:>5.2f} "
f"MaxDD={row['MaxDD']*100:>6.1f}% "
f"Calmar={row['Calmar']:>5.2f}")
def main() -> None:
print("Loading point-in-time price data…")
raw = pit.load_pit_prices()
masked = pit.pit_universe(raw)
# Preserve SPY even though it's not in the membership intervals.
if BENCHMARK in raw.columns:
masked[BENCHMARK] = raw[BENCHMARK]
tickers = [c for c in masked.columns if c != BENCHMARK]
print(f" tickers={len(tickers)} rows={len(masked)} "
f"range={masked.index[0].date()}→{masked.index[-1].date()}")
all_results: dict[int, pd.DataFrame] = {}
for years in (10, 5, 3):
sliced = slice_years(masked, years)
strategies = build_strategies(tickers)
print("\n" + "=" * 110)
print(f"Window = last {years} years ({sliced.index[0].date()} → {sliced.index[-1].date()})")
print("=" * 110)
rows = []
for name, strat in strategies.items():
try:
rows.append(run_one(name, strat, sliced, tickers))
except Exception as exc: # noqa: BLE001
print(f" [skip] {name}: {type(exc).__name__}: {exc}")
continue
df = pd.DataFrame(rows).sort_values("Sharpe", ascending=False)
for _, r in df.iterrows():
print(fmt(r))
out = os.path.join(DATA_DIR, f"sweep_{years}y.csv")
df.to_csv(out, index=False)
all_results[years] = df
print(f" → saved {out}")
# Cross-window comparison: only strategies present in all windows.
print("\n" + "=" * 110)
print("Cross-window CAGR comparison (sorted by 10y Sharpe)")
print("=" * 110)
pivot = pd.DataFrame({
f"CAGR_{y}y": all_results[y].set_index("strategy")["CAGR"]
for y in (10, 5, 3)
})
sharpe10 = all_results[10].set_index("strategy")["Sharpe"]
pivot["Sharpe_10y"] = sharpe10
pivot = pivot.sort_values("Sharpe_10y", ascending=False)
print(pivot.to_string(formatters={
"CAGR_10y": "{:.1%}".format, "CAGR_5y": "{:.1%}".format,
"CAGR_3y": "{:.1%}".format, "Sharpe_10y": "{:.2f}".format,
}))
if __name__ == "__main__":
main()

468
research/trade_analysis.py
View File

@@ -1,468 +0,0 @@
"""
Trade-level analysis of SharpeBoostedEnsembleStrategy.
1. Extract every rebalance event: what was bought/sold and why
2. Measure holding-period return of each position
3. Attribute each trade to the signal that selected it
4. Identify effective vs ineffective trades
5. Overfitting analysis: signal decay, regime dependence, parameter sensitivity
"""
from __future__ import annotations
import os, sys
import numpy as np
import pandas as pd
from collections import defaultdict
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import data_manager
from universe import get_sp500
from strategies.base import Strategy
def _rank(df):
return df.rank(axis=1, pct=True, na_option="keep")
def main():
# --- Load data ---
tickers = get_sp500()
data_manager.update("us", tickers)
data = data_manager.load("us")
p = data
ret = p.pct_change()
# === Reproduce signals step by step (need intermediate signals for attribution) ===
rec_126 = p / p.rolling(126, min_periods=126).min() - 1
mom_filter = p.shift(21).pct_change(105)
rec_mfilt = rec_126.where(mom_filter > 0, np.nan)
rec_mfilt_r = _rank(rec_mfilt)
up_vol = ret.where(ret > 0, 0).rolling(20, min_periods=15).sum()
deep_upvol = _rank(rec_126) * _rank(up_vol)
deep_upvol_r = _rank(deep_upvol)
signal_a = 0.5 * rec_mfilt_r + 0.5 * deep_upvol_r # rec_mfilt+deep_upvol
rec_63 = p / p.rolling(63, min_periods=63).min() - 1
mom_12_1 = p.shift(21).pct_change(231)
rec_63_r = _rank(rec_63)
mom_r = _rank(mom_12_1)
signal_b = 0.5 * rec_63_r + 0.5 * mom_r # recovery63+momentum
ensemble = 0.5 * signal_a + 0.5 * signal_b
# === Generate weights (same as strategy but track rebal dates) ===
top_n = 12
rebal_freq = 42
warmup = 252
rank_df = ensemble.rank(axis=1, ascending=False, na_option="bottom")
n_valid = ensemble.notna().sum(axis=1)
enough = n_valid >= top_n
top_mask = (rank_df <= top_n) & enough.values.reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0, np.nan)
signals = raw.div(row_sums, axis=0).fillna(0.0)
rebal_mask = pd.Series(False, index=data.index)
rebal_indices = list(range(warmup, len(data), rebal_freq))
rebal_mask.iloc[rebal_indices] = True
rebal_dates = data.index[rebal_mask]
signals_rebal = signals.copy()
signals_rebal[~rebal_mask] = np.nan
signals_rebal = signals_rebal.ffill().fillna(0.0)
signals_rebal.iloc[:warmup] = 0.0
weights = signals_rebal.shift(1).fillna(0.0) # PIT
# Trim to eval period
eval_start = "2016-04-01"
eval_end = "2026-05-13"
rebal_dates = rebal_dates[(rebal_dates >= eval_start) & (rebal_dates <= eval_end)]
print("=" * 100)
print("TRADE-LEVEL ANALYSIS: SharpeBoostedEnsembleStrategy (10 years)")
print("=" * 100)
print(f"Total rebalance events: {len(rebal_dates)}")
print(f"Rebalance frequency: every {rebal_freq} trading days (~2 months)")
print(f"Positions per rebalance: {top_n}")
print()
# === Track each rebalance: positions entered, exited, held ===
all_trades = [] # list of dicts
prev_holdings = set()
for i, rebal_date in enumerate(rebal_dates):
# Portfolio at this rebalance
row = signals.loc[rebal_date]
current_holdings = set(row[row > 0].index)
entered = current_holdings - prev_holdings
exited = prev_holdings - current_holdings
held = current_holdings & prev_holdings
# Next rebal date (or end of data)
if i + 1 < len(rebal_dates):
next_rebal = rebal_dates[i + 1]
else:
next_rebal = data.index[data.index <= eval_end][-1]
# Holding period return for each position
for ticker in current_holdings:
try:
entry_price = p.loc[rebal_date, ticker]
exit_price = p.loc[next_rebal, ticker]
if pd.notna(entry_price) and pd.notna(exit_price) and entry_price > 0:
hpr = exit_price / entry_price - 1
else:
hpr = np.nan
except (KeyError, IndexError):
hpr = np.nan
# Signal attribution
sa = signal_a.loc[rebal_date, ticker] if ticker in signal_a.columns else np.nan
sb = signal_b.loc[rebal_date, ticker] if ticker in signal_b.columns else np.nan
ens = ensemble.loc[rebal_date, ticker] if ticker in ensemble.columns else np.nan
rnk = rank_df.loc[rebal_date, ticker] if ticker in rank_df.columns else np.nan
# Raw signal components
rec126_val = rec_126.loc[rebal_date, ticker] if ticker in rec_126.columns else np.nan
rec63_val = rec_63.loc[rebal_date, ticker] if ticker in rec_63.columns else np.nan
mom_val = mom_12_1.loc[rebal_date, ticker] if ticker in mom_12_1.columns else np.nan
action = "ENTER" if ticker in entered else ("HOLD" if ticker in held else "???")
all_trades.append({
"rebal_date": rebal_date,
"next_rebal": next_rebal,
"ticker": ticker,
"action": action,
"hpr": hpr,
"signal_a": sa,
"signal_b": sb,
"ensemble": ens,
"rank": rnk,
"rec_126d": rec126_val,
"rec_63d": rec63_val,
"mom_12_1": mom_val,
"holding_days": (next_rebal - rebal_date).days,
})
prev_holdings = current_holdings
trades_df = pd.DataFrame(all_trades)
trades_df = trades_df.dropna(subset=["hpr"])
# === Summary statistics ===
print("=" * 100)
print("OVERALL TRADE STATISTICS")
print("=" * 100)
n_total = len(trades_df)
n_win = (trades_df["hpr"] > 0).sum()
n_lose = (trades_df["hpr"] <= 0).sum()
print(f"Total position-rebalances: {n_total}")
print(f"Win rate: {n_win}/{n_total} = {n_win/n_total*100:.1f}%")
print(f"Average HPR: {trades_df['hpr'].mean()*100:.2f}%")
print(f"Median HPR: {trades_df['hpr'].median()*100:.2f}%")
print(f"Avg winning trade: {trades_df.loc[trades_df['hpr']>0, 'hpr'].mean()*100:.2f}%")
print(f"Avg losing trade: {trades_df.loc[trades_df['hpr']<=0, 'hpr'].mean()*100:.2f}%")
print(f"Best trade: {trades_df['hpr'].max()*100:.1f}% ({trades_df.loc[trades_df['hpr'].idxmax(), 'ticker']} "
f"on {trades_df.loc[trades_df['hpr'].idxmax(), 'rebal_date'].strftime('%Y-%m-%d')})")
print(f"Worst trade: {trades_df['hpr'].min()*100:.1f}% ({trades_df.loc[trades_df['hpr'].idxmin(), 'ticker']} "
f"on {trades_df.loc[trades_df['hpr'].idxmin(), 'rebal_date'].strftime('%Y-%m-%d')})")
print()
# === ENTER vs HOLD comparison ===
print("--- New entries (ENTER) vs Continued holds (HOLD) ---")
for action in ["ENTER", "HOLD"]:
sub = trades_df[trades_df["action"] == action]
if len(sub) > 0:
print(f" {action}: n={len(sub)}, win_rate={((sub['hpr']>0).mean())*100:.1f}%, "
f"avg_hpr={sub['hpr'].mean()*100:.2f}%, median={sub['hpr'].median()*100:.2f}%")
print()
# === Turnover analysis ===
print("--- Turnover per rebalance ---")
turnover_data = []
prev_set = set()
for rd in rebal_dates:
row = signals.loc[rd]
cur_set = set(row[row > 0].index)
if prev_set:
n_new = len(cur_set - prev_set)
n_exit = len(prev_set - cur_set)
n_hold = len(cur_set & prev_set)
turnover_data.append({
"date": rd, "new": n_new, "exit": n_exit, "held": n_hold,
"turnover_pct": (n_new + n_exit) / (2 * top_n) * 100
})
prev_set = cur_set
turn_df = pd.DataFrame(turnover_data)
print(f" Avg stocks replaced per rebal: {turn_df['new'].mean():.1f} / {top_n}")
print(f" Avg turnover: {turn_df['turnover_pct'].mean():.1f}%")
print(f" Median turnover: {turn_df['turnover_pct'].median():.1f}%")
print(f" Min/Max turnover: {turn_df['turnover_pct'].min():.0f}% / {turn_df['turnover_pct'].max():.0f}%")
print()
# === Yearly breakdown ===
print("=" * 100)
print("YEARLY TRADE ANALYSIS")
print("=" * 100)
trades_df["year"] = trades_df["rebal_date"].dt.year
for year in sorted(trades_df["year"].unique()):
yr = trades_df[trades_df["year"] == year]
n = len(yr)
wr = (yr["hpr"] > 0).mean() * 100
avg = yr["hpr"].mean() * 100
med = yr["hpr"].median() * 100
# Count unique tickers
n_tickers = yr["ticker"].nunique()
# Top winners
top3 = yr.nlargest(3, "hpr")[["ticker", "hpr", "rebal_date"]].values
# Worst 3
bot3 = yr.nsmallest(3, "hpr")[["ticker", "hpr", "rebal_date"]].values
print(f"\n {year}: {n} positions, {n_tickers} unique stocks, "
f"WR={wr:.0f}%, avg={avg:+.1f}%, median={med:+.1f}%")
print(f" Top 3: ", end="")
for t, h, d in top3:
print(f"{t} {h*100:+.1f}%({d.strftime('%m/%d')})", end=" ")
print(f"\n Bot 3: ", end="")
for t, h, d in bot3:
print(f"{t} {h*100:+.1f}%({d.strftime('%m/%d')})", end=" ")
print()
# === Effective vs Ineffective trades ===
print("\n" + "=" * 100)
print("EFFECTIVE vs INEFFECTIVE TRADE ANALYSIS")
print("=" * 100)
# Market benchmark: SPY return over same holding period
spy = data["SPY"]
trades_df["spy_hpr"] = trades_df.apply(
lambda r: spy.loc[r["next_rebal"]] / spy.loc[r["rebal_date"]] - 1
if r["rebal_date"] in spy.index and r["next_rebal"] in spy.index
else np.nan, axis=1
)
trades_df["excess"] = trades_df["hpr"] - trades_df["spy_hpr"]
n_beat = (trades_df["excess"] > 0).sum()
n_lag = (trades_df["excess"] <= 0).sum()
print(f"Positions beating SPY: {n_beat}/{n_total} = {n_beat/n_total*100:.1f}%")
print(f"Avg excess return: {trades_df['excess'].mean()*100:.2f}%")
print(f"Median excess return: {trades_df['excess'].median()*100:.2f}%")
print()
# Categorize trades
trades_df["category"] = "neutral"
# Effective: made money AND beat SPY
trades_df.loc[(trades_df["hpr"] > 0) & (trades_df["excess"] > 0), "category"] = "effective"
# Effective loss: lost money but lost less than SPY (good stock picking in downturn)
trades_df.loc[(trades_df["hpr"] <= 0) & (trades_df["excess"] > 0), "category"] = "effective_loss"
# Ineffective: made money but lagged SPY (would have been better in index)
trades_df.loc[(trades_df["hpr"] > 0) & (trades_df["excess"] <= 0), "category"] = "ineffective_gain"
# Ineffective: lost money AND lagged SPY
trades_df.loc[(trades_df["hpr"] <= 0) & (trades_df["excess"] <= 0), "category"] = "ineffective"
print("--- Trade Categories ---")
for cat, desc in [
("effective", "Won + beat SPY (good pick, right market)"),
("effective_loss", "Lost but beat SPY (good pick, bad market)"),
("ineffective_gain", "Won but lagged SPY (worse than index)"),
("ineffective", "Lost + lagged SPY (bad pick)"),
]:
sub = trades_df[trades_df["category"] == cat]
n = len(sub)
pct = n / n_total * 100
avg_hpr = sub["hpr"].mean() * 100 if n > 0 else 0
avg_exc = sub["excess"].mean() * 100 if n > 0 else 0
print(f" {cat:<20s}: {n:>4d} ({pct:>5.1f}%) avg HPR={avg_hpr:>+6.2f}% excess={avg_exc:>+6.2f}%")
# === Yearly effective rate ===
print("\n--- Yearly effectiveness ---")
print(f" {'Year':>4s} {'effective':>10s} {'eff_loss':>10s} {'ineff_gain':>10s} {'ineff':>10s} {'alpha':>8s}")
for year in sorted(trades_df["year"].unique()):
yr = trades_df[trades_df["year"] == year]
cats = yr["category"].value_counts()
eff = cats.get("effective", 0) + cats.get("effective_loss", 0)
ineff = cats.get("ineffective", 0) + cats.get("ineffective_gain", 0)
alpha = yr["excess"].mean() * 100
print(f" {year:>4d} {cats.get('effective', 0):>10d} {cats.get('effective_loss', 0):>10d} "
f"{cats.get('ineffective_gain', 0):>10d} {cats.get('ineffective', 0):>10d} {alpha:>+7.2f}%")
# === Signal attribution: which signal drives winners? ===
print("\n" + "=" * 100)
print("SIGNAL ATTRIBUTION")
print("=" * 100)
print("Which signal component drove winning vs losing trades?")
# For each trade, determine if signal_a or signal_b contributed more
trades_df["dominant_signal"] = np.where(
trades_df["signal_a"] > trades_df["signal_b"], "A (rec_mfilt+upvol)", "B (rec63+mom)"
)
for sig_name in ["A (rec_mfilt+upvol)", "B (rec63+mom)"]:
sub = trades_df[trades_df["dominant_signal"] == sig_name]
n = len(sub)
wr = (sub["hpr"] > 0).mean() * 100
avg = sub["hpr"].mean() * 100
exc = sub["excess"].mean() * 100
print(f" Signal {sig_name}: n={n}, WR={wr:.0f}%, avg_hpr={avg:+.1f}%, avg_excess={exc:+.1f}%")
# === PIT audit: what information was available at each trade ===
print("\n" + "=" * 100)
print("PIT (POINT-IN-TIME) AUDIT")
print("=" * 100)
print("""
Signal construction timeline (what's known at rebalance date T):
- rec_126d: price[T] / min(price[T-126:T]) - 1
→ Uses current price and 126-day trailing window. Available at T. ✓
- mom_filter: price[T-21].pct_change(105) = (P[T-21] - P[T-126]) / P[T-126]
→ Uses price 21 days ago vs 126 days ago. Both available at T. ✓
→ The shift(21) avoids short-term reversal contamination.
- deep_upvol: rank(rec_126) × rank(up_vol_20d)
→ up_vol uses 20-day trailing sum of positive returns. Available at T. ✓
- rec_63d: price[T] / min(price[T-63:T]) - 1. Available at T. ✓
- mom_12_1: price[T-21].pct_change(231) = (P[T-21] - P[T-252]) / P[T-252]
→ Classic 12-1 month momentum. shift(21) ensures no current-month data. ✓
Execution timeline:
- Signals computed at close of day T
- weights = signals.shift(1) → trade at OPEN of day T+1
- This is conservative (most backtests assume same-day execution)
Risk overlay PIT:
- asym_vol: uses 20-day vol and returns of portfolio, .shift(1) → yesterday's data ✓
- dd_dampen: uses market equity curve drawdown, .shift(1) → yesterday's data ✓
VERDICT: All signals are strictly PIT-compliant. No look-ahead bias.
""")
# === Overfitting analysis ===
print("=" * 100)
print("OVERFITTING RISK ANALYSIS")
print("=" * 100)
# 1. Signal decay: does the signal predict well in early vs late years?
print("\n--- 1. Signal Predictive Power Over Time ---")
print(" IC (rank correlation between ensemble signal and forward return)")
for year in sorted(trades_df["year"].unique()):
yr = trades_df[trades_df["year"] == year]
if len(yr) > 10:
ic = yr["ensemble"].corr(yr["hpr"], method="spearman")
print(f" {year}: IC = {ic:+.3f} (n={len(yr)})")
# 2. Concentration in specific stocks
print("\n--- 2. Stock concentration ---")
top_stocks = trades_df.groupby("ticker").agg(
n=("hpr", "count"),
avg_hpr=("hpr", "mean"),
total_hpr=("hpr", "sum"),
first_seen=("rebal_date", "min"),
last_seen=("rebal_date", "max"),
).sort_values("total_hpr", ascending=False)
print(" Top 15 most held stocks (by total return contribution):")
print(f" {'Ticker':<8s} {'Times':>5s} {'Avg HPR':>8s} {'Total':>8s} {'First':>12s} {'Last':>12s}")
for ticker, row in top_stocks.head(15).iterrows():
print(f" {ticker:<8s} {row['n']:>5.0f} {row['avg_hpr']*100:>+7.1f}% "
f"{row['total_hpr']*100:>+7.1f}% {row['first_seen'].strftime('%Y-%m'):>12s} "
f"{row['last_seen'].strftime('%Y-%m'):>12s}")
print(f"\n Total unique stocks traded: {trades_df['ticker'].nunique()}")
print(f" Top 15 stocks contribute: {top_stocks.head(15)['total_hpr'].sum()*100:.0f}% "
f"of total {top_stocks['total_hpr'].sum()*100:.0f}% cumulative HPR")
# 3. Is alpha concentrated in specific market regimes?
print("\n--- 3. Regime dependence ---")
# Compute market return for each holding period
trades_df["mkt_regime"] = pd.cut(
trades_df["spy_hpr"],
bins=[-1, -0.05, 0.0, 0.05, 0.10, 1],
labels=["crash(<-5%)", "down(0~-5%)", "flat(0~5%)", "up(5~10%)", "rally(>10%)"]
)
print(" Alpha by market regime:")
for regime in ["crash(<-5%)", "down(0~-5%)", "flat(0~5%)", "up(5~10%)", "rally(>10%)"]:
sub = trades_df[trades_df["mkt_regime"] == regime]
if len(sub) > 0:
print(f" {regime:<16s}: n={len(sub):>4d}, avg_excess={sub['excess'].mean()*100:>+6.2f}%, "
f"WR_vs_SPY={(sub['excess']>0).mean()*100:>5.1f}%")
# 4. Parameter sensitivity (rebal frequency)
print("\n--- 4. Parameter sensitivity: rebalance frequency ---")
print(" (From v4 sweep results)")
print(" rebal=30d: Sharpe 1.33 | rebal=35d: Sharpe 1.42")
print(" rebal=42d: Sharpe 1.42 | rebal=50d: Sharpe 1.40")
print(" rebal=63d: Sharpe 1.32")
print(" → Broad plateau from 35-50d. Not sitting on a cliff. ✓")
print("\n Parameter sensitivity: top_n")
print(" top_n=8: Sharpe 1.43 | top_n=10: Sharpe 1.42")
print(" top_n=12: Sharpe 1.44 | top_n=15: Sharpe 1.32 (drops off)")
print(" → Broad plateau from 8-12. Not sitting on a cliff. ✓")
print("\n Parameter sensitivity: DD dampener")
print(" dd_denom=0.25: Sharpe 1.51 | dd_denom=0.30: Sharpe 1.51")
print(" dd_denom=0.35: Sharpe 1.52 | dd_floor 0.5-0.7: all Sharpe 1.50-1.52")
print(" → Very flat surface. Not overfit. ✓")
# 5. Overfitting risk summary
print("\n" + "=" * 100)
print("OVERFITTING RISK SUMMARY FOR NEXT 10 YEARS")
print("=" * 100)
print("""
RISKS (what could go wrong):
1. ALPHA SOURCE DECAY: Recovery+momentum signals have been documented in
academic literature since the 1990s. If more capital chases these signals,
alpha erodes. However, the recovery signal is relatively niche (most quants
use pure momentum, not recovery-from-bottom).
RISK: MEDIUM
2. REGIME CHANGE: If the market enters a prolonged low-volatility sideways
period (like Japan 1990-2010), recovery signals produce no alpha because
there are no drawdowns to recover from. 2021 was a mild version of this.
RISK: MEDIUM
3. CONCENTRATION RISK: top_n=12 means ~2.4% of S&P 500. Single-stock events
(fraud, regulatory action) can cause -30% in a day for 8% of the portfolio.
This is structural and won't improve.
RISK: HIGH (but accepted for higher alpha)
4. SURVIVORSHIP BIAS: We use current S&P 500 constituents back to 2016.
Stocks that were removed (bankrupt/delisted) are not in our backtest.
This flatters results, especially for the recovery signal which would
have selected some of these troubled stocks.
RISK: MEDIUM (partially mitigated by the momentum filter)
MITIGANTS (why it's not pure overfitting):
1. FEW PARAMETERS: Only 4 meaningful degrees of freedom (rebal_freq, top_n,
asym_vol_floor, dd_denom). Hard to overfit with so few knobs.
2. ECONOMIC LOGIC: Every signal has a clear economic story:
- Recovery from bottom → mean reversion after forced selling
- Momentum → behavioral underreaction to positive news
- Asymmetric vol → panic selling is temporary, don't exit good positions
- DD dampener → systemic risk warrants de-risking
3. PARAMETER INSENSITIVITY: Adjacent parameter values produce similar results
(no cliff edges). This is the #1 sign of a robust strategy.
4. OOS PERFORMANCE: IS (2016-2022) Sharpe 1.05, OOS (2023-2026) Sharpe 2.24.
OOS is BETTER than IS — the opposite of overfitting. Though this may
partly reflect the strong 2023-2025 bull market.
HONEST ASSESSMENT:
- Expected Sharpe in next 10 years: 0.8-1.2 (below backtest's 1.52)
- Haircut reasons: transaction costs in practice, alpha decay, survivorship bias
- The strategy IS real (economically grounded, few parameters, OOS holds up)
- But backtest Sharpe is always optimistic — expect 60-75% of backtest performance
""")
if __name__ == "__main__":
main()

419
research/trend_rider_p0.py
View File

@@ -1,419 +0,0 @@
"""P0 robustness validation for TrendRiderV3.
P0.1 Walk-forward / OOS split — IS = 2015-2020, OOS = 2021-2026-05.
Optimize parameters on IS by CAGR, evaluate the IS-best config on OOS,
then compare to the default config evaluated on the same windows.
P0.2 Block bootstrap on daily returns (block_len=21, n_boot=5000) to compute
CIs for CAGR / Sharpe / MaxDD / Calmar / FinalMultiple.
P0.3 De-leveraged comparison — replace risk_on=(TQQQ, UPRO) with (SPY, QQQ)
to isolate timing edge from leverage edge. Compare to SPY/QQQ B&H.
Run:
uv run python -m research.trend_rider_p0
"""
from __future__ import annotations
import argparse
import os
import sys
from dataclasses import asdict
from itertools import product
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.trend_rider_robustness import (
Evaluation,
buy_hold_weights,
evaluate_strategy,
evaluate_weights,
load_price_panel,
portfolio_returns,
)
from strategies.permanent import TrendRiderV3
IS_START = "2015-01-02"
IS_END = "2020-12-31"
OOS_START = "2021-01-01"
OOS_END = "2026-05-07"
def _fmt_pct(x: float) -> str:
return f"{x * 100:7.2f}%"
def _print_eval(label: str, ev: Evaluation) -> None:
print(
f" {label:<24s} "
f"CAGR {_fmt_pct(ev.cagr)} "
f"Sharpe {ev.sharpe:5.2f} "
f"MDD {_fmt_pct(ev.max_drawdown)} "
f"Calmar {ev.calmar:5.2f} "
f"FinalX {ev.final_multiple:6.2f} "
f"Switches {ev.switches:4d}"
)
# ---------------------------------------------------------------------------
# P0.1 — Walk-forward / OOS
# ---------------------------------------------------------------------------
def is_oos_grid() -> list[dict]:
"""Slightly larger sweep than default to expose IS-optimal corners."""
return [
{
"vol_enter": ve,
"vol_exit": vx,
"dd_stop": dd,
"peak_enter": pe,
"mom_lookback": mom,
"regime_min_hold": mh,
"stop_loss_pct": sl,
}
for ve, vx, dd, pe, mom, mh, sl in product(
[0.12, 0.14, 0.16],
[0.20],
[0.04, 0.05, 0.07],
[0.01, 0.02, 0.03],
[42, 63, 84],
[10, 15, 20],
[0.10, 0.15, 0.20],
)
]
def walk_forward(prices: pd.DataFrame, transaction_cost: float = 0.001) -> dict:
"""Optimize on IS, evaluate IS-best on OOS, compare to defaults."""
grid = is_oos_grid()
is_rows = []
for kwargs in grid:
strat = TrendRiderV3(**kwargs)
weights = strat.generate_signals(prices)
ev = evaluate_weights(
"is",
weights,
prices[weights.columns],
transaction_cost=transaction_cost,
start=IS_START,
end=IS_END,
)
row = asdict(ev)
row.update(kwargs)
is_rows.append(row)
is_df = pd.DataFrame(is_rows).sort_values("cagr", ascending=False).reset_index(drop=True)
is_top = is_df.iloc[0]
is_best_kwargs = {k: is_top[k] for k in grid[0].keys()}
# Cast numeric grid values to native types
is_best_kwargs = {
k: (int(v) if isinstance(v, (int, np.integer)) else float(v))
for k, v in is_best_kwargs.items()
}
# mom_lookback / regime_min_hold are ints
for k in ("mom_lookback", "regime_min_hold"):
is_best_kwargs[k] = int(is_best_kwargs[k])
# OOS evaluation of IS-best
strat_isbest = TrendRiderV3(**is_best_kwargs)
w_isbest = strat_isbest.generate_signals(prices)
isbest_oos = evaluate_weights(
"is_best_OOS",
w_isbest,
prices[w_isbest.columns],
transaction_cost=transaction_cost,
start=OOS_START,
end=OOS_END,
)
# Defaults on IS and OOS
default = TrendRiderV3()
w_def = default.generate_signals(prices)
def_is = evaluate_weights(
"default_IS",
w_def,
prices[w_def.columns],
transaction_cost=transaction_cost,
start=IS_START,
end=IS_END,
)
def_oos = evaluate_weights(
"default_OOS",
w_def,
prices[w_def.columns],
transaction_cost=transaction_cost,
start=OOS_START,
end=OOS_END,
)
# SPY B&H benchmark on each window
spy_w = buy_hold_weights(prices, "SPY")
qqq_w = buy_hold_weights(prices, "QQQ")
spy_is = evaluate_weights("spy_IS", spy_w, prices[spy_w.columns], 0.0, IS_START, IS_END)
spy_oos = evaluate_weights("spy_OOS", spy_w, prices[spy_w.columns], 0.0, OOS_START, OOS_END)
qqq_is = evaluate_weights("qqq_IS", qqq_w, prices[qqq_w.columns], 0.0, IS_START, IS_END)
qqq_oos = evaluate_weights("qqq_OOS", qqq_w, prices[qqq_w.columns], 0.0, OOS_START, OOS_END)
# Decay metric: how much CAGR fell from IS-fitted to OOS
return {
"is_grid": is_df,
"is_best_kwargs": is_best_kwargs,
"is_best_IS_cagr": float(is_top["cagr"]),
"is_best_OOS": isbest_oos,
"default_IS": def_is,
"default_OOS": def_oos,
"spy_IS": spy_is,
"spy_OOS": spy_oos,
"qqq_IS": qqq_is,
"qqq_OOS": qqq_oos,
}
# ---------------------------------------------------------------------------
# P0.2 — Block bootstrap on daily returns
# ---------------------------------------------------------------------------
def block_bootstrap(
returns: pd.Series,
n_boot: int = 5000,
block_len: int = 21,
seed: int = 42,
) -> pd.DataFrame:
"""Stationary block bootstrap on daily returns.
Resamples with replacement in fixed-length blocks to preserve short-horizon
autocorrelation / volatility clustering. Returns a DataFrame with columns
[cagr, sharpe, max_drawdown, calmar, final_multiple] of length n_boot.
"""
r = returns.values
n = len(r)
rng = np.random.default_rng(seed)
n_blocks = int(np.ceil(n / block_len))
# Pre-allocate
cagrs = np.empty(n_boot)
sharpes = np.empty(n_boot)
mdds = np.empty(n_boot)
finals = np.empty(n_boot)
span_years = n / 252.0
for b in range(n_boot):
starts = rng.integers(0, n - block_len + 1, size=n_blocks)
idx = (starts[:, None] + np.arange(block_len)[None, :]).ravel()[:n]
sample = r[idx]
equity = np.cumprod(1.0 + sample)
finals[b] = equity[-1]
cagrs[b] = equity[-1] ** (1.0 / span_years) - 1.0
std = sample.std(ddof=1)
sharpes[b] = (sample.mean() / std * np.sqrt(252)) if std > 0 else 0.0
running_max = np.maximum.accumulate(equity)
mdds[b] = float(np.min(equity / running_max - 1.0))
df = pd.DataFrame({
"cagr": cagrs,
"sharpe": sharpes,
"max_drawdown": mdds,
"final_multiple": finals,
})
df["calmar"] = df["cagr"] / df["max_drawdown"].abs().replace(0.0, np.nan)
return df
def bootstrap_summary(boot: pd.DataFrame) -> pd.DataFrame:
qs = [0.025, 0.05, 0.25, 0.50, 0.75, 0.95, 0.975]
summary = boot.quantile(qs).T
summary.columns = [f"p{int(q * 1000):04d}" for q in qs]
summary["mean"] = boot.mean()
summary["std"] = boot.std(ddof=1)
summary["prob_neg_cagr"] = np.nan
summary["prob_below_spy"] = np.nan
return summary
# ---------------------------------------------------------------------------
# P0.3 — De-leveraged comparison
# ---------------------------------------------------------------------------
def deleveraged_evaluations(
prices: pd.DataFrame, transaction_cost: float = 0.001
) -> dict[str, Evaluation]:
out: dict[str, Evaluation] = {}
# Standard (leveraged)
levered = TrendRiderV3()
w_lev = levered.generate_signals(prices)
out["TR_v3_leveraged"] = evaluate_weights(
"TR_v3_leveraged",
w_lev,
prices[w_lev.columns],
transaction_cost=transaction_cost,
start=IS_START,
end=OOS_END,
)
# No leverage on equity (risk_on = SPY/QQQ), commodity risk_off
nolev = TrendRiderV3(risk_on=("SPY", "QQQ"))
w_nl = nolev.generate_signals(prices)
out["TR_v3_nolev_SPYQQQ"] = evaluate_weights(
"TR_v3_nolev_SPYQQQ",
w_nl,
prices[w_nl.columns],
transaction_cost=transaction_cost,
start=IS_START,
end=OOS_END,
)
# No leverage AND cash-only risk_off (most conservative — pure timing edge on equity)
nolev_shy = TrendRiderV3(risk_on=("SPY", "QQQ"), risk_off=("SHY",))
w_nl_shy = nolev_shy.generate_signals(prices)
out["TR_v3_nolev_SHYoff"] = evaluate_weights(
"TR_v3_nolev_SHYoff",
w_nl_shy,
prices[w_nl_shy.columns],
transaction_cost=transaction_cost,
start=IS_START,
end=OOS_END,
)
# Buy-and-hold benchmarks
spy_w = buy_hold_weights(prices, "SPY")
qqq_w = buy_hold_weights(prices, "QQQ")
out["SPY_BH"] = evaluate_weights("SPY_BH", spy_w, prices[spy_w.columns], 0.0, IS_START, OOS_END)
out["QQQ_BH"] = evaluate_weights("QQQ_BH", qqq_w, prices[qqq_w.columns], 0.0, IS_START, OOS_END)
# 50/50 SPY+QQQ rebalanced (passive, no timing) — fairer "equity passive" benchmark
cols = [c for c in ["SPY", "QQQ"] if c in prices.columns]
if len(cols) == 2:
eq_w = pd.DataFrame(0.5, index=prices.index, columns=cols)
out["SPY_QQQ_5050"] = evaluate_weights(
"SPY_QQQ_5050", eq_w, prices[cols], 0.0, IS_START, OOS_END
)
return out
# ---------------------------------------------------------------------------
# main
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(description="P0 validation suite for TrendRiderV3")
parser.add_argument("--n-boot", type=int, default=5000)
parser.add_argument("--block-len", type=int, default=21)
parser.add_argument("--transaction-cost", type=float, default=0.001)
parser.add_argument("--out-dir", default="data")
args = parser.parse_args()
os.makedirs(args.out_dir, exist_ok=True)
prices = load_price_panel()
print(f"Panel: {prices.index.min().date()} to {prices.index.max().date()}, "
f"{prices.shape[1]} columns")
# ---------- P0.1 ----------
print("\n" + "=" * 78)
print("P0.1 Walk-forward / Out-of-sample")
print(f" IS = {IS_START} → {IS_END}")
print(f" OOS = {OOS_START} → {OOS_END}")
print("=" * 78)
wf = walk_forward(prices, transaction_cost=args.transaction_cost)
is_grid = wf["is_grid"]
is_grid.to_csv(os.path.join(args.out_dir, "p0_walkforward_isgrid.csv"), index=False)
print(f"\nGrid size: {len(is_grid)} | top 3 by IS CAGR:")
cols_show = ["cagr", "sharpe", "max_drawdown", "vol_enter", "dd_stop", "peak_enter",
"mom_lookback", "regime_min_hold", "stop_loss_pct"]
print(is_grid[cols_show].head(3).to_string(index=False))
print(f"\nIS-best params: {wf['is_best_kwargs']}")
print(f" IS CAGR : {_fmt_pct(wf['is_best_IS_cagr'])}")
print(f" OOS perf of IS-best params:")
_print_eval("IS-best (OOS)", wf["is_best_OOS"])
_print_eval("Default (IS)", wf["default_IS"])
_print_eval("Default (OOS)", wf["default_OOS"])
_print_eval("SPY B&H (IS)", wf["spy_IS"])
_print_eval("SPY B&H (OOS)", wf["spy_OOS"])
_print_eval("QQQ B&H (IS)", wf["qqq_IS"])
_print_eval("QQQ B&H (OOS)", wf["qqq_OOS"])
decay = wf["is_best_IS_cagr"] - wf["is_best_OOS"].cagr
print(f"\n Performance decay (IS→OOS) of IS-best : {_fmt_pct(decay)}")
decay_def = wf["default_IS"].cagr - wf["default_OOS"].cagr
print(f" Performance decay (IS→OOS) of default : {_fmt_pct(decay_def)}")
# ---------- P0.2 ----------
print("\n" + "=" * 78)
print("P0.2 Block bootstrap (block_len="
f"{args.block_len}, n_boot={args.n_boot})")
print("=" * 78)
default = TrendRiderV3()
weights = default.generate_signals(prices)
rets = portfolio_returns(weights, prices[weights.columns],
transaction_cost=args.transaction_cost)
rets = rets[(rets.index >= IS_START) & (rets.index <= OOS_END)]
print(f" Returns series : {len(rets)} days, "
f"mean {rets.mean()*252:.4f}, vol {rets.std(ddof=1)*np.sqrt(252):.4f}")
boot_full = block_bootstrap(
rets, n_boot=args.n_boot, block_len=args.block_len, seed=42
)
boot_full.to_csv(os.path.join(args.out_dir, "p0_bootstrap_full.csv"), index=False)
print("\nFull-sample bootstrap (2015-2026):")
print(bootstrap_summary(boot_full).round(4).to_string())
# Probability statements
spy_oos_cagr = wf["spy_OOS"].cagr
p_below_spy = float((boot_full["cagr"] < spy_oos_cagr).mean())
p_neg = float((boot_full["cagr"] < 0).mean())
p_dd_50 = float((boot_full["max_drawdown"] < -0.50).mean())
p_sharpe_below_05 = float((boot_full["sharpe"] < 0.5).mean())
print(
f"\n P(CAGR<0) = {p_neg:.3f}\n"
f" P(CAGR<SPY OOS={spy_oos_cagr:.3f}) = {p_below_spy:.3f}\n"
f" P(MaxDD<-50%) = {p_dd_50:.3f}\n"
f" P(Sharpe<0.5) = {p_sharpe_below_05:.3f}"
)
# OOS-only bootstrap (the more honest "future" estimate)
rets_oos = rets[rets.index >= OOS_START]
boot_oos = block_bootstrap(
rets_oos, n_boot=args.n_boot, block_len=args.block_len, seed=43
)
print("\nOOS-only bootstrap (2021-2026):")
print(bootstrap_summary(boot_oos).round(4).to_string())
# ---------- P0.3 ----------
print("\n" + "=" * 78)
print("P0.3 De-leveraged comparison")
print("=" * 78)
de = deleveraged_evaluations(prices, transaction_cost=args.transaction_cost)
rows = []
for name, ev in de.items():
rows.append(asdict(ev))
_print_eval(name, ev)
pd.DataFrame(rows).to_csv(os.path.join(args.out_dir, "p0_deleveraged.csv"), index=False)
# Also break by IS / OOS
print("\n Same comparison, split IS vs OOS:")
for label, (start, end) in {"IS": (IS_START, IS_END), "OOS": (OOS_START, OOS_END)}.items():
print(f" --- {label} ({start} → {end}) ---")
subs = {}
# Recompute on the slice
for nm, ctor in {
"TR_v3_leveraged": TrendRiderV3(),
"TR_v3_nolev_SPYQQQ": TrendRiderV3(risk_on=("SPY", "QQQ")),
"TR_v3_nolev_SHYoff": TrendRiderV3(risk_on=("SPY", "QQQ"), risk_off=("SHY",)),
}.items():
w = ctor.generate_signals(prices)
subs[nm] = evaluate_weights(
nm, w, prices[w.columns], args.transaction_cost, start, end
)
spy_w = buy_hold_weights(prices, "SPY")
qqq_w = buy_hold_weights(prices, "QQQ")
subs["SPY_BH"] = evaluate_weights("SPY_BH", spy_w, prices[spy_w.columns], 0.0, start, end)
subs["QQQ_BH"] = evaluate_weights("QQQ_BH", qqq_w, prices[qqq_w.columns], 0.0, start, end)
for nm, ev in subs.items():
_print_eval(nm, ev)
if __name__ == "__main__":
main()

312
research/trend_rider_robustness.py
View File

@@ -1,312 +0,0 @@
"""Robustness analysis for TrendRiderV3.
Run:
uv run python -m research.trend_rider_robustness
The module is import-safe for tests; price loading only happens in ``main``.
"""
from __future__ import annotations
import argparse
import os
from dataclasses import asdict, dataclass
from itertools import product
from typing import Iterable
import numpy as np
import pandas as pd
from strategies.permanent import (
ETF_UNIVERSE,
GLOBAL_ETF_UNIVERSE,
HK_ETF_UNIVERSE,
PermanentV4,
TREND_RIDER_V4_UNIVERSE,
TrendRiderV3,
TrendRiderV4,
)
@dataclass
class Evaluation:
name: str
start: str
end: str
days: int
cagr: float
volatility: float
sharpe: float
max_drawdown: float
calmar: float
final_multiple: float
switches: int
avg_daily_turnover: float
avg_gross_exposure: float
def portfolio_returns(
weights: pd.DataFrame,
prices: pd.DataFrame,
transaction_cost: float = 0.001,
) -> pd.Series:
aligned = weights.reindex(index=prices.index, columns=prices.columns).fillna(0.0)
returns = prices.pct_change(fill_method=None).fillna(0.0)
gross = (returns * aligned).sum(axis=1)
turnover = aligned.diff().abs().sum(axis=1).fillna(0.0)
return gross - turnover * transaction_cost
def evaluate_weights(
name: str,
weights: pd.DataFrame,
prices: pd.DataFrame,
transaction_cost: float = 0.001,
start: str | None = None,
end: str | None = None,
) -> Evaluation:
prices = prices.reindex(columns=weights.columns).dropna(how="all")
returns = portfolio_returns(weights, prices, transaction_cost=transaction_cost)
if start:
returns = returns[returns.index >= start]
weights = weights[weights.index >= start]
if end:
returns = returns[returns.index <= end]
weights = weights[weights.index <= end]
if returns.empty:
raise ValueError(f"No returns available for {name}")
equity = (1.0 + returns).cumprod()
span_years = max((returns.index[-1] - returns.index[0]).days / 365.25, 1 / 252)
cagr = float(equity.iloc[-1] ** (1 / span_years) - 1)
vol = float(returns.std(ddof=1) * np.sqrt(252)) if len(returns) > 1 else 0.0
sharpe = float(returns.mean() / returns.std(ddof=1) * np.sqrt(252)) if returns.std(ddof=1) > 0 else 0.0
drawdown = equity / equity.cummax() - 1.0
max_dd = float(drawdown.min())
turnover = weights.reindex(returns.index).fillna(0.0).diff().abs().sum(axis=1).fillna(0.0)
gross_exposure = weights.reindex(returns.index).fillna(0.0).abs().sum(axis=1)
return Evaluation(
name=name,
start=str(returns.index[0].date()),
end=str(returns.index[-1].date()),
days=int(len(returns)),
cagr=cagr,
volatility=vol,
sharpe=sharpe,
max_drawdown=max_dd,
calmar=float(cagr / abs(max_dd)) if max_dd < 0 else 0.0,
final_multiple=float(equity.iloc[-1]),
switches=int((turnover > 0.01).sum()),
avg_daily_turnover=float(turnover.mean()),
avg_gross_exposure=float(gross_exposure.mean()),
)
def evaluate_strategy(
name: str,
strategy: TrendRiderV3,
prices: pd.DataFrame,
transaction_cost: float = 0.001,
start: str | None = None,
end: str | None = None,
) -> tuple[Evaluation, pd.DataFrame]:
weights = strategy.generate_signals(prices)
result = evaluate_weights(
name,
weights,
prices[weights.columns],
transaction_cost=transaction_cost,
start=start,
end=end,
)
return result, weights
def default_parameter_grid() -> list[dict]:
return [
{
"vol_enter": vol_enter,
"dd_stop": dd_stop,
"peak_enter": peak_enter,
"mom_lookback": mom,
}
for vol_enter, dd_stop, peak_enter, mom in product(
[0.12, 0.14, 0.16],
[0.04, 0.05, 0.07],
[0.01, 0.02, 0.03],
[42, 63, 84],
)
]
def parameter_sweep(
prices: pd.DataFrame,
variants: Iterable[dict] | None = None,
transaction_cost: float = 0.001,
start: str | None = None,
end: str | None = None,
) -> pd.DataFrame:
rows = []
for kwargs in variants or default_parameter_grid():
strategy = TrendRiderV3(**kwargs)
result, _ = evaluate_strategy(
"param",
strategy,
prices,
transaction_cost=transaction_cost,
start=start,
end=end,
)
row = asdict(result)
row.update(kwargs)
rows.append(row)
return pd.DataFrame(rows).sort_values("cagr", ascending=False).reset_index(drop=True)
def annual_returns(returns: pd.Series) -> pd.Series:
return (1.0 + returns).groupby(returns.index.year).prod() - 1.0
def buy_hold_weights(prices: pd.DataFrame, symbol: str) -> pd.DataFrame:
weights = pd.DataFrame(0.0, index=prices.index, columns=[symbol])
if symbol in prices.columns:
first_valid = prices[symbol].first_valid_index()
if first_valid is not None:
weights.loc[weights.index >= first_valid, symbol] = 1.0
return weights
def candidate_weights(prices: pd.DataFrame) -> dict[str, pd.DataFrame]:
baseline = TrendRiderV3().generate_signals(prices)
diversified = TrendRiderV4().generate_signals(prices)
shy_defense = TrendRiderV3(risk_off=("GLD", "DBC", "SHY")).generate_signals(prices)
cash_defense = TrendRiderV3(risk_off=("SHY",)).generate_signals(prices)
permanent = PermanentV4().generate_signals(prices)
cols = sorted(set(baseline.columns) | set(permanent.columns))
base_aligned = baseline.reindex(columns=cols).fillna(0.0)
perm_aligned = permanent.reindex(index=baseline.index, columns=cols).fillna(0.0)
return {
"TrendRiderV3-US": baseline,
"TrendRiderV4": diversified,
"RiskOff+SHY": shy_defense,
"RiskOff=SHY": cash_defense,
"Blend75_TR25_PermanentV4": base_aligned * 0.75 + perm_aligned * 0.25,
"Blend50_TR50_PermanentV4": base_aligned * 0.50 + perm_aligned * 0.50,
"SPY Buy&Hold": buy_hold_weights(prices, "SPY"),
"QQQ Buy&Hold": buy_hold_weights(prices, "QQQ"),
}
def load_price_panel() -> pd.DataFrame:
from research.permanent_yearly import load_etfs
tickers = sorted(set(ETF_UNIVERSE + GLOBAL_ETF_UNIVERSE + HK_ETF_UNIVERSE + TREND_RIDER_V4_UNIVERSE))
etfs = load_etfs(tickers, start="2013-06-01")
nyse_index = etfs["SPY"].dropna().index
return etfs.reindex(nyse_index).ffill()
def _format_percent_frame(df: pd.DataFrame, cols: list[str]) -> pd.DataFrame:
out = df.copy()
for col in cols:
out[col] = out[col].map(lambda x: f"{x * 100:,.2f}%")
return out
def main() -> None:
parser = argparse.ArgumentParser(description="TrendRiderV3 robustness report")
parser.add_argument("--start", default="2015-01-01")
parser.add_argument("--end", default=None)
parser.add_argument("--transaction-cost", type=float, default=0.001)
parser.add_argument("--out-dir", default="data")
args = parser.parse_args()
prices = load_price_panel()
if args.end:
prices = prices[prices.index <= args.end]
print(f"ETF panel: {prices.index.min().date()} to {prices.index.max().date()} | {prices.shape[1]} columns")
rows = []
weight_map = candidate_weights(prices)
for name, weights in weight_map.items():
rows.append(asdict(evaluate_weights(
name,
weights,
prices[weights.columns],
transaction_cost=args.transaction_cost,
start=args.start,
end=args.end,
)))
summary = pd.DataFrame(rows).sort_values(["max_drawdown", "cagr"], ascending=[False, False])
annual_map = {}
for name, weights in weight_map.items():
returns = portfolio_returns(
weights,
prices[weights.columns],
transaction_cost=args.transaction_cost,
)
returns = returns[returns.index >= args.start]
if args.end:
returns = returns[returns.index <= args.end]
annual_map[name] = annual_returns(returns)
years = pd.DataFrame(annual_map)
sweep = parameter_sweep(
prices,
transaction_cost=args.transaction_cost,
start=args.start,
end=args.end,
)
cost_rows = []
baseline_weights = weight_map["TrendRiderV3-US"]
for cost in [0.0, 0.001, 0.002, 0.005, 0.01]:
result = evaluate_weights(
f"cost_{cost:.3f}",
baseline_weights,
prices[baseline_weights.columns],
transaction_cost=cost,
start=args.start,
end=args.end,
)
row = asdict(result)
row["transaction_cost"] = cost
cost_rows.append(row)
costs = pd.DataFrame(cost_rows)
os.makedirs(args.out_dir, exist_ok=True)
summary_path = os.path.join(args.out_dir, "trend_rider_robustness_summary.csv")
years_path = os.path.join(args.out_dir, "trend_rider_robustness_years.csv")
sweep_path = os.path.join(args.out_dir, "trend_rider_robustness_params.csv")
costs_path = os.path.join(args.out_dir, "trend_rider_robustness_costs.csv")
summary.to_csv(summary_path, index=False)
years.to_csv(years_path)
sweep.to_csv(sweep_path, index=False)
costs.to_csv(costs_path, index=False)
metric_cols = ["cagr", "volatility", "sharpe", "max_drawdown", "calmar", "final_multiple", "switches"]
print("\nCandidate summary")
print(_format_percent_frame(summary[["name", *metric_cols]], ["cagr", "volatility", "max_drawdown"]).to_string(index=False))
print("\nAnnual returns")
annual_cols = [c for c in ["TrendRiderV3-US", "TrendRiderV4", "SPY Buy&Hold", "QQQ Buy&Hold"] if c in years.columns]
print(_format_percent_frame(years[annual_cols].reset_index(), annual_cols).to_string(index=False))
quant = sweep[["cagr", "max_drawdown", "sharpe", "final_multiple"]].quantile([0, 0.1, 0.25, 0.5, 0.75, 0.9, 1.0])
print("\nParameter-neighborhood quantiles")
print(_format_percent_frame(quant, ["cagr", "max_drawdown"]).to_string())
print("\nCost sensitivity")
print(_format_percent_frame(costs[["transaction_cost", "cagr", "max_drawdown", "final_multiple"]], ["transaction_cost", "cagr", "max_drawdown"]).to_string(index=False))
print(f"\nSaved: {summary_path}")
print(f"Saved: {years_path}")
print(f"Saved: {sweep_path}")
print(f"Saved: {costs_path}")
if __name__ == "__main__":
main()

150
research/trend_rider_v5_eval.py
View File

@@ -1,150 +0,0 @@
"""Evaluate TrendRiderV5 vs V3 baseline and benchmarks.
Run:
uv run python -m research.trend_rider_v5_eval
"""
from __future__ import annotations
import argparse
import os
import sys
from dataclasses import asdict
from itertools import product
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.trend_rider_robustness import (
buy_hold_weights,
evaluate_weights,
load_price_panel,
portfolio_returns,
)
from strategies.permanent import TrendRiderV3
from strategies.trend_rider_v5 import TrendRiderV5
IS_START = "2015-01-02"
IS_END = "2020-12-31"
OOS_START = "2021-01-01"
OOS_END = "2026-05-07"
FULL_START = IS_START
FULL_END = OOS_END
def _fmt(x: float) -> str:
return f"{x * 100:7.2f}%"
def print_eval(label: str, ev) -> None:
print(
f" {label:<32s} "
f"CAGR {_fmt(ev.cagr)} Vol {_fmt(ev.volatility)} "
f"Sharpe {ev.sharpe:5.2f} MDD {_fmt(ev.max_drawdown)} "
f"Calmar {ev.calmar:5.2f} X {ev.final_multiple:6.2f} "
f"Sw {ev.switches:4d} Turn {ev.avg_daily_turnover*100:5.2f}%"
)
def evaluate_panel(name: str, weights: pd.DataFrame, prices: pd.DataFrame,
start: str, end: str, transaction_cost: float = 0.001):
return evaluate_weights(name, weights, prices[weights.columns],
transaction_cost=transaction_cost,
start=start, end=end)
def annual_returns_table(weights_map: dict, prices: pd.DataFrame,
transaction_cost: float = 0.001) -> pd.DataFrame:
out = {}
for name, w in weights_map.items():
rets = portfolio_returns(w, prices[w.columns], transaction_cost=transaction_cost)
rets = rets[(rets.index >= FULL_START) & (rets.index <= FULL_END)]
out[name] = (1.0 + rets).groupby(rets.index.year).prod() - 1.0
return pd.DataFrame(out)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--transaction-cost", type=float, default=0.001)
parser.add_argument("--out-dir", default="data")
parser.add_argument("--vol-target", type=float, default=0.30)
args = parser.parse_args()
os.makedirs(args.out_dir, exist_ok=True)
prices = load_price_panel()
print(f"Panel: {prices.index.min().date()} to {prices.index.max().date()}, {prices.shape[1]} cols")
candidates = {
"V3 default": TrendRiderV3(),
"V5 default": TrendRiderV5(),
# Tighter panic detection
"V5 panic 1.4 / 3%": TrendRiderV5(
panic_vol_ratio=1.4, panic_peak_drop_pct=0.03
),
"V5 panic 1.5 / 3.5%": TrendRiderV5(
panic_vol_ratio=1.5, panic_peak_drop_pct=0.035
),
"V5 panic 1.8 / 5%": TrendRiderV5(
panic_vol_ratio=1.8, panic_peak_drop_pct=0.05
),
# Combine panic + harder promote
"V5 panic+conserv": TrendRiderV5(
promote_thresholds=(0.45, 0.70),
demote_thresholds=(0.35, 0.55),
panic_vol_ratio=1.4, panic_peak_drop_pct=0.03,
),
# No panic at all (pure conviction)
"V5 no panic": TrendRiderV5(
panic_vol_ratio=99.0, panic_peak_drop_pct=0.99
),
}
weights_map = {}
print("\n=== Generating signals ===")
for name, strat in candidates.items():
weights_map[name] = strat.generate_signals(prices)
print("\n=== FULL period (2015-01 → 2026-05) ===")
rows = []
for name, w in weights_map.items():
ev = evaluate_panel(name, w, prices, FULL_START, FULL_END, args.transaction_cost)
rows.append(asdict(ev) | {"name": name})
print_eval(name, ev)
spy_w = buy_hold_weights(prices, "SPY")
qqq_w = buy_hold_weights(prices, "QQQ")
bench = {
"SPY B&H": evaluate_panel("SPY B&H", spy_w, prices, FULL_START, FULL_END, 0.0),
"QQQ B&H": evaluate_panel("QQQ B&H", qqq_w, prices, FULL_START, FULL_END, 0.0),
}
for name, ev in bench.items():
print_eval(name, ev)
print("\n=== IS (2015 → 2020) ===")
for name, w in weights_map.items():
ev = evaluate_panel(name, w, prices, IS_START, IS_END, args.transaction_cost)
print_eval(name, ev)
for name, w in [("SPY B&H", spy_w), ("QQQ B&H", qqq_w)]:
ev = evaluate_panel(name, w, prices, IS_START, IS_END, 0.0)
print_eval(name, ev)
print("\n=== OOS (2021 → 2026-05) ===")
for name, w in weights_map.items():
ev = evaluate_panel(name, w, prices, OOS_START, OOS_END, args.transaction_cost)
print_eval(name, ev)
for name, w in [("SPY B&H", spy_w), ("QQQ B&H", qqq_w)]:
ev = evaluate_panel(name, w, prices, OOS_START, OOS_END, 0.0)
print_eval(name, ev)
print("\n=== Annual returns ===")
annual = annual_returns_table(weights_map, prices, args.transaction_cost)
annual = annual.applymap(lambda x: f"{x*100:6.1f}%")
print(annual.to_string())
pd.DataFrame(rows).to_csv(os.path.join(args.out_dir, "v5_eval_full.csv"), index=False)
annual.to_csv(os.path.join(args.out_dir, "v5_eval_annual.csv"))
if __name__ == "__main__":
main()

197
research/trend_rider_v6_eval.py
View File

@@ -1,197 +0,0 @@
"""Evaluate TrendRiderV6 vs V5 baseline.
Run:
uv run python -m research.trend_rider_v6_eval
"""
from __future__ import annotations
import argparse
import os
import sys
from dataclasses import asdict
from datetime import datetime, timedelta
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.permanent_yearly import load_long_stock_history, load_etfs, ETF_CACHE
from research.trend_rider_robustness import (
buy_hold_weights,
evaluate_weights,
portfolio_returns,
)
from strategies.permanent import TrendRiderV3, ETF_UNIVERSE
from strategies.trend_rider_v5 import TrendRiderV5
from strategies.trend_rider_v6 import TrendRiderV6
from strategies.factor_combo import FactorComboStrategy, SIGNAL_REGISTRY
from strategies.recovery_momentum import RecoveryMomentumStrategy
IS_START = "2015-01-02"
IS_END = "2020-12-31"
OOS_START = "2021-01-01"
OOS_END = "2026-05-07"
def _fmt(x: float) -> str:
return f"{x*100:7.2f}%"
def print_eval(label: str, ev) -> None:
print(
f" {label:<42s} "
f"CAGR {_fmt(ev.cagr)} Vol {_fmt(ev.volatility)} "
f"Sharpe {ev.sharpe:5.2f} MDD {_fmt(ev.max_drawdown)} "
f"Calmar {ev.calmar:5.2f} X {ev.final_multiple:6.2f} "
f"Sw {ev.switches:5d} Turn {ev.avg_daily_turnover*100:5.2f}%"
)
def load_combined_panel() -> pd.DataFrame:
"""ETFs + S&P 500 stock panel anchored to SPY trading calendar."""
# ETFs
etf_tickers = sorted(set(ETF_UNIVERSE) | {"SPY", "QQQ", "TQQQ", "UPRO",
"GLD", "DBC", "SHY"})
etfs = load_etfs(etf_tickers, start="2013-06-01")
nyse = etfs["SPY"].dropna().index
# Stocks (large local cache: data/us_long.csv)
stock_cache = "data/us_long.csv"
if not os.path.exists(stock_cache):
raise FileNotFoundError(f"Missing {stock_cache} — run RecoveryMomentum once first.")
stocks = pd.read_csv(stock_cache, index_col=0, parse_dates=True)
# Drop any stock columns that overlap with ETF columns to avoid clash
overlap = set(stocks.columns) & set(etfs.columns)
if overlap:
stocks = stocks.drop(columns=list(overlap))
panel = etfs.reindex(nyse).ffill()
panel = panel.join(stocks.reindex(nyse).ffill(), how="left")
return panel
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--transaction-cost", type=float, default=0.001)
parser.add_argument("--out-dir", default="data")
args = parser.parse_args()
os.makedirs(args.out_dir, exist_ok=True)
panel = load_combined_panel()
print(f"Combined panel: {panel.index.min().date()} → {panel.index.max().date()}, "
f"{panel.shape[1]} columns ({len([c for c in panel.columns if c not in ETF_UNIVERSE])} stocks)")
# Stock-only universe (drop ETFs from the picking universe)
etf_set = set(ETF_UNIVERSE) | {"QQQ", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "SPY",
"YINN", "CHAU", "7200.HK", "7500.HK"}
stock_universe = [c for c in panel.columns if c not in etf_set]
candidates = {}
candidates["V5 (ETF-only baseline)"] = TrendRiderV5()
# V6 regime mode: tier 2 = TQQQ, tier 1 = stocks
candidates["V6 regime_mode top5"] = TrendRiderV6(
signal_name="rec_mfilt+deep_upvol", top_n=5, tier_mode="regime",
stock_universe=stock_universe,
)
candidates["V6 regime_mode top10"] = TrendRiderV6(
signal_name="rec_mfilt+deep_upvol", top_n=10, tier_mode="regime",
stock_universe=stock_universe,
)
candidates["V6 regime_mode mom7m top10"] = TrendRiderV6(
signal_name="mom7m+rec126", top_n=10, tier_mode="regime",
stock_universe=stock_universe,
)
candidates["V6 regime_mode ma200+mom7m top10"] = TrendRiderV6(
signal_name="ma200+mom7m+rec126", top_n=10, tier_mode="regime",
stock_universe=stock_universe,
)
# V6 blend mode best (rec_mfilt top10 + 50% TQQQ)
candidates["V6 blend rec_mfilt top10 +50%TQQQ"] = TrendRiderV6(
signal_name="rec_mfilt+deep_upvol", top_n=10,
tier2_leverage_overlay=0.50,
stock_universe=stock_universe,
)
# Concentrated stock pick: top 5
candidates["V6 blend top5 +50%TQQQ"] = TrendRiderV6(
signal_name="rec_mfilt+deep_upvol", top_n=5,
tier2_leverage_overlay=0.50,
stock_universe=stock_universe,
)
print("\n=== Generating signals ===")
weights_map = {}
for name, strat in candidates.items():
print(f" ... {name}")
weights_map[name] = strat.generate_signals(panel)
print("\n=== FULL period (2015-01 → 2026-05) ===")
rows = []
for name, w in weights_map.items():
ev = evaluate_weights(name, w, panel[w.columns], args.transaction_cost,
IS_START, OOS_END)
rows.append({**asdict(ev), "name": name})
print_eval(name, ev)
spy_w = buy_hold_weights(panel, "SPY")
qqq_w = buy_hold_weights(panel, "QQQ")
print_eval("SPY B&H", evaluate_weights("SPY", spy_w, panel[spy_w.columns], 0.0, IS_START, OOS_END))
print_eval("QQQ B&H", evaluate_weights("QQQ", qqq_w, panel[qqq_w.columns], 0.0, IS_START, OOS_END))
print("\n=== IS (2015 → 2020) ===")
for name, w in weights_map.items():
ev = evaluate_weights(name, w, panel[w.columns], args.transaction_cost, IS_START, IS_END)
print_eval(name, ev)
print("\n=== OOS (2021 → 2026-05) ===")
for name, w in weights_map.items():
ev = evaluate_weights(name, w, panel[w.columns], args.transaction_cost, OOS_START, OOS_END)
print_eval(name, ev)
# ----- V5 + V6 blends — uncorrelated alpha mixing -----
print("\n=== V5 + V6 BLENDS (risk-parity-ish 50/50 and 70/30) ===")
v5_w = weights_map["V5 (ETF-only baseline)"]
best_v6_name = "V6 regime_mode top10"
if best_v6_name in weights_map:
v6_w = weights_map[best_v6_name]
all_cols = sorted(set(v5_w.columns) | set(v6_w.columns))
v5_a = v5_w.reindex(columns=all_cols).fillna(0.0)
v6_a = v6_w.reindex(index=v5_a.index, columns=all_cols).fillna(0.0)
for w5, w6 in [(0.50, 0.50), (0.30, 0.70), (0.70, 0.30), (0.40, 0.60)]:
blend = v5_a * w5 + v6_a * w6
label = f"Blend V5={w5:.0%} + V6={w6:.0%}"
for window_name, (s, e) in {"FULL": (IS_START, OOS_END),
"IS": (IS_START, IS_END),
"OOS": (OOS_START, OOS_END)}.items():
ev = evaluate_weights(label, blend, panel[blend.columns],
args.transaction_cost, s, e)
print(f" [{window_name}] ", end="")
print_eval(label, ev)
# Correlation between V5 and V6 daily returns (full)
v5_rets = portfolio_returns(v5_a, panel[v5_a.columns], args.transaction_cost)
v6_rets = portfolio_returns(v6_a, panel[v6_a.columns], args.transaction_cost)
common = v5_rets.index.intersection(v6_rets.index)
v5_rets, v6_rets = v5_rets.loc[common], v6_rets.loc[common]
v5_rets = v5_rets[(v5_rets.index >= IS_START) & (v5_rets.index <= OOS_END)]
v6_rets = v6_rets[(v6_rets.index >= IS_START) & (v6_rets.index <= OOS_END)]
corr = float(v5_rets.corr(v6_rets))
print(f"\n V5 vs {best_v6_name} daily-return correlation = {corr:.3f}")
print("\n=== Annual returns ===")
annuals = {}
for name, w in weights_map.items():
rets = portfolio_returns(w, panel[w.columns], args.transaction_cost)
rets = rets[(rets.index >= IS_START) & (rets.index <= OOS_END)]
annuals[name] = (1.0 + rets).groupby(rets.index.year).prod() - 1.0
annual_df = pd.DataFrame(annuals)
annual_df = annual_df.map(lambda x: f"{x*100:6.1f}%")
print(annual_df.to_string())
pd.DataFrame(rows).to_csv(os.path.join(args.out_dir, "v6_eval_full.csv"), index=False)
if __name__ == "__main__":
main()

305
research/us_alpha_pipeline.py
View File

@@ -1,305 +0,0 @@
import numpy as np
import pandas as pd
import data_manager
import universe_history as uh
from research.event_factors import breakout_after_compression_score
from research.regime_filters import build_regime_filter
from research.us_alpha_report import summarize_equity_window
from research.us_fundamentals import build_exploratory_fundamental_score
from research.us_universe import build_tradable_mask
MIN_PRICE = 5.0
MIN_DOLLAR_VOLUME = 20_000_000.0
MIN_HISTORY_DAYS = 252
MIN_VALID_VOLUME_DAYS = 40
LIQUIDITY_WINDOW = 60
TREND_WINDOW = 126
RECOVERY_WINDOW = 63
HIGH_PROX_WINDOW = 126
ETF_TICKERS = ["SPY", "QQQ", "IWM", "MDY", "XLK", "XLF", "XLI", "XLV"]
def _price_rank_blend_score(close: pd.DataFrame) -> pd.DataFrame:
"""Simple price-only cross-sectional blend, shifted for next-day trading."""
trend = close.pct_change(TREND_WINDOW, fill_method=None)
recovery = close / close.rolling(RECOVERY_WINDOW, min_periods=RECOVERY_WINDOW).min() - 1
high_proximity = close / close.rolling(HIGH_PROX_WINDOW, min_periods=HIGH_PROX_WINDOW).max().replace(0, np.nan)
trend_rank = trend.rank(axis=1, pct=True, na_option="keep")
recovery_rank = recovery.rank(axis=1, pct=True, na_option="keep")
high_rank = high_proximity.rank(axis=1, pct=True, na_option="keep")
return ((trend_rank + recovery_rank + high_rank) / 3.0).shift(1)
def _build_equal_weight_portfolio(
score: pd.DataFrame,
tradable_mask: pd.DataFrame,
regime_filter: pd.Series,
top_n: int,
) -> pd.DataFrame:
"""Build equal-weight top-n long-only weights from aligned scores."""
aligned_score = score.reindex(index=tradable_mask.index, columns=tradable_mask.columns)
eligible_score = aligned_score.where(tradable_mask)
rank = eligible_score.rank(axis=1, ascending=False, na_option="bottom", method="first")
selected = (rank <= top_n) & eligible_score.notna()
selected = selected & regime_filter.reindex(tradable_mask.index, fill_value=False).to_numpy().reshape(-1, 1)
raw = selected.astype(float)
row_sums = raw.sum(axis=1).replace(0.0, np.nan)
return raw.div(row_sums, axis=0).fillna(0.0)
def _build_close_only_tradable_mask(close: pd.DataFrame, pit_membership: pd.DataFrame | None) -> pd.DataFrame:
if pit_membership is None:
pit_mask = pd.DataFrame(True, index=close.index, columns=close.columns)
else:
pit_mask = pit_membership.reindex(index=close.index, columns=close.columns, fill_value=False)
pit_mask = pit_mask.where(pit_mask.notna(), False).astype(bool)
eligible_close = close.where(pit_mask)
lagged_close = eligible_close.shift(1)
price_ok = lagged_close.gt(MIN_PRICE)
history_ok = (
lagged_close.notna()
.rolling(window=MIN_HISTORY_DAYS, min_periods=MIN_HISTORY_DAYS)
.sum()
.ge(MIN_HISTORY_DAYS)
)
return (price_ok & history_ok & pit_mask).astype(bool)
def _has_ohlcv_inputs(high: pd.DataFrame, low: pd.DataFrame, volume: pd.DataFrame) -> bool:
return not high.empty and not low.empty and not volume.empty and volume.notna().any().any()
def _blend_scores(left: pd.DataFrame, right: pd.DataFrame) -> pd.DataFrame:
left_rank = left.rank(axis=1, pct=True, na_option="keep")
right_rank = right.rank(axis=1, pct=True, na_option="keep")
return (left_rank + right_rank) / 2.0
def _equity_curve(close: pd.DataFrame, weights: pd.DataFrame) -> pd.Series:
"""Convert daily weights into a simple close-to-close equity curve."""
returns = close.pct_change(fill_method=None).fillna(0.0)
portfolio_returns = (returns * weights).sum(axis=1)
return (1.0 + portfolio_returns).cumprod()
def _read_panel_csv(path: str) -> pd.DataFrame:
return pd.read_csv(path, index_col=0, parse_dates=True).sort_index()
def _read_nonempty_panel_csv(path: str) -> pd.DataFrame:
try:
panel = _read_panel_csv(path)
except FileNotFoundError:
return pd.DataFrame()
return panel if not panel.empty else pd.DataFrame()
def load_saved_pit_market_data(data_dir: str = "data", prefix: str = "us_pit") -> dict[str, pd.DataFrame]:
"""Load saved PIT OHLCV panels from disk."""
panels = {}
for field in ("close", "high", "low", "volume"):
panels[field] = _read_panel_csv(f"{data_dir}/{prefix}_{field}.csv")
return panels
def load_saved_etf_close(data_dir: str = "data", market: str = "us_etf") -> pd.DataFrame:
"""Load saved ETF closes or populate them on demand."""
path = f"{data_dir}/{market}.csv"
try:
return _read_panel_csv(path)
except FileNotFoundError:
original_data_dir = data_manager.DATA_DIR
try:
data_manager.DATA_DIR = data_dir
return data_manager.update_market_data(market, ETF_TICKERS, ["close"])["close"]
finally:
data_manager.DATA_DIR = original_data_dir
def _strategy_scores(
close: pd.DataFrame,
high: pd.DataFrame,
low: pd.DataFrame,
volume: pd.DataFrame,
fundamental_score: pd.DataFrame | None = None,
) -> dict[str, pd.DataFrame]:
strategy_scores = {"rank_blend_regime": _price_rank_blend_score(close)}
if _has_ohlcv_inputs(high, low, volume):
strategy_scores["breakout_regime"] = breakout_after_compression_score(close, high, low, volume)
if fundamental_score is not None:
aligned_fundamental = fundamental_score.reindex(index=close.index, columns=close.columns)
strategy_scores["fundamental_regime"] = aligned_fundamental
if "breakout_regime" in strategy_scores:
strategy_scores["breakout_fundamental_regime"] = _blend_scores(
strategy_scores["breakout_regime"], aligned_fundamental
)
strategy_scores["rank_blend_fundamental_regime"] = _blend_scores(
strategy_scores["rank_blend_regime"], aligned_fundamental
)
return strategy_scores
def build_alpha_equity_curves(
market_data,
etf_close,
pit_membership=None,
top_n=10,
fundamental_score: pd.DataFrame | None = None,
) -> dict[str, pd.Series]:
close = market_data["close"].sort_index()
high = market_data["high"].reindex(index=close.index, columns=close.columns).sort_index()
low = market_data["low"].reindex(index=close.index, columns=close.columns).sort_index()
volume = market_data["volume"].reindex(index=close.index, columns=close.columns).sort_index()
if _has_ohlcv_inputs(high, low, volume):
tradable_mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=pit_membership,
min_price=MIN_PRICE,
min_dollar_volume=MIN_DOLLAR_VOLUME,
min_history_days=MIN_HISTORY_DAYS,
min_valid_volume_days=MIN_VALID_VOLUME_DAYS,
liquidity_window=LIQUIDITY_WINDOW,
)
else:
tradable_mask = _build_close_only_tradable_mask(close, pit_membership)
regime_filter = build_regime_filter(etf_close).reindex(close.index, fill_value=False)
equities = {}
for strategy_name, score in _strategy_scores(close, high, low, volume, fundamental_score).items():
weights = _build_equal_weight_portfolio(score, tradable_mask, regime_filter, top_n)
equities[strategy_name] = _equity_curve(close, weights)
return equities
def summarize_equity_curves(equity_curves: dict[str, pd.Series], windows=(1, 2, 3, 5, 10)) -> pd.DataFrame:
summary_rows = []
for strategy_name, equity in equity_curves.items():
for window_years in windows:
summary_rows.append(summarize_equity_window(equity, strategy_name, window_years))
return pd.DataFrame(summary_rows)
def summarize_yearly_returns(equity_curves: dict[str, pd.Series], years: list[int]) -> pd.DataFrame:
eq_df = pd.DataFrame(equity_curves).sort_index()
rows = []
for year in years:
window = eq_df.loc[(eq_df.index >= pd.Timestamp(year=year, month=1, day=1)) & (eq_df.index <= pd.Timestamp(year=year, month=12, day=31))]
if window.empty:
continue
row = {"Year": year}
for name in eq_df.columns:
series = window[name].dropna()
row[name] = np.nan if len(series) < 2 else (series.iloc[-1] / series.iloc[0] - 1.0)
rows.append(row)
if not rows:
return pd.DataFrame()
return pd.DataFrame(rows).set_index("Year")
def run_alpha_pipeline(
market_data,
etf_close,
pit_membership=None,
windows=(1, 2, 3, 5, 10),
top_n=10,
fundamental_score: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Run a lightweight strict US alpha pipeline and summarize trailing windows."""
equity_curves = build_alpha_equity_curves(
market_data=market_data,
etf_close=etf_close,
pit_membership=pit_membership,
top_n=top_n,
fundamental_score=fundamental_score,
)
return summarize_equity_curves(equity_curves, windows=windows)
def run_saved_pit_alpha_pipeline(
data_dir: str = "data",
windows=(1, 2, 3, 5, 10),
top_n: int = 10,
) -> pd.DataFrame:
"""Load saved PIT OHLCV inputs and run the strict alpha pipeline."""
market_data = load_saved_pit_market_data(data_dir=data_dir)
etf_close = load_saved_etf_close(data_dir=data_dir)
intervals = uh.load_sp500_history()
pit_membership = uh.membership_mask(
market_data["close"].index,
intervals=intervals,
tickers=list(market_data["close"].columns),
)
return run_alpha_pipeline(
market_data=market_data,
etf_close=etf_close,
pit_membership=pit_membership,
windows=windows,
top_n=top_n,
)
def run_exploratory_fundamental_alpha_pipeline(
data_dir: str = "data",
market: str = "us_alpha_exploratory",
windows=(1, 2, 3, 5, 10),
top_n: int = 10,
) -> tuple[pd.DataFrame, pd.DataFrame]:
cached_close = _read_panel_csv(f"{data_dir}/us.csv")
stock_tickers = [ticker for ticker in cached_close.columns if ticker not in ETF_TICKERS]
saved_close = _read_nonempty_panel_csv(f"{data_dir}/{market}.csv")
saved_high = _read_nonempty_panel_csv(f"{data_dir}/{market}_high.csv")
saved_low = _read_nonempty_panel_csv(f"{data_dir}/{market}_low.csv")
saved_volume = _read_nonempty_panel_csv(f"{data_dir}/{market}_volume.csv")
saved_etf = _read_nonempty_panel_csv(f"{data_dir}/us_etf.csv")
if not saved_close.empty and not saved_high.empty and not saved_low.empty and not saved_volume.empty:
market_data = {
"close": saved_close.reindex(columns=stock_tickers),
"high": saved_high.reindex(columns=stock_tickers),
"low": saved_low.reindex(columns=stock_tickers),
"volume": saved_volume.reindex(columns=stock_tickers),
}
else:
close = cached_close.reindex(columns=stock_tickers)
market_data = {
"close": close,
"high": pd.DataFrame(index=close.index, columns=close.columns, dtype=float),
"low": pd.DataFrame(index=close.index, columns=close.columns, dtype=float),
"volume": pd.DataFrame(index=close.index, columns=close.columns, dtype=float),
}
etf_close = saved_etf if not saved_etf.empty and "SPY" in saved_etf.columns else cached_close.reindex(columns=["SPY"]).dropna(how="all")
fundamental_score = build_exploratory_fundamental_score(market_data["close"], data_dir=data_dir)
equity_curves = build_alpha_equity_curves(
market_data=market_data,
etf_close=etf_close,
pit_membership=None,
top_n=top_n,
fundamental_score=fundamental_score,
)
windows_df = summarize_equity_curves(equity_curves, windows=windows)
years = list(range(int(market_data["close"].index.min().year), int(market_data["close"].index.max().year) + 1))
yearly_df = summarize_yearly_returns(equity_curves, years)
windows_df.to_csv(f"{data_dir}/us_alpha_fundamental_windows.csv", index=False)
yearly_df.to_csv(f"{data_dir}/us_alpha_fundamental_10y_yearly.csv")
return windows_df, yearly_df
def main() -> None:
windows_df, yearly_df = run_exploratory_fundamental_alpha_pipeline()
print("=== Window Summary ===")
print(windows_df.to_string(index=False))
print("\n=== Yearly Returns ===")
print((yearly_df * 100.0).round(2).to_string())
if __name__ == "__main__":
main()

37
research/us_alpha_report.py
View File

@@ -1,37 +0,0 @@
import numpy as np
import pandas as pd
TRADING_DAYS_PER_YEAR = 252
def summarize_equity_window(equity: pd.Series, strategy: str, window_years: int | float) -> dict:
"""Summarize a strategy equity curve over a trailing trading-day window."""
window_days = max(int(window_years * TRADING_DAYS_PER_YEAR), 1)
clean_equity = equity.dropna()
if len(clean_equity) < window_days + 1:
return {
"strategy": strategy,
"window_years": window_years,
"CAGR": np.nan,
"Sharpe": np.nan,
"MaxDD": np.nan,
"TotalRet": np.nan,
}
window_equity = clean_equity.tail(window_days + 1)
daily = window_equity.pct_change(fill_method=None).dropna()
total_ret = window_equity.iloc[-1] / window_equity.iloc[0] - 1
years = len(daily) / TRADING_DAYS_PER_YEAR
cagr = (window_equity.iloc[-1] / window_equity.iloc[0]) ** (1 / years) - 1 if years > 0 else np.nan
vol = daily.std() * np.sqrt(TRADING_DAYS_PER_YEAR)
sharpe = (daily.mean() * TRADING_DAYS_PER_YEAR) / vol if vol > 0 else 0.0
max_dd = (window_equity / window_equity.cummax() - 1).min()
return {
"strategy": strategy,
"window_years": window_years,
"CAGR": cagr,
"Sharpe": sharpe,
"MaxDD": max_dd,
"TotalRet": total_ret,
}

234
research/us_combo_sweep.py
View File

@@ -1,234 +0,0 @@
import numpy as np
import pandas as pd
from research.us_alpha_report import summarize_equity_window
from research.us_fundamentals import build_exploratory_fundamental_score
from strategies.recovery_momentum import RecoveryMomentumStrategy
TRADING_DAYS_PER_MONTH = 21
def xsec_rank(df: pd.DataFrame, ascending: bool = True) -> pd.DataFrame:
return df.rank(axis=1, pct=True, na_option="keep", ascending=ascending)
def apply_filter_threshold(score: pd.DataFrame, filter_rank: pd.DataFrame, min_rank: float) -> pd.DataFrame:
aligned_filter = filter_rank.reindex(index=score.index, columns=score.columns)
return score.where(aligned_filter >= min_rank)
def weighted_rank_blend(factors: dict[str, pd.DataFrame], weights: dict[str, float]) -> pd.DataFrame:
total = None
total_weight = 0.0
for name, weight in weights.items():
rank = xsec_rank(factors[name])
component = rank * weight
total = component if total is None else total.add(component, fill_value=0.0)
total_weight += weight
return total / total_weight if total_weight > 0 else total
def build_price_factor_pack(close: pd.DataFrame) -> dict[str, pd.DataFrame]:
monthly_ret = close.pct_change(TRADING_DAYS_PER_MONTH)
rolling_max = close.rolling(252, min_periods=252).max()
drawdown = close / rolling_max - 1.0
return {
"recovery": close / close.rolling(63, min_periods=63).min() - 1.0,
"momentum_12_1": close.shift(21).pct_change(231),
"consistency": monthly_ret.gt(0).rolling(252, min_periods=252).mean(),
"inv_drawdown": -drawdown.rolling(252, min_periods=252).min(),
"low_vol": -close.pct_change().rolling(60, min_periods=60).std(),
"dip_21": -close.pct_change(21),
"value_proxy": close.rolling(250, min_periods=250).min() / close,
"uptrend": (close > close.rolling(150, min_periods=150).mean()).astype(float),
}
def _monthly_score_weights(score: pd.DataFrame, top_n: int, rebal_freq: int = TRADING_DAYS_PER_MONTH) -> pd.DataFrame:
score = score.sort_index()
n_valid = score.notna().sum(axis=1)
enough = n_valid >= top_n
rank = score.rank(axis=1, ascending=False, na_option="bottom", method="first")
top_mask = (rank <= top_n) & enough.to_numpy().reshape(-1, 1)
raw = top_mask.astype(float)
row_sums = raw.sum(axis=1).replace(0.0, np.nan)
weights = raw.div(row_sums, axis=0).fillna(0.0)
first_valid = int(np.argmax(score.notna().any(axis=1).to_numpy())) if score.notna().any().any() else 0
rebal_mask = pd.Series(False, index=score.index)
rebal_mask.iloc[list(range(first_valid, len(score), rebal_freq))] = True
weights[~rebal_mask] = np.nan
weights = weights.ffill().fillna(0.0)
weights.iloc[:first_valid] = 0.0
return weights.shift(1).fillna(0.0)
def _backtest_from_weights(
close: pd.DataFrame,
weights: pd.DataFrame,
initial_capital: float = 10_000.0,
transaction_cost: float = 0.001,
) -> pd.Series:
daily_returns = close.pct_change(fill_method=None).fillna(0.0)
portfolio_returns = (daily_returns * weights.reindex(close.index).fillna(0.0)).sum(axis=1)
turnover = weights.diff().abs().sum(axis=1).fillna(0.0)
portfolio_returns -= turnover * transaction_cost
return (1.0 + portfolio_returns).cumprod() * initial_capital
def _equity_to_yearly_returns(equity: pd.Series) -> pd.Series:
rows = {}
for year in range(int(equity.index.min().year), int(equity.index.max().year) + 1):
window = equity.loc[(equity.index >= pd.Timestamp(year=year, month=1, day=1)) & (equity.index <= pd.Timestamp(year=year, month=12, day=31))]
if len(window.dropna()) >= 2:
rows[year] = window.dropna().iloc[-1] / window.dropna().iloc[0] - 1.0
return pd.Series(rows, name=equity.name)
def _cagr(equity: pd.Series) -> float:
clean = equity.dropna()
years = (clean.index[-1] - clean.index[0]).days / 365.25
if years <= 0:
return np.nan
return (clean.iloc[-1] / clean.iloc[0]) ** (1 / years) - 1
def _max_dd(equity: pd.Series) -> float:
clean = equity.dropna()
return (clean / clean.cummax() - 1.0).min()
def _candidate_scores(price_factors: dict[str, pd.DataFrame], fundamental_score: pd.DataFrame) -> dict[str, pd.DataFrame]:
factors = {**price_factors, "fundamental": fundamental_score}
base_rm = weighted_rank_blend(factors, {"recovery": 0.5, "momentum_12_1": 0.5})
candidates = {
"rm_fund_filter_50": apply_filter_threshold(base_rm, xsec_rank(fundamental_score), min_rank=0.50),
"rm_fund_filter_70": apply_filter_threshold(base_rm, xsec_rank(fundamental_score), min_rank=0.70),
"rm_fund_tilt_20": weighted_rank_blend(factors, {"recovery": 0.4, "momentum_12_1": 0.4, "fundamental": 0.2}),
"rm_fund_tilt_35": weighted_rank_blend(factors, {"recovery": 0.325, "momentum_12_1": 0.325, "fundamental": 0.35}),
"rm_quality_fund": weighted_rank_blend(
factors,
{"recovery": 0.35, "momentum_12_1": 0.35, "consistency": 0.10, "inv_drawdown": 0.10, "fundamental": 0.10},
),
"rm_quality_lowvol_fund": weighted_rank_blend(
factors,
{"recovery": 0.30, "momentum_12_1": 0.25, "consistency": 0.10, "inv_drawdown": 0.10, "low_vol": 0.10, "fundamental": 0.15},
),
"mega_quality_fund": weighted_rank_blend(
factors,
{
"recovery": 0.20,
"momentum_12_1": 0.20,
"consistency": 0.15,
"inv_drawdown": 0.15,
"low_vol": 0.10,
"dip_21": 0.05,
"value_proxy": 0.05,
"fundamental": 0.10,
},
),
"mega_filter_fund_50": apply_filter_threshold(
weighted_rank_blend(
factors,
{
"recovery": 0.25,
"momentum_12_1": 0.20,
"consistency": 0.10,
"inv_drawdown": 0.10,
"low_vol": 0.10,
"value_proxy": 0.10,
"fundamental": 0.15,
},
),
xsec_rank(fundamental_score),
min_rank=0.50,
),
"trend_rm_fund": apply_filter_threshold(
weighted_rank_blend(factors, {"recovery": 0.35, "momentum_12_1": 0.35, "fundamental": 0.15, "low_vol": 0.15}),
price_factors["uptrend"],
min_rank=0.50,
),
}
return candidates
def run_combo_backtests(
close: pd.DataFrame,
fundamental_score: pd.DataFrame,
top_n: int = 10,
transaction_cost: float = 0.001,
) -> tuple[pd.DataFrame, pd.DataFrame]:
benchmark_col = "SPY" if "SPY" in close.columns else None
stock_close = close.drop(columns=[benchmark_col], errors="ignore").dropna(axis=1, how="all")
fund = fundamental_score.reindex(index=stock_close.index, columns=stock_close.columns)
price_factors = build_price_factor_pack(stock_close)
equities: dict[str, pd.Series] = {}
baseline = RecoveryMomentumStrategy(top_n=top_n)
baseline_weights = baseline.generate_signals(stock_close)
equities["Recovery+Mom Top10"] = _backtest_from_weights(stock_close, baseline_weights, transaction_cost=transaction_cost)
for name, score in _candidate_scores(price_factors, fund).items():
weights = _monthly_score_weights(score.reindex(index=stock_close.index, columns=stock_close.columns), top_n=top_n)
equities[name] = _backtest_from_weights(stock_close, weights, transaction_cost=transaction_cost)
if benchmark_col is not None:
spy = close[benchmark_col].dropna()
equities["SPY"] = (spy / spy.iloc[0]) * 10_000.0
yearly = pd.DataFrame({name: _equity_to_yearly_returns(eq) for name, eq in equities.items()}).sort_index()
baseline_yearly = yearly["Recovery+Mom Top10"]
summary_rows = []
for name, equity in equities.items():
row = {
"strategy": name,
"CAGR": _cagr(equity),
"MaxDD": _max_dd(equity),
"TotalRet": equity.dropna().iloc[-1] / equity.dropna().iloc[0] - 1.0,
"AvgAnnual": yearly[name].mean(),
"MedianAnnual": yearly[name].median(),
"YearsBeatRecovery": int(yearly[name].gt(baseline_yearly).sum()) if name != "Recovery+Mom Top10" else np.nan,
}
row.update({f"Win{window}Y": summarize_equity_window(equity / equity.dropna().iloc[0], name, window)["CAGR"] for window in (1, 3, 5, 10)})
summary_rows.append(row)
summary = pd.DataFrame(summary_rows).sort_values("AvgAnnual", ascending=False).reset_index(drop=True)
return yearly, summary
def load_default_inputs(data_dir: str = "data") -> tuple[pd.DataFrame, pd.DataFrame]:
close = pd.read_csv(f"{data_dir}/us.csv", index_col=0, parse_dates=True).sort_index()
stock_close = close.drop(columns=["SPY"], errors="ignore")
fundamental_score = build_exploratory_fundamental_score(stock_close, data_dir=data_dir)
return close, fundamental_score
def main() -> None:
close, fundamental_score = load_default_inputs()
yearly, summary = run_combo_backtests(close, fundamental_score, top_n=10)
yearly.to_csv("data/us_factor_combo_yearly.csv")
summary.to_csv("data/us_factor_combo_summary.csv", index=False)
print("=== Yearly Returns ===")
print((yearly * 100.0).round(2).to_string())
print("\n=== Summary ===")
display_cols = ["strategy", "AvgAnnual", "MedianAnnual", "CAGR", "MaxDD", "YearsBeatRecovery", "Win1Y", "Win3Y", "Win5Y", "Win10Y"]
print((summary[display_cols].assign(
AvgAnnual=lambda df: df["AvgAnnual"] * 100.0,
MedianAnnual=lambda df: df["MedianAnnual"] * 100.0,
CAGR=lambda df: df["CAGR"] * 100.0,
MaxDD=lambda df: df["MaxDD"] * 100.0,
Win1Y=lambda df: df["Win1Y"] * 100.0,
Win3Y=lambda df: df["Win3Y"] * 100.0,
Win5Y=lambda df: df["Win5Y"] * 100.0,
Win10Y=lambda df: df["Win10Y"] * 100.0,
).round(2)).to_string(index=False))
if __name__ == "__main__":
main()

273
research/us_fundamentals.py
View File

@@ -1,273 +0,0 @@
import json
import time
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
import numpy as np
import pandas as pd
DEFAULT_SEC_USER_AGENT = "quant-research/0.1 gahow@example.com"
DEFAULT_LAG_DAYS = 60
FRAME_SLEEP_SECONDS = 0.2
QUARTERLY_DURATION_CONCEPTS = {
"net_income": [("NetIncomeLoss", "USD"), ("ProfitLoss", "USD")],
"gross_profit": [("GrossProfit", "USD")],
}
QUARTERLY_INSTANT_CONCEPTS = {
"equity": [
("StockholdersEquityIncludingPortionAttributableToNoncontrollingInterest", "USD"),
("StockholdersEquity", "USD"),
],
"assets": [("Assets", "USD")],
"shares": [
("CommonStockSharesOutstanding", "shares"),
("EntityCommonStockSharesOutstanding", "shares"),
],
}
def _normalize_ticker(ticker: str) -> str:
return ticker.upper().replace(".", "-")
def _frame_code(period_end: pd.Timestamp, instant: bool) -> str:
quarter = ((period_end.month - 1) // 3) + 1
suffix = "I" if instant else ""
return f"CY{period_end.year}Q{quarter}{suffix}"
def _cache_dir(data_dir: str) -> Path:
path = Path(data_dir) / "sec_frames"
path.mkdir(parents=True, exist_ok=True)
return path
def load_sec_ticker_map(data_dir: str = "data", user_agent: str = DEFAULT_SEC_USER_AGENT) -> pd.DataFrame:
cache_path = Path(data_dir) / "sec_company_tickers.json"
if cache_path.exists():
raw = json.loads(cache_path.read_text())
else:
request = Request(
"https://www.sec.gov/files/company_tickers.json",
headers={"User-Agent": user_agent, "Accept": "application/json"},
)
with urlopen(request, timeout=30) as response:
raw = json.loads(response.read().decode("utf-8"))
cache_path.write_text(json.dumps(raw))
rows = []
for item in raw.values():
rows.append(
{
"ticker": _normalize_ticker(item["ticker"]),
"cik": int(item["cik_str"]),
"title": item["title"],
}
)
return pd.DataFrame(rows).drop_duplicates(subset=["ticker"]).sort_values("ticker").reset_index(drop=True)
def _load_or_fetch_frame(
tag: str,
unit: str,
frame_code: str,
data_dir: str = "data",
user_agent: str = DEFAULT_SEC_USER_AGENT,
) -> dict | None:
cache_path = _cache_dir(data_dir) / f"{tag}_{unit}_{frame_code}.json"
if cache_path.exists():
return json.loads(cache_path.read_text())
url = f"https://data.sec.gov/api/xbrl/frames/us-gaap/{tag}/{unit}/{frame_code}.json"
request = Request(url, headers={"User-Agent": user_agent, "Accept": "application/json"})
try:
with urlopen(request, timeout=60) as response:
payload = json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
if exc.code == 404:
return None
raise
except URLError:
raise
cache_path.write_text(json.dumps(payload))
time.sleep(FRAME_SLEEP_SECONDS)
return payload
def _frame_to_series(payload: dict | None, cik_to_ticker: dict[int, str]) -> pd.Series:
if not payload:
return pd.Series(dtype=float)
frame = pd.DataFrame(payload.get("data", []))
if frame.empty:
return pd.Series(dtype=float)
frame = frame.loc[frame["cik"].isin(cik_to_ticker)]
if frame.empty:
return pd.Series(dtype=float)
frame["ticker"] = frame["cik"].map(cik_to_ticker)
frame = frame.dropna(subset=["ticker", "val"])
frame = frame.sort_values(["ticker", "end"])
series = frame.groupby("ticker")["val"].last()
series.index.name = None
return series.astype(float)
def _combine_quarterly_panels(panels: list[pd.DataFrame]) -> pd.DataFrame:
combined = pd.DataFrame()
for panel in panels:
if panel.empty:
continue
if combined.empty:
combined = panel.copy()
continue
combined = combined.combine_first(panel)
return combined.sort_index()
def fetch_sec_quarterly_panels(
tickers: list[str],
price_index: pd.Index,
data_dir: str = "data",
user_agent: str = DEFAULT_SEC_USER_AGENT,
) -> dict[str, pd.DataFrame]:
normalized_to_original = {_normalize_ticker(t): t for t in tickers}
ticker_map = load_sec_ticker_map(data_dir=data_dir, user_agent=user_agent)
ticker_map = ticker_map.loc[ticker_map["ticker"].isin(normalized_to_original)]
cik_to_ticker = {
int(row.cik): normalized_to_original[row.ticker]
for row in ticker_map.itertuples(index=False)
if row.ticker in normalized_to_original
}
if not cik_to_ticker:
return {name: pd.DataFrame(index=pd.Index([], dtype="datetime64[ns]"), columns=tickers) for name in (
list(QUARTERLY_DURATION_CONCEPTS) + list(QUARTERLY_INSTANT_CONCEPTS)
)}
min_year = int(price_index.min().year) - 1
max_year = int(price_index.max().year)
quarter_ends = []
for year in range(min_year, max_year + 1):
for month, day in ((3, 31), (6, 30), (9, 30), (12, 31)):
quarter_ends.append(pd.Timestamp(year=year, month=month, day=day))
results: dict[str, list[pd.DataFrame]] = {name: [] for name in QUARTERLY_DURATION_CONCEPTS | QUARTERLY_INSTANT_CONCEPTS}
for index, quarter_end in enumerate(quarter_ends, start=1):
print(f"--- SEC quarterly frames {index}/{len(quarter_ends)}: {quarter_end.date()} ---")
for factor_name, concept_candidates in QUARTERLY_DURATION_CONCEPTS.items():
panel = pd.DataFrame(index=[quarter_end], columns=tickers, dtype=float)
for tag, unit in concept_candidates:
payload = _load_or_fetch_frame(
tag=tag,
unit=unit,
frame_code=_frame_code(quarter_end, instant=False),
data_dir=data_dir,
user_agent=user_agent,
)
series = _frame_to_series(payload, cik_to_ticker)
if not series.empty:
for ticker, value in series.items():
if pd.isna(panel.at[quarter_end, ticker]):
panel.at[quarter_end, ticker] = value
results[factor_name].append(panel)
for factor_name, concept_candidates in QUARTERLY_INSTANT_CONCEPTS.items():
panel = pd.DataFrame(index=[quarter_end], columns=tickers, dtype=float)
for tag, unit in concept_candidates:
payload = _load_or_fetch_frame(
tag=tag,
unit=unit,
frame_code=_frame_code(quarter_end, instant=True),
data_dir=data_dir,
user_agent=user_agent,
)
series = _frame_to_series(payload, cik_to_ticker)
if not series.empty:
for ticker, value in series.items():
if pd.isna(panel.at[quarter_end, ticker]):
panel.at[quarter_end, ticker] = value
results[factor_name].append(panel)
return {name: _combine_quarterly_panels(panels).reindex(columns=tickers) for name, panels in results.items()}
def quarterly_snapshot_to_daily(quarterly_df: pd.DataFrame, daily_index: pd.Index, lag_days: int) -> pd.DataFrame:
if quarterly_df.empty:
return pd.DataFrame(index=daily_index, columns=quarterly_df.columns, dtype=float)
shifted = quarterly_df.copy()
shifted.index = pd.DatetimeIndex(shifted.index) + pd.Timedelta(days=lag_days)
expanded_index = pd.DatetimeIndex(sorted(set(pd.DatetimeIndex(daily_index)).union(set(shifted.index))))
return shifted.reindex(expanded_index).ffill().reindex(daily_index)
def _xsec_rank(df: pd.DataFrame, ascending: bool = True) -> pd.DataFrame:
return df.rank(axis=1, pct=True, na_option="keep", ascending=ascending)
def build_quarterly_factor_pack(
quarterly_data: dict[str, pd.DataFrame],
close: pd.DataFrame,
lag_days: int = DEFAULT_LAG_DAYS,
) -> dict[str, pd.DataFrame]:
daily_index = close.index
shares_daily = quarterly_snapshot_to_daily(quarterly_data["shares"], daily_index, lag_days)
equity_daily = quarterly_snapshot_to_daily(quarterly_data["equity"], daily_index, lag_days)
assets_daily = quarterly_snapshot_to_daily(quarterly_data["assets"], daily_index, lag_days)
net_income_ttm = quarterly_data["net_income"].rolling(4, min_periods=4).sum()
gross_profit_ttm = quarterly_data["gross_profit"].rolling(4, min_periods=4).sum()
assets_yoy = quarterly_data["assets"].shift(4)
shares_yoy = quarterly_data["shares"].shift(4)
net_income_ttm_daily = quarterly_snapshot_to_daily(net_income_ttm, daily_index, lag_days)
gross_profit_ttm_daily = quarterly_snapshot_to_daily(gross_profit_ttm, daily_index, lag_days)
assets_yoy_daily = quarterly_snapshot_to_daily(assets_yoy, daily_index, lag_days)
shares_yoy_daily = quarterly_snapshot_to_daily(shares_yoy, daily_index, lag_days)
market_cap = close * shares_daily
book_to_market = equity_daily / market_cap.replace(0.0, np.nan)
earnings_yield = net_income_ttm_daily / market_cap.replace(0.0, np.nan)
roe = net_income_ttm_daily / equity_daily.replace(0.0, np.nan)
gross_profitability = gross_profit_ttm_daily / assets_daily.replace(0.0, np.nan)
asset_growth = -(assets_daily / assets_yoy_daily.replace(0.0, np.nan) - 1.0)
share_issuance = -(shares_daily / shares_yoy_daily.replace(0.0, np.nan) - 1.0)
factor_pack = {
"book_to_market": book_to_market,
"earnings_yield": earnings_yield,
"roe": roe,
"gross_profitability": gross_profitability,
"asset_growth": asset_growth,
"share_issuance": share_issuance,
}
ranked = {
"book_to_market": _xsec_rank(factor_pack["book_to_market"]),
"earnings_yield": _xsec_rank(factor_pack["earnings_yield"]),
"roe": _xsec_rank(factor_pack["roe"]),
"gross_profitability": _xsec_rank(factor_pack["gross_profitability"]),
"asset_growth": _xsec_rank(factor_pack["asset_growth"]),
"share_issuance": _xsec_rank(factor_pack["share_issuance"]),
}
factor_pack["composite"] = pd.concat(ranked, axis=1).T.groupby(level=1).mean().T
factor_pack["composite"] = factor_pack["composite"].shift(1)
return factor_pack
def build_exploratory_fundamental_score(
close: pd.DataFrame,
data_dir: str = "data",
lag_days: int = DEFAULT_LAG_DAYS,
user_agent: str = DEFAULT_SEC_USER_AGENT,
) -> pd.DataFrame:
quarterly = fetch_sec_quarterly_panels(
tickers=list(close.columns),
price_index=close.index,
data_dir=data_dir,
user_agent=user_agent,
)
return build_quarterly_factor_pack(quarterly, close, lag_days=lag_days)["composite"]

53
research/us_universe.py
View File

@@ -1,53 +0,0 @@
import pandas as pd
def build_tradable_mask(
close: pd.DataFrame,
volume: pd.DataFrame,
pit_membership: pd.DataFrame | None,
min_price: float,
min_dollar_volume: float,
min_history_days: int,
min_valid_volume_days: int,
liquidity_window: int = 60,
) -> pd.DataFrame:
"""Build a point-in-time tradable universe mask using only lagged inputs."""
close = close.sort_index()
volume = volume.reindex(index=close.index, columns=close.columns).sort_index()
if pit_membership is None:
pit_mask = pd.DataFrame(True, index=close.index, columns=close.columns)
else:
pit_mask = pit_membership.reindex(
index=close.index,
columns=close.columns,
fill_value=False,
)
pit_mask = pit_mask.where(pit_mask.notna(), False).astype(bool)
eligible_close = close.where(pit_mask)
eligible_volume = volume.where(pit_mask)
lagged_close = eligible_close.shift(1)
lagged_volume = eligible_volume.shift(1)
lagged_dollar_volume = lagged_close * lagged_volume
price_ok = lagged_close.gt(min_price)
liquidity_ok = (
lagged_dollar_volume.rolling(window=liquidity_window, min_periods=1).median().gt(min_dollar_volume)
)
history_ok = (
lagged_close.notna()
.rolling(window=min_history_days, min_periods=min_history_days)
.sum()
.ge(min_history_days)
)
valid_volume_ok = (
lagged_dollar_volume.notna()
.rolling(window=liquidity_window, min_periods=1)
.sum()
.ge(min_valid_volume_days)
)
mask = price_ok & liquidity_ok & history_ok & valid_volume_ok
mask = mask & pit_mask
return mask.astype(bool)

66
research/v5_drawdown_trace.py
View File

@@ -1,66 +0,0 @@
"""Trace where V3/V5 maximum drawdowns occur and what holdings they had."""
from __future__ import annotations
import os
import sys
from itertools import product
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.trend_rider_robustness import (
load_price_panel,
portfolio_returns,
)
from strategies.permanent import TrendRiderV3
from strategies.trend_rider_v5 import TrendRiderV5
def trace(name: str, weights: pd.DataFrame, prices: pd.DataFrame,
start: str = "2015-01-02") -> None:
rets = portfolio_returns(weights, prices[weights.columns], 0.001)
rets = rets[rets.index >= start]
eq = (1 + rets).cumprod()
dd = eq / eq.cummax() - 1
trough = dd.idxmin()
peak = eq.loc[:trough].idxmax()
recover = eq.loc[trough:][eq.loc[trough:] >= eq.loc[peak]]
rec_dt = recover.index[0] if len(recover) else None
print(f"\n=== {name} ===")
print(f" MDD = {dd.min()*100:.2f}%")
print(f" Peak : {peak.date()} equity={eq.loc[peak]:.3f}")
print(f" Trough: {trough.date()} equity={eq.loc[trough]:.3f}")
print(f" Recovered: {rec_dt.date() if rec_dt is not None else 'NOT YET'}")
print(f" Days to trough: {(trough - peak).days}")
# Show holdings around the drawdown
print(f"\n Holdings 5 days before peak through 5 days after trough:")
sl = weights.loc[peak - pd.Timedelta(days=10): trough + pd.Timedelta(days=10)]
nonzero = (sl != 0).any(axis=0)
sl = sl.loc[:, nonzero]
sl_disp = sl.copy()
# Show only days when holdings change
changes = (sl_disp != sl_disp.shift(1)).any(axis=1)
sl_disp = sl_disp.loc[changes]
print(sl_disp.round(3).head(40).to_string())
def main() -> None:
prices = load_price_panel()
print(f"Panel: {prices.index.min().date()} to {prices.index.max().date()}")
candidates = {
"V3 default": TrendRiderV3(),
"V5 default (panic 1.6/4%)": TrendRiderV5(),
"V5 panic 1.8/5%": TrendRiderV5(panic_vol_ratio=1.8, panic_peak_drop_pct=0.05),
}
for name, strat in candidates.items():
w = strat.generate_signals(prices)
trace(name, w, prices)
if __name__ == "__main__":
main()

185
research/v5_p0_validate.py
View File

@@ -1,185 +0,0 @@
"""P0 validation for TrendRiderV5 — walk-forward + bootstrap.
Critical question: were V5's panic-demote thresholds curve-fit to the
2024-08 carry-trade unwind? Test by optimizing on IS (2015-2020, which
does NOT contain 2024-08) and evaluating on OOS (2021-2026, which DOES).
If IS-best params still rescue the OOS drawdown, the mechanism is real.
"""
from __future__ import annotations
import os
import sys
from dataclasses import asdict
from itertools import product
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.trend_rider_robustness import (
buy_hold_weights,
evaluate_weights,
load_price_panel,
portfolio_returns,
)
from research.trend_rider_p0 import block_bootstrap, bootstrap_summary
from strategies.permanent import TrendRiderV3
from strategies.trend_rider_v5 import TrendRiderV5
IS_START = "2015-01-02"
IS_END = "2020-12-31"
OOS_START = "2021-01-01"
OOS_END = "2026-05-07"
def _fmt(x: float) -> str:
return f"{x * 100:7.2f}%"
def print_eval(label: str, ev) -> None:
print(
f" {label:<36s} "
f"CAGR {_fmt(ev.cagr)} Sharpe {ev.sharpe:5.2f} "
f"MDD {_fmt(ev.max_drawdown)} Calmar {ev.calmar:5.2f} "
f"X {ev.final_multiple:6.2f}"
)
def panic_grid() -> list[dict]:
return [
{
"panic_vol_ratio": vr,
"panic_peak_drop_pct": pd_,
"panic_vol_short": vs,
"panic_peak_window": pw,
}
for vr, pd_, vs, pw in product(
[1.4, 1.5, 1.6, 1.7, 1.8, 2.0],
[0.03, 0.04, 0.05, 0.06],
[3, 5, 7],
[3, 5, 7],
)
]
def main() -> None:
prices = load_price_panel()
print(f"Panel: {prices.index.min().date()} to {prices.index.max().date()}")
# ----- Walk-forward: choose panic config by IS Calmar (CAGR/|MDD|) -----
print("\n" + "=" * 78)
print(f"P0.1 — Walk-forward (IS panic-grid optimization → OOS test)")
print(f" IS: {IS_START} → {IS_END} (does NOT contain 2024-08 crash)")
print(f" OOS: {OOS_START} → {OOS_END}")
print("=" * 78)
grid = panic_grid()
is_rows = []
oos_rows = []
for kwargs in grid:
strat = TrendRiderV5(**kwargs)
weights = strat.generate_signals(prices)
ev_is = evaluate_weights("is", weights, prices[weights.columns],
0.001, IS_START, IS_END)
ev_oos = evaluate_weights("oos", weights, prices[weights.columns],
0.001, OOS_START, OOS_END)
is_rows.append({**asdict(ev_is), **kwargs, "scope": "IS"})
oos_rows.append({**asdict(ev_oos), **kwargs, "scope": "OOS"})
is_df = pd.DataFrame(is_rows)
oos_df = pd.DataFrame(oos_rows)
is_df["calmar"] = is_df["cagr"] / is_df["max_drawdown"].abs().replace(0.0, np.nan)
oos_df["calmar"] = oos_df["cagr"] / oos_df["max_drawdown"].abs().replace(0.0, np.nan)
# Rank by IS Calmar
is_df = is_df.sort_values("calmar", ascending=False).reset_index(drop=True)
print(f"\n Grid size: {len(grid)}, top 5 by IS Calmar:")
show_cols = ["cagr", "sharpe", "max_drawdown", "calmar",
"panic_vol_ratio", "panic_peak_drop_pct",
"panic_vol_short", "panic_peak_window"]
print(is_df[show_cols].head(5).to_string(index=False))
# IS-best by Calmar
best = is_df.iloc[0]
best_kwargs = {k: best[k] for k in
("panic_vol_ratio", "panic_peak_drop_pct",
"panic_vol_short", "panic_peak_window")}
best_kwargs["panic_vol_short"] = int(best_kwargs["panic_vol_short"])
best_kwargs["panic_peak_window"] = int(best_kwargs["panic_peak_window"])
best_kwargs["panic_vol_ratio"] = float(best_kwargs["panic_vol_ratio"])
best_kwargs["panic_peak_drop_pct"] = float(best_kwargs["panic_peak_drop_pct"])
print(f"\n IS-best (by Calmar): {best_kwargs}")
print(f" IS CAGR {best['cagr']*100:.2f}% MDD {best['max_drawdown']*100:.2f}% "
f"Calmar {best['calmar']:.2f}")
# OOS performance of IS-best
isbest_strat = TrendRiderV5(**best_kwargs)
w_isbest = isbest_strat.generate_signals(prices)
is_best_oos = evaluate_weights("is_best_OOS", w_isbest,
prices[w_isbest.columns],
0.001, OOS_START, OOS_END)
print(f" Same params, OOS performance:")
print_eval("IS-best (OOS)", is_best_oos)
# Compare with V3 default and V5 (default panic = 1.6/4%) on each window
cmp_strats = {
"V3 default": TrendRiderV3(),
"V5 default (1.6 / 4%)": TrendRiderV5(),
f"V5 IS-best (Calmar)": TrendRiderV5(**best_kwargs),
}
print("\n Comparison on full / IS / OOS:")
for window_name, (s, e) in {"FULL": (IS_START, OOS_END), "IS": (IS_START, IS_END),
"OOS": (OOS_START, OOS_END)}.items():
print(f" --- {window_name} ({s} → {e}) ---")
for n, strat in cmp_strats.items():
w = strat.generate_signals(prices)
ev = evaluate_weights(n, w, prices[w.columns], 0.001, s, e)
print_eval(n, ev)
spy_w = buy_hold_weights(prices, "SPY")
ev = evaluate_weights("SPY B&H", spy_w, prices[spy_w.columns], 0.0, s, e)
print_eval("SPY B&H", ev)
# IS-OOS decay analysis
decay_cagr = best["cagr"] - is_best_oos.cagr
print(f"\n Decay (IS-best CAGR IS → OOS): {decay_cagr*100:+.2f}%")
print(f" IS-best preserved OOS MDD: {is_best_oos.max_drawdown*100:.2f}% "
f"(V3 OOS MDD = -37.54%)")
# ----- Bootstrap on V5 default returns -----
print("\n" + "=" * 78)
print("P0.2 — Block bootstrap (V5 default, block_len=21, n_boot=5000)")
print("=" * 78)
v5 = TrendRiderV5()
weights = v5.generate_signals(prices)
rets = portfolio_returns(weights, prices[weights.columns], 0.001)
rets = rets[(rets.index >= IS_START) & (rets.index <= OOS_END)]
boot = block_bootstrap(rets, n_boot=5000, block_len=21, seed=42)
print("\n Full-sample bootstrap (2015-2026):")
print(bootstrap_summary(boot).round(4).to_string())
p_neg = float((boot["cagr"] < 0).mean())
p_below_spy = float((boot["cagr"] < 0.15).mean())
p_dd_30 = float((boot["max_drawdown"] < -0.30).mean())
p_dd_40 = float((boot["max_drawdown"] < -0.40).mean())
p_dd_50 = float((boot["max_drawdown"] < -0.50).mean())
print(f"\n P(CAGR<0) = {p_neg:.3f}")
print(f" P(CAGR<SPY 15%) = {p_below_spy:.3f}")
print(f" P(MaxDD<-30%) = {p_dd_30:.3f}")
print(f" P(MaxDD<-40%) = {p_dd_40:.3f}")
print(f" P(MaxDD<-50%) = {p_dd_50:.3f}")
rets_oos = rets[rets.index >= OOS_START]
boot_oos = block_bootstrap(rets_oos, n_boot=5000, block_len=21, seed=43)
print("\n OOS-only bootstrap (2021-2026):")
print(bootstrap_summary(boot_oos).round(4).to_string())
p_dd_30_oos = float((boot_oos["max_drawdown"] < -0.30).mean())
p_dd_40_oos = float((boot_oos["max_drawdown"] < -0.40).mean())
print(f"\n OOS P(MaxDD<-30%) = {p_dd_30_oos:.3f}")
print(f" OOS P(MaxDD<-40%) = {p_dd_40_oos:.3f}")
if __name__ == "__main__":
main()

115
research/v6_voltarget.py
View File

@@ -1,115 +0,0 @@
"""Vol-targeting overlay on V5/V6 blends — tests if dynamic exposure scaling
can lift realized Sharpe past 1.30 toward 1.50+.
The vol-target post-processor scales total weights by min(1, target_vol /
realized_vol_20d) using the strategy's *own* realized 20-day vol from the
prior backtest output. It shrinks exposure (toward cash) in high-vol
regimes — same effect as a deleveraging manager.
"""
from __future__ import annotations
import os
import sys
import numpy as np
import pandas as pd
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from research.trend_rider_robustness import (
buy_hold_weights,
evaluate_weights,
portfolio_returns,
)
from research.trend_rider_v6_eval import load_combined_panel
from strategies.permanent import ETF_UNIVERSE
from strategies.trend_rider_v5 import TrendRiderV5
from strategies.trend_rider_v6 import TrendRiderV6
IS_START = "2015-01-02"
IS_END = "2020-12-31"
OOS_START = "2021-01-01"
OOS_END = "2026-05-07"
def _fmt(x):
return f"{x*100:7.2f}%"
def vol_target_overlay(weights: pd.DataFrame, prices: pd.DataFrame,
target_vol: float, vol_window: int = 20,
lookback_lag: int = 1) -> pd.DataFrame:
"""Scale weights so realized 20-day portfolio vol ≈ target_vol.
`lookback_lag` ensures PIT-safety: scaling at row t uses vol estimate
available at end of row t-1.
"""
rets = portfolio_returns(weights, prices, transaction_cost=0.0)
realized = rets.rolling(vol_window).std(ddof=1) * np.sqrt(252)
realized = realized.shift(lookback_lag)
realized = realized.fillna(target_vol) # warmup: no scaling
scale = (target_vol / realized.replace(0.0, np.nan)).clip(upper=1.0).fillna(1.0)
out = weights.mul(scale, axis=0)
return out
def evaluate_blend(name, blend, panel, label_prefix="", txn=0.001):
rows = []
for window_name, (s, e) in {"FULL": (IS_START, OOS_END),
"IS": (IS_START, IS_END),
"OOS": (OOS_START, OOS_END)}.items():
ev = evaluate_weights(name, blend, panel[blend.columns], txn, s, e)
print(f" [{window_name}] {label_prefix}{name:<28s} "
f"CAGR {_fmt(ev.cagr)} Vol {_fmt(ev.volatility)} "
f"Sharpe {ev.sharpe:5.2f} MDD {_fmt(ev.max_drawdown)} "
f"Calmar {ev.calmar:5.2f} X {ev.final_multiple:6.2f}")
rows.append({"window": window_name, "name": name, **ev.__dict__})
return rows
def main() -> None:
panel = load_combined_panel()
etf_set = (set(ETF_UNIVERSE)
| {"QQQ", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "SPY",
"YINN", "CHAU", "7200.HK", "7500.HK"})
stock_universe = [c for c in panel.columns if c not in etf_set]
v5 = TrendRiderV5()
v6_best = TrendRiderV6(
signal_name="rec_mfilt+deep_upvol", top_n=10,
tier2_leverage_overlay=0.50,
stock_universe=stock_universe,
)
v5_w = v5.generate_signals(panel)
v6_w = v6_best.generate_signals(panel)
# Align columns
cols = sorted(set(v5_w.columns) | set(v6_w.columns))
v5_a = v5_w.reindex(columns=cols).fillna(0.0)
v6_a = v6_w.reindex(index=v5_a.index, columns=cols).fillna(0.0)
print(f"V5 vs V6 corr = {portfolio_returns(v5_a, panel[cols], 0.001).corr(portfolio_returns(v6_a, panel[cols], 0.001)):.3f}")
print("\n=== V5 + V6 blends WITH vol targeting ===")
blend_ratios = [(0.50, 0.50), (0.70, 0.30), (0.40, 0.60), (0.30, 0.70)]
targets = [0.20, 0.22, 0.25, 0.30]
for w5, w6 in blend_ratios:
blend = v5_a * w5 + v6_a * w6
for tgt in targets:
sized = vol_target_overlay(blend, panel[blend.columns], target_vol=tgt)
evaluate_blend(f"V5={w5:.0%}+V6={w6:.0%} vt{tgt:.2f}", sized, panel,
label_prefix="")
print()
# Vol target on pure V5 / V6 too
print("\n=== Pure strategies WITH vol targeting ===")
for tgt in targets:
for nm, w in [("V5", v5_a), ("V6best", v6_a)]:
sized = vol_target_overlay(w, panel[w.columns], target_vol=tgt)
evaluate_blend(f"{nm} vt{tgt:.2f}", sized, panel)
if __name__ == "__main__":
main()

612
research/v7_breakthrough_eval.py
View File

@@ -1,612 +0,0 @@
"""Three directions to break V7+VT36's ceiling (61% Ann, Sharpe 1.89).
Direction A — Multi-strategy ensemble: V7 + stock pickers, capital split.
Direction B — Cross-market V7: sector 3x ETFs (SOXL, TECL, TNA, FAS).
Direction C — Improved regime engine: alternative signals replacing MA150.
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
import universe_history as uh
from main import backtest
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
from strategies.trend_rider_v7 import TrendRiderV7
from strategies.recovery_momentum import RecoveryMomentumStrategy
from universe import UNIVERSES
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
def run_and_report(label, strategy, data_panel, capital=CAPITAL):
eq = backtest(strategy, data_panel, initial_capital=capital,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
return label, eq, m
def print_table(results: list[tuple[str, pd.Series, dict]]):
print(f"{'#':<4} {'Strategy':<52} {'Ann%':>7} {'Vol%':>7} {'Sharpe':>7} "
f"{'Sortino':>8} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 115)
for i, (label, _, m) in enumerate(results, 1):
marker = " ★" if i <= 3 else ""
print(f"{i:<4} {label:<52} "
f"{m['annualizedReturn']*100:>6.1f}% "
f"{m['annualizedVolatility']*100:>6.1f}% "
f"{m['sharpeRatio']:>7.2f} "
f"{m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>6.1f}% "
f"{m['calmarRatio']:>7.2f}{marker}")
def ensemble_equity(equities: list[pd.Series], weights: list[float] | None = None
) -> pd.Series:
"""Combine independent equity curves with periodic rebalancing.
Each equity is assumed to start at $CAPITAL.
Returns combined equity as if capital were split according to weights.
"""
if weights is None:
weights = [1.0 / len(equities)] * len(equities)
idx = equities[0].index
for eq in equities[1:]:
idx = idx.intersection(eq.index)
aligned = [eq.reindex(idx).ffill() for eq in equities]
# Combine as weighted sum of normalized curves
combined = pd.Series(0.0, index=idx)
for eq, w in zip(aligned, weights):
combined += (eq / eq.iloc[0]) * w
combined = combined * CAPITAL
return combined
# =========================================================================
# Data loading (shared)
# =========================================================================
def load_all_data():
print("=" * 100)
print(" LOADING ALL DATA")
print("=" * 100)
# S&P 500 + PIT
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
pit_intervals = uh.load_sp500_history()
hist_tickers = uh.all_tickers_ever(pit_intervals)
# All ETFs needed across all three directions
core_etfs = ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "TLT"]
sector_etfs = [
"SOXL", "SMH", # 3x semi / semi index
"TECL", "XLK", # 3x tech / tech sector
"TNA", "IWM", # 3x Russell 2000
"FAS", "XLF", # 3x financials
]
regime_etfs = ["VIX", "^VIX"] # VIX for alt regime signals
all_etfs = sorted(set(core_etfs + sector_etfs + regime_etfs))
# Stock data (includes ETFs for mixed strategies)
all_stock_tickers = sorted(set(tickers + hist_tickers + all_etfs))
print(f"\nDownloading {len(all_stock_tickers)} tickers...")
stock_data = data_manager.update("us", all_stock_tickers, with_open=False)
if isinstance(stock_data, tuple):
stock_data = stock_data[0]
cutoff = stock_data.index[-1] - pd.DateOffset(years=YEARS)
stock_data = stock_data[stock_data.index >= cutoff]
stock_data = uh.mask_prices(stock_data, pit_intervals)
# Pure ETF data
etf_data = data_manager.update("etfs", all_etfs, with_open=False)
if isinstance(etf_data, tuple):
etf_data = etf_data[0]
etf_cutoff = etf_data.index[-1] - pd.DateOffset(years=YEARS)
etf_data = etf_data[etf_data.index >= etf_cutoff]
stock_tickers = [t for t in stock_data.columns
if t not in all_etfs and stock_data[t].notna().any()]
print(f"Stocks: {len(stock_tickers)}")
print(f"Period: {stock_data.index[0].date()} → {stock_data.index[-1].date()}")
print(f"ETF columns: {sorted(etf_data.columns.tolist())}")
return stock_data, etf_data, stock_tickers, all_etfs
# =========================================================================
# DIRECTION A: Multi-strategy ensemble
# =========================================================================
def direction_a(stock_data, etf_data, stock_tickers, all_etfs):
print("\n" + "=" * 100)
print(" DIRECTION A: MULTI-STRATEGY ENSEMBLE")
print("=" * 100)
results = []
# Baselines
etf_cols = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"] if t in etf_data.columns]
label, eq_v7, m = run_and_report(
"V7+VT36 (baseline)", TrendRiderV7(target_vol=0.36, min_lev=0.75), etf_data[etf_cols])
results.append((label, eq_v7, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}")
label, eq_rec, m = run_and_report(
"RecoveryMom Top10 (baseline)", RecoveryMomentumStrategy(top_n=10), stock_data[stock_tickers])
results.append((label, eq_rec, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}")
# Ensembles with different splits
for v7_pct in (0.5, 0.6, 0.7, 0.8):
stock_pct = 1.0 - v7_pct
label = f"Ensemble {int(v7_pct*100)}% V7 + {int(stock_pct*100)}% RecMom"
eq = ensemble_equity([eq_v7, eq_rec], [v7_pct, stock_pct])
m = metrics.raw_summary(eq)
results.append((label, eq, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
# Also try V7+VT36 + V7+VT24 (low-vol variant) ensemble
label, eq_v7_lo, m = run_and_report(
"V7+VT24 (low-vol)", TrendRiderV7(target_vol=0.24, min_lev=0.50), etf_data[etf_cols])
results.append((label, eq_v7_lo, m))
eq_v7_duo = ensemble_equity([eq_v7, eq_v7_lo], [0.6, 0.4])
m = metrics.raw_summary(eq_v7_duo)
results.append(("Ensemble 60% V7-VT36 + 40% V7-VT24", eq_v7_duo, m))
print(f" V7-VT36/VT24 blend: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
results.sort(key=lambda x: x[2]["sharpeRatio"], reverse=True)
print(f"\n--- Direction A Results (sorted by Sharpe) ---")
print_table(results)
return results
# =========================================================================
# DIRECTION B: Cross-market V7 (sector 3x ETFs)
# =========================================================================
def direction_b(stock_data, etf_data, stock_tickers, all_etfs):
print("\n" + "=" * 100)
print(" DIRECTION B: CROSS-MARKET V7 (SECTOR 3x ETFs)")
print("=" * 100)
results = []
# Baseline
etf_cols = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"] if t in etf_data.columns]
label, eq_v7, m = run_and_report(
"V7+VT36 SPY→TQQQ/UPRO (baseline)", TrendRiderV7(target_vol=0.36, min_lev=0.75), etf_data[etf_cols])
results.append((label, eq_v7, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%")
# Sector V7 instances
sector_configs = [
("SMH→SOXL (Semiconductor)", "SMH", ("SOXL",)),
("XLK→TECL (Technology)", "XLK", ("TECL",)),
("IWM→TNA (Russell 2000)", "IWM", ("TNA",)),
("XLF→FAS (Financials)", "XLF", ("FAS",)),
]
sector_equities = {}
for desc, signal, risk_on in sector_configs:
needed = [signal] + list(risk_on) + ["GLD", "DBC", "SHY"]
available = [t for t in needed if t in etf_data.columns]
if signal not in available or not any(r in available for r in risk_on):
print(f" SKIP {desc}: missing data ({signal} or {risk_on})")
continue
risk_on_avail = tuple(r for r in risk_on if r in available)
strategy = TrendRiderV7(
signal=signal, risk_on=risk_on_avail, risk_off=("GLD", "DBC"),
target_vol=0.36, min_lev=0.75,
)
label = f"V7+VT36 {desc}"
try:
_, eq, m = run_and_report(label, strategy, etf_data[available])
results.append((label, eq, m))
sector_equities[desc] = eq
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
except Exception as e:
print(f" FAILED {label}: {e}")
# Cross-market ensembles
if sector_equities:
# All sectors + SPY equal weight
all_eqs = [eq_v7] + list(sector_equities.values())
eq_all = ensemble_equity(all_eqs)
m = metrics.raw_summary(eq_all)
label = f"Equal-weight all {len(all_eqs)} V7 instances"
results.append((label, eq_all, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
# Best 2-3 combinations
if "SMH→SOXL (Semiconductor)" in sector_equities:
eq_spy_semi = ensemble_equity([eq_v7, sector_equities["SMH→SOXL (Semiconductor)"]], [0.5, 0.5])
m = metrics.raw_summary(eq_spy_semi)
results.append(("50% SPY-V7 + 50% SOXL-V7", eq_spy_semi, m))
print(f" SPY+SOXL combo: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
eq_spy_semi_70 = ensemble_equity([eq_v7, sector_equities["SMH→SOXL (Semiconductor)"]], [0.7, 0.3])
m = metrics.raw_summary(eq_spy_semi_70)
results.append(("70% SPY-V7 + 30% SOXL-V7", eq_spy_semi_70, m))
if "XLK→TECL (Technology)" in sector_equities:
eq_spy_tech = ensemble_equity([eq_v7, sector_equities["XLK→TECL (Technology)"]], [0.5, 0.5])
m = metrics.raw_summary(eq_spy_tech)
results.append(("50% SPY-V7 + 50% TECL-V7", eq_spy_tech, m))
if len(sector_equities) >= 2:
# SPY + top 2 sectors
sorted_sectors = sorted(sector_equities.items(),
key=lambda x: metrics.raw_summary(x[1])["sharpeRatio"],
reverse=True)
top2 = sorted_sectors[:2]
eq_best3 = ensemble_equity([eq_v7] + [eq for _, eq in top2],
[0.5] + [0.25] * 2)
m = metrics.raw_summary(eq_best3)
label = f"50% SPY-V7 + 25% {top2[0][0][:4]}.. + 25% {top2[1][0][:4]}.."
results.append((label, eq_best3, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
results.sort(key=lambda x: x[2]["sharpeRatio"], reverse=True)
print(f"\n--- Direction B Results (sorted by Sharpe) ---")
print_table(results)
return results
# =========================================================================
# DIRECTION C: Improved regime engine
# =========================================================================
class V7AltRegime(Strategy):
"""V7 with pluggable regime function replacing V3._desired_regime."""
def __init__(
self,
regime_func,
signal: str = "SPY",
risk_on: tuple[str, ...] = ("TQQQ", "UPRO"),
risk_off: tuple[str, ...] = ("GLD", "DBC"),
target_vol: float = 0.36,
vol_window: int = 60,
min_lev: float = 0.75,
max_lev: float = 1.0,
pt_threshold: float = 0.30,
pt_band: float = 0.10,
pt_park: str = "SHY",
ma_long: int = 150,
mom_lookback: int = 63,
min_hold: int = 15,
):
self.regime_func = regime_func
self.signal = signal
self.risk_on = risk_on
self.risk_off = risk_off
self.target_vol = target_vol
self.vol_window = vol_window
self.min_lev = min_lev
self.max_lev = max_lev
self.pt_threshold = pt_threshold
self.pt_band = pt_band
self.pt_park = pt_park
self.mom_lookback = mom_lookback
self.min_hold = min_hold
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
cols = list({self.signal, *self.risk_on, *self.risk_off, self.pt_park})
cols = [c for c in cols if c in data.columns]
w = pd.DataFrame(0.0, index=data.index, columns=cols)
if self.signal not in data.columns:
return w
sig_arr = data[self.signal].to_numpy()
sym_arrays = {s: data[s].to_numpy() for s in cols if s in data.columns}
ron_syms = [s for s in self.risk_on if s in data.columns]
roff_syms = [s for s in self.risk_off if s in data.columns]
need = 252
regime: str | None = None
bars = 0
def pick_best(basket, t):
best_s, best_r = None, -np.inf
for s in basket:
arr = sym_arrays.get(s)
if arr is None or t < self.mom_lookback + 1:
continue
if arr[t-1] <= 0 or np.isnan(arr[t-1]) or arr[t - self.mom_lookback - 1] <= 0:
continue
r = arr[t-1] / arr[t - self.mom_lookback - 1] - 1.0
if np.isfinite(r) and r > best_r:
best_r, best_s = r, s
return best_s
for t in range(len(data)):
if t < need:
continue
closes = sig_arr[:t]
if np.isnan(closes[-1]):
continue
desired = self.regime_func(closes, regime)
changed = False
if regime is None:
regime, bars, changed = desired, 0, True
else:
bars += 1
if desired != regime and bars >= self.min_hold:
regime, bars, changed = desired, 0, True
if not changed and (t - need) % 21 != 0:
continue
basket = ron_syms if regime == "risk_on" else roff_syms
pick = pick_best(basket, t)
if pick:
w.iat[t, cols.index(pick)] = 1.0
w = w.replace(0.0, np.nan).ffill().fillna(0.0)
w = w.shift(1).fillna(0.0)
# Vol-target overlay
daily_ret = data[cols].pct_change(fill_method=None).fillna(0.0)
port_rets = (w * daily_ret).sum(axis=1)
realized_vol = port_rets.rolling(self.vol_window, min_periods=21).std() * np.sqrt(252)
scale = (self.target_vol / realized_vol).clip(lower=self.min_lev, upper=self.max_lev)
scale = scale.shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
# Profit-take
if self.pt_threshold <= 0:
return w
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
park_col = self.pt_park if self.pt_park in w.columns else ""
entry_price = None
current_sym = None
is_stopped = False
restore_level = self.pt_threshold - self.pt_band
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
current_sym, entry_price, is_stopped = None, None, False
continue
if sym != current_sym:
current_sym = sym
entry_price = float(data[sym].iloc[i-1]) if i > 0 and sym in data.columns else None
is_stopped = False
continue
if entry_price is None or entry_price <= 0 or sym not in data.columns:
continue
yesterday = float(data[sym].iloc[i-1]) if i > 0 else float(data[sym].iloc[i])
gain = yesterday / entry_price - 1.0
if is_stopped:
if gain < restore_level:
is_stopped = False
else:
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
else:
if gain >= self.pt_threshold:
is_stopped = True
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
return w
# Regime functions
def regime_ma(window: int):
"""Simple MA crossover: above MA → risk_on."""
def fn(closes, current):
if len(closes) < window:
return "risk_off"
return "risk_on" if closes[-1] > np.mean(closes[-window:]) else "risk_off"
return fn
def regime_dual_ma(short: int = 50, long: int = 200):
"""Golden/death cross: MA_short > MA_long → risk_on."""
def fn(closes, current):
if len(closes) < long:
return "risk_off"
ma_s = np.mean(closes[-short:])
ma_l = np.mean(closes[-long:])
return "risk_on" if ma_s > ma_l else "risk_off"
return fn
def regime_roc(window: int = 63):
"""Rate of change: positive N-day return → risk_on."""
def fn(closes, current):
if len(closes) < window + 1 or closes[-window-1] <= 0:
return "risk_off"
roc = closes[-1] / closes[-window-1] - 1.0
return "risk_on" if roc > 0 else "risk_off"
return fn
def regime_ma_plus_vol(ma_window: int = 150, vol_window: int = 20, vol_cap: float = 0.20):
"""MA + vol filter: above MA AND vol < cap → risk_on."""
def fn(closes, current):
if len(closes) < max(ma_window, vol_window + 1):
return "risk_off"
above_ma = closes[-1] > np.mean(closes[-ma_window:])
if not above_ma:
return "risk_off"
rets = np.diff(closes[-vol_window-1:]) / np.maximum(closes[-vol_window-1:-1], 1e-12)
vol = float(np.std(rets, ddof=1) * np.sqrt(252))
return "risk_on" if vol < vol_cap else "risk_off"
return fn
def regime_ma_slope(ma_window: int = 150, slope_window: int = 10):
"""MA + positive slope: above MA AND MA is rising → risk_on."""
def fn(closes, current):
if len(closes) < ma_window + slope_window:
return "risk_off"
ma_now = np.mean(closes[-ma_window:])
ma_prev = np.mean(closes[-ma_window - slope_window:-slope_window])
above = closes[-1] > ma_now
rising = ma_now > ma_prev
return "risk_on" if (above and rising) else "risk_off"
return fn
def regime_composite(ma_w: int = 150, roc_w: int = 63, vol_w: int = 20,
vol_cap: float = 0.22, threshold: int = 2):
"""Composite: score from MA + ROC + vol. Need ≥ threshold signals for risk_on."""
def fn(closes, current):
if len(closes) < max(ma_w, roc_w + 1, vol_w + 1):
return "risk_off"
score = 0
# Signal 1: above MA
if closes[-1] > np.mean(closes[-ma_w:]):
score += 1
# Signal 2: positive ROC
if closes[-roc_w-1] > 0 and closes[-1] / closes[-roc_w-1] - 1.0 > 0:
score += 1
# Signal 3: vol below cap
rets = np.diff(closes[-vol_w-1:]) / np.maximum(closes[-vol_w-1:-1], 1e-12)
vol = float(np.std(rets, ddof=1) * np.sqrt(252))
if vol < vol_cap:
score += 1
return "risk_on" if score >= threshold else "risk_off"
return fn
def regime_adaptive_ma(fast: int = 100, slow: int = 200, vol_w: int = 60,
vol_threshold: float = 0.18):
"""Adaptive MA: use fast MA in low vol, slow MA in high vol.
High vol → slower signal → fewer whipsaws."""
def fn(closes, current):
if len(closes) < slow:
return "risk_off"
rets = np.diff(closes[-vol_w-1:]) / np.maximum(closes[-vol_w-1:-1], 1e-12)
vol = float(np.std(rets, ddof=1) * np.sqrt(252))
ma_w = slow if vol > vol_threshold else fast
return "risk_on" if closes[-1] > np.mean(closes[-ma_w:]) else "risk_off"
return fn
def direction_c(stock_data, etf_data, stock_tickers, all_etfs):
print("\n" + "=" * 100)
print(" DIRECTION C: IMPROVED REGIME ENGINE")
print("=" * 100)
etf_cols = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"] if t in etf_data.columns]
results = []
# V7+VT36 baseline (uses V3's full regime with MA+vol+dd+peak gates)
label, eq, m = run_and_report(
"V7+VT36 (V3 full regime, baseline)", TrendRiderV7(target_vol=0.36, min_lev=0.75), etf_data[etf_cols])
results.append((label, eq, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}")
regime_configs = [
# Simple MA variants
("Simple MA100", regime_ma(100)),
("Simple MA150", regime_ma(150)),
("Simple MA200", regime_ma(200)),
# Dual MA
("Dual MA 50/200", regime_dual_ma(50, 200)),
("Dual MA 50/150", regime_dual_ma(50, 150)),
("Dual MA 20/100", regime_dual_ma(20, 100)),
# ROC
("ROC 63d", regime_roc(63)),
("ROC 126d", regime_roc(126)),
# MA + vol filter
("MA150 + Vol<20%", regime_ma_plus_vol(150, 20, 0.20)),
("MA150 + Vol<25%", regime_ma_plus_vol(150, 20, 0.25)),
("MA200 + Vol<20%", regime_ma_plus_vol(200, 20, 0.20)),
# MA + slope
("MA150 + Rising (10d)", regime_ma_slope(150, 10)),
("MA150 + Rising (20d)", regime_ma_slope(150, 20)),
# Composite
("Composite 2/3 (MA150+ROC63+Vol)", regime_composite(150, 63, 20, 0.22, 2)),
("Composite 3/3 (all must agree)", regime_composite(150, 63, 20, 0.22, 3)),
# Adaptive MA
("Adaptive MA100/200 (vol pivot 18%)", regime_adaptive_ma(100, 200, 60, 0.18)),
("Adaptive MA100/200 (vol pivot 22%)", regime_adaptive_ma(100, 200, 60, 0.22)),
]
for label, regime_fn in regime_configs:
try:
strategy = V7AltRegime(regime_func=regime_fn)
_, eq, m = run_and_report(f"AltRegime: {label}", strategy, etf_data[etf_cols])
results.append((f"AltRegime: {label}", eq, m))
print(f" {label}: Ann={m['annualizedReturn']*100:.1f}%, Sharpe={m['sharpeRatio']:.2f}, MaxDD={m['maxDrawdown']*100:.1f}%")
except Exception as e:
print(f" FAILED {label}: {e}")
results.sort(key=lambda x: x[2]["sharpeRatio"], reverse=True)
print(f"\n--- Direction C Results (sorted by Sharpe) ---")
print_table(results)
return results
# =========================================================================
# MAIN
# =========================================================================
def main():
stock_data, etf_data, stock_tickers, all_etfs = load_all_data()
results_a = direction_a(stock_data, etf_data, stock_tickers, all_etfs)
results_b = direction_b(stock_data, etf_data, stock_tickers, all_etfs)
results_c = direction_c(stock_data, etf_data, stock_tickers, all_etfs)
# Final summary
print("\n" + "=" * 100)
print(" CROSS-DIRECTION SUMMARY")
print("=" * 100)
all_results = (
[(f"[A] {l}", eq, m) for l, eq, m in results_a] +
[(f"[B] {l}", eq, m) for l, eq, m in results_b] +
[(f"[C] {l}", eq, m) for l, eq, m in results_c]
)
all_results.sort(key=lambda x: x[2]["sharpeRatio"], reverse=True)
print(f"\nTop 10 by Sharpe across all directions:")
print(f"{'#':<4} {'Strategy':<60} {'Ann%':>7} {'Sharpe':>7} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 100)
for i, (label, _, m) in enumerate(all_results[:10], 1):
print(f"{i:<4} {label:<60} "
f"{m['annualizedReturn']*100:>6.1f}% "
f"{m['sharpeRatio']:>7.2f} "
f"{m['maxDrawdown']*100:>6.1f}% "
f"{m['calmarRatio']:>7.2f}")
print(f"\nTop 10 by Ann. Return across all directions:")
all_results.sort(key=lambda x: x[2]["annualizedReturn"], reverse=True)
print(f"{'#':<4} {'Strategy':<60} {'Ann%':>7} {'Sharpe':>7} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 100)
for i, (label, _, m) in enumerate(all_results[:10], 1):
print(f"{i:<4} {label:<60} "
f"{m['annualizedReturn']*100:>6.1f}% "
f"{m['sharpeRatio']:>7.2f} "
f"{m['maxDrawdown']*100:>6.1f}% "
f"{m['calmarRatio']:>7.2f}")
if __name__ == "__main__":
main()

282
research/v7_breakthrough_fixed.py
View File

@@ -1,282 +0,0 @@
"""Fixed re-run for Directions B and C based on review feedback.
Direction B fix: recalibrate V3 thresholds per-sector (scale by vol ratio).
Direction C fix: monkey-patch V3._desired_regime inside real V7, preserving
the full state machine (confirm_days, cooloff, stop_loss, dd_stop).
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from main import backtest
from strategies.trend_rider_v7 import TrendRiderV7
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
def load_etf_data():
all_etfs = sorted(set([
"SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY",
"SOXL", "SMH", "TECL", "XLK", "TNA", "IWM", "FAS", "XLF",
]))
data = data_manager.update("etfs", all_etfs, with_open=False)
if isinstance(data, tuple):
data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
return data[data.index >= cutoff]
def run(label, strategy, panel):
eq = backtest(strategy, panel, initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
print(f" {label:<55} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Sortino={m['sortinoRatio']:.2f} Calmar={m['calmarRatio']:.2f}")
return label, eq, m
# =========================================================================
# DIRECTION B FIX: per-sector calibrated thresholds
# =========================================================================
def direction_b_fixed(etf_data):
print("\n" + "=" * 100)
print(" DIRECTION B FIXED: Sector V7 with recalibrated thresholds")
print("=" * 100)
results = []
core = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"] if t in etf_data.columns]
# Baseline
r = run("V7+VT36 baseline (SPY→TQQQ/UPRO)", TrendRiderV7(target_vol=0.36, min_lev=0.75), etf_data[core])
results.append(r)
eq_v7 = r[1]
# Estimate vol ratios for threshold scaling
rets = etf_data.pct_change(fill_method=None).dropna()
spy_vol = rets["SPY"].std() * np.sqrt(252) if "SPY" in rets.columns else 0.18
print(f"\n SPY realized vol: {spy_vol:.1%}")
sector_configs = [
("SMH", ("SOXL",), "Semiconductor"),
("XLK", ("TECL",), "Technology"),
("IWM", ("TNA",), "Russell 2000"),
("XLF", ("FAS",), "Financials"),
]
sector_eqs = {}
for signal, risk_on, name in sector_configs:
if signal not in etf_data.columns or risk_on[0] not in etf_data.columns:
print(f" SKIP {name}: missing data")
continue
sig_vol = rets[signal].std() * np.sqrt(252) if signal in rets.columns else spy_vol
vol_ratio = sig_vol / spy_vol
print(f" {signal} vol: {sig_vol:.1%}, ratio to SPY: {vol_ratio:.2f}")
needed = [signal] + list(risk_on) + ["GLD", "DBC", "SHY"]
panel = etf_data[[t for t in needed if t in etf_data.columns]]
# Uncalibrated (original V3 thresholds)
v7_raw = TrendRiderV7(
signal=signal, risk_on=risk_on, risk_off=("GLD", "DBC"),
target_vol=0.36, min_lev=0.75,
)
r = run(f" {name} UNCALIBRATED", v7_raw, panel)
results.append(r)
# Calibrated: scale vol/dd/peak thresholds by vol ratio
v7_cal = TrendRiderV7(
signal=signal, risk_on=risk_on, risk_off=("GLD", "DBC"),
target_vol=0.36, min_lev=0.75,
# V3 thresholds scaled by sector vol ratio
vol_enter=0.14 * vol_ratio,
vol_exit=0.20 * vol_ratio,
dd_stop=0.05 * vol_ratio,
peak_enter=0.02 * vol_ratio,
peak_exit=0.05 * vol_ratio,
)
r = run(f" {name} CALIBRATED (×{vol_ratio:.1f})", v7_cal, panel)
results.append(r)
sector_eqs[name] = r[1]
# Ensembles with calibrated sectors
if sector_eqs:
print()
for name, sec_eq in sector_eqs.items():
for v7_pct in (0.5, 0.7):
idx = eq_v7.index.intersection(sec_eq.index)
v7_a = eq_v7.reindex(idx).ffill()
sec_a = sec_eq.reindex(idx).ffill()
ens = (v7_a / v7_a.iloc[0]) * v7_pct + (sec_a / sec_a.iloc[0]) * (1 - v7_pct)
ens = ens * CAPITAL
m = metrics.raw_summary(ens)
label = f" {int(v7_pct*100)}% SPY-V7 + {int((1-v7_pct)*100)}% {name[:8]}-V7 (cal)"
print(f" {label:<55} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Sortino={m['sortinoRatio']:.2f} Calmar={m['calmarRatio']:.2f}")
results.append((label, ens, m))
return results
# =========================================================================
# DIRECTION C FIX: inject alt regime into REAL V3 state machine
# =========================================================================
def direction_c_fixed(etf_data):
print("\n" + "=" * 100)
print(" DIRECTION C FIXED: Alt regimes inside real V3 state machine")
print("=" * 100)
core = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"] if t in etf_data.columns]
results = []
# Baseline
r = run("V7+VT36 (V3 full regime, baseline)", TrendRiderV7(target_vol=0.36, min_lev=0.75), etf_data[core])
results.append(r)
# Alt regimes: monkey-patch V3._desired_regime, preserving full FSM
def make_alt_v7(regime_fn, label):
v7 = TrendRiderV7(target_vol=0.36, min_lev=0.75)
v7.v3._desired_regime = regime_fn
return v7
# --- Simple MA variants ---
for window in (100, 150, 200, 250):
def regime_ma(closes, current, w=window):
if len(closes) < w:
return "risk_off"
return "risk_on" if closes[-1] > np.mean(closes[-w:]) else "risk_off"
r = run(f"Simple MA{window}", make_alt_v7(regime_ma, f"MA{window}"), etf_data[core])
results.append(r)
# --- Dual MA crossover ---
for short, long in ((50, 200), (50, 150), (20, 100)):
def regime_dual(closes, current, s=short, l=long):
if len(closes) < l:
return "risk_off"
return "risk_on" if np.mean(closes[-s:]) > np.mean(closes[-l:]) else "risk_off"
r = run(f"Dual MA {short}/{long}", make_alt_v7(regime_dual, ""), etf_data[core])
results.append(r)
# --- ROC variants ---
for window in (42, 63, 126):
def regime_roc(closes, current, w=window):
if len(closes) < w + 1 or closes[-w-1] <= 0:
return "risk_off"
return "risk_on" if closes[-1] / closes[-w-1] > 1.0 else "risk_off"
r = run(f"ROC {window}d", make_alt_v7(regime_roc, ""), etf_data[core])
results.append(r)
# --- MA + vol filter (simplified V3) ---
for ma_w, vol_cap in ((150, 0.20), (150, 0.25), (200, 0.22)):
def regime_mavol(closes, current, mw=ma_w, vc=vol_cap):
if len(closes) < max(mw, 21):
return "risk_off"
above = closes[-1] > np.mean(closes[-mw:])
if not above:
return "risk_off"
rets = np.diff(closes[-21:]) / np.maximum(closes[-21:-1], 1e-12)
vol = float(np.std(rets, ddof=1) * np.sqrt(252))
return "risk_on" if vol < vc else "risk_off"
r = run(f"MA{ma_w} + Vol<{int(vol_cap*100)}%", make_alt_v7(regime_mavol, ""), etf_data[core])
results.append(r)
# --- Composite (MA + ROC + vol) ---
for thresh in (2, 3):
def regime_comp(closes, current, t=thresh):
if len(closes) < 200:
return "risk_off"
score = 0
if closes[-1] > np.mean(closes[-150:]):
score += 1
if closes[-64] > 0 and closes[-1] / closes[-64] > 1.0:
score += 1
rets = np.diff(closes[-21:]) / np.maximum(closes[-21:-1], 1e-12)
if np.std(rets, ddof=1) * np.sqrt(252) < 0.22:
score += 1
return "risk_on" if score >= t else "risk_off"
r = run(f"Composite {thresh}/3", make_alt_v7(regime_comp, ""), etf_data[core])
results.append(r)
# --- MA + slope (MA must be rising) ---
for slope_w in (10, 20):
def regime_slope(closes, current, sw=slope_w):
if len(closes) < 150 + sw:
return "risk_off"
ma_now = np.mean(closes[-150:])
ma_prev = np.mean(closes[-150-sw:-sw])
return "risk_on" if (closes[-1] > ma_now and ma_now > ma_prev) else "risk_off"
r = run(f"MA150 + Rising({slope_w}d)", make_alt_v7(regime_slope, ""), etf_data[core])
results.append(r)
# --- Adaptive MA (fast in low vol, slow in high vol) ---
for pivot in (0.15, 0.18, 0.22):
def regime_adapt(closes, current, p=pivot):
if len(closes) < 200:
return "risk_off"
rets = np.diff(closes[-61:]) / np.maximum(closes[-61:-1], 1e-12)
vol = np.std(rets, ddof=1) * np.sqrt(252)
w = 200 if vol > p else 100
return "risk_on" if closes[-1] > np.mean(closes[-w:]) else "risk_off"
r = run(f"Adaptive MA (pivot={int(pivot*100)}%)", make_alt_v7(regime_adapt, ""), etf_data[core])
results.append(r)
# Sort and display
results.sort(key=lambda x: x[2]["sharpeRatio"], reverse=True)
print(f"\n--- Direction C FIXED Results (sorted by Sharpe) ---")
for i, (label, _, m) in enumerate(results, 1):
marker = " ★" if i <= 3 else ""
print(f" {i:<3} {label:<55} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Calmar={m['calmarRatio']:.2f}{marker}")
return results
def main():
print("=" * 100)
print(" V7 BREAKTHROUGH EVAL — FIXED RE-RUN (per review feedback)")
print("=" * 100)
etf_data = load_etf_data()
print(f"Period: {etf_data.index[0].date()} → {etf_data.index[-1].date()}")
print(f"ETFs: {sorted(etf_data.columns.tolist())}")
results_b = direction_b_fixed(etf_data)
results_c = direction_c_fixed(etf_data)
# Cross-direction top 10
all_r = [(f"[B] {l}", eq, m) for l, eq, m in results_b] + \
[(f"[C] {l}", eq, m) for l, eq, m in results_c]
all_r.sort(key=lambda x: x[2]["sharpeRatio"], reverse=True)
print(f"\n{'=' * 100}")
print(" FINAL: Top 10 by Sharpe")
print(f"{'=' * 100}")
for i, (label, _, m) in enumerate(all_r[:10], 1):
print(f" {i:<3} {label:<60} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Calmar={m['calmarRatio']:.2f}")
all_r.sort(key=lambda x: x[2]["annualizedReturn"], reverse=True)
print(f"\n FINAL: Top 10 by Ann. Return")
print(f" {'-' * 95}")
for i, (label, _, m) in enumerate(all_r[:10], 1):
print(f" {i:<3} {label:<60} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Calmar={m['calmarRatio']:.2f}")
if __name__ == "__main__":
main()

401
research/v7_literature_alpha.py
View File

@@ -1,401 +0,0 @@
"""Literature-informed alpha research: can we beat V7+VT36?
Grounded in specific academic/industry research:
1. VIX regime overlay — Simon & Campasano (2014): VIX level as exogenous fear signal
2. Kelly-optimal sizing — Kelly (1956), Thorp (2006): return-aware position sizing
3. Multi-timeframe voting — Faber (2007): multiple MAs reduce false signals
4. Cross-asset confirmation — Asness et al. (2013): correlated asset agreement
5. Momentum acceleration — Moskowitz et al. (2012): 2nd derivative of trend
6. VIX mean-reversion entry — Whaley (2009): buy panic, sell complacency
7. Carry-enhanced risk-off — Koijen et al. (2018): hold yield during defensive periods
8. Regime-dependent PT — Optimal stopping theory: vol-drag-aware thresholds
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
from strategies.trend_rider_v7 import TrendRiderV7
from main import backtest
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
class V7Enhanced(Strategy):
"""V7 with pluggable regime enhancer and sizing model."""
def __init__(
self,
regime_enhancer=None,
sizing_model="vol_target",
pt_model="fixed",
target_vol=0.36, min_lev=0.75, max_lev=1.0,
pt_threshold=0.30, pt_band=0.10, pt_park="SHY",
ma_long=150, **v3_kw,
):
self.regime_enhancer = regime_enhancer
self.sizing_model = sizing_model
self.pt_model = pt_model
self.target_vol = target_vol
self.min_lev = min_lev
self.max_lev = max_lev
self.pt_threshold = pt_threshold
self.pt_band = pt_band
self.pt_park = pt_park
self.v3 = TrendRiderV3(
signal="SPY", risk_on=("TQQQ", "UPRO"), risk_off=("GLD", "DBC"),
ma_long=ma_long, **v3_kw,
)
def generate_signals(self, data):
w = self.v3.generate_signals(data)
if self.pt_park and self.pt_park in data.columns and self.pt_park not in w.columns:
w[self.pt_park] = 0.0
# Regime enhancement: override V3's decision in specific conditions
if self.regime_enhancer:
w = self.regime_enhancer(w, data)
# Sizing
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
common = w.columns.intersection(daily_ret.columns)
port_rets = (w[common] * daily_ret[common]).sum(axis=1)
if self.sizing_model == "kelly":
# Kelly: scale = E[r] / Var[r], clipped
roll_mean = port_rets.rolling(60, min_periods=21).mean() * 252
roll_var = port_rets.rolling(60, min_periods=21).var() * 252
kelly_f = (roll_mean / roll_var.clip(lower=0.01)).clip(-1, 2)
scale = kelly_f.clip(lower=self.min_lev, upper=self.max_lev)
scale = scale.shift(1).fillna(1.0)
else:
realized_vol = port_rets.rolling(60, min_periods=21).std() * np.sqrt(252)
scale = (self.target_vol / realized_vol).clip(
lower=self.min_lev, upper=self.max_lev)
scale = scale.shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
# Profit-take
if self.pt_threshold <= 0:
return w
risk_on_set = set(self.v3.risk_on)
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
park_col = self.pt_park if self.pt_park in w.columns else ""
ep, cs, stopped = None, None, False
rl = self.pt_threshold - self.pt_band
if self.pt_model == "vol_adaptive":
# PT threshold inversely proportional to vol drag
# Vol drag ≈ leverage² × σ² / 2; for 3x: 9σ²/2
# Optimal PT ≈ base / (1 + k * σ²)
realized_vol_arr = port_rets.rolling(60, min_periods=21).std().to_numpy() * np.sqrt(252)
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
cs, ep, stopped = None, None, False
continue
if sym != cs:
cs = sym
ep = float(data[sym].iloc[i-1]) if i > 0 and sym in data.columns else None
stopped = False
continue
if sym not in risk_on_set:
continue
if ep is None or ep <= 0 or sym not in data.columns:
continue
y = float(data[sym].iloc[i-1]) if i > 0 else float(data[sym].iloc[i])
g = y / ep - 1.0
if self.pt_model == "vol_adaptive":
rv = realized_vol_arr[i] if i < len(realized_vol_arr) and not np.isnan(realized_vol_arr[i]) else 0.25
# Higher vol → lower threshold (take profits faster)
t = self.pt_threshold * (0.25 / max(rv, 0.10))
t = np.clip(t, 0.15, 0.50)
r = t * (1 - self.pt_band / self.pt_threshold)
else:
t = self.pt_threshold
r = rl
if stopped:
if g < r: stopped = False
else:
w.iloc[i] = 0.0
if park_col: w.at[w.index[i], park_col] = scale.iloc[i]
elif g >= t:
stopped = True
w.iloc[i] = 0.0
if park_col: w.at[w.index[i], park_col] = scale.iloc[i]
return w
# =========================================================================
# Regime enhancers
# =========================================================================
def vix_overlay(vix_high=25, vix_low=15):
"""Force risk-off when VIX > threshold. Simon & Campasano (2014)."""
def enhancer(w, data):
if "^VIX" not in data.columns:
return w
vix = data["^VIX"].shift(1).fillna(20)
risk_on_cols = [c for c in ["TQQQ", "UPRO"] if c in w.columns]
risk_off_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
park = "SHY" if "SHY" in w.columns else ""
for i in range(len(w)):
v = vix.iloc[i]
if np.isnan(v): continue
ron_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in risk_on_cols)
if ron_w > 0.01 and v > vix_high:
for c in risk_on_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if risk_off_cols:
w.iat[i, w.columns.get_loc(risk_off_cols[0])] = ron_w
return w
return enhancer
def multi_timeframe(windows=(50, 150, 200), min_agree=2):
"""Multi-MA voting. Faber (2007). Need majority of MAs bullish."""
def enhancer(w, data):
if "SPY" not in data.columns:
return w
spy = data["SPY"]
votes = pd.DataFrame(index=data.index)
for win in windows:
ma = spy.rolling(win).mean()
votes[f"ma{win}"] = (spy > ma).astype(int)
total_votes = votes.sum(axis=1).shift(2) # PIT: shift 2 to match V3
risk_on_cols = [c for c in ["TQQQ", "UPRO"] if c in w.columns]
risk_off_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
for i in range(len(w)):
ron_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in risk_on_cols)
if ron_w > 0.01 and total_votes.iloc[i] < min_agree:
for c in risk_on_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if risk_off_cols:
w.iat[i, w.columns.get_loc(risk_off_cols[0])] = ron_w
return w
return enhancer
def cross_asset_confirm():
"""Require both SPY and QQQ trends to agree. Asness et al. (2013)."""
def enhancer(w, data):
if "SPY" not in data.columns or "QQQ" not in data.columns:
return w
spy_bull = (data["SPY"] > data["SPY"].rolling(150).mean()).shift(2).fillna(False)
qqq_bull = (data["QQQ"] > data["QQQ"].rolling(150).mean()).shift(2).fillna(False)
both_bull = spy_bull & qqq_bull
risk_on_cols = [c for c in ["TQQQ", "UPRO"] if c in w.columns]
risk_off_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
for i in range(len(w)):
ron_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in risk_on_cols)
if ron_w > 0.01 and not both_bull.iloc[i]:
for c in risk_on_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if risk_off_cols:
w.iat[i, w.columns.get_loc(risk_off_cols[0])] = ron_w
return w
return enhancer
def momentum_accel(accel_window=20):
"""Only risk-on when trend is accelerating. Moskowitz et al. (2012)."""
def enhancer(w, data):
if "SPY" not in data.columns:
return w
spy = data["SPY"]
ma150 = spy.rolling(150).mean()
ma_slope = ma150.diff(accel_window)
accel_positive = (ma_slope > 0).shift(2).fillna(False)
risk_on_cols = [c for c in ["TQQQ", "UPRO"] if c in w.columns]
risk_off_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
for i in range(len(w)):
ron_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in risk_on_cols)
if ron_w > 0.01 and not accel_positive.iloc[i]:
for c in risk_on_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if risk_off_cols:
w.iat[i, w.columns.get_loc(risk_off_cols[0])] = ron_w
return w
return enhancer
def vix_mean_revert_entry(vix_spike=30, lookback=5):
"""After VIX spike + revert, force risk-on. Whaley (2009) mean-reversion."""
def enhancer(w, data):
if "^VIX" not in data.columns:
return w
vix = data["^VIX"].shift(1).fillna(20)
vix_was_high = vix.rolling(lookback).max() > vix_spike
vix_now_falling = vix < vix.rolling(lookback).mean()
buy_signal = vix_was_high & vix_now_falling
risk_on_cols = [c for c in ["TQQQ", "UPRO"] if c in w.columns]
risk_off_cols = [c for c in ["GLD", "DBC"] if c in w.columns]
for i in range(len(w)):
roff_w = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in risk_off_cols)
if roff_w > 0.01 and buy_signal.iloc[i]:
for c in risk_off_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if risk_on_cols:
w.iat[i, w.columns.get_loc(risk_on_cols[0])] = roff_w
return w
return enhancer
def combined_enhancer(*enhancers):
"""Chain multiple enhancers."""
def enhancer(w, data):
for e in enhancers:
w = e(w, data)
return w
return enhancer
# =========================================================================
# Main
# =========================================================================
def main():
print("=" * 100)
print(" LITERATURE-INFORMED ALPHA RESEARCH")
print("=" * 100)
all_etfs = sorted(set([
"SPY", "QQQ", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "TLT",
"^VIX",
]))
data = data_manager.update("etfs", all_etfs, with_open=False)
if isinstance(data, tuple):
data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
data = data[data.index >= cutoff]
has_vix = "^VIX" in data.columns
has_qqq = "QQQ" in data.columns
print(f"Period: {data.index[0].date()} → {data.index[-1].date()}")
print(f"VIX available: {has_vix}, QQQ available: {has_qqq}")
results = []
def run(label, strategy):
eq = backtest(strategy, data, initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
results.append((label, m))
print(f" {label:<55} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Calmar={m['calmarRatio']:.2f}")
# Baseline
print("\n--- Baseline ---")
run("V7+VT36 baseline", V7Enhanced())
# === Idea 1: VIX overlay ===
print("\n--- Idea 1: VIX regime overlay (Simon & Campasano 2014) ---")
if has_vix:
for hi in (20, 25, 30):
run(f"VIX overlay (force off >VIX{hi})", V7Enhanced(regime_enhancer=vix_overlay(hi)))
else:
print(" VIX not available")
# === Idea 2: Kelly sizing ===
print("\n--- Idea 2: Kelly-optimal sizing (Kelly 1956, Thorp 2006) ---")
run("Kelly sizing", V7Enhanced(sizing_model="kelly"))
run("Kelly + VIX>25", V7Enhanced(sizing_model="kelly",
regime_enhancer=vix_overlay(25) if has_vix else None))
# === Idea 3: Multi-timeframe voting ===
print("\n--- Idea 3: Multi-MA voting (Faber 2007) ---")
run("Multi-MA 2/3 (50,150,200)", V7Enhanced(regime_enhancer=multi_timeframe()))
run("Multi-MA 3/3 (all agree)", V7Enhanced(regime_enhancer=multi_timeframe(min_agree=3)))
# === Idea 4: Cross-asset confirmation ===
print("\n--- Idea 4: Cross-asset (Asness et al. 2013) ---")
if has_qqq:
run("SPY+QQQ both bullish", V7Enhanced(regime_enhancer=cross_asset_confirm()))
# === Idea 5: Momentum acceleration ===
print("\n--- Idea 5: Momentum acceleration (Moskowitz et al. 2012) ---")
for w in (10, 20, 40):
run(f"MA150 slope rising ({w}d)", V7Enhanced(regime_enhancer=momentum_accel(w)))
# === Idea 6: VIX mean-reversion entry ===
print("\n--- Idea 6: VIX mean-reversion entry (Whaley 2009) ---")
if has_vix:
for spike in (25, 30, 35):
run(f"VIX spike>{spike} + revert → buy",
V7Enhanced(regime_enhancer=vix_mean_revert_entry(spike)))
# === Idea 7: Vol-adaptive PT ===
print("\n--- Idea 7: Vol-drag-aware PT (optimal stopping theory) ---")
run("Vol-adaptive PT (base=30%)", V7Enhanced(pt_model="vol_adaptive"))
run("Vol-adaptive PT (base=35%)", V7Enhanced(pt_model="vol_adaptive", pt_threshold=0.35))
# === Idea 8: Combined best ideas ===
print("\n--- Idea 8: Combinations ---")
if has_vix:
run("VIX>25 + multi-MA 2/3",
V7Enhanced(regime_enhancer=combined_enhancer(
vix_overlay(25), multi_timeframe())))
run("VIX>25 + cross-asset",
V7Enhanced(regime_enhancer=combined_enhancer(
vix_overlay(25), cross_asset_confirm())) if has_qqq else None)
run("VIX>30 + accel(20d)",
V7Enhanced(regime_enhancer=combined_enhancer(
vix_overlay(30), momentum_accel(20))))
# VIX mean-revert + normal V3
run("V7 + VIX mean-revert entry (>30)",
V7Enhanced(regime_enhancer=vix_mean_revert_entry(30)))
# === Idea 9: Different MA for V3 regime ===
print("\n--- Idea 9: Alternative MA windows ---")
for ma in (100, 120, 130, 150, 170, 200):
run(f"V3 MA{ma} + VT36", V7Enhanced(ma_long=ma))
# Final ranking
results.sort(key=lambda x: x[1]["sharpeRatio"], reverse=True)
print(f"\n{'=' * 110}")
print(" FINAL RANKING (by Sharpe)")
print(f"{'=' * 110}")
print(f"{'#':<4} {'Strategy':<55} {'Ann%':>6} {'Vol%':>6} {'Sharpe':>7} "
f"{'Sortino':>8} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 110)
for i, (label, m) in enumerate(results, 1):
marker = " ★" if i <= 3 else ""
print(f"{i:<4} {label:<55} "
f"{m['annualizedReturn']*100:>5.1f}% "
f"{m['annualizedVolatility']*100:>5.1f}% "
f"{m['sharpeRatio']:>7.2f} {m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>6.1f}% {m['calmarRatio']:>7.2f}{marker}")
print(f"{'=' * 110}")
# Top by Ann Return
results.sort(key=lambda x: x[1]["annualizedReturn"], reverse=True)
print(f"\n Top 5 by Ann Return:")
for i, (label, m) in enumerate(results[:5], 1):
print(f" {i}. {label:<50} Ann={m['annualizedReturn']*100:.1f}% "
f"Sharpe={m['sharpeRatio']:.2f}")
if __name__ == "__main__":
main()

382
research/v7_parameter_sweep.py
View File

@@ -1,382 +0,0 @@
"""V7 parameter sweep: vol-target range + adaptive profit-take variants.
Direction 1: higher vol-target (VT24 → VT48)
Direction 3: adaptive profit-take (vol-scaled, time-decay, combined)
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from main import backtest
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
ETF_TICKERS = ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"]
# ---------------------------------------------------------------------------
# Adaptive V7: modular profit-take that accepts a callable threshold
# ---------------------------------------------------------------------------
class TrendRiderV7Adaptive(Strategy):
"""V7 with pluggable profit-take logic.
pt_func(gain, realized_vol, days_held) -> (threshold, restore_level)
If pt_func is None, no profit-take is applied.
"""
def __init__(
self,
ma_long: int = 150,
signal: str = "SPY",
risk_on: tuple[str, ...] = ("TQQQ", "UPRO"),
risk_off: tuple[str, ...] = ("GLD", "DBC"),
target_vol: float = 0.28,
vol_window: int = 60,
min_lev: float = 0.6,
max_lev: float = 1.0,
pt_func=None,
pt_park: str = "SHY",
**v3_kwargs,
) -> None:
self.target_vol = target_vol
self.vol_window = vol_window
self.min_lev = min_lev
self.max_lev = max_lev
self.pt_func = pt_func
self.pt_park = pt_park
self.v3 = TrendRiderV3(
signal=signal, risk_on=risk_on, risk_off=risk_off,
ma_long=ma_long, **v3_kwargs,
)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
w = self.v3.generate_signals(data)
# Vol-target overlay
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
port_rets = (w * daily_ret).sum(axis=1)
realized_vol = (
port_rets.rolling(self.vol_window, min_periods=21).std()
* np.sqrt(252)
)
scale = (self.target_vol / realized_vol).clip(
lower=self.min_lev, upper=self.max_lev,
)
scale = scale.shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
if self.pt_func is None:
return w
# Adaptive profit-take
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
park_col = self.pt_park if self.pt_park in w.columns else ""
entry_price: float | None = None
current_sym: str | None = None
is_stopped = False
days_held = 0
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
current_sym = None
entry_price = None
is_stopped = False
days_held = 0
continue
if sym != current_sym:
current_sym = sym
entry_price = (
float(data[sym].iloc[i - 1])
if i > 0 and sym in data.columns else None
)
is_stopped = False
days_held = 0
continue
days_held += 1
if entry_price is None or entry_price <= 0 or sym not in data.columns:
continue
yesterday = float(data[sym].iloc[i - 1]) if i > 0 else float(data[sym].iloc[i])
gain = yesterday / entry_price - 1.0
rv = float(realized_vol.iloc[i]) if not np.isnan(realized_vol.iloc[i]) else 0.25
threshold, restore_level = self.pt_func(gain, rv, days_held)
if is_stopped:
if gain < restore_level:
is_stopped = False
else:
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
else:
if gain >= threshold:
is_stopped = True
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
return w
# ---------------------------------------------------------------------------
# Profit-take function factories
# ---------------------------------------------------------------------------
def fixed_pt(threshold: float, band: float):
"""Classic fixed threshold (V7 default)."""
def fn(gain, rv, days_held):
return threshold, threshold - band
return fn
def vol_adaptive_pt(base_threshold: float = 0.30, base_vol: float = 0.25,
band_ratio: float = 0.33, lo: float = 0.15, hi: float = 0.50):
"""Threshold scales inversely with realized vol.
High vol → lower threshold (harvest earlier, vol drag is worse).
Low vol → higher threshold (let profits run, drag is mild).
"""
def fn(gain, rv, days_held):
rv = max(rv, 0.05)
t = np.clip(base_threshold * (base_vol / rv), lo, hi)
return t, t * (1 - band_ratio)
return fn
def time_decay_pt(start_threshold: float = 0.40, end_threshold: float = 0.18,
decay_days: int = 120, band_ratio: float = 0.33):
"""Threshold decays linearly over holding period.
Rationale: longer holds accumulate more vol drag → take profits earlier.
"""
def fn(gain, rv, days_held):
frac = min(days_held / decay_days, 1.0)
t = start_threshold - frac * (start_threshold - end_threshold)
return t, t * (1 - band_ratio)
return fn
def combined_pt(base_threshold: float = 0.30, base_vol: float = 0.25,
time_decay_rate: float = 0.0005, min_threshold: float = 0.12,
max_threshold: float = 0.50, band_ratio: float = 0.33):
"""Vol-adaptive + time decay combined."""
def fn(gain, rv, days_held):
rv = max(rv, 0.05)
vol_adj = base_threshold * (base_vol / rv)
time_adj = vol_adj - days_held * time_decay_rate
t = np.clip(time_adj, min_threshold, max_threshold)
return t, t * (1 - band_ratio)
return fn
def trailing_stop_pt(initial_threshold: float = 0.30, trail_pct: float = 0.15,
band_ratio: float = 0.33):
"""Once gain exceeds threshold, switch to trailing stop from peak gain.
Lets winners run further but protects from reversal.
"""
# We need state across calls, so use a mutable closure
state = {"peak_gain": 0.0, "trailing_active": False}
def fn(gain, rv, days_held):
if days_held == 1:
state["peak_gain"] = 0.0
state["trailing_active"] = False
if state["trailing_active"]:
state["peak_gain"] = max(state["peak_gain"], gain)
trail_level = state["peak_gain"] * (1 - trail_pct)
if gain < trail_level:
return -1.0, -1.0 # trigger immediately
return float("inf"), float("inf") # don't trigger via threshold
else:
if gain >= initial_threshold:
state["trailing_active"] = True
state["peak_gain"] = gain
return float("inf"), float("inf")
return initial_threshold, initial_threshold * (1 - band_ratio)
return fn
# ---------------------------------------------------------------------------
# Main sweep
# ---------------------------------------------------------------------------
def main():
print("=" * 100)
print(" V7 PARAMETER SWEEP: Vol-Target + Adaptive Profit-Take")
print("=" * 100)
# Load ETF data
print("\n[1] Loading ETF data...")
etf_data = data_manager.update("etfs", ETF_TICKERS, with_open=False)
if isinstance(etf_data, tuple):
etf_data = etf_data[0]
cutoff = etf_data.index[-1] - pd.DateOffset(years=YEARS)
etf_data = etf_data[etf_data.index >= cutoff]
tradable = [t for t in ETF_TICKERS if t in etf_data.columns]
print(f" Period: {etf_data.index[0].date()} → {etf_data.index[-1].date()}")
results: list[tuple[str, str, dict]] = []
def run(group: str, label: str, strategy: Strategy):
eq = backtest(strategy, etf_data[tradable], initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
results.append((group, label, m))
print(f" {label:<45} Ann={m['annualizedReturn']*100:.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}%")
# =====================================================================
# SWEEP 1: Vol-target range (with fixed PT30)
# =====================================================================
print("\n[2] Sweep 1: Vol-target range (PT30 fixed)")
print("-" * 70)
vt_configs = [
("VT20", 0.20, 0.45),
("VT24", 0.24, 0.50),
("VT28 (default)", 0.28, 0.60),
("VT32", 0.32, 0.70),
("VT36", 0.36, 0.75),
("VT40", 0.40, 0.80),
("VT44", 0.44, 0.85),
("VT48", 0.48, 0.90),
("No VT (raw V3+PT30)", 0.28, 1.0), # min_lev=max_lev=1.0 → no scaling
]
for label, tv, ml in vt_configs:
if label.startswith("No VT"):
s = TrendRiderV7Adaptive(target_vol=1.0, min_lev=1.0, max_lev=1.0,
pt_func=fixed_pt(0.30, 0.10))
else:
s = TrendRiderV7Adaptive(target_vol=tv, min_lev=ml,
pt_func=fixed_pt(0.30, 0.10))
run("VT sweep", label, s)
# =====================================================================
# SWEEP 2: Profit-take variants (using best VT from sweep 1)
# =====================================================================
print("\n[3] Sweep 2: Profit-take variants (VT32)")
print("-" * 70)
best_vt = 0.32
best_ml = 0.70
pt_configs: list[tuple[str, object]] = [
# Fixed thresholds
("No PT (ablation)", None),
("Fixed PT15 band=5", fixed_pt(0.15, 0.05)),
("Fixed PT20 band=8", fixed_pt(0.20, 0.08)),
("Fixed PT25 band=10", fixed_pt(0.25, 0.10)),
("Fixed PT30 band=10 (default)", fixed_pt(0.30, 0.10)),
("Fixed PT35 band=12", fixed_pt(0.35, 0.12)),
("Fixed PT40 band=15", fixed_pt(0.40, 0.15)),
("Fixed PT50 band=15", fixed_pt(0.50, 0.15)),
# Vol-adaptive
("Vol-adaptive (base=30%, lo=15%)", vol_adaptive_pt(0.30, 0.25, 0.33, 0.15, 0.50)),
("Vol-adaptive (base=25%, lo=12%)", vol_adaptive_pt(0.25, 0.25, 0.33, 0.12, 0.45)),
("Vol-adaptive (base=35%, lo=18%)", vol_adaptive_pt(0.35, 0.25, 0.33, 0.18, 0.55)),
# Time-decay
("Time-decay (40%→18%, 120d)", time_decay_pt(0.40, 0.18, 120)),
("Time-decay (35%→15%, 90d)", time_decay_pt(0.35, 0.15, 90)),
("Time-decay (45%→20%, 150d)", time_decay_pt(0.45, 0.20, 150)),
# Combined
("Combined vol+time (base=30%)", combined_pt(0.30, 0.25, 0.0005, 0.12, 0.50)),
("Combined vol+time (base=25%)", combined_pt(0.25, 0.25, 0.0005, 0.10, 0.45)),
]
for label, pt_fn in pt_configs:
s = TrendRiderV7Adaptive(target_vol=best_vt, min_lev=best_ml,
pt_func=pt_fn)
run("PT sweep", label, s)
# =====================================================================
# SWEEP 3: Best PT × VT grid (narrow search around top combos)
# =====================================================================
print("\n[4] Sweep 3: Best combos (VT × PT grid)")
print("-" * 70)
grid = [
(0.32, 0.70, "Fixed PT30", fixed_pt(0.30, 0.10)),
(0.36, 0.75, "Fixed PT30", fixed_pt(0.30, 0.10)),
(0.40, 0.80, "Fixed PT30", fixed_pt(0.30, 0.10)),
(0.32, 0.70, "Fixed PT25", fixed_pt(0.25, 0.10)),
(0.36, 0.75, "Fixed PT25", fixed_pt(0.25, 0.10)),
(0.40, 0.80, "Fixed PT25", fixed_pt(0.25, 0.10)),
(0.32, 0.70, "Vol-adapt 30%", vol_adaptive_pt(0.30, 0.25, 0.33, 0.15, 0.50)),
(0.36, 0.75, "Vol-adapt 30%", vol_adaptive_pt(0.30, 0.25, 0.33, 0.15, 0.50)),
(0.40, 0.80, "Vol-adapt 30%", vol_adaptive_pt(0.30, 0.25, 0.33, 0.15, 0.50)),
(0.32, 0.70, "Time-decay 40→18", time_decay_pt(0.40, 0.18, 120)),
(0.36, 0.75, "Time-decay 40→18", time_decay_pt(0.40, 0.18, 120)),
(0.40, 0.80, "Time-decay 40→18", time_decay_pt(0.40, 0.18, 120)),
(0.32, 0.70, "Combined 30%", combined_pt(0.30, 0.25, 0.0005, 0.12, 0.50)),
(0.36, 0.75, "Combined 30%", combined_pt(0.30, 0.25, 0.0005, 0.12, 0.50)),
(0.40, 0.80, "Combined 30%", combined_pt(0.30, 0.25, 0.0005, 0.12, 0.50)),
]
for tv, ml, pt_label, pt_fn in grid:
label = f"VT{int(tv*100)} + {pt_label}"
s = TrendRiderV7Adaptive(target_vol=tv, min_lev=ml, pt_func=pt_fn)
run("Grid", label, s)
# =====================================================================
# Final ranking
# =====================================================================
results.sort(key=lambda x: x[2]["annualizedReturn"], reverse=True)
print(f"\n{'=' * 115}")
print(" FINAL RANKING (sorted by annualized return)")
print(f"{'=' * 115}")
print(f"{'#':<4} {'Group':<12} {'Config':<45} {'Ann%':>7} {'Vol%':>7} {'Sharpe':>7} "
f"{'Sortino':>8} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 115)
for i, (group, label, m) in enumerate(results, 1):
ann = m["annualizedReturn"] * 100
vol = m["annualizedVolatility"] * 100
sr = m["sharpeRatio"]
so = m["sortinoRatio"]
dd = m["maxDrawdown"] * 100
ca = m["calmarRatio"]
marker = " ★" if i <= 3 else ""
print(f"{i:<4} {group:<12} {label:<45} {ann:>6.1f}% {vol:>6.1f}% {sr:>7.2f} "
f"{so:>8.2f} {dd:>6.1f}% {ca:>7.2f}{marker}")
print(f"{'=' * 115}")
# Highlight top by Sharpe
by_sharpe = sorted(results, key=lambda x: x[2]["sharpeRatio"], reverse=True)
print("\nTop 5 by Sharpe:")
for i, (group, label, m) in enumerate(by_sharpe[:5], 1):
print(f" {i}. {label:<45} Sharpe={m['sharpeRatio']:.3f} "
f"Ann={m['annualizedReturn']*100:.1f}% MaxDD={m['maxDrawdown']*100:.1f}%")
by_calmar = sorted(results, key=lambda x: x[2]["calmarRatio"], reverse=True)
print("\nTop 5 by Calmar:")
for i, (group, label, m) in enumerate(by_calmar[:5], 1):
print(f" {i}. {label:<45} Calmar={m['calmarRatio']:.3f} "
f"Ann={m['annualizedReturn']*100:.1f}% MaxDD={m['maxDrawdown']*100:.1f}%")
if __name__ == "__main__":
main()

461
research/v7_synthetic_leverage_eval.py
View File

@@ -1,461 +0,0 @@
"""Direction 2: V7 regime + synthetic 2x/3x leveraged individual stocks.
Hypothesis: replacing TQQQ/UPRO with synthetic 2x-leveraged top-momentum
S&P 500 stocks could beat V7 by combining stock-picking alpha with leverage.
Synthetic leverage model:
daily_return_Nx = N * stock_daily_return - (N-1) * daily_borrow_cost
daily_borrow_cost ≈ risk_free_rate / 252 (conservative: 5% annualized)
This captures:
- Leverage amplification
- Financing cost
- Volatility drag (emerges naturally from daily compounding of leveraged returns)
Variants tested:
A. V7 regime + synth 2x top-5 momentum stocks
B. V7 regime + synth 2x top-10 momentum stocks
C. V7 regime + synth 2x top-1 momentum stock (concentrated)
D. V7 regime + synth 3x top-5 (compare to real TQQQ)
E. V7 regime + synth 2x recovery-momentum top-5
F. V7+VT36 baseline (current SOTA)
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
import universe_history as uh
from main import backtest
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
from universe import UNIVERSES
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
BORROW_RATE = 0.05 # 5% annualized
# ---------------------------------------------------------------------------
# Synthetic leveraged returns
# ---------------------------------------------------------------------------
def synthetic_leveraged_prices(prices: pd.DataFrame, leverage: float,
borrow_rate: float = BORROW_RATE) -> pd.DataFrame:
"""Create synthetic leveraged price series from daily returns.
Models daily-rebalanced leverage: each day's return is
r_lev = leverage * r_stock - (leverage - 1) * r_borrow
where r_borrow = borrow_rate / 252.
This captures vol drag naturally (daily compounding of amplified returns).
"""
daily_ret = prices.pct_change(fill_method=None).fillna(0.0)
daily_borrow = borrow_rate / 252
lev_ret = leverage * daily_ret - (leverage - 1) * daily_borrow
lev_prices = (1 + lev_ret).cumprod() * 100 # normalize to 100 start
lev_prices.iloc[0] = 100
return lev_prices
# ---------------------------------------------------------------------------
# Strategy: V7 regime + synthetic leveraged stock picking
# ---------------------------------------------------------------------------
class V7SynthLeverage(Strategy):
"""V7 architecture with synthetic leveraged individual stocks as risk-on.
Layer 1: V3 regime engine on SPY → risk-on vs risk-off
Layer 2: Vol-target overlay
Layer 3: Profit-take with hysteresis
Risk-on: top-N stocks by momentum, synthetically leveraged, equal weight.
Risk-off: momentum leader of (GLD, DBC).
"""
def __init__(
self,
stock_tickers: list[str],
leverage: float = 2.0,
top_n: int = 5,
signal: str = "SPY",
defensive: tuple[str, ...] = ("GLD", "DBC"),
# Momentum ranking
mom_lookback: int = 63,
rebal_every: int = 21,
# Selection method
selection: str = "momentum", # "momentum" or "recovery_momentum"
recovery_window: int = 63,
long_mom_lookback: int = 252,
long_mom_skip: int = 21,
# V3 regime
ma_long: int = 150,
# Vol-target
target_vol: float = 0.36,
vol_window: int = 60,
min_lev: float = 0.75,
max_lev: float = 1.0,
# Profit-take
pt_threshold: float = 0.30,
pt_band: float = 0.10,
pt_park: str = "SHY",
):
self.stock_tickers = stock_tickers
self.leverage = leverage
self.top_n = top_n
self.signal = signal
self.defensive = defensive
self.mom_lookback = mom_lookback
self.rebal_every = rebal_every
self.selection = selection
self.recovery_window = recovery_window
self.long_mom_lookback = long_mom_lookback
self.long_mom_skip = long_mom_skip
self.target_vol = target_vol
self.vol_window = vol_window
self.min_lev = min_lev
self.max_lev = max_lev
self.pt_threshold = pt_threshold
self.pt_band = pt_band
self.pt_park = pt_park
self._v3 = TrendRiderV3(
signal=signal, risk_on=("TQQQ", "UPRO"),
risk_off=defensive, ma_long=ma_long,
)
def _rank_stocks(self, data: pd.DataFrame) -> pd.DataFrame:
"""Return cross-sectional rank (higher = better)."""
avail = [t for t in self.stock_tickers if t in data.columns]
panel = data[avail]
if self.selection == "recovery_momentum":
recovery = panel / panel.rolling(self.recovery_window).min() - 1
momentum = panel.shift(self.long_mom_skip).pct_change(
self.long_mom_lookback - self.long_mom_skip, fill_method=None,
)
rec_r = recovery.rank(axis=1, pct=True, na_option="keep")
mom_r = momentum.rank(axis=1, pct=True, na_option="keep")
composite = 0.5 * rec_r + 0.5 * mom_r
return composite
else:
mom = panel.pct_change(self.mom_lookback, fill_method=None)
return mom
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
"""Build weights on ORIGINAL (unleveraged) price columns.
The backtest engine will track returns using the original data.
We transform the returns in a wrapper (see run_synth_backtest below).
Actually — we build a SYNTHETIC price panel and run the strategy
on that. So weights here are on synthetic-leverage columns.
"""
# This is called on the synthetic data panel.
# Columns: stock tickers (synthetic leveraged) + ETFs (original)
w = pd.DataFrame(0.0, index=data.index, columns=data.columns)
if self.signal not in data.columns:
return w
sig_arr = data[self.signal].to_numpy()
avail_stocks = [t for t in self.stock_tickers if t in data.columns]
avail_def = [t for t in self.defensive if t in data.columns]
park_col = self.pt_park if self.pt_park in data.columns else ""
# Rank using the ORIGINAL unleveraged data — NOT passed here.
# We'll precompute ranks externally and attach them.
# For now, rank on the synthetic data (momentum on leveraged prices
# preserves ranking since leverage is monotone on return).
mom = data[avail_stocks].pct_change(self.mom_lookback, fill_method=None)
if self.selection == "recovery_momentum":
panel = data[avail_stocks]
recovery = panel / panel.rolling(self.recovery_window).min() - 1
long_mom = panel.shift(self.long_mom_skip).pct_change(
self.long_mom_lookback - self.long_mom_skip, fill_method=None,
)
rec_r = recovery.rank(axis=1, pct=True, na_option="keep")
mom_r = long_mom.rank(axis=1, pct=True, na_option="keep")
score = 0.5 * rec_r + 0.5 * mom_r
else:
score = mom
need = max(150, self.mom_lookback + 1, self._v3.vol_window + 1,
self._v3.dd_window, self._v3.peak_window,
self.long_mom_lookback + 1 if self.selection == "recovery_momentum" else 0,
self.recovery_window + 1 if self.selection == "recovery_momentum" else 0) + 1
regime: str | None = None
bars = 0
# Phase 1: build raw weights (regime + stock selection)
raw_w = pd.DataFrame(np.nan, index=data.index, columns=data.columns)
for i in range(len(data)):
if i < need:
continue
closes = sig_arr[:i]
if np.isnan(closes[-1]):
continue
desired = self._v3._desired_regime(closes, regime)
changed = False
if regime is None:
regime, bars, changed = desired, 0, True
else:
bars += 1
if desired != regime and bars >= 15:
regime, bars, changed = desired, 0, True
if not changed and (i - need) % self.rebal_every != 0:
continue
row = {c: 0.0 for c in data.columns}
dt = data.index[i]
if regime == "risk_on":
s = score.iloc[i][avail_stocks].dropna()
valid = s.index[data.loc[dt, s.index].notna()]
s = s[valid]
if self.selection == "momentum":
s = s[s > 0]
top = s.nlargest(min(self.top_n, len(s)))
if len(top) > 0:
wt = 1.0 / len(top)
for t in top.index:
row[t] = wt
elif avail_def:
row[avail_def[0]] = 1.0
else:
if avail_def:
dm = data[avail_def].pct_change(63, fill_method=None).iloc[i].dropna()
best = dm.idxmax() if len(dm) > 0 else avail_def[0]
row[best] = 1.0
for c, v in row.items():
raw_w.at[dt, c] = v
raw_w = raw_w.ffill().fillna(0.0)
raw_w = raw_w.shift(1).fillna(0.0)
# Phase 2: Vol-target overlay
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
port_rets = (raw_w * daily_ret).sum(axis=1)
realized_vol = (
port_rets.rolling(self.vol_window, min_periods=21).std() * np.sqrt(252)
)
scale = (self.target_vol / realized_vol).clip(lower=self.min_lev, upper=self.max_lev)
scale = scale.shift(1).fillna(1.0)
w = raw_w.mul(scale, axis=0)
# Phase 3: Profit-take
if self.pt_threshold <= 0:
return w
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
entry_price: float | None = None
current_sym: str | None = None
is_stopped = False
restore_level = self.pt_threshold - self.pt_band
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
current_sym = None
entry_price = None
is_stopped = False
continue
if sym != current_sym:
current_sym = sym
entry_price = (
float(data[sym].iloc[i - 1])
if i > 0 and sym in data.columns else None
)
is_stopped = False
continue
if entry_price is None or entry_price <= 0 or sym not in data.columns:
continue
yesterday = float(data[sym].iloc[i - 1]) if i > 0 else float(data[sym].iloc[i])
gain = yesterday / entry_price - 1.0
if is_stopped:
if gain < restore_level:
is_stopped = False
else:
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
else:
if gain >= self.pt_threshold:
is_stopped = True
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
return w
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
print("=" * 95)
print(" DIRECTION 2: V7 + SYNTHETIC LEVERAGED INDIVIDUAL STOCKS")
print("=" * 95)
# Load S&P 500 + PIT + ETFs
print("\n[1] Loading data...")
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
pit_intervals = uh.load_sp500_history()
hist_tickers = uh.all_tickers_ever(pit_intervals)
etfs = ["SPY", "GLD", "DBC", "SHY", "TQQQ", "UPRO", "TLT"]
all_tickers = sorted(set(tickers + hist_tickers + etfs))
raw_data = data_manager.update("us", all_tickers, with_open=False)
if isinstance(raw_data, tuple):
raw_data = raw_data[0]
cutoff = raw_data.index[-1] - pd.DateOffset(years=YEARS)
raw_data = raw_data[raw_data.index >= cutoff]
raw_data = uh.mask_prices(raw_data, pit_intervals)
stock_tickers = [t for t in raw_data.columns
if t not in etfs and raw_data[t].notna().any()]
print(f" Stocks: {len(stock_tickers)}, Period: {raw_data.index[0].date()} → {raw_data.index[-1].date()}")
# Build synthetic leveraged price panels
print("\n[2] Building synthetic leveraged prices...")
stock_prices = raw_data[stock_tickers]
synth_2x = synthetic_leveraged_prices(stock_prices, 2.0)
synth_3x = synthetic_leveraged_prices(stock_prices, 3.0)
# Combine synthetic stocks with real ETF prices for each variant
etf_prices = raw_data[etfs]
results: list[tuple[str, dict]] = []
def run(label: str, strategy: Strategy, data_panel: pd.DataFrame):
print(f" {label}...", end=" ", flush=True)
try:
eq = backtest(strategy, data_panel, initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
results.append((label, m))
print(f"Ann={m['annualizedReturn']*100:.1f}% Sharpe={m['sharpeRatio']:.2f} "
f"MaxDD={m['maxDrawdown']*100:.1f}%")
except Exception as e:
print(f"FAILED: {e}")
# =====================================================================
# Run variants
# =====================================================================
print("\n[3] Running strategies...")
# --- V7+VT36 baseline (real TQQQ/UPRO) ---
from strategies.trend_rider_v7 import TrendRiderV7
etf_only = [t for t in ["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"] if t in etf_prices.columns]
run("V7+VT36 baseline (TQQQ/UPRO)",
TrendRiderV7(target_vol=0.36, min_lev=0.75),
etf_prices[etf_only])
# --- Synth 2x: momentum, various top-N ---
for n in (1, 3, 5, 10):
panel_2x = pd.concat([synth_2x, etf_prices], axis=1)
panel_2x = panel_2x.loc[:, ~panel_2x.columns.duplicated()]
run(f"Synth 2x Mom top-{n} (VT36+PT30)",
V7SynthLeverage(stock_tickers=stock_tickers, leverage=2.0,
top_n=n, target_vol=0.36, min_lev=0.75),
panel_2x)
# --- Synth 2x: recovery-momentum ---
for n in (3, 5, 10):
panel_2x = pd.concat([synth_2x, etf_prices], axis=1)
panel_2x = panel_2x.loc[:, ~panel_2x.columns.duplicated()]
run(f"Synth 2x RecMom top-{n} (VT36+PT30)",
V7SynthLeverage(stock_tickers=stock_tickers, leverage=2.0,
top_n=n, selection="recovery_momentum",
target_vol=0.36, min_lev=0.75),
panel_2x)
# --- Synth 3x: direct comparison with real TQQQ ---
for n in (1, 3, 5):
panel_3x = pd.concat([synth_3x, etf_prices], axis=1)
panel_3x = panel_3x.loc[:, ~panel_3x.columns.duplicated()]
run(f"Synth 3x Mom top-{n} (VT36+PT30)",
V7SynthLeverage(stock_tickers=stock_tickers, leverage=3.0,
top_n=n, target_vol=0.36, min_lev=0.75),
panel_3x)
# --- Synth 2x without vol-target (see if raw 2x stocks need less VT) ---
for n in (3, 5):
panel_2x = pd.concat([synth_2x, etf_prices], axis=1)
panel_2x = panel_2x.loc[:, ~panel_2x.columns.duplicated()]
run(f"Synth 2x Mom top-{n} (no VT, PT30)",
V7SynthLeverage(stock_tickers=stock_tickers, leverage=2.0,
top_n=n, target_vol=1.0, min_lev=1.0, max_lev=1.0),
panel_2x)
# --- Synth 2x with higher PT threshold (2x has less vol drag → let profits run) ---
for pt in (0.40, 0.50):
panel_2x = pd.concat([synth_2x, etf_prices], axis=1)
panel_2x = panel_2x.loc[:, ~panel_2x.columns.duplicated()]
run(f"Synth 2x Mom top-5 (VT36+PT{int(pt*100)})",
V7SynthLeverage(stock_tickers=stock_tickers, leverage=2.0,
top_n=5, target_vol=0.36, min_lev=0.75,
pt_threshold=pt, pt_band=pt*0.33),
panel_2x)
# --- Synth 2x: no profit-take (2x might not need it) ---
panel_2x = pd.concat([synth_2x, etf_prices], axis=1)
panel_2x = panel_2x.loc[:, ~panel_2x.columns.duplicated()]
run("Synth 2x Mom top-5 (VT36, no PT)",
V7SynthLeverage(stock_tickers=stock_tickers, leverage=2.0,
top_n=5, target_vol=0.36, min_lev=0.75,
pt_threshold=0),
panel_2x)
# --- SPY benchmark ---
spy = raw_data["SPY"].dropna()
spy_eq = (spy / spy.iloc[0]) * CAPITAL
results.append(("SPY benchmark", metrics.raw_summary(spy_eq)))
# =====================================================================
# Report
# =====================================================================
results.sort(key=lambda x: x[1]["annualizedReturn"], reverse=True)
print(f"\n{'=' * 110}")
print(" RANKING")
print(f"{'=' * 110}")
print(f"{'#':<4} {'Strategy':<45} {'Ann%':>7} {'Vol%':>7} {'Sharpe':>7} "
f"{'Sortino':>8} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 110)
for i, (label, m) in enumerate(results, 1):
marker = " ★" if i <= 3 else ""
print(f"{i:<4} {label:<45} "
f"{m['annualizedReturn']*100:>6.1f}% "
f"{m['annualizedVolatility']*100:>6.1f}% "
f"{m['sharpeRatio']:>7.2f} "
f"{m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>6.1f}% "
f"{m['calmarRatio']:>7.2f}{marker}")
print(f"{'=' * 110}")
if __name__ == "__main__":
main()

385
research/v7_three_ideas_eval.py
View File

@@ -1,385 +0,0 @@
"""Test three structural improvements to V7+VT36 identified by independent review.
Idea 1: PT entry price reset on restore (fix stale anchor)
Idea 2: TMF (3x bonds) in risk-off basket with TLT MA gate
Idea 3: Open-price fast exit overlay for crash protection
All tested against V7+VT36 baseline (61.2% Ann, Sharpe 1.89, MaxDD -29.2%).
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from main import backtest
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
# =========================================================================
# V7 with all three ideas as toggleable flags
# =========================================================================
class TrendRiderV7X(Strategy):
"""V7 extended with three structural improvements.
Flags:
reset_entry_on_restore: Idea 1 — reset entry_price when PT restores.
tmf_risk_off: Idea 2 — include TMF in risk-off when TLT > MA.
fast_exit: Idea 3 — emergency exit when SPY opens below threshold.
"""
def __init__(
self,
# V3 regime
ma_long: int = 150,
signal: str = "SPY",
risk_on: tuple[str, ...] = ("TQQQ", "UPRO"),
risk_off: tuple[str, ...] = ("GLD", "DBC"),
# Vol-target
target_vol: float = 0.36,
vol_window: int = 60,
min_lev: float = 0.75,
max_lev: float = 1.0,
# Profit-take
pt_threshold: float = 0.30,
pt_band: float = 0.10,
pt_park: str = "SHY",
# === Idea 1: reset entry on restore ===
reset_entry_on_restore: bool = False,
# === Idea 2: TMF risk-off with bond gate ===
tmf_risk_off: bool = False,
tmf_symbol: str = "TMF",
tlt_symbol: str = "TLT",
tlt_ma_window: int = 200,
# === Idea 3: fast exit on open ===
fast_exit: bool = False,
fast_exit_gap_pct: float = -0.03,
fast_exit_low_window: int = 20,
# V3 passthrough
**v3_kwargs,
) -> None:
self.target_vol = target_vol
self.vol_window = vol_window
self.min_lev = min_lev
self.max_lev = max_lev
self.pt_threshold = pt_threshold
self.pt_band = pt_band
self.pt_park = pt_park
self.signal = signal
self.risk_off_base = risk_off
self.reset_entry_on_restore = reset_entry_on_restore
self.tmf_risk_off = tmf_risk_off
self.tmf_symbol = tmf_symbol
self.tlt_symbol = tlt_symbol
self.tlt_ma_window = tlt_ma_window
self.fast_exit = fast_exit
self.fast_exit_gap_pct = fast_exit_gap_pct
self.fast_exit_low_window = fast_exit_low_window
self.v3 = TrendRiderV3(
signal=signal, risk_on=risk_on, risk_off=risk_off,
ma_long=ma_long, **v3_kwargs,
)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
# --- Layer 1: V3 regime weights ---
w = self.v3.generate_signals(data)
# --- Idea 2: dynamically swap risk-off pick to TMF when bond regime is bullish ---
if self.tmf_risk_off and self.tmf_symbol in data.columns and self.tlt_symbol in data.columns:
tlt = data[self.tlt_symbol]
tlt_ma = tlt.rolling(self.tlt_ma_window).mean()
tlt_bull = (tlt > tlt_ma).shift(1).fillna(False)
risk_off_cols = [c for c in self.risk_off_base if c in w.columns]
tmf_col = self.tmf_symbol
if tmf_col not in w.columns:
w[tmf_col] = 0.0
for i in range(len(w)):
roff_weight = sum(w.iloc[i].get(c, 0.0) for c in risk_off_cols)
if roff_weight < 1e-8:
continue
if tlt_bull.iloc[i]:
# Candidate: TMF vs best of original risk-off by momentum
mom_lookback = 63
if i >= mom_lookback + 1:
best_sym = tmf_col
best_r = -np.inf
candidates = risk_off_cols + [tmf_col]
for sym in candidates:
if sym not in data.columns:
continue
p_now = data[sym].iloc[i - 1]
p_past = data[sym].iloc[i - 1 - mom_lookback]
if pd.notna(p_now) and pd.notna(p_past) and p_past > 0:
r = p_now / p_past - 1.0
if r > best_r:
best_r, best_sym = r, sym
# Reassign risk-off weight to the winner
for c in risk_off_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
if tmf_col in w.columns:
w.iat[i, w.columns.get_loc(tmf_col)] = 0.0
if best_sym in w.columns:
w.iat[i, w.columns.get_loc(best_sym)] = roff_weight
# --- Idea 3: fast exit overlay (check SPY open for gap-downs) ---
if self.fast_exit and self.signal in data.columns:
spy = data[self.signal]
spy_arr = spy.to_numpy()
risk_on_cols = list(self.v3.risk_on)
risk_off_cols_fast = [c for c in self.risk_off_base if c in w.columns]
park = self.pt_park if self.pt_park in w.columns else ""
for i in range(max(self.fast_exit_low_window + 1, 2), len(w)):
# Check if currently in risk-on
ron_weight = sum(float(w.iloc[i].get(c, 0.0))
for c in risk_on_cols if c in w.columns)
if ron_weight < 1e-8:
continue
prev_close = spy_arr[i - 1]
if np.isnan(prev_close) or prev_close <= 0:
continue
# Gap-down check: today's "effective open" approximated by
# checking if yesterday's close is below N-day low
low_window = spy_arr[max(0, i - 1 - self.fast_exit_low_window):i - 1]
if len(low_window) == 0:
continue
low_val = np.nanmin(low_window)
# Trigger 1: close below N-day low
trigger_low = prev_close <= low_val
# Trigger 2: large single-day drop (gap-down proxy using close-to-close)
if i >= 2:
prev2_close = spy_arr[i - 2]
daily_ret = (prev_close / prev2_close - 1.0) if prev2_close > 0 else 0.0
trigger_gap = daily_ret <= self.fast_exit_gap_pct
else:
trigger_gap = False
if trigger_low or trigger_gap:
# Emergency: zero out risk-on, move to park
for c in risk_on_cols:
if c in w.columns:
w.iat[i, w.columns.get_loc(c)] = 0.0
if park and park in w.columns:
w.iat[i, w.columns.get_loc(park)] = 1.0
# --- Layer 2: Vol-target overlay ---
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
# Only use columns present in w
common_cols = w.columns.intersection(daily_ret.columns)
port_rets = (w[common_cols] * daily_ret[common_cols]).sum(axis=1)
realized_vol = (
port_rets.rolling(self.vol_window, min_periods=21).std() * np.sqrt(252)
)
scale = (self.target_vol / realized_vol).clip(
lower=self.min_lev, upper=self.max_lev,
)
scale = scale.shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
# --- Layer 3: Profit-take with hysteresis ---
if self.pt_threshold <= 0:
return w
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
park_col = self.pt_park if self.pt_park in w.columns else ""
entry_price: float | None = None
current_sym: str | None = None
is_stopped = False
restore_level = self.pt_threshold - self.pt_band
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
current_sym = None
entry_price = None
is_stopped = False
continue
if sym != current_sym:
current_sym = sym
entry_price = (
float(data[sym].iloc[i - 1])
if i > 0 and sym in data.columns else None
)
is_stopped = False
continue
if entry_price is None or entry_price <= 0 or sym not in data.columns:
continue
yesterday = float(data[sym].iloc[i - 1]) if i > 0 else float(data[sym].iloc[i])
gain = yesterday / entry_price - 1.0
if is_stopped:
if gain < restore_level:
is_stopped = False
# === Idea 1: reset entry price on restore ===
if self.reset_entry_on_restore:
entry_price = yesterday
else:
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
else:
if gain >= self.pt_threshold:
is_stopped = True
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
return w
# =========================================================================
# Main
# =========================================================================
def main():
print("=" * 100)
print(" THREE IDEAS EVALUATION")
print("=" * 100)
# Load data including TMF and TLT
all_etfs = sorted(set([
"SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY", "TLT", "TMF",
]))
print(f"\nLoading ETFs: {all_etfs}")
data = data_manager.update("etfs", all_etfs, with_open=False)
if isinstance(data, tuple):
data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
data = data[data.index >= cutoff]
avail = sorted(data.columns.tolist())
print(f"Available: {avail}")
print(f"Period: {data.index[0].date()} → {data.index[-1].date()}")
results: list[tuple[str, dict]] = []
def run(label, strategy):
eq = backtest(strategy, data, initial_capital=CAPITAL,
transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m = metrics.raw_summary(eq)
results.append((label, m))
print(f" {label:<60} Ann={m['annualizedReturn']*100:>5.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}% "
f"Sortino={m['sortinoRatio']:.2f} Calmar={m['calmarRatio']:.2f}")
# Baseline
print("\n--- Baseline ---")
run("V7+VT36 (baseline)",
TrendRiderV7X(target_vol=0.36, min_lev=0.75))
# === Idea 1: PT entry reset ===
print("\n--- Idea 1: PT Entry Price Reset ---")
run("V7+VT36 + PT reset",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
reset_entry_on_restore=True))
# Also test with different PT thresholds to see if reset changes the optimum
for pt in (0.20, 0.25, 0.30, 0.35, 0.40):
run(f" PT reset + PT{int(pt*100)}",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
reset_entry_on_restore=True,
pt_threshold=pt, pt_band=pt * 0.33))
# === Idea 2: TMF risk-off ===
print("\n--- Idea 2: TMF in Risk-Off ---")
if "TMF" in data.columns and "TLT" in data.columns:
run("V7+VT36 + TMF risk-off (TLT gate)",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
tmf_risk_off=True))
# TMF + PT reset combo
run("V7+VT36 + TMF + PT reset",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
tmf_risk_off=True, reset_entry_on_restore=True))
# Different TLT MA windows
for tlt_ma in (100, 150, 200, 250):
run(f" TMF risk-off (TLT MA{tlt_ma})",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
tmf_risk_off=True, tlt_ma_window=tlt_ma))
else:
print(" TMF or TLT not available, skipping")
# === Idea 3: Fast exit ===
print("\n--- Idea 3: Fast Exit ---")
for gap in (-0.02, -0.03, -0.04):
for low_w in (10, 20):
run(f"V7+VT36 + fast exit (gap={gap:.0%}, low={low_w}d)",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
fast_exit=True, fast_exit_gap_pct=gap,
fast_exit_low_window=low_w))
# === All three combined ===
print("\n--- All Three Combined ---")
if "TMF" in data.columns:
run("V7+VT36 + ALL (reset+TMF+fast exit)",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
reset_entry_on_restore=True,
tmf_risk_off=True,
fast_exit=True, fast_exit_gap_pct=-0.03,
fast_exit_low_window=20))
# Best combo tuning
for pt in (0.25, 0.30, 0.35):
run(f" ALL + PT{int(pt*100)}",
TrendRiderV7X(target_vol=0.36, min_lev=0.75,
reset_entry_on_restore=True,
tmf_risk_off=True,
fast_exit=True, fast_exit_gap_pct=-0.03,
fast_exit_low_window=20,
pt_threshold=pt, pt_band=pt * 0.33))
# Final ranking
results.sort(key=lambda x: x[1]["sharpeRatio"], reverse=True)
print(f"\n{'=' * 110}")
print(" FINAL RANKING (by Sharpe)")
print(f"{'=' * 110}")
print(f"{'#':<4} {'Strategy':<60} {'Ann%':>6} {'Vol%':>6} {'Sharpe':>7} "
f"{'Sortino':>8} {'MaxDD%':>7} {'Calmar':>7}")
print("-" * 110)
for i, (label, m) in enumerate(results, 1):
marker = " ★" if i <= 3 else ""
print(f"{i:<4} {label:<60} "
f"{m['annualizedReturn']*100:>5.1f}% "
f"{m['annualizedVolatility']*100:>5.1f}% "
f"{m['sharpeRatio']:>7.2f} "
f"{m['sortinoRatio']:>8.2f} "
f"{m['maxDrawdown']*100:>6.1f}% "
f"{m['calmarRatio']:>7.2f}{marker}")
print(f"{'=' * 110}")
# Also rank by Ann return
results.sort(key=lambda x: x[1]["annualizedReturn"], reverse=True)
print(f"\n Top 5 by Annualized Return:")
for i, (label, m) in enumerate(results[:5], 1):
print(f" {i}. {label:<55} Ann={m['annualizedReturn']*100:.1f}% "
f"Sharpe={m['sharpeRatio']:.2f} MaxDD={m['maxDrawdown']*100:.1f}%")
if __name__ == "__main__":
main()

501
research/v7_trade_audit.py
View File

@@ -1,501 +0,0 @@
"""V7+VT36 full trade log: regime changes, instrument switches, PT events,
vol-target scale, contribution analysis.
Run from /home/gahow/projects/quant:
uv run python research/v7_trade_audit.py
"""
from __future__ import annotations
import sys
sys.path.insert(0, ".")
import numpy as np
import pandas as pd
import data_manager
import metrics
from main import backtest
from strategies.trend_rider_v7 import TrendRiderV7
YEARS = 10
CAPITAL = 100_000
TX_COST = 0.001
FIXED_FEE = 2.0
def load_data():
tickers = sorted(set(["SPY", "TQQQ", "UPRO", "GLD", "DBC", "SHY"]))
data = data_manager.update("etfs", tickers, with_open=False)
if isinstance(data, tuple):
data = data[0]
cutoff = data.index[-1] - pd.DateOffset(years=YEARS)
data = data[data.index >= cutoff]
cols = [c for c in tickers if c in data.columns]
return data[cols]
def trace_v7(data: pd.DataFrame):
"""Replicate V7+VT36 signal generation step-by-step, logging every event."""
v7 = TrendRiderV7(target_vol=0.36, min_lev=0.75)
# --- Layer 1: V3 regime weights ---
w_v3 = v7.v3.generate_signals(data)
# --- Layer 2: Vol-target overlay ---
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
port_rets_v3 = (w_v3 * daily_ret).sum(axis=1)
realized_vol = (
port_rets_v3.rolling(v7.vol_window, min_periods=21).std() * np.sqrt(252)
)
scale_raw = (v7.target_vol / realized_vol).clip(lower=v7.min_lev, upper=v7.max_lev)
scale = scale_raw.shift(1).fillna(1.0)
w_vt = w_v3.mul(scale, axis=0)
# --- Layer 3: Profit-take (replicate the loop, logging events) ---
w = w_vt.copy()
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
park_col = v7.pt_park if v7.pt_park in w.columns else ""
entry_price = None
current_sym = None
is_stopped = False
restore_level = v7.pt_threshold - v7.pt_band
pt_events = [] # (date, type, sym, entry_price, exit_price, gain%)
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
current_sym = None
entry_price = None
is_stopped = False
continue
if sym != current_sym:
current_sym = sym
entry_price = float(data[sym].iloc[i - 1]) if i > 0 and sym in data.columns else None
is_stopped = False
continue
if entry_price is None or entry_price <= 0 or sym not in data.columns:
continue
yesterday = float(data[sym].iloc[i - 1]) if i > 0 else float(data[sym].iloc[i])
gain = yesterday / entry_price - 1.0
if is_stopped:
if gain < restore_level:
is_stopped = False
pt_events.append((data.index[i], "RESTORE", sym, entry_price, yesterday, gain))
else:
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
else:
if gain >= v7.pt_threshold:
is_stopped = True
pt_events.append((data.index[i], "CLEAR", sym, entry_price, yesterday, gain))
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
return w_v3, w_vt, w, scale, pt_events
def analyze_regime(w_v3: pd.DataFrame, data: pd.DataFrame):
"""Trace regime changes from V3 weights."""
held_v3 = w_v3.idxmax(axis=1)
max_w = w_v3.max(axis=1)
held_v3[max_w < 1e-8] = ""
risk_on_syms = {"TQQQ", "UPRO"}
risk_off_syms = {"GLD", "DBC"}
events = []
prev_regime = None
prev_sym = None
regime_start = None
for i in range(len(held_v3)):
sym = held_v3.iloc[i]
if not sym:
continue
regime = "risk_on" if sym in risk_on_syms else ("risk_off" if sym in risk_off_syms else "other")
if regime != prev_regime:
if prev_regime is not None:
events.append({
"date": data.index[i],
"type": "REGIME_CHANGE",
"from": prev_regime,
"to": regime,
"from_sym": prev_sym,
"to_sym": sym,
})
prev_regime = regime
regime_start = data.index[i]
prev_sym = sym
elif sym != prev_sym:
events.append({
"date": data.index[i],
"type": "INSTRUMENT_SWITCH",
"regime": regime,
"from_sym": prev_sym,
"to_sym": sym,
})
prev_sym = sym
return events, held_v3
def compute_contributions(data, w_v3, w_vt, w_final, scale, pt_events):
"""Return attribution of returns to risk-on, risk-off, PT-park, and vol-target."""
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
# Identify which days are in PT-park (cleared to cash by PT layer)
# NOTE: SHY is not in V3's output columns, so PT clears to 0% (cash), not SHY.
risk_on_syms = {"TQQQ", "UPRO"}
risk_off_syms = {"GLD", "DBC"}
vt_sum = w_vt.abs().sum(axis=1)
final_sum = w_final.abs().sum(axis=1)
held_final = w_final.idxmax(axis=1)
max_w_final = w_final.max(axis=1)
held_final[max_w_final < 1e-8] = ""
held_vt = w_vt.idxmax(axis=1)
max_w_vt = w_vt.max(axis=1)
held_vt[max_w_vt < 1e-8] = ""
# Classify each day
day_class = pd.Series("none", index=data.index)
for i in range(len(data)):
sym_vt = held_vt.iloc[i]
sym_final = held_final.iloc[i]
# PT-park: VT layer has allocation but final layer cleared to 0
if vt_sum.iloc[i] > 0.01 and final_sum.iloc[i] < 0.01:
day_class.iloc[i] = "pt_park"
elif not sym_final:
continue
elif sym_final in risk_on_syms:
day_class.iloc[i] = "risk_on"
elif sym_final in risk_off_syms:
day_class.iloc[i] = "risk_off"
elif sym_final == "SHY":
day_class.iloc[i] = "risk_off_shy"
else:
day_class.iloc[i] = "other"
# Daily portfolio return for each layer
port_ret_v3 = (w_v3 * daily_ret).sum(axis=1)
port_ret_vt = (w_vt * daily_ret).sum(axis=1)
port_ret_final = (w_final * daily_ret).sum(axis=1)
# Contribution by regime state
contrib = {}
for cls in ["risk_on", "risk_off", "risk_off_shy", "pt_park", "none"]:
mask = day_class == cls
contrib[cls] = {
"days": int(mask.sum()),
"cum_return": float((1 + port_ret_final[mask]).prod() - 1) if mask.any() else 0.0,
"avg_daily_ret": float(port_ret_final[mask].mean()) if mask.any() else 0.0,
}
# Vol-target impact: compare V3-only vs V3+VT
eq_v3 = (1 + port_ret_v3).cumprod() * CAPITAL
eq_vt = (1 + port_ret_vt).cumprod() * CAPITAL
eq_final = (1 + port_ret_final).cumprod() * CAPITAL
return day_class, contrib, eq_v3, eq_vt, eq_final
def main():
data = load_data()
print(f"Data: {data.index[0].date()} to {data.index[-1].date()}, {len(data)} days")
print(f"Columns: {list(data.columns)}")
w_v3, w_vt, w_final, scale, pt_events = trace_v7(data)
# =========================================================================
# 1. Regime changes and instrument switches
# =========================================================================
regime_events, held_v3 = analyze_regime(w_v3, data)
print(f"\n{'='*90}")
print(" REGIME CHANGES AND INSTRUMENT SWITCHES")
print(f"{'='*90}")
regime_changes = [e for e in regime_events if e["type"] == "REGIME_CHANGE"]
instrument_switches = [e for e in regime_events if e["type"] == "INSTRUMENT_SWITCH"]
print(f"\nTotal regime changes: {len(regime_changes)}")
print(f"Total instrument switches (within regime): {len(instrument_switches)}")
print(f"\n--- Regime Changes ---")
for e in regime_changes:
print(f" {e['date'].date()} {e['from']:>9} -> {e['to']:<9} ({e['from_sym']} -> {e['to_sym']})")
print(f"\n--- Instrument Switches ---")
for e in instrument_switches:
print(f" {e['date'].date()} [{e['regime']:>9}] {e['from_sym']} -> {e['to_sym']}")
# Regime holding periods
risk_on_syms = {"TQQQ", "UPRO"}
risk_off_syms = {"GLD", "DBC"}
regime_series = pd.Series("none", index=data.index)
for i in range(len(held_v3)):
sym = held_v3.iloc[i]
if sym in risk_on_syms:
regime_series.iloc[i] = "risk_on"
elif sym in risk_off_syms:
regime_series.iloc[i] = "risk_off"
active = regime_series[regime_series != "none"]
if len(active) > 0:
pct_risk_on = (active == "risk_on").mean()
pct_risk_off = (active == "risk_off").mean()
print(f"\n Time in risk_on: {pct_risk_on:.1%}")
print(f" Time in risk_off: {pct_risk_off:.1%}")
# Average holding period per regime stint
regime_shifts = active != active.shift(1)
stint_id = regime_shifts.cumsum()
stint_lengths = stint_id.groupby(stint_id).count()
avg_stint = stint_lengths.mean()
print(f" Avg regime stint: {avg_stint:.0f} trading days")
# =========================================================================
# 2. Profit-take events
# =========================================================================
print(f"\n{'='*90}")
print(" PROFIT-TAKE EVENTS")
print(f"{'='*90}")
print(f"\nTotal PT events: {len(pt_events)}")
clears = [e for e in pt_events if e[1] == "CLEAR"]
restores = [e for e in pt_events if e[1] == "RESTORE"]
print(f" CLEAR events: {len(clears)}")
print(f" RESTORE events: {len(restores)}")
if clears:
print(f"\n--- CLEAR Events ---")
gains = []
for date, typ, sym, entry, exit_p, gain in clears:
print(f" {date.date()} {sym:>5} entry=${entry:.2f} exit=${exit_p:.2f} gain={gain:+.1%}")
gains.append(gain)
print(f"\n Avg gain at CLEAR: {np.mean(gains):.1%}")
print(f" Min gain at CLEAR: {np.min(gains):.1%}")
print(f" Max gain at CLEAR: {np.max(gains):.1%}")
if restores:
print(f"\n--- RESTORE Events ---")
for date, typ, sym, entry, exit_p, gain in restores:
print(f" {date.date()} {sym:>5} entry=${entry:.2f} price=${exit_p:.2f} gain={gain:+.1%}")
# PT-park days
day_class, contrib, eq_v3, eq_vt, eq_final = compute_contributions(
data, w_v3, w_vt, w_final, scale, pt_events)
pt_park_days = (day_class == "pt_park").sum()
total_active_days = (day_class != "none").sum()
print(f"\n Days in PT-park (SHY): {pt_park_days} ({pt_park_days/total_active_days:.1%} of active days)")
# =========================================================================
# 3. Vol-target scale analysis
# =========================================================================
print(f"\n{'='*90}")
print(" VOL-TARGET SCALE ANALYSIS")
print(f"{'='*90}")
active_scale = scale[scale < 1.0 - 1e-6]
print(f"\n Scale stats (when < 1.0):")
print(f" Days at full scale (1.0): {(scale >= 1.0 - 1e-6).sum()}")
print(f" Days below 1.0: {len(active_scale)}")
print(f" Days below 0.90: {(scale < 0.90).sum()}")
print(f" Days below 0.80: {(scale < 0.80).sum()}")
print(f" Days at floor (0.75): {(scale <= 0.75 + 1e-6).sum()}")
print(f" Min scale: {scale.min():.3f}")
print(f" Mean scale: {scale.mean():.3f}")
# When did scale hit the floor?
at_floor = scale[scale <= 0.75 + 1e-6]
if len(at_floor) > 0:
print(f"\n Periods at floor (scale=0.75):")
floor_mask = scale <= 0.75 + 1e-6
shifts = floor_mask != floor_mask.shift(1)
stint_ids = shifts.cumsum()
floor_stints = stint_ids[floor_mask]
for stint_id_val in floor_stints.unique():
stint_dates = floor_stints[floor_stints == stint_id_val].index
if len(stint_dates) > 0:
print(f" {stint_dates[0].date()} to {stint_dates[-1].date()} ({len(stint_dates)} days)")
# =========================================================================
# 4. Contribution analysis
# =========================================================================
print(f"\n{'='*90}")
print(" RETURN CONTRIBUTION BY STATE")
print(f"{'='*90}")
for cls in ["risk_on", "risk_off", "risk_off_shy", "pt_park"]:
c = contrib[cls]
print(f"\n {cls:>14}: {c['days']:>5} days "
f"cum_return={c['cum_return']:>+8.1%} "
f"avg_daily={c['avg_daily_ret']*10000:>+6.1f}bps")
# =========================================================================
# 5. With vs Without vol-target
# =========================================================================
print(f"\n{'='*90}")
print(" VOL-TARGET IMPACT (V3 alone vs V3+VT)")
print(f"{'='*90}")
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
# V3-only with turnover cost
eq_v3_bt = backtest(
type("V3Only", (), {"generate_signals": lambda self, d: TrendRiderV7(target_vol=0.36, min_lev=0.75).v3.generate_signals(d)})(),
data, initial_capital=CAPITAL, transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m_v3 = metrics.raw_summary(eq_v3_bt)
# V7+VT36 full
eq_full = backtest(
TrendRiderV7(target_vol=0.36, min_lev=0.75),
data, initial_capital=CAPITAL, transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m_full = metrics.raw_summary(eq_full)
# V7 with VT but no PT
eq_no_pt = backtest(
TrendRiderV7(target_vol=0.36, min_lev=0.75, pt_threshold=0.0),
data, initial_capital=CAPITAL, transaction_cost=TX_COST, fixed_fee=FIXED_FEE)
m_no_pt = metrics.raw_summary(eq_no_pt)
print(f"\n {'Variant':<30} {'Ann':>7} {'Sharpe':>7} {'MaxDD':>7} {'Sortino':>8} {'Calmar':>7}")
print(f" {'-'*67}")
for label, m in [("V3 only (no VT, no PT)", m_v3),
("V3 + VT (no PT)", m_no_pt),
("V7+VT36 full (VT+PT)", m_full)]:
print(f" {label:<30} {m['annualizedReturn']*100:>6.1f}% {m['sharpeRatio']:>7.2f} "
f"{m['maxDrawdown']*100:>6.1f}% {m['sortinoRatio']:>8.2f} {m['calmarRatio']:>7.2f}")
# =========================================================================
# 6. Drawdown analysis
# =========================================================================
print(f"\n{'='*90}")
print(" WORST DRAWDOWN PERIODS")
print(f"{'='*90}")
rolling_peak = eq_full.cummax()
dd = (eq_full - rolling_peak) / rolling_peak
# Find top 5 drawdown troughs
dd_sorted = dd.sort_values()
seen_windows = []
top_dds = []
for date, dd_val in dd_sorted.items():
too_close = False
for prev_date in seen_windows:
if abs((date - prev_date).days) < 60:
too_close = True
break
if not too_close:
top_dds.append((date, dd_val))
seen_windows.append(date)
if len(top_dds) >= 5:
break
for trough_date, trough_dd in top_dds:
# Find start of this drawdown (last peak before trough)
pre_trough = rolling_peak.loc[:trough_date]
peak_val = pre_trough.iloc[-1]
peak_dates = eq_full[eq_full >= peak_val * 0.999].loc[:trough_date]
if len(peak_dates) > 0:
peak_date = peak_dates.index[0]
else:
peak_date = trough_date
# What was the strategy holding?
held_during = day_class.loc[peak_date:trough_date]
regime_during = held_during.value_counts()
# Recovery date
post_trough = eq_full.loc[trough_date:]
recovered = post_trough[post_trough >= peak_val]
recovery_date = recovered.index[0] if len(recovered) > 0 else None
print(f"\n DD {trough_dd:.1%} | {peak_date.date()} -> {trough_date.date()}"
f" | Recovery: {recovery_date.date() if recovery_date else 'N/A'}")
print(f" State during DD: {dict(regime_during)}")
# What instrument was held at the trough?
held_final = w_final.idxmax(axis=1)
max_w_final = w_final.max(axis=1)
held_final[max_w_final < 1e-8] = ""
trough_sym = held_final.loc[trough_date] if trough_date in held_final.index else "?"
print(f" Held at trough: {trough_sym}")
# =========================================================================
# 7. Disproportionate trades
# =========================================================================
print(f"\n{'='*90}")
print(" TOP 10 BEST AND WORST SINGLE-DAY RETURNS")
print(f"{'='*90}")
port_ret_final = (w_final * daily_ret).sum(axis=1)
# Apply turnover cost
turnover = w_final.diff().abs().sum(axis=1).fillna(0.0)
port_ret_final_net = port_ret_final - turnover * TX_COST
held_final = w_final.idxmax(axis=1)
max_w_final = w_final.max(axis=1)
held_final[max_w_final < 1e-8] = ""
best = port_ret_final_net.nlargest(10)
worst = port_ret_final_net.nsmallest(10)
print("\n BEST DAYS:")
for date, ret in best.items():
sym = held_final.loc[date] if date in held_final.index else "?"
sc = scale.loc[date] if date in scale.index else 1.0
print(f" {date.date()} {ret:>+7.2%} holding={sym} scale={sc:.2f}")
print("\n WORST DAYS:")
for date, ret in worst.items():
sym = held_final.loc[date] if date in held_final.index else "?"
sc = scale.loc[date] if date in scale.index else 1.0
print(f" {date.date()} {ret:>+7.2%} holding={sym} scale={sc:.2f}")
# Compound impact of top/bottom 20 days
sorted_rets = port_ret_final_net.sort_values()
total_days = len(sorted_rets)
eq_without_top20 = (1 + sorted_rets.iloc[:-20]).prod()
eq_without_bottom20 = (1 + sorted_rets.iloc[20:]).prod()
eq_all = (1 + sorted_rets).prod()
print(f"\n Total compound growth factor: {eq_all:.2f}x")
print(f" Without best 20 days: {eq_without_top20:.2f}x")
print(f" Without worst 20 days: {eq_without_bottom20:.2f}x")
# =========================================================================
# 8. Annual returns breakdown
# =========================================================================
print(f"\n{'='*90}")
print(" ANNUAL RETURNS BREAKDOWN")
print(f"{'='*90}")
yearly = port_ret_final_net.groupby(port_ret_final_net.index.year)
print(f"\n {'Year':>6} {'Return':>8} {'Vol':>7} {'MaxDD':>7} {'Days_ON':>8} {'Days_OFF':>9} {'Days_PT':>8}")
print(f" {'-'*55}")
for year, rets in yearly:
ann_r = (1 + rets).prod() ** (252 / len(rets)) - 1 if len(rets) > 0 else 0
vol = rets.std() * np.sqrt(252) if len(rets) > 1 else 0
eq_yr = (1 + rets).cumprod()
mdd = ((eq_yr / eq_yr.cummax()) - 1).min()
yr_class = day_class.loc[rets.index]
n_on = (yr_class == "risk_on").sum()
n_off = ((yr_class == "risk_off") | (yr_class == "risk_off_shy")).sum()
n_pt = (yr_class == "pt_park").sum()
print(f" {year:>6} {ann_r:>+7.1%} {vol:>6.1%} {mdd:>+6.1%} {n_on:>8} {n_off:>9} {n_pt:>8}")
if __name__ == "__main__":
main()

116
scripts/run_daily.py
View File

@@ -1,116 +0,0 @@
"""Daily live-trading update for all registered strategies.
Iterates STRATEGY_REGISTRY and calls trader.cmd_auto for each, sharing
fetched price panels across strategies (we still call cmd_auto so the
state-file format stays consistent with manual runs).
Usage:
uv run python scripts/run_daily.py [--market us|cn|all] [--capital 10000]
[--only NAME ...] [--skip NAME ...]
Designed for cron:
# 5pm ET weekdays, after US close
0 17 * * 1-5 cd /path/to/quant && uv run python scripts/run_daily.py --market us
# 4pm CST weekdays, after CN close
0 16 * * 1-5 cd /path/to/quant && uv run python scripts/run_daily.py --market cn
"""
from __future__ import annotations
import argparse
import os
import sys
import time
import types
from datetime import datetime
# Make the project root importable when invoked as `python scripts/run_daily.py`
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
# We change into the project root so trader's relative `data/` paths resolve.
os.chdir(PROJECT_ROOT)
from trader import ( # noqa: E402
ETF_STRATEGY_UNIVERSES,
STRATEGY_REGISTRY,
cmd_auto,
)
from bridge import STRATEGY_META # noqa: E402
def _strategy_markets(name: str) -> list[str]:
"""Return the markets a strategy targets (per its META declaration)."""
meta = STRATEGY_META.get(name, {"markets": ["us"]})
return list(meta.get("markets") or ["us"])
def _make_args(market: str, name: str, capital: float) -> argparse.Namespace:
"""Build the argparse namespace cmd_auto expects."""
return argparse.Namespace(
market=market,
strategy=name,
capital=capital,
tx_cost=0.001,
fixed_fee=0.0, # cmd_auto falls back to MARKET_FEES + tier
integer_shares=True, # match the existing live state convention
)
def main() -> int:
parser = argparse.ArgumentParser(description="Daily live-trading update")
parser.add_argument("--market", default="all",
help="Which market(s): 'us', 'cn', 'all' (default: all)")
parser.add_argument("--capital", type=float, default=10_000,
help="Initial capital for any newly-initialized strategy (default: 10000)")
parser.add_argument("--only", nargs="+", default=None,
help="If set, only run these strategy names")
parser.add_argument("--skip", nargs="+", default=None,
help="Strategy names to skip (e.g. broken ones)")
parser.add_argument("--dry-run", action="store_true",
help="Print what would be run without executing")
args = parser.parse_args()
markets = ["us", "cn"] if args.market == "all" else [args.market]
only = set(args.only) if args.only else None
skip = set(args.skip) if args.skip else set()
plan: list[tuple[str, str]] = []
for name in STRATEGY_REGISTRY:
if only and name not in only:
continue
if name in skip:
continue
for mk in _strategy_markets(name):
if mk in markets:
plan.append((mk, name))
print(f"[run_daily] {datetime.now().isoformat(timespec='seconds')} "
f"will process {len(plan)} (market, strategy) pairs")
if args.dry_run:
for mk, name in plan:
print(f" [DRY] {mk:<3} {name}")
return 0
failures = []
t0 = time.time()
for mk, name in plan:
ns = _make_args(mk, name, args.capital)
try:
cmd_auto(ns)
except SystemExit:
raise
except Exception as e: # noqa: BLE001
failures.append((mk, name, repr(e)))
print(f"[run_daily] FAILED {mk} {name}: {e}", file=sys.stderr)
elapsed = time.time() - t0
print(f"[run_daily] done in {elapsed:.1f}s — {len(plan) - len(failures)} ok, {len(failures)} failed")
for mk, name, err in failures:
print(f" fail {mk} {name}: {err}", file=sys.stderr)
return 0 if not failures else 1
if __name__ == "__main__":
sys.exit(main())

21
scripts/run_daily.sh
View File

@@ -1,21 +0,0 @@
#!/usr/bin/env bash
# Daily live-trading update entrypoint for cron.
#
# Examples:
# 0 17 * * 1-5 /path/to/quant/scripts/run_daily.sh us >> /var/log/quant_us.log 2>&1
# 0 16 * * 1-5 /path/to/quant/scripts/run_daily.sh cn >> /var/log/quant_cn.log 2>&1
#
# The script changes into the quant project root and calls run_daily.py via uv.
# Defaults to "all" markets when no argument is passed.
set -euo pipefail
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
PROJECT_ROOT="$( cd "$SCRIPT_DIR/.." && pwd )"
cd "$PROJECT_ROOT"
MARKET="${1:-all}"
shift || true
exec uv run python scripts/run_daily.py --market "$MARKET" "$@"

190
strategies/adaptive_blend.py
View File

@@ -1,190 +0,0 @@
"""
PIT-Compliant Adaptive Multi-Strategy Blend — Sharpe 1.5+ target.
Combines three uncorrelated alpha sources with adaptive (trailing-Sharpe)
weighting and a vol-target overlay:
Leg 1: TrendRiderV5 (ETF regime/leverage timing) — inherently PIT
Leg 2: PIT Stock Momentum (cross-sectional 12-1 mom, masked universe)
Leg 3: Cross-Asset Time-Series Momentum (6m, top 2, inv-vol) — inherently PIT
Blending: weights proportional to trailing 126d rolling Sharpe (clipped ≥ 0).
Vol-target: scale combined exposure so realized 20d vol ≈ 22%.
Backtest (2017-06 to 2026-05, PIT-compliant):
Full: Sharpe 1.69, CAGR 33.6%, MaxDD -20.6%, Calmar 1.63
IS: Sharpe 1.36 (2017-2022)
OOS: Sharpe 2.21 (2023-2026)
Bootstrap P(Sharpe > 1.5) = 72%
All components are PIT-safe:
- V5: ETF-only (SPY/UPRO/TQQQ/GLD/DBC)
- Stock: uses universe_history.mask_prices() for S&P 500 membership
- XA: ETF-only (SPY/GLD/TLT/IEF/DBC)
- Adaptive weights: trailing 126d (no future info)
- Vol-target: 1-day lagged realized vol
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from strategies.cross_asset_momentum import CrossAssetMomentum
from strategies.trend_rider_v5 import TrendRiderV5
class AdaptiveBlend:
"""
Adaptive multi-strategy blend with vol-target overlay.
Usage:
blend = AdaptiveBlend()
daily_returns = blend.run(etf_panel, pit_masked_prices)
"""
def __init__(
self,
# V5 params
v5_promote_thresholds=(0.35, 0.55),
v5_demote_thresholds=(0.25, 0.45),
# Stock params
stock_top_n: int = 12,
stock_rebal_freq: int = 42,
stock_mom_blend: float = 1.0,
stock_asym_vol: bool = True,
stock_asym_vol_floor: float = 0.50,
# XA params
xa_lookback: int = 126,
xa_top_k: int = 2,
xa_rebal_freq: int = 21,
# Blend params
adaptive_lookback: int = 126,
vol_target: float = 0.22,
vol_window: int = 20,
# Backtest range
start: str = "2017-06-01",
):
self.v5 = TrendRiderV5(
promote_thresholds=v5_promote_thresholds,
demote_thresholds=v5_demote_thresholds,
)
self.stock_params = dict(
top_n=stock_top_n, rebal_freq=stock_rebal_freq,
mom_blend=stock_mom_blend, asym_vol=stock_asym_vol,
asym_vol_floor=stock_asym_vol_floor, dd_dampen=False,
)
self.xa = CrossAssetMomentum(
lookback=xa_lookback, top_k=xa_top_k,
rebal_freq=xa_rebal_freq, vol_scale=True,
)
self.adaptive_lookback = adaptive_lookback
self.vol_target = vol_target
self.vol_window = vol_window
self.start = start
def run(
self,
etf_panel: pd.DataFrame,
pit_masked: pd.DataFrame,
transaction_cost: float = 0.001,
) -> pd.Series:
"""
Execute the full blend and return daily returns.
Parameters
----------
etf_panel : price panel including ETFs (SPY, UPRO, TQQQ, GLD, DBC, TLT, IEF, etc.)
pit_masked : S&P 500 prices masked by membership (NaN outside membership)
transaction_cost : one-way transaction cost (default 10bps)
Returns
-------
pd.Series of daily returns (after vol-target scaling)
"""
from research.pit_optimization import PITEnsemble
from research.trend_rider_robustness import portfolio_returns
# --- Leg 1: V5 ---
w_v5 = self.v5.generate_signals(etf_panel)
rets_v5 = portfolio_returns(w_v5, etf_panel, transaction_cost=transaction_cost)
rets_v5 = rets_v5.loc[self.start:]
# --- Leg 2: PIT Stock Momentum ---
strat_stock = PITEnsemble(**self.stock_params)
w_stock = strat_stock.generate_signals(pit_masked)
rets_stock = (w_stock * pit_masked.pct_change(fill_method=None).fillna(0.0)).sum(axis=1)
rets_stock = rets_stock.loc[self.start:]
# --- Leg 3: Cross-Asset Momentum ---
w_xa = self.xa.generate_signals(etf_panel)
rets_xa = portfolio_returns(w_xa, etf_panel, transaction_cost=transaction_cost)
rets_xa = rets_xa.loc[self.start:]
# --- Adaptive blending ---
idx = rets_v5.index.intersection(rets_stock.index).intersection(rets_xa.index)
df = pd.DataFrame({
"v5": rets_v5.loc[idx],
"stock": rets_stock.loc[idx],
"xa": rets_xa.loc[idx],
}).fillna(0.0)
roll_mu = df.rolling(self.adaptive_lookback).mean()
roll_std = df.rolling(self.adaptive_lookback).std()
roll_sharpe = (roll_mu / roll_std * np.sqrt(252)).clip(lower=0)
w_sum = roll_sharpe.sum(axis=1).replace(0, 1)
adaptive_weights = roll_sharpe.div(w_sum, axis=0).fillna(1.0 / 3)
combined_rets = (df * adaptive_weights).sum(axis=1)
# --- Vol-target overlay ---
realized = combined_rets.rolling(self.vol_window).std(ddof=1) * np.sqrt(252)
realized = realized.shift(1).fillna(self.vol_target)
scale = (self.vol_target / realized.replace(0.0, np.nan)).clip(upper=1.0).fillna(1.0)
final_rets = combined_rets * scale
return final_rets
def run_with_diagnostics(
self,
etf_panel: pd.DataFrame,
pit_masked: pd.DataFrame,
transaction_cost: float = 0.001,
) -> dict:
"""Run and return diagnostics (individual rets, weights, combined)."""
from research.pit_optimization import PITEnsemble
from research.trend_rider_robustness import portfolio_returns
w_v5 = self.v5.generate_signals(etf_panel)
rets_v5 = portfolio_returns(w_v5, etf_panel, transaction_cost=transaction_cost).loc[self.start:]
strat_stock = PITEnsemble(**self.stock_params)
w_stock = strat_stock.generate_signals(pit_masked)
rets_stock = (w_stock * pit_masked.pct_change(fill_method=None).fillna(0.0)).sum(axis=1).loc[self.start:]
w_xa = self.xa.generate_signals(etf_panel)
rets_xa = portfolio_returns(w_xa, etf_panel, transaction_cost=transaction_cost).loc[self.start:]
idx = rets_v5.index.intersection(rets_stock.index).intersection(rets_xa.index)
df = pd.DataFrame({"v5": rets_v5.loc[idx], "stock": rets_stock.loc[idx], "xa": rets_xa.loc[idx]}).fillna(0)
roll_mu = df.rolling(self.adaptive_lookback).mean()
roll_std = df.rolling(self.adaptive_lookback).std()
roll_sharpe = (roll_mu / roll_std * np.sqrt(252)).clip(lower=0)
w_sum = roll_sharpe.sum(axis=1).replace(0, 1)
adaptive_weights = roll_sharpe.div(w_sum, axis=0).fillna(1.0 / 3)
combined_rets = (df * adaptive_weights).sum(axis=1)
realized = combined_rets.rolling(self.vol_window).std(ddof=1) * np.sqrt(252)
realized = realized.shift(1).fillna(self.vol_target)
scale = (self.vol_target / realized.replace(0.0, np.nan)).clip(upper=1.0).fillna(1.0)
final_rets = combined_rets * scale
return {
"rets_v5": rets_v5,
"rets_stock": rets_stock,
"rets_xa": rets_xa,
"adaptive_weights": adaptive_weights,
"combined_rets": combined_rets,
"vol_scale": scale,
"final_rets": final_rets,
}

93
strategies/cross_asset_momentum.py
View File

@@ -1,93 +0,0 @@
"""
Cross-asset time-series momentum strategy (ETF-only, inherently PIT).
Alpha source: Moskowitz, Ooi, Pedersen (2012) — assets with positive
12-month returns continue to trend. Earns during equity crises when
bonds/gold trend up while stocks trend down.
Universe: SPY, GLD, TLT, IEF, DBC (broad, liquid ETFs)
Signal: 12-month total return; go long top K assets with positive momentum
Rebalance: monthly (21 trading days)
If no asset has positive 12m return → 100% cash (SHY proxy = 0 weights)
"""
import numpy as np
import pandas as pd
class CrossAssetMomentum:
"""Time-series momentum across major asset classes."""
UNIVERSE = ["SPY", "GLD", "TLT", "IEF", "DBC"]
def __init__(
self,
lookback: int = 252,
top_k: int = 3,
rebal_freq: int = 21,
vol_scale: bool = True,
vol_window: int = 63,
):
self.lookback = lookback
self.top_k = top_k
self.rebal_freq = rebal_freq
self.vol_scale = vol_scale
self.vol_window = vol_window
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
"""
Parameters
----------
data : DataFrame with columns including UNIVERSE ETFs (prices).
Returns
-------
DataFrame of weights aligned to data.index and data.columns.
"""
# Restrict to our universe (ignore missing gracefully)
available = [t for t in self.UNIVERSE if t in data.columns]
prices = data[available]
# 12-month return signal (shifted 1 day for execution lag)
mom = prices.pct_change(self.lookback).shift(1)
# Inverse-vol for position sizing (optional)
if self.vol_scale:
daily_ret = prices.pct_change()
vol = daily_ret.rolling(self.vol_window).std() * np.sqrt(252)
inv_vol = (1.0 / vol).replace([np.inf, -np.inf], np.nan)
else:
inv_vol = None
weights = pd.DataFrame(0.0, index=data.index, columns=data.columns)
n = len(prices.index)
last_w = pd.Series(0.0, index=available)
for i in range(self.lookback + 1, n, self.rebal_freq):
row_mom = mom.iloc[i]
# Only go long assets with positive momentum
positive = row_mom[row_mom > 0].sort_values(ascending=False)
if positive.empty:
last_w = pd.Series(0.0, index=available)
else:
selected = positive.head(self.top_k).index.tolist()
if self.vol_scale and inv_vol is not None:
iv = inv_vol.iloc[i][selected]
if iv.sum() > 0:
w = iv / iv.sum()
else:
w = pd.Series(1.0 / len(selected), index=selected)
else:
w = pd.Series(1.0 / len(selected), index=selected)
last_w = pd.Series(0.0, index=available)
last_w[selected] = w.values
# Hold until next rebalance
end_i = min(i + self.rebal_freq, n)
for col in available:
weights.iloc[i:end_i, weights.columns.get_loc(col)] = last_w[col]
return weights

124
strategies/long_hedged.py
View File

@@ -1,124 +0,0 @@
"""Market-hedged long-only stock portfolio.
Architecture
------------
Long: top-N stock portfolio (factor-selected, inv-vol weighted).
Short hedge: SPY (or SH ETF) at hedge_ratio × long_gross.
This isolates cross-sectional stock-selection alpha while removing the
broad-market beta. Unlike L/S of individual stocks, the short leg is on
the index — so:
* no meme-stock blowups on short side (GME/AMC type events)
* borrow cost on SPY is ≈ 5-15 bps annualized (very cheap)
* no short-dividend pass-through issue (pay SPY div, but that's offset
by long-side dividends roughly)
Because the long leg is monthly-rebalanced and the short hedge is fixed
at -1.0 × long_gross, total turnover is dominated by the long leg —
similar to V6 long-only.
Output is PIT-safe via terminal `.shift(1)`.
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from strategies.base import Strategy
from strategies.factor_combo import SIGNAL_REGISTRY
class LongHedgedStock(Strategy):
"""Long-only stock momentum hedged with SPY short."""
def __init__(
self,
signal_name: str = "rec_mfilt+deep_upvol",
top_n: int = 15,
rebal_freq: int = 21,
hedge_symbol: str = "SPY",
hedge_ratio: float = 1.0,
long_gross: float = 1.0,
invvol_window: int = 60,
invvol_floor: float = 0.10,
invvol_cap: float = 0.20,
stock_universe: list[str] | None = None,
# Regime gate: zero out positions when regime_signal < its MA(ma_window)
regime_gate: bool = False,
regime_signal: str = "SPY",
ma_window: int = 200,
) -> None:
if signal_name not in SIGNAL_REGISTRY:
raise ValueError(f"Unknown signal: {signal_name}")
self.signal_name = signal_name
self.signal_func = SIGNAL_REGISTRY[signal_name]
self.top_n = top_n
self.rebal_freq = rebal_freq
self.hedge_symbol = hedge_symbol
self.hedge_ratio = hedge_ratio
self.long_gross = long_gross
self.invvol_window = invvol_window
self.invvol_floor = invvol_floor
self.invvol_cap = invvol_cap
self.stock_universe = stock_universe
self.regime_gate = regime_gate
self.regime_signal = regime_signal
self.ma_window = ma_window
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
if self.hedge_symbol not in data.columns:
raise ValueError(f"hedge_symbol {self.hedge_symbol!r} missing from panel")
universe = self.stock_universe or [
c for c in data.columns if c != self.hedge_symbol
]
universe = [c for c in universe if c in data.columns]
stock_panel = data[universe]
sig = self.signal_func(stock_panel)
rank = sig.rank(axis=1, ascending=False, na_option="bottom")
n_valid = sig.notna().sum(axis=1)
enough = n_valid >= self.top_n
top_mask = (rank <= self.top_n) & enough.values.reshape(-1, 1)
# Inv-vol weighting within selection
rets = stock_panel.pct_change(fill_method=None)
vol = rets.rolling(self.invvol_window,
min_periods=self.invvol_window // 2).std() * np.sqrt(252)
vol_clipped = vol.clip(lower=self.invvol_floor, upper=self.invvol_cap)
invvol = (1.0 / vol_clipped).where(top_mask, 0.0)
row_sums = invvol.sum(axis=1).replace(0, np.nan)
long_w = invvol.div(row_sums, axis=0).fillna(0.0) * self.long_gross
# Monthly rebalance
warmup = 252
rebal_mask = pd.Series(False, index=data.index)
rebal_idx = list(range(warmup, len(data), self.rebal_freq))
rebal_mask.iloc[rebal_idx] = True
long_w[~rebal_mask] = np.nan
long_w = long_w.ffill().fillna(0.0)
long_w.iloc[:warmup] = 0.0
# Build full weights frame: longs in stocks, short in SPY
out = pd.DataFrame(0.0, index=data.index, columns=data.columns)
for c in long_w.columns:
if c in out.columns:
out[c] = long_w[c]
# Short hedge: only when long leg is active (gross > 0)
long_gross_now = long_w.abs().sum(axis=1)
active = (long_gross_now > 0)
out[self.hedge_symbol] = -self.hedge_ratio * long_gross_now * active.astype(float)
# Regime gate: zero everything when regime signal is in bear regime.
# Avoids the negative-carry case where long stocks tank with SPY but
# the hedge can't fully offset (since long has higher beta).
if self.regime_gate and self.regime_signal in data.columns:
regime_px = data[self.regime_signal]
ma = regime_px.rolling(self.ma_window).mean()
risk_on = (regime_px > ma).astype(float).fillna(0.0)
out = out.mul(risk_on, axis=0)
return out.shift(1).fillna(0.0)
__all__ = ["LongHedgedStock"]

251
strategies/ls_momentum.py
View File

@@ -1,251 +0,0 @@
"""Industry-neutral long/short momentum on the S&P 500.
Strategy
--------
At each rebalance date (default: monthly):
1. Compute 12-1 month momentum for every stock in the panel.
2. Group stocks by GICS sector.
3. Within each sector, rank by momentum.
4. Long the top `long_pct` (default 20%) of each sector.
5. Short the bottom `short_pct` (default 20%) of each sector.
6. Equal-weight within long-leg and short-leg, scaled so gross long = 1.0
and gross short = 1.0 → 200% gross exposure, ~0 net (β ≈ 0).
The β-neutrality comes from sector-level matching: each sector contributes
both long and short positions in equal $-amounts, so sector and (mostly)
market exposures cancel out.
Output
------
A weights DataFrame with positive (long) and negative (short) entries.
PIT-safe via terminal `.shift(1)`.
Costs
-----
Realistic backtest of L/S requires three additional costs not present in
long-only:
* borrow fee on the short leg (handled by the eval script, not here)
* higher slippage per turnover (this strategy churns more than V5)
* dividend payment on shorts (small for SP500 ~ 1.5% × |short_w|)
The strategy reports raw weights; the eval script applies costs.
"""
from __future__ import annotations
import os
import urllib.request
import io
import json
import numpy as np
import pandas as pd
from strategies.base import Strategy
SECTOR_CACHE = "data/us_sectors.csv"
WIKIPEDIA_SP500_URL = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
def fetch_sp500_sectors(force: bool = False) -> pd.DataFrame:
"""Return a DataFrame indexed by ticker with GICS sector / sub-industry.
Cached at data/us_sectors.csv. Wikipedia is the canonical source for
current S&P 500 sector membership; for backtest purposes we use today's
sector — sector membership is stable enough year-over-year that this
introduces minimal lookahead bias for an industry-neutral strategy.
"""
if not force and os.path.exists(SECTOR_CACHE):
df = pd.read_csv(SECTOR_CACHE, index_col=0)
if "GICS Sector" in df.columns and len(df) > 100:
return df
print("--- Fetching S&P 500 GICS sectors from Wikipedia ---")
headers = {"User-Agent": "Mozilla/5.0 (quant-backtest)"}
req = urllib.request.Request(WIKIPEDIA_SP500_URL, headers=headers)
with urllib.request.urlopen(req) as resp:
html = resp.read().decode("utf-8")
tables = pd.read_html(io.StringIO(html))
df = tables[0]
df = df.rename(columns={"Symbol": "ticker"})
df["ticker"] = df["ticker"].str.replace(".", "-", regex=False)
df = df.set_index("ticker")
keep = [c for c in df.columns if c in ("GICS Sector", "GICS Sub-Industry",
"Security")]
df = df[keep]
os.makedirs(os.path.dirname(SECTOR_CACHE), exist_ok=True)
df.to_csv(SECTOR_CACHE)
print(f"--- Cached {len(df)} sector mappings to {SECTOR_CACHE} ---")
return df
def _signal_mom_12_1(prices: pd.DataFrame) -> pd.DataFrame:
"""12-1 month cross-sectional momentum (highest = long)."""
return prices.shift(21).pct_change(231)
def _signal_reversal_1m(prices: pd.DataFrame) -> pd.DataFrame:
"""1-month reversal: highest 21-day return → SHORT (so we negate)."""
return -prices.pct_change(21)
def _signal_reversal_5d(prices: pd.DataFrame) -> pd.DataFrame:
"""Short-term 5-day reversal."""
return -prices.pct_change(5)
def _signal_recovery_63(prices: pd.DataFrame) -> pd.DataFrame:
"""Recovery factor: price / 63d low (V-shape continuation, long-only-friendly)."""
return prices / prices.rolling(63, min_periods=63).min() - 1
def _signal_low_vol(prices: pd.DataFrame) -> pd.DataFrame:
"""Low-vol: invert 60-day realized vol so low vol → high signal."""
rets = prices.pct_change(fill_method=None)
vol = rets.rolling(60, min_periods=40).std() * np.sqrt(252)
return -vol
def _signal_quality_mom(prices: pd.DataFrame) -> pd.DataFrame:
"""Composite: 12-1 mom + consistency (% positive days over 252d) + low-vol.
Combines a positive long-side selection (mom × consistency) and avoids the
fragile far-tail of pure momentum by inverse-vol weighting.
"""
mom = prices.shift(21).pct_change(231)
rets = prices.pct_change(fill_method=None)
pos_days = (rets > 0).rolling(252, min_periods=126).mean()
vol = rets.rolling(60, min_periods=40).std() * np.sqrt(252)
mom_r = mom.rank(axis=1, pct=True, na_option="keep")
cons_r = pos_days.rank(axis=1, pct=True, na_option="keep")
inv_vol_r = (-vol).rank(axis=1, pct=True, na_option="keep")
return 0.4 * mom_r + 0.3 * cons_r + 0.3 * inv_vol_r
def _signal_mom_x_lowvol(prices: pd.DataFrame) -> pd.DataFrame:
"""Momentum filtered by low-vol — long winners, short LOW-vol losers.
Reduces meme-stock blowups on the short leg by avoiding high-vol losers.
"""
mom = prices.shift(21).pct_change(231)
rets = prices.pct_change(fill_method=None)
vol = rets.rolling(60, min_periods=40).std() * np.sqrt(252)
mom_r = mom.rank(axis=1, pct=True, na_option="keep")
inv_vol_r = (-vol).rank(axis=1, pct=True, na_option="keep")
return 0.5 * mom_r + 0.5 * inv_vol_r
SIGNAL_REGISTRY = {
"mom_12_1": _signal_mom_12_1,
"reversal_1m": _signal_reversal_1m,
"reversal_5d": _signal_reversal_5d,
"recovery_63": _signal_recovery_63,
"low_vol": _signal_low_vol,
"quality_mom": _signal_quality_mom,
"mom_x_lowvol": _signal_mom_x_lowvol,
}
class IndustryNeutralLSMomentum(Strategy):
"""Industry-neutral long/short portfolio with selectable signal."""
def __init__(
self,
rebal_freq: int = 21,
mom_lookback: int = 252,
mom_skip: int = 21,
long_pct: float = 0.20,
short_pct: float = 0.20,
min_sector_size: int = 5,
sector_map: pd.Series | None = None,
gross_long: float = 1.0,
gross_short: float = 1.0,
signal_name: str = "mom_12_1",
) -> None:
self.rebal_freq = rebal_freq
self.mom_lookback = mom_lookback
self.mom_skip = mom_skip
self.long_pct = long_pct
self.short_pct = short_pct
self.min_sector_size = min_sector_size
self.sector_map = sector_map
self.gross_long = gross_long
self.gross_short = gross_short
if signal_name not in SIGNAL_REGISTRY:
raise ValueError(f"Unknown signal: {signal_name}")
self.signal_name = signal_name
self.signal_func = SIGNAL_REGISTRY[signal_name]
def _resolve_sector_map(self, columns: list[str]) -> pd.Series:
if self.sector_map is not None:
return self.sector_map.reindex(columns)
df = fetch_sp500_sectors()
s = df["GICS Sector"]
return s.reindex(columns)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
cols = list(data.columns)
sectors = self._resolve_sector_map(cols)
mom = self.signal_func(data)
weights = pd.DataFrame(0.0, index=data.index, columns=cols)
warmup = self.mom_lookback + 5
# Pre-compute which rows are rebal days
rebal_idx = list(range(warmup, len(data), self.rebal_freq))
rebal_set = set(rebal_idx)
# Group columns by sector
sector_to_cols: dict[str, list[str]] = {}
for c in cols:
s = sectors.get(c)
if pd.isna(s):
continue
sector_to_cols.setdefault(s, []).append(c)
for t in rebal_idx:
row_mom = mom.iloc[t]
longs: dict[str, float] = {}
shorts: dict[str, float] = {}
for sector, members in sector_to_cols.items():
ms = row_mom.reindex(members).dropna()
if len(ms) < self.min_sector_size:
continue
n_long = max(1, int(round(len(ms) * self.long_pct)))
n_short = max(1, int(round(len(ms) * self.short_pct)))
ranked = ms.sort_values(ascending=False)
long_picks = ranked.head(n_long).index
short_picks = ranked.tail(n_short).index
for sym in long_picks:
longs[sym] = longs.get(sym, 0.0) + 1.0
for sym in short_picks:
shorts[sym] = shorts.get(sym, 0.0) - 1.0
if not longs or not shorts:
continue
# Equal-weight within long leg and short leg
n_l = sum(longs.values())
n_s = -sum(shorts.values())
for sym in longs:
longs[sym] = self.gross_long * longs[sym] / n_l
for sym in shorts:
shorts[sym] = self.gross_short * shorts[sym] / n_s
for sym, w in longs.items():
weights.iat[t, cols.index(sym)] = w
for sym, w in shorts.items():
weights.iat[t, cols.index(sym)] = w
# Forward-fill between rebal dates
non_rebal_mask = pd.Series(True, index=data.index)
for i in rebal_idx:
non_rebal_mask.iat[i] = False
weights[non_rebal_mask.values] = np.nan
weights = weights.ffill().fillna(0.0)
weights.iloc[:warmup] = 0.0
return weights.shift(1).fillna(0.0)
__all__ = ["IndustryNeutralLSMomentum", "fetch_sp500_sectors"]

215
strategies/trend_rider_v8.py
View File

@@ -1,215 +0,0 @@
"""TrendRider V8 — V7 + TMF risk-off upgrade.
Extends V7 (V3 regime + vol-target + profit-take) with a smarter risk-off
basket: when TLT is in a bull regime (above its MA200), TMF (3x long-duration
Treasuries) joins the risk-off momentum selection alongside GLD and DBC.
Why this works
--------------
V7 spends ~30-40% of time in risk-off, holding GLD or DBC which average
low single-digit returns. During equity bear markets the Fed typically cuts
rates, driving long bonds higher — TMF (3x TLT) captures this convexity.
2020 crash: TMF ~+60% while equities fell 34%.
The TLT MA200 gate prevents TMF allocation during bond bear markets
(e.g. 2022 rate-hiking cycle where TLT fell 31%).
PIT safety
----------
V3's generate_signals uses prices through day t-1 internally, then applies
a terminal shift(1). So V3's output weight at row i uses data through day
i-2. The TMF swap and TLT gate must match this information set: all
lookups use data through day i-2 (shift(2) for vectorized signals,
iloc[i-2] for point lookups).
Profit-take is applied ONLY to risk-on assets (TQQQ/UPRO). Risk-off
assets (GLD, DBC, TMF) are exempt because:
1. TMF can gain 30%+ during rate-cut cycles — PT would sell at the
worst possible time.
2. Risk-off is already regime-gated; PT on defensive assets is redundant.
"""
from __future__ import annotations
import numpy as np
import pandas as pd
from strategies.base import Strategy
from strategies.permanent import TrendRiderV3
class TrendRiderV8(Strategy):
"""V7 architecture + TMF risk-off with bond-regime gate.
Pipeline:
Layer 1 — V3 regime engine → risk-on / risk-off weights
Layer 1b — TMF swap (PIT-aligned to V3's info set: data through i-2)
Layer 2 — Vol-target overlay
Layer 3 — Profit-take (risk-on assets only; risk-off exempt)
"""
def __init__(
self,
# V3 regime engine
ma_long: int = 150,
signal: str = "SPY",
risk_on: tuple[str, ...] = ("TQQQ", "UPRO"),
risk_off: tuple[str, ...] = ("GLD", "DBC"),
# TMF risk-off
tmf_symbol: str = "TMF",
tlt_symbol: str = "TLT",
tlt_ma_window: int = 200,
tmf_mom_lookback: int = 63,
# Vol-target overlay
target_vol: float = 0.36,
vol_window: int = 60,
min_lev: float = 0.75,
max_lev: float = 1.0,
# Profit-take overlay
pt_threshold: float = 0.30,
pt_band: float = 0.10,
pt_park: str = "SHY",
# V3 passthrough
**v3_kwargs,
) -> None:
self.target_vol = target_vol
self.vol_window = vol_window
self.min_lev = min_lev
self.max_lev = max_lev
self.pt_threshold = pt_threshold
self.pt_band = pt_band
self.pt_park = pt_park
self.risk_on = risk_on
self.risk_off = risk_off
self.tmf_symbol = tmf_symbol
self.tlt_symbol = tlt_symbol
self.tlt_ma_window = tlt_ma_window
self.tmf_mom_lookback = tmf_mom_lookback
self.v3 = TrendRiderV3(
signal=signal, risk_on=risk_on, risk_off=risk_off,
ma_long=ma_long, **v3_kwargs,
)
def generate_signals(self, data: pd.DataFrame) -> pd.DataFrame:
# --- Layer 1: V3 regime weights (already shift(1)'d) ---
# w[i] uses data through day i-2.
w = self.v3.generate_signals(data)
# --- Layer 1b: TMF risk-off swap ---
# PIT: must use data through day i-2 to match V3's info set.
# shift(2) on vectorized signals; iloc[i-2] for point lookups.
tmf = self.tmf_symbol
tlt = self.tlt_symbol
if tmf in data.columns and tlt in data.columns:
tlt_ma = data[tlt].rolling(self.tlt_ma_window).mean()
tlt_bull = (data[tlt] > tlt_ma).shift(2).fillna(False)
roff_cols = [c for c in self.risk_off if c in w.columns]
if tmf not in w.columns:
w[tmf] = 0.0
lb = self.tmf_mom_lookback
for i in range(lb + 3, len(w)):
roff_weight = sum(float(w.iat[i, w.columns.get_loc(c)]) for c in roff_cols)
if roff_weight < 1e-8:
continue
if not tlt_bull.iloc[i]:
continue
best_sym, best_r = None, -np.inf
for sym in roff_cols + [tmf]:
if sym not in data.columns:
continue
p_now = data[sym].iloc[i - 2]
p_past = data[sym].iloc[i - 2 - lb]
if pd.notna(p_now) and pd.notna(p_past) and p_past > 0:
r = float(p_now / p_past - 1.0)
if r > best_r:
best_r, best_sym = r, sym
if best_sym is not None:
for c in roff_cols:
w.iat[i, w.columns.get_loc(c)] = 0.0
w.iat[i, w.columns.get_loc(tmf)] = 0.0
w.iat[i, w.columns.get_loc(best_sym)] = roff_weight
# --- Layer 2: Vol-target overlay ---
daily_ret = data.pct_change(fill_method=None).fillna(0.0)
common = w.columns.intersection(daily_ret.columns)
port_rets = (w[common] * daily_ret[common]).sum(axis=1)
realized_vol = (
port_rets.rolling(self.vol_window, min_periods=21).std()
* np.sqrt(252)
)
scale = (self.target_vol / realized_vol).clip(
lower=self.min_lev, upper=self.max_lev,
)
scale = scale.shift(1).fillna(1.0)
w = w.mul(scale, axis=0)
# --- Layer 3: Profit-take with hysteresis (risk-on only) ---
# Risk-off assets (GLD, DBC, TMF) are exempt from PT.
if self.pt_threshold <= 0:
return w
risk_off_set = set(self.risk_off) | {self.tmf_symbol}
held = w.idxmax(axis=1)
max_w = w.max(axis=1)
held[max_w < 1e-8] = ""
park_col = self.pt_park if self.pt_park in w.columns else ""
entry_price: float | None = None
current_sym: str | None = None
is_stopped = False
restore_level = self.pt_threshold - self.pt_band
for i in range(len(w)):
sym = held.iloc[i]
if not sym or max_w.iloc[i] < 1e-8:
current_sym = None
entry_price = None
is_stopped = False
continue
if sym != current_sym:
current_sym = sym
entry_price = (
float(data[sym].iloc[i - 1])
if i > 0 and sym in data.columns
else None
)
is_stopped = False
continue
# Skip PT for risk-off assets
if sym in risk_off_set:
continue
if (
entry_price is None
or entry_price <= 0
or sym not in data.columns
):
continue
yesterday = float(data[sym].iloc[i - 1]) if i > 0 else float(data[sym].iloc[i])
gain = yesterday / entry_price - 1.0
if is_stopped:
if gain < restore_level:
is_stopped = False
else:
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
else:
if gain >= self.pt_threshold:
is_stopped = True
w.iloc[i] = 0.0
if park_col:
w.at[w.index[i], park_col] = scale.iloc[i]
return w
__all__ = ["TrendRiderV8"]

118
tests/test_alpha_signals.py
View File

@@ -1,118 +0,0 @@
import unittest
import warnings
import numpy as np
import pandas as pd
class AlphaSignalTests(unittest.TestCase):
def test_build_regime_filter_requires_market_trend_and_non_market_leader(self):
from research.regime_filters import build_regime_filter
dates = pd.date_range("2023-01-01", periods=260, freq="D")
spy = pd.Series([100.0 + i for i in range(260)], index=dates)
qqq_leader = pd.Series([100.0 + 1.4 * i for i in range(260)], index=dates)
xlu = pd.Series([100.0 + 0.2 * i for i in range(260)], index=dates)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
bullish = build_regime_filter(pd.DataFrame({"SPY": spy, "QQQ": qqq_leader, "XLU": xlu}))
qqq_laggard = pd.Series([100.0 + 0.5 * i for i in range(260)], index=dates)
no_leader = build_regime_filter(pd.DataFrame({"SPY": spy, "QQQ": qqq_laggard, "XLU": xlu}))
self.assertEqual(len(caught), 0)
self.assertFalse(bool(bullish.iloc[199]))
self.assertTrue(bool(bullish.iloc[-1]))
self.assertFalse(bool(no_leader.iloc[-1]))
def test_build_regime_filter_handles_internal_missing_prices_without_warnings(self):
from research.regime_filters import build_regime_filter
dates = pd.date_range("2023-01-01", periods=260, freq="D")
spy = pd.Series([100.0 + i for i in range(260)], index=dates)
qqq = pd.Series([100.0 + 1.4 * i for i in range(260)], index=dates)
qqq.iloc[120] = np.nan
etf_close = pd.DataFrame({"SPY": spy, "QQQ": qqq, "XLU": 100.0}, index=dates)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
regime = build_regime_filter(etf_close)
self.assertEqual(len(caught), 0)
self.assertEqual(str(regime.dtype), "bool")
def test_breakout_after_compression_score_is_shifted_and_rewards_breakout_profile(self):
from research.event_factors import breakout_after_compression_score
dates = pd.date_range("2024-01-01", periods=80, freq="D")
aaa_close = [100.0 + i for i in range(60)] + [159.0 + 0.05 * i for i in range(20)]
bbb_close = [100.0 + i for i in range(60)] + [150.0 - i for i in range(20)]
close = pd.DataFrame({"AAA": aaa_close, "BBB": bbb_close}, index=dates)
high = pd.DataFrame(
{
"AAA": [value + 0.4 for value in aaa_close],
"BBB": [value + 4.0 for value in bbb_close],
},
index=dates,
)
low = pd.DataFrame(
{
"AAA": [value - 0.4 for value in aaa_close],
"BBB": [value - 4.0 for value in bbb_close],
},
index=dates,
)
volume = pd.DataFrame(
{
"AAA": [1_000.0] * 79 + [1_000.0],
"BBB": [1_000.0] * 80,
},
index=dates,
)
volume.loc[dates[-2], "AAA"] = 6_000.0
shifted_result = breakout_after_compression_score(close, high, low, volume)
self.assertGreater(
shifted_result.loc[dates[-1], "AAA"],
shifted_result.loc[dates[-1], "BBB"],
)
changed_last_day = close.copy()
changed_last_day_high = high.copy()
changed_last_day_low = low.copy()
changed_last_day_volume = volume.copy()
changed_last_day.loc[dates[-1], "AAA"] = 120.0
changed_last_day_high.loc[dates[-1], "AAA"] = 130.0
changed_last_day_low.loc[dates[-1], "AAA"] = 110.0
changed_last_day_volume.loc[dates[-1], "AAA"] = 20_000.0
last_day_changed_result = breakout_after_compression_score(
changed_last_day,
changed_last_day_high,
changed_last_day_low,
changed_last_day_volume,
)
self.assertEqual(
shifted_result.loc[dates[-1], "AAA"],
last_day_changed_result.loc[dates[-1], "AAA"],
)
def test_breakout_after_compression_score_keeps_float_output_when_denominators_hit_zero(self):
from research.event_factors import breakout_after_compression_score
dates = pd.date_range("2024-01-01", periods=70, freq="D")
close = pd.DataFrame({"AAA": [10.0] * 70}, index=dates)
high = pd.DataFrame({"AAA": [10.0] * 70}, index=dates)
low = pd.DataFrame({"AAA": [10.0] * 70}, index=dates)
volume = pd.DataFrame({"AAA": [0.0] * 70}, index=dates)
score = breakout_after_compression_score(close, high, low, volume)
self.assertEqual(str(score.dtypes["AAA"]), "float64")
self.assertTrue(pd.isna(score.iloc[-1]["AAA"]))
if __name__ == "__main__":
unittest.main()

46
tests/test_fetch_historical.py
View File

@@ -1,46 +0,0 @@
import tempfile
import unittest
from pathlib import Path
from unittest import mock
import pandas as pd
from research import fetch_historical
class FetchHistoricalTests(unittest.TestCase):
def test_fetch_all_historical_ohlcv_writes_field_specific_csvs(self):
dates = pd.to_datetime(["2024-01-02", "2024-01-03"])
raw = pd.DataFrame(
{
("Close", "AAA"): [10.0, 11.0],
("Close", "BBB"): [20.0, 21.0],
("High", "AAA"): [10.5, 11.5],
("High", "BBB"): [20.5, 21.5],
("Low", "AAA"): [9.5, 10.5],
("Low", "BBB"): [19.5, 20.5],
("Volume", "AAA"): [1000.0, 1100.0],
("Volume", "BBB"): [2000.0, 2100.0],
},
index=dates,
)
raw.columns = pd.MultiIndex.from_tuples(raw.columns)
with tempfile.TemporaryDirectory() as tmpdir:
with mock.patch.object(fetch_historical, "DATA_DIR", tmpdir):
with mock.patch.object(fetch_historical, "OUT_PATH", str(Path(tmpdir) / "us_pit.csv")):
with mock.patch("research.fetch_historical.uh.load_sp500_history", return_value={"AAA": [[None, None]], "BBB": [[None, None]]}):
with mock.patch("research.fetch_historical.uh.all_tickers_ever", return_value=["AAA", "BBB"]):
with mock.patch("research.fetch_historical.yf.download", return_value=raw):
panels = fetch_historical.fetch_all_historical_ohlcv(force=True)
self.assertEqual(set(panels.keys()), {"close", "high", "low", "volume"})
self.assertTrue((Path(tmpdir) / "us_pit.csv").exists())
self.assertTrue((Path(tmpdir) / "us_pit_close.csv").exists())
self.assertTrue((Path(tmpdir) / "us_pit_high.csv").exists())
self.assertTrue((Path(tmpdir) / "us_pit_low.csv").exists())
self.assertTrue((Path(tmpdir) / "us_pit_volume.csv").exists())
if __name__ == "__main__":
unittest.main()

91
tests/test_trend_rider_robustness.py
View File

@@ -1,91 +0,0 @@
import unittest
import numpy as np
import pandas as pd
from research import trend_rider_robustness as robustness
class TrendRiderRobustnessTests(unittest.TestCase):
def test_evaluate_weights_reports_core_risk_metrics(self):
dates = pd.bdate_range("2024-01-01", periods=6)
prices = pd.DataFrame(
{
"AAA": [100, 110, 105, 120, 118, 130],
"BBB": [50, 49, 51, 50, 52, 53],
},
index=dates,
)
weights = pd.DataFrame(
{
"AAA": [0, 1, 1, 0, 0, 1],
"BBB": [0, 0, 0, 1, 1, 0],
},
index=dates,
)
result = robustness.evaluate_weights("synthetic", weights, prices, transaction_cost=0.001)
self.assertEqual(result.name, "synthetic")
self.assertGreater(result.final_multiple, 1.0)
self.assertLessEqual(result.max_drawdown, 0.0)
self.assertGreater(result.switches, 0)
self.assertGreater(result.avg_daily_turnover, 0.0)
def test_parameter_sweep_returns_rankable_rows(self):
dates = pd.bdate_range("2023-01-02", periods=320)
trend = np.linspace(100, 180, len(dates))
prices = pd.DataFrame(
{
"SPY": trend,
"TQQQ": trend * 1.5,
"UPRO": trend * 1.4,
"GLD": np.linspace(100, 105, len(dates)),
"DBC": np.linspace(90, 95, len(dates)),
},
index=dates,
)
sweep = robustness.parameter_sweep(
prices,
variants=[
{"vol_enter": 0.14, "dd_stop": 0.05, "peak_enter": 0.02, "mom_lookback": 63},
{"vol_enter": 0.16, "dd_stop": 0.07, "peak_enter": 0.03, "mom_lookback": 84},
],
start="2023-01-02",
)
self.assertEqual(len(sweep), 2)
self.assertIn("cagr", sweep.columns)
self.assertIn("max_drawdown", sweep.columns)
self.assertTrue(sweep["cagr"].notna().all())
def test_candidate_weights_include_v4_and_market_benchmarks(self):
dates = pd.bdate_range("2023-01-02", periods=320)
trend = np.linspace(100, 180, len(dates))
prices = pd.DataFrame(
{
"SPY": trend,
"QQQ": trend * 1.1,
"SSO": trend * 1.5,
"QLD": trend * 1.6,
"UPRO": trend * 2.0,
"TQQQ": trend * 2.2,
"SHY": np.linspace(100, 103, len(dates)),
"IEF": np.linspace(100, 104, len(dates)),
"TLT": np.linspace(100, 105, len(dates)),
"GLD": np.linspace(100, 115, len(dates)),
"DBC": np.linspace(90, 105, len(dates)),
},
index=dates,
)
candidates = robustness.candidate_weights(prices)
self.assertIn("TrendRiderV4", candidates)
self.assertIn("SPY Buy&Hold", candidates)
self.assertIn("QQQ Buy&Hold", candidates)
if __name__ == "__main__":
unittest.main()

276
tests/test_us_alpha_pipeline.py
View File

@@ -1,276 +0,0 @@
import unittest
from pathlib import Path
from unittest import mock
import pandas as pd
class USAlphaPipelineTests(unittest.TestCase):
def test_build_equal_weight_portfolio_caps_holdings_under_ties(self):
from research.us_alpha_pipeline import _build_equal_weight_portfolio
dates = pd.date_range("2024-01-01", periods=2, freq="D")
score = pd.DataFrame(
{
"AAA": [0.9, 0.9],
"BBB": [0.9, 0.9],
"CCC": [0.9, 0.9],
},
index=dates,
)
tradable_mask = pd.DataFrame(True, index=dates, columns=score.columns)
regime = pd.Series([True, True], index=dates)
weights = _build_equal_weight_portfolio(score, tradable_mask, regime, top_n=2)
self.assertEqual(int((weights.iloc[-1] > 0).sum()), 2)
self.assertAlmostEqual(float(weights.iloc[-1].sum()), 1.0)
def test_equity_curve_uses_prior_day_weights_for_returns(self):
from research.us_alpha_pipeline import _equity_curve
dates = pd.date_range("2024-01-01", periods=3, freq="D")
close = pd.DataFrame({"AAA": [1.0, 2.0, 4.0]}, index=dates)
weights = pd.DataFrame({"AAA": [0.0, 1.0, 0.0]}, index=dates)
equity = _equity_curve(close, weights)
self.assertEqual(float(equity.iloc[1]), 2.0)
self.assertEqual(float(equity.iloc[2]), 2.0)
def test_summarize_equity_window_returns_nans_when_history_is_too_short(self):
from research.us_alpha_report import summarize_equity_window
dates = pd.date_range("2024-01-01", periods=10, freq="D")
equity = pd.Series([1.0 + 0.01 * i for i in range(10)], index=dates)
summary = summarize_equity_window(equity, "demo", window_years=1)
self.assertTrue(pd.isna(summary["CAGR"]))
self.assertTrue(pd.isna(summary["Sharpe"]))
self.assertTrue(pd.isna(summary["MaxDD"]))
self.assertTrue(pd.isna(summary["TotalRet"]))
def test_run_alpha_pipeline_returns_expected_strategy_summary(self):
from research.us_alpha_pipeline import run_alpha_pipeline
dates = pd.date_range("2023-01-01", periods=400, freq="D")
aaa_close = [50.0 + 0.20 * i for i in range(400)]
bbb_close = [55.0 + 0.12 * i for i in range(400)]
ccc_close = [60.0 + 0.05 * i for i in range(400)]
close = pd.DataFrame(
{
"AAA": aaa_close,
"BBB": bbb_close,
"CCC": ccc_close,
},
index=dates,
)
high = pd.DataFrame(
{
"AAA": [value + 0.5 for value in aaa_close],
"BBB": [value + 1.0 for value in bbb_close],
"CCC": [value + 1.5 for value in ccc_close],
},
index=dates,
)
low = pd.DataFrame(
{
"AAA": [value - 0.5 for value in aaa_close],
"BBB": [value - 1.0 for value in bbb_close],
"CCC": [value - 1.5 for value in ccc_close],
},
index=dates,
)
volume = pd.DataFrame(
{
"AAA": [1_500_000.0] * 400,
"BBB": [1_400_000.0] * 400,
"CCC": [1_300_000.0] * 400,
},
index=dates,
)
volume.loc[dates[-2], "AAA"] = 4_000_000.0
etf_close = pd.DataFrame(
{
"SPY": [300.0 + 0.8 * i for i in range(400)],
"QQQ": [280.0 + 1.1 * i for i in range(400)],
"XLF": [200.0 + 0.4 * i for i in range(400)],
},
index=dates,
)
market_data = {
"close": close,
"high": high,
"low": low,
"volume": volume,
}
summary = run_alpha_pipeline(
market_data=market_data,
etf_close=etf_close,
pit_membership=None,
windows=(1,),
top_n=2,
)
required_columns = {"strategy", "window_years", "CAGR", "Sharpe", "MaxDD", "TotalRet"}
self.assertTrue(required_columns.issubset(summary.columns))
self.assertEqual(set(summary["strategy"]), {"breakout_regime", "rank_blend_regime"})
self.assertEqual(set(summary["window_years"]), {1})
self.assertEqual(len(summary), 2)
self.assertTrue(summary[["CAGR", "Sharpe", "MaxDD", "TotalRet"]].notna().all().all())
def test_run_alpha_pipeline_includes_fundamental_variants_when_score_is_supplied(self):
from research.us_alpha_pipeline import run_alpha_pipeline
dates = pd.date_range("2023-01-01", periods=400, freq="D")
close = pd.DataFrame(
{
"AAA": [50.0 + 0.20 * i for i in range(400)],
"BBB": [55.0 + 0.12 * i for i in range(400)],
"CCC": [60.0 + 0.05 * i for i in range(400)],
},
index=dates,
)
high = close + 1.0
low = close - 1.0
volume = pd.DataFrame(
{
"AAA": [1_500_000.0] * 400,
"BBB": [1_400_000.0] * 400,
"CCC": [1_300_000.0] * 400,
},
index=dates,
)
etf_close = pd.DataFrame(
{
"SPY": [300.0 + 0.8 * i for i in range(400)],
"QQQ": [280.0 + 1.1 * i for i in range(400)],
"XLF": [200.0 + 0.4 * i for i in range(400)],
},
index=dates,
)
market_data = {
"close": close,
"high": high,
"low": low,
"volume": volume,
}
fundamental_score = pd.DataFrame(
{
"AAA": [0.9] * 400,
"BBB": [0.6] * 400,
"CCC": [0.3] * 400,
},
index=dates,
)
summary = run_alpha_pipeline(
market_data=market_data,
etf_close=etf_close,
pit_membership=None,
windows=(1,),
top_n=2,
fundamental_score=fundamental_score,
)
self.assertEqual(
set(summary["strategy"]),
{
"breakout_regime",
"rank_blend_regime",
"fundamental_regime",
"breakout_fundamental_regime",
"rank_blend_fundamental_regime",
},
)
def test_run_alpha_pipeline_close_only_fallback_skips_breakout_and_uses_spy_regime(self):
from research.us_alpha_pipeline import run_alpha_pipeline
dates = pd.date_range("2023-01-01", periods=400, freq="D")
close = pd.DataFrame(
{
"AAA": [50.0 + 0.20 * i for i in range(400)],
"BBB": [55.0 + 0.12 * i for i in range(400)],
"CCC": [60.0 + 0.05 * i for i in range(400)],
},
index=dates,
)
market_data = {
"close": close,
"high": pd.DataFrame(index=dates, columns=close.columns),
"low": pd.DataFrame(index=dates, columns=close.columns),
"volume": pd.DataFrame(index=dates, columns=close.columns),
}
etf_close = pd.DataFrame({"SPY": [300.0 + 0.8 * i for i in range(400)]}, index=dates)
fundamental_score = pd.DataFrame(
{
"AAA": [0.9] * 400,
"BBB": [0.6] * 400,
"CCC": [0.3] * 400,
},
index=dates,
)
summary = run_alpha_pipeline(
market_data=market_data,
etf_close=etf_close,
pit_membership=None,
windows=(1,),
top_n=2,
fundamental_score=fundamental_score,
)
self.assertEqual(
set(summary["strategy"]),
{
"rank_blend_regime",
"fundamental_regime",
"rank_blend_fundamental_regime",
},
)
self.assertTrue(summary[["CAGR", "Sharpe", "MaxDD", "TotalRet"]].notna().all().all())
def test_run_saved_pit_alpha_pipeline_reads_saved_inputs(self):
from research.us_alpha_pipeline import run_saved_pit_alpha_pipeline
dates = pd.date_range("2024-01-01", periods=320, freq="D")
close = pd.DataFrame(
{
"AAA": [50.0 + 0.2 * i for i in range(320)],
"BBB": [40.0 + 0.1 * i for i in range(320)],
},
index=dates,
)
high = close + 1.0
low = close - 1.0
volume = pd.DataFrame({"AAA": [2_500_000.0] * 320, "BBB": [2_000_000.0] * 320}, index=dates)
etf_close = pd.DataFrame(
{"SPY": [300.0 + 0.8 * i for i in range(320)], "QQQ": [280.0 + 1.1 * i for i in range(320)]},
index=dates,
)
with self.subTest("saved_inputs"):
import tempfile
with tempfile.TemporaryDirectory() as tmpdir:
close.to_csv(Path(tmpdir) / "us_pit_close.csv")
high.to_csv(Path(tmpdir) / "us_pit_high.csv")
low.to_csv(Path(tmpdir) / "us_pit_low.csv")
volume.to_csv(Path(tmpdir) / "us_pit_volume.csv")
etf_close.to_csv(Path(tmpdir) / "us_etf.csv")
intervals = {"AAA": [[None, None]], "BBB": [[None, None]]}
with mock.patch("research.us_alpha_pipeline.uh.load_sp500_history", return_value=intervals):
summary = run_saved_pit_alpha_pipeline(data_dir=tmpdir, windows=(1,), top_n=1)
self.assertEqual(set(summary["strategy"]), {"breakout_regime", "rank_blend_regime"})
if __name__ == "__main__":
unittest.main()

65
tests/test_us_combo_sweep.py
View File

@@ -1,65 +0,0 @@
import unittest
import pandas as pd
class USComboSweepTests(unittest.TestCase):
def test_apply_filter_threshold_masks_names_below_rank_cutoff(self):
from research.us_combo_sweep import apply_filter_threshold
index = pd.DatetimeIndex([pd.Timestamp("2024-01-31")])
score = pd.DataFrame({"AAA": [0.9], "BBB": [0.8], "CCC": [0.7]}, index=index)
filter_rank = pd.DataFrame({"AAA": [0.2], "BBB": [0.6], "CCC": [0.9]}, index=index)
filtered = apply_filter_threshold(score, filter_rank, min_rank=0.5)
self.assertTrue(pd.isna(filtered.iloc[0]["AAA"]))
self.assertEqual(float(filtered.iloc[0]["BBB"]), 0.8)
self.assertEqual(float(filtered.iloc[0]["CCC"]), 0.7)
def test_run_combo_backtests_returns_candidates_and_yearly_summary(self):
from research.us_combo_sweep import run_combo_backtests
dates = pd.date_range("2022-01-01", periods=800, freq="D")
close = pd.DataFrame(
{
"AAA": [50.0 + 0.12 * i for i in range(800)],
"BBB": [40.0 + 0.08 * i for i in range(800)],
"CCC": [35.0 + 0.06 * i for i in range(800)],
"DDD": [30.0 + 0.04 * i for i in range(800)],
"EEE": [25.0 + 0.03 * i for i in range(800)],
"FFF": [20.0 + 0.02 * i for i in range(800)],
"GGG": [18.0 + 0.015 * i for i in range(800)],
"HHH": [16.0 + 0.010 * i for i in range(800)],
"III": [14.0 + 0.008 * i for i in range(800)],
"JJJ": [12.0 + 0.005 * i for i in range(800)],
"SPY": [300.0 + 0.20 * i for i in range(800)],
},
index=dates,
)
fundamental_score = pd.DataFrame(
{
"AAA": [0.95] * 800,
"BBB": [0.90] * 800,
"CCC": [0.85] * 800,
"DDD": [0.80] * 800,
"EEE": [0.75] * 800,
"FFF": [0.70] * 800,
"GGG": [0.65] * 800,
"HHH": [0.60] * 800,
"III": [0.55] * 800,
"JJJ": [0.50] * 800,
},
index=dates,
)
yearly, summary = run_combo_backtests(close, fundamental_score, top_n=3)
self.assertIn("Recovery+Mom Top10", yearly.columns)
self.assertIn("rm_fund_tilt_20", yearly.columns)
self.assertIn("rm_fund_filter_50", yearly.columns)
self.assertIn("mega_quality_fund", set(summary["strategy"]))
if __name__ == "__main__":
unittest.main()

52
tests/test_us_fundamentals.py
View File

@@ -1,52 +0,0 @@
import unittest
import pandas as pd
class USFundamentalTransformsTests(unittest.TestCase):
def test_build_quarterly_factor_pack_derives_expected_signals(self):
from research.us_fundamentals import build_quarterly_factor_pack
quarter_ends = pd.to_datetime(
["2023-03-31", "2023-06-30", "2023-09-30", "2023-12-31", "2024-03-31"]
)
close = pd.DataFrame(
{"AAA": [100.0, 105.0], "BBB": [50.0, 48.0]},
index=pd.to_datetime(["2024-06-03", "2024-06-04"]),
)
quarterly = {
"net_income": pd.DataFrame(
{"AAA": [10.0, 11.0, 12.0, 13.0, 14.0], "BBB": [4.0, 4.0, 5.0, 5.0, 5.0]},
index=quarter_ends,
),
"gross_profit": pd.DataFrame(
{"AAA": [30.0, 31.0, 32.0, 33.0, 34.0], "BBB": [10.0, 10.0, 11.0, 11.0, 11.0]},
index=quarter_ends,
),
"equity": pd.DataFrame(
{"AAA": [200.0, 205.0, 210.0, 215.0, 220.0], "BBB": [80.0, 81.0, 82.0, 83.0, 84.0]},
index=quarter_ends,
),
"assets": pd.DataFrame(
{"AAA": [300.0, 305.0, 310.0, 315.0, 320.0], "BBB": [130.0, 131.0, 132.0, 133.0, 134.0]},
index=quarter_ends,
),
"shares": pd.DataFrame(
{"AAA": [10.0, 10.0, 10.0, 10.0, 10.0], "BBB": [10.0, 10.0, 11.0, 11.0, 11.0]},
index=quarter_ends,
),
}
factor_pack = build_quarterly_factor_pack(quarterly, close, lag_days=60)
self.assertIn("composite", factor_pack)
self.assertIn("book_to_market", factor_pack)
self.assertEqual(list(factor_pack["composite"].columns), ["AAA", "BBB"])
self.assertGreater(
float(factor_pack["composite"].iloc[-1]["AAA"]),
float(factor_pack["composite"].iloc[-1]["BBB"]),
)
if __name__ == "__main__":
unittest.main()

213
tests/test_us_universe.py
View File

@@ -1,213 +0,0 @@
import unittest
import warnings
import pandas as pd
class BuildTradableMaskTests(unittest.TestCase):
def test_build_tradable_mask_uses_only_lagged_price_and_liquidity_inputs(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=4, freq="D")
close = pd.DataFrame({"AAA": [4.0, 10.0, 10.0, 10.0]}, index=dates)
volume = pd.DataFrame({"AAA": [float("nan"), 200.0, 200.0, 200.0]}, index=dates)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=None,
min_price=5.0,
min_dollar_volume=1000.0,
min_history_days=2,
min_valid_volume_days=2,
liquidity_window=2,
)
expected = pd.DataFrame({"AAA": [False, False, False, True]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_uses_only_lagged_history(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=4, freq="D")
close = pd.DataFrame({"AAA": [10.0, float("nan"), 10.0, 10.0]}, index=dates)
volume = pd.DataFrame({"AAA": [200.0, 200.0, 200.0, 200.0]}, index=dates)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=None,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=2,
min_valid_volume_days=1,
liquidity_window=1,
)
expected = pd.DataFrame({"AAA": [False, False, False, False]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_requires_membership_history_before_first_eligible_day(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=4, freq="D")
close = pd.DataFrame({"AAA": [10.0, 10.0, 10.0, 10.0]}, index=dates)
volume = pd.DataFrame({"AAA": [200.0, 200.0, 200.0, 200.0]}, index=dates)
pit_membership = pd.DataFrame({"AAA": [False, False, True, True]}, index=dates)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=pit_membership,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=1,
min_valid_volume_days=1,
liquidity_window=1,
)
expected = pd.DataFrame({"AAA": [False, False, False, True]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_aligns_pit_membership_without_truthy_carryover(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=3, freq="D")
close = pd.DataFrame(
{
"AAA": [10.0, 10.0, 10.0],
"BBB": [12.0, 12.0, 12.0],
},
index=dates,
)
volume = pd.DataFrame(
{
"AAA": [1_000_000.0, 1_000_000.0, 1_000_000.0],
"BBB": [1_000_000.0, 1_000_000.0, 1_000_000.0],
},
index=dates,
)
pit_membership = pd.DataFrame(
{
"BBB": [True, True, False],
"CCC": [True, True, True],
},
index=pd.date_range("2024-01-02", periods=3, freq="D"),
)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=pit_membership,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=1,
min_valid_volume_days=1,
liquidity_window=1,
)
self.assertEqual(len(caught), 0)
expected = pd.DataFrame(
{
"AAA": [False, False, False],
"BBB": [False, False, True],
},
index=dates,
dtype=bool,
)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_treats_missing_membership_cells_as_false(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=3, freq="D")
close = pd.DataFrame({"AAA": [10.0, 10.0, 10.0]}, index=dates)
volume = pd.DataFrame({"AAA": [1_000_000.0, 1_000_000.0, 1_000_000.0]}, index=dates)
pit_membership = pd.DataFrame(
{"AAA": [True, pd.NA, True]},
index=dates,
dtype="boolean",
)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=pit_membership,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=1,
min_valid_volume_days=1,
liquidity_window=1,
)
expected = pd.DataFrame({"AAA": [False, False, False]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_uses_strict_thresholds(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=3, freq="D")
close = pd.DataFrame({"AAA": [5.0, 5.0, 5.0]}, index=dates)
volume = pd.DataFrame({"AAA": [300.0, 300.0, 300.0]}, index=dates)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=None,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=1,
min_valid_volume_days=1,
liquidity_window=1,
)
expected = pd.DataFrame({"AAA": [False, False, False]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_uses_strict_dollar_volume_threshold(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=3, freq="D")
close = pd.DataFrame({"AAA": [8.0, 8.0, 8.0]}, index=dates)
volume = pd.DataFrame({"AAA": [125.0, 125.0, 125.0]}, index=dates)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=None,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=1,
min_valid_volume_days=1,
liquidity_window=1,
)
expected = pd.DataFrame({"AAA": [False, False, False]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
def test_build_tradable_mask_requires_valid_dollar_volume_history(self):
from research.us_universe import build_tradable_mask
dates = pd.date_range("2024-01-01", periods=4, freq="D")
close = pd.DataFrame({"AAA": [10.0, float("nan"), 10.0, 10.0]}, index=dates)
volume = pd.DataFrame({"AAA": [200.0, 200.0, 200.0, 200.0]}, index=dates)
mask = build_tradable_mask(
close=close,
volume=volume,
pit_membership=None,
min_price=5.0,
min_dollar_volume=1_000.0,
min_history_days=1,
min_valid_volume_days=2,
liquidity_window=2,
)
expected = pd.DataFrame({"AAA": [False, False, False, False]}, index=dates, dtype=bool)
pd.testing.assert_frame_equal(mask, expected)
if __name__ == "__main__":
unittest.main()

159
yearly_sweep.py
View File

@@ -1,159 +0,0 @@
"""Run all US strategies and report yearly performance vs SPY."""
import pandas as pd
import numpy as np
import data_manager
from universe import UNIVERSES
from strategies.adaptive_momentum import AdaptiveMomentumStrategy
from strategies.buy_and_hold import BuyAndHoldStrategy
from strategies.dual_momentum import DualMomentumStrategy
from strategies.inverse_vol import InverseVolatilityStrategy
from strategies.mean_reversion import MeanReversionStrategy
from strategies.momentum import MomentumStrategy
from strategies.momentum_quality import MomentumQualityStrategy
from strategies.multi_factor import MultiFactorStrategy
from strategies.recovery_momentum import RecoveryMomentumStrategy
from strategies.trend_following import TrendFollowingStrategy
from main import backtest
def build_strategies(tickers, benchmark, data, top_n):
return {
"Buy & Hold (EW)": (BuyAndHoldStrategy(), data[tickers]),
"Momentum": (MomentumStrategy(lookback=252, skip=21, top_n=top_n), data[tickers]),
"Inverse Volatility": (InverseVolatilityStrategy(vol_window=20), data[tickers]),
"Multi-Factor": (MultiFactorStrategy(tickers=tickers, benchmark=benchmark, top_n=top_n), data),
"Mean Reversion": (MeanReversionStrategy(top_n=top_n), data[tickers]),
"Trend Following": (TrendFollowingStrategy(ma_window=150, momentum_period=126, top_n=top_n), data[tickers]),
"Dual Momentum": (DualMomentumStrategy(top_n=top_n), data[tickers]),
"Momentum+Quality": (MomentumQualityStrategy(momentum_period=252, skip=21, top_n=top_n), data[tickers]),
"Mom+InvVol": (AdaptiveMomentumStrategy(top_n=top_n), data[tickers]),
"Recovery+Mom Top20": (RecoveryMomentumStrategy(top_n=min(20, top_n)), data[tickers]),
"Recovery+Mom Top10": (RecoveryMomentumStrategy(top_n=10), data[tickers]),
}
def annual_return(eq: pd.Series) -> float:
return eq.iloc[-1] / eq.iloc[0] - 1
def max_dd(eq: pd.Series) -> float:
return ((eq / eq.cummax()) - 1).min()
def main():
universe = UNIVERSES["us"]
tickers = universe["fetch"]()
benchmark = universe["benchmark"]
all_tickers = sorted(set(tickers + [benchmark]))
data = data_manager.update("us", all_tickers, with_open=False)
tickers = [t for t in tickers if t in data.columns]
top_n = max(5, len(tickers) // 10)
print(f"Universe: {len(tickers)} stocks + {benchmark}. top_n={top_n}")
print(f"Data range: {data.index[0].date()} to {data.index[-1].date()}")
strategies = build_strategies(tickers, benchmark, data, top_n)
equity = {}
for name, (strat, strat_data) in strategies.items():
print(f"Running {name}...")
equity[name] = backtest(strat, strat_data, initial_capital=10_000)
# SPY benchmark normalized
bench = data[benchmark].dropna()
equity["SPY"] = (bench / bench.iloc[0]) * 10_000
eq_df = pd.DataFrame(equity).sort_index()
# Yearly returns table
years = list(range(2017, 2027))
rows = []
for yr in years:
start = pd.Timestamp(f"{yr}-01-01")
end = pd.Timestamp(f"{yr}-12-31")
window = eq_df.loc[(eq_df.index >= start) & (eq_df.index <= end)].dropna(how="all")
if window.empty:
continue
row = {"Year": yr}
for col in eq_df.columns:
s = window[col].dropna()
if len(s) < 2:
row[col] = np.nan
else:
row[col] = annual_return(s)
rows.append(row)
yr_df = pd.DataFrame(rows).set_index("Year")
# Excess over SPY
excess = yr_df.sub(yr_df["SPY"], axis=0).drop(columns=["SPY"])
print("\n=== Yearly Total Return (%) ===")
print((yr_df * 100).round(2).to_string())
print("\n=== Excess vs SPY (pp) ===")
print((excess * 100).round(2).to_string())
# Best strategy each year (excluding SPY)
strat_only = yr_df.drop(columns=["SPY"])
best_per_year = strat_only.idxmax(axis=1)
best_ret = strat_only.max(axis=1)
spy_ret = yr_df["SPY"]
print("\n=== Best Strategy per Year ===")
print(f"{'Year':<6}{'Strategy':<22}{'Return':>10}{'SPY':>10}{'Excess':>10}")
for yr in best_per_year.index:
s = best_per_year.loc[yr]
r = best_ret.loc[yr]
b = spy_ret.loc[yr]
print(f"{yr:<6}{s:<22}{r*100:>9.2f}%{b*100:>9.2f}%{(r-b)*100:>9.2f}pp")
# Average metrics per strategy
print("\n=== Full-period Summary (across years) ===")
summary = pd.DataFrame({
"Avg Annual Return": strat_only.mean() * 100,
"Median": strat_only.median() * 100,
"Std": strat_only.std() * 100,
"Years Beat SPY": strat_only.gt(spy_ret, axis=0).sum(),
"Best Years": (strat_only.idxmax(axis=1).value_counts()
.reindex(strat_only.columns, fill_value=0)),
})
summary = summary.sort_values("Avg Annual Return", ascending=False)
print(summary.round(2).to_string())
# Overall equity-curve CAGR (compound) across all available years
def cagr(col):
s = eq_df[col].dropna()
yrs = (s.index[-1] - s.index[0]).days / 365.25
if yrs <= 0:
return np.nan
return (s.iloc[-1] / s.iloc[0]) ** (1 / yrs) - 1
print("\n=== Compound Over Full Window (CAGR, Max DD) ===")
cagr_rows = []
for c in eq_df.columns:
s = eq_df[c].dropna()
cagr_rows.append({
"Strategy": c,
"CAGR %": cagr(c) * 100,
"Max DD %": max_dd(s) * 100,
"Total %": (s.iloc[-1] / s.iloc[0] - 1) * 100,
})
cagr_df = pd.DataFrame(cagr_rows).sort_values("CAGR %", ascending=False)
print(cagr_df.round(2).to_string(index=False))
# Best "average" strategy (by mean annual return across full years)
best_avg = summary["Avg Annual Return"].idxmax()
print(f"\n>>> Best average strategy: {best_avg} "
f"({summary.loc[best_avg, 'Avg Annual Return']:.2f}% avg annual return, "
f"beat SPY in {int(summary.loc[best_avg, 'Years Beat SPY'])}/{len(strat_only)} years)")
# Save CSV
out = "data/yearly_sweep.csv"
yr_df.to_csv(out)
print(f"\nSaved yearly returns to {out}")
if __name__ == "__main__":
main()