Gahow Wang
92af4e5ac0
Yahoo purges delisted members: the S&P research panel was missing 209 ever-members (AGN/CELG/ESRX/ATVI/TWTR/XLNX/...), the production store 89, and the S&P 400/600 panels ~180 more. The gap inflated momentum-family factors (overnight_mom holdout 16.9->14.8%/yr, rec_126 14.7->10.7 on the research panel; factor_panel_check). research/lab/fetch_tiingo.py pulls full EOD histories (free tier: 500 unique symbols/month, 50/hour; paced, resume-safe, key never logged). research/lab/tiingo_merge.py overlays the caches on the research S&P panel at build time (>=50% PIT-coverage gate, median-overlap cross-check, stub-column replacement, rename splices GPS<-GAP, DISCA<-WBD, WLTW<-WTW, ADS<-BFH, CCE<-CCEP, ESV<-VAL, FBHS<-FBIN, HFC<-DINO) and can apply the same to the production store (--production [--apply]; data/us*.csv, .bak-tiingo backups, under the panel lock). load_us() rebuilds when the overlay changes (_tiingo_sig); fetch_us_mid.load_bars falls back to converted CSVs. Round 1 (2026-10-08): 165/209 S&P names and 183 mid/small PIT members restored (2011-19 snapshots were 15-20% short of PIT members). prod_impact A/B on registered replays: overnight_recovery_top10 23.8%/0.83/-38.3% vs 27.8%/0.88/-47.3% backfill-off; lowrisk_top20 6.1%/0.58 vs 8.6%/0.60 - honest long returns with shallower drawdowns; trend_rider_v10 (ETF path) unchanged at 55.0%/1.64. Official precompute refreshed (backup at data/precomputed/backtest_results.json.pretiingo). midsmall_scan on the corrected panel: no factor beats the EW benchmark (0.60 holdout Sharpe vs 0.51-0.52 best) - no mid/small sleeve.
74 lines
2.8 KiB
Python
74 lines
2.8 KiB
Python
"""Pre-vs-post Tiingo overlay factor check for the S&P panel.
|
|
|
|
Builds the panel twice in-process — Yahoo-only (pre, survivorship-biased)
|
|
and with the Tiingo overlay (post) — and runs the sleeve-relevant factors
|
|
through the standard top-N phase-averaged backtest on both. The delta is the
|
|
survivorship effect on the factors behind the live stock sleeves
|
|
(overnight_recovery: overnight_mom + rec_126; lowrisk: ivol + low_beta).
|
|
|
|
PYTHONPATH=. uv run python -m research.factor_panel_check
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import time
|
|
import warnings
|
|
|
|
import pandas as pd
|
|
|
|
import research.lab.panel as pnl
|
|
from research.lab.factor_scan import PERIODS, phase_avg
|
|
from research.lab.factors import FACTORS
|
|
from research.lab.report import forward_returns, ic_series, ic_summary
|
|
|
|
warnings.filterwarnings("ignore")
|
|
|
|
NAMES = ["overnight_mom", "rec_126", "mom_12_1", "low_vol",
|
|
"ivol", "low_beta", "sharpe_mom", "amp_cut_mom"]
|
|
RESULTS = os.path.join("data", "research", "results")
|
|
|
|
|
|
def scan(p, label: str) -> pd.DataFrame:
|
|
fwd5 = forward_returns(p, 5)
|
|
rows = []
|
|
for name in NAMES:
|
|
t0 = time.time()
|
|
s = FACTORS[name](p)
|
|
row = {"panel": label, "factor": name}
|
|
for per, (a, b) in PERIODS["us"].items():
|
|
st = phase_avg(p, s, a, b, 20, 5)
|
|
ic = ic_summary(ic_series(s.loc[a:b], fwd5.loc[a:b], p.member.loc[a:b], 5))
|
|
row.update({f"{per}_cagr": st["cagr"], f"{per}_sharpe": st["sharpe"],
|
|
f"{per}_ic5": ic["ic"], f"{per}_ic5_t": ic["t"]})
|
|
rows.append(row)
|
|
print(f"[{label}] {name:14s} D {row['design_cagr']:6.1%}/{row['design_sharpe']:.2f} "
|
|
f"H {row['holdout_cagr']:6.1%}/{row['holdout_sharpe']:.2f} [{time.time() - t0:.0f}s]",
|
|
flush=True)
|
|
return pd.DataFrame(rows)
|
|
|
|
|
|
def main() -> None:
|
|
post = pnl.load_us()
|
|
orig = pnl.apply_sp500
|
|
pnl.apply_sp500 = lambda raw, **kw: raw
|
|
try:
|
|
pre = pnl._build_us()
|
|
finally:
|
|
pnl.apply_sp500 = orig
|
|
a = scan(pre, "pre")
|
|
b = scan(post, "post")
|
|
df = pd.concat([a, b], ignore_index=True)
|
|
os.makedirs(RESULTS, exist_ok=True)
|
|
df.to_csv(os.path.join(RESULTS, "factor_panel_check.csv"), index=False)
|
|
m = a.merge(b, on="factor", suffixes=("_pre", "_post"))
|
|
print("\nfactor D_cagr pre->post D_sh pre->post H_cagr pre->post H_sh pre->post")
|
|
for _, r in m.iterrows():
|
|
print(f"{r['factor']:14s} {r['design_cagr_pre']:7.1%}->{r['design_cagr_post']:6.1%} "
|
|
f"{r['design_sharpe_pre']:5.2f}->{r['design_sharpe_post']:5.2f} "
|
|
f"{r['holdout_cagr_pre']:7.1%}->{r['holdout_cagr_post']:6.1%} "
|
|
f"{r['holdout_sharpe_pre']:5.2f}->{r['holdout_sharpe_post']:5.2f}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|