feat(research): Tiingo backfill of delisted names; honest PIT panels
Yahoo purges delisted members: the S&P research panel was missing 209 ever-members (AGN/CELG/ESRX/ATVI/TWTR/XLNX/...), the production store 89, and the S&P 400/600 panels ~180 more. The gap inflated momentum-family factors (overnight_mom holdout 16.9->14.8%/yr, rec_126 14.7->10.7 on the research panel; factor_panel_check). research/lab/fetch_tiingo.py pulls full EOD histories (free tier: 500 unique symbols/month, 50/hour; paced, resume-safe, key never logged). research/lab/tiingo_merge.py overlays the caches on the research S&P panel at build time (>=50% PIT-coverage gate, median-overlap cross-check, stub-column replacement, rename splices GPS<-GAP, DISCA<-WBD, WLTW<-WTW, ADS<-BFH, CCE<-CCEP, ESV<-VAL, FBHS<-FBIN, HFC<-DINO) and can apply the same to the production store (--production [--apply]; data/us*.csv, .bak-tiingo backups, under the panel lock). load_us() rebuilds when the overlay changes (_tiingo_sig); fetch_us_mid.load_bars falls back to converted CSVs. Round 1 (2026-10-08): 165/209 S&P names and 183 mid/small PIT members restored (2011-19 snapshots were 15-20% short of PIT members). prod_impact A/B on registered replays: overnight_recovery_top10 23.8%/0.83/-38.3% vs 27.8%/0.88/-47.3% backfill-off; lowrisk_top20 6.1%/0.58 vs 8.6%/0.60 - honest long returns with shallower drawdowns; trend_rider_v10 (ETF path) unchanged at 55.0%/1.64. Official precompute refreshed (backup at data/precomputed/backtest_results.json.pretiingo). midsmall_scan on the corrected panel: no factor beats the EW benchmark (0.60 holdout Sharpe vs 0.51-0.52 best) - no mid/small sleeve.
This commit is contained in:
@@ -18,6 +18,7 @@ import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from research.lab.engine import Costs, Panel
|
||||
from research.lab.tiingo_merge import apply_sp500
|
||||
|
||||
RAW_DIR = os.path.join("data", "research")
|
||||
US_ETFS = {"SPY", "RSP", "QQQ", "SHY", "BIL", "IEF", "TLT", "GLD",
|
||||
@@ -54,6 +55,7 @@ def _us_membership(dates: pd.DatetimeIndex, tickers: list[str]) -> pd.DataFrame:
|
||||
def _build_us() -> Panel:
|
||||
with open(os.path.join(RAW_DIR, "us_ohlcv.pkl"), "rb") as fh:
|
||||
raw = pickle.load(fh)
|
||||
raw = apply_sp500(raw)
|
||||
close = raw["close"]
|
||||
# keep rows where most listed tickers printed (drops stray holiday rows)
|
||||
alive = close.notna().sum(axis=1)
|
||||
@@ -92,8 +94,31 @@ def _us_sectors(stocks: list[str]) -> pd.Series:
|
||||
return sec.reindex(stocks)
|
||||
|
||||
|
||||
def _tiingo_sig() -> list:
|
||||
d = os.path.join(RAW_DIR, "tiingo")
|
||||
if not os.path.isdir(d):
|
||||
return [0, 0]
|
||||
files = [f for f in os.listdir(d) if f.endswith(".csv")]
|
||||
mtime = max((os.path.getmtime(os.path.join(d, f)) for f in files), default=0)
|
||||
return [len(files), round(mtime, 3)]
|
||||
|
||||
|
||||
def load_us() -> Panel:
|
||||
return _cached("us", _build_us)
|
||||
"""Cached US panel, rebuilt whenever the Tiingo overlay changes."""
|
||||
path = os.path.join(RAW_DIR, "us.panel.pkl")
|
||||
key = os.path.join(RAW_DIR, "us.panel.key")
|
||||
sig = _tiingo_sig()
|
||||
if os.path.exists(path) and os.path.exists(key):
|
||||
with open(key) as fh:
|
||||
if json.load(fh) == sig:
|
||||
with open(path, "rb") as fh:
|
||||
return pickle.load(fh)
|
||||
p = _build_us()
|
||||
with open(path, "wb") as fh:
|
||||
pickle.dump(p, fh)
|
||||
with open(key, "w") as fh:
|
||||
json.dump(sig, fh)
|
||||
return p
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user