feat(research): Tiingo backfill of delisted names; honest PIT panels
Yahoo purges delisted members: the S&P research panel was missing 209 ever-members (AGN/CELG/ESRX/ATVI/TWTR/XLNX/...), the production store 89, and the S&P 400/600 panels ~180 more. The gap inflated momentum-family factors (overnight_mom holdout 16.9->14.8%/yr, rec_126 14.7->10.7 on the research panel; factor_panel_check). research/lab/fetch_tiingo.py pulls full EOD histories (free tier: 500 unique symbols/month, 50/hour; paced, resume-safe, key never logged). research/lab/tiingo_merge.py overlays the caches on the research S&P panel at build time (>=50% PIT-coverage gate, median-overlap cross-check, stub-column replacement, rename splices GPS<-GAP, DISCA<-WBD, WLTW<-WTW, ADS<-BFH, CCE<-CCEP, ESV<-VAL, FBHS<-FBIN, HFC<-DINO) and can apply the same to the production store (--production [--apply]; data/us*.csv, .bak-tiingo backups, under the panel lock). load_us() rebuilds when the overlay changes (_tiingo_sig); fetch_us_mid.load_bars falls back to converted CSVs. Round 1 (2026-10-08): 165/209 S&P names and 183 mid/small PIT members restored (2011-19 snapshots were 15-20% short of PIT members). prod_impact A/B on registered replays: overnight_recovery_top10 23.8%/0.83/-38.3% vs 27.8%/0.88/-47.3% backfill-off; lowrisk_top20 6.1%/0.58 vs 8.6%/0.60 - honest long returns with shallower drawdowns; trend_rider_v10 (ETF path) unchanged at 55.0%/1.64. Official precompute refreshed (backup at data/precomputed/backtest_results.json.pretiingo). midsmall_scan on the corrected panel: no factor beats the EW benchmark (0.60 holdout Sharpe vs 0.51-0.52 best) - no mid/small sleeve.
This commit is contained in:
@@ -56,6 +56,7 @@ import pandas as pd
|
||||
ROOT = os.path.join("data", "research", "ext")
|
||||
HOLD_DIR = os.path.join(ROOT, "ishares")
|
||||
PX_DIR = os.path.join(ROOT, "yahoo_mid")
|
||||
TIINGO_DIR = os.path.join(ROOT, "tiingo_mid")
|
||||
HOLDINGS = os.path.join(HOLD_DIR, "holdings.csv")
|
||||
FUNDS = {"IJH": 239763, "IJR": 239774} # S&P MidCap 400, S&P SmallCap 600
|
||||
ETFS = ["IJH", "IJR", "MDY", "SLY", "IWM", "SPY", "QQQ", "BIL", "SHY"]
|
||||
@@ -394,10 +395,14 @@ def fetch_prices(start: str = "2005-01-01", retry_missing: bool = False) -> None
|
||||
|
||||
|
||||
def load_bars(t: str) -> pd.DataFrame | None:
|
||||
"""Adjusted OHLCV (as yfinance's auto_adjust=True) plus the unadjusted close."""
|
||||
"""Adjusted OHLCV (as yfinance's auto_adjust=True) plus the unadjusted
|
||||
close. Falls back to a Tiingo-converted CSV (see lab.tiingo_merge)."""
|
||||
path = _px_path(t)
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
alt = os.path.join(TIINGO_DIR, os.path.basename(path))
|
||||
if not os.path.exists(alt):
|
||||
return None
|
||||
path = alt
|
||||
h = pd.read_csv(path, index_col=0, parse_dates=True)
|
||||
h = h[~h.index.duplicated(keep="last")].sort_index()
|
||||
ratio = (h["Adj Close"] / h["Close"]).where(h["Close"] > 0)
|
||||
|
||||
Reference in New Issue
Block a user