Gahow Wang
92af4e5ac0
Yahoo purges delisted members: the S&P research panel was missing 209 ever-members (AGN/CELG/ESRX/ATVI/TWTR/XLNX/...), the production store 89, and the S&P 400/600 panels ~180 more. The gap inflated momentum-family factors (overnight_mom holdout 16.9->14.8%/yr, rec_126 14.7->10.7 on the research panel; factor_panel_check). research/lab/fetch_tiingo.py pulls full EOD histories (free tier: 500 unique symbols/month, 50/hour; paced, resume-safe, key never logged). research/lab/tiingo_merge.py overlays the caches on the research S&P panel at build time (>=50% PIT-coverage gate, median-overlap cross-check, stub-column replacement, rename splices GPS<-GAP, DISCA<-WBD, WLTW<-WTW, ADS<-BFH, CCE<-CCEP, ESV<-VAL, FBHS<-FBIN, HFC<-DINO) and can apply the same to the production store (--production [--apply]; data/us*.csv, .bak-tiingo backups, under the panel lock). load_us() rebuilds when the overlay changes (_tiingo_sig); fetch_us_mid.load_bars falls back to converted CSVs. Round 1 (2026-10-08): 165/209 S&P names and 183 mid/small PIT members restored (2011-19 snapshots were 15-20% short of PIT members). prod_impact A/B on registered replays: overnight_recovery_top10 23.8%/0.83/-38.3% vs 27.8%/0.88/-47.3% backfill-off; lowrisk_top20 6.1%/0.58 vs 8.6%/0.60 - honest long returns with shallower drawdowns; trend_rider_v10 (ETF path) unchanged at 55.0%/1.64. Official precompute refreshed (backup at data/precomputed/backtest_results.json.pretiingo). midsmall_scan on the corrected panel: no factor beats the EW benchmark (0.60 holdout Sharpe vs 0.51-0.52 best) - no mid/small sleeve.
51 lines
1.8 KiB
Python
51 lines
1.8 KiB
Python
"""Convert mid/small Tiingo caches, rebuild the PIT panels, report the
|
|
survivorship change (pre-Tiingo backup vs rebuilt member counts per year).
|
|
|
|
PYTHONPATH=. uv run python -m research.midsmall_rebuild
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import pickle
|
|
import shutil
|
|
|
|
import pandas as pd
|
|
|
|
|
|
def main() -> None:
|
|
stage = [l.strip() for l in open("data/research/tiingo_midsmall.txt")][:230]
|
|
cached = [n for n in stage if os.path.exists(f"data/research/tiingo/{n}.csv")]
|
|
print(f"mid/small tickers: {len(stage)} listed, {len(cached)} cached", flush=True)
|
|
|
|
from research.lab.tiingo_merge import write_yahoo_mid
|
|
write_yahoo_mid(cached)
|
|
|
|
for f in ("us_mid_bars.pkl", "us_mid.panel.pkl", "us_small.panel.pkl", "us_midsmall.panel.pkl"):
|
|
p = os.path.join("data", "research", f)
|
|
if os.path.exists(p):
|
|
os.remove(p)
|
|
print(f"removed {p}", flush=True)
|
|
|
|
from research.lab.panel_mid import load_us_mid
|
|
new = load_us_mid("midsmall")
|
|
with open("data/research/us_midsmall.panel.pre_tiingo.pkl", "rb") as fh:
|
|
old = pickle.load(fh)
|
|
|
|
n, o = new.member.sum(axis=1), old.member.sum(axis=1)
|
|
comp = pd.DataFrame({"post": n.resample("YE").last(), "pre": o.resample("YE").last()}).dropna()
|
|
comp["delta"] = comp["post"] - comp["pre"]
|
|
comp.index = comp.index.year
|
|
print("\nPIT members per year (pre -> post):")
|
|
print(comp.to_string())
|
|
from research.lab.fetch_us_mid import load_bars
|
|
restored = [t for t in cached if t in new.close.columns
|
|
and new.member[t].sum() > 0 and load_bars(t) is not None
|
|
and t not in old.close.columns]
|
|
print(f"\nnew PIT-covered names not in old panel: {len(restored)}")
|
|
print(new.close.shape, flush=True)
|
|
print("REBUILD DONE", flush=True)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|