Gahow Wang
c34367161d
- orb_stocks_in_play: 5-min bounds -70%/yr (conservative) vs +71%/yr (optimistic); the optimistic bound breaks even at 3.6 bp slippage a side. - overnight_only: the momentum sleeve's names earn their return overnight (Sharpe 1.57) and nothing intraday; held close->open only, net Sharpe 1.29-1.42 at 1 bp a side, below the 1.5 gate. - overnight_paper.py + systemd unit: MOC buy / MOO sell of the top 10 on the Alpaca paper account from 2026-10-12, with a fills-vs-prints report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
285 lines
14 KiB
Python
285 lines
14 KiB
Python
"""
|
|
Hold the momentum sleeve's names only overnight (buy at the close, sell at the
|
|
next open) and stay flat during the day. Intraday trading is allowed since
|
|
2026-10-09, so an overnight-only book is tradable with market-on-close and
|
|
market-on-open orders, and it leaves the capital free for an intraday strategy
|
|
during the session.
|
|
|
|
PYTHONPATH=. uv run python -m research.lab.overnight_only
|
|
|
|
Background: Lou, Polk & Skouras, "A tug of war: Overnight versus intraday
|
|
expected returns" (JFE 2019) find that momentum profits accrue overnight and
|
|
partly reverse intraday. overnight_recovery_top10 ranks on 12-1 month
|
|
overnight returns, so its own return may sit mostly in the overnight leg.
|
|
|
|
Rules (one book per candidate, equal weight, rebuilt every night):
|
|
night t from the close of t-1 to the open of t
|
|
selection top 10 by the candidate score read at the close of t-2 (one day
|
|
of slack so the MOC order can be sent before the close of t-1),
|
|
PIT S&P 500 members on t-2
|
|
return open_t / close_{t-1} - 1 per name (Yahoo adjusted bars, so
|
|
dividends and splits net out); a missing open falls back to
|
|
close_t
|
|
costs every night trades the whole book twice: c per side, baseline
|
|
c = 1 bp (auction fills, $0 commission, SEC fee ~0.3 bp on
|
|
sells), stress c = 3 bp
|
|
cash earns nothing; Sharpe = mean / sd of the book's own returns
|
|
|
|
Candidates:
|
|
sleeve_score overnight_mom (12-1) + rec_126 percentile blend, the live
|
|
sleeve's score
|
|
overnight_12_1 overnight_mom alone
|
|
overnight_1m last month's overnight return (LPS: overnight returns
|
|
persist at short horizons)
|
|
|
|
Reported only: the overnight / intraday split of the sleeve's names held for
|
|
10 days (the live cadence, no hysteresis), and the equal-weight
|
|
member universe held overnight only.
|
|
|
|
Open-price fidelity (N5): Yahoo opens against the 09:30 bar of the Alpaca 5-min
|
|
SIP files on disk at run time (2016+), on all name-days and on the name-days
|
|
the candidates select. If N5 fails the verdict is DATA.
|
|
|
|
Every period here was examined before (the sleeve's score was chosen on this
|
|
panel). 2019+ is after the LPS publication, not independent OOS.
|
|
|
|
The rule is written to prereg.json before the run; the script refuses to
|
|
overwrite results.json.
|
|
|
|
Result (2026-10-09): STOP. No candidate reaches N1 (full-span net Sharpe 1.5).
|
|
N5 passes: on 288 names, 2016+, Yahoo and Alpaca opens give overnight returns
|
|
with corr 0.96 on all name-days and 0.996 on selected ones; the mean
|
|
difference is 0.03-0.3 bp.
|
|
|
|
candidate (net, 1 bp) CAGR Sharpe halves 2019+ 3 bp MaxDD
|
|
sleeve_score 30.4% 1.29 1.34 / 1.24 1.17 0.84 -31%
|
|
overnight_12_1 38.0% 1.42 1.59 / 1.24 1.16 1.02 -31%
|
|
overnight_1m 25.9% 1.08 1.47 / 0.53 0.54 0.66 -33%
|
|
|
|
Reported only:
|
|
- The live sleeve's names held 10 days (gross) earn the whole return
|
|
overnight: overnight leg 13.7 bp/day, Sharpe 1.57 (halves 1.62 / 1.50,
|
|
2019+ 1.43); intraday leg -0.8 bp/day, Sharpe -0.07; close to close
|
|
Sharpe 0.87.
|
|
- Nightly re-selection adds nothing over the 10-day list (13.5 vs 13.7
|
|
bp/day). The equal-weight universe earns 3.5 bp/day overnight (Sharpe 0.73).
|
|
- Costs decide the result: 2 sides a night at 1 bp take 2 bp of the 13.5
|
|
bp/day gross. Breakeven is about 6.7 bp a side.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
|
|
import numpy as np
|
|
import pandas as pd
|
|
|
|
from research.lab.factor_scan import RESULTS, load
|
|
from research.lab.factors import overnight_mom, overnight_mom_1, rec_126
|
|
from research.lab.us_sweep import _blend
|
|
|
|
OUT = os.path.join(RESULTS, "overnight_only")
|
|
BARS = os.path.join("data", "research", "intraday", "alpaca", "5Min")
|
|
SPAN = ("2006-10-02", "2026-09-24")
|
|
HALVES = [("2006-10-02", "2016-12-30"), ("2017-01-03", "2026-09-24")]
|
|
POST_PUB = ("2019-01-01", "2026-09-24")
|
|
TOP_N, HOLD_DAYS = 10, 10
|
|
COST_BASE, COST_STRESS = 0.0001, 0.0003
|
|
|
|
PREREG = {
|
|
"question": "Does holding the momentum sleeve's names only overnight (MOC buy, MOO sell) earn a net "
|
|
"Sharpe high enough to be a stand-alone stream that frees intraday capital?",
|
|
"rules": "top 10 equal weight by score at close t-2, held close t-1 -> open t, rebuilt nightly; "
|
|
"costs 2 x c per night on the whole book, c = 1 bp base / 3 bp stress; no cash interest",
|
|
"candidates": ["sleeve_score", "overnight_12_1", "overnight_1m"],
|
|
"span": SPAN, "halves": HALVES, "post_publication": POST_PUB,
|
|
"gates": {"N1": "full-span net Sharpe (1 bp) >= 1.5",
|
|
"N2": "net Sharpe (1 bp) >= 1.2 in both halves",
|
|
"N3": "net Sharpe (1 bp) >= 1.0 in 2019-01 .. 2026-09",
|
|
"N4": "full-span net Sharpe at 3 bp/side >= 1.0",
|
|
"N5": "open fidelity vs Alpaca 09:30 bar (2016+): corr of overnight returns >= 0.9 on all "
|
|
"name-days and |mean diff| <= 3 bp on selected name-days"},
|
|
"decision": {"CONTINUE": "a candidate passes N1-N4 and N5 passes -> check MOC/MOO auction fills on Alpaca "
|
|
"intraday data and the combination with an intraday stream on the same capital",
|
|
"DATA": "N5 fails -> rebuild overnight returns from Alpaca opens before any verdict",
|
|
"STOP": "otherwise"},
|
|
"multiplicity": "3 candidates; a lone pass by one is weaker evidence than the sleeve score passing",
|
|
"data_status": "all periods previously examined; 2019+ is post-publication, not independent OOS",
|
|
}
|
|
|
|
|
|
def describe(r: pd.Series) -> dict:
|
|
r = r.dropna()
|
|
eq = (1 + r).cumprod()
|
|
yrs = len(r) / 252
|
|
return {"cagr": float(eq.iloc[-1] ** (1 / yrs) - 1), "vol": float(r.std() * np.sqrt(252)),
|
|
"sharpe": float(r.mean() / r.std() * np.sqrt(252)), "mdd": float((eq / eq.cummax() - 1).min()),
|
|
"mean_bp_per_day": float(r.mean() * 1e4)}
|
|
|
|
|
|
def legs(p) -> tuple[pd.DataFrame, pd.DataFrame]:
|
|
"""Per name-day overnight (open_t / close_{t-1}) and intraday (close_t / open_t) returns."""
|
|
c = p.close.ffill()
|
|
o = p.open.where(p.open.notna(), p.close)
|
|
return o / c.shift(1) - 1, p.close / o - 1
|
|
|
|
|
|
def top_mask(score: pd.DataFrame, n: int) -> pd.DataFrame:
|
|
rk = score.rank(axis=1, ascending=False, method="first")
|
|
return rk <= n
|
|
|
|
|
|
def nightly(sel: pd.DataFrame, on: pd.DataFrame) -> pd.Series:
|
|
"""Equal-weight overnight return of the selected names (selection already aligned to night t)."""
|
|
x = on.where(sel)
|
|
return x.mean(axis=1).where(sel.sum(axis=1) > 0)
|
|
|
|
|
|
def held_split(score: pd.DataFrame, on: pd.DataFrame, intra: pd.DataFrame) -> dict:
|
|
"""Sleeve names held 10 days (refresh at close t-1 with score through t-2), daily reset to equal
|
|
weight, averaged over the 10 phases: overnight leg, intraday leg, close-to-close."""
|
|
rows = {"overnight": [], "intraday": [], "close_to_close": []}
|
|
sel_daily = top_mask(score.shift(2), TOP_N)
|
|
idx = np.arange(len(score))
|
|
for off in range(HOLD_DAYS):
|
|
sel = sel_daily.astype(float)
|
|
sel.loc[(idx % HOLD_DAYS) != off] = np.nan
|
|
sel = sel.ffill().fillna(0.0).astype(bool)
|
|
o = on.where(sel).mean(axis=1)
|
|
i = intra.where(sel).mean(axis=1)
|
|
cc = ((1 + on) * (1 + intra) - 1).where(sel).mean(axis=1)
|
|
rows["overnight"].append(o); rows["intraday"].append(i); rows["close_to_close"].append(cc)
|
|
return {k: pd.concat(v, axis=1).mean(axis=1) for k, v in rows.items()}
|
|
|
|
|
|
def alpaca_opens(columns) -> tuple[pd.DataFrame, pd.DataFrame, int]:
|
|
"""Daily 09:30 bar open and last-bar close per symbol from the 5-min files on disk."""
|
|
opens, closes = {}, {}
|
|
files = [f[:-4] for f in os.listdir(BARS) if f.endswith(".pkl")] if os.path.isdir(BARS) else []
|
|
for s in files:
|
|
if s not in columns:
|
|
continue
|
|
df = pd.read_pickle(os.path.join(BARS, f"{s}.pkl"))
|
|
day = df.index.tz_localize(None).normalize()
|
|
first = df.groupby(day).head(1)
|
|
first = first[(first.index.hour == 9) & (first.index.minute == 30)]
|
|
opens[s] = pd.Series(first["o"].values.astype(float), index=first.index.tz_localize(None).normalize())
|
|
closes[s] = df["c"].groupby(day).last().astype(float)
|
|
return pd.DataFrame(opens), pd.DataFrame(closes), len(opens)
|
|
|
|
|
|
def fidelity(p, sels: dict[str, pd.DataFrame]) -> dict:
|
|
ao, ac, n = alpaca_opens(set(p.close.columns))
|
|
if n == 0:
|
|
return {"symbols": 0}
|
|
cols = list(ao.columns)
|
|
dates = p.dates[p.dates >= "2016-01-04"]
|
|
ao, ac = ao.reindex(index=dates, columns=cols), ac.reindex(index=dates, columns=cols)
|
|
# overnight with each source's own prior close (adjustment factors cancel within a source)
|
|
on_y = (p.open / p.close.ffill().shift(1) - 1).reindex(index=dates, columns=cols)
|
|
on_a = ao / ac.shift(1) - 1
|
|
mem = p.member.reindex(index=dates, columns=cols).fillna(False)
|
|
ok = mem & on_y.notna() & on_a.notna()
|
|
y, a = on_y.where(ok), on_a.where(ok)
|
|
out = {"symbols": n, "name_days": int(ok.sum().sum()),
|
|
"corr_all": float(np.corrcoef(y.stack().values, a.stack().values)[0, 1]),
|
|
"mean_diff_all_bp": float((y - a).stack().mean() * 1e4),
|
|
"median_abs_diff_all_bp": float((y - a).abs().stack().median() * 1e4)}
|
|
for name, sel in sels.items():
|
|
s = sel.reindex(index=dates, columns=cols).fillna(False).astype(bool) & ok
|
|
d = (y - a).where(s).stack()
|
|
out[f"{name}_name_days"] = int(s.sum().sum())
|
|
out[f"{name}_mean_diff_bp"] = float(d.mean() * 1e4) if len(d) else float("nan")
|
|
out[f"{name}_corr"] = (float(np.corrcoef(y.where(s).stack().values, a.where(s).stack().values)[0, 1])
|
|
if len(d) > 10 else float("nan"))
|
|
return out
|
|
|
|
|
|
def main() -> None:
|
|
os.makedirs(OUT, exist_ok=True)
|
|
res_path = os.path.join(OUT, "results.json")
|
|
if os.path.exists(res_path):
|
|
raise SystemExit(f"{res_path} exists; this study does not overwrite results")
|
|
with open(os.path.join(OUT, "prereg.json"), "w") as fh:
|
|
json.dump(PREREG, fh, indent=2)
|
|
|
|
p, _ = load("us")
|
|
mem = p.member
|
|
on, intra = legs(p)
|
|
scores = {
|
|
"sleeve_score": _blend(*[f.where(mem).rank(axis=1, pct=True)
|
|
for f in (overnight_mom(p), rec_126(p))]).where(mem),
|
|
"overnight_12_1": overnight_mom(p).where(mem),
|
|
"overnight_1m": overnight_mom_1(p).where(mem),
|
|
}
|
|
a, b = SPAN
|
|
res, daily, sels = {"candidates": {}}, {}, {}
|
|
for name, sc in scores.items():
|
|
sel = top_mask(sc.shift(2), TOP_N)
|
|
sels[name] = sel
|
|
gross = nightly(sel, on).loc[a:b]
|
|
net, stress = gross - 2 * COST_BASE, gross - 2 * COST_STRESS
|
|
clipped = nightly(sel, on.clip(-0.2, 0.2)).loc[a:b] - 2 * COST_BASE
|
|
big = on.where(sel & (on.abs() > 0.10)).sum(axis=1).loc[a:b] / TOP_N
|
|
daily[name] = net
|
|
d = {"gross": describe(gross), "net": describe(net), "stress": describe(stress),
|
|
"halves": [describe(net.loc[x:y]) for x, y in HALVES],
|
|
"post_pub": describe(net.loc[POST_PUB[0]:]),
|
|
"net_clipped_20pct": describe(clipped),
|
|
"share_of_gross_from_moves_over_10pct": float(big.sum() / gross.sum()),
|
|
"yearly": {int(k): float(v) for k, v in
|
|
((1 + net).groupby(net.index.year).prod() - 1).items()}}
|
|
res["candidates"][name] = d
|
|
print(f"{name:15s} gross {d['gross']['mean_bp_per_day']:5.1f}bp/d sh {d['gross']['sharpe']:5.2f} | net "
|
|
f"cagr {d['net']['cagr']*100:5.1f}% vol {d['net']['vol']*100:4.1f}% sh {d['net']['sharpe']:5.2f} "
|
|
f"mdd {d['net']['mdd']*100:6.1f}% | halves {d['halves'][0]['sharpe']:5.2f} {d['halves'][1]['sharpe']:5.2f}"
|
|
f" | 2019+ {d['post_pub']['sharpe']:5.2f} | stress {d['stress']['sharpe']:5.2f} | clip "
|
|
f"{d['net_clipped_20pct']['sharpe']:5.2f} | >10% moves {d['share_of_gross_from_moves_over_10pct']:.2f}",
|
|
flush=True)
|
|
print(" yearly net %:", {k: round(v * 100, 1) for k, v in d["yearly"].items()})
|
|
|
|
uni = on.where(mem.shift(1).fillna(False).astype(bool)).mean(axis=1).loc[a:b]
|
|
res["universe_overnight_gross"] = {"full": describe(uni), "halves": [describe(uni.loc[x:y]) for x, y in HALVES],
|
|
"post_pub": describe(uni.loc[POST_PUB[0]:])}
|
|
u = res["universe_overnight_gross"]
|
|
print(f"EW members overnight gross: {u['full']['mean_bp_per_day']:.1f}bp/d sh {u['full']['sharpe']:.2f} "
|
|
f"halves {u['halves'][0]['sharpe']:.2f} {u['halves'][1]['sharpe']:.2f} 2019+ {u['post_pub']['sharpe']:.2f}")
|
|
|
|
split = held_split(scores["sleeve_score"], on, intra)
|
|
res["sleeve_held_10d_split_gross"] = {}
|
|
for k, r in split.items():
|
|
r = r.loc[a:b]
|
|
res["sleeve_held_10d_split_gross"][k] = {"full": describe(r), "halves": [describe(r.loc[x:y]) for x, y in HALVES],
|
|
"post_pub": describe(r.loc[POST_PUB[0]:])}
|
|
s = res["sleeve_held_10d_split_gross"][k]
|
|
print(f"sleeve held 10d, {k:14s}: {s['full']['mean_bp_per_day']:5.1f}bp/d cagr {s['full']['cagr']*100:5.1f}% "
|
|
f"sh {s['full']['sharpe']:5.2f} halves {s['halves'][0]['sharpe']:5.2f} {s['halves'][1]['sharpe']:5.2f} "
|
|
f"2019+ {s['post_pub']['sharpe']:5.2f}")
|
|
|
|
fid = fidelity(p, sels)
|
|
res["fidelity"] = fid
|
|
print("fidelity:", {k: (round(v, 3) if isinstance(v, float) else v) for k, v in fid.items()})
|
|
|
|
n5 = (fid.get("symbols", 0) > 0 and fid["corr_all"] >= 0.9
|
|
and all(abs(fid[f"{c}_mean_diff_bp"]) <= 3 for c in scores if fid.get(f"{c}_name_days", 0) > 0))
|
|
checks = {}
|
|
for name, d in res["candidates"].items():
|
|
checks[name] = {"N1": d["net"]["sharpe"] >= 1.5, "N2": all(h["sharpe"] >= 1.2 for h in d["halves"]),
|
|
"N3": d["post_pub"]["sharpe"] >= 1.0, "N4": d["stress"]["sharpe"] >= 1.0}
|
|
passing = [n for n, c in checks.items() if all(c.values())]
|
|
decision = "DATA" if not n5 else ("CONTINUE" if passing else "STOP")
|
|
res["verdict"] = {"checks": checks, "N5": n5, "passing": passing, "decision": decision}
|
|
print(f"checks {checks}\nN5 {n5} passing {passing} -> {decision}")
|
|
with open(res_path, "w") as fh:
|
|
json.dump(res, fh, indent=2)
|
|
pd.DataFrame(daily).to_pickle(os.path.join(OUT, "daily.pkl"))
|
|
from research.lab.trials import append_trials
|
|
append_trials(os.path.join(RESULTS, "trials_us.jsonl"),
|
|
[{"name": f"overnight_only:{n}", **res["candidates"][n]["net"]} for n in scores],
|
|
{"study": "overnight_only", "role": "preregistered quick check"})
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|