fix(data): judge late newest rows as sparse; require SPY print only in the ETF panel
Tickers count as alive for 10 rows after their last bar, so a newest row printed by a few early tickers is dropped until the rest arrive. The benchmark-print session rule now applies only to the mixed US/HK ETF panel: Yahoo's 510300.SS lags CN stocks by days and would stall the CN book. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -34,6 +34,9 @@ REFRESH_DAYS = 30
|
||||
# A row is kept only if at least this fraction of tickers alive on that date
|
||||
# have a bar. Drops e.g. US-holiday rows created by HK-listed tickers.
|
||||
MIN_ROW_COVERAGE = 0.5
|
||||
# A ticker counts as alive for this many rows after its last bar, so a newest
|
||||
# row printed by only a few tickers (late data) is still judged sparse.
|
||||
ALIVE_GRACE_ROWS = 10
|
||||
# Relative change in the adjusted-price overlap that triggers a rescale.
|
||||
RESCALE_TOLERANCE = 1e-3
|
||||
UNAVAILABLE_RETRY_DAYS = 7
|
||||
@@ -203,7 +206,7 @@ def _drop_sparse_rows(df: pd.DataFrame) -> pd.DataFrame:
|
||||
first = np.where(has_any, notna.argmax(axis=0), n)
|
||||
last = np.where(has_any, n - 1 - notna[::-1].argmax(axis=0), -1)
|
||||
pos = np.arange(n)[:, None]
|
||||
alive = (pos >= first) & (pos <= last)
|
||||
alive = (pos >= first) & (pos <= last + ALIVE_GRACE_ROWS)
|
||||
coverage = notna.sum(axis=1) / np.maximum(alive.sum(axis=1), 1)
|
||||
keep = coverage >= MIN_ROW_COVERAGE
|
||||
if not keep.all():
|
||||
|
||||
Reference in New Issue
Block a user