---
name: project-xlow-backtest-league-slice
description: 2026-08-23 DECISION — the forward test's PRIMARY scope is now the 20 leagues the backtest was measured on (n=237, +18.75%); leagues added later are tracked separately and never pooled. Also the trap that live_data has NO Poly before July.
metadata:
  type: project
---

## The decision (user, 2026-08-23)

**Primary forward scope = the leagues the backtest ran on.** Leagues added since
are still scored, separately, as data collection — never pooled into the
headline.

## Why (and a correction to my first read)

I initially called the restricted slice a post-hoc pick and argued against it.
**That was wrong, and the user corrected it:** the league set came from the
backtest, i.e. it was fixed BEFORE the forward test started. The extra leagues
were switched on deliberately, to keep collecting while most of the backtest set
sat out of season. So restricting is the *pre-registered* universe and the added
leagues are the addition — pooling them is what silently redefines the rule's
universe mid-test.

The seasonality is real and checkable: post-freeze live_data fixtures are
**D1 0**, E0 9, F1 9, I1 8, I2 10, G1 6 — against MLS 97, BSA 61. A forward test
that waited for the backtest leagues would have stood still all summer.

**Lesson for me:** "was this slice chosen before or after seeing the data" is a
question about INTENT and cannot be inferred from the numbers. Ask before
calling something post-hoc.

## Numbers

Backtest (poly_backfill, 2026-04-18..06-17, 975 fixtures), reproduced exactly:
n=286, WR 29.7% vs 23.4%, ROI **+24.71%**, z+2.59. **20 leagues fired.**

| forward slice | n | WR | implied | ROI/bet | z |
|---|---|---|---|---|---|
| **PRIMARY · backtest lgs** | 237 | 27.4% | 23.7% | **+18.75%** | +1.37 |
| all leagues, pooled | 442 | 25.1% | 23.8% | +2.46% | +0.65 |
| added lgs, collecting | 205 | 22.4% | 24.0% | −16.37% | −0.53 |

The scoped forward (+18.75%) does track the backtest (+24.7%) far more closely
than the pooled number did.

## SCOPE EDIT 2026-08-23 (user) — and the caveat it carries

**Removed BSA, CL1** ("I don't trust them"). **Added NWSL, PL1, USL.**
Scope is now 21 leagues, in `F.PRIMARY_LEAGUES`; `F.BACKTEST_LEAGUES` keeps the
original 20 verbatim as the audit trail.

| scope | n | ROI/bet | z |
|---|---|---|---|
| backtest 20 | 237 | +18.75% | +1.37 |
| **edited scope** | **212** | **+34.46%** | **+2.55** |
| removals alone, no adds | 189 | +36.14% | +2.32 |

**This one IS post-hoc.** BSA (−40.2%, n=35) and CL1 (−75.2%, n=13) were the two
worst leagues in the scope and were removed after that was visible. Nearly the
whole jump is the removals, and it carries z across 1.96 — that significance is
manufactured by the choice, not earned by the rule. `SCOPE_FROZEN = "2026-08-23"`
marks it: everything before that date is IN-SAMPLE for this scope, the same
discipline [[project-xhigh]]'s XHIGH_FREEZE exists for. Only bets struck after
2026-08-23 test it.

Note the contrast with the original scoping, which was legitimate precisely
because the league set predated the forward test. Same shape of edit, opposite
provenance — which is the whole lesson.

**PL1/USL REMOVED FROM THE SCOPE 2026-08-27 (user).** Scope is 19 leagues.
They were only ever in it through the Poly-US substitution below -- a confirming
book the backtest never used -- and that slice ran flat (+0.8% on n=71) against
+34.5% for the Poly.com-confirmed legs. They keep scoring in `xlow_newlg`
(outside scope) and in the standalone `xlow_polyus` row, so nothing stops being
measured. Current primary: **n=238, +28.02%, z+2.02**.

The paragraph below records why they were added on 08-23 and is kept as the
reasoning that was overturned:

**PL1/USL were IN the primary, via a second confirming book** (user, 2026-08-23).
They have `poly_tag=None` — no Polymarket.com at all — so `fire_xlow` could
never fire there and they were invisible to the forward test while being
actively traded. `fire_xlow_primary()` now substitutes the **Poly-US** confirm
for GAP_LEAGUES (USL, PL1) and uses Poly.com everywhere else. The two paths are
mutually exclusive by league, so nothing is double counted.

Primary is now **n=283, +26.02%, z+2.31**, which decomposes as:

| confirming book | n | ROI | z |
|---|---|---|---|
| Poly.com | 212 | +34.46% | +2.55 |
| Poly-US (USL/PL1) | 71 | +0.81% | +0.26 |

HONEST NOTE: the n=286 backtest confirmed against **Poly.com only**, so Poly-US
as a confirming book is UNVALIDATED — thinner, newer venue. The dashboard keeps
an "of which poly-US confirm" row so the mix can always be decomposed. Note the
Poly-US slice is running flat (+0.8%) while the Poly.com slice carries the
headline.

BSA and CL1 did not vanish: they now fall into `xlow (outside scope, tracking)`,
so they stay measured.

## Honest caveats — these still hold, and are about POWER, not selection

Keep these in view when judging the rule; none of them argue against the scope.

- The backtest-vs-added gap is **not significant**: +35.13pp/bet, Welch t=+1.95,
  permutation p=**0.057** (20k shuffles). +18.75% alone is z+1.37.
- Requiring a minimum backtest presence is **non-monotonic**: min≥1 +18.75%,
  min≥5 +32.53%, min≥10 +45.12%, **min≥20 +13.99% (z+0.02)**.
- Per-league ROI **anti-correlates** backtest→forward, Spearman **−0.370**
  (k=10, ≥5 bets each side). So this is a SCOPE, not a league-quality signal —
  do not go further and filter *within* it. That reproduces
  [[project-league-filter-study]].
- AR1 is the scoped set's biggest contributor (37 bets, +50.9%) off a backtest
  n=1.

The n=1000 decision point in [[project-xlow-rejection-criteria]] is unchanged.

## TRAP: the backtest is NOT in live_data

`live_data` carries **no Polymarket columns before 2026-07** (PRE-row poly fill:
Apr 0.0%, May 0.0%, Jun 0.0%, Jul 74.1%, Aug 63.5%). `fire_xlow` is
shared-price-only, so running it over the pre-FREEZE_DATE slice of live_data
fires **7 times in 3 leagues** — that is the poly rollout, not a backtest. The
real backtest is `poly_backfill/` via `research/kalshi_only_execution.py`.

## Where it lives

- `research/forward_test_unique_book.py` — `BACKTEST_LEAGUES` (frozen, 20,
  hardcoded so it cannot drift with a directory), `in_backtest_leagues()`,
  `classify_xlow_bt()`, `classify_xlow_newlg()`. `log_snapshot` now writes
  `xlow_bt_*` and `xlow_newlg_*`; `xlow_f10` follows the primary's scope.
- `main/dashboard.py` — primary key is `'xlow (PRIMARY · backtest lgs)'`; KPI
  tiles, paper tile, equity curve and daily email all read it. Pooled and added
  rows stay visible. The last-24h panel is deliberately UNSCOPED and says so.
- `main/weekly_strategy_metrics.py` — same scope, so page and email agree.
- `research/xlow_backtest_leagues.py` — the analysis, re-runnable.