---
name: project-mm-fill-prediction-ml
description: 2026-08-22 ML on live maker data — WHOLE-BOOK volume predicts completion (AUC 0.756), min-leg volume does NOT (the paper-era gate metric fails to replicate), and queue depth barely matters. Signal is simple; more model capacity hurts. Re-run scheduled 2026-08-29.
metadata:
  type: project
---

Run on `research/mm_fill_dataset.csv` (Kalshi live maker, 2026-08-18 onward) via
`research/mm_fill_ml.py`. GroupKFold by EVENT throughout — the three legs of a
book share a market and a kickoff, so random k-fold leaks siblings across folds.

## Results

**LEG FILL** — does a resting order get lifted? n=1,467 legs, 174 fills (11.9%)

| model | AUC |
|---|---|
| baseline (no features) | 0.407 |
| **logistic** | **0.673** [0.630, 0.715] |
| GBM depth1 | 0.636 |
| GBM depth3 | 0.633 |

Top permutation importance: `log_volume_24h` +0.054, `above_touch_c` +0.034,
`is_favourite` +0.029, `collected` +0.028, `log_book_volume` +0.024.

**BOOK COMPLETION** — do all three legs sell? n=193 books, 45 complete (23.3%)

| model | AUC |
|---|---|
| baseline | 0.464 |
| **GBM depth1** | **0.756** [0.675, 0.832] |
| GBM depth3 | 0.707 |

Top: `log_book_volume` **+0.091**, `gross_c` +0.045, `hrs_to_ko` +0.026,
`log_min_queue` +0.007, **`min_leg_volume` -0.001**.

## THE CORRECTION: book volume, not min-leg volume

[[project-mm-completion-arithmetic]]'s gate sweep (46.8% -> 82.6% by MIN leg
volume) was measured on PAPER books. It does NOT replicate on live ones:

| metric | Q1 -> Q4 completion | spearman | p |
|---|---|---|---|
| min leg volume | 20.8% -> 25.0% (non-monotonic) | +0.095 | **0.189** |
| sum of leg volumes | 12.2% -> 35.4% | +0.189 | 0.009 |
| **whole-book volume** | **10.2% -> 35.4%** | **+0.215** | **0.003** |

`mm_maker_live.books_in_window` was changed twice on 2026-08-22 because of this:
first to rank on min-leg volume (wrong, from the memo), then corrected to
whole-book volume. **If a re-run stops showing book volume on top, that ranking
is resting on nothing and should revert to kickoff order.**

## Queue depth is not the problem it was thought to be

[[project-mm-queue-reality-check]] called queue position "our central unsolved
problem". Here `log_min_queue` scores +0.007 and `log_max_queue` +0.006 —
essentially nothing, against book volume's +0.091. Not a refutation of the
queue-aware FILL MODEL, but as a book-SELECTION feature it is dominated.

## The signal is simple

depth3 < depth1 in BOTH models (0.633 vs 0.636; 0.707 vs 0.756). That is the
signature [[project-nonprice-features-dead]] flags as null-like: when capacity
hurts, believe the simple model. Two or three features, roughly linear. Do not
build anything larger on this.

## Traps handled (each one has bitten this project before)

- `minutes_exposed` / `n_posts` DROPPED as leaky — an episode ends when it
  fills, so a filled leg is short by construction and those features just
  restate the label. They look like the strongest predictors in the file.
- `side` collapsed — it is 127 team abbreviations. Replaced with is_tie /
  is_favourite / is_longshot.
- A no-feature baseline is reported alongside every model.

## Honest limits

193 books over ~5 days, one venue, one regime, 45 completions. AUC CIs exclude
0.5 so the signal is real, but two live selection rules now depend on a sample
this size. Re-run scheduled: task **SoccerMMFillMLRerun**, one-off
2026-08-29 09:00, runs `research/mm_fill_ml_rerun.py`, which rebuilds the
dataset, re-runs the model, emails the output and prints the baseline above for
comparison.