---
name: Pinnacle has NO measurable edge over Kalshi/B365/Poly — the anchor is a convention, not a superiority
description: Paired log-loss on 1,218 same-snapshot fixtures — Pinnacle vs B365 t=+0.46, vs Kalshi t=-0.07, vs Poly t=-1.06. All null. Test resolves only ~3pp errors, so a small real edge stays invisible. The xlow band premise test looked positive pooled (+12.2%) but that was 80% IN-SAMPLE; split at the freeze date it is IN +19.7% / OOS -18.7%. No positive out-of-sample evidence exists. Run research/book_accuracy_compare.py.
metadata:
  type: project
---

Built 2026-07-25 when the user asked "how much more accurate is Pinnacle
compared to the others? Does it have a real edge?" — the natural follow-up to
[[project_pin_calibration_verified]], which proved Pinnacle is *calibrated* but
never that it is *better* than the books we trade against.

## Headline: no general edge

Paired log loss, **same pre-KO snapshot** for both books, proportional devig for
all:

| vs Pinnacle | n | diff (book − pin) | t | verdict |
|---|---|---|---|---|
| B365 | 1218 | +0.0004 | +0.46 | null |
| Kalshi | 959 | −0.0001 | −0.07 | null |
| Poly | 179 | −0.0027 | −1.06 | null |

Pinnacle is **not** measurably sharper than any of them. Using it as the fair-value
anchor is a defensible *convention* (lowest vig, most liquid, quotes every league),
not a demonstrated superiority. Anyone claiming "Pinnacle is the sharp book" should
be shown this table.

**Do not over-read the null.** A 1pp systematic error on a ~22% outcome costs only
~0.0003 log loss; at these n the test resolves ~2.7–3.2pp errors. So it rules out a
LARGE Pinnacle edge, not a small one — and a small one is all xlow needs.

## The band premise test — and the in-sample trap I fell into

xlow doesn't need Pinnacle better *in general* — only better *when they disagree in
the band*. Selecting on prices alone (outcome-blind), every fixture-outcome an
exchange prices 5–12% below Pinnacle fair.

**Pooled, this looked encouraging: Kalshi n=328, actual 23.17% vs Kalshi's 20.14%,
z=+1.39, implied +12.2% ROI. I reported that to the user as "the first thing in
weeks carrying real information." That was wrong.** The band was CHOSEN on data up
to the 2026-07-05 freeze, and 264 of those 328 rows are from before it. Split:

| book | slice | n | book said | ACTUAL | z vs book | ROI@mid |
|---|---|---|---|---|---|---|
| Kalshi | IN | 264 | 19.97% | 24.24% | +1.77 | **+19.7%** |
| Kalshi | **OOS** | 64 | 20.87% | 18.75% | −0.43 | **−18.7%** |
| B365 | IN | 91 | 23.07% | 28.57% | +1.28 | +27.9% |
| B365 | **OOS** | 23 | 24.46% | 17.39% | −0.81 | −46.3% |
| Poly | OOS | 43 | 19.94% | 20.93% | +0.17 | +0.6% |
| Poly-US | OOS | 16 | 20.30% | 25.00% | +0.47 | +22.5% |

Positive in-sample, at-or-below zero out-of-sample. I initially called this "the
textbook overfitting signature" — **that was an overstatement, and testing it
retracts it.** Is the IN−OOS gap bigger than noise?

| book | IN | OOS | gap | SE | z | |
|---|---|---|---|---|---|---|
| Kalshi | +19.7% | −18.7% | +38.4pp | 26.6pp | **+1.45** | within noise |
| B365 | +27.9% | −46.3% | +74.2pp | 34.6pp | +2.14 | marginal, OOS n=23 |

On Kalshi — the book with by far the most data — the gap is **not** distinguishable
from noise. So the correct reading is that the premise test provides **no evidence
either way**, NOT evidence of overfitting. The pooled +12.2% was invalid; the
negative OOS slices are equally uninformative.

Net as of 2026-07-25: every source except the original backtest is uninformative.
That was already true yesterday. Nothing about xlow's standing actually changed —
one item was wrongly added to the positive column, then wrongly added to the
negative one.

**Lesson worth more than the result: always split any test of the band at
FREEZE_DATE before quoting it.** Any analysis that re-tests the selection rule on
the selection data will look good and mean nothing.

## Two methodology traps this script had to fix (both flipped the headline)

1. **Per-book "last quote" pairing.** Scoring each book at its own final snapshot
   hands an edge to whichever quotes nearest kickoff — the exchanges do, far more
   often than Pinnacle. Must pair on the last snapshot where BOTH quote.
2. **Disagreement defined on the realised outcome.** The first cut selected
   fixtures where books differed on the outcome that *happened* and produced a
   spurious "KALSHI SHARPER t=−3.85". Conditioning on the answer manufactures the
   gap. Use total-variation distance, computed before the result.

Also: an early draft eyeballed the log-loss-to-probability scale as "0.01–0.02 per
1pp" — wrong by ~30x. Compute it via KL, never by intuition.

Re-run: `python research\book_accuracy_compare.py [min_n]`. Worth repeating once
Poly/Poly-US coverage grows (Poly n=179 and Poly-US n=29 are both too thin now).

Related: [[project_pin_calibration_verified]], [[project_xlow_rejection_criteria]],
[[project_clv_spread_finding]].