a C7ej44@s4dZddlZddlZddlZddlZddlmZdZeej dkrPe ej dndZ dZ dZ d Zd Zd d gfd d gfddgfdddgfddgfdZgdZddddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Z d2d3Z!e"d4kr0e dS)5ax Which book is actually SHARPEST? (read-only, trades nothing) THE QUESTION: pin_calibration_by_league.py proved Pinnacle is *calibrated* -- implied 25.0% draws really do land 25.5% of the time. That is NOT the same as proving Pinnacle is *better than the exchanges*. Kalshi and Poly could be equally calibrated, in which case CLV-vs-Pinnacle measures noise, not information, and xlow's premise ("when the exchange is cheaper than Pinnacle, the exchange is wrong") is unfounded. Calibration is marginal (do 20% shots win 20% of the time). Sharpness is conditional (does the book separate the 15% games from the 25% games). A book quoting the league base rate every time is perfectly calibrated and useless. So we score with PROPER SCORING RULES, which reward both at once. METHOD -- two controls that the first cut of this script got wrong, both of which flipped the headline: 1. SAME-SNAPSHOT PAIRING. Every pairwise comparison uses the last pre-KO row where BOTH books have a complete quote. Scoring each book at its own last quote silently hands an information edge to whichever book happens to quote nearest kickoff -- and the exchanges quote later than Pinnacle far more often than the reverse. 2. OUTCOME-INDEPENDENT DISAGREEMENT. Disagreement is total-variation distance over all three outcomes, computed before the result is known. Selecting fixtures where the books differed *on the outcome that happened* is conditioning on the answer and manufactures a gap out of nothing. - Every book devigged identically (proportional normalisation to sum 1) so no book is advantaged by the devig choice. Pinnacle/B365 quote decimal odds (raw = 1/odds); Kalshi/Poly/Poly-US quote probabilities (raw = mid). - Log loss (primary; strictly proper, punishes confident errors) and Brier (secondary; bounded, less tail-sensitive), over all 3 outcomes. - Paired t-stat on the per-fixture log-loss difference. Fixture difficulty varies enormously and cancels in the difference, so paired is far more powerful than comparing the two means. READING THE RESULT Lower log loss = sharper. The paired t is the honest test; a raw log-loss gap without it is meaningless at these samples. If Pinnacle does NOT beat the exchanges, xlow's premise is unsupported -- a far bigger finding than any bet-level ROI. Usage: python research\book_accuracy_compare.py [min_n] N) defaultdict)z*E:\soccer\poly_backfill\*_polybackfill.csvzE:\soccer\live_data\*.csvgQ?)g?gQ?z 2026-07-05gư>odds)Zpin_homeZpin_drawZpin_away)Z b365_homeZ b365_drawZ b365_awayZprob)Zkalshi_home_midZkalshi_draw_midZkalshi_away_mid)Z poly_home_midZ poly_draw_midZ poly_away_mid)Zpoly_home_priceZpoly_draw_priceZpoly_away_price)Zpolyus_home_midZpolyus_draw_midZpolyus_away_mid)pinnacleZb365ZkalshiZpolyZpolyusHDAc Cs(z t|WSttfy"YdS0dSN)float TypeError ValueError)xr+E:\soccer\research\book_accuracy_compare.py_fNs rcCs4|D]*}|dpd}|dvr|SqdS)Nftrr)getstripupper)rowsrvrrr_resultUs  rcs|dkr.tdd|DrdSdd|D}ntdd|DrDdSt|}t|dkrhd ksnndSfd d|DS) zFProportional devig -> probabilities summing to 1, or None if unusable.rcss|]}|dup|dkVqdS)N?r.0rrrr `zdevig..NcSsg|] }d|qS)rrrrrr br!zdevig..css0|](}|dup&d|ko dkn VqdS)Nrrrrrrr dr!g?g?csg|] }|qSrrrsrrr"jr!)anylistsum)rawkindimprr$rdevig]sr,c sgt}}tD]}t|D]}tt}zpt|dddN}t|D]2|ddddf qJWdn1s0YWnt yYq Yn0| D]\}}||vs|d sqt |}|t vrqd d |D} | sqg} | D]ji} t D]F\} \} }|D]2}tfd d |D| }|r(|| | <qq(q| r| | q| s|q||| t|d |d || dqq q|S)zJOne record per fixture: {ftr, snaps:[{book: probs}, ...]} in pre-KO order.rzutf-8)newlineencodingZ game_dateZleaguehomeZawayNrcSs&g|]}|dpddkr|qS)ZstatusrZPRE)rr)rrrrrr"r!zload..csg|]}t|qSr)rr)rcrrrr"r!r)gdZlgrsnaps)setSOURCESglobrr'opencsvZ DictReaderrappend ExceptionitemsrIDXBOOKSr,adddict)outseenZpatpathfxZfhkeyrrZprer3dbookr*ZcolsetsZcolsprr1rloadmsN  (   $rHcCs*t|dD]}||vr ||Sq dS)Nr3reversed)rCrFrErrr last_solosrKcCs:t|dD](}||vr ||vr ||||fSq dS)zILast pre-KO snapshot where BOTH books quote -- the fair comparison point.r3NrI)rCabrErrr last_bothsrNcCstt|t|t Sr )mathlogmaxr<EPSrGrrrrloglosssrTcstfddtdDS)Nc3s.|]&}||tkrdnddVqdS)rr#rN)r<rirrGrrr r!zbrier..r(rangerSrrWrbriersr[cs dtfddtdDS)z=Total-variation distance -- outcome-independent disagreement.g?c3s"|]}t||VqdSr )absrUrGqrrr r!ztvd..rXrYr]rr]rtvdsr_cCs|rt|t|StdS)Nnan)r(lenr )Zxsrrrmeansrbcsdt|}|dkrtdSt|tfdd|D|d}t||}|dkr\|StdS)NrXr`c3s|]}|dVqdSrNr)rrEmrrr r!zpaired_t..rr)rar rbr(rOsqrt)diffsnvarZserrdrpaired_tsrjcCs8t|rdSt|dkrdS|dkr*dS|dS)Nzn/ag\(\?zno detectable differencerzPINNACLE SHARPERz SHARPER)rOZisnanr\r)trFrrr verdict_fors   rlcCstd|td|tddddddddd dd d dd d dd dddddtD]8}|dkrrqbgg}}|D]R}t|d|}|sq|\} } || | sq|t| |d|t| |dqt|} | tkrtd|dd| ddtdqbddt||D} t| } t ddt||D}td|dd| ddt |ddt |ddt | dd| dd|| ddd t | |qbdS)!N  rF9s rh>5sZpin_LL>8sZbook_LLZdiff>9srkz>7sZ pin_closer>10sz verdictrr5d (below min_n=)cSsg|]\}}||qSrrrryrrrr"r!z report_pairs..css|]\}}||krdVqdSrNrrxrrrr r!zreport_pairs..8.4fz+9.4fz+7.2fdz9.0fz% ) printORDERrNr9rTraMIN_Nziprjr(rbrl)fixturesZ subset_fntitleZnoterMZplZblrCpairppbprhrgrkZwinsrrr report_pairssD,       8 rc szt}tdtdt|dtdtddddd dd d dd d tD]fdd|D}t|tkrtddt|ddtdqTtddt|ddtfdd|Dddtfdd|DdqTtddddddtddddddt|dddd t|d!dd"t d#d$d%t |td&td'td(td)td*dS)+NzEBook accuracy comparison (proportional devig; SAME-SNAPSHOT pairing)z/fixtures with a result and >=1 complete quote: rmzSTANDALONE (each book on every fixture it quotes -- samples DIFFER and quote times differ, so NOT comparable across rows; coverage only)rFZ10srprhrqrTrsr[rrcsg|]}t|r|qSr)rKrfrMrrr"r!zmain..rurvrwcs g|]}tt||dqSr)rTrKrrrrr"r!z9.4fcs g|]}tt||dqSr)r[rKrrrrr"r!r{z1/3 eachrZ5srXgUUUUUU?z uninformative baselinecSsdS)NTrrrrrrr!zmain..z)PAIRED vs PINNACLE -- ALL shared fixtureszodiff = book_LL - pinnacle_LL; POSITIVE => Pinnacle sharper. pin_closer = % of fixtures Pinnacle scored better.cSst||tkSr )r_ DISAGREE_MINrrrrrr!z0PAIRED vs PINNACLE -- DISAGREEMENT ONLY (TVD >= .0%z, computed WITHOUT the result)zPxlow only fires when the books disagree, so this is the decision-relevant slice.zW SCALE (do not eyeball this -- an early draft of this script guessed it wrong by ~30x):zG a 1pp systematic error on a ~22% outcome costs only ~0.0003 log loss.z[ So the paired test above can only resolve errors of roughly 3pp; a book that is genuinelyzY 1pp sharper than another is INVISIBLE at these sample sizes. "No detectable difference"z/ means "no LARGE difference", not "identical".) rHr}rar~rrbrOrPrr band_premise)rr%rrrmains@& " . rcCstdtdtdddtdddtdtd td td d d ddddddddddddddddddd ddtD]}|dkrqdddfdd dffD]\}}g}|D]}||d!pd"sqt|d|}|sq|\}}td#D]} || dkrq|| || || } td| krNtdkrnn.||| || t|d$| krxdndfqqt|} | d%krtd |d d|dd| d&d'qd(d)|D} d*d)|D} t d+d,|D}t d-d,| D}|dkr|t | t |nt d.}t d/d,|D| }td |d d|dd| d&dt | d0d1d2t | d0d3d2|| d0d4d2|d5d|d0d6d7qqtd8td9d:S);a?THE test of xlow's premise, on all fixtures rather than just our bets. xlow bets an outcome when an exchange prices it 5-12% BELOW Pinnacle fair (relative). The premise is that Pinnacle is right, so the outcome should occur near PINNACLE's rate, not the exchange's. Selection is PRICES ONLY -- never the result -- and uses every qualifying fixture-outcome, which is several times the bet-level sample. Caveat: a fixture can contribute more than one outcome, so rows are not fully independent; in practice multi-qualifying fixtures are rare. z; XLOW PREMISE TEST -- outcome-blind selection, all fixturesz exchange prices outcome i rr-rz6 BELOW Pinnacle fair. Whose number does reality match?z SPLIT BY FREEZE_DATE z4: the band was CHOSEN on the in-sample rows, so onlyzg the OOS rows are evidence. Reporting pooled numbers here was a real mistake once -- do not repeat it.rnrFrorpsliceZ6srhrqZpin_saysrsZ book_saysrtZactualrrZ z_vs_bookzROI@midrINcSs|tkSr  FREEZE_DATEgrrrr'r!zband_premise..ZOOScSs|tkSr rrrrrr(r!r2rrXr ruz (too few)cSsg|] }|dqS)rrrr%rrrr";r!z band_premise..cSsg|] }|dqS)rrrrrrr"<r!css|]}|dVqdSrcrrrrrr =r!zband_premise..css|]}|d|VqdSrzr)rrGrrrr >r!r`css(|] \}}}|rd|dndVqdS)rgNr)r_Zpxwrrrr @r!r|z8.2fz% z9.2fz7.2fz+10.2fz+7.1f%zY Positive z_vs_book (actual ABOVE the exchange price) = exchange too cheap = xlow works.zX ROI is mid-to-mid and ignores the spread; real fills are at the ask, so live is worse.N)r}BANDrr~rNrZr9r<rar(rOrfr rb)rrFZlabelZkeepZselrCrrrrVZrelrhZpin_pZbk_phitsrizZroirrrrsh ",     $4 "(,  r__main__)#__doc__sysr8r6rO collectionsrr5raargvintrrrrrRr=r~r<rrr,rHrKrNrTr[r_rbrjrlrrr__name__rrrrsL-    ( +;