a 5j @s\dZddlZddlZddlZddlZddlmZmZmZddlm Z dZ dZ dZ dZ gd Zd d Zd d ZddZdZdZdZddZddZddZddZddZddddZdd d!Zd"d#Zd$d%ZzPddlZddlZ ej!"de j!#e j!$e j!$e j!%e&d&dd'l'm(Z(Wne)y8d(d)hZ(Yn0d*d+Z*d,d-Z+d.d/Z,e-hd0Z.d1Z/d2Z0e-e.d3d4hd5d6hBZ1d7d8Z2e2Z3d9d:Z4d;d<Z5d=Z6d>d?Z7d@Z8dAZ9dBZ:e9e:fdCdDZ;e9e:fdEdFZdK\Z?Z@dLdMZAdNdOZBdPdQZCdRdSZDdTdUZEdVZFdWZGdXdYZHdZd[ZId\d]ZJd^d_ZKded`daZLdbd&ZMeNdckrXeMdS)fun FORWARD TEST (read-only, trades nothing): the cross-book "exclusive-low" rule (PRIMARY) plus the older "unique vs overlap" baseline, on out-of-sample fixtures. PRIMARY RULE — "exclusive low", STRICT (user's formulation; spec corrected 2026-07-05 — see fire_xlow docstring for the correction history): bet the CHEAPER book's side when (1) its mid is BELOW the other book, (2) its pin-CLV is in [0.05,0.12), AND (3) the other book is NOT in band — one book in band while the other sits near Pinnacle, confirming the fair. Backtest on poly_backfill (975 shared-price fixtures): plain Kalshi pin-CLV [5,12) n=421 +11.0% z+1.26 p=0.105 (current live) exclusive-low STRICT [5,12) n=286 +24.7% z+2.59 p=0.005 -> OOS TAIL (last 40%) +43.7% z+2.81 (Under below-other, "other not in band" == "other CLV < 5%" — the cheaper book always carries the higher CLV, so the formulations coincide; verified bet-for- bet.) Still pin-anchored (pure two-book deviation with no Pinnacle is dead — see project_pure_xbook_deviation). SECONDARY — "unique vs overlap" (2026-06-28): bets where exactly ONE book fires [0.05,0.12) ("unique") beat bets where BOTH fire ("overlap"): full UNIQUE n=117 +61.9% z+3.52 vs OVERLAP n=267 +10.8% z+1.51; held across OOS splits. The exclusive- low rule is the cleaner, stronger productionizable form of the same edge — kept here for continuity. This FREEZES the rules as of FREEZE_DATE and tracks them on LATER live_data fixtures for clean OOS. NOTHING traded. Selection (NON-look-ahead, mirrors live): per book, first pre-KO snapshot whose mid-CLV (multiplicative pin_fair) qualifies. Kalshi settles @ ask; Poly @ real poly_*_ask when present (live_data) else est-ask (mid + near-term half-spread, for backfill). One bet/fixture (max CLV). Append one row/run. BLOCKER (honest): needs live_data rows carrying BOTH kalshi_* AND poly_* prices. The live Poly feed has been dark since 2026-06-18 (club off-season) and live_monitor must be RESTARTED with Poly wired for poly_* to populate. Until then OOS shared-price fixtures = 0 and this logs n=0 -- expected, not a bug. N)datetimedate timedelta) defaultdictzE:\soccer\live_dataz3E:\soccer\research\forward_test_unique_book_log.csvz 2026-07-05)皙?gQ?)z%Y-%m-%d %H:%M:%Sz%m/%d/%Y %H:%Mz%m/%d/%Y %H:%M:%Sz%Y-%m-%d %H:%Mc Cs8tD].}zt||WSttfy0Yq0qdSN)_FMTSrstrptime ValueError TypeError)sfr$research/forward_test_unique_book.py_pts0src Cs(z t|WSttfy"YdS0dSr)floatr r xrrr_f5s rcCs8t|D]*}|dpd}|dvr|SqdS)Nftr)HDA)reversedgetupperstrip)rowsrvrrr_result8s r!)kalshipolyuspolyg ףp= ?gQ?cst|D]dpddkr$qtD]|fdddD}tdd|DrTq(d dt|D}d dt|D}t|d kr(t|d kr(d |dSq(qdS)aResult inferred from a settled market, or None. DISPLAY ONLY. `ftr` is backfilled from API-Football, so a match that finishes at 15:00 has no result until that backfill runs -- and when the API quota is spent, not until the next day. The dashboard then shows finished matches as "open" for hours, which is what it looked like on 2026-08-15. The result is however already sitting in the same row: once a market settles the winning leg goes to ~1.00 and the others to ~0.00, which is exactly how resolve_bets derives its "D via kalshi". Requires status FT AND an unambiguous split, so an in-play blowout at 85' cannot be mistaken for a settled book. Deliberately NOT used by the scoring paths: results that a strategy is graded against must come from the match record, never from the prices that same strategy trades. This exists so a human can see a finished game as finished, and `load_fixtures` tags it ftr_src='market' so it is never mistaken for the real thing. statusrZFTc s&g|]}td|dqS)__mid)rr).0r bookrrr ]z'_result_from_market..)homeZdrawawaycss|]}|duVqdSrr)r(r rrr ^r,z&_result_from_market..cSsg|]\}}|tkr|qSr) _SETTLE_HIr(ir rrrr+`r,cSsg|]\}}|tkr|qSr) _SETTLE_LOr1rrrr+ar,ZHDArN)rrr _SETTLE_BOOKSany enumeratelen)rZlegshilorr)r_result_from_marketEs r<cs(dd|D}t|fdd|DS)NcSsg|] }d|qS)g?r)r(rrrrr+fr,zfair_mult..csg|] }|qSrr)r(Zrir rrr+fr,)sum)orrr=r fair_multesr@cCs(|dkr dS|dkrdS|dkr$dSdS)Nrg~jth?g?g{Gzt?gffffff?g?g{Gz?r)midrrr half_spreadgs   rBcCs|dkr\t|d|d}t|d|d}||rVd|krNdkrVnn|n|fS|dvrrtd|t||d |d}|d urt||d |d }t||d |d}|rd|krdksn|d ur|t|nd }||fS) u (mid, ask) for a book/side. Works on live_data (poly_*_mid/ask) AND backfill (poly_*_price, no ask -> est-ask). Kalshi: mid + real ask. 2026-07-27: `book` is now actually honoured for the non-Kalshi branch. It previously fell through to poly_* for ANY value, so book_quote(r,'polyus',..) silently returned Polymarket.COM prices — a wrong-but-plausible read of exactly the kind that has bitten this project before. Prefix is now derived from `book`, and an unknown book raises rather than guessing.r"Zkalshi_r'Z_askrr4)r$r#zbook_quote: unknown book r&NZ_price)rrr rB)rr*sidemarrr book_quotems ,(rFc Cs|D]}t|d}|d}|dus|dvr2qz|tt|d}WnttfybYqYn0|jddddd}t||dd }|t|d SdS) aKickoff for a fixture, derived from any pre-match snapshot, or None. live_data carries no kickoff column, but `elapsed` is minutes relative to kickoff (negative before it), so KO = timestamp - elapsed falls out of any row. `elapsed` is whole minutes while the timestamp carries seconds, which lands the answer up to a minute early -- 05:59 for an 06:00 kick. Real kickoffs sit on 5-minute boundaries (:00 :05 :15 :30 :45), so rounding to the nearest 5 removes the artifact exactly rather than papering over it. timestampelapsedN)Nr)Zminutesr)ZhourZminutesecondZ microsecondgr@i,)Zseconds) rrrrr r replaceroundZ total_seconds)rrtseltZmidnightZsecsrrr_ko_froms   rOTc Cshtt}ttjtdD]p}t|dddN}t |D]2}|| d| d| d| df |q:Wd q1s0Yqg}| D]\\}}}} } |sqt d d | Dd d d} | sqt| } | d ur|rq| | rdnd } }| d ur|st| } | rdnd }| t||| | |tdd| Ddd | Ddq|jdd d|S)aoFixtures keyed to their pre-kickoff snapshots. require_result=True (the default, and what every SCORING path uses) drops fixtures with no final result: a bet that cannot be settled is not evidence and must never reach stat(). Pass False to see matches that have not been played yet -- the dashboard's live xhigh panel does, because "what has the rule taken on tonight's games" is exactly the question it answers. Those rows have ftr=None, and a caller showing them MUST render that as open rather than letting the `won` flag (which reads False on an unplayed fixture) print as a loss. z*.csvrutf-8newlineencodingZ game_dateleaguer-r.NcSs>g|]6}|dpddkrt|drt|d|fqS)r%rZPRErG)rrr)r(rrrrr+s"z!load_fixtures..cSs|dS)Nrr)rNrrrr,zload_fixtures..keyZapiZmarketcss|]\}}|VqdSrrr(r&rrrrr/r,z load_fixtures..cSsg|] \}}|qSrrrXrrrr+r,)gdrTr ftr_displayftr_srckoprecSs|dS)NrYrrrrrrUr,)rlistglobospathjoinLIVE_DIRopencsv DictReaderrappenditemssortedr!r<dictrOsort)Zrequire_resultZfixturesrafhrpreppedrYrTr-r.rr]rrZr[rrr load_fixturess: &   rnr5r-r.cCs|dD]v}t|dt|dt|d}}}d|||fvs|dkr|dkr|dksdqt|||f}g}tD]\}}t|d|\} } t|d|\} } | dus~| dus~d | krdkr~nq~d | krdksq~q~| | kr| | | df\} }}}n| | | df\} }}}|dus~d |kr=100 review — NOT wired now). Equivalence note (verified 2026-07-05): 'other NOT in band' == 'other CLV < 5%' under the below-other condition — the cheaper book always has the HIGHER CLV, so the other book can never sit above band while we're in it. Backtest difference set is empty; the n=286 numbers apply to this rule exactly as written. Shared-price only. One bet/fixture (first qualifying pre-KO snapshot, max CLV).r]pin_homepin_drawpin_awayNr4r"r$rrrr-rr.rGrYr\rTrZr[clvpxwonrCr*rAaskfairrLrYr\rTr-r.rrZ won_displayr[cSs|dSNrtrdrrrrU r,zfire_xlow..rV) rrr@SIDXrhrFBANDrgrjmaxfxrphpd_papfcandsrCsikmkapmpa_rAruotherr*rtZ other_clvrvrrr fire_xlows<.(@ $,&.  rcCsdddd|DDS)NcSsg|] }|r|qSrrr(brrrr+ r,z!classify_xlow..css|]}t|VqdSr)rr(rrrrr/ r,z classify_xlow..rrmrrr classify_xlow srmain) GAP_LEAGUESZUSLZPL1cCs|dtvrdS|dD]v}t|dt|dt|d}}}d|||fvs|dkr|dkr|dkstqt|||f}g}tD]\}}t|d|\} } t|d |\} } | dus| dusd | krdkrnqd | krdksqq| | kr| | | df\} }}}n| | | d f\} }}}|dusd |krLdksPqq||| ||}|||||}| |krtd |krtdksqqtd |krtdkrqnq|d d kr|d kp|d dko|dk}|t |||||| ||||d|d|d|d|d |d|d |d|d|d krbd ndk|ddq|rt |dddSqdS)zNfire_xlow with Polymarket-US standing in for Polymarket.com, GAP_LEAGUES only.rTNr]rprqrrr4r"r#rrrr-rr.rGrYr\rZr[rscSs|dSrzrr{rrrrUFr,z"fire_xlow_polyus..rV) rrrr@r}rhrFr~rgrjrrrrrfire_xlow_polyus's>.(@ $,&.  rcCsdddd|DDS)NcSsg|] }|r|qSrrrrrrr+Jr,z(classify_xlow_polyus..css|]}t|VqdSr)rrrrrr/Jr,z'classify_xlow_polyus..rrrrrclassify_xlow_polyusIsrcCs|dtvrt|St|S)aTHE forward-test firing function: xlow with whichever confirming book the league actually has. Poly.com where it exists; Poly-US in GAP_LEAGUES (USL, PL1), which have poly_tag=None and therefore no Poly.com quotes at all. Without this the two are permanently invisible to the forward test while being actively traded -- which is why they were put in the scope (user, 2026-08-23). HONEST NOTE. The n=286 backtest confirmed against Poly.com only, so Poly-US as a confirming book is UNVALIDATED -- it is a thinner, newer venue. The primary is now a mix of two confirmations. Those bets are a small minority (USL 50, PL1 21 vs ~190 Poly.com) and the row `xlow_polyus` still reports them alone, so the mix can always be decomposed. No double counting: the two paths are mutually exclusive by league, because fire_xlow_polyus returns None outside GAP_LEAGUES and fire_xlow cannot fire inside them (no Poly.com prices to compare against). rT)rrrrrrrrfire_xlow_primaryMsr>ZP1ZAR1ZN1ZD2ZD1ZSP1ZSP2ZE0ZE2ZNO1BSAZSC0ZT1ZF1ZI2CL1ZE1ZSE1ZMLSZI1z 2026-08-23z 2026-09-01rrZNWSLZKO1cCs|dtvS)NrT)rPRIMARY_LEAGUESrrrrin_primary_leaguessrcCsdddd|DDS)zxlow over PRIMARY_LEAGUES -- the primary forward scope. Uses fire_xlow_primary, so USL/PL1 are included via the Poly-US confirm.cSsg|] }|r|qSrrrrrrr+sz$classify_xlow_bt..css|]}t|rt|VqdSrrrrrrrr/sz#classify_xlow_bt..rrrrrclassify_xlow_btsrcCsdddd|DDS)axlow over everything outside the primary scope. Tracking only, never pooled into the primary -- see PRIMARY_LEAGUES. Note this now includes the two leagues REMOVED from the scope (BSA, CL1), so they stay measured rather than disappearing from the page.cSsg|] }|r|qSrrrrrrr+sz'classify_xlow_newlg..css|]}t|st|VqdSrrrrrrr/sz&classify_xlow_newlg..rrrrrclassify_xlow_newlgsrg?cCsddt|DS)NcSsg|]}|dtkr|qSruXLOW_PRICE_FLOORrrrrr+r,z%classify_xlow_f10..)rrrrrclassify_xlow_f10srz 2026-09-20c Csttttd}}}gg}}t|dddD]}|d|krt|D]\} } || | qJtt|d}}||drt||dnd} t| o| d|ko| d|k} | r|n||||d|q2||fS) aPoint-in-time exclusion. Returns (kept, dropped). A bet is dropped when its league's z -- computed over that league's bets settled on STRICTLY EARLIER game dates -- is below `floor`, once there are at least `min_n` of them. Same-day bets cannot inform each other: they wait in `pending` and fold in when the date rolls, because a bet struck at 13:00 cannot know the result of one kicking off at 15:00. NcSs|d|dpdfS)NrYr\rr)rrrrrU+r,zzgate_walk..rVrYrTnz)rr^rirhextendstatboolrg) betsmin_nfloorZhistpendingZcurkeptdroppedrlgbsr Zbadrrr zgate_walk s     rc Csztt}|D]}||d|q i}|D]D\}}t|}|sFq0t||d|ko`|d|k|d|kd||<q0|S)aSWhich leagues stand in DROP state now, over every settled bet given. CURRENT state, for display -- what the gate would do to the next fixture -- not the walk-forward record. Leagues still under the warmup come back flagged `warm` so the panel can show what it is waiting on rather than silently treating them as fine. rTrr)rZwarm)rr^rgrhrrj) rrrZbyroutrrr rrr zgate_state6s rcCs,dd|D}tt|\}}dd|DS)zPrimary scope, z-gate applied, POST-ZGATE_FREEZE bets only. The walk runs over the whole OOS history so the gate's state is warm, then the result is cut to post-freeze fixtures: state may use the history, evidence may not. cSsg|]}|dtkr|qSrY FREEZE_DATErrrrr+Qr,z'classify_xlow_zgate..cSsg|]}|dtkr|qSr ZGATE_FREEZErrrrr+Sr,)rr)rmoosrr&rrrclassify_xlow_zgateJsrcCs dd|D}ddt|DS)zThe UNGATED primary over the same post-freeze window -- the control. Without it the gated number has nothing to be read against and would get compared to the all-history primary, which is a different window. cSsg|]}|dtkr|qSrrrrrrr+[r,z'classify_xlow_postz..cSsg|]}|dtkr|qSrrrrrrr+\r,)r)rmrrrrclassify_xlow_postzUsr)i-cCsd\}}|dD]B}t|d}|dus|dkr4qt|t}||kr||}}q|dusd|tkrhdS|}t|dt|dt|d}}}d|||fvs|dkr|dkr|dksdSt|||f} g} tD]\} } t|d | \} }t|d | \}}| dus|dusd| kr.dkrnqd|krJdksNqq| | | | | }| | || | }t d|kot dknt d|kot dkn}}||krq|r|rd|krdkrnn|n| n|}|dusd|krdks qq|d d kr8| d kpL|d dkoL| dk}| t t ||||dq| rt | dddSdS)N)NgeAr]rHrrprqrrr4r"r$rrr-rr.rtrurvcSs|dSrzrr{rrrrUr,zfire_t60..rV) rrabs T60_TARGETT60_TOLr@r}rhrFr~rgrjr)rZbestZbdrrMr|rrrrrrCrrrrrZckZcpZk_inZp_inrurvrrrfire_t60ks.  .*F(F 4$.rcCsdddd|DDS)NcSsg|] }|r|qSrrrrrrr+r,z classify_t60..css|]}t|VqdSr)rrrrrr/r,zclassify_t60..rrrrr classify_t60src Csg}d}dD]Z\}}|dD]F}t|dt|dt|d}}} d||| fvs|dkr|dkr| dkszqt|||\} } | dus| dusd | krdkrnqd | krdksqqd }t||| f|} | r| | | nd} | dustd | krtdks qq|d d kr8|d kpL|d dkoL|dk}|t| | |dq qq |sxd|fSt|ddd|fS)z6book='kalshi' / 'poly'. Returns bet dict + have_price.F))r-r)r.r5r]rprqrrNr4rTrrr-rr.rcSs|dSrzrr{rrrrUr,zfire..rV)rrrFr@r~rgrjr)rr*rZhaverCZsidxrrrrrArurrtrvrrrfires.(@,.rcCs|gg}}|D]d}t|d\}}t|d\}}|r|s8q|rL|rL||q|r`|s`||q|r|s||q||fS)z>Return (unique_bets, overlap_bets) over shared-price fixtures.r"r$)rrg)rmuniqoverlaprkZkhprrrrclassifys  rcCst|}|sdStdd|D}tdd|D}tdd|D}tdd|D}|dkrr||t|nd}t|||d||d||d|d S) Ncss|]}|dVqdS)rvNrrrrrr/r,zstat..css.|]&}|dr"d|d|dndVqdS)rvr4rurNrrrrrr/r,css|]}|dVqdS)ruNrrrrrr/r,css"|]}|dd|dVqdS)rur4Nrrrrrr/r,rgd)rwriwrroir)r9r>mathZsqrtrj)rrZwinsZnetZmuvarrrrrrs$rz 2026-08-09g?c Cs|dD]}t|dt|dt|d}}}d|||fvs|dkr|dkr|dksdqt|||f}dD]0}t|d|\}}|dusv|dusvd |krdkrvnqvd |krdksqvqv|tkrqv|t|} | d krqv| || } td | krtdkrvnqv||d |d kr0d nd k|||| | |d|d|d|d|d |d|d|d kr~d nd k|d|d |ddSqvqdS)aAs fire_xhigh, but returns the FULL context of the qualifying snapshot. THE RULE LIVES HERE, ONCE. fire_xhigh() is a thin wrapper over this. The dashboard's live xhigh panel needs the side, price and CLV that fired, not just the settled bet, and re-implementing the conditions there is exactly how a displayed "signal" drifts into disagreeing with the forward test that scores it. Anything that wants to know what xhigh does asks this function. `won` is meaningless for a fixture with no result yet (ftr is None, so it reads False); a caller showing live fixtures must ignore it rather than render it as a loss. r]rprqrrNr4ror"rrr-rrrGrYr\rTrZr[r.)rurvrCrArwrxrtrLrYr\rTrrZryr[r-r.)rrr@rFXHIGH_PRICE_FLOORr}r~) rrrrrrrCrDrErxrtrrrfire_xhigh_details: .& >  "   rcCs"t|}|r|d|ddSdS)aKalshi mid-CLV in BAND on a side priced at/above the floor. No second book required. One bet per fixture: FIRST qualifying pre-KO snapshot, settled at the real Kalshi ask -- same non-look-ahead selection as fire_xlow, so the two are comparable bet-for-bet. Returns only what the scoring path needs, so adding fields to fire_xhigh_detail can never quietly change what stat() averages. rurv)rurvN)r)rr|rrr fire_xhighs rcCs6g}|D](}|dtkrqt|}|r||q|S)zEBets for fixtures AFTER XHIGH_FREEZE only -- see the provenance note.rY) XHIGH_FREEZErrg)rmrrrrrrclassify_xhighs  rcstjtsdSttddd,}t|}|jp2g}t|}Wdn1sP0Y||krfdSttddddH}tj ||d}| |D]| fdd|DqWdn1s0YdS) zSMigrate the log header if columns changed (preserve old rows, blank-fill new cols).NrrPrQw fieldnamescsi|]}||dqS)rr)r(rrowrr r,z&_ensure_log_schema..) r`raexistsLOG_CSVrdrerfrr^ DictWriter writeheaderwriterow)rrlZrdoldrrrrr_ensure_log_schemas: rc Cs|durt}dd|D}|r(t|ng}|r8t|ng}|rHt|ng}dd|D}|rft|nggf\}}|r~t|ng}t|} |rt|ng} |rt|ng} t t t td} d|fd|fd|fd|fd | fd | fd | fd |fd |fd|ff D]\} }t|}|r|dnd| | d<|r@t|ddnd| | d<|rbt|ddnd| | d<|rt|ddnd| | d<qt| }t|tjt }ttdddd6}tj||d}|r||| Wdn1s0Y| S)NcSsg|]}|dtkr|qSrrrrrrr+r,z log_snapshot..cSsg|]}|dtkr|qSrrrrrrr+%r,)Zrun_dateZ freeze_dateZ xhigh_freezeZ zgate_freezexlow_btxlow xlow_newlgxlow_f10xhigh xlow_zgate xlow_postzt60uniquerrrZ_nrr4rZ_wrrZ_roirr5Z_zrErPrQr)rnrrrrrrrrrZtodayZ isoformatrrrrrKr^keysrr`rarrrdrerrr)rmrrrrrrrrrrrrnamerr rnewrlrrrr log_snapshotsB     ""& *rc Cst}tdtdt|dddd|Dfddd|DffD]\}}td |d |sjtd qDt|}d d|D}t|\}}t|}d |fd|fd|fd|fd|ffD]r\}} t| } | rtd|d| ddd| ddd| ddd| ddd| dd  qtd|d!qtqDt|} t| } td"t d#t d$| rtd%| ddd| ddd| ddd| ddd| dd  ntd&td'td(t |td)t dS)*Nz FREEZE_DATE=z total resolved fixtures= zIN-SAMPLE (<=freeze, reference)cSsg|]}|dtkr|qSrrr(r rrrr+Fr,zmain..zFORWARD (OOS, >freeze)cSsg|]}|dtkr|qSrrrrrrr+Gr,z--- z ---z (no fixtures yet) cSsg|]}|dtkr|qSrrrrrrr+Lr,zXLOW* ZXLOWf10zt60 zunique rz z n=rZ4dz WR=rz4.1fz% impl=rz % ROIturn=rz+6.1fz% z=rz+.2fz6 (no shared-price bets -- Poly feed off / off-season)z--- XHIGH (frozen z, Kalshi-only, mid>=z) ---z XHIGH n=zA XHIGH n= 0 (no fixtures after the freeze yet -- expected)zE needs ~480 bets to resolve; pre-freeze data is EXCLUDED by design. zK * XLOW = PRIMARY rule (cheaper book, below other book AND below Pinnacle)zlogged snapshot -> ) rnprintrr9rrrrrrrrr) rmZlabelsubrrrrrrrr ZxhZshrrrrCsH   *. ( __main__)T)N)O__doc__rer_r`rrrr collectionsrrcrrr~rrrr!r6r0r3r<r@rBrFrOrnr}rrsys_sys_osrainsertrbdirnameabspath__file__Z live_monitorr Exceptionrrr frozensetZBACKTEST_LEAGUESZ SCOPE_FROZENZSCOPE_FROZEN_KO1rrZin_backtest_leaguesrrrrrZ ZGATE_MIN_NZ ZGATE_FLOORrrrrrrrrrrrrrrrrrrr__name__rrrrs%    / > ", !?  -  *$