a $œjàCã @s°dZddlZddlZddlZddlmZdZdZdZ dZ gd¢Z d d d d d ddddddddœ Z dd„Z dd„Zdd„Zgd¢Zdd„Zdd „Zd!d"„Zd#d$„Zed%kr¬eƒdS)&aNBuild a SMALL, downloadable fill-analysis dataset from mm_live_legs.csv. WHY THIS EXISTS. mm_live_legs.csv is ~31MB and growing ~10MB/day, so the dashboard has to gzip it to embed it, and a .gz is exactly what an outside analysis environment tends to refuse. This writes a plain CSV small enough to embed and hand to anything. TWO THINGS IT FIXES BESIDES SIZE: 1. ONE ROW PER RESTING EPISODE, NOT PER RE-POST. The legs log writes a row every time a leg is re-posted, so a leg that rested for an hour contributes ~50 rows and one that filled in 90s contributes one. Treating those as independent observations is a trap: it weights the sample toward legs that DID NOT fill, which is the outcome being predicted. Collapsing to (ticker, price) episodes makes each resting order one observation and moves exposure into its own column, where it belongs as a feature. 2. FEATURES ONLY EXIST FROM 2026-08-18. ask/bid/mid/pin_fair/league logging was added mid-stream: 0% of 08-16 rows have them, 21.7% of 08-17, 100% from 08-18. Rows without them cannot support the analysis and are dropped rather than silently becoming empty cells that a model reads as zeros. The outcome column is `filled`, joined from mm_live_realfills.csv on (ticker, price_cents). Usage: python research\mm_fill_dataset.py Writes: research\mm_fill_dataset.csv éN)Ú defaultdictz#E:\soccer\research\mm_live_legs.csvz(E:\soccer\research\mm_live_realfills.csvz&E:\soccer\research\mm_fill_dataset.csvz 2026-08-18))Úts_firstz1 means the touches do NOT cover what the book still needs -- which gates a new book and forces a committed one to quote above the touch.)Úfreshz|1 if this was the first time this leg had ever been offered, 0 if it was already in the offered set from an earlier session.)ÚfilledzOUTCOME. 1 if the order was filled, 0 if it expired or was cancelled. Joined from mm_live_realfills.csv on (ticker, price_cents).)Úfill_qtyzFContracts filled, blank if never filled. Kalshi can fill fractionally.)Úseconds_to_fillz1Seconds from post to fill, blank if never filled.)Úis_xlowzð1 if the frozen xlow rule ALSO fired on this exact fixture and side, 0 if not. Both signal rules are home/away only, so a draw leg is always 0. BLANK means the fixture could not be resolved (unmapped team abbreviation) -- unknown, not "no".)Úis_xhighzæ1 if the frozen xhigh rule ALSO fired on this exact fixture and side, 0 if not. Same blank/0 convention as is_xlow. xhigh overlaps xlow heavily -- both fire on Kalshi CLV in the same band -- so the two columns are not independent.ééééééééé é é é ) ZJANZFEBZMARZAPRZMAYZJUNZJULZAUGZSEPZOCTZNOVZDECcCsŠddl}|pd dd¡d}| d|¡}|r:| d¡tvr>d S| d¡| d¡| d ¡| d ¡f\}}}}d |›dt|d ›d|›|fS)a~'KXEPLGAME-26AUG23MCIBOU' -> ('2026-08-23', 'MCIBOU'), or (None, None). The Kalshi event code packs the date and BOTH team abbreviations with no separator, and the abbreviations are 2-4 characters, so the pair cannot be split on width. It does not need to be: the HOME abbr is a prefix of the pair and the AWAY abbr is a suffix, which is enough to place a leg. rNÚú-r"éÿÿÿÿz%^(\d{2})([A-Z]{3})(\d{2})([A-Z0-9]+)$r#©NNr$r%Z20Z02d)ÚreÚrsplitÚmatchÚgroupÚ_MONTHS)rr2ÚtailÚmZyyZmonZddÚteams©r:úresearch/mm_fill_dataset.pyÚ _parse_eventws ,r<cCsJ|r|s dS|dkrdS| |¡| |¡}}|r:|s:dS|rF|sFdSdS)zF'home' | 'draw' | 'away' for a leg abbreviation, or None if ambiguous.NZTIEÚdrawÚhomeÚaway)Ú startswithÚendswith)r9ÚlegZstartsZendsr:r:r;Ú _leg_sideˆsrCc Csœzddl}ddl}WnBtyV}z*tdt|ƒj›d|›dƒWYd}~dSd}~00tƒtƒ}}zè|jddD]Ö}|d rŒ|d dni}| d ¡| d ¡}}|rt|s°qt|  |¡|  |¡} } | rt| sÐqt|d |d | ›| ›  ¡f} |  |¡} | r|  d¡r|  | | df¡|  |¡} | rt|  d¡rt|  | | df¡qtWnDty’}z*tdt|ƒj›d|›dƒWYd}~dSd}~00||fS)a¯{(game_date, league, teams_code, side)} for xlow and for xhigh. WHY THIS JOIN EXISTS. The maker quotes all three legs of a book blind to what the signal rules think. Tagging each leg with whether xlow or xhigh also fired on that exact fixture AND side makes it possible to ask whether a leg the signal rules call mispriced behaves differently -- fills faster, completes more often -- which is a question neither dataset can answer alone. Keyed on the FIXTURE, not on (date, league): several matches in one league share a date, and a coarser key would tag the wrong game. The Kalshi abbreviations come from team_map via maps.kalshi_abr, so a team missing there simply produces no key and the row is left blank rather than a confident 0. Returns ({}, {}) rather than raising if anything is unavailable -- this is an annotation on a dataset, and it must never stop the dataset building. rNz" [warn] signal join unavailable (z: z!); is_xlow/is_xhigh will be blankr1F)Zrequire_resultZprer>r?Úgdrr z [warn] signal join failed ()Zforward_test_unique_bookÚmapsÚ ExceptionÚprintÚtypeÚ__name__ÚsetZ load_fixturesÚgetZ kalshi_abrÚupperZ fire_xlowÚaddZfire_xhigh_detail)ÚFrEÚeZxlowZxhighZfxZr0ÚhÚaZhaZaaÚkeyÚbÚdr:r:r;Ú signal_hits–s6   rU©rrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!c Cs@t|dƒ ¡}d|vr"| dd¡}tt t |jddd¡¡ƒS)z@Read a CSV that may contain NUL bytes from an interrupted write.Úrbóóúutf-8Úreplace)Úerrors) ÚopenÚreadr[ÚlistÚcsvZ DictReaderÚioÚStringIOÚdecode)ÚpathÚrawr:r:r;Ú_readÕs rfc Cs(z t|ƒWSttfy"YdS0dS)N)ÚfloatÚ TypeErrorÚ ValueError)Úvr:r:r;Ú_fÝs rkcCs~|dur dSt| dd¡ƒ\}}|p&d dd¡d}t||ƒ}|rN|rN|durRdS|dkr^dS|| d d¡||f|vrzdSdS) ao1 / 0 / '' for whether a signal rule fired on this exact leg. BLANK, NOT ZERO, when the fixture cannot be resolved -- an unmatched team abbreviation or an unparseable event code is 'unknown', and writing 0 there would tell a model the rule declined this leg when in fact nobody asked. Both rules are home/away only, so a draw leg is a genuine 0. Nr.rr/r"r0r=rr)r<rKr3rC)ÚhitsrQrrDr9rBr r:r:r;Ú_hitäs rmc#%Csàtƒ\}}ttƒ}ttƒD]L}z|dtt|dƒƒf}Wnttt fyVYqYn0||  |¡qtt ƒ}t dt |ƒd›ƒdd„|Dƒ}t dt›dt |ƒd›ƒi}|D]°}ztt|dƒƒ}Wntt fyæYq´Yn0|d|f}| |¡} | dur||d d œ||<q´| d d 7<|d | d d krH|| d <|d | dd kr´|| d<q´t dt |ƒd›ƒg} d} | ¡D]T\\} }} | d| d } }d}}d}z4tj | d ¡}tj |d ¡}|| ¡d}Wntt fyYn0t|  d¡ƒt|  d¡ƒ}}t|  d¡ƒ}|durP|durP||dnd}|durn|d|dnd}| | |f¡p„g}g}|rš|rš|ngD]¨}| d¡p°d}tjdd}|sà| d ¡pÒd}tjdd}|sêqžztj |¡}Wnt yYqžYn0|||kr6||kržnn |  |¡qž|rV| d 7} tdd„|Dƒƒ}|r€t|d d¡ƒnd}|   | d |dur |d›nd| d |  d d¡|  d!d¡| |  d"d¡||  dd¡|  dd¡|  d#d¡|durú|d›nd|  dd¡|dur|d›nd|  d$d¡|  d%d¡|  d&d¡|  d'd¡|  d(d¡|  d)d¡|  d*d¡|  d+d¡|  d,d¡|  d-d¡|  d.d¡|  d/d¡|rœd nd|r¬|d›nd|durÀ|d0›ndt|| | ƒt|| | ƒd1œ¡qŠ| jd2d3„d4ttd5dd6d70}tj|td8} |  ¡|   | ¡Wdƒn1s:0Yddl!}!|!j" #t¡}"t ƒt d9t›ƒt d:t | ƒd›ƒt d;| d›d|"d?d›d@ƒt dA|"dBd?d›d@ƒdS)CNrr zlegs rows read : ú,cSs*g|]"}| d¡pddd…tkr|‘qS)Útsr.Nr+)rKÚ FEATURES_FROM)Ú.0Úrr:r:r;Ú rYzmain..zrows >= z : r")ÚfirstÚlastÚnrvrorurtzresting episodes : rgN@r r rédgY@Z posted_atr.é<)Zsecondsi¤css |]}t| d¡ƒpdVqdS)ÚcountrN)rkrK)rqÚxr:r:r;Ú OrYzmain..rz.2frrr r rrrrrrrrrrrrz.1frVcSs|dS)Nrr:)rrr:r:r;ÚtrYzmain..)rRÚwrZ)ÚnewlineÚencoding)Z fieldnameszwrote z rows : z filled : z (z%)z size : g€„.Az MBz base64 embed : gq= ×£põ?)%rUrr_rfÚFILLSÚintrgÚKeyErrorrhriÚappendÚLEGSrGÚlenrprKÚitemsÚdtÚdatetimeZ fromisoformatZ total_secondsrkZ timedeltaÚsumrmÚsortr]ÚOUTr`Z DictWriterÚOUT_COLSZ writeheaderZ writerowsÚosrdÚgetsizeÚmax)#Z xlow_hitsZ xhigh_hitsZfillsrrrRZlegsZkeptZepZpcrOZrowsZn_filledZtkrrQrSZt0Út1ZminsZaskZbidZpinZspreadZ ask_vs_pinZcandZflrzZstampZslackZxdZqtyZstfZfhr}rÚsizer:r:r;ÚmainøsÜ        $   $                      á "* ,r’Ú__main__)Ú__doc__r`rarˆr‡Ú collectionsrr„r€r‹rpZ FIELD_DOCSr6r<rCrUrŒrfrkrmr’rIr:r:r:r;Ús.  B  ÿ3