a j@sdZddlZddlZddlZddlZddlZddlZddlmZej ej e Z ej de ej e dZej e dZgdZedZdd Zd d eDZd d ZddZddZddZddZddZddZddZddZd"dd Z e!d!kre"e dS)#aXBook-level dataset: what makes a BOOK complete, rather than a leg fill. WHY BOOK LEVEL. project_mm_completion_arithmetic reduces the maker to one number -- the completion rate -- because a complete book is near-deterministic (+1.78c on Kalshi, SD 1.69) while a partial one is a coin flip at ~25x the SD. Leg fill rate is an input to that; it is not the thing being decided. A model fitted on legs answers "will this order fill", which is the wrong question: three legs filling INDEPENDENTLY is not the same event as a book completing, because the legs of one book share a market, a moment and a budget. BUILT FROM mm_fill_dataset.csv, not from the raw log, so it inherits that file's two corrections: one row per resting order (not per re-post), and rows dropped before features existed. Run mm_fill_dataset.py first, or just run this -- it rebuilds automatically. THE UNIT IS A BOOK WE QUOTED. Features are taken AT OPEN (the first resting order on each leg), because the question is what could have been known when the decision to open the book was made. Anything measured later is contaminated by the outcome: a book that fills one leg gets re-priced by the budget rule, so its later above_touch and scale are consequences of filling, not causes. WATCH OUT FOR legs_quoted. A book where only two legs were ever offered CANNOT complete, and including those in a completion rate silently understates it. `complete` is defined against 3 legs; filter on legs_quoted==3 when the question is "given a full book was offered, did it complete". Usage: python research\mm_book_dataset.py Writes: research\mm_book_dataset.csv N) defaultdictzmm_fill_dataset.csvzmm_book_dataset.csv)")venuea'kalshi' or 'polyus'. Both makers quote the same fixtures on different venues with different economics -- Kalshi CHARGES the maker 0.0175*p(1-p), Poly-US CREDITS 0.0125*p(1-p) -- so break-even completion differs and the two must never be pooled without splitting on this.)eventz.Event/market identifier -- one match, one row.)leaguezDivision code.)ts_openz1When the first leg of this book was first posted.)hrs_to_ko_openaHours to kickoff when the book was opened. LEG fill rate climbs into kickoff (26.1% inside 30min vs 6.8% beyond 24h) but BOOK completion moves the other way: among books that took a leg, with the entry gate held fixed, strand rate is ~17% under 6h against ~9% over it. A late book fills a leg and runs out of time to finish, which is the expensive half. MIN_OPEN_H=6 exists for this.) legs_quotedz~How many of the three legs were ever offered. A book with <3 CANNOT complete -- filter on ==3 for a clean completion question.) gross_c_openzSum of the three asks at open, minus 1.00, in cents. The overround the book was opened on, and the variable the entry gate tests. Blank unless all three legs were quoted.) pin_sum_openzpSum of Pinnacle fair probabilities across the legs. Should be ~1.00; deviation flags a stale or mismatched fair.) ask_max_openzHighest leg ask at open -- the favourite. Favourites fill far more often (49.7% vs 17.7% for draws), which is how a book strands short the leg most likely to win.) ask_min_openz'Lowest leg ask at open -- the longshot.) spread_c_meanzcMean ask-minus-bid across the legs at open, in cents. Mostly 1c: the venue prices on a 1-cent grid.)queue_ahead_minzDSmallest queue ahead of us across the legs. The easiest leg to fill.)queue_ahead_maxz_Largest queue ahead of us. The hardest leg -- and a book completes only if its WORST leg sells.)book_volume_openzContracts traded across all three legs. The strongest single completion predictor found so far (+0.091 AUC); min-leg volume was null (rho +0.095, p=0.189).)leg_volume_minzVolume on the thinnest leg. Kept because the paper-era gate used it -- it did NOT replicate live, so treat a result here with suspicion.)open_interest_sumz%Open interest summed across the legs.) n_posts_totalzDRe-posts across every leg. A proxy for how long the book was worked.)minutes_exposed_maxz'Longest any single leg sat on the book.) is_xlow_anyzr1 if the frozen xlow rule fired on this fixture (either side), else 0. Blank if the fixture could not be resolved.) is_xhigh_anyzl1 if the frozen xhigh rule fired on this fixture. xhigh overlaps xlow heavily -- not an independent feature.) legs_filledzOUTCOME. How many legs filled, 0-3, per the EXCHANGE fill record -- the authority. Maker fills only (post_only, so never taker), which also excludes xlow taker buys on the same tickers.)legs_filled_loga_The same count derived from the LOCAL fills log, kept for comparison. It undercounts: the log joins on (ticker, price_cents) so a re-priced leg misses, and the log only records while the maker is up. On the 2026-08-28 audit the exchange said 115 books complete and the log-derived figure said 45. A large gap here means fills are being missed locally.) fill_sourcezw'exchange' normally; 'log' if the exchange could not be read, in which case legs_filled falls back and will undercount.)completezxOUTCOME. 1 if all three legs filled. This is the number the strategy lives or dies on -- break-even was 61.4% on Kalshi.)leg_sold_firstaWhich OUTCOME sold first, in 1X2 terms: '1' home, '2' away, 'X' draw. Blank when no leg filled, or when the abbreviation cannot be resolved against the event code. Derived from first_fill_side, which carries the Kalshi leg abbreviation ('CAT') rather than a side -- readable only if you know which team that was. A Kalshi event ticker ends in the two team codes concatenated (26AUG17GEMCAT = GEM at home to CAT), so the abbreviation is matched against that pair: prefix means home, suffix means away, TIE means draw.)first_fill_sidez=Leg abbreviation of the first leg to fill, blank if none did.)first_fill_secsz1Seconds from that leg being posted to it filling.)sizezContracts per leg the maker was working, from the largest leg. SIZE went 1 -> 3 on 2026-08-23 and mixing the two eras is a trap: a size-1 completion pays ~+$0.05 and a size-3 one ~+$0.20, so an average across both understates a good book threefold.)shapeazOUTCOME, stricter than `complete`. 'unfilled' = quoted but never touched. 'clean' = three legs at the SAME size, the only shape that is actually hedged. 'ragged' = three legs at different sizes (a leg that took a 0.01-contract fill and never finished still reads complete). 'over' = a leg exceeded size, which means two orders rested at once. 'stranded' = fewer than three legs.) has_takeraW1 if WE traded this event as a taker -- xlow, or a manual trade. The maker is post_only so it never takes. Books with this set have P&L that is not the maker own: it is the difference between a -$3.22 maker and a -$53.05 account. NOT the same as is_xlow_any, which only says the SIGNAL fired (169 books) rather than that a trade happened (13).)armzWhich pricing arm owned the book: 'budget' (a part-filled leg prices at exactly what the book still owes) or 'latch3' (the ratchet survives the fill but sits at most 3c above that budget). Blank for books opened before the split. From mm_arm_log.csv.)settledz.1 once every leg has settled and pnl is final.)pnlzOUTCOME. Realised dollars, from the venue settlement records (revenue - cost - fees) summed across the legs. Blank until settled. This is the column every P&L question should be answered from; reconstructing it by hand is where the errors came from.)premium_over_budget_medzMedian cents a part-filled leg was offered ABOVE what the book still needed, over the book's life. 0 under the 'budget' arm by construction; the old unbounded latch ran a median +1c with a tail to +54c, and that tail is what stranded books.z^\d{2}[A-Z]{3}\d{2}cCs|sdSt|}|dvr$dSt|p,dddd}td|}|rX||vr\dS||||}}|r~|s~dS|r|sdSdS) a '1' / '2' / 'X' for the leg abbreviation, or '' if it cannot be resolved. A Kalshi event ticker is SERIES-, e.g. KXARGPREMDIVGAME-26AUG17GEMCAT: GEM at home to CAT. So stripping the date leaves the two abbreviations concatenated, and the leg is home if it is the prefix and away if it is the suffix. Returns '' rather than guessing when both or neither match -- a code like 'AAA' vs leg 'A' is genuinely ambiguous, and a wrong side here would be invisible in the data. )ZTIEZDRAWXr&-12)strupperstriprsplit_DATE_REsub startswithendswith)rlegtailZpairZstartsZendsr6research/mm_book_dataset.py leg_to_1x2s   r8cCsg|] \}}|qSr6r6).0c_r6r6r7 r<c Cs(z t|WSttfy"YdS0dSN)float TypeError ValueErrorvr6r6r7_fs rDc CsdzBtjdtjtddddl}|ddddidp>g}WnBt y}z*t d t |j d |d WYd}~dSd}~00t t}|D]}t|d d dks|drqt|dd dkrq|dpd }d|vrq|dd\}}t|dpdt|d}} |||d| f\} } | || durL| n| f|||<qt|S)aTevent -> {leg: (qty, price)} from Kalshi's OWN fill record. WHY NOT THE LOCAL LOG. `filled` in the leg dataset is a join of mm_live_realfills.csv on (ticker, price_cents) PLUS a time window. Both halves lose fills: * the price key misses a leg that was re-priced between the resting order going on and the fill being recorded; * the local log is itself incomplete -- 354 legs against 391 the exchange reports, because the fill detector only runs while the maker is up. Audited 2026-08-28: the exchange said 115 books completed, the dataset said 45. Completion is the one number this strategy is judged on, so it cannot be derived from a lossy local join. Maker fills only: our orders are post_only, so `is_taker` is always false and the side is never 'yes'. That is the same filter mm_maker_live uses to attribute its own fills, and it excludes xlow's taker buys on the same tickers. Returns None if the exchange cannot be read, so the caller can fall back to the log rather than silently reporting every book as incomplete. r..mainN/portfolio/fillsfillslimitz$ [warn] exchange fills unreadable (: z&); falling back to the local fills logactionr%sellis_takersideZyestickerr'r(Zcount_fpZyes_price_dollars)syspathinsertosjoinHERE kalshi_trader_get_allget Exceptionprinttype__name__rdictr,lowerr/rD) ktrowseoutftevr4qpxZprev_qZprev_pxr6r6r7exchange_fillss6"$$rjc Cszrtjdtjtddddl}g}dD]B}z&||ddd|d dpPg7}Wq,t ylYq,Yq,0q,Wnt yt YS0d d |DS) aEvents where WE traded as a taker -- xlow, or a manual trade. The maker is post_only so it never takes. An event with a taker fill has P&L that is not the maker's, and conflating the two is how a -$3.22 maker was reported as losing $53 for most of a day. rrErFNrr(rGrHrJrIZ subaccountcSsBh|]:}|drd|dp dvr|dp0ddddqS)rNr'rPr%r(r)rZr/)r9rer6r6r7 sztaker_events..) rRrSrTrUrVrWrXrYrZr[set)rarb_subr6r6r7 taker_eventss(    rpc CsTzrtjdtjtddddl}g}dD]B}z&||ddd|d dpPg7}Wq,t ylYq,Yq,0q,Wnt yiYS0t d d }|D]}|d pd }d|vrq| dd\}}t |dpdd}t |dpdt |dpd} ||} | d|| t |dp*d7<| d |qdd|DS)a;event -> (realised dollars, all_legs_settled) from the venue's own books. revenue - cost - fees, per leg, summed. This is the authority: every P&L figure should come from here rather than being rebuilt from fill prices, which is where the size-1/size-3 mixing and the taker contamination crept in. rrErFNrkz/portfolio/settlementsZ settlementsi,rlcSs dtgS)NrQ)rnr6r6r6r7/r=zsettled_pnl..rPr%r'r(ZrevenuerQY@Zno_total_cost_dollarsZyes_total_cost_dollarsZfee_costcSs(i|] \}}|t|dd|dfqS)rr()round)r9rgrCr6r6r7 Er=zsettled_pnl..)rRrSrTrUrVrWrXrYrZr[rr/rDadditems) rarbroaggrrfrgr4ZrevZcostar6r6r7 settled_pnls:     &( r{cCshtjtd}tj|siShd}i}t|D]0}|dr2|d|vr2||d|dq2|S)z:event -> pricing arm, from the append-only assignment log.zmm_arm_log.csv>budgetZlatch3Ztieloadrr!)rUrSrVrWexists_readrZ setdefault)rSZvalidrdryr6r6r7arm_ofHs   rcCstjtd}tj|siStt}t|d,}| dd dd d}|D]}| dd}z*t t|ttt| dd}WntyYqZYn0|dd krqZt|d t|d }}t|d t|d }} d|||| fvsZ|dkrqZ| dkr&|| n|} ||dt|d| dqZWdn1sf0Yi} |D]&\} } | | t| d| | <q|| S)zevent -> median cents a PART-FILLED leg was offered above its budget. The whole subject of the arm test. Recomputed from the leg log because nothing else records what the book was actually asked to fetch. zmm_live_legs.csvrbr=utf-8replace,Zliver*Z collectedZ price_cents touch_askZscaleNr?rrrd)rUrSrVrWr}rlistopenreadlinerdecoder.splitr_zipnextcsvreaderioStringIOr[rZrDappendrtrwsortlen)rSperfhZhdrlinerycolriZtouchZscr|rdrgrCr6r6r7budget_premium^s6 "    FrcCszBtjdtjtddddl}|ddddidp>g}Wnt yXiYS0i}|D]|}t |d d  d ksb|d rqb|d pd }d|vrqb| ddd|dpd }}||vs|||krb|||<qb|S)zGevent -> earliest maker fill timestamp. Used to place backfilled books.rrErFNrGrHrIrJrLr%rMrNrPr'r(Z created_time) rRrSrTrUrVrWrXrYrZr[r,r`r/)rarbrdrerfrgtsr6r6r7 fill_timess.  "  rc Cs2t|ddd}ttt|ddS)Nrrr=rr) rreadrrrZ DictReaderrrr)rSrawr6r6r7r~sr~cCsrtjtd}tj|sgSz&tjdtddl}|p@i}WnDt y}z,t dt |j d|di}WYd}~n d}~00t dd}t|D]>}|d pd }d |vrq|d d \}}||||qg} |D]\}} d d| D} dd| D} dd| D} t| dkrDt| ddnd} t||pXi}dd| D}| d|| tt| dpd ddtdd| Ddt| | dur| dnd d | rt| nd| rt| nd|r dt|t|dnd dddddtdd| Ddd d t|t||rHdndt|dkr\d ndd d d d q| S)!aQPoly-US books as rows in the same shape, from its own logs + venue record. FEWER FEATURES ON PURPOSE. pus_live_legs.csv records ts/slug/side/price/ touch/above_touch/gross/collected/need -- it has no volume, queue depth, open interest or Pinnacle fair, because that maker was written to a narrower log. Those columns come out BLANK rather than zero: a model that reads 0 for "queue depth unknown" is being lied to. Completion comes from the VENUE's settlement record, same as the Kalshi side, for the same reason -- the local state stops writing during 503s. zpus_live_legs.csvrNz) [warn] poly-US settlement unavailable (rK)cSsttSr>)rrr6r6r6r7rqr=zpolyus_books..slugr%r'r(cSs&i|]\}}|t|ddddqS)cSs |ddS)Nrr%rZxr6r6r7rqr=z)polyus_books...keyr)sorted)r9ZlgrCr6r6r7rusz polyus_books..cSsg|]}t|dqS)rrDrZr9ryr6r6r7r<r=z polyus_books..cSsg|]}|dur|qSr>r6r9rzr6r6r7r<r=rrrcSs0g|](}t|dpdt|dp(dqS)rrZ touch_bidrrr6r6r7r<sZpolyusrcss$|]}|D]}|ddVq qdS)rr%Nr)r9rCryr6r6r7 r=zpolyus_books...2frcss|]}t|VqdSr>)rr9rCr6r6r7rr=exchangelog)rrrrrrr r r r r rrrrrrrrrrrrrrrr)rUrSrVrWr}rRrTZ pus_maker_pnlZ settled_booksr[r\r]r^rr~rZr/rrwvaluesrsumrkeysriterminmax)Zlegs_csvPr"rcrryrrgr4rdlegsfirstsasksgrossfilledZspreadsr6r6r7 polyus_bookssr    ""  rTc!(sd|r:tjtstdtjtjtj t dgddddt }t }t }t}t}tt}tt}|D] drl|dqlg}|D]\} } i} t| ddd D]| d d gqd d | Dfdd} t} ddD}dd|D}| dkrFt|dkrFt|ddnd}dd| D}|durr|d}}nt|| pid}}d}t| ddd D]4ddkrtddur}qڐqfdd}t|pi| i}|r&td d!|pi| iDng}|r@|t|d"nd#|sPd$}nRt|dkrdd%}n>tfd&d!|Drd'}n |d(|d)d*krd+}nd,}td-d!| D}|d.| t t!d/d || d0t"| |dur|d1nd | dkr td2d!d3dDDd4nd | d5t"| d5t| d6d7ddurX| d6d8dd1nd | d9t| d9t"| d:t"| d;t| dt"|d?|d@t|t||t|dkrdAnd)t#| |r|d nd |r|d d nd |r|dd nd rdBnd || |vr(dAnd)|| d | |vrZ|rZ||| dAkrZdAn| |vrhd)nd | |vr|| d)d4nd || d dC"qdDdE|D}t$}|piD]r\} } | |vs| sڐqtdFd!| D}|r|t|d"nd#|sd$}nRt|dkr*d%}n>tfdGd!|DrHd'}n |d(|d)d*krdd+}nd,}|d.| d || d t| t| d dHt| dkrdAnd)rdBnd || |vrdAnd)|| d | |vrt| || dAkrdAn| |vrd)nd | |vr|| d)d4nd || d dIq|%t&|j'dJdd t(t)dKd dLdMJ}t*j+|t,dN}|-|D]|.fdOd t,DqzWdn1s0YdPd|D}dQd|D} tdRt)tdSt|tdTt|tdUt| |r:dVdWt| t|dXdYnd tdZtj/t)d[d1d\d)S)]a rebuild=False when the caller has ALREADY refreshed the leg dataset. The dashboard rebuilds mm_fill_dataset itself before embedding, so letting this re-run it as a subprocess would parse ~350k log rows twice on every page build for the same answer. z&leg dataset missing; building it firstzmm_fill_dataset.pyTi)Zcapture_outputtextZtimeoutrcSs |ddSNts_firstr%rrr6r6r7rqr=zmain..rrOr%cSsi|]\}}||dqS)rr6r9srCr6r6r7rur=zmain..cs4fddD}dd|D}|r0||SdS)Ncsg|]}t|qSr6rrnamer6r7r<r=z%main..col..cSsg|]}|dur|qSr>r6rr6r6r7r<r=)r)rrxvalsrrr7rszmain..colcSsg|]}t|dqS) ask_at_postrrr6r6r7r< r=zmain..cSsg|]}|dur|qSr>r6rr6r6r7r< r=rrrrNcSs&g|]\}}tdd|Dr|qS)css|]}|ddkVqdS)rr*Nrr9rr6r6r7rr=z"main...)anyrr6r6r7r<s rrcSs |ddSrrrr6r6r7rqr=rr*Zseconds_to_fillcsFfddD}tdd|Dr,dStdd|DrBdSdS) Ncsg|]}|qSr6rrrr6r7r<r=z)main..anyflag..css|]}|dkVqdS)r%Nr6rr6r6r7r r=z(main..anyflag..r%css|]}|dkVqdS)r*Nr6rr6r6r7r"r=r(r)rallr)rrrrr7anyflagszmain..anyflagcss|]\}}|VqdSr>r6r9rhr;r6r6r7r'r=zmain..rrQZunfilledZstrandedc3s|]}|dkVqdSư>Nr6r9rh)_sizer6r7r0r=Zoverr)rrZraggedZcleancss|]}|ddVqdS)rr%Nrrr6r6r7r7r=ZkalshirZ hrs_to_korcss|]}|dur|VqdSr>r6rr6r6r7r@r=cSsg|]}t|dqS)Zpin_fairrrr6r6r7r<@r=z.4frZspread_ccSst|t|Sr>rrrBr6r6r7rqEr=cSst|t|Sr>rrBr6r6r7rqDr=Z queue_aheadZ book_volumeZvolumeZ open_interestcss$|]}tt|dpdVqdS)Zn_postsrN)intrDrZrr6r6r7rKr=Zminutes_exposedZis_xlowZis_xhighr(g)"rrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#r$cSsh|] }|dqS)rr6rr6r6r7rmrr=zmain..css|]\}}|VqdSr>r6rr6r6r7rwr=c3s|]}|dkVqdSrr6r)szr6r7r}r=z!exchange (backfilled, no leg log))rrrrrrrrrrrr r!r"r#r$cSs |dp dS)Nrr%r6ryr6r6r7rqr=wr)newlineencoding)Z fieldnamescs*i|]"}||durdn|qS)Nr%r)r9krr6r7rur=cSsg|]}|ddkr|qS)rrr6rr6r6r7r<r=cSsg|]}|dr|qS)rr6rr6r6r7r<r=zwrote z books : z full 3-leg books : z completed : z (rz.1fz%)z size : g.Az MB)0rUrSr}LEGS_DSr\ subprocessrunrR executablerVrWrjrpr{rrr~rrrZrrwrrrrrrrDrnrrrrrr8rextendrrrOUTrZ DictWriterOUT_COLSZ writeheaderZwriterowgetsize)!ZrebuildexZtakZstlZarmsZpremrbZbooksrdrgrZby_legrZn_legsrrZ log_filledZ filled_legssrcZffrZ_heldZ_qtyZ_shaper_seenZ_ftZcsrrrZfullZcompr6)rrryrr7rFs   ,  $  .( & :  $  <&rF__main__)T)#__doc__rrrUrerrR collectionsrrSdirnameabspath__file__rWrTrVrrZ FIELD_DOCScompiler0r8rrDrjrpr{rrrr~rrFr^exitr6r6r6r7s8 u /0"F I