"""Queue-aware fill model: what happens once you stop ignoring the orders ahead?

WHY. Every fill number this project has produced -- historical replays, the live
paper maker, the 0-5c price ladder -- uses the same test: "did any trade print at
or above our ask?" That ignores QUEUE POSITION. In a FIFO book, a buyer lifting
the ask takes the size already resting there first. With 1,749-44,427 contracts
ahead of us at the touch (measured on Kalshi, 2026-07-28 onward), a print at our
price is very often somebody else's fill, not ours.

The bias is DIRECTIONAL, which is what makes it dangerous rather than merely
imprecise:
  - at +0c we join the touch, so the queue ahead is huge -> fills badly inflated
  - at +3c/+5c we are the only order at that price -> fills roughly honest
So the measured fill curve (steep decline from 100%) is flattered at the bottom
and fair at the top. The TRUE curve must be flatter, which changes the answer to
"should we charge more" and inflates the concede-at-T-60 policy specifically,
since conceding puts us exactly where the inflation is worst.

THE MODEL. When we post at price P with Q contracts already resting at P:
    filled once cumulative volume of trades at price >= P (after we post)
    exceeds Q.
Trades at a HIGHER price than ours mean the book walked up through our level, so
they count too.

Q is taken as the depth at the touch at POST time when we join the touch
(our_ask <= market ask), and 0 when we post above the touch (we are first in line
at a price nobody else is showing).

KNOWN LIMITATIONS, stated rather than buried:
  - Q is pessimistic: makers ahead of us can CANCEL, shrinking the real queue.
    So this is a lower bound on fills, as the optimistic model is an upper bound.
    The truth is between them, and the gap is the honest uncertainty.
  - We only have depth at the BEST ask, not the full ladder. For orders above the
    touch we assume nothing is queued at our level, which is optimistic there.
  - Depth logging only began 2026-07-28 17:17Z, so earlier legs cannot be scored.

Usage:  python research\\queue_aware_fill.py [YYYY-MM-DD]
"""
import csv
import datetime as dt
import sys
from collections import defaultdict

sys.path.insert(0, r'E:\soccer\main')
sys.path.insert(0, r'E:\soccer\research')
import mm_settle as S
import kalshi_trader as kt

QUOTES = r'E:\soccer\research\mm_paper_quotes.csv'


def trades_sized(ticker, since, cap=40000):
    """[(ts, price, count)] after `since`, paginated back far enough."""
    out, cursor = [], None
    while len(out) < cap:
        p = {'ticker': ticker, 'limit': 200}
        if cursor:
            p['cursor'] = cursor
        try:
            d = kt._get('/markets/trades', p)
        except Exception:
            break
        tr = d.get('trades', [])
        out += tr
        cursor = d.get('cursor')
        if not cursor or not tr:
            break
        oldest = min((S._pt(x.get('created_time')) for x in tr
                      if S._pt(x.get('created_time'))), default=None)
        if oldest and since and oldest <= since:
            break
    res = []
    for x in out:
        t = S._pt(x.get('created_time'))
        if not t or (since and t <= since):
            continue
        try:
            # Kalshi names trade size `count_fp`, NOT `count`. Using the wrong
            # key silently yields size 0 for every trade, so no cumulative volume
            # ever exceeds the queue and the model reports 0% fills everywhere --
            # which is how this bug announced itself (0 fills even where the
            # queue was 0, an arithmetic impossibility).
            res.append((t, float(x.get('yes_price_dollars') or 0),
                        float(x.get('count_fp') or x.get('count') or 0)))
        except (TypeError, ValueError):
            continue
    res.sort(key=lambda z: z[0])
    return res


def main():
    day = sys.argv[1] if len(sys.argv) > 1 else '2026-07-29'
    rows = defaultdict(list)
    with open(QUOTES, encoding='utf-8') as fh:
        for r in csv.DictReader(fh):
            if r['ts'][:10] == day and r.get('our_ask') and r.get('k_ask'):
                rows[r['leg_ticker']].append(r)
    print(f'{day}: {len(rows)} Kalshi legs with quotes\n')

    per_arm = defaultdict(lambda: {'n': 0, 'opt': 0, 'q': 0, 'queues': []})
    for tk, rs in rows.items():
        rs.sort(key=lambda x: x['ts'])
        r0 = rs[0]
        if not (r0.get('ask_size') or '').strip():
            continue                      # before depth logging began
        start, ko = S._pt(r0['ts']), S._pt(r0['kickoff'])
        our = float(r0['our_ask'])
        mkt = float(r0['k_ask'])
        arm = round(our * 100 - mkt * 100)
        if arm < 0:
            continue
        queue = float(r0['ask_size']) if our <= mkt + 1e-9 else 0.0

        tr = [(t, p, c) for t, p, c in trades_sized(tk, start) if t < ko]
        optimistic = any(p >= our - 1e-9 for _, p, _ in tr)
        cum = 0.0
        queued_fill = False
        for _, p, c in tr:
            if p >= our - 1e-9:
                cum += c
                if cum > queue:
                    queued_fill = True
                    break

        a = per_arm[arm]
        a['n'] += 1
        a['opt'] += optimistic
        a['q'] += queued_fill
        a['queues'].append(queue)

    print('FILL RATE: optimistic (ignores queue) vs queue-aware')
    print(f"  {'arm':>5s} {'legs':>5s} {'optimistic':>11s} {'queue-aware':>12s} "
          f"{'median queue':>13s}")
    tn = to = tq = 0
    for arm in sorted(per_arm):
        a = per_arm[arm]
        n = a['n']
        med = sorted(a['queues'])[n // 2] if n else 0
        print(f"  +{arm}c {n:5d} {a['opt']}/{n} {a['opt']/n*100:6.0f}% "
              f"{a['q']}/{n} {a['q']/n*100:7.0f}% {med:12.0f}")
        tn += n
        to += a['opt']
        tq += a['q']
    if tn:
        print(f"\n  TOTAL {tn:3d} legs: optimistic {to}/{tn} ({to/tn*100:.0f}%)  "
              f"queue-aware {tq}/{tn} ({tq/tn*100:.0f}%)")
        print(f"  => the optimistic model overstates fills by "
              f"{(to-tq)/tn*100:.0f} percentage points")
    print('\n  Optimistic is an UPPER bound; queue-aware is a LOWER bound (it')
    print('  assumes nobody ahead of us cancels). Truth lies between.')


if __name__ == '__main__':
    main()