# Project Memory - Options Trading Pipeline

## Project Location
- **Code**: `E:/options_code/`
- **Data**: `E:/options_data/` (821 stocks, JSON options chains, organized by date e.g. `2022-01-03/AAPL.json`)
- **Processed DFs**: `E:/options_dfs/calls/` and `E:/options_dfs/puts/`

## Active Scripts
| File | Purpose |
|------|---------|
| `monthly_recommendations.py` | C2 strategy scanner — emails HTML report |
| `weekly_update.py` | Daily sync + process (no suggest/email) |
| `run_monthly_recs.ps1` | Task Scheduler wrapper (last-day-of-month check) |
| `sync_options_data.py` | Google Drive → local data sync |
| `process_data_fast.py` | Processes raw JSON into CSVs |
| `config.py` | Centralized config |
| `models.py` | Model classes and exceptions |
| `utils.py` | Shared utilities |
| `etrade.py` | E*Trade API integration |
| `backtest_options.py` | Backtester class |
| `backtest_cli.py` | CLI for backtests |
| `master_run.py` / `master_run_fast.py` | Pipeline runners |
| `quantile_regression.py` | Quantile regression (LightGBM) |
| `multi_stock_pipeline.py` | Multi-stock pipeline |
| `options.py` / `options_pipeline_enhanced.py` | Original pipelines |
| `random_forest_model.py` / `regime_detection.py` / `tft_model.py` | Model modules |

## Task Scheduler Jobs
- `\OptionsDailySync` — Mon–Fri 4:30 PM, runs `weekly_update.py`
- `\OptionsMonthlyRecs` — Daily 5 PM, fires `monthly_recommendations.py` only on last day of month

## Active Strategy: C2
See `memory/project_c2_strategy.md` for full details.

Signal: stock down 20–40% in 20d + still falling (<−5% over 5d) + 30–50% below 52wk high.
Buy near-money call (0–8% OTM), DTE 21–60, ask ≤5% of underlying.
Backtest PF 2.73, Sharpe 0.83, ~8 trades/month. OOS validated PASS.

## Soccer/Betting Pipeline
- [Full pipeline notes](project_soccer_pipeline.md) — Strategy, all files, league coverage, live_monitor behavior, task scheduler jobs
- [File storage locations](feedback_file_locations.md) — All files in E:\soccer (consolidated from football_analytics on 2026-04-12)
- [Execution platform cascade](project_execution_plan.md) — B365 first, cascade soft books, Kalshi as final fallback
- [Strategy C research](project_strategy_C_research.md) — Backtested home-pick strategy via highest B365 CLV (~54% WR, +6-10% ROI). Not yet live, to revisit.
- [Both-sides-fire dedupe](feedback_dedupe_both_sides.md) — When HOME+AWAY both fire on same match, keep only higher-CLV side (implemented 2026-05-23)
- [Manual execution gap](project_manual_execution_gap.md) — User hesitates placing all signals; expect real P&L to lag tracked P&L. Auto-trade plan is the fix.
- [Kalshi auto-trade rollout](project_kalshi_autotrade.md) — Phase 1 (auth) DONE 2026-05-25. Phase 2 (order placement) is next.
- [Kalshi volume vs edge study](project_kalshi_volume_edge_study.md) — Open question: does high Kalshi volume erode CLV edge? EPL outlier; inconclusive at n=19. Re-check at n>=40.
- [Team-map silent-miss diagnosis](feedback_team_map_silent_miss.md) — Persistent `Kalshi=?A/?H` despite a real orderbook means a team is MISSING or has a WRONG abbr in team_map.json (kalshi_live falls through and 404s the per-market call).
- [Rescheduled-game orphans](feedback_rescheduled_game_orphans.md) — Reschedules leave PRE-only orphan entries under the original date; the new-date record carries the result. NEVER fill an orphan ftr (double-counts in the tracker) — check later dates for the matchup first.
- [Ride out drawdowns](feedback_ride_out_variance.md) — On cold streaks the user wants to hold the line, not pause; don't reflexively offer the kill switch — reinforce rule-based discipline.
- [Restart live_monitor after config edits](feedback_restart_live_monitor_after_config.md) — Editing AUTO_TRADE_* / place_auto_order does NOTHING until live_monitor.py is restarted (it's the 24/7 primary auto-trade path, loads daily_signals once at startup).
- [No sub-ask limit orders](project_no_subask_limits.md) — Sub-ask limits are adverse-selection traps (fill 38%, miss 71% of winners); take liquidity AT the ask. Reverses Phase 4 of auto-trade plan.
- [League filtering buys little](project_league_filter_study.md) — Win-rate gates & warmup-skips cut on noise (25% is natural WR); only rolling-ROI gate w/ >=8 warmup helps, marginally & in-sample. Revisit at 40+ bets/league.
- [Highest-CLV-pick research shelved](project_highest_clv_pick_research.md) — research.ipynb side-picking variants; debugged 4 bugs (net returns, pin_fair-as-p_win, Kelly numerator, TARGET_MINS sign-flip) but don't beat live Kalshi. Don't reopen.
- [Round-robin/parlay study](project_round_robin_study.md) — 2/3-leg round robins look +ROI but it's a few clustered longshot days (54% from 1 day); non-independent, unproven edge, not tradeable on Kalshi. Don't pursue.
- [Strategy-improvement probes](project_strategy_improvement_probes.md) — 5 probes: edge is Kalshi-SPECIFIC (B365 confirmation anti-correlates — never require it); Kelly under-bets longshots; sub-band/favorite/spread mostly in-sample noise.
- [Edge significance status](project_edge_significance_status.md) — Core edge UNPROVEN & weakening (p 0.056→0.115 as n 365→394); ROI still + everywhere. OOS test killed the gate. Don't add rules; collect data. Don't cheerlead OR capitulate.
- [Draw outcome tested](project_draw_outcome_test.md) — Draws have NO CLV edge (ROI -1.8%/p=0.56, OOS TEST slice NEGATIVE). Edge is home/away-specific. Stay home/away only; don't add draws.
- [No World Cup](project_no_world_cup.md) — Decided (and reaffirmed 2026-07-04): no WC tracking or betting. Neutral venues break the home/away edge, max-liquidity market, would contaminate the frozen xlow forward test. Applies to all international tournaments.
- [Cash-out rules tested](project_cashout_study.md) — Every in-game exit (HT always/if-trailing, 75', sell-when-up-2) tests <= hold on 453 bets, aggregate AND per-league; in-game bids run ~3pp under realized value. USL/CanPL "profitable" cells are small-n lotteries. Hold to settlement, no exceptions. Closed 2026-07-08.
- [Kalshi-only execution (geo-restriction)](project_kalshi_only_execution.md) — Poly geo-restricted: reroute ALL xlow signals to Kalshi execution (don't skip Poly-cheap ones). Full n=286 kept, only ~1-2pp ROI cost (+17.8% fee-adj net, still z+2.4/p=.008). Skipping instead halves the sample for no benefit.
- [K/P gap size (no band)](project_kp_gap_size.md) — Raw Kalshi-vs-Poly CLV gap size, unconditional (no [5,12) band gate), is NOT predictive: non-monotonic buckets, OOS sign-flips at every threshold, "significant" cell is 8 longshot bets. Confirms the strict-xlow band conditions do real selection work, not an arbitrary cutoff.
- [xlow + momentum combo tested](project_xlow_momentum_combo_tested.md) — Applying the momentum drift filter to xlow is DOMINATED: cuts n 286->239 for worse ROI (24.7%->23.6%) and z. Discarded bets still +13.4% alone. Don't combine, closed.
- [xlow TEST-segment decomposition](project_xlow_test_segment_decomp.md) — The backtest's +48u last-40% is one hot mid-May week (top-3 days = 74%); driver is the Poly-cheap leg's WR 26%→43%. True edge plausibly ~+10%/bet; median week flat. Weight WR-gap over ROI at the checkpoint; don't re-tune rules off this.
- [Pinnacle anchor VERIFIED calibrated](project_pin_calibration_verified.md) — Pooled 1,202 fixtures: implied 25.0% draws / 44.3% homes vs actual 25.5% / 44.5% (z+0.45/+0.12). The assumption the whole strategy rests on, tested for the first time and sound. No league structurally mis-calibrated; per-league deviations at n=30-50 are randomly signed = noise. Re-run research/pin_calibration_by_league.py when a league "looks broken" — a |z|>2 flag at n>=100 is the ONE legitimate reason to drop a league.
- [South America anomaly REFUTED on 11,746 games](project_south_america_calibration.md) — A 73-fixture sample said SA draws run 16.4% vs 27.4% implied; 11,746 Brazil+Argentina games with Pinnacle CLOSING odds say 28.7% vs 28.6% (z+0.20, perfect). Homes +0.97pp only, and the Mexico/USA control matches. Both trade ideas dead (fade-draw -0.1%, buy-home +2.2% vs a 3.3% overround). Keep as THE worked example of a small-sample mirage that passed sign-consistency AND a drop-the-motivating-league control and was still noise. Byproduct: the 20-25% implied bucket runs ~5% below implied and that is exactly where xlow bets — unverified on European leagues. Run research/sa_history_calibration.py.
- [Non-price features AND Elo add NOTHING to the closing line](project_nonprice_features_dead.md) — Two attempts. Rolling form (8,704 fixtures) and proper Elo+rest-days (31,963 fixtures, walk-forward). Market log loss 0.98501; market+Elo 0.98470 looks like a win but paired t=+0.71 = noise. A betting sim showed +2.7% ROI — the no-model CONTROL shows betting favourites alone gives +1.89%, so it was a favourite-longshot/devig artifact, and at devigged prices nobody can bet anyway (3.3% overround). RULE: when a model shows betting profit but a log-loss null, run the no-model control before believing it. THIRD attempt (full_model_test.py): 43 features x 12 model configs x 31,963 fixtures — 0 met a pre-set t>1.96. Every increase in model CAPACITY made it significantly worse (GBM d6 t=-5.71), the signature of no signal. Do NOT re-run with "more features".
- [Price-derived feature space EXHAUSTED](project_edge_mine_exhausted.md) — 57 slices + a grouped-CV gradient booster vs net=close_mid-entry_ask. EVERY slice negative (best -0.43c). The model WORKS (OOF corr +0.15, top decile moves +0.44c) but corr(predicted movement, half-spread)=+0.143 — the edge sits exactly where the toll is highest. Best config anywhere -0.418c. LESSON: edge and cost are the same variable, not independent; the null is structural. Mine against NET CENTS, never win rate. Run research/edge_mine.py.
- ["Wisdom of the Crowd" tested](project_wisdom_of_crowd_tested.md) — football-data.co.uk's method on ~48k fixtures. B365 ALONE has no edge at any margin (0-8%, all t~0). Best-of-40-books gives +1.39% t=+1.21, not significant — so the value is LINE SHOPPING, not soft books. A first run showed +6-7% t=+5-8: that was LOOK-AHEAD (book OPENING vs Pinnacle CLOSING; the "C" infix = closing — always match open-to-open or close-to-close). Key insight: bookmakers have NO spread to cross, exchanges do — same signal, opposite sign, purely by venue.
- [Paper market maker: THREE bugs inflated every result on 2026-07-26](project_mm_paper_implementation.md) — main/mm_quoter.py (every 5 min) + mm_settle.py (daily 08:45), paper only. State only tracked competitive legs (completion rule fired 0x), timestamps compared as raw STRINGS (dropped ~all trades), no kickoff cutoff (counted IN-PLAY trades as fills, 83% vs real 13-19%) — the earlier "8-for-8 complete books are riskless arbitrage" claim came from bug #3 and is FALSE, do not cite it. The independent historical finding (project_market_making_early_edge, +30% opening decile) WAS re-derived from scratch with sanity checks and DOES replicate exactly — that one is real. Corrected quoting (cap ask at market's own ask, floor at fair) raised fills to 61%, but corrected backtest replay shows COMPLETE books earn ~$0 at that pricing (no overround left) — the historical edge may require being FIRST into a wide book, not joining an already-tightened one. That is now the open question, on top of capture.
- [Market making: the edge is ONLY in a market's first ~30% of volume](project_market_making_early_edge.md) — Maker P&L by volume decile, 209 events/$16.9M: 0-10% +25.8%, 10-20% +17.7%, 20-30% +13.1%, then -0.2% to -4.2%. Monotone. Mature market making LOSES; early liquidity provision pays. Makers pay $0 fees vs takers' 4.4% of stake. BUT aggregate shrank +7.9%->+2.3% as n went 40->205 and the median event is still negative, so only the DECILE STRUCTURE is trustworthy. Pinnacle-conditioned sizing is UNTESTED (my analysis was broken, not the idea). Watch out: taker_book_side is the TAKER's side; use cents/contract not ROI. Run research/mm_feasibility.py then mm_analyse.py.
- [xlow VERDICT: edge is real, smaller than the spread](project_xlow_spread_verdict.md) — THE conclusion (2026-07-25). Line moves +0.31c (95% CI [-0.05,+0.67]) vs a 0.67c half-spread = net -0.36c/bet; tight-spread bets net -0.02c. Best case the data supports is break-even before fees. Implies a ~1.4% edge, not the backtest's +24.7% — which explains the flat forward test and every failed hunt. Selection is fine; execution eats it. CORRECTION same day: "wait for tighter spreads" is IMPOSSIBLE — all three venues price on a 1-CENT grid, so the minimum half-spread is 0.5c and 77% of Kalshi quotes are ALREADY there. Against a 0.31c edge that is -0.19c at the best execution the venue structurally permits. The tick is the binding constraint, not liquidity. Do NOT re-enable pending "spread improvement" — there is none available. Only a >0.5c edge, a sub-cent-tick venue, or MAKING instead of taking. FEE GROUND TRUTH (100 real fills): taker = 0.07*P*(1-P)/contract = 4.39% of stake; MAKER = $0. So true taking cost was ~1.7c (0.67c spread + ~1c fee), not 0.67c. Market making is the structurally correct side (earn spread, pay no fee, median 175k contracts/market) BUT the spread sits at the 1c tick floor 77% of the time = already competed away; infrastructure business, not a script. Cite before re-running any xlow analysis.
- [Pinnacle has NO measurable edge over the other books](project_book_sharpness_compare.md) — Paired log-loss, same-snapshot, n=1218: vs B365 t+0.46, vs Kalshi t-0.07, vs Poly t-1.06. All null; the anchor is a convention, not a proven sharper book. BUT resolves only ~3pp errors. The band premise test looked positive pooled (+12.2%) — 80% of it was IN-SAMPLE; split at the freeze date it's IN +19.7% / OOS -18.7%. But that gap is z=+1.45 on Kalshi = within noise, so the premise test is NO evidence either way, not evidence of overfitting. Always split at FREEZE_DATE before quoting, and test the gap before calling it overfitting. Run research/book_accuracy_compare.py.
- [pin_fair >= 0.15 EXECUTION-only floor](project_pin_fair_execution_floor.md) — User instruction 2026-07-25: no real money under 15% Pinnacle fair. Look-ahead-derived ON PURPOSE and confined to order placement — signals still fire and still count unfiltered in the forward test. NEVER apply this threshold in research/. Lives in daily_signals.pin_fair_ok(), shared by both placers; needs a live_monitor restart.
- [xlow decision rule (power-corrected)](project_xlow_rejection_criteria.md) — Per-bet SD 1.99u means n=100 only detects edges >=+32.7%, so it could NEVER confirm the +17.8% backtest. Real decision point moved to n=1000 (detects +10.3%); n>=100 kept only as a disaster tripwire. ~6 weeks at current 25/day fire rate, ~$100-150 fee cost if edge is zero. Cite when asked "is it dead."
- [t60 variant (passive)](project_t60_variant.md) — One-book-in-band at T-60min pre-KO: backtest +45.3% z+2.02 vs +15.1% same-subset xlow control. Tracked passively as `t60` column; decision only at the n=1000 review (see power-corrected decision rule), don't promote mid-test.
- [xlow sizing: flat beats Kelly](project_xlow_flat_vs_kelly.md) — Flat ~1% units outperform 0.25-Kelly ($1,619 vs $1,141 end bankroll on the 286-bet stream); Kelly refuses 88 still-profitable bets (f<=0 at Kalshi ask) and under-stakes longshots. Bet flat, don't skip on Kelly f<=0.
- [Pure Kalshi-vs-Poly deviation](project_pure_xbook_deviation.md) — Dropping Pinnacle and trading the two books against each other (bet cheaper) is flat-to-NEGATIVE (>=5% dev -23% ROI), worsens as deviation widens, falsification leg also loses; pin_fair signal on same 975 fixtures is +11%. The edge NEEDS the sharp Pinnacle anchor. Don't build a two-book-spread strategy.

## Data Quality
- 378/821 stocks pass quality filter (exclude >25% single-day underlying price change)
- Bad data was root cause of phantom +$3M profit — fixed by price change filter

## Environment
- Platform: Windows 10, WSL available
- Python: `C:\Users\lirse\AppData\Local\Programs\Python\Python39\python.exe`
- Deps: pandas, numpy, scikit-learn, xgboost, lightgbm, joblib, yfinance
- `.env`: Gmail SMTP credentials, Google Drive API key
- [Kalshi drift toward Pinnacle fair](project_kalshi_drift_to_fair.md) - no general drift (t~0, MDE 0.11c); +0.26c only in the xlow band, matching history
- [MM gross-margin gate](project_mm_gross_margin_gate.md) - book P&L = sum of leg margins; gate threshold depends on the natural-completion rate; kalshi_backfill needs 2 filters
- [MM charge-more A/B](project_mm_charge_more_ab.md) - live paper test of posting 1c above the ask; replays cannot answer it (queue priority)
- [Pinnacle line staleness](project_pin_line_staleness.md) - our "Pinnacle" is an API-Football relay updating every ~5.8h; xlow loses -42% on stale lines, wins on fresh; age gate + real-time feed queued
- [MM logging fields](project_mm_logging_fields.md) - queue depth, market age, OI/volume, NO side, league-local hour, fill timing; what is backfillable and what is not
- [Pinnacle coverage + consensus fallback](project_pin_coverage_and_consensus.md) - ~6-8% of fixtures have no Pinnacle line; consensus tracks it to 0.52c but degrades where books thin
- [Verify coverage claims](feedback_verify_coverage_claims.md) - never assert "venue lacks X" from a failed lookup; say UNKNOWN and ask the user to check
- [MM queue reality check](project_mm_queue_reality_check.md) - with queue position modelled, the paper maker LOSES on 07-29 under every policy; supersedes the +$2.26 projection
- [Season rollover checklist (every August)](project_season_rollover_checklist.md) - four SILENT breakages at each season change: football-data serving the WRONG league's CSV (E0 held National League rows for 5 days; the downloader now validates the Div column), fdco RENAMING clubs that never moved, stale `div` fields, and unmapped promotions. Never guess a Kalshi abbr — harvest it. Six divisions still return HTTP 300 with no 2627 file at all.
- [Single-machine consolidation + task inventory](project_single_machine_consolidation.md) - 2026-08-20: the pipeline was running on TWO machines for days (same Kalshi account, same API key, same Pages repo, divergent ledgers). Now MINI only. Full scheduled-task table, the persistent HKLM `E:` mapping that all 44 scripts depend on, the two one-off reminder dates, and the symptom that exposes a second publisher (page updates but your code changes don't land).
- [Fill-prediction ML: book volume predicts completion, min-leg volume does NOT](project_mm_fill_prediction_ml.md) - grouped-CV models on live maker data: book completion AUC 0.756, leg fill AUC 0.673. WHOLE-BOOK volume is the top feature (+0.091); the min-leg-volume gate from the paper era fails to replicate live (rho +0.095, p=0.189) and live selection was corrected to book volume. Queue depth is near-worthless as a selection feature. Capacity HURTS in both models = simple signal. Re-run 2026-08-29 (task SoccerMMFillMLRerun).
- [MM in-play verdict: needs ~89% completion, so no](project_mm_inplay_verdict.md) - opening books after kickoff is measured dead: in-play jumps average 15.6c and persist 63% of the time vs a ~2c completed book, so break-even needs ~89% completion (best ever observed is 82.6%, pre-match). The tempting 5-10x thinner in-play queue is thin BECAUSE other makers pulled. Byproduct: 30% of legs sit in the 24h+ window at a 6.8% fill rate.
- [MM ratchet vs chase — do NOT walk stranded legs down](project_mm_ratchet_vs_chase.md) - 259 books, BOTH fill bounds agree: chasing the touch down loses (-2.76c/-1.26c per book) because it completes at a median 99c against a 1.00 payout; decay is adversely selected. Ratchet stays. The lever that pays is the ENTRY GATE (gate+ratchet best on both bounds), not conceding on stranded books.
- [MM completion arithmetic](project_mm_completion_arithmetic.md) - the maker reduces to ONE number: completion rate, break-even 61.4%. Complete book = near-deterministic +1.78c (SD 1.69, t=+8.9); partial book = coin flip at 25x the SD. Completion replicates across a time split (69.2%/68.8% at the v4 gate), realised P&L does NOT (-3.15c/+5.13c) and needs ~841 books to resolve. ALWAYS estimate EV as p*1.78+(1-p)*(-2.82), never from realised c/book. The v4 cap at 500 rests on 4 partial books and is probably too low - payout is flat in volume while completion climbs to 82.6%. Weak link: the -2.82c partial cost has SE +/-4.8c, which swings break-even from ~0% to ~81%.
- [SYSTEM processes are invisible to a non-elevated check](feedback_system_process_invisible.md) - a non-elevated `Win32_Process` query returns a BLANK CommandLine for SYSTEM-owned processes, so any "is live_monitor/maker already running?" match silently says "down" and you launch a DUPLICATE (happened 2026-08-23 after the power cut; ~90s of two live_monitors, no data damage). Cross-check with lock files, log mtimes, or the guard logs — and prefer letting the 5-15min guards do the restart. ALSO: you cannot KILL them non-elevated (Access denied), so a live_monitor config change (AUTO_TRADE_*, stakes) CANNOT be made active without the user running an elevated restart - say so up front, don't report the change as done.
- [xlow forward scope = backtest leagues](project_xlow_backtest_league_slice.md) - DECISION 2026-08-23: the PRIMARY forward number is now the 20 leagues the backtest ran on (n=237, +18.75%, z+1.37), not the pooled 442 (+2.46%). The added leagues were switched on deliberately to keep collecting while the backtest set was out of season (D1 has ZERO post-freeze fixtures), so they are tracked separately and never pooled. I first called this post-hoc selection and was WRONG - whether a slice was chosen before or after seeing data is a question about intent, ASK. Caveats are about power, not selection: gap p=0.057, threshold non-monotonic, per-league ROI anti-correlates (Spearman -0.370) so do NOT filter further within the scope. TRAP: live_data has NO Poly columns before 2026-07, so an in-sample xlow computed from it fires 7 times and is meaningless - the real backtest is poly_backfill/. SCOPE EDITED 2026-08-23: BSA+CL1 REMOVED (user distrust) and NWSL/PL1/USL added -> n=212 +34.46% z+2.55, but that IS post-hoc (they were the two worst leagues, removals alone explain the jump, z crossing 1.96 is manufactured) - SCOPE_FROZEN='2026-08-23', everything before it is in-sample. PL1/USL are now IN the primary via fire_xlow_primary(), which substitutes the POLY-US confirm where Poly.com has no coverage -> n=283 +26.02% z+2.31, decomposing as poly.com n=212 +34.5% / poly-US n=71 +0.8%. The Poly-US confirm is UNVALIDATED (backtest used Poly.com only).
- [xlow and the maker share Kalshi positions](feedback_xlow_maker_position_netting.md) - 2026-08-23: xlow buys the SAME tickers the maker quotes, and Kalshi nets both into one position, so our -3 alongside xlow's +5 reads +2. THREE paths in mm_maker_live read that netted position as maker fill quantity (a position fallback, `held = abs(positions[tkr])`, and no recovery) and booked phantom fills at $0 - which made the budget rule demand the whole dollar and the ratchet latch 60c/94c floors. RACBOC was cancelled by hand; ROMFIO stopped quoting a leg entirely. Fixed by order_id attribution + a self-heal. RULE: a position is NEVER evidence of a maker fill; attribute by identifier or not at all. ACCEPTED RISK (user, 2026-08-24, do not re-raise): the two also TRADE against each other - the maker sold 3 of xlow's own 5 FIO contracts at the same 18c, so the bet is really 2 contracts while the ledger says 5. Left as is.