---
name: Soccer betting pipeline — architecture and strategy notes
description: Full picture of the soccer CLV/Kalshi betting system: files, strategy, data sources, recent changes
type: project
---

> **2026-06-06 folder reorg:** all live code moved from `E:\soccer\` root into `E:\soccer\main\`. Any bare script name below (e.g. `daily_signals.py`) now lives at `E:\soccer\main\<name>`. Live-state JSON/CSV/logs + data dirs stay at root; research → `research/`, retired → `misc/`. See feedback_file_locations.md for the full map.

## Strategy
Two-signal CLV strategy using Pinnacle closing odds as fair value benchmark:
- **Away CLV 5% to 12%**: bet Away when CLV in [5%, 12%) — upper cap added 2026-05-23 (≥12% AWAY was -ROI in out-of-sample audit; long-shot trap)
- **Home CLV 5% to 12%**: bet Home when CLV in [5%, 12%) — lowered from 8% on 2026-05-23 (5% threshold +31.9% ROI on 47 bets vs +24.3% on 21 at 8%); 12% upper cap added same day after Kalshi-CLV bucket audit showed 14 historical bets above 12% returned -41% ROI / -$50 (the 13-15% bucket was 0/3, suggesting the long-shot cliff hits HOME too)
- Draws excluded (no edge historically)
- Pinnacle fair = remove overround: `p_fair = (1/odds) / (1/h + 1/d + 1/a)`
- **B365 CLV formula**: `(b365_odds - pin_fair_odds) / pin_fair_odds` — positive = B365 better than fair
- **Kalshi CLV formula**: `(pin_fair_prob - kalshi_mid) / pin_fair_prob` — positive = Kalshi underpricing = edge
- Kelly sizing: `f = (p*b - (1-p)) / b`, stake = `bankroll * f * 0.25`, capped at 25%
- **Kalshi sizing rule**: mid for CLV comparison, ask for Kelly sizing; falls back to mid if ask missing
- **Kelly floor**: `max(stake, $1.00)` when f > 0; skip bet only when f ≤ 0 (no edge at ask)
- Execution: B365 first, cascade to soft books as accounts get limited, Kalshi as parallel track (not just fallback)
- Kalshi tracks Home YES and Away YES; NO bets logic exists in live_monitor but suppressed pending validation
- Focus leagues: top tier + 2nd tier **when available on Kalshi** (see coverage below)

### Bankrolls
- B365 bankroll: `bankroll.json` (started $1,000, currently ~$1,048.11 as of 2026-05-03)
- Kalshi bankroll: `kalshi_bankroll.json` (started $1,000 on ~2026-04-25, currently ~$1,104.87 as of 2026-05-03)
- Both shown in CLV alert emails and weekly summary

### Backtest results (2021–2025, Kelly 25%, start $1k)
| Signal | Bets/yr | WR | AvgOdds | CAGR | Sharpe |
|---|---|---|---|---|---|
| Away CLV >5% | ~630 | 31% | 4.78 | +224% | 1.27 |
| Home CLV >8% | ~305 | 40% | 3.77 | +251% | 2.10 |
| Combined | ~935 | — | — | higher | — |

RF+CLV model (Home, p>=0.30 + CLV >5%): Sharpe 3.63, 52% WR — exists in `rf_clv_model.py`, not yet wired into live pipeline.

## Key Files (all in E:\soccer)
| File | Purpose |
|------|---------|
| `live_monitor.py` | 24/7 monitor — polls scores, logs PRE snapshots, emails CLV alerts, detects HT |
| `daily_signals.py` | Core module: CLV scanner, `resolve_pending_bets()`, `scan_fixtures()`, `build_html()`. Not scheduled directly but imported by `resolve_bets.py` and runnable manually (`python daily_signals.py resolve`) |
| `daily_watchlist.py` | Midnight job — pulls ALL fixtures from API-Football for all tracked leagues, tags [K] where Kalshi available, writes watchlist.txt |
| `kalshi_live.py` | Kalshi API wrapper — prices, abbreviations, series lookup. `find_event_by_abrs(series, h_abr, a_abr)` (added 2026-05-19) searches the entire series for an event matching the two abbreviations regardless of date — automatic fallback in `get_all_prices` and `get_market_price` for rescheduled/abandoned-resumed games where API-Football's date no longer matches Kalshi's event ticker |
| `maps.py` | Shared loader for league_map.json + team_map.json (hot-reload capable) |
| `league_map.json` | 23 leagues: api_football_id, kalshi_series (null if not on Kalshi) |
| `team_map.json` | ~420 teams: API-Football name → fdco name, Kalshi abbreviation, aliases |
| `refresh_api_football.py` | Weekly pull of finished fixtures + stats from API-Football |
| `resolve_live_data.py` | Nightly — fills ftr/result_home/result_away in live_data CSVs via FT rows or API fallback |
| `watchlist.txt` | DATE= + pipe-delimited match entries for live_monitor |
| `live_data/YYYY-MM-DD.csv` | Per-day PRE + live snapshot log (elapsed negative = pre-match) |
| `clv_kalshi_log.csv` | Pre-match CLV check log (B365 vs Pinnacle fair vs Kalshi) |
| `bankroll.json` | B365 bankroll |
| `kalshi_bankroll.json` | Kalshi bankroll (separate tracker) |
| `pending_bets.json` | Open bets awaiting result |
| `bet_history.json` | Resolved bet history — B365 entries have `stake/pnl/bankroll_after`; Kalshi entries have `kalshi_ask/kalshi_stake/kalshi_pnl/kalshi_bankroll_after` |
| `weekly_bet_summary.py` | Emails previous Mon–Sun bets for both B365 and Kalshi (runs Tuesdays 8AM) |
| `recompute_kalshi_min1.py` | Utility — rebuilds all Kalshi bet_history entries from live_data CSVs using ask-based Kelly |
| `resolve_bets.py` | Thin wrapper around `resolve_pending_bets()` from `daily_signals.py` — called by `SoccerResolveNightly` Task Scheduler job |

## Shared Config System (maps.py) — added 2026-04-18
All team/league name translations live in two JSON files — no more hardcoded dicts in scripts.
- `league_map.json`: div code → name, api_football_id, kalshi_series, **optional `season` override**
- `team_map.json`: API-Football name (key) → {div, fdco, kalshi, aliases[]}
- `maps.py`: loads both files at import, exposes `fdco_name()`, `kalshi_abr()`, `league_api_id()`, `league_kalshi_series()`, `league_season()`, `reload()`, `changed()`
- **Hot-reload**: `live_monitor.py` calls `maps.changed()` every poll cycle — edit either JSON and the monitor picks it up within 60 seconds, no restart needed
- `kalshi_live.py` also uses maps; reloads on next run
- **NOTE**: Python module code (`kalshi_live.py`, `maps.py` logic) does NOT hot-reload — only the JSON data files do. After editing Python code, restart `live_monitor.py`.

## daily_watchlist.py — added 2026-04-19
Runs at midnight via Task Scheduler. Queries Kalshi for all tomorrow's matches across all series, fetches fixture IDs + real kickoff times (ET) from API-Football, writes watchlist.txt.
- `TEAM_ALIASES` dict handles known Kalshi→API-Football name mismatches (PSG, Bayern Munich, Mögladbach, Alaves, etc.)
- `_normalize()` strips umlauts/accents before fuzzy matching (ü→u, ö→o, etc.)
- `best_fixture_match()` applies aliases + normalization + substring matching in both directions
- If a match is missing a fixture ID in the watchlist, add an alias to `TEAM_ALIASES` in `daily_watchlist.py`
- live_monitor hot-reloads watchlist.txt on save — no restart needed when watchlist updates

## League Coverage
**Kalshi-enabled (21 leagues)** — all have full fdco (where applicable) + Kalshi abbreviations:
E0, E1, E2, SP1, SP2, I1, I2, D1, D2, F1, N1, B1, P1, T1, G1, SC0, MLS, BSA, USL, SE1, NO1, AR1

**Added 2026-05-20**: AR1 (Liga Profesional Argentina, AF id 128, series `KXARGPREMDIVGAME`, season 2026). 30 teams with abbreviations extracted from 268 historical Kalshi events. Kalshi has zero open per-game events at the moment — Clausura starts ~July; revisit then to verify events populate. Note: AR1 splits the year into two championships (Apertura Feb–May, Clausura Jul–Dec) with a zone-based round-robin + 16-team knockout playoff. Playoff games may settle on "advance" rather than 90' result — verify per-event settlement when Kalshi lists them.

**fdco-only (5 leagues)** — Kalshi series is null, teams have fdco names for bet resolution:
E3, EC, F2, SC1, SC2, SC3

**Summer-active leagues with calendar-year season** (require per-league season override — see below):
- BSA (Brazilian Serie A) — March–December
- MLS — February–November
- USL Championship — March–October
- SE1 (Allsvenskan) — April–November
- NO1 (Eliteserien) — March–December
- AR1 (Liga Profesional) — Feb–May Apertura + Jul–Dec Clausura (one AF season per calendar year)

**Per-league season override**: `league_map.json` supports an optional `"season"` field; `maps.league_season(div, default)` returns it or falls back to the global default. `daily_signals.py` uses this when querying API-Football fixtures, so cross-year European leagues stay on the global default (2025/26 → 2025) while calendar-year leagues use 2026.

**Empirically-verified Kalshi abbreviations** — All BSA, USL, SE1, NO1, B1 abbreviations were queried from real Kalshi event tickers (not guessed). On 2026-05-19 a full audit of SE1/NO1/USL/BSA found and fixed:
- Coritiba (BSA): CTB → CFC (Kalshi changed mid-season)
- Sarpsborg 08 (NO1): S08 → SAR
- Ham-Kam (NO1): HAM → HAK

## resolve_bets.py — Nightly Email Report
After resolving bets, `resolve_bets.py` now sends an email if any bets were resolved:
- Subject: `[XW/YL] NIGHTLY RESULTS — YYYY-MM-DD | B365 $X,XXX | Kalshi $X,XXX`
- Body: same HTML table as daily_signals email (resolved results only, no new signals section)
- Only sends if `len(resolved) > 0`

## daily_signals.py — Email Fixes (2026-05-03)
- **P&L column**: now shows combined Kalshi P&L for Kalshi-only bets (was showing $0.00)
- **Subject line**: now shows `Kalshi $XX.XX` total when no B365 stakes; both if mixed
- **Ghost signals**: Kalshi-only pending bets no longer injected as reminder signals (had no B365 data → empty columns). Only pending bets with `b365` field are shown as reminders.
- **Bankroll footer**: shows both `B365 bankroll` and `Kalshi bankroll`

## resolve_pending_bets() Behavior (in daily_signals.py)
- Called by `resolve_bets.py` (nightly) and directly via `python daily_signals.py resolve`
- **Result lookup order**:
  1. `live_data/YYYY-MM-DD.csv` — checks FT/AET/PEN status rows, matches by home+away name (exact then fuzzy substring). Primary source; no API quota used.
  2. API-Football `/fixtures?id=` — fallback for same-day results not yet in live_data
  3. football-data.co.uk CSVs — final fallback for older results
- **Kalshi-only bets** (no `stake`/`b365` fields): B365 P&L = 0, B365 bankroll unchanged; Kalshi P&L computed normally
- **Kelly floor**: all bets with f > 0 are floored at $1 minimum stake

## Known Bugs Fixed (2026-05-03)

### live_monitor.py — `_log_pending_bets()`
1. **opp_key bug**: `AWAY YES` bets stored `kalshi_home_ask` instead of `kalshi_away_ask` (and vice versa for `HOME YES`). The Kelly stake was computed correctly (using the right ask from the row), but the stored `kalshi_ask` in pending_bets.json was wrong. This caused:
   - Wrong payout calculation at resolution time (resolver uses stored ask for win P&L)
   - Wrong ask displayed in emails
   - Fix: `side_key = 'home' if 'HOME' in side else 'away'`
2. **Kalshi Kelly used B365 bankroll**: `_kelly_stake()` always used `bankroll` (B365). Now has `use_kalshi=True` param that uses `kalshi_bankroll` instead.
   - Historical correction applied to all May 2 bets on 2026-05-03
   - All prior bets (Apr 25–May 1) used the old `send_clv_alert` which correctly used Kalshi bankroll — no correction needed

### Pending bet ask correction
- When the monitor is restarted with old code running, pending bets logged during that session have wrong asks
- Fix: reverse-engineer correct ask from stored `pin_fair` + `kalshi_stake` using: `b = (1-pf)/(pf-f)` where `f = stake/(KELLY_MULT * K_BANK)`, then `ask = 1/(b+1)`
- Applied manually to May 2 and May 3 pending bets

## live_monitor.py Behavior
- Reads `watchlist.txt` — hot-reloads on file save (no restart)
- **Pre-kickoff**: skips live polling until 60 min before kickoff
- **Pre-match snapshots**: logs hourly rows (status=PRE, elapsed=negative) from -1450 min to -60 min before kickoff, pulling B365+Pinnacle from `/odds` endpoint and Kalshi prices; CLV columns computed and stored on each row
- **CLV alerts**: after each PRE snapshot, if `clv_kalshi_away ≥ 5%` or `clv_b365_away ≥ 5%` (Away) or `≥ 8%` (Home), sends one email per match with odds, CLV, and Kelly stake — `clv_alerted` flag prevents duplicates
- **Kelly in alerts**: uses ask for sizing, falls back to mid if ask missing; shows `$X.XX` if f > 0 (floored at $1), `—` if f ≤ 0
- **Alert email header**: shows both B365 and Kalshi bankrolls
- **NO bet logic**: code exists (fires when opposite side mid ≥ 0.60) but suppressed with `if is_no: continue`
- **Live polling**: Kalshi fetched every 60s (POLL_SECS); API-Football scores batched every 180s (AF_POLL_SECS) across all live matches in one call (`/fixtures?ids=1-2-3`)
- **API quota fallback**: if remaining calls < AF_LOW_WATER (300), sets api_ok=False. Clock-based elapsed used for status estimate (1H/HT/2H/FT). Kalshi still fetched and logged every poll — data continues even with no API-Football access.
- **HT detection**: triggers `check_halftime()` when API returns status=HT (or clock window 46-62min when api_ok=False)
- **FT dedup**: only logs the first terminal (FT/AET/PEN) row per match — no duplicate rows after full-time; elapsed > 120 rows filtered out
- **fid lookup backoff**: 10-min cooldown between failed fixture ID lookups to avoid burning API quota
- **Auto-exit**: stops 30 min after all matches reach FT
- **UTF-8 encoding**: all CSV writes (live_data, clv_kalshi_log) use `encoding='utf-8'` to handle non-ASCII team names (e.g. Beşiktaş)
- Live Pinnacle/B365 odds are NOT available via API-Football live endpoint (only pre-match via `/odds`)
- **Runs 24/7** — watchlist is regenerated at midnight by daily_watchlist.py

## Task Scheduler Jobs (Windows)
| Task | Schedule | Script | Purpose |
|------|----------|--------|---------|
| `BettingSignals` | 3× daily (8AM/4PM/12AM UTC) | `run_daily_signals.bat` → `daily_signals.py` | Scans fixtures, identifies threshold-crossing bets, emails CLV alert. **Path was wrong (`E:\options_code\run_signals.bat`) until 2026-05-19 — fixed to `E:\soccer\run_daily_signals.bat`** |
| `SoccerDailyWatchlist` | Daily midnight | `daily_watchlist.py` | Regenerates watchlist.txt for the new day |
| `SoccerDailyWatchlist8AM` | Daily 8 AM | `daily_watchlist.py` | Re-runs watchlist to catch D1/D2 markets (open ~8-9 AM ET, kick off ~9:30 AM ET) |
| `SoccerResolveNightly` | Daily midnight | `resolve_bets.py` | Resolves previous day's bets, updates bankroll |
| `SoccerResolveLiveData` | Daily 12:05 AM | `resolve_live_data.py` | Fills ftr/result columns in live_data CSVs; API fallback for games with no FT rows |
| `SoccerRefreshAPIFootball` | Weekly | `refresh_api_football.py` | Pulls last 8 days fixtures, rebuilds match_features.csv |
| `SoccerRefreshData` | Weekly Mon 6 AM | `refresh_league_data.py` | Downloads latest result CSVs from football-data.co.uk |
| `SoccerWeeklySummary` | Tuesdays 8 AM | `weekly_bet_summary.py` | Emails Mon–Sun bet summary for both B365 and Kalshi |
| `SoccerWeeklyStrategyMetrics` | Tuesdays 8:15 AM | `main/weekly_strategy_metrics.py` | Emails forward-test metrics (xlow/f10/t60/unique/overlap, league breakdown, strict-xlow ledger, n>=100 progress). Interactive-only task (runs when logged in). Added 2026-07-19 |

- `live_monitor.py` runs 24/7 as a persistent process (not via Task Scheduler)

## API Keys / Limits
- API-Football key: hardcoded in live_monitor.py, daily_signals.py, resolve_live_data.py; 7,400 calls/day limit
- Kalshi: public REST API, no key needed for price reads
- football-data.co.uk: public CSV downloads, no key

## Data Sources
- **Odds (pre-match)**: API-Football `/odds` — has Pinnacle (ID=4) and B365 (ID=8)
- **Scores**: API-Football `/fixtures?id=`
- **Results (bet resolution)**: football-data.co.uk CSV per season/div
- **Live exchange prices**: Kalshi REST API (`/trade-api/v2/markets/{ticker}`)

## Novig — third exchange leg (dormant, 2026-07-04)
`main/novig_live.py` skeleton exists (read-only, same get_all_prices shape as kalshi/poly). Awaiting user-obtained API creds (`NOVIG_CLIENT_ID/SECRET` in .env — requested via Novig in-app chat; no self-serve portal; "EMM" program). On first authenticated run: `python main\novig_live.py` self-test, verify the UNVERIFIED order-book schema (TODO in module), fill NOVIG_LEAGUE from `list_leagues()`. Don't nag the user about it — they'll add creds when ready. Not wired into live_monitor until verified.