# Options Pipeline - Detailed Architecture Notes

## Data Flow
```
E:/options_data/{date}/{STOCK}.json
  → multi_stock_pipeline.py: combine_data_for_stock()
  → E:/options_dfs/calls/{STOCK}.csv
  → E:/options_dfs/puts/{STOCK}.csv
  → train_and_test_models_for_stock()
  → E:/models/{STOCK}_*.sav or *.joblib
  → analyze_stock_performance()
  → E:/analysis.csv
```

## MultiStockOptionsPipeline Key Methods
- `get_stock_list()` — discover stocks from data dir
- `combine_data_for_stock(stock)` → `(calls_df, puts_df)`
- `get_price_at_expiry_for_stock(df, 'calls'|'puts')` → df with expiry prices
- `train_and_test_models_for_stock(stock, 'calls'|'puts')` → result dict
- `analyze_stock_performance(stock)` → analysis dict
- `run_complete_pipeline()` → bool success
- `run_pipeline_for_top_stocks(tokens)` — uses etrade tokens

## OptionsBacktester (backtest_options.py)
- Walk-forward validation: configurable splits, train_days, test_days
- `_suggest_stock_worker` — used with ProcessPoolExecutor for parallel backtesting
- CLI: `python backtest_cli.py --all-stocks --top-n 20 --months 3`

## Quantile Regression (quantile_regression.py)
- LightGBM backend (`LGBMRegressor`)
- Trains quantiles: 0.10, 0.25, 0.50, 0.75, 0.90
- Calibration step to hit target coverage
- Target: `return_pct` = return / (ask * 100)
- Deduplication: removes duplicate option contracts before training
- Warning: `X does not have valid feature names` (known sklearn/lightgbm compat issue — non-blocking)
- Model saved as `{STOCK}_p0_{CALLS|PUTS}_quantile_reg.joblib`

## Regime Detection (regime_detection.py)
- VIX-based market regime buckets
- `add_regime_features(df)` — adds regime columns to dataframe

## Gate Stack (in backtest_options.py run_walk_forward)
Applied sequentially after model scoring, before top-N selection:
1. **Direction gate**: mom20d >= 1% AND mom5d >= -3% for CALLS; reversed for PUTS
2. **Delta gate**: |delta| >= 0.25 (or moneyness within 10% OTM if delta missing)
3. **IV/HV gate**: iv_hv_ratio <= 2.0 (avoid overpriced options)
4. **Ask% gate**: ask <= 4% of underlying price
5. **Win-prob gate**: predicted win_probability >= 50%
Then composite score threshold (default 30, was 50)

## Score Adjustments After Gates
- **Delta bonus**: +0 to +10 pts for delta 0.25→0.50+
- **DTE bonus**: +5 pts for 7-21 DTE, +2 for 3-6 DTE, -5 for <3 or >45 DTE

## Composite Score Components
- 30% Kelly score, 25% win_prob score, 20% EV score, 15% downside score, 10% consistency
- All scored 0-100, composite = weighted sum (0-100)

## Validation Quality Gate
- `min_val_profit_factor` (default 0.8): skip walk-forward period if model's own val PF < threshold
- Secondary: skip if val WR < 25% when gate is active and val_n >= 10

## Suggest Mode (--suggest flag)
- Trains on all historical data with known outcomes
- Scores currently-open (future-expiry) options through same gate stack
- Outputs HTML report + CSV to `reports/suggest_*.html`

## Known Issues / Observations
- Backtest win rate ~30% per-stock top-N, 14.5% global top-N — poor
- CAR stock shows many -100% returns despite high composite score (98.7)
- Quantile calibration offsets vary widely (e.g. +0.49 for q=0.90 on some stocks)
- Fix 7 (regime features) and Fix 8 (binary classifier) were both tried and reverted
- Large volume of HTML reports generated (reports/ dir, ~50+ files from Feb 21-25)

## Running the Pipeline

### Quick single stock
```bash
python master_run.py quantile --stock AAPL
```

### All stocks backtest
```bash
python backtest_cli.py --all-stocks --top-n 20 --months 3
# With new flags (unstaged):
python backtest_cli.py --all-stocks --top-n 20 --months 3 --min-val-pf 0.8 --max-otm-pct 0.10 --workers 4
# Suggest mode (unstaged):
python backtest_cli.py --all-stocks --suggest --top-n 5 --suggest-months 3 --workers 4
```

### Full pipeline (821 stocks)
```python
from multi_stock_pipeline import MultiStockOptionsPipeline
pipeline = MultiStockOptionsPipeline()
pipeline.run_complete_pipeline()
```

### Sync data from Google Drive
```bash
python sync_options_data.py
python sync_options_data.py --dry-run  # preview
```