a ri( @sdZddlZddlZddlZddlZddlZddlm Z ddl m Z dZ de iZ e dZ e dZe d Ze d Ze d Zd Zd ZddZddZddZddZedededeeZedeedeedeZe r>e!e*Z"edde"#$DZWdn1s40Yeed%ej&dd Z'ed!ed"ee'dd#eeddee'dkred$ne\Z(Z)ed%e(d&e)d'e)e(d(e reene*Z+gZ,e(Z-dZ.e/d0d)Z1e'2D]\Z3Z4e5e4dZ6e-ekr`e 7Z7e7j8d*d+d*Z9e:dd*e7j8d,e7j;d-e7j<d-Z=e=d,Z>e=d,d-Z?ed.e.d/e,re*e,Z@ee+rejAe+e@gdd0ne@ZBeBjCed1d2ed3eeBdd4ee!ed5Z"Wdn1s0Yed6e>d7e?d8eDe=e\Z(Z)e(Z-ed%e(d&e)d'e)e(d(dZEeEd9krz,ee6ZFe-d+7Z-e.d+7Z.eFree6e1e6eFZGe,HeGe!ed5 Z"e"Ie6d:Wdn1s0YeJe6e.edkre*e,Z@ee+r$ejAe+e@gdd0ne@ZBeBjCed1d2eBZ+gZ,ed;ee+ddd?eEZQed@eQdAeEd+dBeDeQeEd+7ZEnePdCkre!ed5 Z"e"Ie6d:Wdn1sL0YeJe6e.d+7Z.eDd+WYdZO[Oqn,edDe6dEeOeDd?WYdZO[OqWYdZO[OnZdZO[O0eRyZOz8edDe6dEeOeDd?WYdZO[OqWYdZO[On dZO[O00qdqe,rve*e,Z@ee+rHejAe+e@gdd0ne@ZBeBjCed1d2ed3eeBdd4en$ee+red:ee+ddFeedGeeZSeeSZTeTjCed1d2dHdIeTjUDZVedJeeTddKeedLeVdS)Mu API-Football data pull — fetches match statistics for all fixtures. Saves to E:\soccer aw_api\ (never OneDrive, avoids disk-full eviction). Resumes from done_fixtures.txt on restart. N)Path)datetimeZ 81b487be73030f5dec980d2a1ef2c406zx-apisports-keyzE:\soccer\raw_apizmatch_stats.csvzdone_fixtures.txtzfixtures_index.csvzmatch_features.csviicCsZd}zBtj|tdd}|}|ddd}|ddd}||fWSYdS0dS) Nz(https://v3.football.api-sports.io/status )headerstimeoutresponserequestsZcurrentZ limit_day)riL)rgetHEADERSjson)urlrdatausedlimitr#E:\soccer\misc\pull_api_football.pyget_requests_useds rcCs2d}tj|td|idd}||dgS)Nz5https://v3.football.api-sports.io/fixtures/statisticsZfixture)rparamsrr)rr r Zraise_for_statusr ) fixture_idr r rrr fetch_stats!srcsg}|D]}|dd}||dk}dd|dDd fdd }|||d |d |d |t||d|d |d |d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|ddq|S)!Nteamname home_teamcSsi|]}|d|dqS)typevaluer).0srrr ,zparse_stats..Z statisticscsn||}|dur|St|trN|drNzt|dWS|YS0z t|WS|YS0dS)N%)r isinstancestrendswithfloatrstrip)keydefaultvZstatsrrr -s   zparse_stats..getdatedivseason away_team home_goals away_goalsz Shots on GoalzShots off Goalz Total Shotsz Blocked ShotszShots insideboxzShots outsideboxZFoulsz Corner KicksZOffsideszBall Possessionz Yellow Cardsz Red CardszGoalkeeper Savesz Total passeszPasses accuratezPasses %)rr+r,r-ris_homerr.r/r0 shots_on_goalshots_off_goal total_shots blocked_shots shots_inside shots_outsidefoulscornersoffsides possession yellow_cards red_cardssaves total_passesaccurate_passespass_pct)N)appendint)rZ fixture_rowrZrowsZ team_dataZ team_namer1r rr*r parse_stats'sF   rDc stdd<ddgjddgd}dd<dd kd d k@ddkd d k@B}d d k}d j|df<d j|df<jd dd dd<jddd dd<gdfdd|D}g}dD]t\}}|djdd}dD]J}|D]>} | |jvr||  d j |d d |d| d|<qq| |qtj |ddgdddjD} fdd| DS)Nr+rTZdrop)r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArptsr1r/r0cSs|dr|dS|dS)Nr1r/r0rr rrrbr z build_features..)ZaxisgfcSs|dr|dS|dS)Nr1r0r/rrIrrrrJcr ga)rFrKrLcsg|]}|jvr|qSrcolumnsrc)dfrr dr z"build_features..)r)Z min_periodsapi_Z_wZ ignore_index) rr+r,r-rr1rr.r/r0cSsg|]}|dr|qSrT startswithrOrrrrRpr csg|]}|jvr|qSrrMrO)feat_dfrrrRqr )copypdZ to_datetimeZ sort_values reset_indexZlocZapplygroupbyrNshiftZrollingZmeanrBconcat) rQZ stat_colsZwin_maskZ draw_maskZ roll_colsZ feat_rowsrZgrpwcol feat_colsr)rQrYrbuild_featuresSs6 4 rcz=== API-Football Data Pull === z Loading existing fixture index ()z Total fixtures in index: ,r,ccs|]}|rt|VqdS)N)striprC)rxrrr r rhrTrEz" === Pulling match statistics === zFixtures to process: z (already done: zAll fixtures already pulled!z API requests today: /z (z remaining)indexrGi<u(Approaching daily limit — stopping at z requests todayrUF)rjz Saved z team-match rows to az Daily limit reached. Sleeping zh zm until quota resets...rH z Checkpoint: z rows saved, iu 429 rate limit — sleeping z s (retry z/3)iz Error fixture z: z team-match rows already in z" === Building rolling features ===cCsg|]}|dr|qSrVrWrOrrrrRr rRzFeatures saved: z rows -> z Columns: )W__doc__rZpandasr[ZnumpyZnptimeospathlibrrZAPI_KEYr ZOUT_DIRZ STATS_CSVZ DONE_FILEZ FIXTURES_CSVZ FEATURES_CSVZ DAILY_LIMITZ CHECKPOINTrrrDrcprintZread_csvZindex_dflenZ value_countsZ to_stringsetZdoneexistsopenfreadsplitZisinr\ZtodorrZ DataFrameexistingZnew_rowsZrequests_todayZ done_this_runZ set_indexZto_dictZfixture_lookupZiterrowsirowrCZfidZnowZhourZ reset_hourmaxZminutesecondZ wait_secsZhrsZminschunkr_ZcombinedZto_csvsleepZretriesrZparsedextendwriteaddZused2_ exceptionsZ HTTPErroreZ status_codewait ExceptionZstats_dfrYrNrbrrrrs(   ,!   :"     (          0             0   " ,