a 8j@sdZddlZddlmZmZddlmZddlmZddlm Z m Z m Z ddl Z ddlmZmZdd lmZmZmZmZmZdd lmZmZmZmZmZed Zd Zd diZddddddZ Gdd d eZ!ej"e j#dddZ$e%e j#e%e&e j#dddZ'e j#e j#dddZ(dS)zScraper for http://fbref.com.N)datetimetimezone)reduce)Path)CallableOptionalUnion)etreehtml)BaseRequestsReader SeasonCodeadd_alt_team_names make_game_idstandardize_colnames)DATA_DIRNOCACHENOSTORETEAMNAME_REPLACEMENTSloggerFBrefzhttps://fbref.comz sec-ch-uaz@"Not)A;Brand";v="8", "Chromium";v="138", "Google Chrome";v="138"z ITA-Serie Az FRA-Ligue 1z ESP-La LigazENG-Premier LeaguezGER-Bundesliga)zSerie AzLigue 1zLa LigazPremier LeagueZ Bundesligac seZdZdZdddeeefeee e e feee e e feee e e e ge ffe e edfdd Zee e dddZeee e fdfd d Zeedd d Ze e e d fdd Zd2e ejdddZd3e ejdddZd4e e ejdddZd5e e eee e e fe ejdddZd6e ejddd Zd7e ejd!d"d#Ze j!e ed$d%d&Z"d8e eee e e fe ejd(d)d*Z#d9eee e e fe ejd+d,d-Z$d:eee e e fe ejd+d.d/Z%d;eee e e fe ejd+d0d1Z&Z'S)tt|dddjddtjd}tjtjd|kSt||S) zCheck if a season is complete.rN%yrr )tzinfo)tz)rstrptimeyearrutcnowr" _is_complete)r+r;r<Z season_endsr,r.r/rEzszFBref._is_completeF) split_up_big5r0c Cstd}|jd}|||}g}t|}|dD]$}t|}|d|d<||q8t | t j ddid |j jdd d} | d |jj| d <| d |jj| d <|j} d |jvr|rtt|jd httB} | | j| S) a)Retrieve selected leagues from the datasource. Parameters ---------- split_up_big5: bool If True, it will load the "Big 5 European Leagues Combined" instead of each league individually. Returns ------- pd.DataFrame z /en/comps/z leagues.htmlz//table[contains(@id, 'comps')]z'.//th[@data-stat='league_name']/a/@hrefurlcompetition_namer;columns)subsetZ first_seasonZ last_seasonr) FBREF_APIrgetr parsexpath _parse_tableappendpdconcatpiperrename_translate_leaguedrop_duplicates set_index sort_indexapplyr:rr2r%r&r'indexisin) r+rFrGfilepathreaderdfstree html_tabledf_tabledfrr.r.r/ read_leaguess>        zFBref.read_leaguescCs8d}||}g}|D]\}}t|j}|j||}|||} t| } | d\} t | } | d| d<|| d<d| j vr| j ddidd d | j vrd | d <nd | d <| | qt|t} | j ddid} | d|jj| d<| jddgddd| ddg} | jtd|jfddgfS)a2Retrieve the selected seasons for the selected leagues. Parameters ---------- split_up_big5: bool If True, it will load the "Big 5 European Leagues Combined" instead of each league individually. Returns ------- pd.DataFrame zseasons_{}.htmlz//table[@id='seasons']z7//th[@data-stat='year_id' or @data-stat='year']/a/@hrefrGzCompetition NameZYearZSeasonT)rJinplaceFinal eliminationZFormatz round-robinrHr;rIr<first)rKZkeepreNformat)rditerrowsrLrGrrirMr rNrOrPrJrUrQrRrSrTrrZr:rWrXrYlocslicer)r+rFfilemaskZ df_leaguesrlkeyr;rGr]r^r`rarbrcr.r.r/ read_seasonss4          zFBref.read_seasonsstandard) stat_typeopponent_statsr0cCsgd}d}||vr"td||dkr0d}n@|dkr>d}n2|dkrLd }n$|d kr^d }d }n|d krld }n|}|r~|d7}n|d7}|}g}|D]:\\}} } |dk} | ddk} |j||| | r|n|} td| jddd| rd|dn| rd|dnd| jdd}| || }t |}| d|d|d\}t |}||d<| |d<| d|d<| r|jdddd t|d<|jdddd!d"|jd#ddd!d"||qt|ddgd$jd%d&d'd(d%tigd)S)*aRetrieve aggregated season stats for all teams in the selected leagues and seasons. The following stat types are available: * 'standard' * 'keeper' * 'keeper_adv' * 'shooting' * 'passing' * 'passing_types' * 'goal_shot_creation' * 'defense' * 'possession' * 'playing_time' * 'misc' Parameters ---------- stat_type: str Type of stats to retrieve. opponent_stats: bool If True, will retrieve opponent stats. Raises ------ ValueError If ``stat_type`` is not valid. Returns ------- pd.DataFrame rpkeeper keeper_advshootingpassing passing_typesgoal_shot_creationdefense possession playing_timemisczteams_{}_{}_{}.html)Invalid argument: stat_type should be in rpstatsrtkeepersru keepersadvrygcar| playingtimeZ_againstZ_forrrirg/Nz/squads/z//table[@id='stats_teams_z' or @id='stats_squads_']r;r<z.//*[@data-stat='team']/a/@hrefrGCompr axislevelTrrreRkkeyteamZ players_used)Squadz# PlrIr;r<r) ValueErrorrorjrrirLjoinrGsplitrMr rNrOrPxssqueezemapr&droprQ_concatrUreplacerrXrY)r+rqrr team_statsrmpagerteamsrnskeyr<big_fiveZ tournamentr]rGr^r`rarbr.r.r/read_team_season_statssp"  &   zFBref.read_team_season_statsschedule)rqrrr force_cacher0c CsRgd}d}||vr"td||dkr6|r6td|dkrBd}|rJdnd }|}|d urt|} |j|jj| d d d d f} t| d krtdn|} g} | jD]\\} } }}|j | || |}t| ddkr t | ddd ddd|}nt| tjkr\tj| }d t|d d djt|d d dj}n tj| }t | ddd d|ddd|}|| |  }|j|||o| d}t|}|d|d\}|dD] }d|_q|dD]}||qt|}| |d<||d<dd|d D|d!<d"d|d#D|d$<|jj}|d kr|j d$d!gddd%}| !|qt"| gd&d'#d(t$ij%d)d*id+&|j'j&t(gd,d-}t)*|d.+|d.<|gd/,}gd/|_|j-d0d1dd2|d3<|j-d4d1dd2|d5<|j-t.dd2|d6<|/gd70j|j1S)8aRetrieve the match logs for all teams in the selected leagues and seasons. The following stat types are available: * 'schedule' * 'keeper' * 'shooting' * 'passing' * 'passing_types' * 'goal_shot_creation' * 'defense' * 'possession' * 'misc' Parameters ---------- stat_type: str Type of stats to retrieve. opponent_stats: bool If True, will retrieve opponent stats. team: str or list of str, optional Team(s) to retrieve. If None, will retrieve all teams. force_cache: bool By default no cached data is used for the current season. If True, will force the use of cached data anyway. Raises ------ ValueError If ``stat_type`` is not valid. Returns ------- pd.DataFrame ) rrvrtrwrxryrzr{r}zmatchlogs_{}_{}_{}.htmlr~rz=Opponent stats are not available for the 'schedule' stat typeryrZagainstforN)rrz:No data found for the given teams in the selected seasons.rr z /matchlogsz /all_compsz{}-{}r>rz//table[@id='matchlogs_rz%//th[@data-stat='header_for_against']z//tfootr<rcSsg|]}|ddqS)ZcskN)rM).0xr.r.r/ sz/FBref.read_team_match_stats..z.//td[@data-stat='start_time']TimecSs<g|]4}|dr4|ddjdkr4|ddndqSz./ar Match Reportz ./a/@hrefNrOtextrZmlinkr.r.r/rs  .//td[@data-stat='match_report']rrrrOpponentrr;rI) ZTeamrZVenueDaterZRoundZDayResultrcolsdate)ropponentvenuercSs|ddkr|dS|dS)NrHomerrr.rr.r.r/z-FBref.read_team_match_stats..r home_teamcSs|ddkr|dS|dS)NrAwayrrr.rr.r.r/rr away_teamgame)r;r<rr)2rrrrkr[r\lenrGr1rrirrLrsplitr Z from_leaguer9rNrrArBZ SINGLE_YEARrErMr rOr getparentremoverPrJZnlevelsrrQrrrrUrTrVrrR to_datetimeffillcopyrZrrXrYr)r+rqrrrrrrmZopp_typeZdf_teamsZteams_to_checkiteratorrrnrZteam_urlr]rGZ_skeyZ season_formatcurrent_seasonr^r`raelemrbZ nb_levelsrcZdf_tmpr.r.r/read_team_match_statsYs)                  zFBref.read_team_match_stats)rqr0cCsFgd}d}||vr"td||dkr0d}n@|dkrBd}d}n.|dkrPd }n |d kr^d }n|d krld }n|}|}g}|D]R\\}}} |dk} |j||||} td| jdddd|| rdnd| jdd} | | | } t | }| dD]}| |q| rpt|}|jddddt|d<||d<|jdddddnV| d|d\}tjdd}t|j| d|d\}t|}||d<||d<t|}||qt|dd gd!}||jd"k}|jd#dd$djd%dd$djd&d'id(d'tij t!gd)d*"gd+#S),aRetrieve players from the datasource for the selected leagues and seasons. The following stat types are available: * 'standard' * 'shooting' * 'passing' * 'passing_types' * 'goal_shot_creation' * 'defense' * 'possession' * 'playing_time' * 'misc' * 'keeper' * 'keeper_adv' Parameters ---------- stat_type :str Type of stats to retrieve. Raises ------ TypeError If ``stat_type`` is not valid. Returns ------- pd.DataFrame rszplayers_{}_{}_{}.htmlr~rprryrr|rrtrrurrrNrz /players/z#//td[@data-stat='comp_level']//spanrr r)zUnnamed: leaguer;)zUnnamed: seasonr<Trz"//comment()[contains(.,'div_stats_z')])Zrecoverz//table[contains(@id, 'stats_r;r<rPlayerMatchesrrrrrI)rNationPosAgeZBornr)r;r<rplayer)$ TypeErrorrorjrrirLrrGrrMr rNrOrrrPrrrr&rr HTMLParserZ fromstringr_fix_nation_colrQrrrUrrrTrrXrY)r+rqZ player_statsrmrrZplayersrnrr<rr]rGr^r`rrbelparserrarcr.r.r/read_player_season_statss      zFBref.read_player_season_stats)rr0cCs|jdd}g}|D]\\}}}t|j}|jd|d|d}|||} t| } t| ddd} |jd |d|d} | || } |j| | | o| d } t| } | d d}t |}d d |dD|d<||d<||d<|j dd}| |qt |jdddddddttdt}t |d|d<|jtdd|d<|j|jd fjd!jd"|j|jd#f<|gd$S)%aHRetrieve the game schedule for the selected leagues and seasons. Parameters ---------- force_cache : bool By default no cached data is used for the current season. If True, will force the use of cached data anyway. Returns ------- pd.DataFrame T)rFZteams_r4z.htmlz//a[text()='Scores & Fixtures']rhrefZ schedule_rz//table[contains(@id, 'sched')]cSs<g|]4}|dr4|ddjdkr4|ddndqSrrrr.r.r/rs z'FBref.read_schedule..rrr;r<allhowweekrrZhome_xgZaway_xg)ZWkrrZxGzxG.1rI)rrrr rr match_reportrgame_idr;r<r)rorjrLrGrrMr rNrOrErPdropnarQrRrSrUrrrTrrrrZrrkrisnastrrrXrY)r+rrrrnrr<Z url_statsZfilepath_statsr^r`Z url_fixturesZfilepath_fixturesrrarbrcr.r.r/ read_schedulesZ          "zFBref.read_schedule)r`r0cCsJ|ddd}g}|D]*}||ddd|jdq|S)zParse the teams from a match summary page. Parameters ---------- tree : etree.ElementTree The match summary page. Returns ------- list of dict z"//div[@class='scorebox']//strong/aNrrrr)idname)rOrQrMrrstrip)r+r`Z team_nodesrrr.r.r/ _parse_teamss (zFBref._parse_teamssummary)rqmatch_idrr0cCsgd}td}d}||vr*td|||}||j|j@}|dur||jt |t rt|gn|}t |dkrt dn|}g} | D]\} } || d} td | d t || d|j|| d} || | }t|}||\}}|d kr d nd |}|d||dd}|durt|}|d|d<| d|d<| d|d<| d|d<| d|d<| |ntd| d|d||dd}|dur$t|}|d|d<| d|d<| d|d<| d|d<| d|d<| |qtd| dqt| dgd}||jj d}|jddiddtij t!gdd"gd#S) aRetrieve the match stats for the selected leagues and seasons. The following stat types are available: * 'summary' * 'keepers' * 'passing' * 'passing_types' * 'defense' * 'possession' * 'misc' Parameters ---------- stat_type : str Type of stats to retrieve. match_id : int or list of int, optional Retrieve the event stream for a specific game. force_cache : bool By default no cached data is used to scrape the list of available games for the current season. If True, will force the use of cached data anyway. Raises ------ ValueError If no games with the given IDs were found for the selected seasons and leagues. TypeError If ``stat_type`` is not valid. Returns ------- pd.DataFrame )rrrwrxrzr{r}/en/matches/{} match_{}.htmlr~Nr:No games found with the given IDs in the selected seasons.r"[%s/%s] Retrieving game with id=%sr rzkeeper_stats_{}z stats_{}_ //table[@id='rrrrrr;r<z0No stats found for home team for game with id=%sz0No stats found for away team for game with id=%srz^\d+\sPlayers$# jersey_numberrI)rrrrMinr)r;r<rrr)$rLrr reset_indexrrrisnullr\ isinstancerrrrjririnforrMr rNrfindrPrQwarningrrcontainsrUrrrTrrXrY)r+rqrrZ match_statsurlmaskrm df_schedulerrirrGr]r^r`rrZ id_formatrarbrcr.r.r/read_player_match_statsst'                   zFBref.read_player_match_stats)rrr0c Csjtd}d}||}||j|j@}|durt||jt|t rV|gn|}t |dkrxt dn|}g}| D]\}} | | d} td|dt || d|j| | d} || | } t| } || }| d }t|D]N\}}t|}d d g|_||d |d <d|jjvr|j|jdkd}d|jd|df<d|j|ddf<| d|d<| d|d<| d|d<| d|d<|j|dd| dd ||dd}t|}|jdddgd}gd|_|d  d|d <|d  d|d <t!j"||d d gdd}|d #d|d <|$|qqt!%|&gd!S)"abRetrieve lineups for the selected leagues and seasons. Parameters ---------- match_id : int or list of int, optional Retrieve the lineup for a specific game. force_cache : bool By default no cached data is used to scrape the list of available games for the current season. If True, will force the use of cached data anyway. Raises ------ ValueError If no games with the given IDs were found for the selected seasons and leagues. Returns ------- pd.DataFrame. rrNrrrrr z//div[@class='lineup']rrrrZBenchTZ is_starterFrr;r<)rerzstats_{}_summaryrrr)rrrr)rrpositionminutes_playedZInt64left)onrrr)'rLrrrrrrr\rrrrrjrirrrrMr rNrrO enumeraterPrJrr'r[rkrrZ droplevelZastyperRmergeZfillnarQrSrX)r+rrrrmrrZlineupsrrrGr]r^r`rZ html_tablesrarbZ bench_idxZhtml_stats_tableZdf_stats_tabler.r.r/ read_lineupVsh             zFBref.read_lineupc CsZtd}d}||}||j|j@}|durt||jt|t rV|gn|}t |dkrxt dn|}g}| D]\}} g} | | d} td|dt || d|j| | d} || | } t| }||}t|d d gD]\}}|d |d }|D]}|d djdd}|ddj}|dd}|dr|ddj}nd}|dddd}| |d|||||dq qt| }| d|d<| d|d<| d|d<t |dkr|jddd||qt |dkr0tSt |dt!i"gd#j$dd S)!aRetrieve match events for the selected seasons or selected matches. The data returned includes the timing of goals, cards and substitutions. Also includes the players who are involved in the event. Parameters ---------- match_id : int or list of int, optional Retrieve the events for a specific game. force_cache : bool By default no cached data is used to scrape the list of available games for the current season. If True, will force the use of cached data anyway. Raises ------ ValueError If no games with the given IDs were found for the selected seasons and leagues. Returns ------- pd.DataFrame. rrNrrrrr abz/////*[@id='events_wrap']/div/div[@class='event rz./div[1]z’rz./div[1]/small/spanz./div[2]/div[2]/divz./div[2]/div[2]/small/az./div[2]/div[1]/@class r)rminutescoreplayer1player2 event_typerr;r<r T)Zbyrerrrr)%rLrrrrrrr\rrrrrjrirrrrMr rNrziprOrrrZ text_contentrrQrR DataFrameZ sort_valuesrSrrXrYr)r+rrrrmrreventsrrZ match_eventsrGr]r^r`rrtidZ html_eventser r r r rZdf_match_eventsr.r.r/ read_eventssz            zFBref.read_eventscCstd}d}||}||j|j@}|durt||jt|t rV|gn|}t |dkrxt dn|}g}| D]\}} | | d} td|dt || d|j| | d} || | } t| } | d }|dur0t|}| d |d <| d |d <| d |d <||qtd | dqt |dkrXtSt|d gdjddiddtijtgdd gd!j"ddS)aQRetrieve shooting data for the selected seasons or selected matches. The data returned includes who took the shot, when, with which body part and from how far away. Additionally, the player creating the chance and also the creation before this are included in the data. Parameters ---------- match_id : int or list of int, optional Retrieve the shots for a specific game. force_cache : bool By default no cached data is used to scrape the list of available games for the current season. If True, will force the use of cached data anyway. Raises ------ ValueError If no games with the given IDs were found for the selected seasons and leagues. Returns ------- pd.DataFrame. rrNrrrrr z//table[@id='shots_all']r;r<rz&No shot data found for game with id=%srrrrI)ZOutcomeZMinuteZDistancerz Body PartZNotesEventrrrr)#rLrrrrrrr\rrrrrjrirrrrMr rNrrPrQrrRrrrUrrrTrrXrYr)r+rrrrmrrZshotsrrrGr]r^r`rarbr.r.r/read_shot_eventss\          zFBref.read_shot_events)F)F)rpF)rFNF)rp)F)rNF)NF)NF)NF)(__name__ __module__ __qualname____doc__rr FBREF_DATADIRrrrr2intrboolrr#propertyr classmethoddictr5r r:rErRrrdrorrrrr Z ElementTreerrrrr __classcell__r.r.r,r/r$s!  -2 w ;sJ v ] e)rar0cCs~|dD]}tj|dddq |dD]}||q.|dD]}||qLtjt|dd\}| S) zParse HTML table into a dataframe. Parameters ---------- html_table : lxml.html.HtmlElement HTML table to clean up. Returns ------- pd.DataFrame z//span[contains(@class, 'f-i')]spanF)Z with_tailz&//tbody/tr[contains(@class, 'spacer')]z%//tbody/tr[contains(@class, 'thead')]lxml)Zflavor) rOr Zstrip_elementsrrrRZ read_htmlr ZtostringZconvert_dtypes)rarrbr.r.r/rPqs rP)r_rr0c Csg}t|dddd}|D]}t|j}|jddddd|jdd kr|jd ddd t|d}|j|dd gfj |j|d dgf<| | t|d }|j|dd gfj |j|d dgf<d |j|df<tj |jd d|_qt|r|d jdd krt|D]\}}|d|d ds|d j|ddd}tjdj|d jdd|j ||jdd|j dtt|j|ddkd dgfjd ddtt|j|ddkd dgfjd ddddqt|r|d jdd krtdd|}t|d }|j|dd gfj |j|d dgf<d |j|df<tj |jd d}t|D]F\}}|j|rt|jt|kr||_n|j|dd||<q~t|S)a9Merge matching tables scraped from different pages. The level 0 headers are not consistent across seasons and leagues, this function tries to determine uniform column names. If there are dataframes with different columns, we will use the ones from the dataframe with the most columns. Parameters ---------- dfs : list(pd.DataFrame) Input dataframes. key : list(str) List of columns that uniquely identify each df. Returns ------- pd.DataFrame Concatenated dataframe with uniform column names. cSs t|jSN)rrJrr.r.r/rrz_concat..T)rreversez ^Unnamed:.*N) to_replacevalueregexrer rr)r&r'rerF)r[outer)Z indicatorrzDifferent columns found for {first} and {cur}. The following columns are missing in {first}: {extra_cols}. The following columns are missing in {cur}: {missing_cols}. The columns of the dataframe with the most columns will be used.z, _merge left_only right_only)rhcurZ extra_colsZ missing_colsr cSs ||Sr$)Z combine_first)rrightr.r.r/rr)rJZ fill_value)sortedrRrrJtolistrshaperrkr'rQrZ MultiIndex from_tuplesZ to_recordsrrequalsrr)r*riZilocrrrrZreindexrS) r_rZ all_columnsrcrJmaskrr8Z column_idxr.r.r/rsl"" "r)rbr0c Csfd|jdvrb|jdddddd|jddtddff<|dd |jdddd|S) zFix the "Nation" column. Adds a 'nations' column for international games based on the team's name. Parameters ---------- df_table : pd.DataFrame Input dataframe. Returns ------- pd.DataFrame rr rrcSst|tr|dkr|SdS)Nr)rrrr.r.r/r rz!_fix_nation_col..Nr)zUnnamed: nationr)rJZget_level_valuesrrrZrkrlinsert)rbr.r.r/rsr))rr)rr functoolsrpathlibrtypingrrrZpandasrRr#r r _commonr r rrr_configrrrrrrrLr$r&rZ HtmlElementrrPr2rrrr.r.r.r/s>   Ul