a j?@sddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZmZddlmZdd Zejd gd d d Zejd gd ddZejdejejejgejdejejejgddZejdejejejgddZddZddZddZ ddZ!ddZ"ejjd gd!gd"ge#gd#gd$gej#gd%gd&ge$d'ej#gd(d)ej%d*gge$d'ej#gd(d)e&d+d*gge$d'ej#gd,gd-ge$d'ej#gd.d)ej%dgge$d'ej#gd.d)e&d+dgge$d'ggd/d0d1d2Z'ejd gd ejd3d4d5gejd6dd7gd8d9Z(ejd3d4d5gejd:d;dgd?gd>gfd@d)gdAd)gdBdCgdAd)gggdDgdEgdFgfgdGdHZ)dIdJZ*ejd6gdKejdLgdKdMdNZ+ejdOdPdQgejd d=d;ge#dRdSggdTdUZ,ejdOdPdQgdVdWZ-ejjdXdYdejde&e?gddZ@ddZAddZBddZCddZDddZEddZFejd6dyd7gddZGejdej%de&d+gddZHejd6dYd[ggdgddZIejjdd5d4gddgd0ejjd6d7gdgd7dgd0ddZJejdoe e gddZKejddd;iddiddid;dddeddgejddgdggddZLejd6dyd7dCggddZMejd6d)gdbggddZNejddd[iddiddiddiddid[dddeddgddńZOejd6d7dCggddDŽZPejd6d)gdbggddɄZQdd˄ZRejdd[d=dddeigdd̈́ZSddτZTddфZUddӄZVddՄZWddׄZXejddd=dٜgddۄZYejdd;d[dٜgdd݄ZZejdgdߢejdgdddZ[ddZ\ejdej%dgddZ]ddZ^ejd gd ejdddgddZ_ejd gd ddZ`ejd gd ddZaejd gd ddZbddZcddZdejdej%dgddZeejdddgejdej%dgddZfejjdaej#d)ej%gge$d'j2ej#d)dCgge$d'j2ej#d)dbej%ge$d'gej.fej#d)ej%gge$d'j2ej#d)dCgge$d'j2ej#d)dbej%ge$d'gej.fej#dej%ggejd'j2ej#dRggejd'j2e#ddSej%ggejfggdd0ddZgejdoe e gddZhejde#dej%dRggj2e#dej%dggj2e#dSggfe#gdgj2e#gd gj2e#ej%ggfej#d ej%dCgge$d'j2e#dej%dggj2ej#dbgge$d'fej#gd ge$d'j2e#gd gj2ej#ej%gge$d'fgd dZiejdeddZjddZkejddd]ggej#dd]ggdd'ej#dd]ggdd'gejdd\d]ggej#d\d]ggdd'ej#d\d]ggdd'gddZlddZmddZndd Zoejd!d5d4gd"d#Zpejd$ej#d)gd%gge$d'dgej%gej%ggejqd%gdgdgge$d'fej#ej%gd%gd)gge$d'dgej%gej%ggejqd%gej%gej%gge$d'fgd&d'Zrd(d)Zsd*d+Ztd,d-Zud.d/Zvd0d1Zwejddd[iddiddiddiddid[dddeddgd2d3Zxd4d5Zyd6d7Zzd8d9Z{d:d;Z|ejdddidd;igd<d=Z}ejddd=idd>igd?d@Z~dAdBZdCdDZejdoe e gdEdFZdS(GN)sparse)NotFittedError) OneHotEncoderOrdinalEncoder) is_scalar_nan)_convert_containerassert_allcloseassert_array_equal)CSR_CONTAINERScCstgdgdg}t}tdd}||}||}|jdksHJ|jdksVJt|sdJt|rrJt|gdgdgt||dS)NrrrF sparse_outputr )?rrr)rrrrr) nparrayr fit_transformshaperissparser toarray)XZ enc_sparseZ enc_denseX_trans_sparseZ X_trans_denserOE:\soccer\ml_env\Lib\site-packages\sklearn/preprocessing/tests/test_encoders.py!test_one_hot_encoder_sparse_denses   r handle_unknown)ignoreinfrequent_if_existwarncCstgdgdgdg}tgdg}tdd}||tjtdd||Wdn1sn0Yt|d}|||}t || tgd gt ||dS) N)rr r)rrr )rrr )rrerrorr!Found unknown categoriesmatch)rrrrrrr) rrrfitpytestraises ValueError transformcopyr rrr!rX2ohZ X2_passedrrr#test_one_hot_encoder_handle_unknown*s  (   r4cCsxtgdd}tddgd}t|d}|||}t||tgdgdgt||dS)N)Z1111111122Z333Z4444)rZ55555r5r')rrrrrrrr) rrreshaperr+r0r r/rr1rrr+test_one_hot_encoder_handle_unknown_stringsBs   r9 output_dtype input_dtypecCstjddgg|dj}tjddgddgg|d}td|d}t|||t||||td|dd}t|||t||||dS)Nrrdtypeauto) categoriesr=F)r?r=r) rasarrayTrr rrr+r/)r;r:r X_expectedr3rrrtest_one_hot_encoder_dtypeUs rCcCstd}|ddgddgd}tjgdgdg|d }t|d }t|||t| | ||t|d d }t|||t| | ||dS) Npandasabrr ABrrrrrrrrr<F)r=r) r, importorskip DataFramerrrr rrr+r/)r:pdX_dfrBr3rrr!test_one_hot_encoder_dtype_pandasds   rPcCst}gdgdgdgdg}|||}tgd||gd}tgd|tjtdd |d d gWdn1s0YdS) N)Malergirlr r )Female)rRr )rQ3boy r )rQ[rR)Z x0_FemaleZx0_MaleZx1_1Zx1_41Zx1_51Zx1_91Zx2_boyZx2_girlZx3_1Zx3_2Zx3_12Zx3_21Zx4_3Zx4_10Zx4_30)onetwothreefourfive)Z one_FemaleZone_MaleZtwo_1Ztwo_41Ztwo_51Ztwo_91Z three_boyZ three_girlZfour_1Zfour_2Zfour_12Zfour_21Zfive_3Zfive_10Zfive_30z!input_features should have lengthr)r\r])rr+get_feature_names_outr r,r-r.)encr feature_namesZfeature_names2rrr"test_one_hot_encoder_feature_namests& rdcCs\t}tjddggtdj}|||}tddg||jdgd}tdd g|dS) Nuc❤t1Zdat2r<u x0_c❤t1Zx0_dat2un👍meZinput_featuresun👍me_c❤t1u n👍me_dat2)rrrobjectrAr+rar )rbrrcrrr*test_one_hot_encoder_feature_names_unicodes rgcCsdd}t|d}tjddggtdj}|||}tddg||jd gd }td d g|d d}t|d|}d}tj t |d|Wdn1s0YdS)z=Check the behaviour of `feature_name_combiner` as a callable.cSs|dt|S)N_)reprfeaturecategoryrrr name_combinerszHtest_one_hot_encoder_custom_feature_name_combiner..name_combiner)Zfeature_name_combinerNoneNr<z x0_'None'Zx0_NonerEreza_'None'Za_NonecSsdS)Nrrrjrrrwrong_combinerszItest_one_hot_encoder_custom_feature_name_combiner..wrong_combinerzMWhen `feature_name_combiner` is a callable, it should return a Python string.r)) rrrrfrAr+rar r,r- TypeError)rmrbrrcroerr_msgrrr1test_one_hot_encoder_custom_feature_name_combiners  rrcCstddggj}t}|jgdgd|dgdgksDJ||jdks\J|jgdgd||jdksJdS) Nrr )rrr r r?r?)r r%)rrr r r%r) rrrAr set_params get_paramsrrr)rr3rrrtest_one_hot_encoder_set_paramssrvcCsXtdd}||}tddd}||}t||t|rL|jdksPJ|S)Nr>rsFr?rZcsr)rrrrrrformat)rrbZXtr1ZXtr2rrrcheck_categorical_onehots    ryrdefr7abcr r|)rUrr|)rr r|)rFrHcat)rErIrr<)rFrrrErnan)Nrr)rEr r)NrN)mixednumericrfz mixed-nanzmixed-float-nanz mixed-Nonezmixed-None-nanzmixed-None-float-nan)ZidscCstt|dddgf}t|ddgddggtt|ddddgf}t|gdgdgtdd|}t|gdgdgdS) Nrr)rrrrrrrrr>rs)rrrrr)rrrrr)ryrrrrrr)rXtrrrrtest_one_hot_encoders rsparse_FTdropfirstcCsgdgdgdg}t||d}||}tj|td}t|||ddgddgd dgg}t|d |d }||}t|}t||||durrgdgdgdg}t||d d gddggdgd}||}tj|td}d|d<t|||ddgddgd dgg}t|ddgddgg|d}||}tj|td}d|d<d|dddf<t|||tgdgdg}td}t j t |d||Wdn1s0YdS)Nr}rz)r~r r|rrr<r r|rr r>)rr?rr~r{)6r|8)rr!r?)r rrr)rr?r!r rrrrr)Shape of the passed X data is not correctr)) rrrrrfr inverse_transformreescaper,r-r.)r!rrrrbX_trexpmsgrrrtest_one_hot_encoder_inversesH        rz X, X_transr r|rrrrrr\r]r^rFrrrrr)rrrrr)rrrrrcCs^t|d|}d}|r"t|d}tjt|d||Wdn1sP0YdS)zCheck that `inverse_transform` raise an error with unknown samples, no dropped feature, and `handle_unknow="error`. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/14934 rzqSamples \[(\d )*\d\] can not be inverted when drop=None and handle_unknown='error' because they contain all zerosrr)N)rr+rr,r-r.r)rX_transrrbrrrr?test_one_hot_encoder_inverse_transform_raise_error_with_unknownAs rcCsJtjddgddgddggtd}tddd }||}t|||dS) NrQrrSr r r< if_binaryFrr)rrrfrrr r)roherrrr&test_one_hot_encoder_inverse_if_binaryas   r)rrN reset_dropcCstjddgddgddggtd}t|dd}||||}|}|j|d t| ||t |||t||dS) NrQrrSr r r<Frr) rrrfrr+r/rartr rr)rrrrrrcrrrtest_one_hot_encoder_drop_reseths     rmethodr+r@@cCsJt}d}tjt|dt|||Wdn1s<0YdS)Nz'Expected 2D array, got 1D array insteadr))rr,r-r.getattr)rrr3rrrrtest_X_is_not_1DwsrcCsntd}|gd}t}dt|d}tjt|dt|||Wdn1s`0YdS)NrD)r r%rz+Expected a 2-dimensional container but got z instead.r))r,rLSeriesrtyper-r.r)rrNrr3rrrrtest_X_is_not_1D_pandass  rzX, cat_exp, cat_dtyper~r{r rHrI)rrrfstringz missing-floatzmissing-np.nan-objectzmissing-float-nan-objectcCs||dddfD]}tdd}||t|jts:Jt|j|D]l\}}|}t|drt|dsrJ|dd|ddksJn||ksJt |j |sFJqFqdS)Nr6r>rs) rr+ isinstance categories_listziptolistrr issubdtyper=)rZcat_exp cat_dtypeXirbresrZres_listrrrtest_one_hot_encoder_categoriess#   rzX, X2, cats, cat_dtypedrErFcint64r%rr r )NrEz)rErFr)rENr)rfrz object-stringzobject-string-nonezobject-string-nanzobject-None-and-nancCst|d}tgdgdg}t|||t|jdt|dksRJ|jd t|dkspJ|jdj |ksJt|d}t j t dd||Wdn1s0Yt||d}tgdgdg}t||||dS) Nrsrrrrrrrr(r)r?r!)rrr)rrrr rrrr?rrr=r,r-r.r+r/)rr2catsrr!rbrrrr)test_one_hot_encoder_specified_categoriess3  ( rcCstjddggtdj}tgdgd}tgdgdg}t||||t| |||j d gdksJt |j dj tjsJtd d ggj}tgd gd}d }tjt|d | |Wdn1s0YdS)NrErFr<)rFrErrsrrrrr )r rr z%Unsorted categories are not supportedr))rrrfrArr r+r/rrrrrr=object_r,r-r.)rrbrrrrr(test_one_hot_encoder_unsorted_categoriessrEncodercCsptdtjdgg}||d}tjddggtdj}tjtdd||Wdn1sb0YdS)zTest encoder for specified categories that nan is at the end. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27088 rrrsr<zNan should be the last elementr)N) rrrrfrAr,r-r.r+rrrbrrrr,test_encoder_nan_ending_specified_categoriess  rcCstjddgddggtdj}tgdgdgd}tgd gd g}t||||jd gdksvJt |jdj tj sJ|jd  gdksJt |jd j tj sJdS) NrErFrr r<r)rrr rs)rrrrrr)rrrrrrr) rrrfrArr rrrrrr=rrrbrrrr7test_one_hot_encoder_specified_categories_mixed_columns$srcCsDtd}|ddgddgd}t|}t|gdgdgdS) NrDrErFrr rGrJrK)r,rLrMryr)rNrOrrrrtest_one_hot_encoder_pandas1s rzdrop, expected_namesx0_cx2_br)rZx1_2r)rr rFx0_bZx2_a)rbinarymanualcCs:gdgdg}t|d}|||}t||dS)N)rr rE)rFr rFr)rr+rar )rZexpected_namesrrrcrrr'test_one_hot_encoder_feature_names_drop:s   rcCsddgddgddgg}tgdgdgdg}tdd g}td d d }||}t|j|t||d dgddgddgg}tddgddgddgg}td dg}td d d }||}t|j|t||dS)NrUyesnor[)rrrrr7)rrrrrrFrtruerEfalserr)rrrrr drop_idx_r)rexpectedZexpected_drop_idxrresultrrr*test_one_hot_encoder_drop_equals_if_binaryLs        r)rUr r|)rrr|)rrrfcCsTt}tjgdgdgdd}t|||dtdd}t|||dS)Nrrrrrrrr<float64)rrrr rZastyperrrrtest_ordinal_encoderds  r)rfrzobject-string-catcCst|d}tdgdgg}t|||t|jdt|dksJJ|jdt|dkshJ|jdj |ks|Jt|d}t j t dd| |Wdn1s0YdS)Nrsrrrr(r))rrrr rrr?rrr=r,r-r.r+)rr2rrrbrrrr)test_ordinal_encoder_specified_categoriesus  rcCsgdgdg}t}||}tj|td}t|||tgdgdg}td}t j t |d||Wdn1s0YdS)Nr}rzr<)rrrr rJrr)) rrrrrfr rrrr,r-r.)rrbrrrrrrtest_ordinal_encoder_inverses  rcCstddd}tjddgddgdd ggtd }tjdd gd dgddggtd }||||}tjd dgddgddggdd }t||||}tjddgddgddggtd }t||dS)Nuse_encoded_valuer! unknown_valuerExrFyrrr<ZxyZblar rrr)rrrrfr+r/r r)rbX_fitr X_trans_encr X_trans_invinv_exprrr+test_ordinal_encoder_handle_unknowns_strings         rr=cCstddd}tjddgddgdd gg|d }tjdd gd dgddgg|d }||||}tjddgddgd d ggdd }t||||}tjddgddgddggtd }t||dS)Nrrrr r r<rXrr)rrrr+r/r rrf)r=rbrrrrrrrrr,test_ordinal_encoder_handle_unknowns_numerics         rcCs`tdtjd}tdgdgdgg}|||dgdgdgg}t|dgdgtjggdS)Nrrrr r r%r)rrrrr+r/r )rbrrrrr(test_ordinal_encoder_handle_unknowns_nans  rcCsbtdtjtd}tdgdgdgg}tjtdd||Wdn1sT0YdS)Nr)r!rr=rr r z'dtype parameter should be a float dtyper)) rrrintrr,r-r.r+)rbrrrr8test_ordinal_encoder_handle_unknowns_nan_non_float_dtypes rcCshtjgdgtdj}gd}t|d}d}tjt|d||Wdn1sZ0YdS)N)LowMediumHighrrr<)rrrrsz*Shape mismatch: if categories is an array,r)) rrrfrArr,r-r.r+)rrrbrrrr+test_ordinal_encoder_raise_categories_shapes  rcs|tddtjgdgdgdd}tjddgd d ggd dtjddgd d ggddtd d gddggtddgddggtjdd gd d ggddfD]BtfddtdDsJt|qddgd d ggtfddtdDsJt|dd gd d ggtfddtdDsdJt|dS)Nr>rs)rrrr)rrrrrr<rr r r%rrErFrrabcdrfcsg|]}j|jjkqSrrr=.0irrbrr z'test_encoder_dtypes..cs"g|]}tj|jtjqSr)rrrr=integerrrbrrrrcsg|]}j|jdkqS)rfrrrrrrr) rrrr+allranger r/r)rrrrtest_encoder_dtypess&       rcstd}tddtjgdgdgdd}|jdd gd d gd d gddd}|tfddtd DsxJt  | ||dd gddgddgd}|dj |dj |dj g|tfddtd DsJt  | |dS)NrDr>rs)rrrrrr)rrrrrrrr<rr r r%rrrHrICrcsg|]}j|jdkqS)rrrrrrrrz.test_encoder_dtypes_pandas..rErFrrrHrIrcs g|]}j|j|kqSrrrZX_typerbrrrr) r,rLrrrrMr+rrr r/rr=)rNrrrrrtest_encoder_dtypes_pandass  "   rcCsVt}ddgddgg}t$td||Wdn1sH0YdS)NrQrrSr r&)rwarningscatch_warnings simplefilterr)rbrrrrtest_one_hot_encoder_warnings   r cCsddgddgddgg}t|ddddgddggd}||d dgg}tddgg}d }tjt|d ||}Wd n1s0Yt||d S) z,Check handle_unknown='warn' works correctly.rErrFr rFr$rrr!r?rqFound unknown categories in columns \[0\] during transform. These unknown categories will be encoded as all zerosr)N rr+rrr,warns UserWarningr/r)rrrX_testrBwarn_msgrrrrtest_ohe_handle_unknown_warn%s  (r missing_valuec Csrdddd|g}t|d}gdgddddd|gg}||}gdgd gd g}t|||j|uspJd d t|j|jD}||}t j |t d }t |drZt|dd|ddt |dsJt |dsJt|ddddf|ddddft|dddf|dddft |dsFJt |dsnJnt||t||dS)Nr{rXr rr)r~rXr r|rE)r{rXrr|rE)rrrrr)rrrrrrcSsg|]\}}||qSrr)rrrkrrrrMsz4test_one_hot_encoder_drop_manual..r<r6)r6r6) rrrr rrrrrrrrfr) rZ cats_to_droprbrZtransrZ dropped_catsZ X_inv_transZX_arrayrrr test_one_hot_encoder_drop_manual?s2     *" r)r~r rTrEcCs\t|d}d}tjt|d,|gdgdgdgWdn1sN0YdS)Nrz-`drop` should have length equal to the numberr)r}rz)r{r ;)rr,r-r.r+)rrbrqrrrtest_invalid_drop_lengthds rdensityrZdenserEr rFrcCst|d}t||d}gdgdg}||||t|j|j|dkr^t|jdn0t||j|jD]\}}}|t||ksnJqnt|jtj sJ|jj t ksJdS)Nrr)rrrErrr) rr+r rrrrrrZndarrayr=rf)rrZohe_baseZohe_testrZdrop_catZdrop_idxZcat_listrrrtest_categoriesls     rcCs|jjsJdS)N)Z__sklearn_tags__Z input_tags categorical)rrrr"test_encoders_has_categorical_tagssrkwargsmax_categories min_frequency g(\?r)rr rXr?r>rErFrrc Cstdgddgddgddgdgj}tf|d d d ||}t|jgd gdgdgdgdgd gg}tddgddgddgddgddgg}||}t||dddgdgdD}| |}t||| } tddg| dS)zpTest that different parameters for combine 'a', 'c', and 'd' into the infrequent category works as expected.rErrFrrrUrr r#F)r?r!rrErrerrcSsg|] }|gqSrrrcolrrrrrz2test_ohe_infrequent_two_levels..infrequent_sklearnr%rx0_infrequent_sklearnN rrrArr+r infrequent_categories_r/rrra) rr?X_trainrrrr expected_invX_invrcrrrtest_ohe_infrequent_two_levelss(2(    r.cCstdgddgddgddgdgj}td d d |d |}|jd |jd dksdJtdgdgg}||}td gdgg|| }t dg|| |}t dgdgg|dS)z3Test two levels and dropping the frequent category.rErrFrrrUrr r#Fr r!rrrrrr(r'N) rrrArr+rrr/rrar r)rr+rrrrc X_inverserrr,test_ohe_infrequent_two_levels_drop_frequents"2   r1cCstdgddgddgddgdgj}td d d |d }d |dd}tjt|d||Wdn1s0YdS)z_Test two levels and dropping any infrequent category removes the whole infrequent category.rErrFrrrUrr r#Fr r/Unable to drop category r( from feature 0 because it is infrequentr)NrrrArr,r-r.r+rr+rrrrr5test_ohe_infrequent_two_levels_drop_infrequent_errorss2r6rgQ?g{Gz?rc Cstdgddgddgddgdgj}tfd d d ||}t|jddggdgdgdgdgd gg}tgd gdgdgdgdg}||}t||dgdgdgdgdgg}| |}t||| }tgd|dS)zkTest that different parameters for combing 'a', and 'd' into the infrequent category works as expected.rErrFrrrUrr r#Fr!rr$rrrrrr')rrr(Nr)) rr+rrrrr,r-rcrrr test_ohe_infrequent_three_levelss.2(    r9cCstdgddgddgddgdgj}td d d|d |}tdgdgdgg}td d gd d gd d gg|||jdd|d}tj t |d"|dgdgg}Wdn1s0Ytd d gd d gg|dS)z5Test three levels and dropping the frequent category.rErrFrrrUrr r#Fr/rrr"r'r(r)r$N) rrrArr+rr/rtr,rr)rr+rrrrrrr.test_ohe_infrequent_three_levels_drop_frequents 2"0r:cCstdgddgddgddgdgj}td d d|d }d |d d}tjt|d||Wdn1s0YdS)z7Test three levels and dropping the infrequent category.rErrFrrrUrr r#Fr/r2rr3r)Nr4r5rrr7test_ohe_infrequent_three_levels_drop_infrequent_errorss2r;cCstdgddgddgddgdgj}td d dd |}t|jddggdgdgdgdgg}tgd gd gdgd g}||}t||dgg}d}t j t |d||Wdn1s0YdS)zmTest that different parameters for combining 'a', and 'd' into the infrequent category works as expected.rErrFrrrUrr r&F)r!rrrr8rbadz.Found unknown categories \['bad'\] in column 0r)N) rrrArr+r r*r/rr,r-r.)r+rrrrrrrr(test_ohe_infrequent_handle_unknown_error's2"  r=cCstjdgddgdgtdj}tfgdgddd ||}dgd gd gd gdgg}td dgdd gdd gdd gdd gg}||}t||dddgg}dgd gg}|D].}|j|d|tdgd gg||qdS)zG'a' is the only frequent category, all other categories are infrequent.rErr$r[r<rrrErFFr#r?rr!rFrrrrrrrN) rrrfrArr+r/rrt)rr+rrrrZdropsrrrr5test_ohe_infrequent_two_levels_user_cats_one_frequent?s&"(    r@cCstjdgddgddgddgdgtd j}tgd gd d d d|}t|jgdgdgdgdgdgdgg}tddgddgddgddgddgg}||}t ||dddgdgdD}| |}t||dS)zFTest that the order of the categories provided by a user is respected.rErrFrrrUrr r<r>Fr#r r?rr!r)rrrEr$rrcSsg|] }|gqSrrr%rrrrqrz.r'r%N rrrfrArr+r r*r/rrr+rrrrr,r-rrr(test_ohe_infrequent_two_levels_user_cats[s&*(   rDcCstjdgddgddgddgdgtd j}tgd gd d dd |}t|jddggdgdgdgdgdgg}tgdgdgdgdgdg}||}t ||dgdgdgdgdgg}| |}t||dS)zTest that the order of the categories provided by a user is respected. In this case 'c' is encoded as the first category and 'b' is encoded as the second one.rErrFrrrUrr r<rrrFrEFr#rAr$rr8rr'NrBrCrrr*test_ohe_infrequent_three_levels_user_catsvs0*(   rFcCsbtjgdgdf}tdddd}||ddgddgg}||}t|gd gd gd S) zaTest infrequent categories where feature 0 has infrequent categories, and feature 1 does not. rrr r r r r rr rrrrrrrrrr rF)rrrrrrrrr)rrrrN)rc_rr+r/r)rrrrrrrtest_ohe_infrequent_mixeds   rKc Cstjgdgdgdf}tdddd}||}t|jdd d gt|jd d d gt|jd d |}tgd |gdgdgdgdgdgdgdgdgdg }t||gdgdg}| |}gdgdg}t||| |}tj gdgdgt d}t||tdddd |}tjtdd| |Wd n1st0Ygd gd!g}| |}gd"gdg}t||| |}tj gd#gd$gt d}t||d S)%z?Test infrequent categories with feature matrix with 3 features.rG) rrrrrrUrrr) rrrrrrrrrr>r r#r?rr!rrr rUN)Zx0_0Zx0_3r(Zx1_0Zx1_5Zx1_infrequent_sklearnZx2_0Zx2_1)rrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrr)r rr )r%rr )rrrrrrrr)rrrrrrrr)r r'N)r'rNr<r&r(r))rrr)r rUr)rrrrrrrr)r'r'r)r r'r)rrJrrrr r*rarr/rrrfr+r,r-r.) rrrrcrr X_test_transr-r,rrr'test_ohe_infrequent_multiple_categoriessn     *  rNc Cstd}|jgdgddddgd}tdd d d }||}t|jd d dgt|jdgdgdgdgdgdgdgdgdgdgdg }t|||jddgddgdddgd}gdgdg}| |}t||| |}t j ddgddggt d}t|||jddgddgdddgd}| |}gdgdg}t||| |}t j ddgddggt d}t||dS)zHTest infrequent categories with a pandas dataframe with multiple dtypes.rD rEfrrPrPrErrFrF rr rrUrUrXrr r)strrrRrcolumnsr>r r#rLrrErFrrr rX)rrrrrr)rrrrrr)rrrrrr)rrrrrr)rrrrrrrPrXr'r<rrN)r,rLrMrrrr r*rr/rrrrf) rNrrrrrrMr-r,rrr.test_ohe_infrequent_multiple_categories_dtypessV          rWrZ)r rcCsptdgddgddgddgdgj}tfd d d |}|||dgg}t|d ggd S),All user provided categories are infrequent.rErrFrrrUrr r#Fr7rN)rrrArr+r/rrr+rrrrr$test_ohe_infrequent_one_level_errorsHs2 rZcCsbtjdgdgtdj}tfgdgddd||}|dgdgg}t|d gd ggd S) rXr$r r<r>Fr#r?rErN)rrrfrArr+r/rrYrrr5test_ohe_infrequent_user_cats_unknown_training_errorsVsr[zinput_dtype, category_dtype)ZOOZOUZUOZUUSOZSUZSS array_type)rrZ dataframec Cstjdgdgg|d}tjddg|dg}t|dd|}tdgdgdgdgg||d}||}tddgddgddgddgg}t||t|d|} | |}tdgdgdgdgg}t||d S) a"Check that encoding work with object, unicode, and byte string dtypes. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/15616 https://github.com/scikit-learn/scikit-learn/issues/15726 https://github.com/scikit-learn/scikit-learn/issues/19677 rFrEr<FrwrrrsN) rrrr+rr/rrr ) r;Zcategory_dtyper]rr?rrrroerrrtest_encoders_string_categoriesgs  "  r_cCs|tjdgdggdd}tjddgddg}t|dd}td}tjt|d ||Wd n1sn0Yd S) zCheck that this mixture of predefined categories and X raises an error. Categories defined as bytes can not easily be compared to data that is a string. rFrEUr<SFrwzjIn column 0, the predefined categories have type 'bytes' which is incompatible with values of type 'str_'.r)N) rrrrrr,r-r.r+)rr?rrrrr$test_mixed_string_bytes_categoricalss rbcCsPtjdd|d|ggtdj}tddd|}|}t|ddd |gdS) NrErFr<Fr"rr!Zx0_arZx0_)rrrfrArr+rar )rrrnamesrrr)test_ohe_missing_values_get_feature_namessrecCsrtd}|jgdtjdddtjgtdddd gd }tgd gd gd gdg}t|}t||dS)NrD)dogrNrr rr%r<)col1col2rgrhrS)rrrrrrr)rrrrrrr)rrrrrrr)rrrrrrr) r,rLrMrrrfloatryr)rNdfexpected_df_transrrrr%test_ohe_missing_value_support_pandass   rl pd_nan_typepd.NAznp.nanc Cstd}|dkr|jntj}|d|jdd|ddgddi}tgd gd gd gd gd g}td |d}| |}t ||t |j dksJt |j dddgdt|j ddsJdS)NrDrnrgrrErFrlr<)rrrr)rrrr)rrrrrIFrcrrr6r)r,rLNArrrMrrrrrlenrr isnan)rmr!rNpd_missing_valuerjrkrdf_transrrr1test_ohe_missing_value_support_pandas_categoricals(    rtcCsddgddgddgg}tdd|d}||}tgd gd gd g}t||d d gg}tgd g}d}tjt|d||}Wdn1s0Yt||| |}t |tjddggt ddS)zZCheck drop='first' and handle_unknown='ignore'/'infrequent_if_exist' during transform.rErrFr rrFrrr!rr)rrrrr tFound unknown categories in columns \[0, 1\] during transform. These unknown categories will be encoded as all zerosr)Nr< rrrrrr,rrr/rr rfr!rrrrBrrr-rrr/test_ohe_drop_first_handle_unknown_ignore_warnss*   (  rycCsddgddgddgg}tdd|d}||}tgd gd gd g}t||d d gg}tgdg}d}tjt|d||}Wdn1s0Yt||| |}t |tjddggt ddS)zDCheck drop='if_binary' and handle_unknown='ignore' during transform.rErrFr rrFrurIrrJrr )rrrrrvr)Nr<rwrxrrr3test_ohe_drop_if_binary_handle_unknown_ignore_warnss*   (  rzcCsddgddgddgg}tdd|ddgddggd}||d dgg}tddgg}d }tjt|d ||}Wd n1s0Yt||d S) znCheck drop='first' and handle_unknown='ignore'/'infrequent_if_exist' during fit with categories passed in.rErrFr rrFr rrr)Nr)r!rrrrBrrrrr'test_ohe_drop_first_explicit_categories&s  (r{cCstd}|jgdgddddgd}tdd }|jdd d }tjt|d ||Wd n1sr0Y||tjt|d | |Wd n1s0Yd S)zJRaise informative error message when pandas output and sparse_output=True.rDr)rrFrF)rErFrErFrSTrr/zxPandas output does not support sparse data. Set sparse_output=False to output pandas dataframes or disable Pandas outputr)N) r,rLrMr set_outputr-r.rr+r/)rNrjrrrrr'test_ohe_more_informative_error_messageAs    ( r~cCslttjdddggj}ttjd}dtj}tjt|d| |Wdn1s^0YdS)zDTest ordinal encoder with nan passthrough fails when dtype=np.int32.rrr<zdThere are missing values in features \[0\]. For OrdinalEncoder to encode missing values with dtype: r)N) rrrrArint32r,r-r.r+)rr^rrrrBtest_ordinal_encoder_passthrough_missing_values_float_errors_dtypeUs rencoded_missing_valuercCstjtjdddggtjdj}t|d|}t|jdks@Jt |jdddtjg| |}t ||gdgdgdgg| |}t ||dS) z.Test ordinal encoder with nan on float dtypes.rrr<rrrrN) rrrrrArr+rprrr/r)rrr^rr0rrr5test_ordinal_encoder_passthrough_missing_values_floatcs  rc Cs"td}|dkr|jntj}|d|jdd|ddgddi}t|d |}t |j d ksbJt |j d d d gdt |j d dsJ| |}t|dgdg|gdgdgg||}|jdksJt |d dd fddgt |d d d fddgt |dsJd S)z0Check ordinal encoder is compatible with pandas.rDrnrgrrErFrlr<rrrNr rr6@rr)rrr r)r,rLrorrrMrrr+rprr rqr/rrr)rmrrNrrrjr^rsr0rrr=test_ordinal_encoder_missing_value_support_pandas_categoricalus"   rr)zobject-None-missing-valuezobject-nan-missing_valueznumeric-missing-valuecCst|d}tdgtjgg}t||||jdj|ksBJt|d}tj t dd| |Wdn1sz0YdS)z.Test ordinal encoder for specified categories.rsrrr(r)N) rrrrr rrr=r,r-r.r+)rr2rrr^rrrr=test_ordinal_encoder_specified_categories_missing_passthroughs&  rcCsptjgdtdg}||d}tjddggtdj}tjtdd||Wdn1sb0YdS) zTest encoder for specified categories have duplicate values. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27088 )rErFrEr<rsrErFz5the predefined categories contain duplicate elements.r)N)rrrfrAr,r-r.r+rrrr+test_encoder_duplicate_specified_categoriess rzX, expected_X_trans, X_testrr)rrr)rrrr)rrErF)rrrcCs8tddd}||}t||t||dggdS)z>Test the interaction between missing values and handle_unknownrr6rgN)rrrr/)rZexpected_X_transrr^rrrr/test_ordinal_encoder_handle_missing_and_unknowns   r csr_containercCstgdgdg}||}t}d}tjt|d||Wdn1sV0Ytjt|d||Wdn1s0Y||}||}tjt|d||Wdn1s0YdS)zCheck that we raise proper error with sparse input in OrdinalEncoder. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/19878 r rz2Sparse data was passed, but dense data is requiredr)N) rrrr,r-rpr+rr)rrZX_sparseencoderrqrrrrrtest_ordinal_encoder_sparses(( rcCstgdddtjf}tgdgddd}||tgdgdd}tjtd d ||Wdn1sz0YdS) zCheck OrdinalEncoder.fit works with unseen category when `handle_unknown="use_encoded_value"`. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/19872 )rrrrr rN)r6rrrr)r?r!rr&rr(r))rrZnewaxisrr+r,r-r.)rr^rrr-test_ordinal_encoder_fit_with_unseen_categorys  rr+ZAAOr`rcCs4tddd}||||}t|ddggdS)zChecks that `OrdinalEncoder` transforms string dtypes. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/19872 rirrN)rr+r/r)r+rrbrrrr1test_ordinal_encoder_handle_unknown_string_dtypes&s   rcCsbtgddd}t|}t|jtj|ddj| |}t|dgdgdgdggdS) zCheck that `OrdinalEncoder` accepts Python integers that are potentially larger than 64 bits. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/20721 )l HP 1&l H]viel :?i}GalIRK2e6kr6rr)Zaxisr r N) rrr8rr+r rsortrAr/)rrrrrr#test_ordinal_encoder_python_integerBs  rcCsHtd}gd}|jgdg|d}t|}|}t||dS)z-Check feature names out is same as the input.rD)rFrrErrSN)r,rLrMrr+rar )rNrdrrbZfeature_names_outrrr.test_ordinal_encoder_features_names_out_pandasVs   rcCstjdgdgtjggtd}tdtjdd|}||}t|dgdgdggtjd gtjggtd}||}t|tjgdgg||}|ddd usJt |ddsJd S) zECheck interactions between encode_unknown and missing value encoding.rErFr<rr!rrrrrN) rrrrfrr+r/rrrq)rr^rrrM X_roundtriprrr0test_ordinal_encoder_unknown_missing_interactionbs    r with_pandascCstjddgddgdtjggtd}d}|rPtd}|j|d d gd }|d }n|d }tdd}tjt |d| |Wdn1s0YdS)zXCheck OrdinalEncoder errors when encoded_missing_value is used by an known category.rErfrFrrr<zTencoded_missing_value \(1\) is already used to encode a known category in features: rDletterZpetrSz \['pet'\]z\[1\]rrr)N) rrrrfr,rLrMrr-r.r+)rr error_msgrNr^rrr0test_ordinal_encoder_encoded_missing_value_errors"   rz4X_train, X_test_trans_expected, X_roundtrip_expected1c Cstdtjtjd|}tdgtjgdgg}||}t||||}|jd}t |D]V}||df} ||df} | dur| dusJq`t | rt | sJq`| | ks`Jq`dS)znCheck transform when unknown_value and encoded_missing_value is nan. Non-regression test for #24082. rrrrFrN) rrrr+rr/rrrrrrq) r+ZX_test_trans_expectedZX_roundtrip_expectedr^rrMrZ n_samplesrZ expected_valvalrrr9test_ordinal_encoder_unknown_missing_interaction_both_nans(       rcCstd}|ddgddgd}t}|jddd}tjt|d ||Wd n1sf0Ytd d jd d}td d jdd}||}||}t| |t | |j d S)z*Check OneHotEncoder works with set_output.rDrErFrr rGr|zCPandas output does not support sparse data. Set sparse_output=Falser)NFrdefault) r,rLrMrr}r-r.rrto_numpyr rarT)rNrOrr*Z ohe_defaultZ ohe_pandas X_defaultX_pandasrrrtest_one_hot_encoder_set_outputs  (  rcCsttd}|ddgddgd}tjdd}tjdd}||}||}t||t| |j d S) z+Check OrdinalEncoder works with set_output.rDrErFrr rGrr|N) r,rLrMrr}rrrr rarT)rNrOZ ord_defaultZ ord_pandasrrrrrtest_ordinal_set_outputs   rcCstgdddgg}t|d}|ddggt|t|jks@Jt|jD]$\}}|jtks`Jt|||qJdS)zjCheck that the categories_ dtype is `object` for string categories Regression test for gh-25171. )asZmmasZeasZrasZacsr2rsrN)rr+rpr enumerater=rfr )r?rbnrrrr test_predefined_categories_dtypes rcCs~tjdgdgtjggtd}tdd|}t|dgdgdggtddd |}td gg}||}t|dggd S) zBCheck missing value or unknown encoding can equal the cardinality.rfrr<r rrrrrsnakeN) rrrrfrrrr+r/)rrrbrrrr1test_ordinal_encoder_missing_unknown_encoding_maxs rcCstjdgddgddgddgddgdgtdj}tdd d d |}t|gd |jd |j d dksJtjdgddgddgdgtdj}tdd dd |}t|dg|jd |j d dksJtjdgddgddgddgddgdgtdj}tdd dgd |}t|gd|jd |j d dkstJtdd dd |}t|gd|j dusJdS)zkCheck drop_idx is defined correctly with infrequent categories. Non-regression test for gh-25550. rEr rFr%rrr$r<Fr)r rr)rx0_dx0_er(rrUrr()rrrr(N)rrrrr() rrrfrArr+r rarr)rrrrr#test_drop_idx_infrequent_categoriess44 ,4 rcCstdgddgddgddgdgj}tfd d d ||}t|jgd gt|jddggdgdgdgdgd gg}dgdgdgdgd gg}||}t ||| |}dgdgdgdgdgg}t||dS)zGTest parameters for grouping 'a', and 'd' into the infrequent category.rErrFrrrUrr rr6rr"rr rrr'N) rrrArr+r rr*r/rr)rr+ordinalrexpected_transrr0expected_inverserrr,test_ordinal_encoder_infrequent_three_levels6s,2   rcCstjdgddgddgddgdgtd j}tgd gdd d d |}t|jgd gt|jddggdgdgdgdgdgg}dgdgdgdgd gg}| |}t ||| |}dgdgdgdgdgg}t||dS)zTest that the order of the categories provided by a user is respected. In this case 'c' is encoded as the first category and 'b' is encoded as the second one. rErrFrrrUrr r<rErr6)r?rr!rrr rrr'N) rrrfrArr+r rr*r/rr)r+rrrrr0rrrr6test_ordinal_encoder_infrequent_three_levels_user_cats]s2*   rcCstgdgdf}tdd|}t|jdddg|jddusLJddgddgg}ddgddgg}||}t||||}tj ddgd dggt d }t||dS) zETest when feature 0 has infrequent categories and feature 1 does not.rGrHr rrrr Nr'r<) rZ column_stackrr+r r*r/rrrrf)rrrrrr0rrrr%test_ordinal_encoder_infrequent_mixeds   rcCs td}|gd}|jgdgd|jdgddgdd gd g|d d gd d }tdd|}t|jdddgt|jdgdt|jdd d g|jgdgd|jdgd gd gdg|d d gd d }gdgdgdgdg}| |}t ||dS)zHTest infrequent categories with a pandas DataFrame with multiple dtypes.rD)birdrrfrrOrQrfr%rr rrr<)rRrrrSrrrErFrrUr )rErFrPr)rXrrUr)r r r)r r r )rrr r8N) r,rLZCategoricalDtyperMrrr+r r*r/r)rNZcategorical_dtyperrrrrrrr:test_ordinal_encoder_infrequent_multiple_categories_dtypess:     rcCstjdgddgddgddgdtjggtd j}td d d dd |}t|jgd gtjdgdgdgdgdgtjggtd }dgdgdgdgd gdgg}| |}t ||dS)zJCheck behavior of unknown_value and encoded_missing_value with infrequent.rErrFrrrUrr r<rr )r!rrrr#r$rrN) rrrrfrArr+r r*r/r)r+rrrrrrr.test_ordinal_encoder_infrequent_custom_mappings 2( rcCstjdgddgddgddgdgtd j}tfi|d d d |}td d d |}dgdgdgdgd gg}t||||dS)zMAll categories are considered frequent have same encoding as default encoder.rErrFrrrUrr r<rr6rr$NrrrfrArr+rr/)rr+Zadjusted_encoderZdefault_encoderrrrr!test_ordinal_encoder_all_frequents& * rdcCstjdgddgddgddgdgtd j}tfi|d d d |}dgdgdgdgd gg}t||dgdgdgdgd ggdS)zAWhen all categories are infrequent, they are all encoded as zero.rErrFrrrUrr r<rr6rr$rNr)rr+rrrrr#test_ordinal_encoder_all_infrequents * rcCstjtjgddgddgddgdggtdj}td d |}tjdddtjggtdj}||}t|d gd gd gtjggdS)z5Check behavior when missing value appears frequently.rrfrUrrrdeerr<r rr rrN rrrrfrArr+r/rrrrrrrr-test_ordinal_encoder_missing_appears_frequent s, rcCstjtjgdgddgddgdgdgdd gdgtd j}td d |}tjddgdd gtjd gdd gddggtd }||}t|d dgd dgtjdgddgddggdS)z7Check behavior when missing value appears infrequently.rfrUrrrrredrgreenr<r%)r r rrNrrrrr/test_ordinal_encoder_missing_appears_infrequent s$& rcCsbtjdgdgdggtd}|gdgd}tt||Wdn1sT0YdS)a!Check that we raise a `NotFittedError` by calling transform before fit with the encoders. One could expect that the passing the `categories` argument to the encoder would make it stateless. However, `fit` is making a couple of check, such as the position of `np.nan`. rHrIrr<rrsN)rrrfr,r-rr/)rrrrrrtest_encoder_not_fitted3 s  r)rr numpyrr,ZscipyrZsklearn.exceptionsrZsklearn.preprocessingrrZsklearn.utils._missingrZsklearn.utils._testingrrr Zsklearn.utils.fixesr r markZ parametrizer4r9rZfloat32rrCrPrdrgrrrvryrrfrrirrrrrrrrrZstr_rrArrrrrrrrrrrrrrrrrrrr rrrrrr.r1r6r9r:r;r=r@rDrFrKrNrWrZr[r_rbrerlrtryrzr{r~rrrrrrrrrrrrrr@rrrrrrrrrrrrrrrrrrrrs        <     /*     & &* ! &1               $           $[A      % $     "       !    $  $  0