a jN @sDdZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZm Z ddl!m"Z"m#Z#eddd\Z$Z%ee$e%dd\Z$Z%e&e$Z$gdZ'dhdde(DBZ)doddZ*ej+,deddZ-dd Z.ej+,d!ge#e"d"d#Z/d$d%Z0ej+,d&e'ej+,d'ed(d)Z1d*d+Z2ej+,d,d-d.d/Z3d0d1Z4d2d3Z5d4d5Z6d6d7Z7ej+,d8d9d:gd;d<Z8ej+,d=e#d>d?Z9ej+,d@e'dAdBZ:dCdDZ;dEdFZgej>dIggfd'dHidIdJgdJdIggfidIdJgdKdLggfgdMdNZ?ej+,d=e#dOdPZ@ej+,d=e#dQdRZAdSdTZBdUdVZCdWdXZDej+,dYdZd[gej+,d\dd]gd^d_ZEd`daZFej+,dbdcddgdedfZGej+,dgdhdigdjdkZHej+,dld9d:gdmdnZIdS)pzF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)stats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal)CSC_CONTAINERSCSR_CONTAINERS ) n_samples random_state)r)kd_tree ball_treebruteautocCsh|]\}}|dqS)label).0_outr r HE:\soccer\ml_env\Lib\site-packages\sklearn/cluster/tests/test_hdbscan.py &r%Gz?cCs2tt|t}|dksJt|t|ks.JdS)N)lenset OUTLIER_SETr y)labels threshold n_clustersr r r$check_label_quality)s r0 outlier_typec Cstjtjd|}ddddd|}t|d}t|d}t}|dg|d<||g|d <t|}|j|k \}t |dd g||j | \}t |dd gt t dd t t d d } t|| } t | j|j| d S) O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingcSs||kSNr xr,r r r$9r&z#test_outlier_data..cSs t|Sr5)npisnanr6r r r$r8:r&rprobrrN)r9infnanrXcopyrfitlabels_ZnonzerorZprobabilities_listrange) r1ZoutlierZ prob_checkrr; X_outliermodelmissing_labels_idxZmissing_probs_idxZ clean_indices clean_modelr r r$test_outlier_data/s.     rKcCstt}|}tddd|}t||t|d}tjt |d"tdddtWdn1sn0Yd}d|d <d |d <tjt |d tdd |Wdn1s0YdS) zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedT)metricrBz*The precomputed distance matrix.*has shapematchNz'The precomputed distance matrix.*valuesr)rr<r<)r<rrM) r rArBr fit_predictrr0pytestraises ValueError)DZ D_originalr-msgr r r$test_hdbscan_distance_matrixOs 0rWsparse_constructorcCsfttt}|t|}t|d}d|||k<||}| t dd |}t |dS)zA Tests that HDBSCAN works with sparse distance matrices. 2rLrPN) rZ squareformZpdistrAr9maxrZscoreatpercentileflattenZeliminate_zerosrrQr0)rXrUr.r-r r r$#test_hdbscan_sparse_distance_matrixgs r]cCstt}t|dS)z Tests that HDBSCAN works with feature array, including an arbitrary goodness of fit check. Note that the check is a simple heuristic. N)rrQrAr0r-r r r$test_hdbscan_feature_arrayys r_algorMcCst|dt}t||dvr$dSttd}dttjdidt tjdiddidt tjdd d  |d}t|||d }|||j vrt t|tWdn1s0YnL|d krt t|tWdn1s0Yn |tdS) z Tests that HDBSCAN works with the expected combinations of algorithms and metrics, or raises the expected errors. ) algorithm)rrN)rrVr<p)rcw)Z mahalanobis seuclideanZ minkowski wminkowski)rarM metric_paramsrg)rrQrAr0rr r9eyeshapeonesget valid_metricsrRrSrTrCZwarns FutureWarning)r`rMr-Z ALGOS_TREESrhhdbr r r$test_hdbscan_algorithmss4 *  ,rpcCs&tt}|d}t|dddS)z Tests that HDBSCAN can generate a sufficiently accurate dbscan clustering. This test is more of a sanity check than a rigorous evaluation. 333333?gq= ףp?)r.N)rrCrAdbscan_clusteringr0)Z clustererr-r r r$test_dbscan_clusterings  rs cut_distance)皙??r<c Cstdd}tdd}t}tjdg|d<dtjg|d<tjtjg|d<t|}|j|d}t ||k}t |ddgt ||k}t |dgt t t d t ||}t||} | j|d} t | ||d S) r2r4rr3r<rrdr=)rtrN)rrArBr9r?r@rrCrrZ flatnonzerorrEr*rF) rtZ missing_labelZinfinite_labelrGrHr-rIZinfinite_labels_idxZ clean_idxrJZ clean_labelsr r r$#test_dbscan_clustering_outlier_datas       rwcCs.tddttjdidt}t|dS)z4 Tests that HDBSCAN using `BallTree` works. rfrbr<)rMrhN)rr9rkrArjrQr0r^r r r$!test_hdbscan_best_balltree_metrics rxcCs.tttddt}t|ts*JdS)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. r<min_cluster_sizeN)rr)rArQr*issubsetr+r^r r r$test_hdbscan_no_clusterssr|cCs\tdttdD]F}t|dt}dd|D}t|dkrtt||ksJqdS)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points rdr<rycSsg|]}|dkr|qS)rr )r!rr r r$ r&z1test_hdbscan_min_cluster_size..rN)rFr)rArrQr9minZbincount)rzr-Z true_labelsr r r$test_hdbscan_min_cluster_sizes  rcCs"tj}t|dt}t|dS)zA Tests that HDBSCAN works when passed a callable metric. rPN)r euclideanrrQrAr0)rMr-r r r$test_hdbscan_callable_metricsrtreerrcCsLtd|d}d}tjt|d|tWdn1s>0YdS)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rLrMraz%precomputed is not a valid metric forrNN)rrRrSrTrCrA)rrorVr r r$"test_hdbscan_precomputed_non_brutes r csr_containerc Cs ttj}t||t}|}t|j}t||tjdftj dffD]n\}}t}||d<t|j}t||dt |dksJ|}||d<t|j}t||qRd}t j t |d"tdd d |Wd n1s0Yd S) z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. r3r4rrrrz4Sparse data matrices only support algorithm `brute`.rNrrrN)rrCrArDr0rBrr9r?r@rrRrSrT) rZ dense_labelsZ _X_sparseZX_sparseZ sparse_labelsZ outlier_valr1ZX_denserVr r r$test_hdbscan_sparse s&  rracCsddg}tdd|dd\}}tdd|}t||j|jD]*\}}}t||d d d t||d d d q.zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.rNr)rarMNrr) rRrSrTrrCrArEr*r rmrr))Zmetric_callablerVZmetrics_not_kdr r r$ test_hdbscan_tree_invalid_metrics00 rcCsRtttdd}d}tjt|d|tWdn1sD0YdS)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. r<)rz min_samples (.*) must be at mostrNN)rr)rArRrSrTrC)rorVr r r$!test_hdbscan_too_many_min_samplessrcCs\t}tj|d<d}tdd}tjt|d||Wdn1sN0YdS)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserLrPrNN) rArBr9r@rrRrSrTrC)ZX_nanrVror r r$"test_hdbscan_precomputed_dense_nans   rrTFepsilonruc sd}t||ddgddgddggd\}t|}t|j|jd}|d|d|dh}|dd|dd |ddi}t|||||d fd d ttDfd d ttD} t | j } t | dS)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrryrdr(rr<condensed_treeclusterscluster_label_maprrcs$i|]}|t|kddqS)r)r9wherer!Z_y)r,r r$ r&z+test_labelling_distinct..csi|]}||qSr r r)first_with_labelr-r r$rr&N) r rrCrZ_single_linkage_tree_rzrrEr*r9Z vectorizerlr) Zglobal_random_seedrrrrAZestrrrZ y_to_labelsZaligned_targetr )rr-r,r$test_labelling_distincts4 rcCsd}d}tjdd|dfddd|dfddgtd }t||h|d|ddid dd }|d dk}t|t|d kkstJt||h|d|ddid dd }|d |k}t|t|d kksJdS)z Tests that the `_do_labelling` helper function correctly thresholds the incoming lambda values given various `cluster_selection_epsilon` values. r=g?rdr<)r=r<rur<r)r=r(rr<)r=rrqr<)ZdtypeTrvaluerN)r9arrayrrsum)rZ MAX_LAMBDArr-Z num_noiser r r$test_labelling_thresholdings:     rrrrcCsftjd}|d}t|}d}tjt|d"td|d|Wdn1sX0YdS)zCheck that we raise an error if the centers are requested together with a precomputed input matrix. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27893 r)drdz>Cannot store centers when using a precomputed distance matrix.rNrL)rMrN) r9rrr rRrSrTrrC)rrrAZX_disterr_msgr r r$0test_hdbscan_error_precomputed_and_store_centers%s   r valid_algorrcCstd|dtdS)zTest that HDBSCAN works with the "cosine" metric when the algorithm is set to "brute" or "auto". Non-regression test for issue #28631 cosinerN)rrQrA)rr r r$*test_hdbscan_cosine_metric_valid_algorithm5sr invalid_algocCsHtd|d}tjtdd|tWdn1s:0YdS)zTest that HDBSCAN raises an informative error is raised when an unsupported algorithm is used with the "cosine" metric. rrzcosine is not a valid metricrNN)rrRrSrTrQrA)rZhdbscanr r r$,test_hdbscan_cosine_metric_invalid_algorithm?s r)r')J__doc__numpyr9rRZscipyrZ scipy.spatialrZsklearn.clusterrZsklearn.cluster._hdbscan._treerrrZ sklearn.cluster._hdbscan.hdbscanrZsklearn.datasetsr Zsklearn.metricsr Zsklearn.metrics.pairwiser r Zsklearn.neighborsr rZsklearn.preprocessingrZ sklearn.utilsrZsklearn.utils._testingrrZsklearn.utils.fixesrrrAr,Z fit_transformZ ALGORITHMSitemsr+r0markZ parametrizerKrWr]r_rprsrwrxr|rrrrrrrrr?rrrrrrrrrrrr r r r$s                '        "  !        $)