a &jd @sxUddlmZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z ddlmZmZmZmZmZdd lmZdd lmZmZdd lmZmZmZmZmZmZe d Z!e"Z#e#$e%d e&edddZ'de(d<d.dddddddddddd d d!Z)d/d"ddddddddddd# d$d%Z*d0d&ddddddddddd' d(d)Z+d1d*ddddddddddd+ d,d-Z,dS)2) annotationsN) lru_cache)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDIANA_SUPPORTED_SIMILARTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bomis_multi_byte_encodingshould_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)scCs t| SN)rencodingr.r)keyz list[str]IANA_SUPPORTED_MB_FIRST皙?TF皙?zbytes | bytearrayintfloatzlist[str] | Noneboolr) sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainlanguage_thresholdenable_fallbackreturnc < Cs t|ttfs tdt||r>tj} tt t t t |} | dkrt d|rptt t | tt|dddgdgS|durtt d d |d d |D}ng}|durtt d d |dd |D}ng}| ||krtt d||| d}| }|dkr4| ||kr4t| |}t |tk} t |tk} | rbtt d| n| rvtt d| g}|rt|nd}|dur||tt d|t}g}g}t}tddt}tddt}d}t}d}d}d}d}d}d}t}t}t|\} }!| durB|d| tt dt |!| |dd|vr`|d|tD ]}"|r|"|vrqh|r|"|vrqh|"|vrqh||"d}#| |"k}$|$ot |"}%|"dvr|$stt d|"qh|"dvr |$s tt d|"qh|"|vr(tt d|"qhz t!|"}&Wn,t"t#fy`tt d|"YqhYn0|r|&s|tt$|"}'n tt%|"}'|'&|stt d|"|'|qh|r|&s||krtt d|"||qh|r|&stt d |"qh|& o| }(z| rD|&sDt'|%s(|dtd!n|t |!td!|"d"nf|(s|"d#kr|$rt'||"d"}#|#r|#dd$kr|#dd}#n"t'|%s|n|t |!d|"d"}#Wnbt(t)fy})zDt|)t)stt d%|"t'|)||"WYd})~)qhWYd})~)n d})~)00t*|$sdnt |!| t| |}*|&oL|#duoLt |#| k}+|+rbtt d&|"tt |*d'},t+|,d(},d}-d}.g}/g}0zt,||"|*||$|%|!|&|#|( D]t}1|/|1|0||1||odt |kod(kn|0d)|kr|-d7}-|-|,ks|$r|%sqqWnxt(t)fy})zZ|(rhtt d%|"t'|)||"WYd})~)qhtt d*|"t'|)|,}-d+}.WYd})~)n d})~)00|0rt-|0t |0nd}2|.sL| rL|&sL|2|krL|-|,krLz|td,dj.|"d-d.WnRt(yJ})z8tt d/|"t'|)||"WYd})~)qhWYd})~)n d})~)00|2|ks`|-|,krf||"|"t/vr|0t/|"tt d0|"|-t1|2d1d2d3| rh|"dd|d4d5fvrh|.sh|#dur(z&t'|%s|n|t |!d|"d"}#Wn,t(t)fytt d6|"YqhYn0| r(d}#t||"||$g|#|d7}3|"|krN|3}n|"dkr^|3}n|3}qh|(rz&t'|%sz|n|t |!d|"d"}#WnVt(t)fy})z8tt d%|"t'|)||"WYd})~)qhWYd})~)n d})~)00tt d8|"t1|2d1d2d3|& st$|"}4nt%|"}4|4 r4tt d9|"|4g}5|"dk rx|4 rRd:|4nd}6|/D]}1||1||6}7|5|7 qZt2|5}8|8 rtt d;|8|"t||"|2|$|8| r|"|ddfv r|#nd|d7}9||9| r|& s|2d|9j3| r:tt t | t|9gS||9t | r|du sp||v rd|v rd|v r|4}:|:du sJt d>|:j3| rtt t | t|:gS| s:|& s:|8 rt+d?d@|8DddAnd};|;dBk r:d|v r:d|v r:d+}|0|4tt dC|"|2|;| s|& r|+ r|#du rt |#| dDk r|"dEv rd|v rd|v rd+}tt dF|"|2t |#| t |#| d1|"| krht dG|"| rtt t | t||"gSqht |dk r| s| s| r"tt dH| rBt dI|j3||nd| rR|du sv| rl| rl|j5|j5k sv|du rt dJ||n| rt dK||| rt dL|4j3t |dn t dM| rtt t | |S)Naf Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z3Expected object of type bytes or bytearray, got: {}rzlrzfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSsg|]}t|dqSr6r7r8rrrr;wrz^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of (%i) byte(s).zIUsing lazy str decoding because the payload is quite large, (%i) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.)maxsizezIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.zY%s is deemed too similar to a code page that was already considered unsuited. Continuing!z2Encoding %s does not provide an IncrementalDecoderzbSkipping %s: definitive match already found, this encoding targets different languages (%s vs %s).zXSkipping %s: already accumulated %d same-family results after definitive match (cap=%d).zCSkipping single-byte %s: multi-byte definitive match already found.gArrAuz9Code page %s does not fit given bytes sequence at ALL. %szpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sTgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsr?r@z>%s does not decode the whole payload: fallback entry withheld.)Zpreemptive_declarationz=%s passed initial chaos probing. Mean measured chaos is %f %%z&%s should target any language(s) of %s,z We detected language %s using %sg{Gz?r%z.Encoding detection: %s is most likely the one.css|]\}}|VqdSrr)r9_vrrr rzfrom_bytes..)defaultg?zyDefinitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.g\(\?> utf_32_be utf_16_be utf_32_le utf_16_ler@Z utf_8_sigr4rAr?zjMulti-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)6 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelrlendebug removeHandlerrrlogjoinr&rr rappendsetrrrrinsertr!addrrModuleNotFoundError ImportErrorrr intersectionstrUnicodeDecodeError LookupErrorrangemaxrsumdecoder updateroundr rbest fingerprint)s~      $$ $!