a ñ»jozã@s`dZddlZddlZddlZddlZddlZddlZddlZddlZ ddl m Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZdZe e ¡Z!da"a#a$a%a&a'Gdd„de(ƒZ)dd„Z*Gdd„dƒZ+Gdd„deƒZ,dd„Z-Gdd„de ƒZ.dd„Z/dd„Z0Gdd„deƒZ1dS) a/ Overview ======== The multiprocess plugin enables you to distribute your test run among a set of worker processes that run tests in parallel. This can speed up CPU-bound test runs (as long as the number of work processeses is around the number of processors or cores available), but is mainly useful for IO-bound tests that spend most of their time waiting for data to arrive from someplace else. .. _multiprocessing : http://code.google.com/p/python-multiprocessing/ How tests are distributed ========================= The ideal case would be to dispatch each test to a worker process separately. This ideal is not attainable in all cases, however, because many test suites depend on context (class, module, or package) fixtures. The plugin can't know (unless you tell it -- see below!) if a context fixture can be called many times concurrently (is re-entrant), or if it can be shared among tests running in different processes. Therefore, if a context has fixtures, the default behavior is to dispatch the suite to a worker as a unit. Controlling distribution ======================== There are two context-level variables to control this default behavior: * If a context's fixtures are re-entrant, set `_multiprocess_can_split_ = True` in the context, and the plugin will dispatch tests in suites bound to that context as if the context had no fixtures. This means that the fixtures will execute concurrently and multiple times, typically once per test. * If a context's fixtures can be shared by tests running in different processes (such as a package-level fixture that starts an external http server or initializes a shared database) then set ``_multiprocess_shared_ = True`` in the context. These fixtures will then execute in the primary nose process, and tests in those contexts will be individually dispatched to run in parallel. How results are collected and reported ====================================== As each test or suite executes in a worker process, results (failures, errors, and specially handled exceptions like SkipTest) are collected in that process. When the worker process finishes, it returns results to the main nose process. There, any progress output is printed (dots!), and the results from the test run are combined into a consolidated result set. When results have been received for all dispatched tests, or all workers have died, the result summary is output as normal. Beware! ======= Not all test suites will benefit from, or even operate correctly using this plugin. For example, CPU-bound tests will run more slowly if you don't have multiple processors. There are also some differences in plugin interactions and behaviors due to the way in which tests are dispatched and loaded. In general, tests loaded under this plugin operate as if they were always in directed mode instead of discovered mode. For instance, doctests in test modules will always be found when using this plugin with the doctest plugin. But the biggest issue you will face is probably concurrency. Unless you have kept your tests as religiously pure unit tests, with no side-effects, no ordering issues, and no external dependencies, chances are you will experience odd, intermittent and unexplainable failures and errors when using this plugin. This doesn't necessarily mean the plugin is broken; it may mean that your test suite is not safe for concurrency.éN)ÚTextTestRunner)Úfailure)Úloader)ÚPlugin)Úbytes_)Ú ContextSuite)Ú test_address)Ú_WritelnDecorator)ÚEmpty)Úwarn)ÚStringIOc@seZdZddd„Zdd„ZdS)ÚTimedOutExceptionú Timed OutcCs ||_dS©N)Úvalue)Úselfr©rú?E:\soccer\ml_env\Lib\site-packages\nose/plugins/multiprocess.pyÚ__init__VszTimedOutException.__init__cCs t|jƒSr)Úreprr©rrrrÚ__str__YszTimedOutException.__str__N)r)Ú__name__Ú __module__Ú __qualname__rrrrrrr Ts r cCs|zZddlm}mat tjtj¡}|ƒ}t tj|¡|j|j|j|j |j f\aaaa a Wnt yvt dt ƒYn0dS)Nr)ÚManagerÚProcesszKmultiprocessing module is not available, multiprocess plugin cannot be used)ÚmultiprocessingrrÚsignalÚSIGINTÚSIG_IGNÚQueueÚPoolÚEventÚValueÚArrayÚ ImportErrorr ÚRuntimeWarning)rÚoldÚmrrrÚ _import_mp]sÿ ÿr*c@s,eZdZdd„Zdd„Zdd„Zdd„Zd S) ÚTestLetcCs:z| ¡|_Wnty Yn0| ¡|_t|ƒ|_dSr)ÚidÚ_idÚAttributeErrorÚshortDescriptionÚ_short_descriptionÚstrÚ_str)rÚcaserrrrms   zTestLet.__init__cCs|jSr)r-rrrrr,usz TestLet.idcCs|jSr)r0rrrrr/xszTestLet.shortDescriptioncCs|jSr)r2rrrrr{szTestLet.__str__N)rrrrr,r/rrrrrr+lsr+c@s8eZdZdZdZiZdd„Zdd„Zdd„Zd d „Z d S) Ú MultiProcessz¡|  ?| |¡|jj @| ¡|dur‚t d¡|D]}| &¡rb|jAdddqbt%|ƒD]>\}}| &¡rŠt d |¡| !¡| &¡rŠt d|¡qŠWnNt6t7fyt 8d ¡|D]}| &¡rî| /¡qî|r|‚n‚Yn0| S)!aExecute the test (which may be a suite). If the test is a suite, distribute it out among as many processes as have been configured, at as fine a level as is possible given the context fixtures defined in the suite or any sub-suites.z%s.run(%s) (%s)NzStarting %s workerszStarted worker process %séz5Waiting for results (%s/%s tasks), next timeout=%.3fs©Útimeoutz1Results received for worker %d, %s, new tasks: %dz)worker %s failed to remove from tasks: %szGot result for unknown task? %sz current: %srzjoining worker %sz!starting new process on worker %sz3Timeout - %s tasks pending (empty testQueue=%r): %sFÚasciigš™™™™™¹?zHWorker %d finished its work item, but is not exiting. Do we wait for it?TzTimed out worker %s: %sr…zterminating worker %szAll workers are deadzCompleted %s tasks (%s remain)z4parent received ctrl-c when waiting for test resultsz#Tearing down shared fixtures for %szTell all workers to stopÚSTOP©Úblockzfailed to join worker %sz=parent received ctrl-c when shutting down: stop all processes)BrgrhÚosÚgetpidrKÚpluginsZ prepareTestZsetOutputStreamrXr!r#Z _makeResultr‰r|r6Úranger•rxr~r=rAÚremoverNÚwarningÚextendrGr1ÚlistÚ consolidateZ stopOnErrorZ wasSuccessfulÚsetr?ÚjoinÚis_setÚemptyr Ú enumerateÚis_aliverrŒrrrŽÚclearÚ waitkilltimeÚerrorÚ terminateÚkillÚpidrrÚsleepÚminrrrsÚinforwrurvÚtearDownrtZ printErrorsZ printSummaryÚfinalizeÚput) rrmÚwrapperÚwrappedrr’r€Ú completedrQrr“r‚rÚir”Z total_tasksZ nexttimeoutZ thrownErrorr‘ÚaddrZ newtask_addrsÚ batch_resultÚmsgZ any_aliveÚwZ worker_addrZtimeprocessingZ startkilltimeÚer3ÚstoprJrrrÚrunsn ÿ  þ ÿ  þþ    ÿÿ  ÿû  ÿ ÿþþ ÿ ÿþ  ÿþ ÿÿ ûÿ   ÿþ ÿ &         ÿ  zMultiProcessTestRunner.runcCstd}t|tjjƒr.t|jdƒr.d|j_|jj}t  |¡}|j ||fdd|dur^|t |ƒ7}|durp|  |¡|S)NÚargFr›) rjrkr3rlrHrmZ descriptorrÃrZÚaddressr·r1rx)rr€r3rÃrƒrrrr}Ús   zMultiProcessTestRunner.addtaskcCsºt|dƒr| ¡\}}}n(t|dƒr6t|jƒ\}}}n td|ƒ‚g}|durp|durdtd|ƒ‚q˜| |¡n(tj |¡\}}|  d¡rŽ|}| |¡|durª| |¡d  t t |ƒ¡S)NrÄrozUnable to convert %s to addresszUnaddressable case %srú:) rHrÄrrortrxrÚpathÚsplitÚ startswithr§Úmapr1)r3ÚfileÚmodÚcallÚpartsÚdirnameÚbasenamerrrrÄès"       zMultiProcessTestRunner.addressccs¬t|dƒrt|jddƒsdSt|tƒr2| |j¡sJt|ddƒrJt|tjƒsŽt|tƒr†t |ƒ}t |ƒdkr†t|dddƒ|jkr†|d}|Vn|D]}|  |¡EdHq’dS)NroZ_multiprocess_TZ can_splitr–r) rHrzrorjrZ hasFixturesÚ checkCanSplitÚunittestZ TestSuiter¤r~rf)rrmZ containedr3rrrrfÿs* þ ý û ú  ÿþz MultiProcessTestRunner.nextBatchcCs|sdSt|ddƒrdSdS)zÉCallback to check whether the fixtures found in a context or ancestor are ones we care about. Contexts can tell us that their fixtures are reentrant by setting _multiprocess_can_split_.Fr{T©rz)roZfixtrrrrÐs  z$MultiProcessTestRunner.checkCanSplitcCs t|ddƒ}|sdSt|ddƒS)NroFrerÒ)rr3rorrrrp#s z%MultiProcessTestRunner.sharedFixturescCsèt d|¡z|\}}}}}Wn2tyPt d|¡tjt ¡Ž|ƒYdS0|j |¡|j |7_ |j   |¡|j   |¡t | ¡ƒD]B\}\} } } ||jvr¸g| | f|j|<|j|\} } } |   | ¡qt d||j ¡dS)Nzbatch result is %szresult in unexpected format %szRan %s tests (total: %s))rgrhrNrrnrurvrXÚwriteÚtestsRunÚfailuresr£Úerrorsr¤ÚitemsÚ errorClasses)rr‚r½ÚoutputrÔrÕrÖrØÚkeyÚstorageÚlabelÚisfailZ mystorageZ_junkrrrr¥)s"        z"MultiProcessTestRunner.consolidate)rrrr­rr|r•rÂr}Ú staticmethodrÄrfrÐrpr¥Ú __classcell__rrrcrrZÏs (?  rZc CsfzBzt|||||||||| ƒ WWSty>t d|¡Yn0Wnty`t d|¡Yn0dS)Nz&Worker %s keyboard interrupt, stoppingz%Worker %s timed out waiting for tasks)Ú__runnerrrrgrhr ) Úixrr’rŒrrŽr“rVÚ resultClassrKrrrr[=s  þ  r[c  sØt ˆ¡‰ˆ ¡} tdurBtD]"} | ƒ} |  | i¡ˆj | ¡qˆj ˆjˆ¡ˆj  ¡t   d|t   ¡¡|ˆd} t| j_‡‡fdd„}‡‡fdd„}dd„}t|d ƒD]\}}| ¡rÒt  d |¡qÈ|ƒ}|  |g¡}ˆ|_g|_||_t   d |||¡zZ|dur|t|ƒ}t|ƒ|_t ¡|_||ƒtd ƒ|_| |||j||ƒf¡WnXty<}zÂt|tƒ}|rŠ| ¡t |jƒrê|r¢d }nd}t  |||¡td ƒ|_t!j"t# $¡Ž|ƒ| |||j||ƒf¡n6|röd}nd}t   |||¡| |||j||ƒf¡|s(‚WYd}~n†d}~0t%yftd ƒ|_t  d|¡‚YnTt&y¸td ƒ|_t  d|¡t!j"t# $¡Ž|ƒ| |||j||ƒf¡Yn0ˆj'r®qÈq®t   d|¡dS)NzWorker %s executing, pid=%d)rKcsˆjˆjdS)Nr—)rAr=r)rKrrrrA^sz__runner..getcs4ttƒƒ}ˆ|dˆjˆd}ˆj |¡}|r0|S|S)Nr–)Z descriptionsrYrK)r r rYrŸZprepareTestResult)rXr‚Z plug_result)rKrârrÚ makeResultas  ÿ z__runner..makeResultcSspdd„|jDƒ}dd„|jDƒ}i}t|j ¡ƒD]&\}\}}}dd„|Dƒ||f||<q2|j ¡|j|||fS)NcSsg|]\}}t|ƒ|f‘qSr©r+©Ú.0r„ÚerrrrrÚ lóz+__runner..batch..cSsg|]\}}t|ƒ|f‘qSrrärårrrrèmrécSsg|]\}}t|ƒ|f‘qSrrärårrrrèqré)rÕrÖr¤rØr×rXÚgetvaluerÔ)r‚rÕrÖrØrÚrÛrÜrÝrrrÚbatchksÿ ûz__runner..batchršzWorker %d STOPPEDzWorker %s Test is %s (%s)r…z,Worker %s timed out, failing current test %sz5Worker %s keyboard interrupt, failing current test %szWorker %s test %s timed outz$Worker %s test %s keyboard interruptzWorker %s system exitz1Worker %s error running test or returning resultszWorker %s ending)(rŠÚloadsZ parserClassÚ_instantiate_pluginsZ addOptionsrŸZ addPluginrTrDÚbeginrgrhrržÚNoSharedFixtureContextSuiteZ suiteClassÚiterr¨Ú exceptionZloadTestsFromNamesrr€rÃr1rrr‰r·rrrjr r¦r~rrnrurvrsrtr?)rárr’rŒrrŽr“rVrârKZ dummy_parserZ pluginclassÚpluginrrArãrërƒrÃr‚rmrÀr˜r¾r)rKrârrràMs†              ÿ    ÿràcs@eZdZdZdZdZdZ‡fdd„Z‡fdd„Zdd„Z ‡Z S) rïzîContext suite that never fires shared fixtures. When a context sets _multiprocess_shared_, fixtures in that context are executed by the main process. Using this class prevents them from executing in the runner process as well.Ncs$t|ddƒrdStt|ƒ |¡dS©NreF)rzrarïÚ setupContext©rrorcrrrô½s z(NoSharedFixtureContextSuite.setupContextcs$t|ddƒrdStt|ƒ |¡dSró)rzrarïÚteardownContextrõrcrrröÂs z+NoSharedFixtureContextSuite.teardownContextcCs.t dt|ƒ||j¡|jr0| ||¡|}}n ||}}z | ¡Wn<tyZ‚Yn*ty‚d|_|  ||  ¡¡YdS0zN|jD]ð}t |t j jƒr¶|jdur¶|j|j_n|j|_|j|_|j|_|jrät d¡q€z ||ƒWqŽty|}zrt |tƒ}|rd}nd}t |||¡ttt|ƒƒt ¡df}|jj  ||¡|  ||¡|sh‚WYd}~qŽd}~00qŽWd|_z | ¡Wn>tyª‚Yn*tyÒd |_|  ||  ¡¡Yn0nTd|_z | ¡Wn>tyþ‚Yn*ty&d |_|  ||  ¡¡Yn00dS) z,Run tests in suite inside of suite fixtures.z#suite %s (%s) run called, tests: %sÚsetupNZstoppingz(Timeout when running test %s in suite %sz2KeyboardInterrupt when running test %s in suite %srdTZteardown)rgrhr,Z_testsZ resultProxyrqrrrtZ error_contextrwZ _exc_inforjrkr3rlrÃrmrr€r“r r1rurvrKrŸZhas_runrµ)rr‚ÚorigrmrÀr˜r¾rçrrrrÂÇsrÿ         ÿ  ý  ø zNoSharedFixtureContextSuite.run) rrrr]rr€rÃrôrörÂrßrrrcrrï´s  rï)2r]Úloggingrrur‰rÑrŠrZ nose.caserkZ nose.corerrrZnose.plugins.baserZnose.pyversionrZ nose.suiterZ nose.utilrZunittest.runnerr Úqueuer Úwarningsr Úior ríÚ getLoggerrrgrr!r"r#r$r%rrr r*r+r4r`rZr[ràrïrrrrÚsB:             Lpg