o ckF[3#@sddlmZGdddeZGdddeZGdddeZGdd d eZGd d d eZGd d d eZGdddeZ GdddeZ Gddde Z Gddde Z dS))sixc@s(eZdZdZddZddZddZdS) Stepz! Jobflow Step base class cCt)z= :rtype: str :return: URI to the jar NotImplementedselfr //usr/lib/python3/dist-packages/boto/emr/step.pyjarzStep.jarcCr)zS :rtype: list(str) :return: List of arguments for the step rrr r r args%r z Step.argscCr)zB :rtype: str :return: The main class name rrr r r main_class,r zStep.main_classN)__name__ __module__ __qualname____doc__r r rr r r r rs  rc@s6eZdZdZ  d ddZddZdd Zd d ZdS) JarStepz Custom jar step NTERMINATE_JOB_FLOWcCs4||_||_||_||_t|tjr|g}||_dS)a  A elastic mapreduce step that executes a jar :type name: str :param name: The name of the step :type jar: str :param jar: S3 URI to the Jar file :type main_class: str :param main_class: The class to execute in the jar :type action_on_failure: str :param action_on_failure: An action, defined in the EMR docs to take on failure. :type step_args: list(str) :param step_args: A list of arguments to pass to the step N)name_jar _main_classaction_on_failure isinstancer string_types step_args)rrr rrrr r r __init__8s  zJarStep.__init__cC|jSNrrr r r r Sz JarStep.jarcCsg}|jr ||j|Sr)rextend)rr r r r r Vs z JarStep.argscCrr)rrr r r r^r zJarStep.main_class)NrN)rrrrrr r rr r r r r4s  rc@sDeZdZdZ     dddZddZd d Zd d Zd dZdS) StreamingStepz Hadoop streaming step Nr3/home/hadoop/contrib/streaming/hadoop-streaming.jarc CsX||_||_||_||_||_||_||_| |_| |_| |_ t |t j r'|g}||_ dS)a A hadoop streaming elastic mapreduce step :type name: str :param name: The name of the step :type mapper: str :param mapper: The mapper URI :type reducer: str :param reducer: The reducer URI :type combiner: str :param combiner: The combiner URI. Only works for Hadoop 0.20 and later! :type action_on_failure: str :param action_on_failure: An action, defined in the EMR docs to take on failure. :type cache_files: list(str) :param cache_files: A list of cache files to be bundled with the job :type cache_archives: list(str) :param cache_archives: A list of jar archives to be bundled with the job :type step_args: list(str) :param step_args: A list of arguments to pass to the step :type input: str or a list of str :param input: The input uri :type output: str :param output: The output uri :type jar: str :param jar: The hadoop streaming jar. This can be either a local path on the master node, or an s3:// URI. N)rmapperreducercombinerr cache_filescache_archivesinputoutputrrrrr) rrr$r%r&rr'r(rr)r*r r r r rfs#  zStreamingStep.__init__cCrrrrr r r r r zStreamingStep.jarcCsdSrr rr r r rszStreamingStep.main_classcCsg}|jr ||j|d|jg|jr|d|jg|jr*|d|jgn|ddg|jrPt|jtrH|jD] }|d|fq=n|d|jf|jr[|d|jf|j rk|j D] }|d|fqa|j r{|j D] }|d |fqq|S) Nz-mapperz -combinerz-reducerz-jobconfzmapred.reduce.tasks=0z-inputz-outputz -cacheFilez -cacheArchive) rr!r$r&r%r)rlistr*r'r()rr r) cache_file cache_archiver r r r s0     zStreamingStep.argsc Cs<d|jj|jj|j|j|j|j|j|j|j |j |j |j f S)Nz%s.%s(name=%r, mapper=%r, reducer=%r, action_on_failure=%r, cache_files=%r, cache_archives=%r, step_args=%r, input=%r, output=%r, jar=%r)) __class__rrrr$r%rr'r(rr)r*rrr r r __repr__s    zStreamingStep.__repr__) NNrNNNNNr#) rrrrrr rr r/r r r r r"bs 3 %r"cs eZdZdZfddZZS)ScriptRunnerStepzEs3n://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jarc s tt|j||jfi|dSr)superr0rScriptRunnerJar)rrkwr.r r rs zScriptRunnerStep.__init__)rrrr2r __classcell__r r r4r r0sr0c@eZdZgdZdS)PigBase)z4s3n://us-east-1.elasticmapreduce/libs/pig/pig-script --base-pathz*s3n://us-east-1.elasticmapreduce/libs/pig/NrrrBaseArgsr r r r r7 r7cs&eZdZdZdZdfdd ZZS)InstallPigStepz! Install pig on emr step z Install PiglatestcsDg}||j|dg|d|gtt|j|j|ddS)Nz --install-pig--pig-versionsr)r!r:r1r<rInstallPigName)r pig_versionsrr4r r rs   zInstallPigStep.__init__)r=)rrrrr@rr5r r r4r r<sr<cs&eZdZdZdgffdd ZZS)PigStepz Pig script step r=csRg}||j|d|g|ddd|g||tt|j||ddS)Nr>z--run-pig-script--args-fr?)r!r:r1rBr)rrpig_filerApig_argsrr4r r rs   zPigStep.__init__rrrrrr5r r r4r rBsrBc@r6)HiveBase)z6s3n://us-east-1.elasticmapreduce/libs/hive/hive-scriptr8z+s3n://us-east-1.elasticmapreduce/libs/hive/Nr9r r r r rHr;rHcs&eZdZdZdZdfdd ZZS)InstallHiveStepz" Install Hive on EMR step z Install Hiver=Ncs\g}||j|dg|d|g|dur!|d|gtt|j|j|ddS)Nz--install-hive--hive-versionsz--hive-site=%sr?)r!r:r1rIrInstallHiveName)r hive_versions hive_siterr4r r rs   zInstallHiveStep.__init__r=N)rrrrrKrr5r r r4r rIsrIcs&eZdZdZ  dfdd ZZS)HiveStepz Hive script step r=NcsZg}||j|d|g|ddd|g|dur!||tt|j||ddS)NrJz--run-hive-scriptrCrDr?)r!r:r1rOr)rr hive_filerL hive_argsrr4r r rs  zHiveStep.__init__rNrGr r r4r rOs rON) boto.compatrobjectrrr"r0r7r<rBrHrIrOr r r r s .j