从Java API启动AWS弹性mapreduce jobflow。我的蜂巢脚本应该去哪里？

Question

我一直在使用Amazon Elastic MapReduce和Hive开发一个数据处理应用程序。现在，当我使用交互模式作业流SSH并运行它们时，Hive脚本就可以工作了，现在我尝试使用AWS Java API创建一个作业流。

使用http://docs.amazonwebservices.com/elasticmapreduce/latest/developerguide/calling-emr-with-java-sdk.html作为起点，我创建了如下步骤配置

StepConfig runScript = new StepConfig().withName("Prepare Admin")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(oStepFactory.newRunHiveScriptStep(scriptPath, args));

我假设/希望scriptPath可以是我的Hive脚本的s3 url，比如:s3://bucketname/hive-script。我找到的唯一文档讨论了如何从主节点的文件系统中使用脚本。但是，如果主节点是为了这个作业流而启动的实例，我不明白如何将任何脚本（配置单元或其他方式）添加到文件系统中。

当我尝试我的想法（将一个s3位置传递给stepFactory方法）时，runScript步骤失败了。

我已经通过AWS控制台检查了日志。stdout日志以

2012-11-19 19:28:33 GMT-执行cmd时出错：/home/hadoop/.versions/hive-0.7.1/bin/hive'-f''s3://anet-emr/scripts/admin.q''-d rawdatalocation=s3://anet-emr/raw-d year=2010-d cycle=1'

stderr日志以

java.lang.nosuchmethoderror：org.apache.commons.cli.commandline.getOptionProperties（ljava/lang/string；）ljava/util/properties；在org.apache.hadoop.hive.cli.optionsprocessor.process_stage1(optionsprocessor.java:115)在org.apache.hadoop.hive.cli.cli.clidriver.main(clidriver.java:399)在sun.reflect.nativeMethodAccessorImpl.invoke(nativeMethodAccessorImpl.invoke(nativeMethodAccessorImpl.invoke(nativeMethodAccessorImpl.java:39)在ar.main(runjar.java:155)在org.apache.hadoop.mapred.jobshell.run(jobshell.java:54)在org.apache.hadoop.util.toolrunner.run(toolrunner.java:65)在org.apache.hadoop.util.toolrunner.run(toolrunner.java:79)在org.apache.hadoop.mapred.jobshell.main(jobshell.java:68)

控制器日志中有

2012-11-19T19:28:27.406 z信息执行log.file=syslog-dhadoop.home.dir=/home/hadoop-dhadoop.id.str=hadoop-dhadoop.root.logger=INFO,DRFA-djava.io.tmpdir=/mnt/var/lib/hadoop/steps/3/tmp-djava.library.path=/home/hadoop/library.path=/home/hadoop/lib/native/linux-i386-32 org.apache.hadoop.mapred.jobshell：//US-EAST-1.ElasticMapReduce/libs/hive/--hive-版本最新--run-hive-script--args-f s3:/aNet-emr/scripts/admin.q-d rawdatalocation=s3://aNet-emr/raw-d year=2010-d cycle=1 2012-11-19t19:28:34.143 z信息执行以ret val 255 201 2-11-19t19:28:34.143 z警告步骤失败，retval错误

问题似乎在于我通过Amazon的API传递给Hive对Apache CLI库的调用的参数...我尝试过用“-d arg1=val1-d arg2=val2”传递单个字符串，我尝试过“-d,arg1=val1等”。我已经尝试了各种方法来分解成字符串数组-例如{“-d”，“arg1=val1”...}。找不到任何文档的适当方式做到这一点！

谢谢你的帮助，科尔曼

Answer 1

嗨，这段代码适合我：

String accessKey = "";
String secretKey = "";
 AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
   AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(credentials);

   StepFactory stepFactory = new StepFactory();

   StepConfig enabledebugging = new StepConfig()
       .withName("Enable debugging")
       .withActionOnFailure("TERMINATE_JOB_FLOW")
       .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

   StepConfig installHive = new StepConfig()
       .withName("Install Hive")
       .withActionOnFailure("TERMINATE_JOB_FLOW")
       .withHadoopJarStep(stepFactory.newInstallHiveStep());

   StepConfig runScript = new StepConfig()
        .withName("Run Script")
        .withActionOnFailure("TERMINATE_JOB_FLOW")
        .withHadoopJarStep(stepFactory.newRunHiveScriptStep("s3://dummy/dummy.hive"));


   RunJobFlowRequest request = new RunJobFlowRequest()
       .withName("Hive Interactive")
       .withSteps(enabledebugging, installHive, runScript)
       .withLogUri("s3://dummy/")
       .withInstances(new JobFlowInstancesConfig()
           .withHadoopVersion("0.20.205")
           .withInstanceCount(1)
           .withKeepJobFlowAliveWhenNoSteps(false)
           .withMasterInstanceType("m1.small")
           .withSlaveInstanceType("m1.small"));

   RunJobFlowResult result = emr.runJobFlow(request);

希望这有所帮助：）