elastic-map-reduce - 弹性地图减少java api中的“保持活力”规范

Question

如何在 java api 中将作业流设置为“保持活动状态”，就像我使用这样的命令一样：

elastic-mapreduce --create --alive ...

我尝试添加 withKeepJobFlowAlivewhenNoSteps(true) 但这仍然会使作业流在步骤失败时关闭（例如，如果我提交了一个坏 jar）

Answer 1

您需要设置withActionOnFailure让 API 在步骤失败时知道该怎么做，并且必须基于每个步骤进行设置。

你一定有withActionOnFailure("TERMINATE_JOB_FLOW")你StepConfig的s。将它们更改为withActionOnFailure("CANCEL_AND_WAIT").

以下是使用从此处获取的 Java API 启动集群的完整代码，只是替换了需要的部分：

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
   AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(credentials);

   StepFactory stepFactory = new StepFactory();

   StepConfig enabledebugging = new StepConfig()
       .withName("Enable debugging")
       .withActionOnFailure("CANCEL_AND_WAIT") //here is the change
       .withHadoopJarStep(stepFactory.newEnabledebuggingStep());

   StepConfig installHive = new StepConfig()
       .withName("Install Hive")
       .withActionOnFailure("CANCEL_AND_WAIT") //here is the change
       .withHadoopJarStep(stepFactory.newInstallHiveStep());

   RunJobFlowRequest request = new RunJobFlowRequest()
       .withName("Hive Interactive")
       .withSteps(enabledebugging, installHive)
       .withLogUri("s3://myawsbucket/")
       .withInstances(new JobFlowInstancesConfig()
           .withEc2KeyName("keypair")
           .withHadoopVersion("0.20")
           .withInstanceCount(5)
           .withKeepJobFlowAliveWhenNoSteps(true)
           .withMasterInstanceType("m1.small")
           .withSlaveInstanceType("m1.small"));

   RunJobFlowResult result = emr.runJobFlow(request);