如何将 AWS 凭证密钥添加到 intelliJ 中的 Hadoop 配置中

Question

我试图通过带有AWS的IntelliJ使用Hadoop执行操作，但遇到Hadoop问题——无法访问我的凭据密钥。我的IntelliJ帐户已经连接到AWS，我可以使用这些服务，但使用Hadoop，我遇到了以下问题。

这是代码-

public static void main(String[] args) throws Exception {

    String s3Path = "adoop-map-reduce-collocation-extraction/corpus-cardinality-by-decade";
    String output = "s3a://adoop-map-reduce-collocation-extraction/statistic-extraction";

    Configuration conf = new Configuration();
    conf.set("CorpusCardinalityFilePath", s3Path);

    Job job = Job.getInstance(conf, "Appearances Calculation");
    job.setJarByClass(AppearancesCalculation.class);
    job.setMapperClass(AC_Mapper.class);
    job.setCombinerClass(AC_Combiner.class);
    job.setReducerClass(AC_Reducer.class);
    job.setOutputKeyClass(nGramKey.class);
    job.setOutputValueClass(LongWritable.class);
    job.setMapOutputValueClass(LongWritable.class);
    job.setMapOutputKeyClass(nGramKey.class);
    job.setOutputValueClass(nGramValue.class);

    FileInputFormat.addInputPath(job, new Path(inPath));
    FileOutputFormat.setOutputPath(job, new Path(output));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
}

以及所有依赖项-

3.0.0

<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.531</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-kinesis-client</artifactId>
        <version>1.2.1</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.jetbrains</groupId>
        <artifactId>annotations-java5</artifactId>
        <version>17.0.0</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-aws</artifactId>
        <version>${hadoop.version}</version>
    </dependency>

</dependencies>

这就是整个错误-

过程结束，退出代码为1

Answer 1

查看错误，我会说您需要将文件核心站点.xml添加到资源中，其属性为fs.s3a.access.key，fs.s3a.secret.key和fs.s3a.endpoint