Hadoop HDFS以编程方式写入操作

Question

public static boolean fileWriteHDFS(InputStream input, String fileName) {   
    try {
        System.setProperty("HADOOP_USER_NAME", "hduser");

        //Get Configuration of Hadoop system
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://localhost:9000");
        //conf.get("fs.defaultFS");     

        //Extract destination path
        URI uri = URI.create(DESTINATION_PATH+fileName);
        Path path = new Path(uri);

        //Destination file in HDFS
        FileSystem fs = FileSystem.get(uri, conf); //.get(conf);

        //Check if the file already exists
        if (fs.exists(path))
        {
            //Write appropriate error to log file and return.
            return false;
        }

        //Create an Output stream to the destination path
        FSDataOutputStream out = fs.create(path);

        //Copy file from input steam to HDFSs
        IOUtils.copyBytes(input, out, 4096, true);

        //Close all the file descriptors
        out.close();
        fs.close();
        //All went perfectly as planned
        return true;    
    } catch (Exception e) {
        //Something went wrong
        System.out.println(e.toString());
        return false;
    }
}

/home/hduser/bin/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar/home/hduser/bin/hadoop-common-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar/home/hduser/bin/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar

您可以看到，我的hadoop安装位置是/home/hduser/bin/hadoop-2.9.0/...当我运行这段代码时，它会抛出一个异常。即。

Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
at com.ws.filewrite.fileWrite.fileWriteHDFS(fileWrite.java:21)
at com.ws.main.listenerService.main(listenerService.java:21)
Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 2 more

特别是在谎言处抛出异常：

Configuration conf=new Configuration（）；

我是不是漏了什么？是什么导致了这个问题？我对HDFS是全新的，所以请原谅，这是一个明显的问题。

Answer 1

hadoop 2.9依赖项与hadoop 2.6不相似。

我也遇到过同样的情况，试着找到依赖项JAR。这很难，下次可能又有一罐漏了...

因此，我使用Maven来管理依赖项。

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.9.0</version>
        <!--<scope>provided</scope>-->
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.9.0</version>
    </dependency>