在Scala/Spark中在HDFS上将文件从一个文件夹移动到另一个文件夹

Question

val fs = FileSystem.get(sc.hadoopConfiguration)
fs.moveFromLocalFile(something, something2)

我得到以下错误：

线程“main”java.lang.IllegalArgumentException：错误的fs：hdfs：/user/o/datasets/data.txt，应为：file：///

movetolocalfile()也是如此，因为它们用于在文件系统之间传输文件，而不是在文件系统内传输文件。我还尝试了fs.rename()，但根本没有任何作用（没有错误或任何东西）。

Answer 1

尝试下面的Scala代码。

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path

val hadoopConf = new Configuration()
val hdfs = FileSystem.get(hadoopConf)

val srcPath = new Path(srcFilePath)
val destPath = new Path(destFilePath)

hdfs.copyFromLocalFile(srcPath, destPath)

您还应该检查Spark是否在conf/spark-env.sh文件中设置了HADOOP_CONF_DIR变量。这将确保Spark找到Hadoop配置设置。

build.sbt文件的依赖关系：

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0"
libraryDependencies += "org.apache.commons" % "commons-io" % "1.3.2"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.6.0"

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import org.apache.commons.io.IOUtils;



val hadoopconf = new Configuration();
val fs = FileSystem.get(hadoopconf);

//Create output stream to HDFS file
val outFileStream = fs.create(new Path("hdfs://<namenode>:<port>/output_path"))

//Create input stream from local file
val inStream = fs.open(new Path("hdfs://<namenode>:<port>/input_path"))

IOUtils.copy(inStream, outFileStream)

//Close both files
inStream.close()
outFileStream.close()