使用spark sql在配置单元表中从列表中插入数据

Question

我有一个文件名、FilePath和FileSize的列表，我想使用spark SQL将这些细节插入到我的配置单元表中。

var fs1 = FileSystem.get(sparksession.sparkContext.hadoopConfiguration)
var file_path = fs1.listStatus(new  Path("path")).filter(_.isFile).map(_.getPath).toList
var new_files = fs1.listStatus(new  Path("path")).filter(_.isFile).map(_.getPath.getName).toList
var file_size = fs1.listStatus(new Path("path")).filter(_.isFile).map(_.getLen).toList
var file_data = file_path zip new_files zip file_size

for ((filedetail, size) <- file_size){
  var filepath = filedetail._1
  var filesize: Long = size
  var filename = filedetail._2
  var df = spark.sql(s"insert into mytable(file_path,filename,file_size)  select '${file_path}' as file_path,'${new_files}' as filename,'${file_size}' as file_size")
  df.write.insertInto("dbname.tablename")
}

insert into mytable(file_path,filename,file_size) select  'List(path/filename.txt,path/filename4.txt,path/filename5.txt)' as file_path,'List(filename.txt, filename4.txt, filename5.txt)' as filename,'List(19, 19, 19)' as file_size;

file_path                 filename      file_size
path/filename.txt         filename.txt  19
path/filename4.txt        filename4.txt  19
path/filename5.txt        filename5.txt  19

是否有任何方法可以再次将此查询拆分为3个不同的insert hive语句。

    insert into mytable(file_path,filename,file_size) select 'path/filename.txt' as file_path,'filename.txt' as filename,'19' as file_size;
    insert into mytable(file_path,filename,file_size) select 'path/filename3.txt' as file_path,'filename3.txt' as filename,'19' as file_size;
    insert into mytable(file_path,filename,file_size) select 'path/filename4.txt' as file_path,'filename4.txt' as filename,'19' as file_size;

Answer 1

您可以简单地创建一个包含所有行的列表，每行将有3个元素。创建一个dataframe，然后使用dataframe持久化数据。

参考这篇文章