使用spark到绝对路径写入Google云存储

Question

我正试图将一个spark dataframe写入谷歌云存储。这个数据文件有一些更新，所以我需要一个分区策略。所以我需要把它写进GCS的精确文件中。

        .config("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")\
        .config("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")\
        .config("fs.gs.project.id", project_id)\
        .config("fs.gs.auth.service.account.enable", "true")\
        .config("fs.gs.auth.service.account.project.id",project_id)\
        .config("fs.gs.auth.service.account.private.key.id",private_key_id)\
        .config("fs.gs.auth.service.account.private.key",private_key)\
        .config("fs.gs.auth.service.account.client.email",client_email)\
        .config("fs.gs.auth.service.account.email",client_email)\
        .config("fs.gs.auth.service.account.client.id",client_id)\
        .config("fs.gs.auth.service.account.auth.uri",auth_uri)\
        .config("fs.gs.auth.service.account.token.uri",token_uri)\
        .config("fs.gs.auth.service.account.auth.provider.x509.cert.url",auth_provider_x509_cert_url)\
        .config("fs.gs.auth.service.account.client_x509_cert_url",client_x509_cert_url)\
        .config("spark.sql.avro.compression.codec", "deflate")\
        .config("spark.sql.avro.deflate.level", "5")\
        .getOrCreate())

df.write.format(file_format).save('gs://'+bucket_name+path+'/'+table_name+'/file_name.avro')

gs://bucket_name/table_name/file_name.avro

有谁能帮我做到这一点吗？

Answer 1

这看起来像是标准Spark库的局限性。也许这个答案会有帮助。

您还可以使用带有Apache Spark的Cloud Storage Connector从Spark查看与Google Cloud Storage交互的替代方式。