AWS Sagemaker Spark S3访问问题

Question

我是新来的。我在一个VPC中创建了一个笔记本，它有私有子网，kms默认加密密钥，根访问，没有直接的internet访问。我已附上政策，根据文件，该政策可完全访问IAM中的Sagemaker和S3。现在，当一位数据科学家试图在jupyter中运行他的代码时，出现了以下错误。我可以看到jar文件（/home/ec2 user/anaconda3/envs/python3/lib/python3.6/site packages/sagemaker_pyspark/jars/），我甚至在代码中提供了访问密钥和密钥，我们在这里做了什么错误吗

import os
import boto3

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

import sagemaker
from sagemaker import get_execution_role
import sagemaker_pyspark
import pyspark

role = get_execution_role()
spark = SparkSession.builder \
            .appName("app_name2") \
            .getOrCreate()

sc=pyspark.SparkContext.getOrCreate()
sc.setSystemProperty("com.amazonaws.services.s3.enableV4", "true")

hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
spark._jsc.hadoopConfiguration().set("fs.s3a.access.key", 'access_key')
spark._jsc.hadoopConfiguration().set("fs.s3a.secret.key", 'secret_key')
spark._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.us-east-2.amazonaws.com")
spark._jsc.hadoopConfiguration().set("com.amazonaws.services.s3a.enableV4", "true")
spark._jsc.hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false");
spark._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
df= spark.read.csv("s3a://mybucket/ConsolidatedData/my.csv",header="true")


Py4JJavaError: An error occurred while calling o579.csv.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:709)

Answer 1

/home/ec2 user/anaconda3/envs/python3/lib/python3中缺少Jar文件。6/site packages/pyspark/jars，我正在查看（/home/ec2 user/anaconda3/envs/python3/lib/python3.6/site packages/sagemaker_pyspark/jars/），在第一个位置复制文件解决了问题。