用boto3从S3桶读取文件内容

Question

我通过执行以下操作读取S3存储桶中的文件名

objs = boto3.client.list_objects(Bucket='my_bucket')
    while 'Contents' in objs.keys():
        objs_contents = objs['Contents']
        for i in range(len(objs_contents)):
            filename = objs_contents[i]['Key']

现在，我需要获取文件的实际内容，类似于打开（filename）.readlines（）。最好的方法是什么？

Answer 1

使用客户端而不是资源：

s3 = boto3.client('s3')
bucket='bucket_name'
result = s3.list_objects(Bucket = bucket, Prefix='/something/')
for o in result.get('Contents'):
    data = s3.get_object(Bucket=bucket, Key=o.get('Key'))
    contents = data['Body'].read()
    print(contents.decode("utf-8"))

Answer 2

您还可以考虑smart_open模块，它支持迭代器：

from smart_open import smart_open

# stream lines from an S3 object
for line in smart_open('s3://mybucket/mykey.txt', 'rb'):
    print(line.decode('utf8'))

和上下文管理器：

with smart_open('s3://mybucket/mykey.txt', 'rb') as s3_source:
    for line in s3_source:
         print(line.decode('utf8'))

    s3_source.seek(0)  # seek to the beginning
    b1000 = s3_source.read(1000)  # read 1000 bytes

在以下位置查找smart\u openhttps://pypi.org/project/smart_open/

Answer 3

BOTO3提供了一个资源模型，使得像迭代对象这样的任务更加容易。不幸的是，StreamingBody不提供readline或readline。

s3 = boto3.resource('s3')
bucket = s3.Bucket('test-bucket')
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
    key = obj.key
    body = obj.get()['Body'].read()