是否有任何可行的方法可以直接将动态生成的文件上传到亚马逊 s3,而无需先创建本地文件然后上传到 s3 服务器?我用蟒蛇。谢谢
12 回答
这是一个下载图像(使用请求库)并将其上传到 s3 的示例,而不写入本地文件:
import boto
from boto.s3.key import Key
import requests
#setup the bucket
c = boto.connect_s3(your_s3_key, your_s3_key_secret)
b = c.get_bucket(bucket, validate=False)
#download the file
url = "http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r = requests.get(url)
if r.status_code == 200:
#upload the file
k = Key(b)
k.key = "image1.png"
k.content_type = r.headers['content-type']
k.set_contents_from_string(r.content)
您可以使用Python 标准库中的BytesIO。
from io import BytesIO
bytesIO = BytesIO()
bytesIO.write('whee')
bytesIO.seek(0)
s3_file.set_contents_from_file(bytesIO)
有关使用 set_contents_from_string 的示例,请参阅boto 文档的存储数据部分,为了完整起见,粘贴在此处:
>>> from boto.s3.key import Key
>>> k = Key(bucket)
>>> k.key = 'foobar'
>>> k.set_contents_from_string('This is a test of S3')
我假设你正在使用boto
. boto
'sBucket.set_contents_from_file()
将接受一个StringIO
对象,并且您为将数据写入文件而编写的任何代码都应该很容易适应写入StringIO
对象。或者,如果您生成一个字符串,您可以使用set_contents_from_string()
.
def upload_to_s3(url, **kwargs):
'''
:param url: url of image which have to upload or resize to upload
:return: url of image stored on aws s3 bucket
'''
r = requests.get(url)
if r.status_code == 200:
# credentials stored in settings AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, host=AWS_HOST)
# Connect to bucket and create key
b = conn.get_bucket(AWS_Bucket_Name)
k = b.new_key("{folder_name}/{filename}".format(**kwargs))
k.set_contents_from_string(r.content, replace=True,
headers={'Content-Type': 'application/%s' % (FILE_FORMAT)},
policy='authenticated-read',
reduced_redundancy=True)
# TODO Change AWS_EXPIRY
return k.generate_url(expires_in=AWS_EXPIRY, force_http=True)
在boto3中,有一种上传文件内容的简单方法,无需使用以下代码创建本地文件。我已经修改了 boto3 的 JimJty 示例代码
import boto3
from botocore.retries import bucket
import requests
from io import BytesIO
# set the values
aws_access_key_id=""
aws_secret_access_key=""
region_name=""
bucket=""
key=""
session = boto3.session.Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key, region_name=region_name)
s3_client = session.client('s3')
#download the file
url = "http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r = requests.get(url)
if r.status_code == 200:
#convert content to bytes, since upload_fileobj requires file like obj
bytesIO = BytesIO(bytes(r.content))
with bytesIO as data:
s3_client.upload_fileobj(data, bucket, key)
我有一个 dict 对象,我想将其存储为 S3 上的 json 文件,而不创建本地文件。下面的代码对我有用:
from smart_open import smart_open
with smart_open('s3://access-key:secret-key@bucket-name/file.json', 'wb') as fout:
fout.write(json.dumps(dict_object).encode('utf8'))
您可以尝试使用smart_open
(https://pypi.org/project/smart_open/)。我正是为此使用它:直接在 S3 中写入文件。
鉴于静态加密现在是一个非常需要的数据标准,smart_open 不支持这个 afaik
此实现是将图像列表(NumPy 列表、OpenCV 图像对象)直接上传到 S3 的示例
注意:您需要在上传文件时将图像对象转换为字节或缓冲区转换为字节,这样您就可以上传文件而不会出现损坏错误
#Consider you have images in the form of a list i.e. img_array
import boto3
s3 = boto3.client('s3')
res_url = []
for i,img in enumerate(img_array):
s3_key = "fileName_on_s3.png"
response = s3.put_object(Body=img.tobytes(), Bucket='bucket_name',Key=s3_key,ACL='public-read',ContentType= 'image/png')
s3_url = 'https://bucket_name.s3.ap-south-1.amazonaws.com/'+s3_key
res_url.append(s3_url)
#res_url is the list of URLs returned from S3 Upload
boto3的更新:
aws_session = boto3.Session('my_access_key_id', 'my_secret_access_key')
s3 = aws_session.resource('s3')
s3.Bucket('my_bucket').put_object(Key='file_name.txt', Body=my_file)
我遇到了类似的问题,想知道是否有最终答案,因为使用下面的代码,“starwars.json”会继续在本地保存,但我只想将每个循环的 .json 文件推送到 S3 中并且没有文件存储在本地。
for key, value in star_wars_actors.items():
response = requests.get('http:starwarsapi/' + value)
data = response.json()
with open("starwars.json", "w+") as d:
json.dump(data, d, ensure_ascii=False, indent=4)
s3.upload_file('starwars.json', 'test-bucket',
'%s/%s' % ('test', str(key) + '.json'))