5

到目前为止,我已经经历了这个问题的多次迭代,搜索了许多不同的例子,并且已经通过了文档。

我正在尝试将 Plupload ( http://www.plupload.com/ ) 与 AWS S3 直接发布方法 ( http://aws.amazon.com/articles/1434 ) 结合起来。但是,我认为我构建用于传输的策略和签名的方式存在问题。当我提交表单时,我没有得到服务器的响应,而是我与服务器的连接被重置。

我尝试在示例中使用 python 代码:

import base64
import hmac, sha

policy = base64.b64encode(policy_document)

signature = base64.b64encode(
hmac.new(aws_secret_key, policy, sha).digest())

我还尝试在 python 中使用更新的 hashlib 库。无论我使用什么方法来构建我的策略和签名,我总是得到与此处生成的值不同的值:

http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html

我已经阅读了这个问题:

如何使 Plupload 直接上传到 Amazon S3?

但我发现提供的示例过于复杂,无法准确实现。

我最近的尝试是使用部分 boto 库:

http://boto.cloudhackers.com/ref/s3.html#module-boto.s3.connection

但是使用 S3Commection.build_post_form_args 方法对我也不起作用。

如果有人可以提供如何使用 python 创建帖子表单的适当示例,我将非常感激。即使是关于为什么总是重置连接的一些简单见解也会很好。

一些警告:

如果可能的话,我想使用 hashlib。我想从亚马逊获得 XML 响应(大概是“success_action_status = '201'”)我需要能够上传较大的类型文件,最大大小约为 2GB。

最后一点,当我在 Chrome 中运行它时,它会提供上传进度,并且上传通常会在 37% 左右失败。

4

4 回答 4

5

内森的回答帮助我开始了。我已经包含了两个目前对我有用的解决方案。

第一个解决方案使用纯 Python。第二个使用boto。

我试图让 boto 先工作,但不断出错。所以我回到了 Amazon ruby​​ 文档,让 S3 使用没有 boto 的 python 接受文件。(浏览器使用 HTML POST 上传到 S3

在了解发生了什么之后,我能够修复我的错误并使用 boto,这是一个更简单的解决方案。

我包括解决方案 1,因为它明确显示了如何使用 python 设置策略文档和签名。

我的目标是将 html 上传页面创建为动态页面,以及用户在成功上传后看到的“成功”页面。方案一展示了表单上传页面的动态创建,方案二展示了上传表单页面和成功页面的创建。

解决方案1:

import base64
import hmac, hashlib

###### EDIT ONLY THE FOLLOWING ITEMS ######

DEBUG = 1
AWS_SECRET_KEY = "MySecretKey"
AWS_ACCESS_KEY = "MyAccessKey"
HTML_NAME = "S3PostForm.html"
EXPIRE_DATE = "2015-01-01T00:00:00Z" # Jan 1, 2015 gmt
FILE_TO_UPLOAD = "${filename}"
BUCKET = "media.mysite.com"
KEY = ""
ACL = "public-read" # or "private"
SUCCESS = "http://media.mysite.com/success.html"
CONTENT_TYPE = ""
CONTENT_LENGTH = 1024**3 # One gigabyte
HTTP_OR_HTTPS = "http" # Or "https" for better security
PAGE_TITLE = "My Html Upload to S3 Form"
ACTION = "%s://%s.s3.amazonaws.com/" % (HTTP_OR_HTTPS, BUCKET)

###### DON'T EDIT FROM HERE ON DOWN ######

policy_document_data = {
"expire": EXPIRE_DATE,
"bucket_name": BUCKET,
"key_name": KEY,
"acl_name": ACL,
"success_redirect": SUCCESS,
"content_name": CONTENT_TYPE,
"content_length": CONTENT_LENGTH,
}

policy_document = """
{"expiration": "%(expire)s",
  "conditions": [ 
    {"bucket": "%(bucket_name)s"}, 
    ["starts-with", "$key", "%(key_name)s"],
    {"acl": "%(acl_name)s"},
    {"success_action_redirect": "%(success_redirect)s"},
    ["starts-with", "$Content-Type", "%(content_name)s"],
    ["content-length-range", 0, %(content_length)d]
  ]
}
""" % policy_document_data

policy = base64.b64encode(policy_document)
signature = base64.b64encode(hmac.new(AWS_SECRET_KEY, policy, hashlib.sha1).digest())

html_page_data = {
"page_title": PAGE_TITLE,
"action_name": ACTION,
"filename": FILE_TO_UPLOAD,
"access_name": AWS_ACCESS_KEY,
"acl_name": ACL,
"redirect_name": SUCCESS,
"policy_name": policy,
"sig_name": signature,
"content_name": CONTENT_TYPE,
}

html_page = """
<html> 
 <head>
  <title>%(page_title)s</title> 
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 </head>
<body>
 <form action="%(action_name)s" method="post" enctype="multipart/form-data">
  <input type="hidden" name="key" value="%(filename)s">
  <input type="hidden" name="AWSAccessKeyId" value="%(access_name)s">
  <input type="hidden" name="acl" value="%(acl_name)s">
  <input type="hidden" name="success_action_redirect" value="%(redirect_name)s">
  <input type="hidden" name="policy" value="%(policy_name)s">
  <input type="hidden" name="signature" value="%(sig_name)s">
  <input type="hidden" name="Content-Type" value="%(content_name)s">

  <!-- Include any additional input fields here -->

  Browse to locate the file to upload:<br \> <br \>

  <input name="file" type="file"><br> <br \>
  <input type="submit" value="Upload File to S3"> 
 </form> 
</body>
</html>
""" % html_page_data

with open(HTML_NAME, "wb") as f:
    f.write(html_page)

###### Dump output if testing ######
if DEBUG:

    if 1: # Set true if not using the LEO editor
        class G:
            def es(self, data):print(data)
        g = G()

    items = [
    "",
    "",
    "policy_document: %s" % policy_document,
    "ploicy: %s" % policy,
    "signature: %s" % signature,
    "",
    "",
    ]
    for item in items:
        g.es(item)

解决方案2:

from boto.s3 import connection

###### EDIT ONLY THE FOLLOWING ITEMS ######

DEBUG = 1
AWS_SECRET_KEY = "MySecretKey"
AWS_ACCESS_KEY = "MyAccessKey"
HTML_NAME = "S3PostForm.html"
SUCCESS_NAME = "success.html"
EXPIRES = 60*60*24*356 # seconds = 1 year
BUCKET = "media.mysite.com"
KEY = "${filename}" # will match file entered by user
ACL = "public-read" # or "private"
SUCCESS = "http://media.mysite.com/success.html"
CONTENT_TYPE = "" # seems to work this way
CONTENT_LENGTH = 1024**3 # One gigabyte
HTTP_OR_HTTPS = "http" # Or https for better security
PAGE_TITLE = "My Html Upload to S3 Form"

###### DON'T EDIT FROM HERE ON DOWN ######

conn = connection.S3Connection(AWS_ACCESS_KEY,AWS_SECRET_KEY)
args = conn.build_post_form_args(
    BUCKET,
    KEY,
    expires_in=EXPIRES,
    acl=ACL,
    success_action_redirect=SUCCESS,
    max_content_length=CONTENT_LENGTH,
    http_method=HTTP_OR_HTTPS,
    fields=None,
    conditions=None,
    storage_class='STANDARD',
    server_side_encryption=None,
    )

form_fields = ""
line = '  <input type="hidden" name="%s" value="%s" >\n'
for item in args['fields']:
    new_line = line % (item["name"], item["value"])
    form_fields += new_line

html_page_data = {
"page_title": PAGE_TITLE,
"action": args["action"],
"input_fields": form_fields,
}

html_page = """
<html> 
 <head>
  <title>%(page_title)s</title> 
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 </head>
<body>
 <form action="%(action)s" method="post" enctype="multipart/form-data" >
%(input_fields)s
  <!-- Include any additional input fields here -->

  Browse to locate the file to upload:<br \> <br \>

  <input name="file" type="file"><br> <br \>
  <input type="submit" value="Upload File to S3"> 
 </form> 
</body>
</html>
""" % html_page_data

with open(HTML_NAME, "wb") as f:
    f.write(html_page)

success_page = """
<html>
  <head>
    <title>S3 POST Success Page</title>
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
      <script src="jquery.js"></script>
      <script src="purl.js"></script>
<!--

    Amazon S3 passes three data items in the url of this page if
        the upload was successful:
        bucket = bucket name
        key = file name upload to the bucket
        etag = hash of file

    The following script parses these values and puts them in
    the page to be displayed.

-->

<script type="text/javascript">
var pname,url,val,params=["bucket","key","etag"];
$(document).ready(function()
{
  url = $.url();
  for (param in params)
  {
    pname = params[param];
    val = url.param(pname);
    if(typeof val != 'undefined')
      document.getElementById(pname).value = val;
  }
});
</script>

  </head>
  <body>
      <div style="margin:0 auto;text-align:center;">
      <p>Congratulations!</p>
      <p>You have successfully uploaded the file.</p>
        <form action="#" method="get"
          >Location:
        <br />
          <input type="text" name="bucket" id="bucket" />
        <br />File Name:
        <br />
          <input type="text" name="key" id="key" />
        <br />Hash:
        <br />
          <input type="text" name="etag" id="etag" />
      </form>
    </div>
  </body>
</html>
"""

with open(SUCCESS_NAME, "wb") as f:
    f.write(success_page)

###### Dump output if testing ######
if DEBUG:

    if 1: # Set true if not using the LEO editor
        class G:
            def es(self, data):print(data)
        g = G()

    g.es("conn = %s" % conn)
    for key in args.keys():
        if key is not "fields":
            g.es("%s: %s" % (key, args[key]))
            continue
        for item in args['fields']:
            g.es(item)
于 2013-12-01T23:24:25.973 回答
3

我尝试使用 Boto,但发现它并没有让我输入我想要的所有标题。您可以在下面看到我为生成策略、签名和发布表单值字典所做的工作。

请注意,所有 x-amz-meta-* 标记都是自定义标头属性,您不需要它们。另请注意,表单中的几乎所有内容都需要在经过编码和签名的策略中。

def generate_post_form(bucket_name, key, post_key, file_id, file_name, content_type):
  import hmac
  from hashlib import sha1
  from django.conf import settings
  policy = """{"expiration": "%(expires)s","conditions": [{"bucket":"%(bucket)s"},["eq","$key","%(key)s"],{"acl":"private"},{"x-amz-meta-content_type":"%(content_type)s"},{"x-amz-meta-file_name":"%(file_name)s"},{"x-amz-meta-post_key":"%(post_key)s"},{"x-amz-meta-file_id":"%(file_id)s"},{"success_action_status":"200"}]}"""
  policy = policy%{
    "expires":(datetime.utcnow()+settings.TIMEOUT).strftime("%Y-%m-%dT%H:%M:%SZ"), # This has to be formatted this way
    "bucket": bucket_name, # the name of your bucket
    "key": key, # this is the S3 key where the posted file will be stored
    "post_key": post_key, # custom properties begin here
    "file_id":file_id,
    "file_name": file_name,
    "content_type": content_type,
  }
  encoded = policy.encode('utf-8').encode('base64').replace("\n","") # Here we base64 encode a UTF-8 version of our policy.  Make sure there are no new lines, Amazon doesn't like them.
  return ("%s://%s.s3.amazonaws.com/"%(settings.HTTP_CONNECTION_TYPE, self.bucket_name),
          {"policy":encoded,
           "signature":hmac.new(settings.AWS_SECRET_KEY,encoded,sha1).digest().encode("base64").replace("\n",""), # Generate the policy signature using our Amazon Secret Key
           "key": key,
           "AWSAccessKeyId": settings.AWS_ACCESS_KEY, # Obviously the Amazon Access Key
           "acl":"private",
           "x-amz-meta-post_key":post_key,
           "x-amz-meta-file_id":file_id,
           "x-amz-meta-file_name": file_name,
           "x-amz-meta-content_type": content_type,
           "success_action_status":"200",
          })

然后可以使用返回的元组生成一个表单,该表单将字典中的所有键值对作为隐藏字段和您的实际文件输入字段(其名称/id 应为“文件”)发布到生成的 S3 url。

希望作为一个例子有所帮助。

于 2011-08-24T04:36:29.007 回答
1

几天来,我一直在努力解决同样的问题,使用几乎相同的代码。(请参阅Python Generated Signature for S3 Post)刚刚尝试根据 White Box Dev 的代码对我的策略进行编码,但仍然没有像 AWS 建议的那样提出我应该有的。我最终放弃并使用了...

http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html

...并将它返回的值插入 HTML 表单中。效果很好。

@先生。Oodles:如果您将 aws_secret_key 存储在单独的文件中,请使用 bash 命令 ls -al 在生成签名之前检查它的字节数。它应该是 40 字节长。正如 White Box Dev 指出的那样,AWS 不喜欢 \n,并且您可能在保存时将这个隐藏字符(或回车符或 ^M)与 aws_secret_key 字符串捆绑在一起......因此使其长度为 41 个字节. 您可以尝试 .replace("\n", "") 或 .rstrip() 在将其读入脚本后摆脱它, .encode("utf-8") 也可能对您有用。然而,这些都不适合我。好奇您是否在 Windows 或 Unix 上运行 Python ...您也可以尝试使用 emacs 保存字符串,而不会由编辑器自动插入 \n。

于 2012-01-31T01:24:41.640 回答
0

尝试检查https://github.com/burgalon/plupload-s3mixin 它结合了 PLUPLOAD 和直接 S3 上传

于 2011-09-05T08:39:05.320 回答