0

我最近编写了一个 Python 脚本，用于将本地的、以换行符分隔的 JSON 文件上传到 BigQuery 表。它与此处官方文档中提供的示例非常相似。我遇到的问题是我尝试上传的文件中的非 ASCII 字符正在使我的 POST 请求无效。

这是脚本的相关部分......

def upload(dataFilePath, loadJob, recipeJSON, logger):
    body = '--xxx\n'
    body += 'Content-Type: application/json; charset=UTF-8\n\n'
    body += loadJob
    body += '\n--xxx\n' 
    body += 'Content-Type: application/octet-stream\n\n'

    dataFile = io.open(dataFilePath, 'r', encoding = 'utf-8')
    body += dataFile.read()
    dataFile.close()

    body += '\n--xxx--\n'

    credentials = buildCredentials(recipeJSON['keyPath'], recipeJSON['accountEmail'])
    http = httplib2.Http()
    http = credentials.authorize(http)
    service = build('bigquery', 'v2', http=http)

    projectId = recipeJSON['projectId']

    url = BIGQUERY_URL_BASE + projectId + "/jobs"

    headers = {'Content-Type': 'multipart/related; boundary=xxx'}
    response, content = http.request(url, method="POST", body=body, headers=headers)

...这是我在运行时得到的堆栈跟踪...

Traceback (most recent call last):
  File "/usr/local/uploader/upload_data.py", line 179, in <module>
    main(sys.argv)
  File "/usr/local/uploader/upload_data.py", line 170, in main
    if (upload(unprocessedFile, loadJob, recipeJSON, logger)):
  File "/usr/local/uploader/upload_data.py", line 100, in upload
    response, content = http.request(url, method="POST", body=body, headers=headers)
  File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 128, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 490, in new_request
redirections, connection_type)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1253, in _conn_request
conn.request(method, request_uri, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 833, in _send_output
    self.send(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/ssl.py", line 229, in sendall
    v = self.send(data[count:])
  File "/usr/local/lib/python2.7/ssl.py", line 198, in send
    v = self._sslobj.write(data)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 4586-4611: ordinal not in range(128)

我正在使用 Python 2.7 和以下库：distribute (0.6.36) google-api-python-client (1.1) httplib2 (0.8) oauth2client (1.1) pyOpenSSL (0.13) python-gflags (2.0) wsgiref (0.1.2) )

有没有其他人有这个问题？

似乎 httplib2 的请求方法将“body”作为字符串，这意味着它稍后需要在通过网络发送之前进行编码。我一直在寻找一种将编码覆盖为 UTF-8 的方法，但到目前为止还没有运气。

提前致谢！

编辑：

我可以通过做两件事来解决这个问题：1.) 读取我的原始文件内容而不进行解码。（我也可以在上面的第一次尝试中对“body”进行编码......） 2.）将 url 和 headers 编码为字节。

代码最终看起来像这样：

def upload(dataFilePath, loadJob, recipeJSON, logger):
    part_one = '--xxx\n'
    part_one += 'Content-Type: application/json; charset=UTF-8\n\n'
    part_one += loadJob
    part_one += '\n--xxx\n'
    part_one += 'Content-Type: application/octet-stream\n\n'

    dataFile = io.open(dataFilePath, 'rb')
    part_two = dataFile.read()
    dataFile.close()

    part_three = '\n--xxx--\n'

    body = part_one.encode('utf-8')
    body += part_two
    body += part_three.encode('utf-8')

    credentials = buildCredentials(recipeJSON['keyPath'], recipeJSON['accountEmail'])
    http = httplib2.Http()
    http = credentials.authorize(http)
    service = build('bigquery', 'v2', http=http)

    projectId = recipeJSON['projectId']

    url = BIGQUERY_URL_BASE + projectId + "/jobs"

    headers = {'Content-Type'.encode('utf-8'): 'multipart/related; boundary=xxx'.encode('utf-8')}
    response, content = http.request(url.encode('utf-8'), method="POST", body=body, headers=headers)
4

1 回答 1

1

io.open()将文件作为unicode文本打开。要么使用 plain open()，要么使用二进制模式：

dataFile = io.open(dataFilePath, 'rb')

您正在通过网络直接发送文件内容，因此您需要发送字节，而不是 unicode，并且正如您发现的那样，混合 Unicode 和字节会导致痛苦的错误，因为 python 尝试使用 ASCII 编解码器自动编码回字节连接两种不同的类型。这里根本不需要解码为 Unicode 。

于 2013-05-21T23:08:49.140 回答