7

Docker API 镜像创建/拉取(/v1.6/images/create)显然总是返回

HTTP/1.1 200 OK
Content-Type: application/json

不管这个过程是成功还是失败。

此外,有效负载不是有效的 json。

例如:/v1.6/images/create?fromImage=whatevertheflush

返回:

{"status":"Pulling repository whatevertheflush"}{"error":"Server error: 404 trying to fetch remote history for whatevertheflush","errorDetail":{"code":404,"message":"Server error: 404 trying to fetch remote history for whatevertheflush"}}

不是有效的 json,并且 HTTP 错误没有被转发/使用使得为客户端处理错误变得很尴尬。

实际上,docker-py 只是吐出有效负载(https://github.com/dotcloud/docker-py/blob/master/docker/client.py#L374)。并且来自 openstack 的 DockerHTTPClient 尝试根据 http 错误代码返回一个值,该值始终为 200 ...(https://github.com/openstack/nova/blob/master/nova/virt/docker/client.py# L191 )

现在,我知道拉取可能需要很长时间,并且开始向客户端发送答案有点意义,但我不禁认为这里出了点问题。

所以,这是三折:

  • 我在这里完全错过了什么吗?
  • 如果不是:如果您正在实现一个客户端应用程序(例如,在 Python 中),您将如何处理这个(如果可能的话,优雅地:))?尝试检测有效的 json 块,加载它们,并在我们“认为”有问题时退出?
  • 如果不是:这会在未来的 docker 版本中改变(变得更好)吗?
4

3 回答 3

6

这个问题有点老了,但是对于登陆这个页面的未来读者,我想让你知道你并不孤单,我们感受到你的痛苦。这个 API 确实和看起来一样可怕。

TL;DR 答案是/images/create响应格式未记录;/images/XXX/json在您的创建调用完成后丢弃输出和查询。”

几年前我写了一些编排工具,我发现/images/createAPI 非常烦人。但让我们深入了解:

  • 没有记录在案的200响应模式;v1.19 文档仅给出了一些记录的示例。v1.37(我写这篇文章时最新的)文档甚至没有走那么远,没有提供任何详细信息。
  • 响应以 发送Transfer-Encoding: chunked,并且发送的每条记录前面都有十六进制的字节数。这是一个低级的摘录(绕过 curl,所以我们可以看到实际通过网络发送的内容):
    host-4:~ rg$ telnet localhost 2375
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.
    POST /images/create?fromImage=jenkins/jenkins:latest HTTP/1.1
    Host: localhost:2375
    User-Agent: foo/1.0
    Accept: */*

    HTTP/1.1 200 OK
    Api-Version: 1.39
    Content-Type: application/json
    Docker-Experimental: true
    Ostype: linux
    Server: Docker/18.09.1 (linux)
    Date: Wed, 06 Feb 2019 16:53:19 GMT
    Transfer-Encoding: chunked

    39
    {"status":"Pulling from jenkins/jenkins","id":"latest"}

    5e
    {"status":"Digest: sha256:abd3e3f96fbc3445c420fda590f37e2bd3377f69affd47b63b3d826d084c5ddc"}

    45
    {"status":"Status: Image is up to date for jenkins/jenkins:latest"}

    0
  • 是的,它流式传输图像下载进度——不提供对分块记录的低级访问权限的客户端库可能只是在将数据提供给您之前将其连接起来。如您所见,API 的早期版本返回 JSON 记录,唯一的分隔符是分块传输编码,因此客户端代码收到一个未分隔的 JSON 连接块,并且必须通过跟踪 curlies/quotes/escape 字符来解析它!它已经更新为现在发出由换行符分隔的记录,但我们可以指望它们总是在那里吗?谁知道!此行为无需任何仪式即可更改,并且如果您在较新的守护程序上调用较旧版本的 API,则不会保留此行为。
  • 200 OK立即返回,不代表成功或失败。(考虑到调用的性质,我想它可能应该返回202 Accepted。理想情况下,我们会得到一个Location指向一个新 URL 的标头,我们可以用它来查询进度/状态。)
  • 返回的响应数据是巨大的、垃圾邮件的,而且只是......愚蠢。如果您有一个在 TCP 上侦听的 docker 实例,请尝试curl -Nv -X POST http://yourdocker:2375/images/create?fromImage=jenkins/jenkins:latest -o /tmp/omgwtf.txt. 你会很惊讶。传输服务器渲染的 ASCII 条形图浪费了大量带宽!!!. 事实上,记录以三种不同的方式返回每一层的进度,作为当前和总字节数的数字字段、作为条形图以及作为具有 MB 或 GB 单位的漂亮打印字符串。为什么这不只是在客户端上呈现?好问题。
    相反,您需要您的客户端解析千字节或兆字节的垃圾邮件。
  • 条形图有一个随机转义的>字符 unicode 表示,尽管它安全地位于 JSON 字符串中。有人只是在墙上发出逃生电话,看看是什么卡住了?¯\_(ツ)_/¯
  • 记录本身是相当随意的。有一个id字段会更改它所引用的内容,并且是了解解析人类可读字符串的记录类型的唯一方法。 Pulling from XXXvs Pulling fs layervsDownloading等等。据我所知,知道它是否完成的唯一真正方法是跟踪所有 id,并确保Pull complete在套接字关闭时获得一个 for each。
  • 您也许可以寻找,Status: Downloaded newer image for XXX但我不确定是否有多种可能的响应。
  • /images/XXX/json正如我在开始时提到的那样,在声明完成后,您可能会获得最好的运气/images/create。这两个调用的组合将给出一个非常可靠的指示是否/images/create有效。

这是一个较长的串联客户端响应块,显示了几种不同的记录类型。为简洁起见编辑:


    {"status":"Pulling from jenkins/jenkins","id":"latest"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"ab1fc7e4bf91"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"35fba333ff52"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"f0cb1fa13079"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"3d1dd648b5ad"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"a9f886e483d6"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"4346341d3c49"}
    ..
    "status":"Waiting","progressDetail":{},"id":"3d1dd648b5ad"}
    {"status":"Waiting","progressDetail":{},"id":"a9f886e483d6"}
    {"status":"Waiting","progressDetail":{},"id":"4346341d3c49"}
    {"status":"Waiting","progressDetail":{},"id":"006f2208d67a"}
    {"status":"Waiting","progressDetail":{},"id":"fb85cf26717d"}
    {"status":"Waiting","progressDetail":{},"id":"52ca068dbca7"}
    {"status":"Waiting","progressDetail":{},"id":"82f4759b8d12"}
    ...
    {"status":"Downloading","progressDetail":{"current":110118,"total":10780995},"progress":"[\u003e                                                  ]  110.1kB/10.78MB","id":"35fba333ff52"}
    {"status":"Downloading","progressDetail":{"current":457415,"total":45344749},"progress":"[\u003e                                                  ]  457.4kB/45.34MB","id":"ab1fc7e4bf91"}
    {"status":"Downloading","progressDetail":{"current":44427,"total":4340040},"progress":"[\u003e                                                  ]  44.43kB/4.34MB","id":"f0cb1fa13079"}
    {"status":"Downloading","progressDetail":{"current":817890,"total":10780995},"progress":"[===\u003e                                               ]  817.9kB/10.78MB","id":"35fba333ff52"}
    {"status":"Downloading","progressDetail":{"current":1833671,"total":45344749},"progress":"[==\u003e                                                ]  1.834MB/45.34MB","id":"ab1fc7e4bf91"}
    {"status":"Downloading","progressDetail":{"current":531179,"total":4340040},"progress":"[======\u003e                                            ]  531.2kB/4.34MB","id":"f0cb1fa13079"}
    {"status":"Downloading","progressDetail":{"current":1719010,"total":10780995},"progress":"[=======\u003e                                           ]  1.719MB/10.78MB","id":"35fba333ff52"}
    {"status":"Downloading","progressDetail":{"current":3205831,"total":45344749},"progress":"[===\u003e                                               ]  3.206MB/45.34MB","id":"ab1fc7e4bf91"}
    {"status":"Downloading","progressDetail":{"current":1129195,"total":4340040},"progress":"[=============\u003e                                     ]  1.129MB/4.34MB","id":"f0cb1fa13079"}
    {"status":"Downloading","progressDetail":{"current":2640610,"total":10780995},"progress":"[============\u003e                                      ]  2.641MB/10.78MB","id":"35fba333ff52"}
    {"status":"Downloading","progressDetail":{"current":1719019,"total":4340040},"progress":"[===================\u003e                               ]  1.719MB/4.34MB","id":"f0cb1fa13079"}
    {"status":"Downloading","progressDetail":{"current":4586183,"total":45344749},"progress":"[=====\u003e                                             ]  4.586MB/45.34MB","id":"ab1fc7e4bf91"}
    {"status":"Downloading","progressDetail":{"current":3549922,"total":10780995},"progress":"[================\u003e                                  ]   3.55MB/10.78MB","id":"35fba333ff52"}
    {"status":"Downloading","progressDetail":{"current":2513643,"total":4340040},"progress":"[============================\u003e                      ]  2.514M
    ...
    {"status":"Pull complete","progressDetail":{},"id":"6d9b49fc8a28"}
    {"status":"Extracting","progressDetail":{"current":380,"total":380},"progress":"[==================================================\u003e]     380B/380B","id":"6302e8b6563c"}
    {"status":"Extracting","progressDetail":{"current":380,"total":380},"progress":"[==================================================\u003e]     380B/380B","id":"6302e8b6563c"}
    {"status":"Pull complete","progressDetail":{},"id":"6302e8b6563c"}
    {"status":"Extracting","progressDetail":{"current":1548,"total":1548},"progress":"[==================================================\u003e]  1.548kB/1.548kB","id":"7348f018cf93"}
    {"status":"Extracting","progressDetail":{"current":1548,"total":1548},"progress":"[==================================================\u003e]  1.548kB/1.548kB","id":"7348f018cf93"}
    {"status":"Pull complete","progressDetail":{},"id":"7348f018cf93"}
    {"status":"Extracting","progressDetail":{"current":3083,"total":3083},"progress":"[==================================================\u003e]  3.083kB/3.083kB","id":"c651ee7bd59e"}
    {"status":"Extracting","progressDetail":{"current":3083,"total":3083},"progress":"[==================================================\u003e]  3.083kB/3.083kB","id":"c651ee7bd59e"}
    {"status":"Pull complete","progressDetail":{},"id":"c651ee7bd59e"}
    {"status":"Digest: sha256:abd3e3f96fbc3445c420fda590f37e2bd3377f69affd47b63b3d826d084c5ddc"}
    {"status":"Status: Downloaded newer image for jenkins/jenkins:latest"}

这段代码现在在 Internet 上运行。=8-O

于 2019-02-06T17:20:07.813 回答
0

这个特定的端点实际上返回分块编码。通过 curl 的示例:

$ curl -v -X POST http://localhost:4243/images/create?fromImage=base
* About to connect() to localhost port 4243 (#0)
*   Trying ::1...
* Connection refused
*   Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 4243 (#0)
> POST /images/create?fromImage=base HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
> Host: localhost:4243
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 07 Feb 2014 04:21:59 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
{"status":"Pulling repository base"}{"status":"Pulling image (ubuntu-quantl) from      base","progressDetail":{},"id":"b750fe79269d"}{"status":"Pulling image (ubuntu-quantl) from base, endpoint: https://cdn-registry-1.docker.io/v1/","progressDetail":{},"id":"b750fe79269d"}{"status":"Pulling dependent layers","progressDetail":{},"id":"b750fe79269d"}{"status":"Download complete","progressDetail":{},"id":"27cf78414709"}{"status":"Download complete","progressDetail":{},"id":"b750fe79269d"}{"status":"Download complete","progressDetail":{},"id":"b750fe79269d"}* Closing connection #0

现在我不确定你如何在 Python 中解析它,但在 Ruby 中,我可以像这样使用Yajl

parts = []
Yajl::Parser.parse(body) { |o| parts << o }
puts parts
{"status"=>"Pulling repository base"}
{"status"=>"Pulling image (ubuntu-quantl) from base", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Pulling image (ubuntu-quantl) from base, endpoint: https://cdn-registry-1.docker.io/v1/", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Pulling dependent layers", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"27cf78414709"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"b750fe79269d"}
{"status"=>"Download complete", "progressDetail"=>{}, "id"=>"b750fe79269d"}
于 2014-02-07T04:31:06.440 回答
0

使用 Docker v1.9 我仍然有这个问题需要处理。在 Docker Github 存储库上也发现了一个问题:Docker 在某些 API 函数中使用了无效的 JSON 格式 #16925

一些贡献者建议使用Content-Type这样的 HTTP 标头:application/json; boundary=NL 这对我不起作用。

然后,在使用我的自定义解析器时,发现了这个问题 StackOverflow:如何处理大量 JSON 字典?

于 2016-02-08T15:44:22.517 回答