我很困惑，也许我误解了一些东西，所以我会发布并看看有什么见解回来。
我有一个使用 JWT 进行身份验证的 Flask API 应用程序（bjoern WSGI）。我向烧瓶应用程序发送一些凭据和本地 URL，以访问将提供图像的 API。然后，flask 应用程序在图像上运行一些 ML，并回复检测数据。一旦客户端收到来自烧瓶应用程序的响应，它也会从 API 请求匹配的图像（双击）。
我想要做的是将flask应用程序中的匹配图像连同JSON格式的检测数据一起传递回客户端。我通过使用 requests-toolbox 和 MultiPartEncoder 实现了所有这些。我的问题是当我尝试使用 cv2.imdecode 将 response.content 字节编码为客户端的 jpeg 时。这是相关代码，我还将发布输出和错误。我不想将图像写入磁盘并将其读回 numpy 数组，我试图在内存中完成所有操作。
这是在客户端，这部分是将请求发送到烧瓶应用程序的地方。
do_raise = True
try:
from requests_toolbelt.multipart import decoder
r = requests.post(
url=ml_object_url,
headers=auth_header,
params=params,
# json doesnt send when sending files to mlapi, so we send the file and the json together
json=mlapi_json if not files else None,
files=files,
)
r.raise_for_status()
except ValueError as v_ex:
# pass as we will do a retry loop? -> urllib3 has a retry loop built in but idk if that works here
if v_ex == "BAD_IMAGE":
pass
except requests.exceptions.HTTPError as http_ex:
if http_ex.response.status_code == 400:
if args.get('file'):
g.logger.error(
f"There seems to be an error trying to send an image from zm_detect to mlapi, looking into it"
)
else:
g.logger.error(f"{http_ex.response.json()}")
elif http_ex.response.status_code == 500:
g.logger.error(f"There seems to be an Internal Error with the mlapi host, check mlapi logs!")
else:
g.logger.error(f"ERR CODE={http_ex.response.status_code} {http_ex.response.content=}")
except urllib3.exceptions.NewConnectionError as urllib3_ex:
g.logger.debug(f"{lp} {urllib3_ex.args=} {urllib3_ex.pool=}")
g.logger.error(
f"There seems to be an error while trying to start a new connection to the mlapi host -> {urllib3_ex}")
except requests.exceptions.ConnectionError as req_conn_ex:
g.logger.error(
f"There seems to be an error while trying to start a new connection to the mlapi host -> "
f"{req_conn_ex.response}"
)
except Exception as all_ex:
g.logger.error(
f"{lp} error during post to mlapi host-> {all_ex}"
)
# g.logger.debug(f"traceback-> {format_exc()}")
else:
do_raise = False
data: Optional[dict] = None
multipart_data: decoder.MultipartDecoder = decoder.MultipartDecoder.from_response(r)
part: decoder.MultipartDecoder.from_response
img: Optional[bytes] = None
for part in multipart_data.parts:
# part = part.headers.decode('utf-8')
if part.headers.get(b'Content-Type') == b'image/jpeg':
print(f' got an image')
img = part.content
print(f"{type(img)}")
print(f"{len(img)=}")
elif part.headers.get(b'Content-Type') == b'application/json':
print(f"got json data")
data = part.content.decode('utf-8')
print(data)
np_img = np.asarray(bytearray(img), dtype=np.uint8)
print(f"img after np,asarray(bytearray()) -> {type(np_img) = }")
print(f"{len(np_img)=}")
try:
new_img = cv2.imdecode(img, cv2.IMREAD_UNCHANGED)
except Exception as exc:
print(f"EXCEPTION while cv2.imdecode")
print(exc)
else:
print(f"img after cv2.imdecode -> {type(new_img) = }")
if new_img is not None:
if options.get("resize", 'no') != "no":
new_img = resize_image(img, options.get("resize"))
data["matched_data"]["image"] = new_img
else:
print(f"exiting due to image error")
g.logger.log_close(exit=1)
g.logger.log_close(exit=1)
exit(1)
return data
finally:
if do_raise:
raise ValueError('MLAPI remote detection error!')
这是烧瓶应用程序中的一些代码，用于处理从 API 抓取图像并将其编码以传递到 ML 模型管道。此代码按预期工作。
r = response
img = np.asarray(bytearray(response.content), dtype="uint8")
img = cv2.imdecode(img, cv2.IMREAD_COLOR) # RGB ?
self.orig_h_w = img.shape[:2]
return img
所以现在有一个包含img的变量被cv2.imdecode解码成jpg。然后可以将此图像格式 (numpy.ndarray) 传递给 opencv DNN 模块或传递给 pycoral 模型以进行 TPU 推理。这是我要发回给客户的图像。这是我用来完成此操作的代码。
img = matched_data['image'].tobytes()
# Remove the numpy.ndarray formatted image from matched_data because it is not JSON serializable
matched_data['image'] = None
# Construct a multipart response that contains the detection data and the image
success = False
if matched_data["frame_id"]:
success = True
resp_json = {
'success': success,
'matched_data': matched_data,
'all_matches': all_matches,
}
from requests_toolbelt import MultipartEncoder
multipart_encoded_data = MultipartEncoder(
fields={
'json': (None, json.dumps(resp_json), 'application/json'),
'image': (f"event-{g.eid}-frame-{matched_data['frame_id']}.jpg", img, 'image/jpeg')
}
)
response = Response(multipart_encoded_data.to_string(), mimetype=multipart_encoded_data.content_type)
if success:
g.logger.info(
f"{lp} returning matched detection -> {matched_data}",
)
g.logger.debug(
f"{lp} returning all detections -> {all_matches}")
else:
g.logger.info(
f"{lp} no detections to return"
)
return response
现在在客户端分离 JSON 和图像并将图像转换为可用格式 ->
do_raise = False
data: Optional[dict] = None
multipart_data: decoder.MultipartDecoder = decoder.MultipartDecoder.from_response(r)
part: decoder.MultipartDecoder.from_response
img: Optional[bytes] = None
for part in multipart_data.parts:
# part = part.headers.decode('utf-8')
if part.headers.get(b'Content-Type') == b'image/jpeg':
print(f' got an image')
img = part.content
print(f"{type(img)}")
print(f"{len(img)=}")
elif part.headers.get(b'Content-Type') == b'application/json':
print(f"got json data")
data = part.content.decode('utf-8')
print(data)
np_img = np.asarray(bytearray(img), dtype=np.uint8)
print(f"img after np,asarray(bytearray()) -> {type(np_img) = }")
print(f"{len(np_img)=}")
try:
new_img = cv2.imdecode(img, cv2.IMREAD_UNCHANGED)
except Exception as exc:
print(f"EXCEPTION while cv2.imdecode")
print(exc)
else:
print(f"img after cv2.imdecode -> {type(new_img) = }")
if new_img is not None:
if options.get("resize", 'no') != "no":
new_img = resize_image(img, options.get("resize"))
data["matched_data"]["image"] = new_img
else:
print(f"exiting due to image error")
g.logger.log_close(exit=1)
g.logger.log_close(exit=1)
exit(1)
return data
我收到的错误是无声的，它只返回“无”->
# Grabbing image using an http request and converting into a jpeg
11/07/21 20:44:30.623202 zm_mlapi[37535] DBG1 Media:659 ['std.out' --> image from ZM API as response.content - type(img) = <class 'bytes'> - len(img) = 205125]
11/07/21 20:44:30.627857 zm_mlapi[37535] DBG1 Media:661 ['std.out' --> after np.asarray(bytearray(img), np.uint8) - type(img) = <class 'numpy.ndarray'> - len(img) = 205125]
11/07/21 20:44:30.658582 zm_mlapi[37535] DBG1 Media:663 ['std.out' --> after cv2.imdecode(img, cv2.IMREAD_COLOR) - type(img) = <class 'numpy.ndarray'> - len(img) = 1080]
11/07/21 20:44:30.67595 zm_mlapi[37535] DBG2 pyzm_utils:386 [resize:img: success using resize=800.0 - original dimensions: 1920*1080 - resized dimensions: 450*800]
11/07/21 20:44:30.678568 zm_mlapi[37535] DBG1 Media:681 ['std.out' --> after resize - type(img) = <class 'numpy.ndarray'> - len(img) = 450]
# returned image to the class that requested it (ML Pipeline)
11/07/21 20:44:30.687835 zm_mlapi[37535] DBG1 detect_sequence:1048 ['std.out' --> DETECT STREAM: FRAME RETURNED FROM MEDIA CLASS --> type(frame) = <class 'numpy.ndarray'> - len(frame) = 450]
11/07/21 20:44:33.582062 zm_mlapi[37535] DBG1 detect_sequence:1656 ['std.out' --> before returning matched data - type(matched_data['image']) = <class 'numpy.ndarray'> - len(matched_data['image']) = 450]
# Return image to the flask app, now the flask app has to construct a response with JSON and the image
11/07/21 20:44:33.588139 zm_mlapi[37535] DBG1 mlapi:587 ['std.out' --> type(matched_data['image']) = <class 'numpy.ndarray'> - len(matched_data['image']) = 450]
11/07/21 20:44:33.591981 zm_mlapi[37535] DBG1 mlapi:590 ['std.out' --> before converting using .tobytes() - type(img) = <class 'numpy.ndarray'> - len(img) = 450]
11/07/21 20:44:33.596642 zm_mlapi[37535] DBG1 mlapi:594 ['std.out' --> after converting using .tobytes() - type(img) = <class 'bytes'> - len(img) = 1080000]
11/07/21 20:44:33.611218 zm_mlapi[37535] DBG1 mlapi:611 ['std.out' --> multipart MIME TYPE -> multipart/form-data; boundary=e7f7b825a51d4184ad7f12e7bbc6f411]
# flask app returns the response to the client
11/07/21 21:00:58.393864 zmesdetect_m4[102768] DBG1 zm_detect:418 ['std.out' --> got json data]
11/07/21 21:00:58.395459 zmesdetect_m4[102768] DBG1 zm_detect:414 ['std.out' --> got an image with Content-Type - b'application/octet']
11/07/21 21:00:58.396815 zmesdetect_m4[102768] DBG1 zm_detect:422 ['std.out' --> success = True]
11/07/21 21:00:58.398169 zmesdetect_m4[102768] DBG1 zm_detect:423 ['std.out' --> img - type(img) = <class 'bytes'> - len(img) = 1080000]
11/07/21 21:00:58.39958 zmesdetect_m4[102768] DBG1 zm_detect:424 ['std.out' --> img[:50] = b'\\gu\\gu\\gu]hv^iw_jx`kyalzgr\x80kv\x84it\x82it\x82it\x82it\x82it\x82it\x82ju']
11/07/21 21:00:58.401012 zmesdetect_m4[102768] DBG1 zm_detect:426 ['std.out' --> img after np.frombuffer(img, dtype=np.uint8) -> type(np_img) = <class 'numpy.ndarray'>]
11/07/21 21:00:58.402911 zmesdetect_m4[102768] DBG1 zm_detect:430 ['std.out' --> img after np_img.copy() -> type(np_img) = <class 'numpy.ndarray'>]
11/07/21 21:00:58.404296 zmesdetect_m4[102768] DBG1 zm_detect:432 ['std.out' --> len(np_img)=1080000]
11/07/21 21:00:58.405619 zmesdetect_m4[102768] DBG1 zm_detect:433 ['std.out' --> attempting to decode numpy array into a jpeg]
11/07/21 21:00:58.407144 zmesdetect_m4[102768] DBG1 zm_detect:442 ['std.out' --> img after cv2.imdecode -> type(new_img) = <class 'NoneType'>]
11/07/21 21:00:58.408474 zmesdetect_m4[102768] DBG1 zm_detect:448 ['std.out' --> exiting due to image error]
任何见解将不胜感激！我是新手，所以希望这是一个正确的问题。