1

我已成功运行 ML Pipeline 实验,并顺利发布了 Azure ML Pipeline。当我在成功运行并发布后直接运行以下命令时(即我正在使用 Jupyter 运行所有单元格),测试失败!

interactive_auth = InteractiveLoginAuthentication()
auth_header = interactive_auth.get_authentication_header()

rest_endpoint = published_pipeline.endpoint
response = requests.post(rest_endpoint, 
                         headers=auth_header, 
                         json={"ExperimentName": "***redacted***",
                               "ParameterAssignments": {"process_count_per_node": 6}})
run_id = response.json()["Id"]

这是 azureml-logs/70_driver_log.txt 中的错误:

[2020-12-10T17:17:50.124303] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
3 items cleaning up...
Cleanup took 0.20258069038391113 seconds
Traceback (most recent call last):
  File "driver/amlbi_main.py", line 48, in <module>
    main()
  File "driver/amlbi_main.py", line 44, in main
    JobStarter().start_job()
  File "/mnt/batch/tasks/shared/LS_root/jobs/***redacted***/azureml/***redacted***/mounts/workspaceblobstore/azureml/***redacted***/driver/job_starter.py", line 52, in start_job
    job.start()
  File "/mnt/batch/tasks/shared/LS_root/jobs/***redacted***/azureml/***redacted***/mounts/workspaceblobstore/azureml/***redacted***/driver/job.py", line 105, in start
    master.wait()
  File "/mnt/batch/tasks/shared/LS_root/jobs/***redacted***/azureml/***redacted***/mounts/workspaceblobstore/azureml/***redacted***/driver/master.py", line 301, in wait
    file_helper.start()
  File "/mnt/batch/tasks/shared/LS_root/jobs/***redacted***/azureml/***redacted***/mounts/workspaceblobstore/azureml/***redacted***/driver/file_helper.py", line 206, in start
    self.analyze_source()
  File "/mnt/batch/tasks/shared/LS_root/jobs/***redacted***/azureml/***redacted***/mounts/workspaceblobstore/azureml/***redacted***/driver/file_helper.py", line 69, in analyze_source
    raise Exception(message)
Exception: No temp file found. The job failed. A job should generate temp files or should fail before this. Please check logs for the cause.

例外:找不到临时文件。作业失败。作业应生成临时文件或在此之前失败。请检查日志以查找原因。

以下是 logs/sys/warning.txt 中的错误:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://eastus.experiments.azureml.net/execution/v1.0/subscriptions/***redacted***/resourceGroups/***redacted***/providers/Microsoft.MachineLearningServices/workspaces/***redacted***/experiments/***redacted-experiment-name***/runs/***redacted-run-id***/telemetry

[...]

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url:

具有相同的 URL。

下一个...

当我等待几分钟并重新运行以下代码/单元格时。

interactive_auth = InteractiveLoginAuthentication()
auth_header = interactive_auth.get_authentication_header()

rest_endpoint = published_pipeline.endpoint
response = requests.post(rest_endpoint, 
                         headers=auth_header, 
                         json={"ExperimentName": "***redacted***",
                               "ParameterAssignments": {"process_count_per_node": 2}})
run_id = response.json()["Id"]

顺利完成!?嗯?我在这里更改了进程计数,但我认为这没有什么区别)。此外,日志中没有用户错误。

关于这里可能发生什么的任何想法?

提前感谢您提供的任何见解,并祝您编码愉快!:)

========== 更新#1:==========

在 1 个文件上运行约 300k 行。有时这项工作有效,有时则无效。我们已经尝试了许多具有不同配置设置的版本,都会不时导致失败。将 sklearn 模型更改为使用 n_jobs=1。我们正在对 NLP 工作的文本数据进行评分。

default_ds = ws.get_default_datastore()

# output dataset
output_dir = OutputFileDatasetConfig(destination=(def_file_store, 'model/results')).register_on_complete(name='model_inferences')

# location of scoring script
experiment_folder = 'model_pipeline'    

rit = 60*60*24

parallel_run_config = ParallelRunConfig(
    source_directory=experiment_folder,
    entry_script="score.py",
    mini_batch_size="5",
    error_threshold=10,
    output_action="append_row",
    environment=batch_env,
    compute_target=compute_target,
    node_count=5,
    run_invocation_timeout=rit,
    process_count_per_node=1
)

我们的下一个测试将是 - 将每一行数据放入自己的文件中。我只尝试了 30 行,即 30 个文件,每个文件有 1 条记录用于评分,但仍然得到相同的错误。这次我将错误阈值更改为 1。

2020-12-17 02:26:16,721|ParallelRunStep.ProgressSummary|INFO|112|The ParallelRunStep processed all mini batches. There are 6 mini batches with 30 items. Processed 6 mini batches containing 30 items, 30 succeeded, 0 failed. The error threshold is 1. 
2020-12-17 02:26:16,722|ParallelRunStep.Telemetry|INFO|112|Start concatenating.
2020-12-17 02:26:17,202|ParallelRunStep.FileHelper|ERROR|112|No temp file found. The job failed. A job should generate temp files or should fail before this. Please check logs for the cause.
2020-12-17 02:26:17,368|ParallelRunStep.Telemetry|INFO|112|Run status: Running
2020-12-17 02:26:17,495|ParallelRunStep.Telemetry|ERROR|112|Exception occurred executing job: No temp file found. The job failed. A job should generate temp files or should fail before this. Please check logs for the cause..
Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/jobs/**redacted**/mounts/workspaceblobstore/azureml/**redacted**/driver/job.py", line 105, in start
    master.wait()
  File "/mnt/batch/tasks/shared/LS_root/jobs/**redacted**/mounts/workspaceblobstore/azureml/**redacted**/driver/master.py", line 301, in wait
    file_helper.start()
  File "/mnt/batch/tasks/shared/LS_root/jobs/**redacted**/mounts/workspaceblobstore/azureml/**redacted**/driver/file_helper.py", line 206, in start
    self.analyze_source()
  File "/mnt/batch/tasks/shared/LS_root/jobs/**redacted**/mounts/workspaceblobstore/azureml/**redacted**/driver/file_helper.py", line 69, in analyze_source
    raise Exception(message)
Exception: No temp file found. The job failed. A job should generate temp files or should fail before this. Please check logs for the cause.

在它完成的回合中,只返回一些记录。一次返回的记录数我认为是 25 或 23,另一次是 15。

========== 更新 #2:12/17/2020 ==========

我删除了我的一个模型(我的模型是 15 个模型的重量混合)。我什至清理了我的文本字段,删除了所有的制表符、换行符和逗号。现在我正在为 30 个文件评分,每个文件有 1 条记录,有时工作会完成,但它不会返回 30 条记录。其他时候它返回一个错误,并且仍然得到“没有找到临时文件”错误。

4

1 回答 1

1

我想我可能已经回答了我自己的问题。我认为问题在于

OutputFileDatasetConfig

一旦我切换回使用

PipelineData

一切又开始工作了。我猜 Azure 在他们说 OutputFileDatasetConfig 仍处于试验阶段时并不是在开玩笑。

我仍然不明白的是,我们应该如何在没有 OutputFileDatasetConfig 的情况下从数据工厂管道中获取 ML Studio 管道的结果?PipelineData 根据子步骤运行 ID 将结果输出到文件夹中,那么数据工厂应该如何知道从哪里获取结果呢?很想听听任何人可能有的任何反馈。谢谢 :)

== 更新 ==

要从数据工厂管道中获取 ML Studio 管道的结果,请查看从数据工厂管道中的ML Studio 管道中获取结果

== 更新 #2 ==

https://github.com/Azure/azure-sdk-for-python/issues/16568#issuecomment-781526789

嗨@yeamusic21,感谢您的反馈,在当前版本中,OutputDatasetConfig 无法与 ParallelRunStep 一起使用,我们正在努力修复它。

于 2020-12-18T00:08:45.457 回答