machine-learning - 使用 sk-learn 模型进行大查询预测

Question

我在本地机器上创建了一个 sklearn 模型。然后我把它上传到谷歌存储。我使用相同的模型在 AI Platform 中创建了模型和版本。它适用于在线预测。现在我想执行批量预测并将数据存储到大查询中，例如每次执行预测时都会更新大查询表。

有人可以建议我怎么做吗？

Answer 1

AI Platform does not support writing prediction results to BigQuery at the moment.

You can write the prediction results to BigQuery with Dataflow. There are two options here:

1. Create Dataflow job that makes the predictions itself.
2. Create Dataflow job that uses AI Platform to get the model's predictions. Probably this would use online predictions.

In both cases you can define a BigQuery sink to insert new rows to your table.

Alternatively, you can use Cloud Functions to update a BigQuery table whenever a new file appears in GCS. This solution would look like:

1. Use gcloud to run the batch prediction (`gcloud ml-engine jobs submit prediction ... --output-path="gs://[My Bucket]/batch-predictions/"
2. Results are written in multiple files: gs://[My Bucket]/batch-predictions/prediction.results-*-of-NNNNN
3. Cloud function is triggered to parse and insert the results to BigQuery. This Medium post explains how to this up setup