14

我正在尝试将 Google Cloud Tasks 的速率限制为每秒不超过 1 个已处理的任务。

我创建了我的队列：

gcloud tasks queues create my-queue \
          --max-dispatches-per-second=1 \
          --max-concurrent-dispatches=1 \
          --max-attempts=2 \
          --min-backoff=60s

描述它给了我：

name: projects/my-project/locations/us-central1/queues/my-queue
rateLimits:
  maxBurstSize: 10
  maxConcurrentDispatches: 1
  maxDispatchesPerSecond: 1.0
retryConfig:
  maxAttempts: 2
  maxBackoff: 3600s
  maxDoublings: 16
  minBackoff: 60s
state: RUNNING

在创建了一堆任务后，我可以在日志中看到其中许多任务在 1 秒的时间段内被不希望地处理：

2019-07-27 02:37:48 default[20190727t043306]  Received task with payload: {'id': 51}
2019-07-27 02:37:48 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 52}
2019-07-27 02:37:49 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 53}
2019-07-27 02:37:49 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 54}
2019-07-27 02:37:49 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 55}
2019-07-27 02:37:49 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 56}
2019-07-27 02:37:49 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 57}
2019-07-27 02:37:49 default[20190727t043306]  "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306]  Received task with payload: {'id': 58}

如何正确强制它在这 1 秒的时间间隔内运行不超过 1 个任务？

更新 30/06：

我用基本设置再次尝试过，同样的问题。

有关设置和过程的更多详细信息：

  1. 源代码https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/tasks，无修改
  2. 部署 app.yaml，而不是 app.flexible.yaml
  3. 多次触发任务：python create_app_engine_queue_task.py --project=$PROJECT_ID --queue=$QUEUE_ID --location=$LOCATION_ID --payload=hello
  4. 检查日志：gcloud app logs read

这次他们花了一段时间才开始处理，但在那之后似乎他们都或多或少地同时处理了：

完整日志：

2019-07-30 00:22:37 default[20190730t021951]  [2019-07-30 00:22:37 +0000] [9] [INFO] Starting gunicorn 19.9.0
2019-07-30 00:22:37 default[20190730t021951]  [2019-07-30 00:22:37 +0000] [9] [INFO] Listening at: http://0.0.0.0:8081 (9)
2019-07-30 00:22:37 default[20190730t021951]  [2019-07-30 00:22:37 +0000] [9] [INFO] Using worker: threads
2019-07-30 00:22:37 default[20190730t021951]  [2019-07-30 00:22:37 +0000] [23] [INFO] Booting worker with pid: 23
2019-07-30 00:22:37 default[20190730t021951]  [2019-07-30 00:22:37 +0000] [26] [INFO] Booting worker with pid: 26
2019-07-30 00:27:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:27:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:27:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:27:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:27:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:27:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:42 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:42 default[20190730t021951]  Received task with payload: hello
2019-07-30 00:37:43 default[20190730t021951]  "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:43 default[20190730t021951]  Received task with payload: hello
4

3 回答 3

6

tl; dr 它可能按预期工作。您可以期待最初的任务爆发maxBurstSize，然后减慢到maxDispatchesPerSecond.

这样做的原因是“令牌桶”算法：有一个桶可以容纳最多maxBurstSize令牌，并且最初在其中有那么多令牌。如果计划的时间已经到达并且桶中有一个令牌并且少于maxConcurrentDispatches正在飞行的令牌，则调度任务，否则我们等待满足这些条件。调度任务时，会从存储桶中删除一个令牌。当桶未满时，令牌以maxDispatchesPerSecond. 因此，速率并不是对任务分派的精确限制。只要桶中有令牌并且任务准备好运行，任务就可以以任意速率发送。只有当任务必须等待令牌时，我们才必须放慢到给定的速度。由于存储桶已满，您可以获得初始爆发。

在 Cloud Tasks API 和控制台中，存储桶大小是只读的（API 调用它max_burst_size）。但是使用较旧的queue.yaml配置，您可以控制存储桶大小以及其他参数，例如

queue:
- name: my-appengine-queue
  rate: 2/s
  bucket_size: 20
  max_concurrent_requests: 5

然后gcloud app deploy queue.yaml。但是，如果您这样做，请注意以下陷阱：https ://cloud.google.com/tasks/docs/queue-yaml#pitfalls

仅供参考，有一个问题可以查看是否可以改进文档。

于 2019-08-01T19:01:04.003 回答
0

还没有。设置 --max-burst-size=1。

简短的回答：现在使用 Cloud Tasks API 是不可能的，因为 Max Burst Size 配置没有公开。

解决方法：使用 App Engine 的 queue.yaml https://cloud.google.com/tasks/docs/queue-yaml管理您的任务

如果您也对此功能感兴趣，可以投票支持他们将其移植https://issuetracker.google.com/issues/138813037

于 2019-10-25T12:49:51.840 回答
0

使用 queue.yaml 创建的队列和由 Cloud Tasks 创建的队列（无论您如何操作）都是相同的队列。但是，如果您混合使用 queue.yaml 和 Cloud Tasks 队列管理方法，可能会出现问题。有关更多信息，请参阅https://cloud.google.com/tasks/docs/queue-yaml。

于 2019-10-25T00:39:12.970 回答