rabbitmq - How to handle long asynchronous requests with pyramid and celery?

Question

I'm setting up a web service with pyramid. A typical request for a view will be very long, about 15 min to finish. So my idea was to queue jobs with celery and a rabbitmq broker. I would like to know what would be the best way to ensure that bad things cannot happen. Specifically I would like to prevent the task queue from overflow for example.

A first mesure will be defining quotas per IP, to limit the number of requests a given IP can submit per hour.

However I cannot predict the number of involved IPs, so this cannot solve everything.

I have read that it's not possible to limit the queue size with celery/rabbitmq. I was thinking of retrieving the queue size before pushing a new item into it but I'm not sure if it's a good idea.

I'm not used to good practices in messaging/job scheduling. Is there a recommended way to handle this kind of problems ?

Answer 1

RabbitMQ在 QoS 中内置了流控制。如果 RabbitMQ 无法处理发布速率，它将调整 TCP 窗口大小以减慢发布者的速度。如果发送到服务器的消息过多，它也会溢出到磁盘。这将使您的消费者更加天真，尽管如果您在出错时重新启动连接并淹没连接，您可能会导致问题。

我一直决定花更多时间确保发布者/消费者可以使用多个队列服务器，而不是试图让他们对单个队列服务器更加智能。好处是，如果您真的超载单个服务器，您可以添加另一个（如果使用 RabbitMQ HA，则添加另一对。Pycon 提供了一个有用的视频，关于使用 Celery 和 RabbitMQ进行大规模消息传递，应该有用。