apache-kafka - On demand horizontally scaling event driven architectures

Question

What is the best way to horizontally scale an event driven architecture when load increases?

1. Many people suggest using Kakfa as the message queue source for EDA however Kafka only allows one consumer in a consumer group per partition. Repartitioning especially during heavy load situations can be costly and time consuming.
2. Having many consumers in a consumer group that take work and acknowledge quickly would give some horizontal scaling but now message order needs to be considered as well as load completion.
3. With RabbitMQ queues can be created and deleted on the fly however that would require an additional orchestrator to help manage and distribute load.

Also none of this addresses the load balancing problem that comes with the territory.

Any help would be appreciated. Thanks

Answer 1

回答有点晚了，但这里有，

您认为缩放应该发生在消息总线层的推理并不完全正确。如果我们采用端到端方案，负载增加意味着对前端（API 层）的传入请求增加。请参阅下面链接中的参考事件驱动架构。

假设存在某种形式的自动缩放（Kubernetes 复制因子、Amazon 自动缩放器），前端将向外扩展以处理额外负载。在初始预处理之后，服务会将事件发布到事件驱动架构中的消息队列中。

在 Kafka 中，主题分区是一个横向扩展单元，因为一个生产者可以写入一个分区。通常，您会根据单个分区的吞吐量提前定义分区数。

正如参考文章提到的，如果单分区吞吐量为p并且您需要t作为吞吐量，那么您需要t/p分区。

如果 t 是正常预期负载的吞吐量，您可以提前为 2x、3x、10x 吞吐量创建配置，然后通过创建尽可能多的分区来正常运行。

通常，单个分区的吞吐量超过 10 Mb/s。