问题标签 [vertx-kafka-client]

问问题

For questions regarding programming in ECMAScript (JavaScript/JS) and its various dialects/implementations (excluding ActionScript). Note JavaScript is NOT the same as Java! Please include all relevant tags on your question; e.g., [node.js], [jquery], [json], [reactjs], [angular], [ember.js], [vue.js], [typescript], [svelte], etc.

2 问题
Newest
Active
Bountied
318
Unanswered
0 投票
0 回答
28 浏览

apache-kafka - vertx kafka 客户端使用消息并按顺序执行 REST api 调用

例如，主题中的记录，例如，

当记录{"createtime":"2021-10-29 00:03:49.0"}到来时，处理程序执行了一个 REST api 调用来保存createtime=2021-10-29 00:03:49.0到 mongodb。
当第二个记录到来时，createtime=2021-10-29 00:03:50.0将被保存。然后将保存
第三条记录。 等等。我希望这些消息能够按顺序处理。因此，最后一条消息中的创建时间将是 DB 中的最终值。createtime=2021-10-29 00:03:56.0
2021-10-29 00:13:56.0

当我运行我的代码时，结果值是随机的，前一个值（例如2021-10-29 00:03:56.0）可能是最终值。我怎样才能让处理程序按顺序运行？

0 投票
1 回答
45 浏览

apache-kafka - Cannot scale Kafka Consumer Groups

Setup: 1 Topic (test-topic) with 4 partitions

  • Vert.x: 4.0.3
  • RxJava2 : 2.2.12
  • Vert.x internally uses kafka-client: 2.6.0
  • Kafka broker: 2.8.1

Scenario 1: When 1 consumer group with 1 consumer is used, the system takes 60 seconds to complete processing 100 messages

Scenario 2: When 1 consumer group with 4 consumers are used, then the system takes around 15 seconds to process same 100 messages

Scenario 3: When 4 consumer groups having same group id with 1 consumer per group is used, then the system takes 60 seconds to process.

My assumption was in Scenario 3, the time taken to process all the messages would be around 15 seconds but that is not the case. I verified by logs that the partitions are distributed and all the partitions recieve 25 messages. Can anyone please help me understand what I might be doing wrong here that adding consumer groups does not scale? Is this behavior normal (which I do not think so)

Note:

  • The consumer and producer setting are default and the message delivery is "exactly-once".
  • The code is tested on local setup and on AWS too.

Psuedo Code:


12345678910