0

I have this use case that I have 2 topics

Topic 1 (Units) -> P0 / Topic 2 (Reservations) -> P0

I have a single consumer that needs to have the up to data from both topics/partitions in order to take the correct decision (either delete a unit if not reserved or reserve the unit if it exists)

I decided to have them in 2 different topics for concept segregation as well as at any single time I would need a 3rd service to interact with units only then that's possible in an easy way.

But the idea of how to handle the concurrent operation that could arise or maybe delayed events from a topic over another topic?

Thanks

4

1 回答 1

2

假设每个单独的主题（或分区）都按您的喜好排序，您可以使用pause()、resume()和seek()调用来实现这一点。

为每个主题启动一个消费者，然后如果它“超前”另一个主题，则暂停（）它，并在另一个主题赶上时恢复（）。

这基本上是合并两个排序列表，就在 kafka 之上。

根据您是否使用 subscribe() 或 assign()，您可能仍需要继续轮询暂停的消费者以不触发重新平衡。

如果有问题的主题没有完全排序（意味着某些事件发布的时间太晚），你需要本地状态（基本上是一个窗口流来流连接）。它可以自己实现，但这是流处理框架开始派上用场的地方。

于 2019-09-02T18:37:16.137 回答