I am using debezium mongodb connector 0.3.6 , running inside a docker container. I have been monitoring kafka-connect for some time ,and found that the connector stops tailing mongodb and sending change events to kafka brokers automatically. Upon investigation , I found that sometimes, after some time of inactivity, it's mongo connection is refused, and upon retrial, it connects successfully ,and sends the large number of records that it has not sent during inactive period. But this is not the case everytime, as I have encountered this inactivity for 1 hour straight, and upon restarting the connector deliberately, I can get it to tail the oplog properly, and hence sends an enormous number of records that were pending during inactivity.
I am attaching 2 log files: - 1- consisting logs where the mongo connection is refused, and it automatically retry and connects successfully. (https://ufile.io/c5v5g) 2- consisting logs where for 1 hour, it was not tailing mongodb oplog, and no change events were sent to kafka , even when new events were written in mongodb oplog. (https://ufile.io/9g4gz)
I am not able to find a particular problem here. My kafka-connect container, mongodb host, and kafka brokers are all running on different hosts, and connected by public ip.