0

My application listen on kafka topic and dump data into cassandra. Threads loads some information from mongo too. Lag in kafka topic getting increased. I have seen that mostly threads are blocked while loading some class. I am attaching my thread_dump below.

 "KafkaConsumer-49" prio=10 tid=0x00007f1178fdd000 nid=0x78e0 waiting for monitor entry [0x00007f1155fb5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:403)
        - waiting to lock <0x00000006c0655b58> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
        at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103)
        at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75)
        at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840)
        at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)
        at com.snapdeal.coms.timemachine.mao.TimeMachineMao.getVendorProductsForUploadId(TimeMachineMao.java:32)
        at com.snapdeal.coms.timemachine.service.TimeMachineService.getVendorProductsForUploadIdAndSupc(TimeMachineService.java:35)
        at com.snapdeal.coms.timemachine.event.SupcUploadIdStateUpdateEventHandler.handleEvent(SupcUploadIdStateUpdateEventHandler.java:40)



KafkaConsumer-48" prio=10 tid=0x00007f1178fdb000 nid=0x78df waiting for monitor entry [0x00007f11560b6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:403)
        - waiting to lock <0x00000006c0655b58> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
        at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103)
        at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75)
        at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840)
        at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)
        at com.snapdeal.coms.timemachine.mao.TimeMachineMao.getVendorProductsForUploadId(TimeMachineMao.java:32)
        at com.snapdeal.coms.timemachine.service.TimeMachineService.getVendorProductsForUploadIdAndSupc(TimeMachineService.java:35)
        at com.snapdeal.coms.timemachine.event.SupcUploadIdStateUpdateEventHandler.handleEvent(SupcUploadIdStateUpdateEventHandler.java:40)
        at com.snapdeal.coms.timemachine.TimeMachine.onEvent(TimeMachine.java:109)

"KafkaConsumer-47" prio=10 tid=0x00007f1178fd9800 nid=0x78de waiting for monitor entry [0x00007f11561b7000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:403) - waiting to lock <0x00000006c0655b58> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at org.springframework.util.ClassUtils.forName(ClassUtils.java:258) at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103) at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75) at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840) at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)

"KafkaConsumer-46" prio=10 tid=0x00007f1178fd8000 nid=0x78dd waiting for monitor entry [0x00007f11562b8000]

java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:403) - waiting to lock <0x00000006c0655b58> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at org.springframework.util.ClassUtils.forName(ClassUtils.java:258) at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103) at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75) at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840) at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486) at com.snapdeal.coms.timemachine.mao.TimeMachineMao.getVendorProductsForUploadId(TimeMachineMao.java:32) at com.snapdeal.coms.timemachine.service.TimeMachineService.getVendorProductsForUploadIdAndSupc(TimeMachineService.java:35) at com.snapdeal.coms.timemachine.event.SupcUploadIdStateUpdateEventHandler.handleEvent(SupcUploadIdStateUpdateEventHandler.java:40)

I am not sure why all the threads are blocked. I thought class get loaded only one time and later no need to take any lock .

4

2 回答 2

0

问题在于从 mongo 获取数据。有大量数据并且没有实现分页，并且特定请求没有套接字超时，因此线程被阻塞。

于 2014-06-11T19:15:58.743 回答
0

您是否尝试使用ConsumerOffsetChecker来查看您的消费者是否还活着？您可以从$KAFKA_ROOT_DIR/文件夹中尝试以下命令

  bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect zkhost:zkport --topic topic1

这是从他们的常见问题页面中获取的一些注释

如果消费者偏移量在一段时间后没有移动，那么消费者很可能已经停止。如果消费者偏移量正在移动，但消费者滞后（日志末尾和消费者偏移量之间的差异）正在增加，则消费者比生产者慢。如果消费者速度慢，典型的解决方案是增加消费者的并行度。这可能需要增加主题的分区数。

上面的常见问题页面还解释了您的消费者被阻止的可能原因，可能值得一看。

于 2014-04-01T17:04:49.543 回答