0

I'm trying to emulate the stream drawing from Sources.mapJournal through IMap which receives data from IoT device. The processing of this stream is too slow and I'm getting the big accumulated outcome after 30-60 seconds.

When I started to update the IMap frequently with small data (12 KB per value), the exception is:

com.hazelcast.ringbuffer.StaleSequenceException: sequence:123 is too small and data store is disabled.

I increased the default capacity of IMap journal 10 times. It became stable after that, but very slow. A similar issue is when I'm updating the IMap with big values (about 1.2 MB per 5 seconds). Additionally I have several connected IoT devices and each of them has its own Jet job with the same pipeline:

StreamStage<TagPosition> sourceSteam = 
    p.drawFrom(Sources.<TagPosition, String, TagPosition>mapJournal(
        Constants.IMAP_TAGS_POSITIONS_BUFFER,
        Util.mapPutEvents().and(entry -> ((String) entry.getKey()).startsWith(instanceNumber)),
        Util.mapEventNewValue(),
        JournalInitialPosition.START_FROM_OLDEST));

// Drain to SmartMap
sourceSteam.drainTo(SmartMapSinks.newTagPositionSink(instanceNumber));

Thanks in advance!

UPD:

  • The journal size is EventJournalConfig.DEFAULT_CAPACITY * 10 = 100 000 (1 partition)
  • Jet version is 0.7.2
  • Serialazable classes implements com.hazelcast.nio.serialization.IdentifiedDataSerializable
4

1 回答 1

0

问题在于多个作业使用单个 IMap（和地图日志）。地图日志产生的事件就像一个有延迟的批次，但不是流。

解决了。谢谢！

于 2019-01-25T08:22:35.337 回答