neo4j - Using SQS for batch running

Question

I am starting a new architecture to support a MDM (Master data management) database. For the MDM database I will be using a graph database (Neo4J). So the I need to integrate disparate sources within the MDM database. I will have sources from orders, customers signup, likes from facebook and so on.

So I imagine that I can have one or multiple SQS queues for the sources. So each source will put messages on SQS.

Then I will have many worker nodes that will be responsible to get the messages from queue and update the MDM database.

At the moment, I will use just one worker, because I dont know how neo4j will performance doing multiple writers.

So I see many people using SQS to send messages to launch a batch process. Its like a job queueing.

Is it a common use case use SQS to send data messages? Or maybe using another AWS component.

Answer 1

将 Neo4j 与排队系统一起使用是很常见的。我自己从未使用过 SQS，但我知道有人使用 ActiveMQ（或其他人）将数据输入 Neo4j。

这主要是为了解耦架构并提供一种优雅地处理负载峰值的方法。

根据您放入队列的事件类型，如果消费者是单线程的，这可能是有益的——尤其是。当事件写入图表的同一部分时。否则 Neo4j 的锁定机制将阻止并发写入到相同的节点/rels。