我在生产环境中运行负载测试,发现存在某种并发问题。我正在运行一个查询(平均)需要 900 毫秒才能返回我的结果。我所看到的是:如果我使用 JMeter 以低于 900 毫秒的延迟(在 20 秒内增加 20 个用户)对并发用户运行查询,则响应时间开始以某种滚雪球效应增加。
最后的时间是:
1 15:39:14.641 Thread Group 1-1 Query 981 Success 31020 948 31
2 15:39:15.041 Thread Group 1-2 Query 1451 Success 31020 1412 30
3 15:39:15.446 Thread Group 1-3 Query 1943 Success 31020 1868 30
4 15:39:15.843 Thread Group 1-4 Query 2367 Success 31020 2333 29
5 15:39:16.244 Thread Group 1-5 Query 2842 Success 31020 2805 56
6 15:39:16.648 Thread Group 1-6 Query 3385 Success 31020 3334 30
7 15:39:17.044 Thread Group 1-7 Query 3890 Success 31020 3857 30
8 15:39:17.444 Thread Group 1-8 Query 4423 Success 31020 4375 29
9 15:39:17.851 Thread Group 1-9 Query 4952 Success 31020 4908 31
10 15:39:18.246 Thread Group 1-10 Query 5447 Success 31020 5409 31
11 15:39:18.650 Thread Group 1-11 Query 5952 Success 31020 5918 30
12 15:39:19.047 Thread Group 1-12 Query 6459 Success 31020 6423 32
13 15:39:19.452 Thread Group 1-13 Query 6994 Success 31020 6952 30
14 15:39:19.854 Thread Group 1-14 Query 7494 Success 31020 7459 30
15 15:39:20.253 Thread Group 1-15 Query 7976 Success 31020 7933 31
16 15:39:20.649 Thread Group 1-16 Query 8476 Success 31020 8424 30
17 15:39:21.055 Thread Group 1-17 Query 9001 Success 31020 8959 30
18 15:39:21.451 Thread Group 1-18 Query 9549 Success 31020 9489 30
19 15:39:21.850 Thread Group 1-19 Query 10057 Success 31020 10020 75
20 15:39:22.255 Thread Group 1-20 Query 10616 Success 31020 10562 28
21 15:39:22.657 Thread Group 1-21 Query 11071 Success 31020 11038 29
22 15:39:23.053 Thread Group 1-22 Query 11620 Success 31020 11583 45
23 15:39:23.453 Thread Group 1-23 Query 12125 Success 31020 12080 30
24 15:39:23.854 Thread Group 1-24 Query 12622 Success 31020 12577 32
25 15:39:24.259 Thread Group 1-25 Query 13141 Success 31020 13099 31
26 15:39:24.661 Thread Group 1-26 Query 13623 Success 31020 13586 29
27 15:39:25.056 Thread Group 1-27 Query 14182 Success 31020 14113 30
28 15:39:25.457 Thread Group 1-28 Query 14769 Success 31020 14616 31
29 15:39:25.855 Thread Group 1-29 Query 15158 Success 31020 15100 34
30 15:39:26.258 Thread Group 1-30 Query 15617 Success 31020 15580 48
31 15:39:26.662 Thread Group 1-31 Query 16095 Success 31020 16063 31
32 15:39:27.058 Thread Group 1-32 Query 16639 Success 31020 16556 32
33 15:39:27.457 Thread Group 1-33 Query 17105 Success 31020 17059 34
34 15:39:27.860 Thread Group 1-34 Query 17614 Success 31020 17573 30
35 15:39:28.262 Thread Group 1-35 Query 18113 Success 31020 18076 30
36 15:39:28.658 Thread Group 1-36 Query 18631 Success 31020 18597 38
37 15:39:29.058 Thread Group 1-37 Query 19088 Success 31020 19045 39
38 15:39:29.458 Thread Group 1-38 Query 19577 Success 31020 19541 31
39 15:39:29.861 Thread Group 1-39 Query 20081 Success 31020 20042 30
40 15:39:30.264 Thread Group 1-40 Query 20611 Success 31020 20556 30
41 15:39:30.660 Thread Group 1-41 Query 21082 Success 31020 21046 30
42 15:39:31.065 Thread Group 1-42 Query 21595 Success 31020 21546 32
43 15:39:31.460 Thread Group 1-43 Query 22080 Success 31020 22033 31
44 15:39:31.862 Thread Group 1-44 Query 22597 Success 31020 22553 33
45 15:39:32.262 Thread Group 1-45 Query 23075 Success 31020 23041 129
46 15:39:32.662 Thread Group 1-46 Query 23586 Success 31020 23549 55
47 15:39:33.062 Thread Group 1-47 Query 24100 Success 31020 24061 31
48 15:39:33.463 Thread Group 1-48 Query 24546 Success 31020 24503 30
49 15:39:33.865 Thread Group 1-49 Query 25053 Success 31020 24991 29
50 15:39:34.265 Thread Group 1-50 Query 25512 Success 31020 25470 30
我想这是一些配置问题。
我已经通过http://neo4j.com/developer/in-production/正确设置了我的环境,它使用的是 Amazon AWS m3.large 实例:
- 7.5Gb。内存
- 2核处理器
- 4Gb 堆大小
- 最多 320 个线程
任何人都可以对此有所了解吗?
编辑
查询是:
START station=node:LocationIndex('withinDistance:[51.51766116938769,-0.09774009180783261,8.0]'), user=node(9999)
MATCH (tt:TimeTable)<--(line:Line)<-[h:HasTrainLine]-station
WHERE tt.from <= 2100 AND tt.to >= 2100 and tt-[:validOn]->(:Weekday{dow:5})
WITH station, tt, line, user
OPTIONAL MATCH user-[r:isFav]->station
OPTIONAL MATCH user-[r2:Uses]->line
WITH station, r, sum(r2.timesUsed) as times, tt
RETURN station, times, not(r is NULL) as fav
ORDER BY times DESC, station.name ASC
SKIP 0
LIMIT 16
已编辑
我在运行负载测试时从 jvm 获取了线程转储 - > https://gist.github.com/aabutaleb/1a2cb6d1067bdff96be8
我在第961行发现了一些有趣的东西,它显示 Spatial 获得了一个写锁,但这没有意义,因为这个查询中没有任何文字......
0x00007fbb1bce0414 __pthread_cond_wait + 0xc4
0x00007fbb1ae8cadd _ZN13ObjectMonitor4waitElbP6Thread + 0x9bd
0x00007fbb1acfc378 JVM_MonitorWait + 0x168
0x00007fbb11012d78 * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0x00007fbb11006058 * java.lang.Object.wait() bci:2 line:503 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.kernel.impl.locking.community.RWLock.deadlockGuardedWait(javax.transaction.Transaction, org.neo4j.kernel.impl.locking.community.RWLock$TxLockElement, org.neo4j.kernel.impl.transaction.LockType) bci:29 line:551 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.kernel.impl.locking.community.RWLock.acquireWriteLock(javax.transaction.Transaction) bci:40 line:319 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.kernel.impl.locking.community.LockManagerImpl.getWriteLock(java.lang.Object, javax.transaction.Transaction) bci:7 line:68 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.kernel.impl.locking.community.CommunityLockClient.acquireExclusive(org.neo4j.kernel.impl.locking.Locks$ResourceType, long[]) bci:82 line:73 (Interpreted frame)
0x00007fbb11006706 * org.neo4j.kernel.impl.api.LockingStatementOperations.nodeSetProperty(org.neo4j.kernel.impl.api.KernelStatement, long, org.neo4j.kernel.api.properties.DefinedProperty) bci:35 line:280 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.kernel.impl.api.OperationsFacade.nodeSetProperty(long, org.neo4j.kernel.api.properties.DefinedProperty) bci:17 line:545 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.kernel.impl.core.NodeProxy.setProperty(java.lang.String, java.lang.Object) bci:46 line:254 (Interpreted frame)
0x00007fbb11006706 * org.neo4j.gis.spatial.rtree.RTreeIndex.saveCount() bci:67 line:435 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.gis.spatial.rtree.RTreeIndex.initIndexMetadata() bci:111 line:399 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.gis.spatial.rtree.RTreeIndex.<init>(org.neo4j.graphdb.GraphDatabaseService, org.neo4j.graphdb.Node, org.neo4j.gis.spatial.rtree.EnvelopeDecoder, int) bci:54 line:63 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.gis.spatial.LayerRTreeIndex.<init>(org.neo4j.graphdb.GraphDatabaseService, org.neo4j.gis.spatial.Layer, int) bci:15 line:49 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.gis.spatial.LayerRTreeIndex.<init>(org.neo4j.graphdb.GraphDatabaseService, org.neo4j.gis.spatial.Layer) bci:5 line:45 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.gis.spatial.DefaultLayer.initialize(org.neo4j.gis.spatial.SpatialDatabaseService, java.lang.String, org.neo4j.graphdb.Node) bci:158 line:281 (Interpreted frame)
0x00007fbb11006706 * org.neo4j.gis.spatial.DefaultLayer.makeLayerInstance(org.neo4j.gis.spatial.SpatialDatabaseService, java.lang.String, org.neo4j.graphdb.Node, java.lang.Class) bci:22 line:251 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.gis.spatial.DefaultLayer.makeLayerFromNode(org.neo4j.gis.spatial.SpatialDatabaseService, org.neo4j.graphdb.Node) bci:64 line:219 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.gis.spatial.SpatialDatabaseService.getLayer(java.lang.String) bci:84 line:111 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.gis.spatial.SpatialDatabaseService.getOrCreateLayer(java.lang.String, java.lang.Class, java.lang.Class, java.lang.String) bci:16 line:202 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.gis.spatial.SpatialDatabaseService.getOrCreateEditableLayer(java.lang.String, java.lang.String, java.lang.String) bci:34 line:175 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.gis.spatial.indexprovider.LayerNodeIndex.<init>(java.lang.String, org.neo4j.graphdb.GraphDatabaseService, java.util.Map) bci:187 line:97 (Interpreted frame)
0x00007fbb11006058 * org.neo4j.gis.spatial.indexprovider.SpatialIndexImplementation.nodeIndex(java.lang.String, java.util.Map) bci:10 line:47 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.kernel.impl.coreapi.IndexManagerImpl.getOrCreateNodeIndex(java.lang.String, java.util.Map) bci:39 line:318 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.kernel.impl.coreapi.IndexManagerImpl.forNodes(java.lang.String, java.util.Map) bci:7 line:302 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.kernel.impl.coreapi.IndexManagerImpl.forNodes(java.lang.String) bci:3 line:294 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$NodeOperations.indexQuery(java.lang.String, java.lang.Object) bci:16 line:172 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.cypher.internal.compiler.v2_1.spi.DelegatingOperations.indexQuery(java.lang.String, java.lang.Object) bci:7 line:128 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations.org$neo4j$cypher$internal$compiler$v2_1$spi$ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$super$indexQuery(java.lang.String, java.lang.Object) bci:3 line:142 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$indexQuery$1.apply() bci:12 line:142 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$indexQuery$1.apply() bci:1 line:142 (Interpreted frame)
0x00007fbb11374194 * org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext.org$neo4j$cypher$internal$compiler$v2_1$spi$ExceptionTranslatingQueryContext$$translateException(scala.Function0) bci:1 line:149 (Compiled frame)
* org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations.indexQuery(java.lang.String, java.lang.Object) bci:14 line:142 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.cypher.internal.compiler.v2_1.spi.DelegatingOperations.indexQuery(java.lang.String, java.lang.Object) bci:7 line:128 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.cypher.internal.compiler.v2_1.executionplan.builders.EntityProducerFactory$$anonfun$2$$anonfun$applyOrElse$2.apply(org.neo4j.cypher.internal.compiler.v2_1.ExecutionContext, org.neo4j.cypher.internal.compiler.v2_1.pipes.QueryState) bci:24 line:67 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.cypher.internal.compiler.v2_1.executionplan.builders.EntityProducerFactory$$anonfun$2$$anonfun$applyOrElse$2.apply(java.lang.Object, java.lang.Object) bci:9 line:65 (Interpreted frame)
0x00007fbb118019e4 * org.neo4j.cypher.internal.compiler.v2_1.executionplan.builders.EntityProducerFactory$$anon$1.apply(org.neo4j.cypher.internal.compiler.v2_1.ExecutionContext, org.neo4j.cypher.internal.compiler.v2_1.pipes.QueryState) bci:6 line:35 (Compiled frame)
* org.neo4j.cypher.internal.compiler.v2_1.executionplan.builders.EntityProducerFactory$$anon$1.apply(java.lang.Object, java.lang.Object) bci:9 line:34 (Compiled frame)
* org.neo4j.cypher.internal.compiler.v2_1.pipes.matching.MonoDirectionalTraversalMatcher.findMatchingPaths(org.neo4j.cypher.internal.compiler.v2_1.pipes.QueryState, org.neo4j.cypher.internal.compiler.v2_1.ExecutionContext) bci:6 line:45 (Interpreted frame)
0x00007fbb110068e1 * org.neo4j.cypher.internal.compiler.v2_1.pipes.TraversalMatchPipe$$anonfun$internalCreateResults$1.apply(org.neo4j.cypher.internal.compiler.v2_1.ExecutionContext) bci:14 line:35 (Interpreted frame)
0x00007fbb11006233 * org.neo4j.cypher.internal.compiler.v2_1.pipes.TraversalMatchPipe$$anonfun$internalCreateResults$1.apply(java.lang.Object) bci:5 line:32 (Interpreted frame)
但是,当我从查询中删除地理空间语句时,似乎没有出现任何锁定。https://gist.github.com/aabutaleb/e3d6b1c72644d330cc82