我们已经设置了具有三个节点的 MySQL NDB Cluster,这是设置的详细信息。
Node1 - MultithreadedDataNode1、SQLNode1
Node2 - MultithreadedDataNode2、SQLNode2
Node3 - 管理节点
我们正在测试一些故障转移方案以确保设置正常工作。在客户端,我们使用 MySQLConnector v5.1.30 和 c3p0 连接池。客户端配置了多主控参数。因此,如果其中一个节点或应用程序出现故障,则应在另一个节点上执行 JDBC 查询并继续响应。
但是当 SQLNode(Port: 3306) 出现故障时,我们会观察到异常行为。连接不会故障转移到另一个节点。
配置如下:
<bean id="myDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="com.mysql.jdbc.ReplicationDriver" /> <!-- useConfigs=clusterBase -->
<property name="jdbcUrl" value="jdbc:mysql:replication://192.168.102.22,192.168.102.23/hhmefep?autoReconnect=true&failOverReadOnly=false&roundRobinLoadBalance=true" />
<property name="user" value="root"/>
<property name="password" value="changeit"/>
<property name="acquireIncrement" value="2" />
<property name="minPoolSize" value="50" />
<property name="maxPoolSize" value="100" />
<property name="maxStatements" value="2000" />
<property name="maxIdleTime" value="3600" />
<property name="connectionTesterClassName" value="com.mysql.jdbc.integration.c3p0.MysqlConnectionTester" />
<property name="testConnectionOnCheckin" value="true" />
<property name="testConnectionOnCheckout" value="false" />
<property name="idleConnectionTestPeriod" value="30" />
</bean>
当我启动客户端时,这两个节点已启动并正在运行。之后,我从第一个节点(192.168.102.22)杀死了 mysqld 进程,我得到了以下 stackTrace,
org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:99)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2536)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at com.mi.persistence.dao.cdr.BganActiveCallsDaoImpl.findBganActiveCallsBy_Imsi_Nsapi_Apn(BganActiveCallsDaoImpl.java:53)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy23.findBganActiveCallsBy_Imsi_Nsapi_Apn(Unknown Source)
at com.mi.persistence.service.cdr.BganActiveCallsServiceImpl.findBganActiveCallsBy_Imsi_Nsapi_Apn(BganActiveCallsServiceImpl.java:22)
at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy24.findBganActiveCallsBy_Imsi_Nsapi_Apn(Unknown Source)
at com.mi.persistence.service.callhandler.BganAccessRequestServiceImpl.getPreeptedCalls(BganAccessRequestServiceImpl.java:384)
at com.mi.taskobjects.externalsigobj.radiusobj.BGANAccessRequest.process(BGANAccessRequest.java:175)
at com.mi.entities.scheduler.SchWorkerThread.run(SchWorkerThread.java:165)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 19,578 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
at sun.reflect.GeneratedConstructorAccessor66.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1127)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3715)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4155)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.LoadBalancedMySQLConnection.execSQL(LoadBalancedMySQLConnection.java:159)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2212)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:802)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
... 42 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3161)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615)
... 56 more
Apr 24, 2014 2:13:13 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@18138d9 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Apr 24, 2014 2:13:13 PM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@18138d9 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@2dbdcf (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1088a28 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1a67efd (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@1db547d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@87fa45
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@13fcd69
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1c6dc6d
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@b198d1
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1a7ce01
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@1c5be25
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1469706
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@1654503
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@195f1b4
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@12f893d
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1db4400
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@4d7ab3
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b29aa5
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@1c6fb4f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@9d2c8e
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@d843e0
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask@504b76
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@18a408c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@193e4a4
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@87d60d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1f6002a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@184e113
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@d4bbb0
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6e4a4c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@cfc9f4
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1d3c0f3
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@120989c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1034f45
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@497a1b
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1ef8e0b
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@568dc9
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@76bb9c
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.lang.Thread.sleep(Native Method)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2396)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
com.mysql.jdbc.LoadBalancingConnectionProxy.createConnectionForHost(LoadBalancingConnectionProxy.java:383)
com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:75)
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:711)
com.mysql.jdbc.LoadBalancingConnectionProxy.dealWithInvocationException(LoadBalancingConnectionProxy.java:415)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:654)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:549)
$Proxy22.ping(Unknown Source)
com.mysql.jdbc.ReplicationConnection.ping(ReplicationConnection.java:986)
sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.mchange.v2.c3p0.impl.NewProxyConnection.rawConnectionOperation(NewProxyConnection.java:1315)
com.mysql.jdbc.integration.c3p0.MysqlConnectionTester.activeCheckConnection(MysqlConnectionTester.java:78)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:301)
com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1606)
com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.lang.Thread.sleep(Native Method)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2396)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
com.mysql.jdbc.LoadBalancingConnectionProxy.createConnectionForHost(LoadBalancingConnectionProxy.java:383)
com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:75)
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:711)
com.mysql.jdbc.LoadBalancingConnectionProxy.dealWithInvocationException(LoadBalancingConnectionProxy.java:415)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:654)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:549)
$Proxy22.ping(Unknown Source)
com.mysql.jdbc.ReplicationConnection.ping(ReplicationConnection.java:986)
sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.mchange.v2.c3p0.impl.NewProxyConnection.rawConnectionOperation(NewProxyConnection.java:1315)
com.mysql.jdbc.integration.c3p0.MysqlConnectionTester.activeCheckConnection(MysqlConnectionTester.java:78)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:301)
com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1606)
com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.lang.Thread.sleep(Native Method)
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2396)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2326)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
com.mysql.jdbc.LoadBalancingConnectionProxy.createConnectionForHost(LoadBalancingConnectionProxy.java:383)
com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:75)
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:711)
com.mysql.jdbc.LoadBalancingConnectionProxy.dealWithInvocationException(LoadBalancingConnectionProxy.java:415)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:654)
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:549)
$Proxy22.ping(Unknown Source)
com.mysql.jdbc.ReplicationConnection.ping(ReplicationConnection.java:986)
sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.mchange.v2.c3p0.impl.NewProxyConnection.rawConnectionOperation(NewProxyConnection.java:1315)
com.mysql.jdbc.integration.c3p0.MysqlConnectionTester.activeCheckConnection(MysqlConnectionTester.java:78)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:301)
com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1606)
com.mchange.v2.resourcepool.BasicResourcePool.access$200(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1228)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
如果我遗漏任何东西,请纠正我。
谢谢,
Parimal