java - 如何解决代理示例交通锁和 ssl 握手锁的死锁？

Question

我结合了大部分网络示例 HexdumpProxy (http://netty.io/docs/stable/xref/org/jboss/netty/example/proxy/) 和 SecureChat (http://netty.io/docs/stable/ xref/org/jboss/netty/example/securechat）形成一个支持 SSL（和非 SSL）的代理（到非 SSL 后端）。这似乎是一个好主意，事实证明很简单，这正是我目前所需要的。

代理示例代码使用交通锁锁作为 2010 年报告的竞争条件 (http://markmail.org/message/x7jc6mqx6ripynqf) 的解决方案，该竞争条件发生在饱和通道周围，并设置传入和传出通道的可写和可读状态。

现在，在我的组合示例中，在较高负载下，这会导致死锁，因为 SSL 代码中的另一个锁“握手锁”是交织在一起的。请参阅下面的分析器诊断输出。

我担心，即使在阅读了原始讨论之后，我也无法理解流量锁的潜在问题足以找到解决此死锁的直接解决方案。

（这是netty 3.2.6）

Java-level deadlock has been detected

This means that some threads are blocked waiting to enter a synchronization block or
waiting to reenter a synchronization block after an Object.wait() call, where each thread
owns one monitor while trying to obtain another monitor already held by another thread.

Deadlock:


New I/O client worker #1-2 is waiting to lock java.lang.Object@7900f3c9 which is held by New I/O server worker #1-2
New I/O server worker #1-2 is waiting to lock java.lang.Object@2d854f2f which is held by New I/O client worker #1-2




Thread stacks


New I/O client worker #1-2 [BLOCKED; waiting to lock java.lang.Object@7900f3c9]
org.jboss.netty.handler.ssl.SslHandler.wrap(SslHandler.java:665) <== sync handshakelock
org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:461)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776)
org.jboss.netty.channel.Channels.write(Channels.java:632)
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
org.jboss.netty.channel.Channels.write(Channels.java:611)
org.jboss.netty.channel.Channels.write(Channels.java:578)
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
com.activevideo.frontend.ProxyInboundHandler$OutboundHandler.messageReceived(ProxyInboundHandler.java:162) <== sync trafficlock
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(unknown source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(unknown source)
java.lang.Thread.run(unknown source)


New I/O server worker #1-2 [BLOCKED; waiting to lock java.lang.Object@2d854f2f]
com.activevideo.frontend.ProxyInboundHandler.channelInterestChanged(ProxyInboundHandler.java:138) <== sync trafficlock
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:192)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:192)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:192)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:116)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
org.jboss.netty.channel.Channels.fireChannelInterestChanged(Channels.java:335)
org.jboss.netty.channel.socket.nio.NioWorker.setInterestOps(NioWorker.java:728)
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:129)
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:430)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776)
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
org.jboss.netty.channel.Channels.setInterestOps(Channels.java:652)
org.jboss.netty.channel.AbstractChannel.setInterestOps(AbstractChannel.java:222)
org.jboss.netty.channel.AbstractChannel.setReadable(AbstractChannel.java:244)
com.activevideo.frontend.SSLProxyInboundHandler$BackendConnector$1.operationComplete(SSLProxyInboundHandler.java:92)
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:148)
com.activevideo.frontend.SSLProxyInboundHandler$BackendConnector.operationComplete(SSLProxyInboundHandler.java:87)
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:367)
org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)
org.jboss.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1040)
org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:838)
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:907)    <=== sync handshakelock
org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:620)
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(unknown source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(unknown source)
java.lang.Thread.run(unknown source)

score 2 · Accepted Answer

好吧，我想我修好了。

代理示例中早期竞争条件的解决方案不必要地包括在同步块中调用 write()。

最初的问题是传出线程（TO）和传入线程（TI）之间存在这种（罕见的）竞争条件：

TO：inboundChannel.write()（在 messageReceived 处）
TO：inboundChannel.isWritable() 返回 false（在 messageReceived 处）
然后刷新在 (1) 发出的未决写入
TI：inboundChannel.isWritable() 返回 true（在 channelInterestChanged）
TI：outboundChannel.setReadable(true)（在 channelInterestChanged）
TO：outboundChannel.setReadable(false)（在 messageReceived 处）

2010 年的解决方案是在 inboundChannel 和 outboundChannel（以及 InterestChanged 处理程序）的 messageReceived() 处理程序中设置可读标志时引入同步（使用“trafficlock”），如下所示：

synchronized (trafficLock) {
  outboundChannel.write(msg);
  // If outboundChannel is saturated, do not read until notified in
  // OutboundHandler.channelInterestChanged().
  if (!outboundChannel.isWritable()) {
    e.getChannel().setReadable(false);
  }
}

这确实解决了竞态条件，因为您可以防止步骤 3、4 和 5 干扰步骤 2 和 6。但是，将步骤 1（write()）留在同步块之外是安全的。

对 write() 的调用导致了死锁，因为在 SSLHandler 中调用的过程中，它使用了另一个锁握手锁。

所以.. 我将调用 write() 移到两个位置的同步块之外。现在僵局已经过去了。我建议相应地更改“官方”代理示例。

java - 如何解决代理示例交通锁和 ssl 握手锁的死锁？

1 回答 1

Related

Reference