我们的 WCF 服务存在问题,无法重现。有时,该服务不响应客户的呼叫。在不活动一段时间后,这种情况在星期一更常见。
WCF 服务在 Windows 服务中自承载。实例上下文是每次调用。它使用NetTcpBinding
没有安全性,整个 WCF 配置是在代码中完成的,没有 XML 配置。我们已将ServiceThrottle
会话、调用和实例的参数设置为 1024。这是完整的ServiceHost
配置:
ServiceThrottlingBehavior throttle;
throttle = _svcHost.Description.Behaviors.Find<ServiceThrottlingBehavior>();
if (throttle == null)
{
throttle = new ServiceThrottlingBehavior();
throttle.MaxConcurrentCalls = 1024;
throttle.MaxConcurrentSessions = 1024;
throttle.MaxConcurrentInstances = 1024;
_svcHost.Description.Behaviors.Add(throttle);
}
...
TimeSpan timeout = new TimeSpan(0, 0, 5);
NetTcpBinding binding = new NetTcpBinding(SecurityMode.None);
binding.OpenTimeout = timeout;
binding.CloseTimeout = timeout;
binding.ReceiveTimeout = timeout;
binding.SendTimeout = timeout;
binding.MaxBufferSize = 10485760;
binding.MaxReceivedMessageSize = 10485760;
XmlDictionaryReaderQuotas quotas = new XmlDictionaryReaderQuotas();
binding.ReaderQuotas = quotas;
binding.ReaderQuotas.MaxStringContentLength = 10485760;
binding.ReaderQuotas.MaxArrayLength = 10000;
binding.Security.Message.ClientCredentialType = MessageCredentialType.None;
...
ServiceEndpoint endpoint = _svcHost.AddServiceEndpoint(intfType, binding, serviceBaseAddress + "/" + intfType.Name);
endpoint.Behaviors.Add(new ClientTrackerEndpointBehavior());
问题出现在客户端抛出异常。有 5 到 10 个客户端连接到服务,每个客户端都会抛出这种类型的异常(即使是与服务本身在同一台机器上运行的客户端):
System.ServiceModel.CommunicationException
The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:29.9969997'.
An existing connection was forcibly closed by the remote host
StackTrace:
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
抛出异常后,我们尝试使用手动连接到服务telnet
,并且似乎此连接尝试也被拒绝了。由于本周一我们的 3 位客户在生产系统上遇到了无法将 Visual Studio 调试器附加到的问题,因此我们使用 WinDBG 创建了用户迷你转储来分析问题。我们检查的第一件事是当前值ServiceThrottle
(这里只有一个转储,但输出等同于其他转储生成的输出):
0:032> !dumpheap -type ServiceThrottle -short
01fb65a4
0:032> !do 01fb65a4
Name: System.ServiceModel.Dispatcher.ServiceThrottle
MethodTable: 70cc56f0
EEClass: 70a07ce4
Size: 40(0x28) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ServiceModel\v4.0_4.0.0.0__b77a5c561934e089\System.ServiceModel.dll
Fields:
MT Field Offset Type VT Attr Value Name
70cc572c 400314f 4 ...cher.FlowThrottle 0 instance 01fb6660 calls
70cc572c 4003150 8 ...cher.FlowThrottle 0 instance 01fb6760 sessions
71725030 4003151 c ...her.QuotaThrottle 0 instance 00000000 dynamic
70cc572c 4003152 10 ...cher.FlowThrottle 0 instance 02013110 instanceContexts
70cb653c 4003153 14 ...l.ServiceHostBase 0 instance 01fadd84 host
70cc7cf8 4003154 18 ...manceCountersBase 0 instance 02003590 servicePerformanceCounters
74246788 4003155 20 System.Boolean 1 instance 1 isActive
7423f744 4003156 1c System.Object 0 instance 01fb65cc thisLock
74242ad4 400314d 1134 System.Int32 1 static 128 DefaultMaxConcurrentCallsCpuCount
74242ad4 400314e 1138 System.Int32 1 static 800 DefaultMaxConcurrentSessionsCpuCount
0:032> !do 01fb6660
Name: System.ServiceModel.Dispatcher.FlowThrottle
MethodTable: 70cc572c
EEClass: 70a07d24
Size: 52(0x34) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ServiceModel\v4.0_4.0.0.0__b77a5c561934e089\System.ServiceModel.dll
Fields:
MT Field Offset Type VT Attr Value Name
74242ad4 4002ede 20 System.Int32 1 instance 1024 capacity
74242ad4 4002edf 24 System.Int32 1 instance 0 count
74246788 4002ee0 2c System.Boolean 1 instance 0 warningIssued
74242ad4 4002ee1 28 System.Int32 1 instance 89 warningRestoreLimit
7423f744 4002ee2 4 System.Object 0 instance 01fb6694 mutex
74232914 4002ee3 8 ...ding.WaitCallback 0 instance 01fb6640 release
00000000 4002ee4 c 0 instance 01fb66a0 waiters
7423fb08 4002ee5 10 System.String 0 instance 01fb65d8 propertyName
7423fb08 4002ee6 14 System.String 0 instance 01fb660c configName
74230f78 4002ee7 18 System.Action 0 instance 02007670 acquired
74230f78 4002ee8 1c System.Action 0 instance 02007690 released
0:032> !do 01fb6760
Name: System.ServiceModel.Dispatcher.FlowThrottle
MethodTable: 70cc572c
EEClass: 70a07d24
Size: 52(0x34) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ServiceModel\v4.0_4.0.0.0__b77a5c561934e089\System.ServiceModel.dll
Fields:
MT Field Offset Type VT Attr Value Name
74242ad4 4002ede 20 System.Int32 1 instance 1024 capacity
74242ad4 4002edf 24 System.Int32 1 instance 0 count
74246788 4002ee0 2c System.Boolean 1 instance 0 warningIssued
74242ad4 4002ee1 28 System.Int32 1 instance 560 warningRestoreLimit
7423f744 4002ee2 4 System.Object 0 instance 01fb6794 mutex
74232914 4002ee3 8 ...ding.WaitCallback 0 instance 01fb6740 release
00000000 4002ee4 c 0 instance 01fb67a0 waiters
7423fb08 4002ee5 10 System.String 0 instance 01fb66d0 propertyName
7423fb08 4002ee6 14 System.String 0 instance 01fb6708 configName
74230f78 4002ee7 18 System.Action 0 instance 020076b0 acquired
74230f78 4002ee8 1c System.Action 0 instance 020076d0 released
所有这些值似乎都很好。然后我们检查了线程池和线程:
0:032> !threadpool
CPU utilization: 0%
Worker Thread: Total: 1023 Running: 1017 Idle: 6 MaxLimit: 1023 MinLimit: 1000
Work Request in Queue: 0
--------------------------------------
Number of Timers: 4
--------------------------------------
Completion Port Thread:Total: 32 Free: 0 MaxFree: 16 CurrentLimit: 33 MaxLimit: 1000 MinLimit: 1000
0:032> !threads -special
ThreadCount: 1027
UnstartedThread: 997
BackgroundThread: 28
PendingThread: 997
DeadThread: 1
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 1d80 006b6518 a020 Enabled 00000000:00000000 006ac0b0 0 MTA
2 2 238c 006c1840 b220 Enabled 00000000:00000000 006ac0b0 0 MTA (Finalizer)
XXXX 4 00704f58 1019820 Enabled 00000000:00000000 006ac0b0 0 Ukn (Threadpool Worker)
4 5 21a4 00706480 3009220 Enabled 0bedaf78:0bedb5c8 006ac0b0 0 MTA (Threadpool Worker)
5 6 e8c 03de9428 100a220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
7 7 634 03e0d318 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
8 8 1d38 03ebeb08 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
9 9 1808 03e4fd70 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
10 a 1c48 03e50d70 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
11 b 1d88 073be2b0 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
12 c 1c74 073bf2b8 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
13 d 1ae4 073c0dc8 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
14 e 1818 073c1598 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
15 f 1a58 073c1fa8 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
16 10 13e0 073c4bb8 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
17 11 1a3c 073c5388 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
18 12 1b5c 03e9ffe0 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
19 13 1b80 03ea04e8 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
20 14 900 03ea09f0 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
21 15 1c84 03ea0ef8 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
22 16 da0 03ea1400 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
23 17 13b0 03ea1908 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
24 18 18cc 03ea1e10 3009220 Enabled 00000000:00000000 006ac0b0 0 MTA (Threadpool Worker)
25 1a 1008 03ea2820 3009220 Enabled 0bfe03b8:0bfe21c4 006ac0b0 0 MTA (Threadpool Worker)
27 29 bc4 0b041588 1009220 Enabled 0bdd7c3c:0bdd99f8 006ac0b0 0 MTA (Threadpool Worker)
28 3b ad8 0af7d740 1009220 Enabled 0bf8c0d8:0bf8c1c4 006ac0b0 0 MTA (Threadpool Worker)
29 64 dd8 0ae54890 1009220 Enabled 0bdfbb9c:0bdfd9f8 006ac0b0 0 MTA (Threadpool Worker)
30 2f 440 0b03c010 1009220 Enabled 0be03bf0:0be059f8 006ac0b0 0 MTA (Threadpool Worker)
31 25 2198 0b03f080 1009220 Enabled 0bd6d5b4:0bd6f410 006ac0b0 0 MTA (Threadpool Worker)
32 63 1b9c 0ae41388 1009220 Enabled 0bdb5b14:0bdb79f8 006ac0b0 0 MTA (Threadpool Worker)
XXXX 33 2270 0b09cd20 1400 Enabled 00000000:00000000 006ac0b0 0 Ukn
XXXX 5b 1554 0ae54388 1400 Enabled 00000000:00000000 006ac0b0 0 MTA
XXXX 31 1098 0ae53978 1400 Enabled 00000000:00000000 006ac0b0 0 Ukn
XXXX 34 15c 0af7be18 1400 Enabled 00000000:00000000 006ac0b0 0 Ukn
... -> lots of more threads here
XXXX 403 24e4 0d85b578 1400 Enabled 00000000:00000000 006ac0b0 0 Ukn
XXXX 404 24d8 0d85ba80 1400 Enabled 00000000:00000000 006ac0b0 0 Ukn
XXXX 405 24e0 0d85bf88 1400 Enabled 00000000:00000000 006ac0b0 0 Ukn
OSID Special thread type
1 a88 DbgHelper
2 238c Finalizer
4 21a4 ThreadpoolWorker
5 e8c Timer
7 634 ThreadpoolWorker
8 1d38 ThreadpoolWorker
9 1808 ThreadpoolWorker
10 1c48 ThreadpoolWorker
11 1d88 ThreadpoolWorker
12 1c74 ThreadpoolWorker
13 1ae4 ThreadpoolWorker
14 1818 ThreadpoolWorker
15 1a58 ThreadpoolWorker
16 13e0 ThreadpoolWorker
17 1a3c ThreadpoolWorker
18 1b5c ThreadpoolWorker
19 1b80 ThreadpoolWorker
20 900 ThreadpoolWorker
21 1c84 ThreadpoolWorker
22 da0 ThreadpoolWorker
23 13b0 ThreadpoolWorker
24 18cc ThreadpoolWorker
25 1008 ThreadpoolWorker
26 1b08 Gate
27 bc4 ThreadpoolWorker
28 ad8 ThreadpoolWorker
29 dd8 ThreadpoolWorker
30 440 ThreadpoolWorker
31 2198 ThreadpoolWorker
32 1b9c ThreadpoolWorker
当我们看到这么多线程时,我们感到震惊。现在我们怀疑我们的问题可能与大量线程有关。因此,如果有人可以回答以下问题,将不胜感激:
- 我们的假设是否正确?还是我们在调查线程时走错了路?
- !threadpool 命令输出大量正在运行的线程 (1027)。查看 !threads 的输出,似乎只有 28 个线程在工作。如何解释这些差异?
- 我们有大量未启动/待处理的线程。未启动线程和待处理线程有什么区别?我们试图重现这种行为,但即使在线程池中设置了最小和最大线程,我们也没有得到这么高的数字。更重要的是,调查在进程正常运行 2 小时后创建的转储,没有发现未启动或挂起的线程(此时服务仍在工作)。原始转储的过程正常运行时间约为 14 天。
- 完成端口线程空闲值0是什么意思?
- 我们可以使用 WinDBG 中的任何其他方法/命令来更好地理解我们的问题吗?
我们对我们软件的当前状态非常不满意,在查找有关此主题的信息时,通常会说这是一个未关闭的会话/并发调用问题,但如前所述,这似乎不是我们的问题。很感谢任何形式的帮助!