wcf - 带有 WCF 的 IIS 进程随机崩溃

Question

需要帮助调试服务器崩溃的原因。Win 2008 上的 IIS 服务器，CPU 负载正常，内存使用日志，似乎没有任何问题。

随机 w3p 进程崩溃。在事件日志中，我首先看到：

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: System.Runtime.Serialization.ni.dll, version: 4.0.30319.18044, time stamp: 0x51271348
Exception code: 0xc00000fd
Fault offset: 0x0000000000091ac6
Faulting process id: 0x1928
Faulting application start time: 0x01ceaf1919acdfd4
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runteb92aa12#\9a8ecadcc9e05f435158bedc1f1a3e74\System.Runtime.Serialization.ni.dll
Report Id: 5beef11c-1b0c-11e3-8eec-0026b9347fe0

然后我看到：

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: clr.dll, version: 4.0.30319.18052, time stamp: 0x5173c114
Exception code: 0xc00000fd
Fault offset: 0x00000000000055a7
Faulting process id: 0x1760
Faulting application start time: 0x01ceaf191e746321
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 60deebcd-1b0c-11e3-8eec-0026b9347fe0

接着：

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: clr.dll, version: 4.0.30319.18052, time stamp: 0x5173c114
Exception code: 0xc00000fd
Fault offset: 0x0000000000073dc4
Faulting process id: 0x2208
Faulting application start time: 0x01ceaf193c19060b
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 7e58b5f2-1b0c-11e3-8eec-0026b9347fe0

在该应用程序池进入停止状态之后。

处理了所有的 web 服务异常，我不知道去哪里找。系统可以稳定很长时间（几天）然后随机崩溃。

有什么方法可以解决此类问题的根本原因吗？

如果 IIS 内部发生此类崩溃，有什么方法可以监控服务并重新启动？

编辑：

得到更多细节。调试诊断工具能够读取内存转储。似乎是 WCF 调用导致它。看起来当我创建 WCF 服务实例并开始反序列化消息时，它会进入无限循环。关于如何更好地了解正在发生的事情的任何想法？在这个堆栈中，我看不到任何代码。我把中间部分剪掉了，它只是在序列化 DLL 中重复调用

Process Image   c:\Windows\System32\inetsrv\w3wp.exe 
System Up-Time   21 day(s) 11:44:13 
Process Up-Time   1 day(s) 19:43:33 

Thread 49 - System ID 1104

.NET Call Stack

Full Call Stack

Function     Arg 1     Arg 2     Arg 3     Arg 4   Source 
clr!GetRuntimeAssemblyHelper+1d     000007fe`f9cc471c     00000000`024dd240     00000000`ffba5cb0     00000000`00000000    
clr!RuntimeTypeHandle::GetAssembly+8c     00000000`ffba5cb0     00000003`00053d30     000007fe`f87bd608     00000000`ffd338a8    
mscorlib_ni+4ec579     00000004`0313e578     00000004`015e1ef0     00000001`fff8f2c0     000007fe`f1ee8f0e    
mscorlib_ni+4f0a39     00000004`0313e578     000007fe`f1ed0af3     00000001`ffbbb300     00000004`0313e5a0    
0x000007fe`9ad39c8a     00000000`ffd32520     000007fe`f9c6c4c5     00000000`0000008b     00000000`00000000    
0x000007fe`9b6e2976     00000000`ffd32520     00000000`000054e2     00000000`00000000     000007fe`f1ec2e00    
System_Runtime_Serialization_ni+b2ce1     00000004`015e1ef0     00000003`00023dc0     00000000`0e656548     00000000`0e6564e8    
System_Runtime_Serialization_ni+b29ed     00000002`030caad8     00000001`00b65cc8     00000001`ffc11890     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000000`00000066     00000000`00000000     00000004`015e1f98     00000003`00023dc0    
0x000007fe`9bda513d     00000001`ffc25548     00000000`000054e1     00000000`00000000     000007fe`f1ec2e00    
System_Runtime_Serialization_ni+b2ce1     00000004`015e1ef0     00000003`00023dc0     00000004`0313e578     00000003`00023dc0    
System_Runtime_Serialization_ni+b29ed     00000002`030cbe08     000007fe`f203bb28     00000003`ffe868e8     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000003`ffe86520     000007fe`9ab63858     00000000`00000147     00000004`ffb5afb0    
0x000007fe`9bda4ef9     00000003`ffe86520     00000004`0313e5a0     00000000`00000000     00000003`00083558    
System_Runtime_Serialization_ni+b08c1     00000002`030cbcf0     000007fe`f213c7cd     00000002`030c7c20     00000002`030c7c40    
System_Runtime_Serialization_ni+b29ed     00000004`003e33f0     00000002`030c7c40     00000003`00083558     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000001`ffc1eaa0     00000004`015e2018     00000004`015e2264     00000004`015e2080    
0x000007fe`9bda4687     00000001`ffc1eaa0     00000000`000054dc     00000000`00000000     000007fe`f1ec2e00    
System_Runtime_Serialization_ni+b2ce1     00000004`015e1ef0     00000003`00023dc0     00000004`0313e578     00000003`00023dc0    
System_Runtime_Serialization_ni+b29ed     00000002`018a8af0     000007fe`f203bb28     00000003`ffe85768     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f 

.....

0x000007fe`9b8e780b     00000001`ffcb8690     00000000`00000016     00000000`00000000     000007fe`f1ec2e00    
System_Runtime_Serialization_ni+b2ce1     00000004`015e1ef0     00000003`00023dc0     00000003`028539a0     000007fe`9a97d658    
System_Runtime_Serialization_ni+b29ed     00000001`ffc22190     00000003`028539c0     00000003`000837c0     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000001`ffc1eaa0     000007fe`f213d7a0     00000003`00052e00     00000004`015e2018    
0x000007fe`9bda44cf     00000001`ffc1eaa0     000007fe`f1edf4fc     00000000`0e6cc8e0     00000000`00000000    
System_Runtime_Serialization_ni+b2ce1     00000001`ffc1eaf0     000007fe`f9cf5c2f     00000000`ffba5b60     00000000`00000000    
System_Runtime_Serialization_ni+20c45b     00000004`015e2018     00000004`015e2018     00000000`00000011     000007fe`f1ee5e19    
System_Runtime_Serialization_ni+20c766     00000004`015e1ef0     00000003`00023dc0     00000001`ffbbb3f0     00000000`00000007    
System_Runtime_Serialization_ni+30a177     00000004`0313e578     00000004`0313e5a0     00000001`fff8ac40     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000001`ffbbaec8     000007fe`9ab625a8     80000000`00000000     000007fe`f860d2a2    
0x000007fe`9b6e1823     00000001`fff8a948     000007fe`f2035929     00000000`0e6ccbd0     00000000`00000000    
System_Runtime_Serialization_ni+b08c1     00000001`fff8a990     00000000`00000000     00000001`ffbe0180     00000001`ffbbae00    
System_Runtime_Serialization_ni+20c45b     00000004`015e2018     000007fe`f821e740     00000000`00000010     000007fe`f1ee5e19    
System_Runtime_Serialization_ni+20c766     00000004`015e1ef0     00000003`00023dc0     00000001`ffdadf58     000007fe`f86157d7    
System_Runtime_Serialization_ni+30a177     00000001`fff7d2f0     000007fe`f1ee8efa     00000001`ffdb1d68     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000001`ffdafc80     00000004`0313e5e8     00000000`00000000     000007fe`f1edf848    
0x000007fe`9b6e0d8c     00000001`ffdafc80     000007fe`f1edf4fc     00000000`0e6cced0     00000000`00000000    
System_Runtime_Serialization_ni+b2ce1     00000001`ffdafcd0     000007fe`f9cf5c2f     00000003`ffc3f3a8     00000004`0313e500    
System_Runtime_Serialization_ni+20c45b     00000004`015e2018     000007fe`f821e740     00000000`00000005     000007fe`f1ee5e19    
System_Runtime_Serialization_ni+20c766     00000004`015e1ef0     00000003`00023dc0     02261db5`f87be0c0     000007fe`f9c6797c    
System_Runtime_Serialization_ni+30a177     000007fe`9ac3c088     000007fe`fa074137     00000001`ffdac0f8     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000004`0313cee0     00000001`ffdac0f8     00000004`0313e578     00000000`0e6cd220    
0x000007fe`9a85a630     00000001`ffdab308     000007fe`f2035900     00000000`0e6cd1c0     000007fe`f1edf484    
System_Runtime_Serialization_ni+b2ce1     00000001`ffdab358     000007fe`f1ee7149     00000004`015e1f20     00000000`00000004    
System_Runtime_Serialization_ni+308197     00000004`015e1ef0     00000003`00023dc0     00000004`0313e578     00000003`00023dc0    
System_Runtime_Serialization_ni+b29ed     00000004`0313cf38     000007fe`f203bb28     00000001`ffda1c40     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000004`015e1f98     000007fe`9b01bda0     00000001`ffbbb3f0     00000000`00000003    
0x000007fe`9a85a430     00000001`ffda1938     00000000`00000003     00000000`00000000     000007fe`f1ec2e00    
System_Runtime_Serialization_ni+b08c1     00000004`015e1ef0     00000003`00023dc0     00000004`015e1ef0     000007fe`f20879f9    
System_Runtime_Serialization_ni+b29ed     000007fe`9ac3bf38     000007fe`fa074137     00000002`fffe6060     000007fe`f8e31860    
System_Runtime_Serialization_ni+20c91f     00000003`ffc41ef8     00000004`0313cef8     00000004`0313e5a0     00000004`015e1ef0    
0x000007fe`9a8597e7     00000002`fffe4a28     000007fe`f2137f69     000007fe`9a811e88     000007fe`f1ed0a23    
System_Runtime_Serialization_ni+b2ce1     000007fe`9a811e88     000007fe`f1ed082d     00000001`ffd15c14     00000003`00023c74    
System_Runtime_Serialization_ni+a0c3c     00000000`00000000     00000003`00023cb0     00000004`015e1ef0     00000000`0e6cd760    
System_Runtime_Serialization_ni+1fbd92     00000001`ffd8f520     000007fe`f76e5277     00000004`0313cef8     000007fe`f0e81779    
System_Runtime_Serialization_ni+1fc626     00000003`027f98f8     00000000`00000000     00000000`00000000     00000000`00000000    
System_Runtime_Serialization_ni+a0715     00000001`ffd8f508     00000004`015e2018     00000004`0313cef8     00000004`015e2080    
System_Runtime_Serialization_ni+a03f5     00000000`00000066     000007fe`f1ee66fc     00000000`00000065     00000000`00000000    
System_ServiceModel_ni+5eaf06     00000004`015e1ef0     00000002`fffe2158     00000002`fffe4908     000007fe`f1ec1577    
System_ServiceModel_ni+5eade9     00000001`00000064     00000003`ffb31420     00000001`ffd06138     00000001`ffd06138    
System_ServiceModel_ni+5eac97     00000000`00000000     00000004`0000000e     00000004`015e1ef0     00000004`015e1ef0    
System_ServiceModel_ni+5eab05     00000004`015e1e88     00000003`0000bd68     00000000`00000000     00000000`00000000    
System_ServiceModel_ni+650ed8     00000000`00000000     00000000`00000000     00000001`ffbcf838     00000000`00000000    
System_ServiceModel_ni+650b93     00000000`0e6cdd20     00000004`015e1de8     00000001`ffec0ef0     00000004`0313d388    
System_ServiceModel_ni+635c07     00000000`0e6cde30     00000001`ffbf92b0     00000004`0313d308     000007fe`7fffffff    
0x000007fe`9a6a601c     00000001`ffbcb2f0     000007fe`ef84b555     00000003`04f95618     00000004`0313d488    
System_ServiceModel_ni+5f316c     00000000`0e6ce060     00000000`00000001     00000000`0e6ce060     000007fe`f4ff80f3    
System_ServiceModel_ni+5f2c8e     00000000`00000000     00000000`00000000     00000000`00000000     00000003`04f956c0    
System_ServiceModel_ni+62380a     00000000`0e6ce000     00000003`054831d0     01ceb097`e03b5200     00000000`00000000    
System_ServiceModel_ni+6232d1     00000000`00000000     00000004`0313d308     00000000`00000000     00000000`00000000    
System_ServiceModel_ni+618d4b     00000003`04f95618     00000004`0313d308     00000000`23c34600     00000000`00000000    
System_ServiceModel_ni+618bf3     00000003`0000c190     00000000`0e6ce820     00000003`0000c190     00000001`ffbd7038    
System_ServiceModel_ni+6189ad     00000003`0000c190     00000000`0e6ce820     00000000`00000000     000007fe`f9d5d900    
System_ServiceModel_ni+617a2a     00000003`0000c190     00000000`0e6ce820     000007fe`9a7857b8     00000000`0e6ce820    
System_ServiceModel_ni+6124c6     00000003`0000c190     00000000`0e6ce820     00006a2a`00000000     00000000`0f48e8a0    
System_ServiceModel_ni+611b2b     00000003`000299c8     000007fe`f5f1d74b     00000001`ffbf9d28     00000000`00000000    
System_ServiceModel_ni+61053e     00000000`0e6ce820     000007fe`f1ed0d00     00000000`00000000     000007fe`f8621f10    
System_ServiceModel_ni+60fa57     00000000`ffb46740     00000003`04f95618     00000000`00000000     00000000`00000000    
System_ServiceModel_ni+60de03     00000003`ffc0fb60     00000001`ffbf92b0     80000000`00000000     00000003`000296d0    
System_ServiceModel_ni+60d92b     00000000`00000000     000007fe`f9e33824     00000000`0122f070     00000000`00000000    
System_ServiceModel_Internals_ni+4c352     00000000`00000000     00000000`00000000     00000003`054834a0     00000000`0e6cec18    
System_ServiceModel_Internals_ni+47d2a     00000004`04299c58     000007fe`f76ef200     00000000`00000000     00000000`0e6ceca0    
System_ServiceModel_Internals_ni+4c5f1     00000000`0e6cec20     00000003`04f95890     00000003`04f96060     000007fe`f56a4c65    
System_ServiceModel_Internals_ni+4c1cc     00000000`ffb465a0     00000000`0e6cedf0     00000003`05450001     000007fe`f5efdda6    
System_ServiceModel_Internals_ni+4bd9e     00000001`ffbf9620     00000000`00000000     00000000`00000000     00000000`00000000    
System_ServiceModel_ni+622eb2     00000003`04f95890     00000003`05451a10     00000003`04f957b0     00000003`04f95890    
System_ServiceModel_ni+62281f     00000001`ffbf9620     00000003`04f95790     00000003`04f957b0     00000003`04f95780    
System_ServiceModel_ni+62268a     00000003`04f95890     000007fe`f4f9b001     00000000`00000000     00000000`00000000    
System_ServiceModel_Internals_ni+e93f8     00000003`04f95618     000007fe`f56d0e96     00000003`04f95550     000007fe`f9cc5699    
System_ServiceModel_ni+621582     00000001`ffbf9620     00000003`04f95618     00000000`00000001     00000003`04f956d8    
System_ServiceModel_ni+620c38     00000003`04f956d8     00000003`ffc0f940     00000001`01d742a8     000007fe`ef858cb0    
System_ServiceModel_ni+620a2a     00000000`00000000     00000000`00000000     00000003`ffc0fb60     00000003`04f949b8    
System_ServiceModel_Activation_ni+71d48     00000002`00040004     00000001`02b1f848     00000003`04f953a8     000007fe`9a7aad00    
System_ServiceModel_Activation_ni+6ac44     00000001`01d742a8     00000001`ffb3a450     00000000`0e6cf608     00000000`00000000    
System_ServiceModel_Activation_ni+6ae35     00000001`0173afd8     000007fe`f8617fd7     00000000`0e6cf280     00000000`00000000    
System_ServiceModel_Activation_ni+6aec3     00000003`ffc0daf0     00000000`00000000     00000003`04f94528     000007fe`ef84c79c    
System_ServiceModel_Activation_ni+2c7a2     00000001`ffb3a498     00000001`ffb384a8     00000000`d2d6d2d7     000007fe`f4f9e809    
System_ServiceModel_Activation_ni+6afe3     00000000`00000000     00000000`00000000     00000000`00000000     00000000`00000000    
System_ServiceModel_Internals_ni+4e651     00000000`0e6cf4c0     000007fe`fa0bb84b     00000000`0f48e8a0     00000000`00000002    
System_ServiceModel_Internals_ni+4e5d0     00000000`0e6cf550     00000000`0e6cf470     00000000`0e6cf5e0     00000001`ffb384c8    
mscorlib_ni+534fa6     00000000`00000000     00000000`00000000     00000001`ffb3a498     00000000`11dc0470    
clr!CallDescrWorkerInternal+83     00000000`10b0ec30     00000000`11f43e20     00000000`001d0000     00000000`00000468    
clr!CallDescrWorkerWithHandler+4a     00000000`00000000     00000000`0e6cf5e0     00000000`0e6cf608     000007fe`f854af18    
clr!DispatchCallSimple+85     00000000`0e6cfca0     00000001`ffb3a498     00000000`00000000     00000000`00000000    
clr!BindIoCompletionCallBack_Worker+b8     00000001`ffb3a450     00000000`0e6cfc10     00000000`0e6cfcc0     00000000`0e6cfc10    
clr!DebuggerU2MCatchHandlerFrame::DebuggerU2MCatchHandlerFrame+a0     00000000`0e6cfc10     00000000`00000000     00000000`120a12f8     000007fe`ecad6585    
clr!ManagedPerAppDomainTPCount::DispatchWorkItem+38e     00000000`ffffffff     000007fe`f9c63108     00000000`0f48e8a0     000007fe`f9c63108    
clr!ManagedPerAppDomainTPCount::DispatchWorkItem+2bd     00000000`0e6cf9b8     00000000`0f48e8a0     00000000`0e6cfc10     00000000`0f48e8a0    
clr!ReturnToPreviousAppDomainHolder::Init+39     00000000`00000000     00000000`023e0340     00000000`0e6cfcc0     00000000`0e6cfcc0    
clr!Thread::DoADCallBack+234     00000000`00000002     00000000`00000002     00000000`0e6cfc10     00000000`0e6cfc10    
clr!DebuggerU2MCatchHandlerFrame::DebuggerU2MCatchHandlerFrame+d6     00000000`0e6cfc10     00000000`00000000     00000000`00000030     00000000`120a0000    
clr!ManagedPerAppDomainTPCount::DispatchWorkItem+38e     00000000`80004005     00000000`0e6cfc40     000007fe`fa324dd8     00000000`0f48e8a0    
clr!ManagedPerAppDomainTPCount::DispatchWorkItem+2bd     80000000`00000000     00000000`0f48e8a0     00000000`0f48e8a0     00000000`00000000    
clr!ManagedPerAppDomainTPCount::DispatchWorkItem+23b     00000000`00000002     00000000`0e6cfcf0     00000000`00008000     000007fe`fd5216ad    
clr!BindIoCompletionCallbackStubEx+cd     00000001`00040004     00000000`00000001     00000000`0f48e8a0     00000001`00000001    
clr!BindIoCompletionCallbackStub+9     00000002`00040004     00000002`00040004     00000000`00000001     00000000`00000000    
clr!ThreadpoolMgr::CompletionPortThreadStart+361     00000000`00000000     00000000`00000000     00000000`00000001     00000000`00000000    
clr!Thread::intermediateThreadProc+7d     00000000`00000000     00000000`00000000     00000000`00000000     00000000`00000000    
kernel32!BaseThreadInitThunk+d     00000000`00000000     00000000`00000000     00000000`00000000     00000000`00000000    
ntdll!RtlUserThreadStart+1d     00000000`00000000     00000000`00000000     00000000`00000000     00000000`00000000    

Exception Information
CLR!GETRUNTIMEASSEMBLYHELPER+1DIn WER45CE.tmp.mdmp the assembly instruction at clr!GetRuntimeAssemblyHelper+1d in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll from Microsoft Corporation has caused a stack overflow exception (0xC00000FD) when trying to write to memory location 0x0e655ff8 on thread 49

Answer 1

事件日志中的详细信息确实指出了序列化问题，但是我不记得导致 w3wp 崩溃的序列化问题。首先，打开 WCF 跟踪以获取 WCF 日志并查看确切的序列化问题（如果是这种情况）。 http://msdn.microsoft.com/en-us/library/ms733025.aspx

其次，关于故障转移，如果应用程序池崩溃，它将自动重新启动。但是，如果在重新启动后池再次崩溃，在给定的时间段内（默认为 5 分钟内崩溃 5 次），应用程序池将停止以防止更多崩溃 - 这应该在事件日志中显示。

至于序列化问题 - 如果您无法从跟踪中找出问题，您可以在这里发布 xml 内容吗？或将其上传到某处并提供链接？如果您使用的是 HTTP 绑定，您可以简单地打开 Fiddler 并复制请求内容。如果您使用 TCP 或其他绑定，则可以使用 WCF 消息日志记录消息（不要忘记将 LogEntireMessage 设置为 true）。 http://msdn.microsoft.com/en-us/library/ms730064.aspx

Answer 2

您是否尝试过在配置文件中设置诊断？

<system.diagnostics>
    <trace autoflush="true" />
    <sources>
      <source name="System.ServiceModel"
              switchValue ="Critical, Error"
              propagateActivity="true">
        <listeners>
          <add name="sdt"
              type="System.Diagnostics.XmlWriterTraceListener"
              initializeData= "c:\logs\servicelog.e2e" />
        </listeners>
      </source>
    </sources>
  </system.diagnostics>

您需要向服务添加权限以写入 C:\logs 或您选择的任何文件夹。这多次帮助我获取有关哪个方法发生异常的低级堆栈信息。

Answer 3

您在服务器上有 64 位版本的框架，应用程序是在 32 位系统上开发然后部署的吗？

如果是，请在 64 位系统上重新构建应用程序，然后重新部署。

此外，您在应用程序中引用的任何第 3 方 dll，您可能需要获取他们的 64 位等效项

Answer 4

有一个修复程序可能修复堆栈溢出：

http://support.microsoft.com/kb/2640103

Is there any way to monitor service and restart in case of such crash inside IIS?

您是否尝试过在 IIS 中关闭快速故障保护？如果在指定的时间间隔内发生并发故障，快速故障保护将停止您的 w3p 服务。见http://www.iisunderground.com/disable-rapid-fail-protection/

原因：

A worker process does not start within the startup time limit.

A worker process does not shut down within the shutdown time limit.

A worker process shuts itself down because of a fatal error and sends the WWW service an error code.

A worker process fails to respond to a ping message.

The WWW service detects that a worker process is sending non-standard communications (the worker process may have been taken over).

Answer 5

我看到了类似的问题。在我的例子中，我在 web.config 中添加了 aspNetCompatibilityEnabled=”true”，并且我在 asp.net 2.0 兼容模式下运行。此知识库文章更详细地描述了该问题

https://support.microsoft.com/en-us/kb/977104