1

TaskCanceledException根据我们的日志，我们偶尔会收到一个调用，该调用在我们为请求配置的超时时间内很好地完成。第一个日志条目来自服务器。这是该方法在返回 JsonResult（MVC 4 控制器）之前注销的最后一件事。

{
    "TimeGenerated": "2021-03-19T12:08:48.882Z",
    "CorrelationId": "b1568096-fdbd-46a7-8b69-58d0b33f458c",
    "date_s": "2021-03-19",
    "time_s": "07:08:37.9582",
    "callsite_s": "...ImportDatasets",
    "stacktrace_s": "",
    "Level": "INFO",
    "class_s": "...ReportConfigController",
    "Message": "Some uninteresting message",
    "exception_s": ""
}

在这种情况下，请求大约需要 5 分钟才能完成。然后 30 分钟后，我们的调用者从以下位置抛出任务取消异常HttpClient.SendAsync：

{
    "TimeGenerated": "2021-03-19T12:48:27.783Z",
    "CorrelationId": "b1568096-fdbd-46a7-8b69-58d0b33f458c",
    "date_s": "2021-03-19",
    "time_s": "12:48:17.5354",
    "callsite_s": "...AuthorizedApiAccessor+<CallApi>d__29.MoveNext",
    "stacktrace_s": "TaskCanceledException    
        at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   
        at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n   
        at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n   
        at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   
        at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   
        at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n   
        at ...AuthorizedApiAccessor.CallApi(String url, Object content, HttpMethod httpMethod, AuthenticationType authType, Boolean isCompressed)\r\nIOException    
        at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\r\n   
        at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)\r\n   
        at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)\r\n   
        at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)\r\n   
        at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\nSocketException",
    "Level": "ERROR",
    "class_s": "...AuthorizedApiAccessor",
    "Message": "Nothing good",
    "exception_s": "The operation was canceled."
}

鉴于在发出请求的过程中我们阻止了异步调用（.Result——遇到不支持异步的棕地缓存实现），我的第一个猜测是我们遇到了Stephen Cleary所描述的死锁。但是调用者是 dotnetcore 3.1 应用程序，所以这种死锁是不可能的。

我认为我们的使用HttpClient非常标准。这是最终进行调用的方法：

private async Task<string> CallApi(string url, object content, HttpMethod httpMethod, AuthenticationType authType, bool isCompressed)
{
    try
    {
        var request = new HttpRequestMessage()
        {
            RequestUri = new Uri(url),
            Method = httpMethod,
            Content = GetContent(content, isCompressed)
        };

        AddRequestHeaders(request);

        var httpClient = _httpClientFactory.CreateClient(HTTPCLIENT_NAME);
        httpClient.Timeout = Timeout;

        AddAuthenticationHeaders(httpClient, authType);

        var resp = await httpClient.SendAsync(request);
        var responseString = await (resp.Content?.ReadAsStringAsync() ?? Task.FromResult<string>(string.Empty));

        if (!resp.IsSuccessStatusCode)
        {
            var message = $"{url}: {httpMethod}: {authType}: {isCompressed}: {responseString}";
            if (resp.StatusCode == HttpStatusCode.Forbidden || resp.StatusCode == HttpStatusCode.Unauthorized)
            {
                throw new CustomException(message, ErrorType.AccessViolation);
            }

            if (resp.StatusCode == HttpStatusCode.NotFound)
            {
                throw new CustomException(message, ErrorType.NotFound);
            }

            throw new CustomException(message);
        }

        return responseString;
    }
    catch (CustomException) { throw; }
    catch (Exception ex)
    {
        var message = "{Url}: {HttpVerb}: {AuthType}: {IsCompressed}: {Message}";
        _logger.ErrorFormat(message, ex, url, httpMethod, authType, isCompressed, ex.Message);
        throw;
    }
}

我们对这种行为的理论感到茫然。在几百个成功的请求中，我们已经看到每月 3-5 次任务取消，所以它是间歇性的，但绝非罕见。

我们还应该在哪里寻找像死锁一样的行为的根源？

更新

可能需要注意我们正在使用标准HttpClientHandler。最近添加了重试策略，但是我们不会在长时间运行的 POST 上重试，就是上面的场景。

builder.Services.AddHttpClient(AuthorizedApiAccessor.HTTPCLIENT_NAME)
    .ConfigurePrimaryHttpMessageHandler(_ => new HttpClientHandler()
    {
        AutomaticDecompression = System.Net.DecompressionMethods.Deflate | System.Net.DecompressionMethods.GZip
    })
    .AddRetryPolicies(retryOptions);
4

0 回答 0