6

我正在评估使用Microsoft Health Checks来改进我们内部负载均衡器的路由。到目前为止,我对这个特性和它周围的社区所提供的功能非常满意。但是,我还没有找到一件事,想问一下是否可以开箱即用:

健康检查似乎会在收到请求后立即检索自己的状态。但是因为我们的服务在特定时刻可能很难处理大量请求,所以对 SQL Server 等第三方组件的查询可能需要时间来响应。因此,我们希望定期(例如每隔几秒)预先评估健康检查,并在调用健康检查 api 时返回该状态。

原因是,我们希望我们的负载均衡器尽快获得健康状态。对于我们的用例来说,使用预先评估的结果似乎已经足够了。

现在的问题是:是否可以在 ASP.NET Core 健康检查中添加一种“轮询”或“自动更新”机制?或者这是否意味着我必须从定期预先评估结果的后台服务执行我自己的健康检查返回值?

请注意,我想对每个请求使用预先评估的结果,而不是 HTTP 缓存,其中为下一个请求缓存实时结果。

4

3 回答 3

6

Panagiotis 的回答非常出色,给我带来了一个优雅的解决方案,我很想留给下一个遇到这个问题的开发人员......

为了在不实现后台服务或任何计时器的情况下实现定期更新,我注册了一个IHealthCheckPublisher. 这样,ASP.NET Core 将定期自动运行注册的健康检查并将其结果发布到相应的实现。

在我的测试中,健康报告默认每 30 秒发布一次。

// add a publisher to cache the latest health report
services.AddSingleton<IHealthCheckPublisher, HealthReportCachePublisher>();

我注册了我的实现HealthReportCachePublisher,它只是获取已发布的健康报告并将其保存在静态属性中。

我不太喜欢静态属性,但对我来说,它似乎足以满足这个用例。

/// <summary>
/// This publisher takes a health report and keeps it as "Latest".
/// Other health checks or endpoints can reuse the latest health report to provide
/// health check APIs without having the checks executed on each request.
/// </summary>
public class HealthReportCachePublisher : IHealthCheckPublisher
{
    /// <summary>
    /// The latest health report which got published
    /// </summary>
    public static HealthReport Latest { get; set; }

    /// <summary>
    /// Publishes a provided report
    /// </summary>
    /// <param name="report">The result of executing a set of health checks</param>
    /// <param name="cancellationToken">A task which will complete when publishing is complete</param>
    /// <returns></returns>
    public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
    {
        Latest = report;
        return Task.CompletedTask;
    }
}

现在真正的魔法发生在这里

正如在每个运行状况检查示例中所见,我将运行状况检查映射到路由/health并使用UIResponseWriter.WriteHealthCheckUIResponse返回漂亮的 json 响应。

但我绘制了另一条路线/health/latest。在那里,谓词_ => false阻止执行任何健康检查。但是,我没有返回零健康检查的空结果,而是通过访问 static 来返回先前发布的健康报告HealthReportCachePublisher.Latest

app.UseEndpoints(endpoints =>
{
    // live health data: executes health checks for each request
    endpoints.MapHealthChecks("/health", new Microsoft.AspNetCore.Diagnostics.HealthChecks.HealthCheckOptions()
    {
        ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
    });

    // latest health report: won't execute health checks but return the cached data from the HealthReportCachePublisher
    endpoints.MapHealthChecks("/health/latest", new Microsoft.AspNetCore.Diagnostics.HealthChecks.HealthCheckOptions()
    {
        Predicate = _ => false, // do not execute any health checks, we just want to return the latest health report
        ResponseWriter = (context, _) => UIResponseWriter.WriteHealthCheckUIResponse(context, HealthReportCachePublisher.Latest)
    });
});

这样,调用/health通过对每个请求执行所有健康检查来返回实时健康报告。如果有很多事情要检查或要发出网络请求,这可能需要一段时间。

呼叫/health/latest将始终返回最新的预先评估的健康报告。这是非常快的,如果您有一个负载均衡器等待运行状况报告来相应地路由传入的请求,这可能会有很大帮助。


一点补充:上面的方案使用路由映射取消健康检查的执行,返回最新的健康报告。正如建议的那样,我尝试首先构建进一步的健康检查,它应该返回最新的缓存健康报告,但这有两个缺点:

  • 返回缓存报告本身的新运行状况检查也会出现在结果中(或者必须按名称或标签进行过滤)。
  • 没有简单的方法将缓存的健康报告映射到HealthCheckResult. 如果您复制属性和状态代码,这可能会起作用。但是生成的 json 基本上是一个包含内部健康报告的健康报告。那不是你想要的。
于 2020-10-09T09:13:58.570 回答
4

Short Version

This is already available and can already integrate with common monitoring systems. You may be able to tie Health Check directly into your monitoring infrastructure.

The details

The Health Check middleware covers this by periodically publishing metrics to a target, through any registered classes that implement the IHealthCheckPublisher.PublishAsync interface method.

services.AddSingleton<IHealthCheckPublisher, ReadinessPublisher>();

Publishing can be configured through HealthCheckPublisherOptions. The default period is 30 seconds. The options can be used to add delays, filter the checks to run etc:

services.Configure<HealthCheckPublisherOptions>(options =>
{
    options.Delay = TimeSpan.FromSeconds(2);
    options.Predicate = (check) => check.Tags.Contains("ready");
});

One option would be to cache the results (the HealthReport instance) with a publisher and serve them from another HealthCheck endpoint.

Perhaps a better option would be to push them to a monitoring system like Application Insights or a time-series database like Prometheus. The AspNetCore.Diagnostics.HealthCheck package provides a ton of ready-made checks and publishers for App Insights, Seq, Datadog and Prometheus.

Prometheus uses polling itself. It calls all its registered sources periodically to retrieve metrics. While that works for services, it won't work for eg CLI applications. For that reason, applications can push results to a Prometheus Gateway that caches the metrics until Prometheus itself requests them.

services.AddHealthChecks()
        .AddSqlServer(connectionString: Configuration["Data:ConnectionStrings:Sample"])
        .AddCheck<RandomHealthCheck>("random")
        .AddPrometheusGatewayPublisher();

Apart from pushing to Prometheus Gateway, the Prometheus publisher also offers an endpoint to retrieve live metrics directly, through the AspNetcore.HealthChecks.Publisher.Prometheus package. The same endpoint could be used by other applications to retrieve those metrics :

// default endpoint: /healthmetrics
app.UseHealthChecksPrometheusExporter();
于 2020-10-08T08:04:37.783 回答
0

另一种选择是使用Scrutor和装饰 HealthCheckService。如果您想对多个线程重新发布感到偏执,则必须在从内部 HealthCheckService 获取 HealthCheckReport 时添加锁定机制。一个体面的例子是here

using System.Reflection;
using HealthCheckCache;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

// used by the Decorator CachingHealthCheckService
builder.Services.AddMemoryCache();
builder.Services.AddHttpContextAccessor();

// register all IHealthCheck types - basically builder.Services.AddTransient<AlwaysHealthy>(), but across all types in this assembly.
var healthServices = builder.Services.Scan(scan =>
    scan.FromCallingAssembly()
        .AddClasses(filter => filter.AssignableTo<IHealthCheck>())
        .AsSelf()
        .WithTransientLifetime()
);

// Register HealthCheckService, so it can be decorated.
var healthCheckBuilder = builder.Services.AddHealthChecks();
// Decorate the implementation with a cache
builder.Services.Decorate<HealthCheckService>((inner, provider) =>
    new CachingHealthCheckService(inner,
        provider.GetRequiredService<IHttpContextAccessor>(),
        provider.GetRequiredService<IMemoryCache>()
    )
);

// Register all the IHealthCheck instances in the container
// this has to be a for loop, b/c healthCheckBuilder.Add will modify the builder.Services - ServiceCollection
for (int i = 0; i < healthServices.Count; i++)
{
    ServiceDescriptor serviceDescriptor = healthServices[i];
    var isHealthCheck = serviceDescriptor.ServiceType.IsAssignableTo(typeof(IHealthCheck)) && serviceDescriptor.ServiceType == serviceDescriptor.ImplementationType;
    if (isHealthCheck)
    {
        healthCheckBuilder.Add(new HealthCheckRegistration(
            serviceDescriptor.ImplementationType.Name,
            s => (IHealthCheck)ActivatorUtilities.GetServiceOrCreateInstance(s, serviceDescriptor.ImplementationType),
            failureStatus: null,
            tags: null)
        );
    }

}

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.MapHealthChecks("/health", new HealthCheckOptions()
{
    AllowCachingResponses = true, // allow caching at Http level
});

app.Run();

public class CachingHealthCheckService : HealthCheckService
{
    private readonly HealthCheckService _innerHealthCheckService;
    private readonly IHttpContextAccessor _contextAccessor;
    private readonly IMemoryCache _cache;
    private const string CacheKey = "CachingHealthCheckService:HealthCheckReport";

    public CachingHealthCheckService(HealthCheckService innerHealthCheckService, IHttpContextAccessor contextAccessor, IMemoryCache cache)
    {
        _innerHealthCheckService = innerHealthCheckService;
        _contextAccessor = contextAccessor;
        _cache = cache;
    }

    public override async Task<HealthReport> CheckHealthAsync(Func<HealthCheckRegistration, bool>? predicate, CancellationToken cancellationToken = new CancellationToken())
    {
        HttpContext context = _contextAccessor.HttpContext;


        var forced = !string.IsNullOrEmpty(context.Request.Query["force"]);
        context.Response.Headers.Add("X-Health-Forced", forced.ToString());
        var cached = _cache.Get<HealthReport>(CacheKey);
        if (!forced && cached != null)
        {
            context.Response.Headers.Add("X-Health-Cached", "True");
            return cached;
        }
        var healthReport = await _innerHealthCheckService.CheckHealthAsync(predicate, cancellationToken);
        if (!forced)
        {
            _cache.Set(CacheKey, healthReport, TimeSpan.FromSeconds(30));
        }
        context.Response.Headers.Add("X-Health-Cached", "False");
        return healthReport;
    }
}
于 2022-02-10T22:36:29.457 回答