我们的应用程序使用 SLF4J 的 MDC 和 Logback 的 JSON 编码器将日志行编写为 JSON。然后这些行由日志传送管道处理并作为文档写入 ElasticSearch。他们的logback.xml文件如下所示：
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
</configuration>
并且MDC是这样使用的：
MDC.put("stringVal", "val");
MDC.put("decimalVal", "1.23");
MDC.put("longVal", "123498559348792879267942876");
logger.info("foo");
问题是 MDC 的接口是void put(String key, String val)，因此所有值都必须是字符串或可自动装箱为字符串。然后，这会产生以下日志：
{"@timestamp":"2021-08-23T16:32:04.231+01:00","@version":1,"message":"foo","logger_name":"MDCTest","thread_name":"main","level":"INFO","level_value":20000,"stringVal":"val","decimalVal":"1.23","longVal":"123498559348792879267942876"}
键入的字符串然后在 Elasticsearch 中使用自动类型映射作为字符串类型被拾取decimalVal，longVal然后我们无法对它们执行数字操作。
在这种情况下，这些数值主要来自滥用日志来发送指标，但这种混淆正在其他地方处理。
我不想强迫开发人员在添加更多日志时必须更新 Elasticsearch 索引模板或为日志传送管道编写配置，因此我一直在寻找一种自动执行此操作的方法。我已经生成了一个实现，它通过替换 Logback 的内部来工作，Encoder并且Formatter为了切换MdcJsonProvider负责将 MDC 序列化为 JSON 的类。这感觉非常脆弱且性能不佳。
有没有更优雅的方法来做到这一点，或者有不同的方法来获得相同的效果？我已经看过了，ch.qos.logback.contrib.jackson.JacksonJsonFormatter但我仍然需要在我试图避免的 logback 文件中列出数字 MDC 属性。
import java.io.IOException;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import com.fasterxml.jackson.core.JsonGenerator;
import com.google.common.collect.Maps;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.spi.ContextAware;
import net.logstash.logback.LogstashFormatter;
import net.logstash.logback.composite.CompositeJsonFormatter;
import net.logstash.logback.composite.JsonProvider;
import net.logstash.logback.composite.JsonWritingUtils;
import net.logstash.logback.composite.loggingevent.MdcJsonProvider;
import net.logstash.logback.encoder.LogstashEncoder;
/**
* This class is a hack.
*
* It exists because Logback's MDC interface is a map from String to String, and this cannot
* be changed. Instead, we must modify the logger itself to check if any of the String values
* appear numeric and then log them as such.
*
* This is necessary since we ship the documents directly to ElasticSearch. As such, be warned
* that you may have mapping conflict issues if the type for any given key fluctuates or if you
* occasionally write strings that are fully numeric.
*
* This class implements an Encoder, a Formatter, and a JsonProvider that work together. The
* encoder specifies the formatter, and the formatter works by swapping out the MdcJsonProvider
* implementation normally used to serialise the MDC values with another implementation with tries
* to log them as numbers.
*
* Using this class has a cost. It will result in more short-lived object creation, so it is not
* suitable for high frequency logging.
*/
public class JsonMdcTypePreservingEncoder extends LogstashEncoder {
protected CompositeJsonFormatter<ILoggingEvent> createFormatter() {
return new JsonMdcTypePreservingFormatter(this);
}
protected JsonMdcTypePreservingFormatter getFormatter() {
return (JsonMdcTypePreservingFormatter) super.getFormatter();
}
/**
* This class exists to remove the default MdcJsonProvider, responsible for logging the MDC
* section as JSON, and swap it out for the custom one.
*/
public static class JsonMdcTypePreservingFormatter extends LogstashFormatter {
public JsonMdcTypePreservingFormatter(ContextAware declaredOrigin) {
super(declaredOrigin);
Optional<JsonProvider<ILoggingEvent>> oldProvider = getProviders().getProviders()
.stream()
.filter(o -> o.getClass() == MdcJsonProvider.class)
.findFirst();
if (oldProvider.isPresent()) {
getProviders().removeProvider(oldProvider.get());
getProviders().addProvider(new TypePreservingMdcJsonProvider());
}
}
/**
* This class contains a duplicate of MdcJsonProvider.writeTo but with a small change.
* Instead of taking the MDC Map<String, String> and logging it, it produces a modified
* Map<String, Object> that potentially contains BigDecimal or BigInteger types alongside
* Strings and serialises those instead.
*
* The only new code in this method is the call to getStringObjectMap.
*/
public static class TypePreservingMdcJsonProvider extends MdcJsonProvider {
private Map<String, Object> convertedProperties = Maps.newHashMap();
@Override
public void writeTo(JsonGenerator generator, ILoggingEvent event) throws IOException {
Map<String, String> mdcProperties = event.getMDCPropertyMap();
if (mdcProperties != null && !mdcProperties.isEmpty()) {
if (getFieldName() != null) {
generator.writeObjectFieldStart(getFieldName());
}
if (!getIncludeMdcKeyNames().isEmpty()) {
mdcProperties = new HashMap(mdcProperties);
((Map) mdcProperties).keySet().retainAll(getIncludeMdcKeyNames());
}
if (!getExcludeMdcKeyNames().isEmpty()) {
mdcProperties = new HashMap(mdcProperties);
((Map) mdcProperties).keySet().removeAll(getExcludeMdcKeyNames());
}
Map<String, Object> convertedProperties = getStringObjectMap(mdcProperties);
JsonWritingUtils.writeMapEntries(generator, convertedProperties);
if (getFieldName() != null) {
generator.writeEndObject();
}
}
}
private Map<String, Object> getStringObjectMap(Map<String, String> mdcProperties) {
convertedProperties.clear();
for (String key : mdcProperties.keySet()) {
String value = mdcProperties.get(key);
// If the majority of MDC values are not numbers, this prevents unnecessary
// parsing steps but adds a character-wise inspection per value.
try {
BigInteger parsed = new BigInteger(value);
convertedProperties.put(key, parsed);
} catch (NumberFormatException e) {
try {
BigDecimal parsed = new BigDecimal(vlue);
convertedProperties.put(key, parsed);
} catch (NumberFormatException f) {
// No-op
}
}
if (!convertedProperties.containsKey(key)) {
convertedProperties.put(key, value);
}
}
return convertedProperties;
}
}
}
}
感谢您的任何建议！