When running a native-image of a Micronaut application retrieving Prometheus metrics fails because it cannot get jvm metrics.
Of course native-image pre-compiles the application and includes only what's actually being used, but what kind of JVM is used when running a native-image? I assume it's also a greatly reduced version of the standard hotspot vm.
So is it even possible to get jvm metrics when running a native image or should other means be used to retrieve cpu and memory stats of the running application?
In this case I used Micronaut but I recon it is common for getting metrics from any native-image application with prometheus metrics.
To reproduce:
- install Micronaut cli (e.g. via Homebrew on Mac)
- install GraalVM 19.2.0 (e.g. via Sdkman on Mac)
- install GraalVM native-image tool: gu install native-image
- create application: mn create-app --features=management,micrometer-prometheus,graal-native-image --build maven metrics-graalvm-image
- update src/main/resources/application.yml to allow anonymous access to prometheus endpoint:
endpoints:
all:
enabled: true
sensitive: false
- run application: man exec:exec
- check prometheus metrics can be retrieved: open http://localhost:8080/prometheus
- build application: mvn clean verify
- create native image: native-image --no-server -jar target/metrics-graalvm-image-0.1.jar
- run native image: ./metrics-graalvm-image
- get prometheus metrics: open http://localhost:8080/prometheus
No with the native image, this exception is thrown:
11:06:51.186 [pool-2-thread-4] ERROR i.m.h.s.netty.RoutingInBoundHandler - Unexpected error occurred: ThreadMXBean methods
com.oracle.svm.core.jdk.UnsupportedFeatureError: ThreadMXBean methods
at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:102)
at com.oracle.svm.core.jdk.SubstrateThreadMXBean.getAllThreadIds(ManagementSupport.java:448)
at io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics.getThreadStateCount(JvmThreadMetrics.java:85)
at io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics.lambda$bindTo$0(JvmThreadMetrics.java:75)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:40)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$3(PrometheusMeterRegistry.java:245)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:69)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:95)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:80)
at io.micronaut.configuration.metrics.micrometer.prometheus.management.PrometheusEndpoint.scrape(PrometheusEndpoint.java:52)
at io.micronaut.configuration.metrics.micrometer.prometheus.management.$PrometheusEndpointDefinition$$exec1.invokeInternal(Unknown Source)
at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:144)
at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2792)
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:235)
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$19(RoutingInBoundHandler.java:1408)
at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14868)
at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52)
at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52)
at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:122)
at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)