Metrics

Module for collecting application metrics using Micrometer. It creates a PrometheusMeterRegistry, registers it in the dependency container as a MeterRegistry, and exposes the collected values in the Prometheus format through the system HTTP server. This lets you collect application, JVM, process, and built-in integration metrics in one place and scrape them with an external observability system.

Publishing metrics requires the system HTTP server, which exposes them in the Prometheus format.

For a step-by-step walkthrough before the reference details, see Observability.

Dependency¶

implementation "io.koraframework:micrometer-module"

@KoraApp
public interface Application extends MetricsModule { }

implementation("io.koraframework:micrometer-module")

@KoraApp
interface Application : MetricsModule

MetricsModule lives in the io.koraframework.micrometer.module package. It provides the registry, the application-wide common tags and the scrape endpoint contract — the metric values themselves are produced by the modules you connect (HTTP server, HTTP client, Database, Kafka and so on).

Configuration¶

Metrics are configured in three places: the application-wide metrics section of MetricsModule, the path and port of the scrape endpoint on the system server, and a telemetry.metrics block inside each module that reports metrics.

Application metrics¶

The metrics section is described by the MetricsConfig class from the io.koraframework.micrometer.module package (default values are specified):

metrics {
    enabled = true //(1)!
    tags { //(2)!
        "application" = "example-service"
        "environment" = "stage"
    }
}

metrics:
  enabled: true #(1)!
  tags: #(2)!
    application: "example-service"
    environment: "stage"

With metrics.enabled = false, MetricsModule provides a no-op MeterRegistry instead of PrometheusMeterRegistry: no module, JVM, kora.up or custom metric is recorded, PrometheusMeterRegistryInitializer components are not applied, and /metrics answers with an empty body. This switches metrics off without removing MetricsModule from the application, for example in a test or a local profile.

This MetricsConfig is a different class from the per-module TelemetryConfig.MetricsConfig described in Module metrics, and the defaults are opposite: metrics.enabled is true, while <module>.telemetry.metrics.enabled is false. A module reports metrics only when both switches are on.

Common tags provider¶

When a tag value is known only at startup (a host name, a region read from the environment, a build version), register a MetricsTagsProvider component from the io.koraframework.micrometer.module package:

@Component
public final class HostMetricsTagsProvider implements MetricsTagsProvider {

    @Override
    public Map<String, String> tags() {
        return Map.of("host", System.getenv().getOrDefault("HOSTNAME", "unknown"));
    }
}

@Component
class HostMetricsTagsProvider : MetricsTagsProvider {

    override fun tags(): Map<String, String> {
        return mapOf("host" to (System.getenv("HOSTNAME") ?: "unknown"))
    }
}

Any number of providers can be registered. MetricsModule collects all of them, merges their tags with metrics.tags, and installs one global MeterFilter.commonTags(...) on the registry through a PrometheusMeterRegistryInitializer. On a key conflict the value from metrics.tags wins over every provider. Tags are resolved once, when the registry is created, so a provider must not return per-request values. That initializer is a regular component, not a @DefaultComponent, so the common tags stay applied when the application declares its own PrometheusMeterRegistryInitializer: the common-tags initializer runs alongside them, see Personalization.

Scrape endpoint¶

Example of the system HTTP server path configuration described in the SystemHttpServerConfig class (default values are specified):

httpServer {
    system {
        port = 8085 //(1)!
        metricsPath = "/metrics" //(2)!
    }
}

httpServer:
  system:
    port: 8085 #(1)!
    metricsPath: "/metrics" #(2)!

Module metrics¶

Each metric-collecting module exposes a telemetry.metrics block described in TelemetryConfig.MetricsConfig, letting you turn metrics on, tune histogram buckets, and attach extra tags for that module only. The example below uses the HTTP server module as the host, but the same telemetry.metrics fields apply verbatim to HTTP client, Database, Kafka, gRPC server, gRPC client, Scheduling, Cache, Resilience, and every other integration that reports metrics:

httpServer {
    telemetry {
        metrics {
            enabled = true //(1)!
            slo = ["1ms", "10ms", "50ms", "100ms", "200ms", "500ms", "1s", "2s", "5s", "10s", "20s", "30s", "60s", "90s"] //(2)!
            tags { //(3)!
                "key1" = "value1"
                "key2" = "value2"
            }
        }
    }
}

httpServer:
  telemetry:
    metrics:
      enabled: true #(1)!
      slo: [ "1ms", "10ms", "50ms", "100ms", "200ms", "500ms", "1s", "2s", "5s", "10s", "20s", "30s", "60s", "90s" ] #(2)!
      tags: #(3)!
        key1: value1
        key2: value2

slo values are durations: a string carries its unit ("1ms", "250ms", "1s", PT1S), a bare number is read as milliseconds, so slo = [1, 10, 50] and slo = ["1ms", "10ms", "50ms"] are the same list.

Leaving enabled = false (the default) means the module's telemetry uses a Noop metrics factory: no Meter is created and nothing is registered in the registry. That is also the way to silence a noisy integration after you have enabled metrics globally.

Metrics are only produced when both conditions hold: MetricsModule is connected (so a MeterRegistry exists in the container) and the module's telemetry.metrics.enabled is true. If either is missing, the module falls back to a no-op telemetry implementation. The application-wide metrics.enabled = false overrides both: the module then records into a no-op MeterRegistry and nothing is exported.

Configuration paths¶

The telemetry.metrics block is nested under the module's own configuration section:

Some modules add their own keys next to enabled / slo / tags:

• Database — driverMetrics (default: true) registers the HikariCP pool metrics of the connection pool in the same registry.
• Kafka — driverMetrics (default: false) registers the native Kafka client metrics through Micrometer's KafkaClientMetrics binder, under the kafka.* name prefix.
• Camunda 7 BPMN — engineMetrics (default: false) enables the Camunda engine's own metrics.

Metric collection parameters are also described in the modules that collect metrics: HTTP server, HTTP client, gRPC server, gRPC client, Scheduling, Cache, and other integrations.

Usage¶

Kora follows the notation described in the Prometheus specification.

After the module is connected, PrometheusMeterRegistry is created, registered in Metrics.globalRegistry, and used by all components that collect metrics. When the application stops, this registry is removed from Metrics.globalRegistry and closed.

The registry is provided by the PrometheusMeterRegistryWrapper component, which is a Root component and implements Wrapped<MeterRegistry>, so user code injects the MeterRegistry contract:

@Component
public final class SomeService {

    private final MeterRegistry meterRegistry;

    public SomeService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }
}

@Component
class SomeService(
    private val meterRegistry: MeterRegistry
)

The registry automatically gets standard Micrometer binders: ClassLoaderMetrics, JvmMemoryMetrics, JvmGcMetrics, ProcessorMetrics, JvmThreadMetrics, FileDescriptorMetrics, UptimeMetrics. These are bound when the registry is created and do not depend on any telemetry.metrics.enabled flag; only the application-wide metrics.enabled = false removes them. Kora also registers the kora.up metric with value 1 and the version tag.

Kora additionally bridges the Micrometer registry to an OpenTelemetry MeterProvider (MicrometerMeterProvider from io.opentelemetry.contrib.metrics.micrometer), so libraries instrumented with the OpenTelemetry metrics API publish through the same registry. The bridge is a @DefaultComponent and accepts an optional CallbackRegistrar component if you need to control how asynchronous instruments are polled.

A runnable baseline that wires MetricsModule alongside HoconConfigModule, LogbackModule, JdbcDatabaseModule, UndertowPublicHttpServerModule, and the OpenTelemetry tracing exporter is available in the kora-java-telemetry example.

Prometheus export¶

Metrics are exposed in the Prometheus text format by the system HTTP server on httpServer.system.metricsPath (default /metrics) served at httpServer.system.port (default 8085):

curl http://localhost:8085/metrics

Point your Prometheus scrape target (or any compatible collector) at the same host, port, and path.

The endpoint always answers 200. What it returns depends on what is bound in the container:

• MetricsModule connected — the Prometheus text exposition of the registry snapshot.
• MetricsModule connected with metrics.enabled = false — an empty body, because the registry is a no-op one.
• MetricsModule not connected — the body # Metric Scraper disabled, because no MetricsScraper component exists.
• A custom MeterRegistry that is not a PrometheusMeterRegistry — an empty body, because that registry cannot be scraped in the Prometheus format. Provide your own MetricsScraper implementation in that case: it overrides the @DefaultComponent supplied by MetricsModule.

MetricsScraper is a single-method contract from the io.koraframework.telemetry.common package:

@Module
public interface MetricsScraperModule {

    default MetricsScraper metricsScraper(MeterRegistry registry) {
        return os -> os.write(renderMyFormat(registry).getBytes(StandardCharsets.UTF_8));
    }
}

@Module
interface MetricsScraperModule {

    fun metricsScraper(registry: MeterRegistry): MetricsScraper {
        return MetricsScraper { os -> os.write(renderMyFormat(registry).toByteArray()) }
    }
}

Custom metric¶

For a custom metric, it is better to create a separate component, inject MeterRegistry, and reuse created Meter instances. Do not create a new metric on every method call: if the tag set depends on the operation, use a key with limited cardinality and cache the metric in ConcurrentHashMap. The register(...) call is needed for initial metric registration in MeterRegistry; on the hot path, prefer using an already created Timer / Counter / Gauge and only call record(...) or increment(...). Kora uses the same approach for its internal metrics.

For example, a duration metric for an external operation:

@Component
public final class ExternalOperationMetrics {

    private record Key(String operation, String status) {}

    private final MeterRegistry meterRegistry;
    private final ConcurrentHashMap<Key, Timer> timers = new ConcurrentHashMap<>();

    public ExternalOperationMetrics(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    public void record(String operation, String status, long durationNanos) {
        var key = new Key(operation, status);
        var timer = this.timers.computeIfAbsent(key, k -> Timer.builder("external.operation.duration")
            .tag("operation", k.operation())
            .tag("status", k.status())
            .register(this.meterRegistry));

        timer.record(durationNanos, TimeUnit.NANOSECONDS);
    }
}

@Component
class ExternalOperationMetrics(
    private val meterRegistry: MeterRegistry
) {

    private data class Key(
        val operation: String,
        val status: String
    )

    private val timers = ConcurrentHashMap<Key, Timer>()

    fun record(operation: String, status: String, durationNanos: Long) {
        val key = Key(operation, status)
        val timer = timers.computeIfAbsent(key) {
            Timer.builder("external.operation.duration")
                .tag("operation", it.operation)
                .tag("status", it.status)
                .register(meterRegistry)
        }

        timer.record(durationNanos, TimeUnit.NANOSECONDS)
    }
}

Tag values must have a limited number of variants. Do not use user identifiers, request numbers, full error text, or other high-cardinality values as tags.

Personalization¶

To change PrometheusMeterRegistry configuration, add a PrometheusMeterRegistryInitializer to the container. The initializer receives the created registry before standard system metrics are registered, so it can add common tags, MeterFilter, renaming rules, or custom PrometheusMeterRegistry settings. All initializers found in the container are applied in sequence, each receiving the result of the previous one.

Important, PrometheusMeterRegistryInitializer is applied only once when the application is initialized.

Common tags for all metrics do not need an initializer: use metrics.tags or a MetricsTagsProvider. An initializer is for the rest of the registry settings, for example a MeterFilter that drops metrics you do not collect:

@Module
public interface MetricsConfigModule {

    default PrometheusMeterRegistryInitializer denyGcMetricsInit() {
        return registry -> {
            registry.config().meterFilter(MeterFilter.denyNameStartsWith("jvm.gc"));
            return registry;
        };
    }
}

@Module
interface MetricsConfigModule {

    fun denyGcMetricsInit(): PrometheusMeterRegistryInitializer {
        return PrometheusMeterRegistryInitializer {
            it.config().meterFilter(MeterFilter.denyNameStartsWith("jvm.gc"))
            it
        }
    }
}

Standard metrics also have their own settings, for example the slo histogram buckets for Timer metrics, configured per module under telemetry.metrics. When slo is not overridden, TelemetryConfig.MetricsConfig#DEFAULT_SLO is used — 14 buckets:

1ms, 10ms, 50ms, 100ms, 200ms, 500ms, 1s, 2s, 5s, 10s, 20s, 30s, 60s, 90s

The array is declared in io.koraframework.telemetry.common.TelemetryConfig.MetricsConfig and shared by every module that builds a Timer.

Metric factories¶

The names and tags of framework metrics are produced by per-module metric factories, one Default<Module>MetricsFactory class per integration (package <module>.telemetry.impl). Each module's telemetry factory accepts that class as an optional dependency, so supplying your own subclass as a container component replaces the default one:

@Component
public final class TenantHttpServerMetricsFactory extends DefaultHttpServerMetricsFactory { //(1)!

    @Override
    public DefaultHttpServerMetrics create(DefaultHttpServerTelemetry.TelemetryContext context) {
        return new TenantHttpServerMetrics(context);
    }

    private static final class TenantHttpServerMetrics extends DefaultHttpServerMetrics {

        private TenantHttpServerMetrics(DefaultHttpServerTelemetry.TelemetryContext context) {
            super(context);
        }

        @Override
        protected Timer.Builder createMetricServerDuration(DurationKey metricKey, //(2)!
                                                           HttpServerRequest request,
                                                           HttpServerResponse response,
                                                           @Nullable Throwable throwable) {
            return super.createMetricServerDuration(metricKey, request, response, throwable)
                .tag("tenant", "default");
        }
    }
}

@Component
class TenantHttpServerMetricsFactory : DefaultHttpServerMetricsFactory() { //(1)!

    override fun create(context: DefaultHttpServerTelemetry.TelemetryContext): DefaultHttpServerMetrics {
        return TenantHttpServerMetrics(context)
    }

    private class TenantHttpServerMetrics(
        context: DefaultHttpServerTelemetry.TelemetryContext
    ) : DefaultHttpServerMetrics(context) {

        override fun createMetricServerDuration( //(2)!
            metricKey: DurationKey,
            request: HttpServerRequest,
            response: HttpServerResponse,
            throwable: Throwable?
        ): Timer.Builder {
            return super.createMetricServerDuration(metricKey, request, response, throwable)
                .tag("tenant", "default")
        }
    }
}

The same pattern applies to every integration, the class name follows the module: DefaultHttpClientMetricsFactory, DefaultDatabaseMetricsFactory, DefaultKafkaConsumerMetricsFactory, DefaultKafkaPublisherMetricsFactory, DefaultGrpcServerMetricsFactory, DefaultGrpcClientMetricsFactory, DefaultSoapClientMetricsFactory, DefaultSchedulingMetricsFactory, DefaultCaffeineCacheMetricsFactory, DefaultRedisCacheMetricsFactory, DefaultCircuitBreakerMetricsFactory, DefaultRetryMetricsFactory, DefaultTimeoutMetricsFactory, DefaultFallbackMetricsFactory, DefaultRateLimiterMetricsFactory, DefaultAwsS3ClientMetricsFactory, DefaultJmsConsumerMetricsFactory.

When a tag has to depend on the current request, add it to the metric key instead of the builder: every factory exposes create<Metric>Key(...) methods and key records with a withExtraTags(Tags) copy method for exactly that purpose.

If the extra tags are the same for every metric of a module, do not write a factory at all — use the telemetry.metrics.tags configuration key. Tags shared by every metric of the application belong in metrics.tags or a MetricsTagsProvider.

Standard¶

All Kora metrics follow the OpenTelemetry semantic conventions for names and tags, and each module uses exactly one naming scheme — there is no configurable specification version. Tag keys come from the io.opentelemetry.semconv attribute constants, so for example an HTTP server metric carries http.request.method, http.response.status_code, http.route, url.scheme and error.type.

The Prometheus exposition names are derived from the Micrometer name by the Prometheus naming convention:

• . is replaced with _;
• a Timer gets the _seconds suffix, and the histogram is exposed as _bucket / _count / _sum series plus a separate _max gauge;
• a Counter gets its base unit and the _total suffix (metrics built with BaseUnits.OPERATIONS therefore end with _operations_total);
• a Gauge gets its base unit as the suffix, if the metric declares one.

The error.type tag is always present on metrics that can fail — it holds the canonical exception class name, or an empty string on success.

Metrics reference¶

Micrometer metric types used:

• Timer — operation duration with count, sum, max, and histogram bucket support
• Counter — monotonically increasing counter
• Gauge — current metric value

Every metric listed below additionally carries the tags configured in that module's telemetry.metrics.tags and the application-wide common tags from metrics.tags and MetricsTagsProvider.

For exception tags, error.type uses the binary class name when a local or anonymous exception class has no canonical name. The updated metric names and tags require dashboard and alert query changes.

HTTP server¶

server.name distinguishes the public server (kora-undertow) from the system one (kora-undertow-system); a request that matched no route reports http.route as UNKNOWN_ROUTE. http.response.status_code is the response status code, so the request rate and latency can be split by status on http.server.request.duration; http.server.active_requests has no status tag because it is counted before the response exists. The Host header value is not a metric tag: the client sets it and its values are unbounded, so every new value would create a new series. The address stays on the span as the server.address attribute (see Tracing). If you need it in metrics, override createMetricServerDurationKey and createMetricActiveRequestsGaugeKey in your own metric factory and return the key with withExtraTags(...) (a tag added only in the builder is not part of the meter cache key); bounding the set of values is up to you. A non-standard HTTP method is reported in http.request.method as _OTHER.

See HTTP server module documentation for more details.

HTTP client¶

system.config is the client's configuration path. system.name.simple and system.name.canonical are set per client method: system.name.canonical is the canonical name of the declarative client interface followed by the method name (com.example.PetClient.getPet), and system.name.simple is the method name alone (getPet). Same-named methods of different clients share system.name.simple, so tell clients apart by system.config or system.name.canonical. server.port uses 80 for HTTP and 443 for HTTPS when the URI omits a port; for other schemes without an explicit port, the tag is omitted.

See HTTP client module documentation for more details.

Database¶

db.system.name is taken from the connection string for JDBC (postgresql, mysql, ...) and is cassandra for Cassandra. db.query.text holds the query identifier, not the raw SQL text.

With telemetry.metrics.driverMetrics = true (the default), the connection pool also registers its own metrics in the same registry: HikariCP pool metrics for JDBC, and the DataStax driver metrics selected by cassandra.telemetry.metrics for Cassandra.

See Database module documentation for more details.

Kafka¶

messaging.system is always kafka; messaging.operation.type on the publisher side is send.

See Kafka module documentation for more details.

gRPC server¶

See gRPC server module documentation for more details.

gRPC client¶

rpc.system.name is grpc for both the gRPC server and the gRPC client. rpc.response.status_code contains the gRPC status name (for example OK or UNAVAILABLE).

See gRPC client module documentation for more details.

SOAP client¶

rpc.system.name is soap; soap.fault.code holds the SOAP fault code and is empty when the call did not return a fault.

See SOAP client module documentation for more details.

Scheduling¶

See Scheduling module documentation for more details.

Cache¶

Distributed Redis caches report their own operation metrics:

cache.origin is redis, cache.operation is the Cache contract operation name, and cache.result on cache.requests is hit or miss.

Caffeine caches do not report cache.operation.duration and cache.requests — their telemetry delegates to the standard Micrometer cache binders, which are attached to the underlying cache when telemetry.metrics.enabled = true:

All Caffeine metrics carry the cache tag holding the cache name plus the tags from telemetry.metrics.tags. Kora builds a plain (non-loading) Caffeine cache, so the binder's loading metrics cache.load and cache.load.duration are not registered.

See Cache module documentation for more details.

Redis / Lettuce¶

db.system.name is redis; db.operation.name is the command, and lettuce.type identifies the client kind. server.address and server.port identify the Redis node. error.type holds only the Redis error prefix (for example ERR or MOVED) to keep tag cardinality bounded. The completion timer shares db.client.operation.duration with other database clients; filter on db.system.name=redis.

Resilience¶

Tag values: resilient.status is PERMITTED / REJECTED / DISABLED on call.acquire and SUCCESS / FAILURE / IGNORED_FAILURE / FALLBACK on call.result; resilient.reason is EXHAUSTED_ATTEMPTS or EXHAUSTED_BUDGET; the rate limiter's resilient.status is acquired or rejected; the fallback's resilient.type is executed.

See Resilience module documentation for more details.

JMS¶

messaging.system is always jms.

S3 client¶

rpc.system.name is s3 for the AWS based client. rpc.method is the S3 operation name and system.config is the client's configuration path.

See AWS S3 integration for more details.

Camunda 7 BPMN¶

The engine's own metrics are published separately and require camunda.engine.bpmn.telemetry.metrics.engineMetrics = true.

See Camunda 7 BPMN module documentation for more details.

Camunda REST¶

See Camunda 7 REST module documentation for more details.

Camunda 8 worker¶

The Camunda client additionally publishes its own worker job metrics into the same registry, tagged with the job job.type.

See Camunda 8 worker module documentation for more details.

System¶

JVM¶

Standard JVM and process metrics are collected automatically by the Micrometer binders bound to the registry at startup. They do not depend on any module's telemetry.metrics.enabled setting:

Some of these are registered only when the running JVM and operating system expose the corresponding MXBean value, so the exact set of series in a scrape depends on the platform.