Tracing
Tracing helps link separate application operations into a single execution chain and understand where a request spent time or failed.
Kora uses OpenTelemetry to create Span and to export them in the OTLP format.
Kora registers its own ContextStorage implementation for OpenTelemetry, so the current trace context is carried by a ScopedValue rather than by a thread local.
Because of that, io.opentelemetry.context.Context.current() and io.opentelemetry.api.trace.Span.current() return the correct values anywhere inside a traced operation, including on virtual threads.
Most Span are created for you: the HTTP server and client, the database, the Kafka consumer and producer, the gRPC server and client and other subsystems create their own spans through their telemetry,
and the trace context is carried between services with the W3C Trace Context standard.
Kora provides two mutually exclusive exporter modules, OTLP/gRPC and OTLP/HTTP; choose exactly one depending on the protocol your collector accepts.
Either exporter module transitively provides the core tracing wiring (OpentelemetryTracingModule), so no other tracing dependency is required.
For a step-by-step walkthrough before the reference details, see Observability.
gRPC¶
The module exports tracing data to OpenTelemetry Collector through OTLP/gRPC.
It builds an OtlpGrpcSpanExporter behind a BatchSpanProcessor, and the typical collector endpoint is http://localhost:4317.
Dependency in build.gradle:
Module:
Dependency in build.gradle.kts:
Module:
The module is io.koraframework.opentelemetry.tracing.exporter.grpc.OpentelemetryGrpcExporterModule and it ships the OkHttp sender that the OTLP/gRPC exporter uses.
HTTP¶
The module exports tracing data to OpenTelemetry Collector through OTLP/HTTP.
It builds an OtlpHttpSpanExporter behind a BatchSpanProcessor, and the typical collector endpoint is http://localhost:4318/v1/traces.
Dependency in build.gradle:
Module:
Dependency in build.gradle.kts:
Module:
The module is io.koraframework.opentelemetry.tracing.exporter.http.OpentelemetryHttpExporterModule.
Unlike the OTLP/gRPC module it sends over the JDK HTTP client sender and does not pull OkHttp into the application.
Configuration¶
Tracing is described by two configuration sections.
The tracing section is described by OpentelemetryTracingConfig and is provided by OpentelemetryTracingModule:
enabled— the global tracing switch (default:true). WithfalseKora installs a no-opTracerProvider, so noSpanare recorded and nothing is exported.attributes—OpenTelemetry Resourceattributes (default:{}).
The tracing.exporter section is described by OpentelemetryGrpcExporterConfig (for OTLP/gRPC) and OpentelemetryHttpExporterConfig (for OTLP/HTTP); both interfaces have the same field set, so switching the exporter module does not change the configuration.
If tracing.exporter.endpoint is not specified, no exporter and no span processor are created — the application starts and spans are still created and propagated, they are simply never sent to an external collector.
The tracing.attributes field defines OpenTelemetry Resource attributes that are attached to every exported Span of the whole service.
It is empty by default, so set at least the service name and namespace there, for example service.name and service.namespace, otherwise the collector receives spans without service identity.
These service-wide Resource attributes are different from per-module span attributes configured under <module>.telemetry.tracing.attributes, which are added only to the spans of a specific subsystem — see Module tracing configuration.
tracing {
enabled = true //(1)!
exporter {
endpoint = "http://localhost:4317" //(2)!
connectTimeout = "60s" //(3)!
exportTimeout = "3s" //(4)!
scheduleDelay = "2s" //(5)!
maxExportBatchSize = 512 //(6)!
maxQueueSize = 2048 //(7)!
batchExportTimeout = "30s" //(8)!
compression = "gzip" //(9)!
exportUnsampledSpans = false //(10)!
retryPolicy { //(11)!
maxAttempts = 5 //(12)!
initialBackoff = "1s" //(13)!
maxBackoff = "5s" //(14)!
backoffMultiplier = 1.5 //(15)!
}
}
attributes { //(16)!
"service.name" = "example-service"
"service.namespace" = "kora"
}
}
- Enables tracing for the whole application (default:
true). OpenTelemetry Collectorendpoint for exporting traces (optional, no default).gRPCusually useshttp://localhost:4317, andHTTPusually useshttp://localhost:4318/v1/traces.- Timeout for establishing a connection to the exporter (optional, no default — the
OpenTelemetryexporter default applies). - Maximum time to wait while the exporter sends one
OTLPrequest (default:3s). - Delay between sending accumulated
Spanto the collector (default:2s). - Maximum number of
Spanin one export batch (default:512). - Maximum queue size for
Spanwaiting to be sent (default:2048). - Maximum time the
BatchSpanProcessorwaits for one accumulated batch to be exported; this is distinct fromexportTimeout, which bounds a singleOTLPrequest (default:30s). - Data compression used during export,
gzipornone(default:gzip). - Whether to export
Spanthat were not selected bySampler(default:false). - Retry policy applied to failed export attempts (optional, the whole block may be omitted and every value below then takes its default).
- Maximum number of retry attempts (default:
5). - Initial delay before a retry attempt (default:
1s). - Maximum delay before a retry attempt (default:
5s). - Delay multiplier between retry attempts (default:
1.5). OpenTelemetry Resourceattributes added to every exportedSpan(default:{}).
tracing:
enabled: true #(1)!
exporter:
endpoint: http://localhost:4317 #(2)!
connectTimeout: 60s #(3)!
exportTimeout: 3s #(4)!
scheduleDelay: 2s #(5)!
maxExportBatchSize: 512 #(6)!
maxQueueSize: 2048 #(7)!
batchExportTimeout: 30s #(8)!
compression: gzip #(9)!
exportUnsampledSpans: false #(10)!
retryPolicy: #(11)!
maxAttempts: 5 #(12)!
initialBackoff: 1s #(13)!
maxBackoff: 5s #(14)!
backoffMultiplier: 1.5 #(15)!
attributes: #(16)!
service.name: example-service
service.namespace: kora
- Enables tracing for the whole application (default:
true). OpenTelemetry Collectorendpoint for exporting traces (optional, no default).gRPCusually useshttp://localhost:4317, andHTTPusually useshttp://localhost:4318/v1/traces.- Timeout for establishing a connection to the exporter (optional, no default — the
OpenTelemetryexporter default applies). - Maximum time to wait while the exporter sends one
OTLPrequest (default:3s). - Delay between sending accumulated
Spanto the collector (default:2s). - Maximum number of
Spanin one export batch (default:512). - Maximum queue size for
Spanwaiting to be sent (default:2048). - Maximum time the
BatchSpanProcessorwaits for one accumulated batch to be exported; this is distinct fromexportTimeout, which bounds a singleOTLPrequest (default:30s). - Data compression used during export,
gzipornone(default:gzip). - Whether to export
Spanthat were not selected bySampler(default:false). - Retry policy applied to failed export attempts (optional, the whole block may be omitted and every value below then takes its default).
- Maximum number of retry attempts (default:
5). - Initial delay before a retry attempt (default:
1s). - Maximum delay before a retry attempt (default:
5s). - Delay multiplier between retry attempts (default:
1.5). OpenTelemetry Resourceattributes added to every exportedSpan(default:{}).
The example project uses environment substitution for the endpoint and overrides a few export parameters:
tracing {
exporter {
endpoint = ${METRIC_COLLECTOR_ENDPOINT} //(1)!
exportTimeout = "250s"
scheduleDelay = "50ms"
maxExportBatchSize = 10000
}
attributes {
"service.name" = "kora-java-telemetry"
"service.namespace" = "kora"
}
}
- Resolved from the
METRIC_COLLECTOR_ENDPOINTenvironment variable, see environment substitution.
tracing:
exporter:
endpoint: ${METRIC_COLLECTOR_ENDPOINT} #(1)!
exportTimeout: "250s"
scheduleDelay: "50ms"
maxExportBatchSize: 10000
attributes:
service.name: "kora-java-telemetry"
service.namespace: "kora"
- Resolved from the
METRIC_COLLECTOR_ENDPOINTenvironment variable, see environment substitution.
If the application also has the metrics module, its MeterProvider is handed to the exporter and the span processor, and they report their own internal metrics through the same registry.
Automatic tracing¶
A tracing module in the application graph provides a Tracer component.
Every Kora subsystem that has telemetry picks that Tracer up and starts creating Span for its own operations: for every incoming request, outgoing call, message, query or scheduled run it opens a Span, binds it to the current context, nests it under the currently active Span and closes it when the operation ends.
No annotations or manual code are required for these Span.
For example, the GET /text controller from the telemetry example produces a SERVER span named GET /text, with the repository query nested inside it as a CLIENT span:
@Repository
public interface TraceRepository extends JdbcRepository {
@Query("SELECT 1")
int selectOne();
}
@Component
@HttpController
public final class SimpleController {
private final TraceRepository repository;
public SimpleController(TraceRepository repository) {
this.repository = repository;
}
@HttpRoute(method = HttpMethod.GET, path = "/text")
public HttpServerResponse get() {
var databaseValue = repository.selectOne();
return HttpServerResponse.of(200, HttpBody.plaintext("Hello world: " + databaseValue));
}
}
@Repository
interface TraceRepository : JdbcRepository {
@Query("SELECT 1")
fun selectOne(): Int
}
@Component
@HttpController
class SimpleController(private val repository: TraceRepository) {
@HttpRoute(method = HttpMethod.GET, path = "/text")
fun get(): HttpServerResponse {
val databaseValue = repository.selectOne()
return HttpServerResponse.of(200, HttpBody.plaintext("Hello world: $databaseValue"))
}
}
The table below lists the subsystems that create Span, the resulting span name and kind, and the main attributes.
Attribute names follow the OpenTelemetry Semantic Conventions.
| Subsystem | Span name | Kind | Key attributes |
|---|---|---|---|
| HTTP server | <METHOD> <route>, e.g. GET /text |
SERVER |
http.request.method, http.route, url.scheme, url.path, server.address, server.port, server.name, http.response.status_code, http.response.result_code |
| HTTP client | <METHOD> <uriTemplate> |
CLIENT |
http.request.method, http.route, server.address, server.port, url.scheme, url.path, url.full, http.response.status_code, http.response.result_code |
| Database | <Repository>.<method> |
CLIENT |
db.system.name, db.query.text |
| Kafka consumer | kafka.poll, <topic> process record |
CONSUMER |
messaging.system = kafka, messaging.client.id, messaging.consumer.group.name, messaging.destination.name, messaging.destination.partition.id, messaging.kafka.offset |
| Kafka producer | <topic> send, producer transaction |
PRODUCER / INTERNAL |
messaging.system = kafka, messaging.operation.type = send, messaging.destination.name |
| gRPC server | <service>/<method> |
SERVER |
rpc.system = grpc, rpc.service, rpc.method, server.port, server.name, network.peer.address |
| gRPC client | <fullMethodName> |
CLIENT |
rpc.system = grpc, rpc.service, rpc.method, server.address, server.port |
| SOAP client | SOAP <service> <method> |
CLIENT |
rpc.system = soap, rpc.service, rpc.method, server.address, server.port |
| S3 client | S3.<operation> |
CLIENT |
rpc.system = s3, rpc.method, aws.s3.bucket |
| JMS consumer | <destination> receive |
CONSUMER |
messaging.system = jms, messaging.destination.name, messaging.message.id |
| Scheduling | scheduling <class> |
INTERNAL |
code.function.name |
| Redis cache | cache.operation |
INTERNAL |
operation, origin = redis |
| Camunda BPMN | Camunda Delegate <name> |
INTERNAL |
delegate |
| Camunda REST | <METHOD> <route> |
SERVER |
http.request.method, http.route, url.scheme, url.path, server.address |
| Zeebe worker | Zeebe Worker <type> |
INTERNAL |
jobType, jobName, jobKey, jobWorker, processKey, elementId |
Spans produced by a named component also carry attributes that say which declaration they came from: system.config (the configuration path of the component), system.name.simple and system.name.canonical (the simple and canonical class name of the declaration).
They are set by the HTTP client, the Kafka consumer and producer, the SOAP and S3 clients, caches and scheduled jobs; the AWS S3 client names the first one system.path instead of system.config.
A few more details worth knowing:
- The HTTP server creates a span only for a request that matched a route, because the span name is built from the route template. Requests that end in
404because no route matched produce no span. url.pathandurl.fullare only added whentracePathFull(server) orpathFull(client) is enabled, which is the default — turn them off to keep identifiers out of the trace.- The
Kafkaconsumer opens akafka.pollspan for the whole poll and one<topic> process recordspan per record; the per-record span is parented to the context extracted from the record headers and is linked to thekafka.pollspan. - The
Caffeinecache and the resilience aspects report metrics and logs but do not create spans. - On failure Kora sets the span status to
ERRORand records the exception viaSpan#recordException. Most subsystems also setOKon success; the HTTP server and HTTP client leave a successful spanUNSETand only markERRORfor a4xx/5xxstatus or a connection failure.
Module tracing configuration¶
Tracing of each subsystem is configured under that module's telemetry.tracing section, described by io.koraframework.telemetry.common.TelemetryConfig.TracingConfig.
Two options are available for every subsystem:
enabled(default:true) — turns the subsystem's spans on or off. Set tofalseto stop creating spans for a specific module without removing the exporter.attributes(default:{}) — a map of key/value pairs added to every span produced by that module only. These per-span attributes differ from the service-widetracing.attributes(Resourceattributes) that apply to all spans.
Note that enabled defaults to true here, unlike telemetry.logging.enabled and telemetry.metrics.enabled, which default to false.
Two groups of modules override that default to false: the system HTTP server (httpServer.system.telemetry.tracing) and the resilience aspects.
The telemetry.tracing section lives at the same path as the module's own configuration:
| Subsystem | Configuration path |
|---|---|
| HTTP server | httpServer.telemetry.tracing |
| System HTTP server | httpServer.system.telemetry.tracing |
| HTTP client | httpClient.<name>.telemetry.tracing |
| JDBC database | jdbc.telemetry.tracing |
| Cassandra database | cassandra.telemetry.tracing |
| gRPC server | grpcServer.telemetry.tracing |
| gRPC client | grpcClient.<ServiceName>.telemetry.tracing |
| Kafka consumer | kafka.consumer.<name>.telemetry.tracing |
| Kafka producer | kafka.producer.<name>.telemetry.tracing |
| Scheduling | scheduling.telemetry.tracing |
| Cache | <cache config path>.telemetry.tracing |
Two subsystems add an option of their own on top of enabled and attributes:
httpServer.telemetry.tracing.tracePathFull(default:true) — addsurl.pathwith the real request path to the server span.httpClient.<name>.telemetry.tracing.pathFull(default:true) — addsurl.pathandurl.fullwith the real request URI to the client span.
httpServer {
telemetry {
tracing {
enabled = true //(1)!
tracePathFull = true //(2)!
attributes { //(3)!
"component" = "gateway"
}
}
}
}
jdbc {
telemetry {
tracing {
enabled = false //(4)!
}
}
}
- Enables tracing for the HTTP server (default:
true). - Adds the real request path as
url.pathto the HTTP server span (default:true). - Per-span attributes added only to HTTP server spans (default:
{}). - Disables tracing for database queries (default:
true).
httpServer:
telemetry:
tracing:
enabled: true #(1)!
tracePathFull: true #(2)!
attributes: #(3)!
component: "gateway"
jdbc:
telemetry:
tracing:
enabled: false #(4)!
- Enables tracing for the HTTP server (default:
true). - Adds the real request path as
url.pathto the HTTP server span (default:true). - Per-span attributes added only to HTTP server spans (default:
{}). - Disables tracing for database queries (default:
true).
Module-specific tracing parameters are also described in those modules' own documentation, for example HTTP server, HTTP client, gRPC server, gRPC client, and Kafka.
Context propagation¶
Kora stitches distributed traces together with the W3C Trace Context standard through W3CTraceContextPropagator.
This happens automatically and requires no configuration:
- HTTP server —
traceparentis extracted from the request headers and becomes the parent of the serverSpan; the identifiers of that span are then written back into the response headers, so a caller can correlate the response with the trace. - Kafka —
traceparentis injected into the record headers by the producer and extracted from the record headers by the consumer, so each<topic> process recordspan continues the producer's trace. - gRPC client —
traceparentis injected into the call metadata. - gRPC server —
traceparentis extracted from the call metadata and injected back into the response headers metadata. - JMS consumer —
traceparentis extracted from the message properties. - Camunda REST and Zeebe worker —
traceparentis extracted from the request headers and the job headers respectively.
The Kora HTTP client is the exception: it opens a CLIENT span for the outgoing call so the call is visible in your own trace, but it does not write traceparent into the outgoing request.
If a downstream service has to continue the same trace, add the header yourself in an HttpServerInterceptor-style HTTP client interceptor.
Within one service nothing has to be propagated by hand: the current Span lives in a ScopedValue, so a span you create manually (see Synchronous tracing) automatically becomes the parent of everything called inside it, in the same thread.
Crossing a thread boundary is the one case that needs explicit work — see Asynchronous tracing.
Sampling¶
The core tracing components are provided by OpentelemetryTracingModule as @DefaultComponent, which means each of them can be replaced by declaring your own component of the same type:
Sampler— decides whichSpanare recorded. The default isSampler.parentBased(Sampler.alwaysOn()), i.e. record every rootSpanand follow the parent's decision for childSpan.IdGenerator— generates trace and span identifiers. The default isIdGenerator.random().Supplier<SpanLimits>— limits on attributes, events, and links perSpan. The default isSpanLimits.getDefault().KoraTracer— the helper used for manual spans.
The exporter modules declare SpanExporter and SpanProcessor as @DefaultComponent too, so an application can send spans somewhere else entirely by providing its own.
To apply head-based sampling, override the Sampler factory method in your application, for example to record roughly 10% of root traces:
The exportUnsampledSpans export option controls whether Span that were not selected by the Sampler are still sent to the collector; it is false by default, so only sampled Span are exported.
Tracing context¶
The current trace context is read through the standard OpenTelemetry API — Kora plugs its own storage behind it, so no Kora-specific accessor is needed.
To get the current Span:
To get the current trace identifier:
When there is no current Span, Span.current() returns Span.getInvalid() and its SpanContext reports isValid() == false with an all-zero trace identifier, so these calls never return null and never throw.
The pieces that make this work live in io.koraframework.common.telemetry:
OpentelemetryContext— an implementation ofio.opentelemetry.context.Contextbacked byScopedValue. Kora registers it as anOpenTelemetryContextStorageProvider, which is whyContext.current()andSpan.current()work on any thread that Kora entered, virtual threads included.OpentelemetryContext.VALUE— theScopedValue<Context>itself. Binding it withScopedValue.where(OpentelemetryContext.VALUE, ctx)is how a context is made current;Context#makeCurrent()is deliberately unsupported and throwsIllegalStateException, because a scoped value cannot be attached and detached imperatively.Observation— the per-operation telemetry object of the module that is currently running, also bound to aScopedValue.Observation.current(HttpServerObservation.class)returns it andobservation.span()gives the module's own span; it throws if there is no bound observation of that type.
Log correlation¶
Log correlation is done by the Logback module: KoraAsyncAppender captures Span.current().getSpanContext() at the moment the event is queued, and ConsoleTextRecordEncoder writes traceId= and spanId= into the log line whenever that span context is valid.
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="io.koraframework.logging.logback.ConsoleTextRecordEncoder"/>
</appender>
<appender name="ASYNC" class="io.koraframework.logging.logback.KoraAsyncAppender">
<appender-ref ref="STDOUT"/>
</appender>
As a result every log line emitted within a traced operation carries the traceId and spanId, which lets you jump from a log entry to the corresponding trace in your observability backend and back.
Lines logged outside any traced operation simply have no such fields.
The Kora MDC is a separate mechanism for your own structured fields — the trace identifiers do not go through it, so nothing has to be put into or removed from MDC around a span.
Synchronous tracing¶
In addition to Span created by the framework, you can create your own.
The simplest way is the KoraTracer component: it builds the Span, binds it as the current context for the duration of the call, sets the status, records an exception if one is thrown, and ends the span — all in one call.
traceParent(name, …)— creates aSpannested under the currently active one.traceNew(name, …)— creates a rootSpanwith no parent, for work that must start its own trace.tracer()— returns the underlyingio.opentelemetry.api.trace.Tracerwhen you need full control.
Each of them accepts either a TraceCallable, which returns a value, or a TraceRunnable, which does not; both receive the created Span so that attributes and events can be added to it.
@Component
public final class MyService {
private final KoraTracer tracer;
public MyService(KoraTracer tracer) {
this.tracer = tracer;
}
public String doTraceWork(String userId) {
return tracer.traceParent("myOperation", span -> {
span.setAttribute("user.id", userId);
return doWork(userId);
});
}
private String doWork(String userId) {
// do some work
}
}
@Component
class MyService(private val tracer: KoraTracer) {
fun doTraceWork(userId: String): String {
return tracer.traceParent("myOperation", KoraTracer.TraceCallable<String, RuntimeException> { span -> //(1)!
span.setAttribute("user.id", userId)
doWork(userId)
})
}
private fun doWork(userId: String): String {
// do some work
}
}
traceParentis overloaded forTraceCallableandTraceRunnable, andKotlincannot choose between two functional interfaces on its own — pass the explicit SAM constructor.
If you need something KoraTracer does not cover, such as a custom span kind or a link to another trace, build the Span from the Tracer yourself and bind it as the current context for the duration of the operation:
@Component
public final class MyService {
private final Tracer tracer;
public MyService(Tracer tracer) {
this.tracer = tracer;
}
public String doTraceWork() {
var span = tracer.spanBuilder("myOperation")
.setSpanKind(SpanKind.INTERNAL)
.setParent(io.opentelemetry.context.Context.current())
.startSpan();
return ScopedValue.where(OpentelemetryContext.VALUE, io.opentelemetry.context.Context.current().with(span))
.call(() -> {
try {
var result = doWork();
span.setStatus(StatusCode.OK);
return result;
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
span.end();
}
});
}
private String doWork() {
// do some work
}
}
@Component
class MyService(private val tracer: Tracer) {
fun doTraceWork(): String {
val span = tracer.spanBuilder("myOperation")
.setSpanKind(SpanKind.INTERNAL)
.setParent(io.opentelemetry.context.Context.current())
.startSpan()
val carrier = ScopedValue.where(
OpentelemetryContext.VALUE,
io.opentelemetry.context.Context.current().with(span)
)
return carrier.call<String, RuntimeException> {
try {
val result = doWork()
span.setStatus(StatusCode.OK)
result
} catch (e: Exception) {
span.recordException(e)
span.setStatus(StatusCode.ERROR, e.message)
throw e
} finally {
span.end()
}
}
}
private fun doWork(): String {
// do some work
}
}
Asynchronous tracing¶
The trace context is a ScopedValue, and a scoped value is visible only inside the dynamic scope that bound it.
Handing work to another thread therefore drops the context unless you carry it over explicitly: capture io.opentelemetry.context.Context.current() in the calling thread and re-bind it in the worker thread.
OpentelemetryContext implements the wrap family of the OpenTelemetry Context interface on top of ScopedValue, so wrapping the task is usually all that is needed — wrap(Runnable), wrap(Callable), wrapSupplier, wrapFunction and wrapConsumer are all available.
@Component
public final class MyService {
private final KoraTracer tracer;
private final ExecutorService executor;
public MyService(KoraTracer tracer, ExecutorService executor) {
this.tracer = tracer;
this.executor = executor;
}
public CompletableFuture<String> doTraceWork() {
return tracer.traceParent("myOperation", span -> {
var ctx = io.opentelemetry.context.Context.current(); //(1)!
return CompletableFuture.supplyAsync(ctx.wrapSupplier(this::doWork), executor); //(2)!
});
}
private String doWork() {
// runs on another thread, but Span.current() is still the "myOperation" span
}
}
- Captured while the span is still current, so it already contains the
myOperationspan. wrapSupplierre-binds that context around the call in the worker thread.
@Component
class MyService(private val tracer: KoraTracer, private val executor: ExecutorService) {
fun doTraceWork(): CompletableFuture<String> {
return tracer.traceParent("myOperation", KoraTracer.TraceCallable<CompletableFuture<String>, RuntimeException> { span ->
val ctx = io.opentelemetry.context.Context.current() //(1)!
CompletableFuture.supplyAsync(ctx.wrapSupplier { doWork() }, executor) //(2)!
})
}
private fun doWork(): String {
// runs on another thread, but Span.current() is still the "myOperation" span
}
}
- Captured while the span is still current, so it already contains the
myOperationspan. wrapSupplierre-binds that context around the call in the worker thread.
Note that KoraTracer ends the span as soon as its callback returns, which for the example above is before the future completes.
When the span has to cover the whole asynchronous operation, build it from the Tracer yourself as shown in Synchronous tracing and call span.end() from the completion callback.