Observability from the code: a practical OpenTelemetry guide

An APM platform's auto-instrumentation covers a lot: HTTP requests, database queries, dependencies between services. But it leaves out what only your code knows: which business operation was running, what decision was made and why a technically successful result was a failure for the customer. This guide shows, with minimal Python examples, how to close that gap with OpenTelemetry.

The examples use the OpenTelemetry API for Python. The concepts are the same in Java, .NET, Go or Node.js, because the specification is common to all languages.

Before the code: API, SDK and auto-instrumentation

OpenTelemetry separates two pieces worth telling apart from the start:

• The API (the opentelemetry-api package) is what your code uses to create spans, record metrics and read context. It is stable and lightweight. If no SDK is configured, its calls do nothing, which lets you instrument libraries without imposing an operational dependency.
• The SDK (the opentelemetry-sdk package) decides what happens to that telemetry: sampling, processing and export, usually over the OTLP protocol to an OpenTelemetry Collector or directly to your analysis platform.

This separation has a practical consequence: business code depends only on the API, and destination settings (which platform, which endpoint, which sampling) are resolved when the application starts, often through standard environment variables such as OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT. Python also offers auto-instrumentation (the opentelemetry-instrument command) that covers well known frameworks and clients. What follows is what auto-instrumentation cannot do for you.

Structured logs that carry the trace

The first step, and the one with the best effort to benefit ratio, is making every log line JSON and having it carry the active trace_id and span_id. With that, from any slow or failed trace you reach the exact logs of that request, and from a log you reach the full trace.

import json
import logging

from opentelemetry import trace


class JsonFormatter(logging.Formatter):
    """Emits each record as one JSON line with the active trace context."""

    def format(self, record):
        ctx = trace.get_current_span().get_span_context()
        entry = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            # Same hexadecimal format used by W3C Trace Context
            "trace_id": format(ctx.trace_id, "032x") if ctx.is_valid else None,
            "span_id": format(ctx.span_id, "016x") if ctx.is_valid else None,
        }
        # Business fields passed with extra={"fields": {...}}
        entry.update(getattr(record, "fields", {}))
        return json.dumps(entry)


handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("payments")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info("Payment authorized", extra={"fields": {"operation": "payment", "channel": "web"}})

There are official integrations that inject these identifiers into the logging module automatically, and many platforms do the same through their agents. The example shows the mechanism so you know what to check: if your logs have no trace_id, correlation does not exist, however good the tool.

Two rules complement the format. Fields must be stable (the same names across every service) and the message must describe the event, not concatenate data. "Payment authorized" with separate fields can be filtered and aggregated; "Payment 123 authorized for customer X for 500" cannot.

Custom spans and business attributes

Auto-instrumentation creates spans for the HTTP call and the SQL query. Custom spans mark the steps the business understands and carry the attributes that explain the decision. They are also the place to record domain errors: outcomes that raise no exception but are still failures.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("payments")


def authorize_payment(payment, rule):
    with tracer.start_as_current_span("authorize_payment") as span:
        # Non sensitive business attributes: never accounts, cards or names
        span.set_attribute("payment.channel", payment["channel"])
        span.set_attribute("payment.type", payment["type"])

        if rule is None:
            # Domain error: nothing failed technically, but the flow cannot decide
            span.set_attribute("payment.outcome", "held")
            span.set_status(Status(StatusCode.ERROR, "rule_not_found"))
            return "held"

        span.set_attribute("payment.rule", rule)
        if payment["amount"] > payment["available_balance"]:
            # Expected rejection: a valid business outcome, not a system error
            span.set_attribute("payment.outcome", "rejected")
            span.set_attribute("payment.reason", "insufficient_funds")
            return "rejected"

        span.add_event("payment_authorized")
        span.set_attribute("payment.outcome", "authorized")
        return "authorized"

The example separates two cases worth separating in any domain. A rejection for insufficient funds is a legitimate outcome: it is recorded as an attribute, not as an error, so it does not pollute the error rate. An authorization rule that cannot be found is a system failure, even though no line raises an exception: it is marked with an error status so it shows up in alerts and root cause analysis. Deciding which outcomes are errors is a design decision to make with the process owner, not something left to each developer.

Context propagation across services and queues

A trace is only useful if it does not break. Between HTTP services, auto-instrumentation usually propagates context with the traceparent header from the W3C Trace Context standard. Propagation fails most often in message queues, batch processes and custom integrations. There, the code must inject the context when publishing and extract it when consuming.

from opentelemetry import trace
from opentelemetry.propagate import extract, inject
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("payments")
queue = []  # in production, your message broker client


def publish(body):
    with tracer.start_as_current_span("publish payments", kind=SpanKind.PRODUCER):
        headers = {}
        inject(headers)  # adds traceparent (and tracestate if present)
        queue.append({"body": body, "headers": headers})


def consume():
    message = queue.pop(0)
    ctx = extract(message["headers"])  # restores the producer's context
    with tracer.start_as_current_span("process payments", context=ctx, kind=SpanKind.CONSUMER):
        return message["body"]

With context traveling in the message metadata, the consumer span lands inside the same trace as the producer, even if processing happens minutes later in another service. For batch processes that group many messages, OpenTelemetry provides span links, which relate one span to several source traces without forcing a single hierarchy.

Business metrics with low cardinality

Spans explain individual requests; metrics show aggregate behavior and feed alerts. A well designed business metric answers questions such as "how many payments ended in each state in the last five minutes?" without querying traces.

from opentelemetry import metrics

meter = metrics.get_meter("payments")

payments_processed = meter.create_counter(
    "payments.processed",
    unit="{payment}",
    description="Payments processed by final outcome",
)
authorization_duration = meter.create_histogram(
    "payments.authorization.duration",
    unit="s",
    description="Payment authorization time",
)


def record(outcome, channel, seconds):
    # Low cardinality attributes: few values, known in advance
    attributes = {"payment.outcome": outcome, "payment.channel": channel}
    payments_processed.add(1, attributes)
    authorization_duration.record(seconds, attributes)

Metric attributes are the key design decision. Outcome and channel have few possible values, so they produce few time series. If you added the customer or payment ID, every new value would create a new series, with cost that grows without limit and queries that get slower and slower. Unique identifiers belong in spans and logs; metrics keep bounded dimensions.

Semantic conventions and what to avoid

OpenTelemetry defines semantic conventions: standard names for common attributes, such as service.name for the resource, http.request.method and http.response.status_code for HTTP, or messaging.destination.name for messaging. Using them lets any platform interpret your data without extra configuration, and keeps dashboards consistent across teams. For custom attributes, define a domain prefix (like payment. in the examples) and document the names in a shared catalog. Keep in mind that some conventions still evolve between versions, so check their stability status in the official documentation before fixing them in an internal standard.

Before considering a service's instrumentation done, we recommend reviewing this list:

• No sensitive data in telemetry: no account or card numbers, identity documents, passwords, tokens or health data in attributes, events or logs. Telemetry gets replicated, retained and queried by many people.
• No high cardinality in metrics: customer, transaction or session IDs, or full URLs with parameters, never as metric attributes.
• Stable span names: the name describes the operation ("authorize_payment") and carries no identifiers ("authorize_payment_123"), because that fragments analysis.
• Domain errors flagged: relevant business failures carry an error status; expected rejections, only attributes.
• Context propagated at every hop: HTTP, queues, asynchronous processes and batches.
• Complete resource attributes: service name, version and environment, so you can compare before and after a deployment.

How to verify the instrumentation works

Code that emits telemetry does not guarantee telemetry that answers an incident's questions. The most direct way to check is to rehearse a diagnosis before you need it. We recommend this test, in a pre-production environment with synthetic data:

1. Trigger a known domain failure: for example, disable a test authorization rule so payments end up held.
2. Start from the alert or dashboard: check that the outcome metric shows the change without anyone having to look for it.
3. Jump to an example trace: confirm the trace crosses every service and queue in the flow without breaks, and that the custom span has the right error status and attribute.
4. Jump from the trace to the logs: filter by trace_id and confirm that lines from every involved service appear, in JSON and with no sensitive data.
5. Measure the time: note how long it took someone who did not write the code to reach the cause. That time is the best indicator of instrumentation quality.

If any step fails, that is the next instrumentation task. Repeating this rehearsal whenever a critical flow changes keeps observability from degrading over time without anyone noticing.

Instrumenting well does not mean instrumenting everything. It means emitting, with stable names and no sensitive data, the information the team will need to explain an incident. Start with one critical flow, check that you can go from an alert to the trace and from the trace to the log, and extend the pattern from there.