Edge observability: branches, ATMs and devices

Most observability guidance assumes telemetry is born close to where it is analyzed: in a cluster, in a cloud region, in the same data center as the monitoring platform. But a large part of the customer experience happens far from there. An ATM that does not dispense cash, a branch terminal that will not print the receipt or a self-service kiosk stuck in a reboot loop are real incidents, and they are often the ones customers remember.

Observing the edge is not about installing the same agent on more machines. The conditions change: connectivity is intermittent, bandwidth is limited and expensive, clocks drift, devices are moved, replaced or switched off, and sometimes nobody on site knows what to do when something fails. This article proposes an architecture and a set of decisions so that telemetry from those sites arrives complete, in order and trustworthy.

A layered architecture for the edge

The pattern we recommend has four layers: the device that generates the signal, a local collector on site, the link back to the center and the central platform. The key piece is the local collector, because it decouples the device from the quality of the link.

The OpenTelemetry Collector fits this role well because its documentation describes exactly two deployment patterns: agent, a collector next to the application or on the same machine, and gateway, one or more collectors that receive telemetry from many sources at a single OTLP endpoint. At the edge they are usually combined: an agent on each device with enough capacity, or one collector per branch acting as a local gateway for devices that cannot run their own; and a central gateway that receives all sites and applies common policies before handing data to the platform.

If your applications are already instrumented with OpenTelemetry, as we describe in observability from the code, the change for the edge is in the collection topology, not in the code. If they are not, the collector can read local log files while instrumentation progresses.

Intermittent connectivity: local buffer and retries

A branch can lose its link for minutes or hours, and a mobile device can go days without a stable connection. If telemetry is lost exactly when the link fails, the central platform goes blind at the moment it matters most.

The OTLP protocol defines how to behave on failure: which responses are retried (over HTTP, for example, 429, 502, 503 and 504), which are not, and how to honor the server's backpressure signals with exponential backoff. The collector implements this with two mechanisms worth configuring explicitly:

• Retries (retry_on_failure), with growing intervals and a maximum time.
• Sending queue (sending_queue), which by default lives in memory and is lost if the machine restarts. With the file_storage extension the queue becomes persistent on disk.

The buffer needs sizing. A starting rule: estimate the site's hourly volume, multiply it by the longest outage you want to tolerate and compare the result with available disk. If it does not fit, decide what gets sacrificed first. We recommend always keeping business and security events, and dropping debug logs and high-frequency metrics first.

Here is a minimal configuration example for a branch collector. Memory values, paths and names are illustrative; adjust them to your hardware.

# Local collector for a branch (illustrative example).
# filelog and file_storage ship in the Collector's contrib distribution.
extensions:
  file_storage:
    directory: /var/lib/otelcol/buffer   # the directory must exist

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317         # only applications on this machine
  filelog:
    include: [ /var/log/terminal/*.log ]
    storage: file_storage                # remembers how far it read each file

processors:
  memory_limiter:                        # always the first processor
    check_interval: 1s
    limit_mib: 200
    spike_limit_mib: 50
  resource:
    attributes:
      - key: site.id                     # site identity on every record
        value: branch-0001
        action: upsert
  batch:

exporters:
  otlp:
    endpoint: gateway.example.internal:4317
    tls:
      ca_file: /etc/otelcol/ca.pem       # mutual TLS: the site validates the gateway...
      cert_file: /etc/otelcol/site.pem   # ...and the gateway validates the site
      key_file: /etc/otelcol/site.key
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0                # retry with no time limit
    sending_queue:
      storage: file_storage              # persistent queue on disk

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp]

Two details matter. The memory_limiter processor goes first in the chain, as its documentation recommends, so the collector applies backpressure before running out of memory on modest hardware. And retrying with no time limit only makes sense with a bounded persistent queue: when the queue fills up, the collector drops new data instead of growing without control.

Less data at the source: filtering and sampling

At the edge every byte costs something: bandwidth shared with business operations, mobile data backup links and central storage. Reducing at the source is cheaper than reducing at the destination.

• Filter what will not be used, with the filter processor: debug logs in production, metrics nobody queries, repetitive health checks.
• Aggregate before sending: many questions can be answered with a per-minute count instead of every individual event.
• Sample traces with judgment. Probabilistic sampling in the local collector (probabilistic_sampler) is simple and predictable. Tail sampling (tail_sampling), which decides after seeing the full trace and lets you keep every trace with an error, is best done at the central gateway, where all spans of the same transaction arrive.

One warning: never sample the events that feed business or security indicators. If a dashboard counts failed withdrawals, every one of those events has to arrive. Business signals are what connect the edge to what we describe in business observability in banking.

Time, identity and transmission security

Clock synchronization

Correlating an error at an ATM with a failure in the central authorizer requires both clocks to tell the truth. NIST SP 800-92, the guide to computer security log management, identifies inconsistent timestamps across sources as one of the challenges of log management: each device records events by its own clock, and without a common reference event reconstruction loses value. In practice: NTP configured and monitored on every device, and an alert when drift exceeds a threshold your team defines. The OpenTelemetry log data model helps, because it distinguishes the time the event occurred at the source from the time the collector observed it; keeping both lets you detect drifting clocks and data that arrived late because of a link outage.

Device and site identity

Every record should say where it comes from without relying on an IP address, which changes frequently at the edge. We recommend stable resource attributes: device identifier, site, region, device type and software version, following OpenTelemetry semantic conventions where they exist (for example, host.id or service.instance.id) and documented custom attributes where they do not.

Transmission security

Edge telemetry travels over networks you do not fully control and may contain sensitive data. Three minimum practices: mutual TLS with a certificate per site or per device that can be revoked individually; masking personal or card data in the local collector before it leaves the site; and protecting the on-disk buffer with the same controls as any other local storage on the device. NIST SP 800-92 treats the integrity and confidentiality of logs, both in transit and at rest, as part of the log management infrastructure.

Inventory: knowing what should be reporting

In the cloud, a resource that stops sending data has usually been deleted. At the edge, a device that stops sending data may be switched off, offline, stolen or running a crashed collector. And if nobody knew that device existed, its silence triggers no alert at all.

That is why edge observability starts with a reliable inventory:

1. A list of what is expected. Every site and every device, with its identifier, owner, criticality and operating hours, in the CMDB or whatever source of truth you use.
2. Heartbeat. Each collector sends its own periodic signal, independent of business activity. The collector also exposes its own internal metrics, useful for spotting growing queues or rejected data.
3. Alert on absence. Compare what is expected with what is received and alert when a critical device does not report during its operating hours, with a threshold that tolerates short link outages.
4. Versions and configuration. Record which collector version and which configuration each site runs. A fleet with dozens of different configurations is impossible to operate.

That inventory is also what lets the command center move from "device X is not responding" to "branch Y has been running without two of its three ATMs for an hour", which is the information the business needs.

How to get started

We recommend starting with a limited pilot: one device type and a few representative sites, including those with the worst connectivity. Decide first which questions the telemetry must answer (service availability at the site, failed transaction rate, device health) and choose signals with that in mind, based on what each one answers best, as we explain in logs, metrics and traces. Then test what matters: cut a pilot site's link for as long as you want to tolerate, restore it and verify that the telemetry arrived complete, in order and without significant duplicates. With that measured, rolling out to the whole network is a deployment problem, not a design problem.

If the telemetry includes customer data, validate with Security and your data protection function what gets masked at the source and how long the buffer is retained.