Observability is a federal compliance instrument
In commercial settings, observability is an operational concern: keep the system up, keep the latency low, find the bug before the customer does. In federal settings, observability is also a compliance instrument. The same stream of telemetry that an SRE uses to diagnose a slow API endpoint feeds the agency SIEM that the security operations center uses to detect intrusions, feeds the FedRAMP continuous monitoring program that the JAB or agency authorizing official requires monthly, and feeds the audit log that an inspector general may request years later. A federal observability stack that is designed for one use case and not the others creates either gaps in compliance or duplicated infrastructure that wastes appropriated funds.
Precision Federal designs observability stacks for federal systems that produce data once and serve it to every consumer — SRE, security, compliance, leadership — through fit-for-purpose interfaces.
OpenTelemetry as the instrumentation standard
OpenTelemetry (OTel) is the CNCF project that defines a vendor-neutral standard for application instrumentation. It is the federal default for one reason: federal systems live for a decade or more, and over that span the right backend will change at least once. Datadog might be the right answer in 2026 and Splunk in 2031. Grafana might be the answer at the federal civilian agency and Honeycomb at the DoD program. OpenTelemetry decouples the instrumentation (which lives in the application code) from the backend (which lives in the platform).
What we instrument with OTel:
- Traces — every request, every cross-service hop, with W3C Trace Context propagation through HTTP, gRPC, and Kafka.
- Metrics — RED (Rate, Errors, Duration) per endpoint, USE (Utilization, Saturation, Errors) per resource, plus business metrics tied to mission outcomes.
- Logs — structured JSON with trace correlation IDs so logs can be pivoted to traces and traces to logs.
The OTel Collector sits between the applications and the backend. It batches, samples, redacts PII, enriches with cluster metadata, and routes to one or more backends.
The federal observability backend landscape
Backend selection is constrained by FedRAMP authorization status, agency procurement vehicles, and the deployment substrate. Options we deploy:
- Datadog FedRAMP High — strong APM, infrastructure monitoring, log management. Common in agencies with existing Datadog spend. Authorized for sensitive data.
- Splunk Cloud for Government — FedRAMP High. The dominant SIEM in federal. Increasingly used as an observability backend.
- Grafana Cloud (FedRAMP Moderate) — managed Grafana, Mimir, Loki, Tempo. Lower-cost option for civilian agencies with FedRAMP Moderate workloads.
- Self-hosted Grafana stack — Grafana, Prometheus or Mimir, Loki, Tempo, all deployed on Kubernetes. Inherits the FedRAMP authorization of the underlying cloud. The default for DoD IL4/IL5 and air-gapped environments. Ships in Platform One Big Bang.
- New Relic FedRAMP Moderate — full-stack observability with FedRAMP authorization at moderate.
- AWS CloudWatch / Azure Monitor / Google Cloud Operations — native cloud observability, free tier deeply integrated, FedRAMP High in respective government regions.
- Elastic Cloud for Government — Elasticsearch, Kibana, APM. Common in agencies with existing Elastic investment.
The four golden signals
For every service in a federal system we instrument and dashboard the four golden signals from the Google SRE book:
- Latency — duration of requests, separated for successful and failed.
- Traffic — request rate, broken out by endpoint and consumer.
- Errors — error rate by class (4xx, 5xx, application-level errors).
- Saturation — how full the service is (CPU, memory, queue depth, connection pool).
These map directly to SLOs (see federal SRE) and to error budgets that govern release pace.
Logs, metrics, traces — the three pillars
Each pillar serves a different question. Metrics answer "is the system healthy right now?" — high-cardinality, low-resolution, cheap to store and query. Logs answer "what happened at this exact moment?" — high-detail, structured, indexed for search. Traces answer "where in the request path did this slow down or fail?" — connecting the pieces across services. We instrument all three and ensure they are pivotable: a metric anomaly leads to a log query, a log entry leads to its trace, a trace leads to the upstream and downstream metrics.
Cardinality discipline matters in federal. Unbounded label values (user IDs, request IDs, free-form fields) blow up metric backends. We enforce cardinality budgets per service and use exemplars to bridge from metrics to traces for the high-cardinality questions.
Continuous monitoring alignment
FedRAMP and agency authorization require continuous monitoring (ConMon): monthly vulnerability scans, monthly POA&M updates, annual assessments, ongoing security event reporting, configuration baseline monitoring. The observability stack is the natural producer of much of this data:
- AU-2 (auditable events) — audit logs shipped from every service via the OTel Collector to the SIEM.
- AU-3 (content of audit records) — structured logs with required fields (timestamp, source, user, action, outcome).
- AU-6 (audit review) — SIEM dashboards for audit log review by ISSO.
- SI-4 (system monitoring) — runtime security tooling (Falco, Sysdig) emitting events to the SIEM.
- IR-5 (incident monitoring) — incident detection rules in the SIEM, automated ticket creation in the ITSM.
- CM-3 (configuration change control) — drift detection against the declared cluster state, alerts on out-of-band changes.
SLOs, error budgets, and burn rate alerts
Federal systems benefit from SLO-driven operations as much as commercial systems. We define SLOs per critical user journey, calculate error budgets monthly, and alert on burn rate rather than threshold crossing. A 1-hour burn rate alert at 14.4x the 30-day budget rate is the canonical pattern. This catches genuine incidents fast while avoiding pager fatigue from transient blips. See federal SRE for the full SLO methodology.
Tracing in microservice federal systems
Distributed tracing becomes essential as service count grows. We deploy:
- Tail-based sampling at the OTel Collector to keep all traces with errors and a representative sample of normal traces — instead of head-based sampling that misses the interesting cases.
- Service mesh telemetry from Istio or Linkerd providing infrastructure-level spans for every cross-service hop.
- Backend choices: Tempo (Grafana stack), Datadog APM, Jaeger (legacy installations), or Honeycomb where authorized.
Air-gapped and classified observability
Classified networks do not call out to SaaS observability backends. The Grafana stack — Grafana, Prometheus or Mimir, Loki, Tempo — runs cleanly inside disconnected enclaves. We deploy via Big Bang Helm charts or hand-rolled GitOps manifests, with image mirroring through Harbor or Iron Bank. Storage backends are S3-compatible (MinIO inside the enclave) or block storage from the underlying virtualization platform.
Cost discipline
Observability cost can run away in federal systems where every developer wants to add another metric. Discipline we apply:
- Cardinality budgets per service, enforced at the Collector.
- Log sampling for noisy debug streams, with full retention only for warn/error and audit categories.
- Tail-based trace sampling rather than store-everything.
- Tiered log retention (hot 30 days, warm 90, cold/archive per the agency records schedule).
- Quarterly review of dashboards and alerts; retire what nobody looks at.
Who we build observability for
- DoD — Big Bang-aligned Grafana stacks for IL4/IL5.
- VA — observability for Lighthouse APIs and digital service teams.
- HHS — CMS and component agency monitoring.
- DHS — multi-component observability with SIEM integration.
- Treasury — high-throughput financial system monitoring.