Production Logging: From Printed Strings to Traceable Events

The conclusion first

A production log is not a text file for temporary console.log statements. It is a stream of system events. It should answer:

• What happened, and when?
• Which service, version and environment produced it?
• Which request, job or message did it belong to?
• Which business object was affected?
• Did the operation succeed, fail, retry or defer?
• Which dependency failed, and who should respond next?

Logging best practice can be reduced to four properties: structured, correlated, secret-safe and actionable.

Logs, metrics and traces answer different questions

The three observability signals should not replace one another:

An error-rate metric may reveal an incident. A trace may locate latency at a payment provider. A log explains which provider operation ran, how its result was classified and which action the system took.

Use structured logging

Do not concatenate every fact into a sentence intended only for humans:

Payment failed for order 123 because provider timeout

Emit JSON with a stable schema:

{
  "timestamp": "2026-10-04T10:30:12.431Z",
  "severity": "ERROR",
  "event": "payment.provider_request_failed",
  "service": "payment-api",
  "serviceVersion": "2026.10.04.2",
  "environment": "production",
  "traceId": "4f91...",
  "spanId": "0ab3...",
  "requestId": "req_...",
  "orderId": "ord_...",
  "provider": "provider-a",
  "operation": "capture",
  "outcome": "retry_scheduled",
  "errorType": "UPSTREAM_TIMEOUT",
  "durationMs": 1850
}

This supports queries, aggregation and alerts by event, service, errorType or orderId instead of fragile full-text matching. Google Cloud defines JSON payloads as structured logs whose fields can be queried and indexed; plain text can only be searched as a whole. Google Cloud structured logging

The OpenTelemetry log data model similarly defines timestamp, severity, body, resource, attributes, TraceId and SpanId as standard fields. OpenTelemetry Logs Data Model

Correlate the complete flow

A request ID may be enough in a monolith. Once a flow crosses services, queues and background jobs, propagate richer context:

• traceId: shared by the end-to-end flow.
• spanId: identifies one operation within the flow.
• requestId: identifies one HTTP request.
• messageId: identifies one queue message.
• jobId: identifies one background execution.
• correlationId: connects protocols or systems.
• causationId: identifies the event that caused this event.

flowchart LR
  A[API<br/>traceId: T1] --> Q[Queue message<br/>traceId: T1]
  Q --> W[Worker<br/>traceId: T1]
  W --> P[Provider call<br/>traceId: T1]

Keep these IDs in structured fields rather than hiding them inside a message. OpenTelemetry recommends including TraceId and SpanId in log records so that records from multiple components can be correlated with the same trace. OpenTelemetry log correlation

Which events should be logged?

Not every line of code deserves a log record. Prioritize boundary events that reconstruct system behavior.

Requests and external integrations

• Request outcome, status code and duration.
• Dependency name, operation, result classification and latency.
• Timeouts, retries, circuit breakers and fallbacks.
• Avoid complete request and response bodies.

Business state transitions

• The old and new state of an order, payment, membership or document.
• The actor or system that initiated the change.
• Rule version or reason behind the decision.
• Business identifiers rather than full object snapshots.

Asynchronous work

• Job or message start, completion, retry and dead-letter transition.
• Attempt number, queue delay and processing duration.
• Idempotency decision.

Security and audit events

• Authentication success and failure, and authorization denial.
• High-risk access, configuration or permission changes.
• Key rotation, secret access and administrative operations.
• Audit logs should be governed separately from ordinary debug logs.

OWASP recommends risk-proportionate logging for authentication, authorization, validation, system errors, high-risk operations and data changes rather than blind collection of everything. OWASP Logging Cheat Sheet

Data that must not be logged directly

Logs are copied into collectors, search platforms, backups and analytics systems. Once written, they are harder to erase than primary data. Prefer an allow-list of approved fields instead of logging entire objects and hoping every secret is masked later.

Normally exclude:

• Passwords, OTPs and recovery codes.
• Access tokens, refresh tokens, session cookies and API keys.
• Encryption keys, plaintext DEKs and database connection strings.
• Full payment card, bank account or identity-document data.
• Unnecessary PII, medical or regulated information.
• Complete request or response bodies and Authorization headers.
• Raw user-uploaded file contents.

Use internal identifiers, tokenization or stable HMAC identifiers when correlation is needed. Masking a few middle characters does not automatically make data safe. OWASP explicitly recommends removing, masking, hashing or encrypting session identifiers, tokens, passwords, keys, payment data and sensitive personal data.

Also defend against log injection: use a structured logger, limit untrusted value lengths, and sanitize control characters such as CR and LF so an attacker cannot forge another record.

Give log levels stable meanings

Do not record the same exception and stack trace in the repository, service and controller layers. Usually the boundary that understands the business outcome should log it once, while lower layers propagate the error with its cause. Otherwise one failure produces several alerts and corrupts event counts.

Also, ERROR does not simply mean “an exception occurred.” A retry that fully recovered may be WARN or a metric. A reconciliation mismatch without an exception may still justify an ERROR or audit event.

Centralize collection without letting logging break the service

A common architecture has applications emit JSON to stdout, then lets the execution environment, an agent or an OpenTelemetry Collector batch and forward it:

flowchart LR
  A[Applications<br/>JSON to stdout] --> C[Agent / OTel Collector]
  C --> L[Central log store]
  L --> S[Search]
  L --> R[Rules / Alerts]
  L --> X[Archive]

The main request path should not synchronously wait for a remote logging backend. Define buffering, backpressure and drop policies for collector outages, and cap disk usage so a logging failure cannot become a production outage.

The centralized store needs:

• Encryption in transit and at rest.
• Least-privilege read access and auditing of log access.
• Protection against unauthorized modification or deletion.
• Isolation between development, production and tenants.
• Detection for stopped collectors, pipeline backlog and ingestion failure.

Set retention and cost by purpose

Keeping every log forever is expensive and increases security and privacy exposure. Retention should follow the use case:

• High-volume operational logs remain searchable briefly, then move to cheaper archive or expire.
• Error and incident evidence remains long enough for investigation and review.
• Security and audit logs follow risk, contract and regulatory requirements, with stronger tamper protection.
• Temporary debug logging expires automatically.

Sample high-volume successful traffic when appropriate, but do not casually sample errors, security events or required audit trails. Watch high-cardinality fields as well: indexing complete URLs, arbitrary user input or always-unique values can rapidly increase cost.

OWASP recommends neither deleting logs before their required retention period nor retaining them indefinitely beyond their purpose. Legal, contractual and business requirements should determine the period.

Alerts should lead to action

Do not page everyone for every ERROR record. Alert on actionable conditions:

• An error class exceeds its baseline over a window.
• Payment, login or checkout success rate drops.
• Queue lag, retries or dead-letter volume rises.
• A new error type appears only after a deployment.
• Log ingestion stops unexpectedly.
• Permissions, keys or important configuration change unexpectedly.

Each alert should identify an owner and include impact, a dashboard, a useful query and a runbook. An alert that nobody owns and nobody can act on only creates alert fatigue.

Pre-production checklist

• Are logs emitted as structured JSON?
• Are event names and fields consistent across services?
• Does trace context cross HTTP, queues and workers?
• Does an allow-list prevent tokens, passwords, keys and PII from being logged?
• Can a business state change be reconstructed by its identifier?
• Is each failure recorded once at the correct boundary?
• Can the service continue safely when its logger or collector fails?
• Are access, retention, archive and deletion policies explicit?
• Are ingestion failures, drop rate and storage cost monitored?
• Does every alert have an owner and a runbook?

Good logging does not attempt to preserve everything. It records enough evidence to understand system behavior, trace responsibility and respond to incidents—without sending secrets, noise and unlimited storage cost into the logging platform with it.