Zum Inhalt

structlog for Structured Logging and PII Protection

Einordnung ins Overtime-Projekt

Für Abrechnung könnte Overtime hypothetisch bei Voice- oder Videochat Logs erfassen, wer wann mit wem und wie lange über welches Medium gesprochen hat. Keine dieser Informationen sollten ungefiltert geloggt werden oder im zentralen Log-System landen.

1. Kontext & Problemstellung (Deutsch)

Applikations-Logs müssen maschinenlesbar sein, um in Systemen wie OpenSearch oder FluentBit aggregiert und ausgewertet werden zu können. Gleichzeitig dürfen sie keinesfalls unmaskierte personenbezogene Daten (PII – Personally Identifiable Information) wie Namen, Telefonnummern oder E-Mail-Adressen enthalten.

Die Standard-Logging-Bibliothek (logging) von Python basiert stark auf String-Interpolation. Wenn ein Log-Eintrag den Handler erreicht, sind sensible Daten bereits fest in einen einzigen, undurchsichtigen String "eingebacken". Es gibt ab diesem Punkt keine verlässliche Möglichkeit mehr, diese Daten programmatisch zu entfernen, ohne fehleranfällige und performance-intensive reguläre Ausdrücke (Regex) über Freitext laufen zu lassen.

Ein dict-basiertes (strukturiertes) Logging-Konzept löst dieses Problem: Jedes Feld bleibt ein separates Schlüssel-Wert-Paar, das gezielt inspiziert, transformiert oder verworfen werden kann, bevor die eigentliche Serialisierung (z.B. in JSON) stattfindet.

2. Architecture Decision Record (English)

ADR-011: structlog for Structured Logging and PII Protection

Status

Accepted

Context and Problem Statement

Application logs must be machine-parseable for aggregation in OpenSearch/FluentBit and must not contain PII (names, phone numbers, email addresses, etc.). Python's stdlib logging uses string interpolation: by the time a log record reaches any handler, PII is already baked into a single opaque string. There is no reliable way to remove it programmatically without error-prone regex over free text. A dict-based logging approach keeps each field as a named key-value pair that can be inspected, transformed, or dropped before serialisation.

Decision Drivers

  • Independent Field Accessibility: Each log field must remain independently accessible (e.g., for OpenSearch dashboards analyzing numeric fields like attachment_size_mb).
  • Deterministic PII Scrubbing: PII must be removable at the processor level — not via post-hoc regex over rendered strings.
  • Unified Output Schema: Django internals, Gunicorn access/error logs, and application logs must emit identical JSON schemas to the same stdout stream.

Considered Options

  • stdlib logging with a JSON formatter — All fields are flattened into one message string by %s formatting before any processor can inspect them. PII masking requires fragile regex over free text.
  • structlog — Processors operate on a mutable event_dict (plain Python dictionary) before serialisation. Each keyword argument stays as a named field until the JSONRenderer is called.

Decision Outcome

Chosen: structlog with a two-layer PII processor chain.

Why dict-based logging enables PII control

With stdlib f-strings or %-formatting, the data is concatenated before any handler runs:

# ❌ PII is baked into the string — no processor can extract it safely
log.info(f"User {user_id} {name} sent a message in {channel}")
# emits: {"message": "User 42 Peter Müller sent a message in #general"}

With structlog kwargs, each field stays separate until the JSONRenderer:

# ✅ Processors see 'name' as a key and can drop or redact it
log.info("message sent", user_id=user_id, channel=channel, name=name)
# after processors: {"event": "message sent", "user_id": 42, "channel": "#general"}
# 'name' was dropped by drop_pii_keys_processor before render

Consequences & Trade-offs

Positive Consequences
  • Queryable Documents: OpenSearch receives a JSON document with typed, queryable fields per log line — dashboards on response times or IDs are trivial to build.
  • Scalable Privacy: Adding a new PII field to _PII_KEYS takes one line and zero call-site changes.
  • High Testability: The processor chain is easily testable (pass a dict in, assert on the dict out) without requiring log handler mocks.
Negative Consequences / Trade-offs
  • Exception Redaction Limits: pii_scrub_processor runs after ExceptionRenderer. While emails/phones in exceptions are scrubbed, names and free-text PII are not. Stack trace source lines are also untouched. PII must not be embedded in raise statements (enforced via Semgrep structlog-no-fstring-event rule). Also one must create/ implement custom rules.
  • Format Bypasses: f-string, %-format, and str.format() log calls bypass the processor chain entirely. This must be strictly banned by SAST rules.

3. Implementation & Code Snippets (English)

The implementation spans across the configuration and filter logic, ensuring both app code and framework internals utilize the exact same processing pipelines.

PII Configuration Example (config/structlog_config.py)

The core of the PII protection relies on two lightweight functions processing the event_dict before it gets serialized.

import logging
import re
import structlog

_PHONE_RE = re.compile(r"\+(\d)\d+(\d)(?!\d)")
_PHONE_REDACTED = r"+\1***\2"
_EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}")
_EMAIL_REDACTED = "[email redacted]"

def _scrub(value: str) -> str:
    value = _PHONE_RE.sub(_PHONE_REDACTED, value)
    value = _EMAIL_RE.sub(_EMAIL_REDACTED, value)
    return value

# Keys whose values must never appear in logs regardless of format.
_PII_KEYS = frozenset({"kunde", "name", "email"})

def drop_pii_keys_processor(logger, method, event_dict):
    """Drop known PII fields from the event dict before rendering."""
    for key in _PII_KEYS:
        event_dict.pop(key, None)
    return event_dict

def pii_scrub_processor(logger, method, event_dict):
    """Redact phone numbers and email addresses from every remaining string value."""
    for key, value in list(event_dict.items()):
        if isinstance(value, str):
            event_dict[key] = _scrub(value)
    return event_dict

4. Compliance Einordnung (Deutsch)

Das detaillierte Mapping auf BSI IT-Grundschutz, DSGVO und CIS Benchmarks wird später ergänzt.