Audit Logging Infrastructure¶

Overview¶

The HNS Ticketing System generates audit events across 25+ business domains — orders, payments, seat changes, ticket transfers, queue activity, blacklist enforcement, admin operations, and more. Every meaningful business action is logged so that a product owner can see a complete picture of what's happening inside the system at any time.

All audit logs are stored in Grafana Loki rather than the primary PostgreSQL database.

Key benefits:

Performance isolation — Audit writes never contend with seat reservation or checkout transactions
Purpose-built retention — Per-domain retention policies (90 days to 7 years) without PostgreSQL partition management
Operational observability — Native Grafana dashboards for real-time monitoring, alerting, and forensic investigation
Cost efficiency — Loki's index-free design stores log data at object-storage cost, not database-storage cost
Business visibility — Every log entry includes human-readable context (match name, user email, seat label) so non-technical stakeholders can understand what happened

PostgreSQL remains the system of record for transactional data only. Three tables with transactional requirements (idempotency constraints, foreign key dependencies, or status machines) remain in PostgreSQL.

Architecture Decision¶

Why Loki over ELK/OpenSearch¶

Criterion	Grafana Loki	ELK/OpenSearch
Ops overhead	Minimal — single binary or managed service	High — Elasticsearch cluster management, JVM tuning
Index design	Index-free — labels only, log lines stored as-is	Full-text indexing — expensive at audit log scale
Storage cost	Object storage (S3/GCS/MinIO)	SSD-backed Elasticsearch shards
Query language	LogQL (Prometheus-style, team already knows PromQL)	Lucene/KQL (separate query language to learn)
Grafana integration	Native — first-class data source	Plugin-based — separate Kibana often needed
Cloud-agnostic	Yes — runs on any cloud or on-prem	Yes, but heavier to self-host
Horizontal scaling	Read/write path independently scalable	Requires careful shard/replica management

Decision: Grafana Loki is the audit log store. Grafana provides dashboards and alerting.

Why Backend-Only Logging¶

All audit logging originates from the Ticketing Backend (Symfony). The Admin Portal is a thin UI proxy with no database — every action it performs calls a backend API endpoint, which handles the audit logging. This eliminates duplicate log entries and ensures a single source of truth.

High-Level Architecture¶

┌─────────────────────────────────────────────────────────────────────────────┐
│                      Ticketing Backend (Symfony)                              │
│                                                                              │
│   Business Logic (Controllers / Services)                                    │
│        │                                                                     │
│        ▼                                                                     │
│   AuditLogger::log(type, data)                                               │
│   (PII sanitization, auto-adds performed_by + timestamp)                     │
│        │                                                                     │
│        ▼                                                                     │
│   Monolog 'audit' channel                                                    │
│   (JSON formatter → var/log/audit.log)                                       │
│                                                                              │
└──────────────────────────┬──────────────────────────────────────────────────┘
                           │
                    ┌──────▼──────────────────────┐
                    │   Grafana Alloy (sidecar)    │
                    │   • Tails audit.log (5s)     │
                    │   • Parses JSON fields       │
                    │   • Extracts Loki labels      │
                    │   • Batch push (3s / 512KiB) │
                    │   • Retry: 1s→30s, 5 retries │
                    └──────────────┬───────────────┘
                                   │ HTTP POST
                                   │ /loki/api/v1/push
                                   │ (basic auth)
                    ┌──────────────▼───────────────┐
                    │        Grafana Loki           │
                    │   ┌──────────┐ ┌───────────┐ │
                    │   │ Ingester │→│ Storage   │ │
                    │   └──────────┘ │ (S3/disk) │ │
                    │   ┌──────────┐ └───────────┘ │
                    │   │Compactor │ (retention)    │
                    │   └──────────┘               │
                    └──────────────┬───────────────┘
                                   │
                    ┌──────────────▼───────────────┐
                    │          Grafana              │
                    │   • Audit Dashboards          │
                    │   • Alerting Rules            │
                    │   • Admin Portal (via API)    │
                    └──────────────────────────────┘

Audit Event Categories¶

Complete Event Catalog (25 domains)¶

Every event includes these common fields automatically:

Field	Description
`audit_type`	Event domain (label, used for Loki routing and retention)
`source`	Always `ticketing-backend`
`performed_by`	Email of the user who triggered the action
`timestamp`	ISO 8601 timestamp

Operational Events (13 — implemented)¶

#	`audit_type`	Description	Retention
1	`seat_change`	Seat status transitions (available → reserved → sold, etc.)	2 years
2	`match_change`	Match lifecycle: create, update, publish, cancel, close, reschedule	3 years
3	`transfer`	Ticket holder transfers (self-service + support-initiated)	3 years
4	`emergency_print`	Emergency/duplicate ticket printing at stadium	5 years
5	`blacklist_change`	Blacklist entry create/update/remove/restore	7 years
6	`violation`	Blocked purchase or transfer attempt by blacklisted person	7 years
7	`blacklist_import`	CSV blacklist import batch summary	3 years
8	`blacklist_cancellation`	Automatic ticket cancellation triggered by blacklisting	7 years
9	`email_delivery`	Email send events with delivery status	90 days
10	`push_delivery`	Push notification send events with delivery status	90 days
11	`barcode_export`	Access control barcode file export for event day	3 years
12	`attendance_import`	Post-match attendance/barcode scan file import	3 years
13	`report_export`	Report generation and file export	1 year

Admin & Security Events (3 — implemented)¶

#	`audit_type`	Description	Retention
14	`admin_login`	Admin user authentication	1 year
15	`user_management`	Admin user create, deactivate, reactivate, role change, password reset	3 years
16	`support_refund`	Refund request processed via support	3 years

Business Flow Events (5 — planned, high priority)¶

#	`audit_type`	Description	Retention
17	`order_created`	New order placed after successful checkout	3 years
18	`order_paid`	Payment confirmed (Stripe webhook success)	3 years
19	`order_cancelled`	Order cancellation with reason (refund, timeout, admin)	3 years
20	`payment_event`	Payment attempts, failures, refund processing, webhook events	3 years
21	`sales_phase_change`	Sales phase opened, closed, or modified	3 years

Operational Visibility Events (4 — planned, medium priority)¶

#	`audit_type`	Description	Retention
22	`queue_event`	Queue join, activation, expiration, user leaving	1 year
23	`quota_change`	Quota create, assign, claim, close, subquota delegation	3 years
24	`ticket_cancellation`	Explicit ticket cancellation with reason and context	3 years
25	`cart_event`	Cart created, items added, cart expired, checkout started	90 days

Configuration Audit Events (2 — planned, lower priority)¶

#	`audit_type`	Description	Retention
26	`loyalty_event`	Points awarded, tier changed	2 years
27	`config_change`	Stadium, sector, seat map, price category, or snake config changed	3 years

Remaining in PostgreSQL (3 transactional tables)¶

Table	Reason
`stripe_webhook_logs`	Idempotency — `UNIQUE` constraint on `event_id` prevents duplicate webhook processing; requires atomic check-and-insert
`quota_import_batches`	FK dependency — Referenced by `quotas.batch_id`; batch status drives import validation logic
`cancellation_requests`	Status machine — PENDING → PROCESSING → COMPLETED/FAILED state transitions with concurrent access control

Log Structure & Labels Strategy¶

Loki Labels (low cardinality only)¶

Loki indexes labels, so they must be low cardinality to avoid index explosion:

Label	Values	Purpose
`service`	`ticketing-backend`	Source application (single service)
`environment`	`production`, `staging`, `development`	Deployment environment
`audit_type`	See catalog above (27 values)	Audit domain routing and retention
`level`	`info`, `warning`, `error`	Log severity level
`channel`	`audit`	Monolog channel

Human-Readable Log Design¶

Every log entry must include enough denormalized context that a product owner can read it without looking up UUIDs:

Include match_name alongside match_id (e.g., "Croatia vs England, 2026-09-15 Maksimir")
Include user_email alongside user_id
Include seat_label alongside seat_id (e.g., "Sector East, Row 5, Seat 12")
Include holder_name alongside holder IDs
Include human-readable reason fields

Schema Examples Per Audit Type¶

Operational Events¶

`seat_change`¶

{
    "audit_type": "seat_change",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "match_seat_inventory_id": "uuid",
    "sector_name": "East Stand",
    "seat_label": "Row 5, Seat 12",
    "from_status": "AVAILABLE",
    "to_status": "RESERVED",
    "reason": "cart_reservation",
    "changed_by": "uuid",
    "ticket_id": "uuid|null",
    "order_id": "uuid|null"
}

`match_change`¶

{
    "audit_type": "match_change",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "change_type": "UPDATE|CANCEL|RESCHEDULE|PUBLISH|CLOSE",
    "changes": {
        "kick_off_time": { "old": "2026-09-15T18:00:00Z", "new": "2026-09-15T20:00:00Z" }
    },
    "changed_by": "uuid",
    "notification_sent": true
}

`transfer`¶

{
    "audit_type": "transfer",
    "ticket_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "sector_name": "East Stand",
    "seat_label": "Row 5, Seat 12",
    "from_holder_name": "Ivan Horvat",
    "from_holder_oib_masked": "***********1234",
    "to_holder_name": "Marko Marić",
    "to_holder_oib_masked": "***********5678",
    "to_holder_email": "marko@example.com",
    "transfer_type": "SELF_SERVICE|SUPPORT",
    "initiated_by": "uuid",
    "reason": "Gift to friend"
}

`emergency_print`¶

{
    "audit_type": "emergency_print",
    "ticket_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "sector_name": "East Stand",
    "seat_label": "Row 5, Seat 12",
    "holder_name": "Ivan Horvat",
    "agent_id": "uuid",
    "reason": "Lost mobile phone",
    "identity_verified": true,
    "document_type": "ID_CARD",
    "location": "Gate A",
    "is_duplicate": false
}

`blacklist_change`¶

{
    "audit_type": "blacklist_change",
    "blacklist_id": "uuid",
    "oib_masked": "***********1234",
    "person_name": "Ivan Horvat",
    "action": "CREATE|UPDATE|REMOVE|RESTORE",
    "reason": "Stadium ban — violent behaviour",
    "ban_start": "2026-01-01",
    "ban_end": "2028-01-01",
    "changed_by": "uuid",
    "changes": { "field": { "old": "...", "new": "..." } }
}

`violation`¶

{
    "audit_type": "violation",
    "oib_masked": "***********1234",
    "person_name": "Ivan Horvat",
    "blacklist_id": "uuid",
    "action_type": "PURCHASE_ATTEMPT|TRANSFER_ATTEMPT",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "user_id": "uuid",
    "user_email": "ivan@example.com",
    "session_id": "abc123",
    "ip_address": "203.0.113.42"
}

`blacklist_import`¶

{
    "audit_type": "blacklist_import",
    "file_name": "blacklist-2026-03.csv",
    "total_rows": 150,
    "successful_rows": 142,
    "failed_rows": 5,
    "duplicate_rows": 3,
    "imported_by": "uuid",
    "imported_by_email": "admin@hns.hr"
}

`blacklist_cancellation`¶

{
    "audit_type": "blacklist_cancellation",
    "blacklist_id": "uuid",
    "oib_masked": "***********1234",
    "person_name": "Ivan Horvat",
    "cancelled_ticket_count": 3,
    "cancelled_order_count": 1,
    "affected_matches": ["Croatia vs England", "Croatia vs Germany"],
    "cancelled_by": "uuid"
}

`email_delivery`¶

{
    "audit_type": "email_delivery",
    "template_key": "order_confirmation",
    "template_name": "Order Confirmation",
    "recipient_email": "user@example.com",
    "subject": "Your tickets for Croatia vs. England",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "provider": "mailgun",
    "provider_message_id": "msg-123",
    "status": "SENT|DELIVERED|BOUNCED|FAILED",
    "error_message": null
}

`push_delivery`¶

{
    "audit_type": "push_delivery",
    "user_id": "uuid",
    "user_email": "user@example.com",
    "notification_type": "queue_position_update",
    "title": "Your turn is coming!",
    "priority": "HIGH",
    "fcm_message_id": "fcm-456",
    "status": "SENT|DELIVERED|FAILED"
}

`barcode_export`¶

{
    "audit_type": "barcode_export",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "file_name": "barcodes-croatia-england-2026-09-15.csv",
    "total_barcodes": 12500,
    "extra_barcodes": 50,
    "exported_by": "uuid",
    "exported_by_email": "admin@hns.hr"
}

`attendance_import`¶

{
    "audit_type": "attendance_import",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "file_name": "attendance-scan-2026-09-15.csv",
    "total_scanned": 11800,
    "matched_count": 11750,
    "unmatched_count": 50,
    "imported_by": "uuid",
    "imported_by_email": "admin@hns.hr"
}

`report_export`¶

{
    "audit_type": "report_export",
    "report_type": "sales_summary|financial|attendance",
    "report_name": "Sales Summary — Croatia vs England",
    "match_id": "uuid|null",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "parameters": { "date_from": "2026-09-01", "date_to": "2026-09-30" },
    "requested_by": "uuid",
    "requested_by_email": "admin@hns.hr",
    "status": "COMPLETED|FAILED"
}

Admin & Security Events¶

`admin_login`¶

{
    "audit_type": "admin_login",
    "user_id": "uuid",
    "email": "admin@hns.hr",
    "ip_address": "203.0.113.42",
    "roles": ["ROLE_SUPER_ADMIN"]
}

`user_management`¶

{
    "audit_type": "user_management",
    "action": "create|update_roles|deactivate|reactivate|reset_password",
    "target_user_id": "uuid",
    "target_email": "user@hns.hr",
    "old_roles": ["ROLE_REPORTS_VIEWER"],
    "new_roles": ["ROLE_MATCH_MANAGER"],
    "reason": "Promoted to match operations"
}

`support_refund`¶

{
    "audit_type": "support_refund",
    "order_id": "uuid",
    "refund_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "refund_amount": "120.00",
    "currency": "EUR",
    "reason": "Customer unable to attend",
    "ticket_count": 2,
    "requested_by": "uuid",
    "requested_by_email": "support@hns.hr"
}

Business Flow Events (planned)¶

`order_created`¶

{
    "audit_type": "order_created",
    "order_id": "uuid",
    "user_id": "uuid",
    "user_email": "fan@example.com",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "ticket_count": 2,
    "total_amount": "60.00",
    "currency": "EUR",
    "sales_phase": "GENERAL",
    "channel": "online|box_office|partner"
}

`order_paid`¶

{
    "audit_type": "order_paid",
    "order_id": "uuid",
    "user_id": "uuid",
    "user_email": "fan@example.com",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "total_amount": "60.00",
    "currency": "EUR",
    "payment_method": "stripe",
    "stripe_payment_intent_id": "pi_xxx"
}

`order_cancelled`¶

{
    "audit_type": "order_cancelled",
    "order_id": "uuid",
    "user_id": "uuid",
    "user_email": "fan@example.com",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "total_amount": "60.00",
    "ticket_count": 2,
    "cancellation_reason": "refund_request|cart_timeout|admin_cancel|blacklist",
    "cancelled_by": "uuid"
}

`payment_event`¶

{
    "audit_type": "payment_event",
    "event_type": "intent_created|intent_succeeded|intent_failed|refund_initiated|refund_completed|webhook_received",
    "order_id": "uuid",
    "user_email": "fan@example.com",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "amount": "60.00",
    "currency": "EUR",
    "stripe_event_id": "evt_xxx",
    "error_message": null
}

`sales_phase_change`¶

{
    "audit_type": "sales_phase_change",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "phase_id": "uuid",
    "phase_type": "LOYALTY|PRESALE|GENERAL|LAST_MINUTE",
    "action": "opened|closed|modified",
    "start_time": "2026-09-01T10:00:00Z",
    "end_time": "2026-09-14T23:59:00Z",
    "changed_by": "uuid"
}

Operational Visibility Events (planned)¶

`queue_event`¶

{
    "audit_type": "queue_event",
    "event_type": "joined|activated|expired|left",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "user_id": "uuid",
    "user_email": "fan@example.com",
    "queue_position": 1523,
    "waited_seconds": 300,
    "sales_phase": "GENERAL"
}

`quota_change`¶

{
    "audit_type": "quota_change",
    "action": "created|assigned|claimed|closed|cancelled|subquota_delegated",
    "quota_id": "uuid",
    "match_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "organization_name": "UEFA Delegation",
    "seat_count": 50,
    "claimed_count": 45,
    "changed_by": "uuid"
}

`ticket_cancellation`¶

{
    "audit_type": "ticket_cancellation",
    "ticket_id": "uuid",
    "order_id": "uuid",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "sector_name": "East Stand",
    "seat_label": "Row 5, Seat 12",
    "holder_name": "Ivan Horvat",
    "cancellation_reason": "refund|blacklist|admin|match_cancelled",
    "cancelled_by": "uuid"
}

`cart_event`¶

{
    "audit_type": "cart_event",
    "event_type": "created|item_added|item_removed|expired|checkout_started",
    "session_id": "abc123",
    "user_id": "uuid",
    "user_email": "fan@example.com",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "item_count": 2,
    "total_amount": "60.00"
}

Configuration Audit Events (planned)¶

`loyalty_event`¶

{
    "audit_type": "loyalty_event",
    "event_type": "points_awarded|tier_changed",
    "user_id": "uuid",
    "user_email": "fan@example.com",
    "match_name": "Croatia vs England, 2026-09-15 Maksimir",
    "points_awarded": 10,
    "new_balance": 150,
    "new_tier": "SILVER",
    "old_tier": "BRONZE"
}

`config_change`¶

{
    "audit_type": "config_change",
    "entity_type": "stadium|sector|seat_map|price_category|snake_config",
    "entity_id": "uuid",
    "entity_name": "Maksimir Stadium — East Stand",
    "action": "created|updated|deleted",
    "changes": {
        "capacity": { "old": 5000, "new": 5200 }
    },
    "changed_by": "uuid"
}

PII Handling¶

Audit logs contain sensitive data. The following rules apply:

Data Type	Handling	Example
OIB	Masked to last 4 digits	`***********1234`
Names	Retained (required for audit)	`Ivan Horvat`
Email addresses	Retained (required for audit)	`ivan@example.com`
Passport numbers	Never logged	—
IP addresses	Retained for violation and login logs only	`203.0.113.42`
Date of birth	Not included in audit events	—
Passwords	Never logged	—

Access Control¶

Grafana RBAC restricts audit dashboard access to authorized roles only
Admin Portal users see audit data through the backend's Audit Log API (with filtering and CSV export)
Direct Loki API access restricted to infrastructure team via network policies

Retention Policies¶

Retention is enforced by Loki's compactor based on the audit_type label:

Category	Audit Types	Retention	Rationale
Short-lived delivery	`email_delivery`, `push_delivery`, `cart_event`	90 days	Delivery status and cart activity only needed for recent troubleshooting
Report & queue tracking	`report_export`, `admin_login`, `queue_event`	1 year	Operational tracking
Seat & loyalty	`seat_change`, `loyalty_event`	2 years	Operational audit trail
Core business	`match_change`, `transfer`, `barcode_export`, `attendance_import`, `blacklist_import`, `order_created`, `order_paid`, `order_cancelled`, `payment_event`, `sales_phase_change`, `quota_change`, `ticket_cancellation`, `user_management`, `support_refund`, `config_change`	3 years	Compliance and dispute resolution
Emergency operations	`emergency_print`	5 years	Physical ticket printing requires extended audit
Security & violations	`blacklist_change`, `violation`, `blacklist_cancellation`	7 years	Legal and security compliance

# Loki retention configuration (per-tenant or per-stream)
limits_config:
    retention_period: 2160h  # 90 days default

retention_config:
    enabled: true
    retention_period: 2160h  # 90 days default
    retention_stream:
        - selector: '{audit_type=~"email_delivery|push_delivery|cart_event"}'
          period: 2160h   # 90 days
        - selector: '{audit_type=~"report_export|admin_login|queue_event"}'
          period: 8760h   # 1 year
        - selector: '{audit_type=~"seat_change|loyalty_event"}'
          period: 17520h  # 2 years
        - selector: '{audit_type=~"match_change|transfer|barcode_export|attendance_import|blacklist_import|order_created|order_paid|order_cancelled|payment_event|sales_phase_change|quota_change|ticket_cancellation|user_management|support_refund|config_change"}'
          period: 26280h  # 3 years
        - selector: '{audit_type="emergency_print"}'
          period: 43800h  # 5 years
        - selector: '{audit_type=~"blacklist_change|violation|blacklist_cancellation"}'
          period: 61320h  # 7 years

Ingestion Architecture¶

Ticketing Backend → Alloy → Loki¶

The single ingestion path for all audit events:

┌─────────────────────────────────────────────────────────────────┐
│                  Ticketing Backend (Symfony)                      │
│                                                                  │
│   Controller / Service                                           │
│        │                                                         │
│        ▼                                                         │
│   AuditLogger::log($type, $data)                                │
│   • Sanitizes PII (masks OIBs, strips passports)                │
│   • Adds: audit_type, source, performed_by, timestamp           │
│        │                                                         │
│        ▼                                                         │
│   Monolog 'audit' channel (JSON formatter)                       │
│        │                                                         │
│        ▼                                                         │
│   var/log/audit.log (one JSON object per line)                   │
└─────────────────────────┬───────────────────────────────────────┘
                          │
                   ┌──────▼───────────────────────────┐
                   │  Grafana Alloy (sidecar container) │
                   │  • File tail: /var/log/app/audit.log │
                   │  • Sync period: 5 seconds           │
                   │  • JSON parse: level, channel,       │
                   │    audit_type from context            │
                   │  • Static labels: server, service     │
                   │  • Batch: 3s wait, 512KiB max        │
                   │  • Retry: 1s → 30s, max 5 retries    │
                   │  • Basic auth to Loki                 │
                   └──────────────┬───────────────────────┘
                                  │
                                  ▼
                         Loki HTTP Push API
                     POST /loki/api/v1/push

Key Configuration Files¶

File	Purpose
`hns-ticketing-backend/src/Service/AuditLogger.php`	Core logging service with PII sanitization
`hns-ticketing-backend/config/packages/monolog.yaml`	Monolog audit channel (JSON to file)
`hns-ticketing-backend/docker/alloy/config.alloy`	Alloy pipeline (tail → parse → push)
`hns-ticketing-grafana/docker/loki/s3-config.yaml`	Loki retention policies
`hns-ticketing-grafana/docker/grafana/provisioning/dashboards/`	Pre-configured dashboards

Grafana Dashboards¶

Currently Implemented¶

Access & Error Logs — HTTP request monitoring, PHP errors, Nginx errors
Audit Trail — All audit events with filters by service and audit_type

Planned Dashboards¶

3. Security & Violations Dashboard¶

Purpose: Real-time monitoring of blacklist violations and security events.

Panel	Query (LogQL)	Visualization
Violation rate (24h)	`count_over_time({audit_type="violation"}[1h])`	Time series
Violations by action type	`sum by (action_type) (count_over_time({audit_type="violation"}[24h]))`	Pie chart
Recent violations	`{audit_type="violation"} \\| json \\| line_format "{{.person_name}} - {{.action_type}} - {{.match_name}}"`	Logs panel
Blacklist changes	`{audit_type="blacklist_change"} \\| json`	Logs panel
Auto-cancellations	`{audit_type="blacklist_cancellation"} \\| json`	Table

Alerts: - Violation rate > 10/hour → Slack notification - Blacklist removal → Email to security team

4. Support Operations Dashboard¶

Purpose: Support team visibility into transfers, prints, and refunds.

Panel	Query	Visualization
Transfers today	`count_over_time({audit_type="transfer"}[24h])`	Stat
Refunds today	`count_over_time({audit_type="support_refund"}[24h])`	Stat
Emergency prints	`{audit_type="emergency_print"} \\| json`	Table
Transfers by type	`sum by (transfer_type) (count_over_time({audit_type="transfer"}[7d]))`	Bar chart

5. System Health Dashboard¶

Purpose: Monitor notification delivery and audit pipeline health.

Panel	Query	Visualization
Email delivery rate	`sum by (status) (count_over_time({audit_type="email_delivery"}[1h]))`	Stacked time series
Push delivery failures	`{audit_type="push_delivery"} \\| json \\| status="FAILED"`	Logs panel
Audit ingestion rate	`sum(rate({service="ticketing-backend"}[5m]))`	Gauge
Errors by audit type	`sum by (audit_type) (count_over_time({level="error"}[1h]))`	Table

6. Match Day Operations Dashboard¶

Purpose: Live monitoring during match days.

Panel	Query	Visualization
Seat changes (live)	`{audit_type="seat_change"} \\| json \\| match_id="<variable>"`	Logs panel
Barcode exports	`{audit_type="barcode_export"} \\| json \\| match_id="<variable>"`	Table
Attendance imports	`{audit_type="attendance_import"} \\| json`	Table
Match config changes	`{audit_type="match_change"} \\| json \\| match_id="<variable>"`	Logs panel

7. Business Operations Dashboard¶

Purpose: PO-level visibility into sales, orders, and revenue.

Panel	Query	Visualization
Orders today	`count_over_time({audit_type="order_created"}[24h])`	Stat
Revenue today	`{audit_type="order_paid"} \\| json \\| sum by () (total_amount)`	Stat
Payment failures	`{audit_type="payment_event"} \\| json \\| event_type="intent_failed"`	Table
Queue activity	`sum by (event_type) (count_over_time({audit_type="queue_event"}[1h]))`	Stacked time series
Cart conversion	`count_over_time({audit_type="cart_event"}[24h])`	Time series
Active sales phases	`{audit_type="sales_phase_change"} \\| json \\| action="opened"`	Table

Deployment Architecture¶

Infrastructure Diagram¶

┌─────────────────────────────────────────────────────────────────────────────┐
│                           Production Environment                              │
│                                                                              │
│   ┌──────────────────────────────────┐                                      │
│   │  Ticketing Backend               │                                      │
│   │  + Grafana Alloy (sidecar)       │                                      │
│   └──────────────────┬───────────────┘                                      │
│                      │                                                       │
│           ┌──────────▼───────────┐                                          │
│           │   Loki               │                                          │
│           │   + Nginx gateway    │                                          │
│           │     (basic auth)     │                                          │
│           └──────────┬───────────┘                                          │
│                      │                                                       │
│           ┌──────────▼───────────┐                                          │
│           │  Object Storage      │                                          │
│           │  (S3 / GCS / MinIO)  │                                          │
│           └──────────────────────┘                                          │
│                                                                              │
│           ┌──────────────────────┐                                          │
│           │  Grafana             │                                          │
│           │  (dashboards + RBAC) │                                          │
│           └──────────────────────┘                                          │
└─────────────────────────────────────────────────────────────────────────────┘

Environment Configuration¶

Environment	Loki Mode	Storage
Development	Single binary (monolithic)	Local filesystem
Staging	Single binary	MinIO (S3-compatible)
Production	Simple Scalable (read/write/backend)	Cloud object storage (S3/GCS)

Implementation Status¶

Category	Types	Status
Operational Events (13)	seat_change, match_change, transfer, emergency_print, blacklist_change, violation, blacklist_import, blacklist_cancellation, email_delivery, push_delivery, barcode_export, attendance_import, report_export	Implemented
Admin & Security (3)	admin_login, user_management, support_refund	Implemented
Business Flow (5)	order_created, order_paid, order_cancelled, payment_event, sales_phase_change	Planned — high priority
Operational Visibility (4)	queue_event, quota_change, ticket_cancellation, cart_event	Planned — medium priority
Configuration Audit (2)	loyalty_event, config_change	Planned — lower priority

Architecture Overview — System architecture and data layer
Data Model — Complete schema reference (audit tables annotated with Loki storage)

Last Updated: March 2026