Monitoring & Observability

MemGhost includes a built-in observability stack powered by OpenTelemetry and Grafana. When enabled, it gives you:

AI cost tracking — see exactly how much each feature (chat, classification, hub materialization) costs per user, per model, per day
Usage metrics — vault item counts, storage per user, capture rates
Distributed traces — follow a request from the HTTP handler through command handlers and the event store
System health — request rates, error rates, latency percentiles

Quick Start

Add the monitoring profile when starting your stack:

docker compose --profile standalone --profile monitoring up -d

Add the OTEL endpoint to your .env file so the API sends telemetry:

OTEL_EXPORTER_OTLP_ENDPOINT=otel:4317

Open Grafana at http://localhost:3001 (default login: admin / admin).

What Gets Tracked

AI Usage & Costs

Every LLM and embedding call is logged with:

Field	Example
Feature	`vault.classification`, `spaces.chat`, `hubs.materialization`
Model	`claude-haiku-4-5-20251001`, `qwen3:8b`
Provider	`anthropic`, `bedrock`, `ollama`
Model tier	`lite`, `standard`, `advanced`
Input/output tokens	`1860` / `276`
Cost (USD)	`$0.003240`
User	Per-user attribution

For self-hosted instances running local models (Ollama), token counts are tracked for capacity planning even though the dollar cost is zero.

Storage & Users

Per-user counters are maintained in real time:

Vault item count — incremented on every capture
Storage bytes — tracks body content size
Items captured per day — time series from vault events

These counters are backfilled automatically on first startup for existing users.

Distributed Traces

HTTP requests generate OpenTelemetry traces with spans for:

HTTP handler (method, path, status, duration)
Event store operations (save, load)
Command and query handlers

Traces are stored in Grafana Tempo and can be searched by service, operation, duration, or error status.

HTTP Metrics

Standard RED (Rate, Errors, Duration) metrics are exported via OpenTelemetry:

http_requests_total — request count by method and status
http_request_duration_seconds — latency histogram with p50/p95/p99

Dashboards

Four dashboards are provisioned automatically:

Home

The default landing page with at-a-glance stats: total AI cost, request count, vault items, storage, active users, and HTTP request volume. Includes a cost-over-time chart and links to the detail dashboards.

AI Usage & Costs

Deep dive into AI spending:

Stat panels: total cost, requests, input/output tokens
Pie charts: cost by feature, cost by model, requests by tier
Time series: daily cost and token usage (stacked by feature)
Table: top cost drivers ranked by total spend

Storage & Users

User-level resource tracking:

Stat panels: total items, total storage, active users, average cost per user
Tables: per-user storage and per-user AI cost with username
Time series: items captured per day

API Overview

System health from OpenTelemetry metrics:

Request rate (req/s)
Error rate (%)
Latency percentiles (p50/p95/p99)
Request distribution by HTTP method

Configuration

Environment Variables

Variable	Default	Description
`OTEL_EXPORTER_OTLP_ENDPOINT`	(empty)	OTLP gRPC endpoint. Set to `otel:4317` to enable telemetry. Empty disables OTEL.
`OTEL_SERVICE_NAME`	`memghost-api`	Service name in traces and metrics
`GRAFANA_PORT`	`3001`	Host port for Grafana UI
`GRAFANA_USER`	`admin`	Grafana admin username
`GRAFANA_PASSWORD`	`admin`	Grafana admin password

Ports

Service	Container Port	Host Port	Protocol
Grafana	3000	3001	HTTP
OTLP gRPC	4317	4317	gRPC
OTLP HTTP	4318	4318	HTTP

Data Persistence

Grafana data (dashboards, preferences) is stored in the grafana-data Docker volume. Trace and metric data is stored inside the OTEL container’s /data volume.

Architecture

The monitoring stack runs as a single container (grafana/otel-lgtm) that bundles:

Grafana — dashboards and visualization
Tempo — trace storage (TraceQL)
Loki — log aggregation (LogQL)
Mimir — metrics storage (PromQL)
OpenTelemetry Collector — receives OTLP telemetry from the API

Go API (OTEL SDK) ──► OTLP (4317) ──► otel-lgtm container
                                          ├── Tempo (traces)
                                          ├── Loki (logs)
                                          ├── Mimir (metrics)
                                          └── Grafana UI (3001)
                                                │
                           PostgreSQL ◄─────────┘
                           (metering tables)

The API’s metering tables (ai_usage_log, ai_usage_daily, user_usage) are queried directly by Grafana via a provisioned PostgreSQL datasource. This gives the dashboards access to business metrics (costs, user storage) alongside operational metrics (traces, latency).

CLI Tool

The obs CLI queries observability data from the terminal. Build it with task obs:build.

bin/obs health                          # Check service connectivity
bin/obs cost                            # Recent AI usage with per-call costs
bin/obs cost --from 2026-05-01          # Filter by date range
bin/obs drivers                         # Top cost drivers (feature + model)
bin/obs storage                         # Per-user vault item counts and bytes
bin/obs traces --limit 10               # Recent traces from Tempo
bin/obs trace <trace-id>                # Full span tree with timing
bin/obs logs --level error --since 1h   # Search Loki logs

In development, the tool connects to localhost ports automatically and works out of the box.

Production & Staging Access

For hosted deployments, the observability ports are not publicly exposed. Use the tunnel tasks to connect:

# Production
task otel:prod:up       # Connect (SSH tunnel runs in background)
bin/obs health          # Works with defaults (localhost:9091, localhost:3001)
bin/obs cost            # Check AI spending
open http://localhost:3001  # Grafana dashboards
task otel:prod:down     # Disconnect

# Staging (uses different local ports to avoid conflicts)
task otel:staging:up
OBS_API_URL=http://localhost:9092 OBS_GRAFANA_URL=http://localhost:3002 bin/obs health
open http://localhost:3002
task otel:staging:down

The tunnel requires AWS credentials with EC2 Instance Connect permissions. Grafana and the obs API are accessible from other machines on your network (bound to 0.0.0.0).

Disabling Monitoring

If you don’t want monitoring, simply don’t include the monitoring profile and leave OTEL_EXPORTER_OTLP_ENDPOINT empty. The API gracefully skips all telemetry when no endpoint is configured — no performance impact.