Goal

Turn a DB problem into an AI-triaged incident that names the exact PID to kill.

1. Alert rules (Prometheus)

From bonito.rules.yml — the critical one is BonitoBlocking:

- alert: BonitoBlocking
  expr: bonito_blocking_sessions > 0
  for: 1m
  labels: { severity: critical }

2. Alertmanager receiver

receivers:
  - name: remo
    webhook_configs:
      - url: https://api.remo.sofe.dev/ingest
        send_resolved: false
        http_config:
          http_headers:
            X-Webhook-Secret:
              values: [whsec_bonito_demo_001]

3. Trigger a real blocking

# holder — row lock on orders.id=1 for 10 min
docker exec -d bonito-collector python -c "import psycopg,time; c=psycopg.connect('postgresql://postgres:postgres@dbs-postgres:5432/app'); c.autocommit=False; c.execute('UPDATE orders SET status=%s WHERE id=1',('held',)); time.sleep(600)"

# blocked updater
docker exec -d bonito-collector python -c "import psycopg; c=psycopg.connect('postgresql://postgres:postgres@dbs-postgres:5432/app'); c.autocommit=True; c.execute('UPDATE orders SET status=%s WHERE id=1',('stuck',))"

4. Watch it flow

# metric appears
curl -s "http://localhost:9090/api/v1/query?query=bonito_blocking_sessions" | jq .data.result

# ~1-2 min later the incident exists with enrichment + PID
curl -s -H "X-Remo-Key: rk_bonito_demo_001" "https://api.remo.sofe.dev/incidents?limit=3" | jq

The triage_log shows ingested → enriched → diagnosed, and the diagnosis contains the blocker PID and remediation:

1. SELECT pid, state, query FROM pg_stat_activity WHERE pid = 829705;
2. SELECT pg_cancel_backend(829705);
3. SELECT pg_terminate_backend(829705);
4. CREATE INDEX IF NOT EXISTS idx_orders_id ON orders(id);

Gotchas

• Prometheus needs alerting.alertmanagers (or nothing is sent).
• rule_files glob must be scoped (rules/*.yml), not *.yml next to alertmanager.yml.
• Enrichment failure never blocks triage (3s timeout).