== hermann/hermann-app (prod) ==

[FIRING] critical · Container missing: prod_hermann-app/app
No container of service app (stack prod_hermann-app) is running. Since 2026-10-19 11:57 UTC (13:57 Europe/Berlin).

  Tenant     hermann
  Project    hermann-app
  Env        prod
  Host       hermann-external
  Check      container prod_hermann-app/app
  Value      0 running (exited)
  Threshold  ≥ 1 running
  Started    2026-10-19 11:57 UTC (13:57 Europe/Berlin)

  What to do:
    1. $ docker compose -p prod_hermann-app ps -a app
    2. $ docker compose -p prod_hermann-app logs --tail 200 app
    3. If it exited by mistake, start it again: `docker compose -p prod_hermann-app up -d app` (from the stack's directory).

  Runbook: https://gitea.i.ivo-zilkenat.de/ivo.zilkenat/ops-kit/src/branch/main/README.md#runbook-container

[FIRING] critical · Disk / 91% full
The filesystem holding / is 91% full (7.4 GB free). Since 2026-10-19 11:58 UTC (13:58 Europe/Berlin).

  Tenant     hermann
  Project    hermann-app
  Env        prod
  Host       hermann-external
  Check      disk /
  Value      91% used
  Threshold  80%
  Started    2026-10-19 11:58 UTC (13:58 Europe/Berlin)

  What to do:
    1. $ du -xh --max-depth=2 / | sort -h | tail -20
    2. $ docker system df -v

  Runbook: https://gitea.i.ivo-zilkenat.de/ivo.zilkenat/ops-kit/src/branch/main/README.md#runbook-disk

[RESOLVED] critical · convex returns HTTP 502
Resolved after 15 min: back to normal.

Grafana: hermann dashboard (hermann-app): https://grafana.i.ivo-zilkenat.de/d/ops-tenant-hermann?var-project=hermann-app&var-env=prod&var-host=hermann-external

== hermann/glitchtip (prod) ==

[RESTARTED] warning · Container restarted: glitchtip-web-1
Container glitchtip-web-1 (glitchtip/web, id 3f2a9c1b0d4e) restarted: same container id, new start time. At 2026-10-19 11:59 UTC (13:59 Europe/Berlin).

  Tenant     hermann
  Project    glitchtip
  Env        prod
  Host       hermann-external
  Check      container glitchtip/web
  Value      new start 2026-10-19T11:59:30Z, previous start 2026-10-12T08:00:00Z, Docker RestartCount 1
  Restarted  2026-10-19 11:59 UTC (13:59 Europe/Berlin)

  Evidence:
    curl: (7) Failed to connect to localhost port 8000

  What to do:
    1. $ docker logs --tail 200 glitchtip-web-1

  Runbook: https://gitea.i.ivo-zilkenat.de/ivo.zilkenat/ops-kit/src/branch/main/README.md#runbook-restart

Grafana: hermann dashboard (glitchtip): https://grafana.i.ivo-zilkenat.de/d/ops-tenant-hermann?var-project=glitchtip&var-env=prod&var-host=hermann-external

GlitchTip: hermann-app: https://glitchtip.hermann-stb-gmbh.de/hermann/issues/

--
Sent by ops-hostcheck 0.2.0 on hermann-external. Checks run every 60 s; a problem mails after 2 failed runs in a row and once more when it is resolved (no repeats in between).
Rules: container:missing:prod_hermann-app/app, disk:full:/, http:convex, restart:glitchtip-web-1
Mute a rule: add `- {match: "<rule>", until: "YYYY-MM-DD", reason: "..."}` under `mute:` in hostcheck.yaml (globs allowed; a date mutes through the end of that day, Europe/Berlin).
Runbook: https://gitea.i.ivo-zilkenat.de/ivo.zilkenat/ops-kit/src/branch/main/README.md#runbook
