TASK-012

a monitoring and alerting system for the forge, capable of sending alert

review

Goal

Deploy a Prometheus + Grafana observability stack on k3s so the operator has portfolio-grade dashboards and declarative alerts when steady-state deployments (Argo CD Applications and their workloads) are unhealthy or the single-node host shows resource pressure. Notifications route to ntfy and Slack via Alertmanager — without modifying host-watch, UFW, or factory task-thread Slack routing (TASK-011).

Acceptance criteria

kube-prometheus-stack (Prometheus Operator, Prometheus, Grafana, Alertmanager, kube-state-metrics, node-exporter) deployed via Argo CD Helm Application in a dedicated monitoring namespace; worker does not kubectl apply Argo-managed apps
Grafana reachable on HTTPS Ingress (TLS via cert-manager) with admin credentials from Vault via ExternalSecret; at least three provisioned dashboards documented for portfolio use (cluster overview, node resources, workload health)
PrometheusRule resources in git encode alert conditions A1–A8 (see plan); firing alerts reach ntfy and a dedicated Slack ops channel through Alertmanager receivers
ExternalSecret projects secret/forge/ntfy (existing) and secret/forge/alerts/slack webhook_url into monitoring; no secrets, webhook URLs, or Slack user IDs in git
Minimal shell CronJob forge-node-alert in k8s/platform/metrics/alerts.yaml removed or superseded by Prometheus rules; no parallel custom alert checker
k8s/platform/metrics/README.md documents Vault bootstrap, Grafana URL, dashboard tour, and operator smoke-test procedure (including safe alert trigger)
kubectl kustomize k8s/platform/metrics (and helm values lint/template in CI) passes review; factory/review/CHECKLIST.md completed on the PR
After human merge to main, Argo CD syncs metrics overlay and monitoring Application only (ADR-008)

Assignee: worker-localpower
Branch: factory/task-012-a-monitoring-and-alerting-system-for-the
Sandbox: agent-cell
Risk: medium

Artifacts

pr artifacts/TASK-012-pr.txt
log /media/diestrin/data/forge/factory/artifacts/logs/TASK-012.log
other /media/diestrin/data/forge/factory/artifacts/TASK-012-git.diff

Agent runs

finishedimplement9/1/2026, 5:17:51 AM· composer-2.5· 925 eventsTranscript →
finishedplan9/1/2026, 5:10:44 AM· composer-2.5· 475 eventsTranscript →
finishedplan9/1/2026, 4:41:49 AM· composer-2.5· 1170 eventsTranscript →

Message history

slack· operator· 9/1/2026, 4:33:07 AM
a monitoring and alerting system for the forge, capable of sending alerts via ntfy and slack when deployments are failing or the host resources are going to dangerous levels
orchestrator· plan-worker· 9/1/2026, 4:33:42 AM
Plan PR: https://github.com/diestrin/homelab-forge/pull/27
notify· control-plane· 9/1/2026, 4:33:43 AM
*TASK-012* plan opened (`planning`) • PR: https://github.com/diestrin/homelab-forge/pull/27 • Reply in this thread to revise. • Approve with `approve` / `lgtm` when ready for the worker.
system· watch-checks· 9/1/2026, 4:35:07 AM
CI green on https://github.com/diestrin/homelab-forge/pull/27
notify· control-plane· 9/1/2026, 4:35:09 AM
✅ CI green on https://github.com/diestrin/homelab-forge/pull/27 for `TASK-012`. Next step: human review + merge (ADR-008 — the factory never merges).
slack· operator· 9/1/2026, 4:36:41 AM
update the plan with the implementation details, I’m not sure what I’m approving here
orchestrator· plan-worker· 9/1/2026, 4:37:08 AM
Plan job failed: planner exited rc=1
notify· control-plane· 9/1/2026, 4:37:09 AM
Plan job for `TASK-012` failed: planner exited rc=1
orchestrator· plan-worker· 9/1/2026, 4:44:44 AM
Plan job failed: planner blocked: lint failures remain after fix attempts
notify· control-plane· 9/1/2026, 4:44:45 AM
Plan job for `TASK-012` failed: planner blocked: lint failures remain after fix attempts
orchestrator· plan-worker· 9/1/2026, 4:54:52 AM
Plan updated with implementation details after Slack feedback (manual recover after planner lint/host fix).
notify· control-plane· 9/1/2026, 4:54:54 AM
*TASK-012* plan updated (`planning`) • PR: https://github.com/diestrin/homelab-forge/pull/27 • Reply in this thread to revise. • Approve with `approve` / `lgtm` when ready for the worker. _Note: first Slack revise failed (worker python lacked cursor-sdk); host fix applied and plan rewritten with implementation details._
slack· operator· 9/1/2026, 5:10:28 AM
Following the principle that I want to use this project as a portfolio project, I’d rather implement a monitoring system like prometheus+grafana to be able to get some monitoring dashboards to show, and to configure alerts from there
orchestrator· plan-worker· 9/1/2026, 5:11:55 AM
Plan PR: https://github.com/diestrin/homelab-forge/pull/27
notify· control-plane· 9/1/2026, 5:11:56 AM
*TASK-012* plan updated (`planning`) • PR: https://github.com/diestrin/homelab-forge/pull/27 • Reply in this thread to revise. • Approve with `approve` / `lgtm` when ready for the worker.
system· watch-checks· 9/1/2026, 5:13:16 AM
CI green on https://github.com/diestrin/homelab-forge/pull/27
notify· control-plane· 9/1/2026, 5:13:18 AM
✅ CI green on https://github.com/diestrin/homelab-forge/pull/27 for `TASK-012`. Next step: human review + merge (ADR-008 — the factory never merges).
system· slack· 9/1/2026, 5:17:40 AM
Task approved → proposed; implement job enqueued.
notify· control-plane· 9/1/2026, 5:17:40 AM
`TASK-012` → *proposed* (worker-claimable). PR: https://github.com/diestrin/homelab-forge/pull/27 Worker will implement via Cursor SDK on the pinned branch and update the same PR.
slack· operator· 9/1/2026, 5:41:03 AM
Is there any new adr needed for this PR?
notify· control-plane· 9/1/2026, 5:41:05 AM
Noted on `TASK-012` (status: review) — feedback stored on the task. No new plan PR is opened after approve; CI is watched automatically. Use `/forge plan …` for new work.
system· watch-checks· 9/1/2026, 5:47:51 AM
CI green on https://github.com/diestrin/homelab-forge/pull/27
notify· control-plane· 9/1/2026, 5:47:51 AM
✅ CI green on https://github.com/diestrin/homelab-forge/pull/27 for `TASK-012`. Next step: human review + merge (ADR-008 — the factory never merges).