Metrics in Prometheus, dashboards in Grafana, logs in Loki, distributed traces in Tempo, alerts through Alertmanager — all self-hosted alongside the platform they watch. Monitoring is the part of a project that gets cut when the budget tightens, and it is the part whose absence costs the most.
| Measure | Value | Source | Measured |
|---|---|---|---|
| Observability components running | 5 (Prometheus, Grafana, Loki, Tempo, Alertmanager) | kubectl get deployments,statefulsets -n monitoring | |
| Services under metric collection | 65 deployments in the platform namespace | kubectl get deployments -n kamo |
What building it taught us
Almost every system we are asked to take over has the same gap. There is no way to tell whether it is working except by using it. Nobody knows what normal looks like, so nobody can tell when it stops. The first sign of trouble is a customer complaint, which means the clock started well before anyone looked.
Distributed tracing is the piece most often missing and most often decisive. In a system of forty services, 'the page is slow' is not a diagnosis — it is the beginning of an argument between teams. A trace ends the argument in about a minute by showing exactly which call took the time. We instrument client systems for this before launch, because retrofitting tracing into a running system is several times the work of building it in.
It also changes what a support conversation is like. When a tenant reports that one widget is broken shortly after a schema change, we can tell from the metrics that connection pools are the cause and that it will clear itself, rather than restarting services and making it worse. Knowing when to do nothing is an operational skill and it depends entirely on being able to see.
Built with
- Prometheus
- Grafana
- Loki
- Tempo
- Alertmanager
- kube-state-metrics
Tell us what you are trying to do
A short conversation is usually enough to tell whether we are the right people. If we are not, we will say so and point you somewhere better.