Monitoring Stack Overview: Prometheus, Grafana, Loki, and Alertmanager¶

By Sameer Alam · Updated 24 September 2026

What You'll Learn¶

• What each component in the stack does and how the signals connect
• The recommended order for learning and running the lab
• Where each page fits, from fundamentals to troubleshooting

This section explains a practical observability stack built with Prometheus, Grafana, Alertmanager, Loki, Grafana Alloy, OpenTelemetry, Blackbox Exporter, Node Exporter, and application instrumentation.

What This Stack Covers¶

• Metrics with Prometheus
• Dashboards with Grafana
• Alert routing with Alertmanager
• Logs with Loki and Grafana Alloy (Promtail's supported successor)
• Events and traces with OpenTelemetry
• Synthetic checks with Blackbox Exporter

Why This Setup Is Useful¶

Many examples stop at metrics only. This stack is more practical because it combines the main observability layers in one place:

• Metrics
• Logs
• Events and traces
• Alerts
• Endpoint reachability checks

That makes it a solid learning setup for DevOps and SRE work.

Recommended Flow¶

1. Learn the observability signals and how they answer different questions.
2. Start the lab and verify the services.
3. Review Prometheus, Node Exporter, and Grafana.
4. Add alerting, structured application logs, traces, and blackbox checks.
5. Use the troubleshooting page when a service does not behave as expected.

Pages in This Section¶

• Observability fundamentals: logs, events, metrics, and traces
• Stack walkthrough
• Prometheus
• Node Exporter
• Grafana
• Alertmanager
• Logging with Loki and Grafana Alloy
• Python logging in practice
• OpenTelemetry and other platforms
• Blackbox Exporter
• Troubleshooting

Common Mistakes¶

• Building dashboards but no alerts, so problems are only found by someone looking at the right screen.
• Alerting on causes (CPU is high) instead of symptoms users feel (errors, latency).
• Using high-cardinality values such as user IDs as metric or log labels.
• Not planning retention and storage, then losing history exactly when an incident needs it.
• Not monitoring the monitoring stack itself.

Interview Questions¶

• What question does each signal — metrics, logs, traces — answer best?
• Why does Prometheus pull metrics instead of having services push them?
• Why is Alertmanager a separate component from Prometheus?

Further Reading¶

• Prometheus Documentation
• Grafana Documentation
• OpenTelemetry Documentation

Next¶

Continue to Observability Fundamentals. To apply these tools inside a cluster, see Kubernetes Observability & Health.