Skip to content

Prometheus Guide for the Monitoring Stack

By Updated

What You'll Learn

  • How Prometheus scrapes targets and stores time series
  • Where scrape jobs and alert rules are configured
  • How to write practical alert rules
  • How to check that Prometheus and its targets are healthy

Prometheus is the core metrics and alerting engine in this monitoring lab.

What It Does

  • Scrapes metrics from configured targets
  • Stores time-series data
  • Evaluates alert rules
  • Supplies metrics to Grafana

Important Files

  • prometheus/prometheus.yml
  • prometheus/rules/alerts.yml

Main Scrape Targets

  • Prometheus
  • Alertmanager
  • Node Exporter
  • cAdvisor
  • Blackbox Exporter
  • Blackbox HTTP probe jobs

Why This Layout Works

It combines host metrics, container metrics, and endpoint checks in one place. That gives a more complete view than using only one exporter type.

Example Alerts

  • TargetDown
  • HostHighCPU
  • HostHighMemory
  • ContainerHighCPU
  • ContainerHighMemory
  • SyntheticProbeFailed

Practical Improvements

  • Add severity labels such as warning and critical
  • Add for: windows to reduce alert noise
  • Add ownership labels for teams or services
  • Add recording rules for repeated expensive queries

Quick Check

docker compose logs -f prometheus
curl http://localhost:9090/-/healthy
curl http://localhost:9090/api/v1/targets

FAQ

What is Prometheus used for?

Prometheus collects and stores metrics as time-series data. It is commonly used for infrastructure, container, application, and service health monitoring.

Is Prometheus the same as Grafana?

No. Prometheus stores and queries metrics. Grafana visualizes metrics and logs through dashboards and panels.

What is a scrape target?

A scrape target is an endpoint that exposes metrics for Prometheus to collect. Common targets include Node Exporter, cAdvisor, application metrics endpoints, and Blackbox Exporter probes.

When should I add Alertmanager?

Add Alertmanager when you need alert grouping, routing, silencing, deduplication, and notification delivery.

Common Mistakes

  • Adding high-cardinality labels (user IDs, request paths with IDs), which explode the number of time series.
  • Using rate() on gauges, or reading raw counter values instead of their rate.
  • Alert rules without a for: duration, paging on brief spikes.
  • Treating local Prometheus storage as long-term storage instead of using remote write to a durable backend.
  • Scrape intervals so long that rate() windows have too few samples.

Interview Questions

  • Why does Prometheus use a pull model, and when do you need the Pushgateway?
  • What's the difference between a counter, a gauge, and a histogram?
  • What are recording rules for?
  • How do you keep series cardinality under control?

Next

Continue to Node Exporter.