#monitoring #DevOps #observability #Prometheus #Grafana

Prometheus, Grafana and Loki on one server

Константин Потапов

October 14, 2025

16 min

Production died at 3 a.m. The customer email arrived at 9. A stack for one evening: metrics, logs, alerts, no Datadog, no dedicated DevOps.

Prometheus, Grafana and Loki on one server

Production died at 3 a.m. They learned at 9 from a customer email. In 2023 I advised a startup at $2M ARR: no alerts, monitoring was SSH and tail -f. The disk filled, the database stopped, six hours of work and $15k of revenue.

A sane stack stands up in an evening. No Kubernetes, no Datadog at three thousand a month, no dedicated team.

Three cases where blindness already cost money

Fintech, 2024. The API was slowing down, everyone knew "something was wrong", nobody knew what. Prometheus in one evening. One endpoint made 300+ SQL queries per HTTP request. They killed the N+1, 2.5 s became 120 ms.

A Django shop, 2025. Every two or three days production crawled with no pattern. htop was quiet. Grafana on memory: a Celery worker ate 16 GB, the OOM-killer shot it. A leak in image processing. Two hours to fix.

B2B SaaS, 2023. Customers complained about odd slowness. A latency spike every six hours. Loki found pg_dump without nice on the same box. CPU at the ceiling, the app choking. Backups moved to another machine.

Without metrics, the customer or the till tells you about the problem.

Metrics: numbers. CPU, memory, RPS, latency, error rate. Logs: events. Traces: the path of a request. I skip traces here, that is already microservices.

A speedometer, a line "engine overheated at 2:35 p.m.", a dashcam. Without the first two you drive blind and learn about the breakdown when the car has already stopped.

Why this stack

I have tried Zabbix, Nagios, ELK, Datadog, New Relic. For most projects Prometheus, Grafana and Loki are enough.

Free, no vendor lock. Lives on a box with 2 GB RAM. Exporters exist for Postgres, Redis, Nginx, Node. I have run it on a Hetzner CX21 for five dollars.

Criterion	Prometheus Stack	ELK Stack	Datadog/New Relic	Zabbix
Cost	Free	Free	$100-5000/mo	Free
Setup simplicity	1-2 hours	4-8 hours	30 minutes	2-4 hours
Resources (RAM)	1-2GB	8-16GB	SaaS	2-4GB
Metrics	Excellent	Not the focus	Excellent	Good
Logs	Loki	Excellent (ES)	Excellent	Basic
Alerts	Alertmanager	Complex	Excellent	Good
Dashboards	Grafana	Kibana	Pretty	Weaker

Up to ten servers I take this stack. Need Elasticsearch for audit: ELK. Budget exists and nobody wants to fuss: Datadog. Zabbix already there: leave it.

PrometheusGrafanaLokiDockerNode Exporter

┌─────────────────────────────────────────────────┐
│  Your server (2-4GB RAM)                        │
├─────────────────────────────────────────────────┤
│                                                 │
│  ┌─────────────────┐  ┌──────────────┐          │
│  │ Your app        │  │ PostgreSQL   │          │
│  │ (FastAPI/Django)│  │ / Redis      │          │
│  └────┬────────────┘  └────┬─────────┘          │
│       │                    │                    │
│       │ metrics            │ metrics            │
│       │ + logs             │ (exporter)         │
│       ▼                    ▼                    │
│  ┌─────────────────────────────────┐            │
│  │ Prometheus                      │            │
│  │ (collects metrics every 15s)    │            │
│  └───────────┬─────────────────────┘            │
│              │                                  │
│              │ query                            │
│              ▼                                  │
│  ┌─────────────────────────────────┐            │
│  │ Grafana                         │            │
│  │ (visualization + alerts)        │            │
│  └───────────┬─────────────────────┘            │
│              │ query                            │
│  ┌───────────▼─────────────────────┐            │
│  │ Loki                            │            │
│  │ (stores logs)                   │            │
│  └─────────────────────────────────┘            │
│                                                 │
└─────────────────────────────────────────────────┘

Prometheus walks HTTP every 15-30 seconds. Grafana draws. Loki indexes logs. node_exporter, postgres_exporter, redis_exporter, nginx_exporter speak its format.

Minimum: 2 GB RAM, 2 cores, 20 GB disk. Better 4 GB and 50 GB. Ubuntu 22.04/24.04 or Debian 12 with Docker.

Standing it up

mkdir -p /opt/monitoring
cd /opt/monitoring

services:
  # Prometheus: collects metrics
  prometheus:
    image: prom/prometheus:v3.0.0
    container_name: prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d" # Store metrics for 30 days
      - "--web.enable-lifecycle" # API for hot-reload config
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alerts.yml:/etc/prometheus/alerts.yml
      - prometheus_data:/prometheus
    restart: unless-stopped
    networks:
      - monitoring
 
  # Grafana: visualization
  grafana:
    image: grafana/grafana:11.4.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your_secure_password # CHANGE THIS!
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    restart: unless-stopped
    networks:
      - monitoring
 
  # Loki: logs
  loki:
    image: grafana/loki:3.3.2
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki_data:/loki
    restart: unless-stopped
    networks:
      - monitoring
 
  # Node Exporter: server metrics
  node_exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node_exporter
    command:
      - "--path.rootfs=/host"
    ports:
      - "9100:9100"
    volumes:
      - /:/host:ro,rslave
    restart: unless-stopped
    networks:
      - monitoring
 
volumes:
  prometheus_data:
  grafana_data:
  loki_data:
 
networks:
  monitoring:
    driver: bridge

prometheus/prometheus.yml:

global:
  scrape_interval: 15s # Collect metrics every 15 seconds
  evaluation_interval: 15s # Check alert rules every 15 seconds
 
# Alerts (we'll create later)
rule_files:
  - "/etc/prometheus/alerts.yml"
 
# Where to collect metrics from
scrape_configs:
  # Prometheus itself
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
 
  # Server metrics (CPU, RAM, Disk)
  - job_name: "node"
    static_configs:
      - targets: ["node_exporter:9100"]
 
  # Your application (FastAPI, Django, etc.)
  # Uncomment and specify your app address
  # - job_name: 'app'
  #   static_configs:
  #     - targets: ['app:8000']
 
  # PostgreSQL (if using postgres_exporter)
  # - job_name: 'postgres'
  #   static_configs:
  #     - targets: ['postgres_exporter:9187']
 
  # Redis (if using redis_exporter)
  # - job_name: 'redis'
  #   static_configs:
  #     - targets: ['redis_exporter:9121']

Empty prometheus/alerts.yml:

groups:
  - name: basic_alerts
    interval: 30s
    rules: []

docker compose up -d
docker compose ps

Four containers Up: prometheus, grafana, loki, node_exporter.

Prometheus: http://your-server-ip:9090
Grafana: http://your-server-ip:3000, login admin, password from compose
Loki: http://your-server-ip:3100/ready should return ready

In Grafana: Connections → Add data source → Prometheus, URL http://prometheus:9090. Same for Loki: http://loki:3100.

I do not draw a dashboard from scratch. Dashboards → Import → ID 1860 (Node Exporter Full). Postgres: 9628. Redis: 11835. Nginx: 12708. Docker: 893.

App metrics

pip install prometheus-client

from fastapi import FastAPI
from prometheus_client import Counter, Histogram, make_asgi_app
 
app = FastAPI()
 
# Metrics
REQUEST_COUNT = Counter(
    'app_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
 
REQUEST_DURATION = Histogram(
    'app_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint']
)
 
@app.middleware("http")
async def prometheus_middleware(request, call_next):
    method = request.method
    endpoint = request.url.path
 
    with REQUEST_DURATION.labels(method, endpoint).time():
        response = await call_next(request)
 
    REQUEST_COUNT.labels(method, endpoint, response.status_code).inc()
    return response
 
# Endpoint for Prometheus
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)

Metrics at http://your-app:8000/metrics.

pip install django-prometheus

# settings.py
INSTALLED_APPS = [
    'django_prometheus',
    # ...
]
 
MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    # ... other middleware
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]
 
# urls.py
urlpatterns = [
    path('', include('django_prometheus.urls')),
    # ...
]

npm install prom-client

const express = require("express");
const client = require("prom-client");
 
const app = express();
 
// Create registry
const register = new client.Registry();
 
// Collect default metrics (CPU, memory, event loop)
client.collectDefaultMetrics({ register });
 
// Custom metrics
const httpRequestDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "Duration of HTTP requests in seconds",
  labelNames: ["method", "route", "status_code"],
  registers: [register],
});
 
// Middleware for metrics
app.use((req, res, next) => {
  const start = Date.now();
  res.on("finish", () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration
      .labels(req.method, req.route?.path || req.path, res.statusCode)
      .observe(duration);
  });
  next();
});
 
// Endpoint for Prometheus
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});
 
app.listen(3000);

In prometheus.yml:

scrape_configs:
  # ... existing jobs
 
  - job_name: "myapp"
    static_configs:
      - targets: ["host.docker.internal:8000"] # Your application

If the app is in the same Compose file, the target is myapp:8000.

docker compose restart prometheus

Status → Targets: the app should be UP.

Logs

Promtail picks up files and pushes them to Loki.

promtail:
  image: grafana/promtail:3.3.2
  container_name: promtail
  volumes:
    - /var/log:/var/log:ro # System logs
    - ./promtail/config.yml:/etc/promtail/config.yml
    - ./logs:/app/logs:ro # Your app logs
  command: -config.file=/etc/promtail/config.yml
  restart: unless-stopped
  networks:
    - monitoring

server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://loki:3100/loki/api/v1/push
 
scrape_configs:
  # Your application logs
  - job_name: app
    static_configs:
      - targets:
          - localhost
        labels:
          job: app
          __path__: /app/logs/*.log
 
  # System logs (optional)
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          __path__: /var/log/syslog

docker compose up -d

Or straight from Python:

pip install python-logging-loki

import logging
from logging_loki import LokiHandler
 
logger = logging.getLogger("my-app")
logger.setLevel(logging.INFO)
 
loki_handler = LokiHandler(
    url="http://loki:3100/loki/api/v1/push",
    tags={"application": "my-app", "environment": "production"},
    version="1",
)
 
logger.addHandler(loki_handler)
 
logger.info("Application started")
logger.error("Something went wrong", extra={"user_id": 123})

In Grafana: Explore → Loki → {job="app"}.

# All app logs
{job="app"}
 
# Errors only
{job="app"} |= "ERROR"
 
# Logs for specific user
{job="app"} | json | user_id="123"
 
# Error rate for last 5 minutes
rate({job="app"} |= "ERROR" [5m])

Alerts

prometheus/alerts.yml:

groups:
  - name: critical_alerts
    interval: 30s
    rules:
      # Server unreachable
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.job }} has been down for more than 1 minute."
 
      # CPU above 80%
      - alert: HighCPU
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for 5 minutes (current: {{ $value }}%)"
 
      # RAM above 90%
      - alert: HighMemory
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 90% (current: {{ $value }}%)"
 
      # Disk above 85%
      - alert: DiskSpaceLow
        expr: (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs"} / node_filesystem_size_bytes)) * 100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk {{ $labels.mountpoint }} is {{ $value }}% full"
 
      # High error rate (>5% requests with errors)
      - alert: HighErrorRate
        expr: rate(app_requests_total{status=~"5.."}[5m]) / rate(app_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate in {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} for 5 minutes"
 
      # Slow requests (p95 latency > 1s)
      - alert: SlowRequests
        expr: histogram_quantile(0.95, rate(app_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow requests in {{ $labels.job }}"
          description: "95th percentile latency is {{ $value }}s"

curl -X POST http://localhost:9090/-/reload

Check: http://your-server-ip:9090/alerts

Alertmanager puts letters into Telegram, Slack, email.

alertmanager:
  image: prom/alertmanager:v0.27.0
  container_name: alertmanager
  ports:
    - "9093:9093"
  volumes:
    - ./alertmanager/config.yml:/etc/alertmanager/config.yml
    - alertmanager_data:/alertmanager
  command:
    - "--config.file=/etc/alertmanager/config.yml"
  restart: unless-stopped
  networks:
    - monitoring
 
volumes:
  # ... existing volumes
  alertmanager_data:

At the top of prometheus.yml:

# Add at the beginning of the file
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

global:
  resolve_timeout: 5m
 
route:
  group_by: ["alertname", "cluster"]
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: "telegram"
 
receivers:
  # Telegram (recommended)
  - name: "telegram"
    telegram_configs:
      - bot_token: "YOUR_BOT_TOKEN" # Get from @BotFather
        chat_id: YOUR_CHAT_ID # Your chat_id
        parse_mode: "HTML"
        message: |
          <b>{{ .Status | toUpper }}</b>
          {{ range .Alerts }}
          <b>Alert:</b> {{ .Labels.alertname }}
          <b>Severity:</b> {{ .Labels.severity }}
          <b>Summary:</b> {{ .Annotations.summary }}
          <b>Description:</b> {{ .Annotations.description }}
          {{ end }}
 
  # Slack (alternative)
  # - name: 'slack'
  #   slack_configs:
  #     - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
  #       channel: '#alerts'
  #       title: 'Alert: {{ .GroupLabels.alertname }}'
  #       text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
 
  # Email (alternative)
  # - name: 'email'
  #   email_configs:
  #     - to: 'your-email@example.com'
  #       from: 'alerts@yourapp.com'
  #       smarthost: 'smtp.gmail.com:587'
  #       auth_username: 'your-email@gmail.com'
  #       auth_password: 'your-app-password'

docker compose up -d

Bot: @BotFather → /newbot → token. Then /start the bot and open https://api.telegram.org/bot<bot_token>/getUpdates, look for "chat":{"id":123456789}.

CPU check:

# Load CPU
yes > /dev/null &
yes > /dev/null &
yes > /dev/null &
yes > /dev/null &
 
# After 5 minutes HighCPU alert should trigger
# Check: http://your-server-ip:9090/alerts
 
# Stop the load:
killall yes

In five or six minutes it should arrive in Telegram.

Dashboards I actually watch

RPS: rate(app_requests_total[1m]).

Error rate: (rate(app_requests_total{status=~"5.."}[5m]) / rate(app_requests_total[5m])) * 100. Thresholds: 1% warning, 5% critical.

Latency:

histogram_quantile(0.50, rate(app_request_duration_seconds_bucket[5m]))  # p50
histogram_quantile(0.95, rate(app_request_duration_seconds_bucket[5m]))  # p95
histogram_quantile(0.99, rate(app_request_duration_seconds_bucket[5m]))  # p99

Slowest: topk(5, histogram_quantile(0.95, rate(app_request_duration_seconds_bucket[5m]))).

CPU: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100).

Memory: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100.

Disk: rate(node_disk_read_bytes_total[5m]) and rate(node_disk_written_bytes_total[5m]).

Network: rate(node_network_receive_bytes_total[5m]) and rate(node_network_transmit_bytes_total[5m]).

What to do at 3 a.m.

HighCPU: CPU graph, RPS next to it. If traffic spiked, the code is guilty. In Loki {job="app"} | json | line_format "{{.endpoint}} {{.duration}}". In Prometheus topk(5, rate(app_request_duration_seconds_sum[5m])).

HighMemory: process_resident_memory_bytes, logs OutOfMemory / MemoryError. Temporary: docker compose restart app. Then memory_profiler or py-spy.

HighErrorRate: which endpoints return 5xx, in Loki ERROR and stack traces. Database, someone else's API, a timeout.

SlowRequests: p95 by endpoint, specific requests in the logs, pg_stat_statements.

Every alert needs a short runbook. At 3 a.m. memory is worse than at a whiteboard at noon.

What I no longer do

The default 15 days of retention is short. I set 30-90:

command:
  - "--storage.tsdb.retention.time=90d"

# loki-config.yaml
limits_config:
  retention_period: 30d

I do not monitor clicks on buttons. That is analytics. I watch RPS, errors, latency, CPU, RAM, disk.

Fifty letters a minute is worse than silence. I group. Critical calls, warning goes to Telegram, info stays in logs. During a deploy I mute alerts, otherwise false positives.

Configs in Git. Ports 9090 and 3000 do not face the world: Nginx and a password. The default Grafana password dies the same evening. Prometheus can show things it should not.

If you need metrics for years: Thanos, Cortex, or Grafana Cloud (free up to 10k series).

In Kubernetes: Prometheus Operator and ServiceMonitor.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: metrics
      interval: 30s

For microservices people put Tempo next to it:

tempo:
  image: grafana/tempo:latest
  command: ["-config.file=/etc/tempo.yaml"]
  volumes:
    - ./tempo.yaml:/etc/tempo.yaml
  ports:
    - "3200:3200" # Tempo UI
    - "4317:4317" # OTLP gRPC

Then the OpenTelemetry SDK in the app.

What it costs

Hetzner CX31 4 GB: $7. Another 50 GB: $5. Setup once, 4-8 hours. Upkeep: 1-2 hours a month. Total $12-15 and two hours.

Datadog $100-500. New Relic $99-749. Grafana Cloud from zero to $299.

The main saving is not the SaaS invoice. An hour of downtime costs a thousand to ten, depending on the product. Mine paid for itself in the first week.

Next I usually add the database, cache, queues and external APIs. If you promised 99.9% uptime, forty-three minutes of downtime a month is already an incident. The runbook is written before the second such letter, not after.

← All posts Discuss post

2025-11-13·30 min