Prometheus and Grafana bare-metal monitoring architecture
Prometheus collects metrics from servers, applications, databases, and network devices. Grafana reads those metrics from Prometheus and turns them into dashboards, graphs, and alerts. On bare metal, this stack gives you direct control over CPU capacity, NVMe storage, retention policy, network access, and data locality.
A practical deployment has three components:
- Prometheus server: Scrapes, stores, and queries time-series metrics on TCP port 9090.
- Node Exporter: Runs on each monitored Linux server and exposes host metrics on TCP port 9100.
- Grafana: Provides dashboards and user access on TCP port 3000.
For a small estate, Prometheus and Grafana can share one bare-metal server. Use a separate monitoring server for production databases, mail servers, game servers, streaming nodes, CI/CD runners, or any workload where monitoring must remain available during an application outage.
Prerequisites and minimum server requirements
This guide assumes an x86_64 Ubuntu Server 24.04 LTS host with a static IP address and sudo access. Prometheus needs fast local storage because its write-ahead log and time-series database perform frequent disk writes. NVMe storage is strongly preferred over hard drives.
- 1 bare-metal monitoring server with at least 4 CPU cores, 16 GB RAM, and 200 GB NVMe.
- Ubuntu Server 24.04 LTS or another systemd-based Linux distribution.
- A DNS name such as
monitoring.example.comfor Grafana access. - TCP 9100 reachable from the Prometheus server to monitored hosts.
- TCP 3000 reachable only from administrator IP ranges or through a reverse proxy with TLS.
- Outbound HTTPS access during installation to download signed packages and release archives.
Use a VPS only for small, non-critical environments such as a few websites, a personal VPN, or a single self-hosted application. For sustained monitoring of many servers, long retention periods, high-cardinality application metrics, or isolation-critical workloads, use a dedicated bare-metal server with local NVMe.
Capacity planning for 15-second scrapes and 30-day retention
The following sizing assumes Node Exporter metrics, a 15-second scrape interval, 30-day retention, and modest application metrics. Kubernetes, per-request labels, high-cardinality logs, and database query labels can increase disk and RAM use sharply.
Takeaway: Start with 4 cores, 16 GB RAM, and 200 GB NVMe for up to 50 monitored servers; move to 8 cores and 32 GB RAM before adding hundreds of targets or long retention.
| Monitored Linux servers at 15-second scrape interval | vCPU / CPU cores | RAM | Disk | Monthly bandwidth |
|---|---|---|---|---|
| 10-50 servers | 4 cores | 16 GB | 200 GB NVMe | 250 GB |
| 51-250 servers | 8 cores | 32 GB | 1 TB NVMe | 1 TB |
| 251-1,000 servers | 16 cores | 64 GB | 2 TB enterprise NVMe | 4 TB |
For more than 1,000 targets, more than 90 days of retention, or large application metric volumes, use Prometheus for recent data and add a long-term metrics system such as Thanos, Cortex, or Grafana Mimir with object storage. Do not solve a cardinality problem merely by adding disk.
Step 1: Update the bare-metal server and create service accounts
Connect through SSH, apply security updates, install required tools, and create unprivileged service accounts. Prometheus and Node Exporter should not run as root.
sudo apt update
sudo apt -y upgrade
sudo apt install -y curl wget tar gnupg ufw
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
sudo mkdir -p /etc/prometheus /var/lib/prometheus /var/lib/node_exporter/textfile_collector
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown -R node_exporter:node_exporter /var/lib/node_exporterCheck available memory and NVMe capacity before continuing:
free -h
df -hT /
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTSStep 2: Install Prometheus 2.54.1
Download the official Linux AMD64 archive, install the binaries, and copy the console templates. Before production deployment, compare the archive checksum with the checksum published on the Prometheus GitHub release page.
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.54.1/prometheus-2.54.1.linux-amd64.tar.gz
tar -xzf prometheus-2.54.1.linux-amd64.tar.gz
sudo install -m 0755 prometheus-2.54.1.linux-amd64/prometheus /usr/local/bin/prometheus
sudo install -m 0755 prometheus-2.54.1.linux-amd64/promtool /usr/local/bin/promtool
sudo cp -r prometheus-2.54.1.linux-amd64/consoles /etc/prometheus/
sudo cp -r prometheus-2.54.1.linux-amd64/console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheusCreate the Prometheus configuration. Replace the example private IP addresses with the addresses of your monitored servers. Prometheus supports DNS names, but static private addresses are easier to troubleshoot initially.
sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'EOF'
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
site: bare-metal-dc1
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['127.0.0.1:9090']
- job_name: linux-nodes
static_configs:
- targets:
- '127.0.0.1:9100'
- '10.20.0.11:9100'
- '10.20.0.12:9100'
labels:
environment: production
EOF
sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sudo -u prometheus /usr/local/bin/promtool check config /etc/prometheus/prometheus.ymlCreate a systemd unit with a 30-day retention limit and a 150 GB storage ceiling. The storage size limit prevents metrics from consuming the entire system disk.
sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=150GB \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus --no-pagerStep 3: Install Node Exporter on the monitoring server and monitored hosts
Node Exporter reports CPU, memory, filesystem, disk I/O, network, load average, and other operating-system metrics. Repeat this installation on every Linux server that Prometheus will scrape.
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xzf node_exporter-1.8.2.linux-amd64.tar.gz
sudo install -m 0755 node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/node_exporter
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
--collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
--web.listen-address=:9100
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | headDo not expose port 9100 to the public internet. On each monitored server, allow it only from the Prometheus server IP. Replace 10.20.0.10 with your monitoring server address.
sudo ufw allow from 10.20.0.10 to any port 9100 proto tcp
sudo ufw status numberedStep 4: Install Grafana and provision Prometheus as a data source
Install Grafana from its official APT repository. Grafana and Prometheus are on the same machine in this example, so the data source uses the loopback address rather than a public IP.
sudo install -d -m 0755 /etc/apt/keyrings
wget -qO- https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo 'deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main' | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana
sudo mkdir -p /etc/grafana/provisioning/datasources
sudo tee /etc/grafana/provisioning/datasources/prometheus.yml > /dev/null <<'EOF'
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://127.0.0.1:9090
isDefault: true
editable: false
EOF
sudo systemctl enable --now grafana-server
sudo systemctl restart grafana-server
sudo systemctl status grafana-server --no-pagerOpen http://SERVER_IP:3000 from an authorized administrator network. The initial login is admin with password admin; Grafana prompts for an immediate password change. Import the widely used Node Exporter Full dashboard by selecting Dashboards, New, Import, and entering dashboard ID 1860. Select the provisioned Prometheus data source.
Step 5: Restrict network access and verify collection
Allow SSH only from your administration network, allow Grafana only from the same network, and avoid public access to Prometheus. Replace 198.51.100.0/24 with your real administrator CIDR before running these commands.
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from 198.51.100.0/24 to any port 22 proto tcp
sudo ufw allow from 198.51.100.0/24 to any port 3000 proto tcp
sudo ufw enable
sudo ufw status verbose
curl -s 'http://127.0.0.1:9090/api/v1/query?query=up'
curl -s http://127.0.0.1:3000/api/healthThe Prometheus query should return JSON with a result value of 1 for healthy targets. In the Prometheus web interface, open Status > Targets; every Node Exporter endpoint should show UP. In Grafana Explore, run up{job="linux-nodes"} and confirm a value of 1 for each server.
Step 6: Add a useful alert rule
Alerts catch failures that are easy to miss on dashboards. This rule triggers if a monitored Linux node has been unreachable for five minutes.
sudo mkdir -p /etc/prometheus/rules
sudo tee /etc/prometheus/rules/node-alerts.yml > /dev/null <<'EOF'
groups:
- name: node-health
rules:
- alert: NodeExporterDown
expr: up{job="linux-nodes"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: 'Node Exporter is down on {{ $labels.instance }}'
description: 'Prometheus has not scraped {{ $labels.instance }} for 5 minutes.'
EOF
sudo sed -i '/scrape_configs:/i rule_files:\n - /etc/prometheus/rules/*.yml\n' /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus/rules
sudo -u prometheus /usr/local/bin/promtool check rules /etc/prometheus/rules/node-alerts.yml
sudo systemctl reload prometheusPrometheus records alert state but does not send notifications by itself. Add Alertmanager later for email, Slack, PagerDuty, or webhook delivery. Test alerting safely by stopping Node Exporter on a non-production host: sudo systemctl stop node_exporter, wait five minutes, then start it again with sudo systemctl start node_exporter.
Troubleshooting Prometheus and Grafana on bare metal
Prometheus target shows DOWN
First test connectivity from the Prometheus server: curl -v http://10.20.0.11:9100/metrics. If the connection fails, check firewall rules, routing, and the Node Exporter service with sudo systemctl status node_exporter --no-pager. If it returns metrics but Prometheus still reports DOWN, validate YAML with sudo -u prometheus promtool check config /etc/prometheus/prometheus.yml and reload Prometheus.
Grafana cannot connect to Prometheus
Confirm that Prometheus is listening locally: ss -ltnp | grep 9090. Then test the query API using curl -s http://127.0.0.1:9090/-/ready. The Grafana data source URL should be http://127.0.0.1:9090 if both services share the same server.
Disk usage grows too quickly
Check TSDB storage with sudo du -sh /var/lib/prometheus and identify noisy metric names in Prometheus using queries such as count by (__name__)({__name__=~".+"}). Remove unneeded labels and never use user IDs, request IDs, email addresses, or timestamps as metric labels. Lower retention or add remote storage after fixing excessive cardinality.
Prometheus service fails after configuration changes
Run sudo -u prometheus promtool check config /etc/prometheus/prometheus.yml, then inspect logs with sudo journalctl -u prometheus -n 100 --no-pager. YAML indentation errors and incorrect target quoting are common causes.