Prometheus & Grafana Monitoring on Bare Metal: A How-To Guide

Why Prometheus and Grafana for Server Monitoring?

Prometheus and Grafana form a powerful, open-source stack for server and application monitoring. Prometheus excels at collecting and storing time-series data, offering a flexible query language (PromQL) for analysis and alerting. Grafana, on the other hand, provides rich, customizable dashboards for visualizing this data, making complex metrics easily digestible. Together, they offer a comprehensive solution for tracking server health, resource utilization, and application-specific metrics, essential for maintaining stable and performant services like game servers, web hosting, databases, or CI/CD pipelines.

Prerequisites

Before you begin, ensure you have the following:

• Two Servers: One dedicated server or robust VPS for the Prometheus/Grafana stack (the 'monitoring server'), and at least one server to be monitored (the 'monitored server').
• Operating System: Ubuntu 22.04 LTS or a similar Debian-based distribution on both servers.
• SSH Access: Root or a user with sudo privileges on both servers.
• Firewall: UFW enabled and configured to allow SSH (port 22), Prometheus (port 9090), Grafana (port 3000), and Node Exporter (port 9100) traffic.

Minimum Server Requirements for the Monitoring Stack (Monitoring Server)

For the server hosting Prometheus and Grafana, consider these baseline specifications:

• vCPU: 2 cores
• RAM: 4GB
• Disk: 50GB NVMe (for performance and data storage)
• Bandwidth: 1TB/month

These specifications are suitable for monitoring up to 10-15 endpoints with typical metric volumes. Larger deployments will require more resources.

Step-by-Step Installation and Configuration

Step 1: Update System and Install Dependencies (Monitoring Server)

Start by updating your monitoring server's package list and installing necessary utilities.

sudo apt update
sudo apt upgrade -y
sudo apt install -y wget curl gnupg2 software-properties-common apt-transport-https

Step 2: Install Prometheus Server (Monitoring Server)

Prometheus doesn't typically come in standard repositories, so we'll download its binary directly.

Download and Extract Prometheus

PROMETHEUS_VERSION="2.48.0" # Check for the latest stable version on prometheus.io
wget https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
tar xvfz prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROMETHEUS_VERSION}.linux-amd64 /usr/local/prometheus

Create Prometheus User and Directories

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Configure Prometheus

Create a basic Prometheus configuration file at /etc/prometheus/prometheus.yml:

sudo nano /etc/prometheus/prometheus.yml

Add the following content:

global:
  scrape_interval: 15s # How frequently to scrape targets.
  evaluation_interval: 15s # How frequently to evaluate rules.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090'] # Prometheus scrapes itself

Set Permissions and Create Systemd Service

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/prometheus/prometheus
sudo chown prometheus:prometheus /usr/local/prometheus/promtool
sudo cp /usr/local/prometheus/consoles /usr/local/prometheus/console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

sudo nano /etc/systemd/system/prometheus.service

Add the service configuration:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/prometheus/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.external-url=http://<YOUR_MONITORING_SERVER_IP>:9090

[Install]
WantedBy=multi-user.target

Replace <YOUR_MONITORING_SERVER_IP> with your server's public IP address or domain name.

Start Prometheus and Enable Firewall

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo ufw allow 9090/tcp

Verify Prometheus is running by navigating to http://<YOUR_MONITORING_SERVER_IP>:9090 in your web browser. You should see the Prometheus UI.

Step 3: Install Node Exporter (Monitored Server)

Node Exporter runs on each server you want to monitor, exposing system metrics for Prometheus to scrape.

Download and Extract Node Exporter

On your *monitored* server:

NODE_EXPORTER_VERSION="1.7.0" # Check for the latest stable version
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
tar xvfz node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
sudo mv node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64 /usr/local/node_exporter

Create Node Exporter User and Service

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo chown -R node_exporter:node_exporter /usr/local/node_exporter
sudo nano /etc/systemd/system/node_exporter.service

Add the service configuration:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

Start Node Exporter and Enable Firewall

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo ufw allow 9100/tcp

Verify Node Exporter is running by visiting http://<YOUR_MONITORED_SERVER_IP>:9100/metrics in your browser. You should see a page full of metrics.

Add Node Exporter to Prometheus Configuration (Monitoring Server)

Back on your *monitoring* server, edit /etc/prometheus/prometheus.yml to add the new target:

sudo nano /etc/prometheus/prometheus.yml

Add a new scrape_configs entry:

...
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['<YOUR_MONITORED_SERVER_IP>:9100'] # Replace with actual IP
        labels:
          instance: 'my_first_server'

Reload Prometheus to apply changes:

sudo systemctl reload prometheus

Check Prometheus UI (http://<YOUR_MONITORING_SERVER_IP>:9090/targets) to ensure Node Exporter is listed and healthy.

Step 4: Install Grafana (Monitoring Server)

Grafana provides the visualization layer for your Prometheus data.

Add Grafana GPG Key and Repository

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update

Install Grafana

sudo apt install -y grafana

Start Grafana and Enable Firewall

sudo systemctl start grafana-server
sudo systemctl enable grafana-server
sudo ufw allow 3000/tcp

Access Grafana at http://<YOUR_MONITORING_SERVER_IP>:3000. The default login is admin / admin. You will be prompted to change the password.

Add Prometheus as a Data Source in Grafana

1. Log in to Grafana.
2. Click the gear icon (Configuration) on the left sidebar, then Data sources.
3. Click Add data source and select Prometheus.
4. Set the Name (e.g., Prometheus).
5. For the URL, enter http://localhost:9090 (since Grafana is on the same server).
6. Click Save & Test. You should see a "Data source is working" message.

Step 5: Import Dashboards and Create Alerts (Monitoring Server)

Import a Node Exporter Dashboard

Grafana Labs provides many community-contributed dashboards. A popular one for Node Exporter is ID 1860.

1. On the left sidebar, click the plus icon (Create), then Import.
2. Enter 1860 in the "Import via grafana.com" field and click Load.
3. Select your Prometheus data source and click Import.

You should now see a comprehensive dashboard visualizing metrics from your monitored server.

Basic Alerting with Prometheus and Grafana

Prometheus handles alert rules, while Grafana can visualize and send notifications for these alerts.

1. Prometheus Alert Rules: Create a file like /etc/prometheus/alert.rules.yml with rules (e.g., high CPU usage, low disk space).
2. Prometheus Configuration: Update prometheus.yml to include rule_files: ['/etc/prometheus/alert.rules.yml'].
3. Alertmanager: For advanced notification routing (email, Slack, PagerDuty), install and configure Prometheus Alertmanager.
4. Grafana Alerting: You can also set up alerts directly within Grafana dashboards, which can trigger notifications based on query results.

Scaling Your Monitoring Setup

The resources required for your Prometheus and Grafana monitoring stack depend heavily on the number of monitored endpoints, the frequency of data collection, and the retention period of your metrics. It's crucial to select appropriate server hardware to ensure your monitoring system remains reliable and performant as your infrastructure grows.

Troubleshooting Common Issues

• Prometheus UI Not Accessible (9090): Check firewall (sudo ufw status) and Prometheus service status (sudo systemctl status prometheus). Ensure --web.external-url is correctly set in the service file.
• Node Exporter Not Accessible (9100): Verify firewall on the monitored server and Node Exporter service status.
• Targets Down in Prometheus: Check network connectivity between the monitoring and monitored servers. Ensure Node Exporter is running on the target and its IP/port are correct in prometheus.yml.
• Grafana Data Source Error: Confirm Prometheus is running on port 9090. Check Grafana server logs (sudo journalctl -u grafana-server -f) for errors.
• Missing Metrics in Grafana: Ensure the correct Prometheus data source is selected in your dashboard. Verify that Prometheus is indeed scraping the target (check Prometheus UI -> Status -> Targets).
• Disk Space Running Low: Prometheus stores metrics data. Configure retention policies in prometheus.yml (e.g., --storage.tsdb.retention.time=30d) or consider remote storage solutions for long-term retention.