Prometheus & Grafana Bare Metal Monitoring Tutorial

Why Prometheus and Grafana for Bare Metal Monitoring?

Monitoring is a critical component of maintaining healthy and performant server infrastructure. For bare metal environments, where you have full control and responsibility, a robust monitoring solution like Prometheus and Grafana is indispensable. Prometheus excels at collecting and storing time-series data, while Grafana provides powerful visualization capabilities, turning raw metrics into actionable dashboards. Together, they offer a comprehensive view of your server's CPU usage, memory consumption, disk I/O, network traffic, and application-specific metrics.

Whether you're hosting a high-traffic web application, a demanding game server, a complex database cluster, or a CI/CD pipeline, understanding your server's performance characteristics is key to proactive maintenance, troubleshooting, and scaling. This tutorial focuses on setting up this stack on a dedicated server from Valebyte, providing the isolation and raw power needed for serious monitoring.

Prerequisites

• A dedicated server or a powerful VPS (for the monitoring stack itself) running Ubuntu 22.04 LTS.
• Root or sudo access to the server.
• Basic familiarity with the Linux command line.
• An unprivileged user for running Prometheus and Grafana services.
• Firewall configured to allow access to Prometheus (port 9090), Grafana (port 3000), and Node Exporter (port 9100).

Minimum Server Requirements for the Monitoring Stack

For a small to medium-sized monitoring setup (monitoring 5-10 servers), the monitoring server itself should have:

• vCPU: 2 cores (or 2 vCPU)
• RAM: 4 GB
• Disk: 50 GB NVMe (for Prometheus time-series data)
• Bandwidth: 100 Mbps (for collecting metrics and dashboard access)

For larger deployments, consider higher specifications as detailed in the scaling table below.

Step-by-Step Tutorial: Setting Up Prometheus and Grafana

Step 1: Prepare Your Server

First, update your system packages and install necessary utilities:

sudo apt update
sudo apt upgrade -y
sudo apt install -y wget curl gnupg2 software-properties-common apt-transport-https

Create dedicated system users for Prometheus and Grafana. This enhances security by running services with minimal privileges.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false grafana

Step 2: Install Prometheus Server

Download the latest stable Prometheus release. Check the Prometheus download page for the most recent version. As of this writing, 2.47.0 is current.

PROMETHEUS_VERSION="2.47.0"
wget https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
tar xvfz prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
cd prometheus-${PROMETHEUS_VERSION}.linux-amd64

Create necessary directories and copy the binaries and configuration files:

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp prometheus promtool /usr/local/bin/
sudo cp consoles/ console_libraries/ /etc/prometheus/
sudo cp prometheus.yml /etc/prometheus/prometheus.yml

Set ownership for security:

sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

Step 3: Configure Prometheus

Edit the Prometheus configuration file to define scrape targets. For a basic setup, we'll monitor the Prometheus server itself using a Node Exporter (which we'll install later).

sudo nano /etc/prometheus/prometheus.yml

Replace the existing scrape_configs section with the following, or append a new job for the local Node Exporter:

global:
  scrape_interval: 15s # How frequently to scrape targets
  evaluation_interval: 15s # How frequently to evaluate rules

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100'] # Target for the local Node Exporter

Save and exit (Ctrl+X, Y, Enter).

Step 4: Create Prometheus Systemd Service

Create a systemd service file to manage Prometheus:

sudo nano /etc/systemd/system/prometheus.service

Add the following content:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address=":9090"
Restart=always

[Install]
WantedBy=multi-user.target

Reload systemd, start Prometheus, and enable it to start on boot:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

Verify Prometheus is running by navigating to http://YOUR_SERVER_IP:9090/ in your browser. You should see the Prometheus UI.

Step 5: Install Node Exporter on Monitored Servers (and the Prometheus Server)

Node Exporter collects host-level metrics (CPU, memory, disk, network) and exposes them to Prometheus. Install it on any server you wish to monitor, including the Prometheus server itself.

Download the latest stable Node Exporter release (e.g., 1.6.1):

NODE_EXPORTER_VERSION="1.6.1"
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
tar xvfz node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
cd node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64

Copy the binary and set ownership:

sudo cp node_exporter /usr/local/bin
sudo chown prometheus:prometheus /usr/local/bin/node_exporter

Create a systemd service for Node Exporter:

sudo nano /etc/systemd/system/node_exporter.service

Add the following content:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=":9100"
Restart=always

[Install]
WantedBy=multi-user.target

Reload systemd, start Node Exporter, and enable it:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

Verify Node Exporter is running by navigating to http://YOUR_SERVER_IP:9100/metrics in your browser. You should see a page of raw metrics.

Important: If installing Node Exporter on other servers, remember to open port 9100 in their firewalls and add their IPs to the /etc/prometheus/prometheus.yml file on your Prometheus server:

  - job_name: 'remote_servers'
    static_configs:
      - targets: ['REMOTE_SERVER_IP_1:9100', 'REMOTE_SERVER_IP_2:9100']

After modifying prometheus.yml, restart Prometheus: sudo systemctl restart prometheus.

Step 6: Install Grafana

Add Grafana's GPG key and repository to your system:

sudo wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Update packages and install Grafana:

sudo apt update
sudo apt install -y grafana

Start Grafana and enable it to start on boot:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
sudo systemctl status grafana-server

Verify Grafana is running by navigating to http://YOUR_SERVER_IP:3000/ in your browser. The default login is admin / admin. You'll be prompted to change the password on first login.

Step 7: Configure Grafana with Prometheus Data Source and Dashboard

Log into Grafana (admin/admin).

Add Prometheus as a Data Source:

1. Click the gear icon (Configuration) on the left sidebar, then 'Data sources'.
2. Click 'Add data source'.
3. Select 'Prometheus'.
4. For 'Name', enter Prometheus.
5. For 'URL', enter http://localhost:9090.
6. Scroll down and click 'Save & Test'. You should see 'Data source is working'.

Import a Node Exporter Dashboard:

1. Click the '+' icon (Create) on the left sidebar, then 'Import'.
2. In the 'Import via grafana.com' field, enter 1860 (a popular Node Exporter Full dashboard ID).
3. Click 'Load'.
4. On the next screen, select your 'Prometheus' data source from the dropdown.
5. Click 'Import'.

You should now see a comprehensive dashboard displaying metrics from your Node Exporter instances. You can switch between different Node Exporter targets if you have multiple configured.

Troubleshooting Common Issues

• Service not starting: Check systemd logs with sudo journalctl -u prometheus.service or sudo journalctl -u grafana-server.service for errors.
• Prometheus not scraping targets: Verify target IPs and ports in /etc/prometheus/prometheus.yml. Check Prometheus 'Status -> Targets' page in the UI for scrape errors. Ensure firewalls are open on both the Prometheus server (port 9090, 9100) and monitored servers (port 9100).
• Grafana not connecting to Prometheus: Double-check the Prometheus data source URL in Grafana. Ensure Prometheus is running and accessible from the Grafana server.
• Metrics missing in Grafana: Confirm Node Exporter is running on the target server and Prometheus is successfully scraping it. Check Prometheus 'Graph' page to see if metrics are being collected (e.g., node_cpu_seconds_total).
• Permissions issues: Ensure the prometheus and grafana users own their respective directories and binaries.

Choosing the right server for your monitoring stack, especially when managing numerous or critical services, is paramount. The following table provides guidance on server specifications based on the scale of your monitored infrastructure.

Further Customization and Advanced Monitoring

Once your basic setup is operational, you can extend your monitoring capabilities:

• Alertmanager: Integrate Alertmanager with Prometheus to send notifications (email, Slack, PagerDuty) when critical thresholds are crossed.
• Exporters for specific applications: Install other Prometheus Exporters for databases (e.g., mysqld_exporter, postgres_exporter), web servers (e.g., apache_exporter, nginx_exporter), or messaging queues.
• Custom Dashboards: Create your own Grafana dashboards tailored to your specific application metrics and business needs.
• Recording Rules: Use Prometheus recording rules to pre-aggregate frequently queried data, improving dashboard load times.
• Remote Storage: For long-term retention or high availability, consider integrating Prometheus with remote storage solutions.