Prometheus & Grafana: Bare Metal Server Monitoring Setup Guide

Why Prometheus and Grafana for Bare Metal Monitoring?

Bare metal servers offer unparalleled performance, control, and isolation, making them ideal for demanding workloads like high-traffic web hosting, large databases, game servers, CI/CD pipelines, and streaming services. However, managing these powerful machines effectively requires robust monitoring. Prometheus excels at collecting time-series data, while Grafana provides intuitive dashboards for visualizing that data. Together, they form a potent combination for understanding server health, identifying bottlenecks, and proactively addressing issues.

This tutorial focuses on installing the monitoring stack (Prometheus and Grafana) on a dedicated server or a robust VPS, which will then monitor other bare metal servers using Node Exporter. This ensures the monitoring system itself has dedicated resources and doesn't contend with the services running on the monitored machines.

Prerequisites and Minimum Server Requirements

Before you begin, ensure you have:

• A dedicated server or a robust VPS from Valebyte with a fresh installation of Ubuntu 22.04 LTS (or a similar Debian-based distribution). For the monitoring server itself, we recommend at least 4 vCPU, 8GB RAM, and 100GB NVMe storage for production use, allowing for data retention and dashboard responsiveness.
• Root or sudo access to the monitoring server.
• Root or sudo access to the bare metal servers you wish to monitor.
• Basic understanding of Linux command-line interface.
• Open ports in your firewall: 9090 for Prometheus, 3000 for Grafana, and 9100 for Node Exporter (on monitored servers).

Step-by-Step Installation Guide

Step 1: Update System and Install Dependencies

First, connect to your monitoring server via SSH and update your package lists and installed packages. Install necessary utilities:

sudo apt update
sudo apt upgrade -y
sudo apt install -y wget curl gnupg2 software-properties-common apt-transport-https

Step 2: Install Prometheus Server

Prometheus does not typically reside in standard Ubuntu repositories, so we will download the latest stable release directly.

1. Create Prometheus user and directories:

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

2. Download and extract Prometheus: Replace 2.47.1 with the latest stable version if different. Check Prometheus Downloads.

wget https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz
tar xvfz prometheus-2.47.1.linux-amd64.tar.gz
sudo cp prometheus-2.47.1.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.47.1.linux-amd64/promtool /usr/local/bin/
sudo cp -r prometheus-2.47.1.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.47.1.linux-amd64/console_libraries /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
rm -rf prometheus-2.47.1.linux-amd64.tar.gz prometheus-2.47.1.linux-amd64

3. Create Prometheus configuration file (/etc/prometheus/prometheus.yml):

sudo nano /etc/prometheus/prometheus.yml

Paste the following content. This configuration includes a basic scrape target for Prometheus itself and a placeholder for Node Exporter targets.

global:
  scrape_interval: 15s # How frequently to scrape targets
  evaluation_interval: 15s # How frequently to evaluate rules

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Add your bare metal servers here after installing Node Exporter
  # - job_name: 'node_exporter'
  #   static_configs:
  #     - targets: ['your_server_ip_1:9100', 'your_server_ip_2:9100']

Save and exit (Ctrl+X, Y, Enter).

4. Set ownership for configuration:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

5. Create a systemd service file for Prometheus (/etc/systemd/system/prometheus.service):

sudo nano /etc/systemd/system/prometheus.service

Paste the following:

[Unit]
Description=Prometheus Monitoring System
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.enable-lifecycle

[Install]
WantedBy=multi-user.target

Save and exit.

6. Reload systemd, start, and enable Prometheus:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

7. Verify Prometheus status:

sudo systemctl status prometheus

You should see active (running). Access Prometheus UI in your browser at http://YOUR_MONITORING_SERVER_IP:9090.

Step 3: Install Node Exporter on Monitored Bare Metal Servers

Node Exporter is an agent that collects system metrics (CPU, RAM, disk I/O, network) and exposes them for Prometheus to scrape. Install this on each bare metal server you want to monitor.

1. Create Node Exporter user and directories:

sudo useradd --no-create-home --shell /bin/false node_exporter

2. Download and extract Node Exporter: Replace 1.6.1 with the latest stable version if different. Check Node Exporter Downloads.

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz
sudo cp node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
rm -rf node_exporter-1.6.1.linux-amd64.tar.gz node_exporter-1.6.1.linux-amd64

3. Create a systemd service file for Node Exporter (/etc/systemd/system/node_exporter.service):

sudo nano /etc/systemd/system/node_exporter.service

Paste the following:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Save and exit.

4. Reload systemd, start, and enable Node Exporter:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

5. Verify Node Exporter status:

sudo systemctl status node_exporter

You should see active (running). Access Node Exporter metrics in your browser at http://YOUR_MONITORED_SERVER_IP:9100/metrics.

Step 4: Configure Prometheus to Scrape Node Exporter Targets

Now, go back to your monitoring server. Edit the Prometheus configuration to add your bare metal servers.

1. Edit Prometheus configuration:

sudo nano /etc/prometheus/prometheus.yml

2. Add new job_name entries for your bare metal servers: Replace your_server_ip_1, your_server_ip_2 with the actual IP addresses of your monitored servers.

# ... existing configuration ...

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100', 'your_next_server_ip:9100']
    # Optionally, add labels for better organization
    # labels:
    #   environment: production
    #   location: datacenter_a

Save and exit.

3. Check Prometheus configuration for syntax errors:

/usr/local/bin/promtool check config /etc/prometheus/prometheus.yml

It should output SUCCESS!. If not, fix any reported errors.

4. Reload Prometheus to apply changes:

sudo systemctl reload prometheus

5. Verify targets in Prometheus UI: Navigate to http://YOUR_MONITORING_SERVER_IP:9090/targets. You should see your Node Exporter targets listed with a 'UP' state.

Step 5: Install Grafana

Grafana provides the visualization layer for your Prometheus data.

1. Add Grafana GPG key:

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

2. Add Grafana repository:

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

3. Update apt cache and install Grafana:

sudo apt update
sudo apt install -y grafana

4. Start and enable Grafana service:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

5. Verify Grafana status:

sudo systemctl status grafana-server

You should see active (running). Access Grafana UI in your browser at http://YOUR_MONITORING_SERVER_IP:3000. The default login is admin / admin. You will be prompted to change the password on first login.

Step 6: Configure Grafana Data Source and Dashboard

Now, connect Grafana to Prometheus and import a useful dashboard.

1. Add Prometheus as a Data Source:
   • Log in to Grafana (admin/admin).
   • Click the gear icon (Configuration) on the left sidebar, then 'Data sources'.
   • Click 'Add data source'.
   • Select 'Prometheus'.
   • For 'Name', enter Prometheus.
   • For 'URL', enter http://localhost:9090.
   • Scroll down and click 'Save & test'. You should see 'Data source is working'.
2. Import a Node Exporter Dashboard:
   • Click the '+' icon (Create) on the left sidebar, then 'Import'.
   • In the 'Import via grafana.com' field, enter 1860 (a popular Node Exporter Full dashboard ID).
   • Click 'Load'.
   • On the next screen, select your 'Prometheus' data source from the dropdown.
   • Click 'Import'.

   You should now see a comprehensive dashboard visualizing metrics from all your monitored bare metal servers.

Use Cases for Bare Metal Monitoring

With Prometheus and Grafana, you can monitor a wide array of bare metal workloads:

• Web Hosting: Track Nginx/Apache requests per second, response times, concurrent connections, and resource usage to ensure optimal website performance.
• Databases (e.g., PostgreSQL, MySQL): Monitor query latency, active connections, buffer hit ratios, disk I/O, and replication status to maintain database health.
• Game Servers: Keep an eye on CPU utilization, network latency, player counts, and memory usage to provide a smooth gaming experience.
• Streaming Services: Observe bandwidth usage, concurrent streams, and server load to ensure uninterrupted content delivery.
• CI/CD Pipelines: Monitor build agent resources, disk space, and network activity during continuous integration and deployment processes.
• Self-Hosted Applications: Track resource consumption of custom applications, mail servers (e.g., Postfix, Dovecot), or VPN services (e.g., OpenVPN, WireGuard) to ensure their stability.

Quick pick

Need a dedicated server?

Bare metal with NVMe in 70+ locations — configure and order in minutes.

Browse servers

Troubleshooting Common Issues

• Prometheus/Grafana/Node Exporter not starting: Check sudo systemctl status <service_name> and sudo journalctl -xeu <service_name> for error messages.
• Prometheus not scraping targets: Verify target IP addresses and port numbers in prometheus.yml. Ensure firewalls (e.g., UFW) on both the monitoring and monitored servers allow traffic on ports 9090 (Prometheus), 3000 (Grafana), and 9100 (Node Exporter). Check http://YOUR_MONITORING_SERVER_IP:9090/targets in Prometheus UI for scrape errors.
• Grafana not displaying data: Confirm the Prometheus data source is configured correctly and 'Save & test' shows success. Ensure the dashboard you imported is compatible with Node Exporter metrics.
• Permissions issues: Double-check file and directory ownership (chown) and permissions (chmod) for Prometheus and Node Exporter binaries and configuration files.

The right server infrastructure is crucial for efficient monitoring. Below are general hardware recommendations for your monitoring server based on the scale of your bare metal environment.