Bare Metal Monitoring: Prometheus & Grafana Tutorial

Why Monitor Your Bare Metal Server?

Dedicated servers offer unmatched power, control, and performance, but with great power comes the responsibility of vigilant oversight. Monitoring your bare metal server with tools like Prometheus and Grafana isn't just a best practice; it's essential for:

• Proactive Problem Solving: Detect anomalies, resource bottlenecks, and potential hardware failures before they impact your services.
• Performance Optimization: Understand CPU, RAM, disk I/O, and network usage patterns to fine-tune applications and infrastructure.
• Resource Management: Accurately assess if your current server specifications meet demand or if an upgrade is needed.
• Reliability and Uptime: Ensure your critical applications, whether game servers, high-traffic websites, databases, or CI/CD pipelines, remain operational and responsive.
• Capacity Planning: Gather historical data to make informed decisions about future infrastructure scaling.

For sysadmins, developers, and businesses relying on dedicated hosting, a robust monitoring solution provides the data-driven insights needed to maximize the return on their infrastructure investment.

Understanding Prometheus and Grafana

Prometheus and Grafana are a dynamic duo in the world of open-source monitoring. They are designed to work seamlessly together, each excelling in its specific domain:

Prometheus: The Data Collector and Time-Series Database

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It's renowned for its powerful multi-dimensional data model (time series data identified by metric name and key/value pairs), flexible query language (PromQL), and efficient storage.

• Scraping Model: Prometheus pulls (scrapes) metrics from configured targets at specified intervals.
• Exporters: Small applications that expose metrics from various services (e.g., Node Exporter for server OS metrics, Blackbox Exporter for endpoint probing).
• Service Discovery: Can dynamically discover targets to monitor.
• Alertmanager: Handles alerts sent by Prometheus, deduplicating, grouping, and routing them to various notification channels.

Grafana: The Visualization and Dashboard Powerhouse

Grafana is an open-source analytics and interactive visualization web application. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. When paired with Prometheus, Grafana transforms raw data into intuitive, actionable dashboards.

• Rich Dashboards: Create custom dashboards with a wide array of visualization options (graphs, tables, heatmaps, single stats).
• Multiple Data Sources: Supports Prometheus, InfluxDB, PostgreSQL, MySQL, Elasticsearch, and many others.
• Alerting: Define alert rules directly within Grafana to notify you of critical events.
• Templating: Build dynamic and reusable dashboards.

Prerequisites and Server Requirements

Before diving into the installation, ensure your Valebyte dedicated server meets these requirements:

Operating System

This guide primarily focuses on Debian/Ubuntu-based Linux distributions, which are common choices for bare metal deployments due to their stability and community support. The commands might vary slightly for RHEL/CentOS-based systems, but the core concepts remain the same.

Server Resources

While Prometheus and Grafana are efficient, they do consume resources. For monitoring a single bare metal server and a few applications:

• CPU: 1-2 cores (modern multi-core CPUs are ideal).
• RAM: 2GB - 4GB dedicated to the monitoring stack (Prometheus can be memory-intensive with long retention periods or many metrics).
• Storage: 20GB - 50GB SSD recommended for Prometheus's time-series database (TSDB). The actual requirement depends heavily on the number of metrics, their churn rate, and your desired data retention period. SSDs are crucial for good TSDB performance.
• Network: Standard network connectivity.

Access and Privileges

• Root or Sudo Access: You'll need administrative privileges to install packages, create users, and manage system services.
• SSH Access: Secure Shell access to your dedicated server.

Firewall Configuration

You'll need to open specific ports in your server's firewall (e.g., UFW on Ubuntu or firewalld on CentOS) to allow access to the monitoring components:

• Prometheus: Port 9090 (for the Prometheus UI and scraping).
• Node Exporter: Port 9100 (for Node Exporter metrics).
• Grafana: Port 3000 (for the Grafana web interface).

Step-by-Step Installation Guide

Let's get started with setting up your monitoring stack.

Step 1: Prepare Your Bare Metal Server

First, update your system and install necessary utilities.

sudo apt update
sudo apt upgrade -y
sudo apt install -y wget curl gnupg2 software-properties-common

Create Dedicated Users

It's a security best practice to run Prometheus and Node Exporter under unprivileged users.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false node_exporter

Configure Firewall (UFW Example)

If you're using UFW, enable it and allow necessary ports.

sudo ufw enable
sudo ufw allow ssh
sudo ufw allow 9090/tcp   # Prometheus
sudo ufw allow 9100/tcp   # Node Exporter
sudo ufw allow 3000/tcp   # Grafana
sudo ufw status

Step 2: Install Prometheus

Download and Extract Prometheus

Visit the Prometheus download page to get the latest stable version. Adjust the version number as needed.

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvf prometheus-2.48.0.linux-amd64.tar.gz
sudo mv prometheus-2.48.0.linux-amd64 /opt/prometheus-2.48.0

Create Directories and Set Permissions

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo cp /opt/prometheus-2.48.0/prometheus /usr/local/bin/
sudo cp /opt/prometheus-2.48.0/promtool /usr/local/bin/
sudo cp -r /opt/prometheus-2.48.0/consoles /etc/prometheus
sudo cp -r /opt/prometheus-2.48.0/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

Create Prometheus Configuration File

Create /etc/prometheus/prometheus.yml with the following basic configuration. This initial setup includes Prometheus itself as a target.

sudo nano /etc/prometheus/prometheus.yml

Add the following content:

global:
  scrape_interval: 15s # How frequently to scrape targets
  evaluation_interval: 15s # How frequently to evaluate rules

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Create Systemd Service File for Prometheus

This allows Prometheus to run as a service.

sudo nano /etc/systemd/system/prometheus.service

Add the following content:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address=0.0.0.0:9090 \
    --log.level=info

[Install]
WantedBy=multi-user.target

Reload Systemd, Start, and Enable Prometheus

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Verify Prometheus Installation

sudo systemctl status prometheus

You should see an 'active (running)' status. You can also access the Prometheus UI in your web browser at http://YOUR_SERVER_IP:9090. Navigate to 'Status' -> 'Targets' to see if Prometheus is scraping itself.

Step 3: Install Node Exporter (for Server Metrics)

Node Exporter exposes a wide range of hardware and OS metrics from the server it runs on.

Download and Extract Node Exporter

Get the latest stable version from the Prometheus download page.

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64 /opt/node_exporter-1.7.0

Move Binary and Set Permissions

sudo cp /opt/node_exporter-1.7.0/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create Systemd Service File for Node Exporter

sudo nano /etc/systemd/system/node_exporter.service

Add the following content:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload Systemd, Start, and Enable Node Exporter

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Verify Node Exporter Installation

sudo systemctl status node_exporter

You should see an 'active (running)' status. You can also check its metrics by visiting http://YOUR_SERVER_IP:9100/metrics in your browser.

Configure Prometheus to Scrape Node Exporter

Now, tell Prometheus to collect metrics from Node Exporter. Edit your /etc/prometheus/prometheus.yml file:

sudo nano /etc/prometheus/prometheus.yml

Add a new scrape_configs entry:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100'] # Or YOUR_SERVER_IP:9100 if on a different machine

Save the file and restart Prometheus for the changes to take effect:

sudo systemctl restart prometheus

Verify in the Prometheus UI (http://YOUR_SERVER_IP:9090 -> 'Status' -> 'Targets') that node_exporter is listed and healthy.

Step 4: Install Grafana

Grafana is typically installed from its official APT repository.

Add Grafana GPG Key and Repository

sudo wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Install Grafana

sudo apt update
sudo apt install grafana -y

Start and Enable Grafana

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Verify Grafana Installation

sudo systemctl status grafana-server

You should see an 'active (running)' status. Access the Grafana web interface at http://YOUR_SERVER_IP:3000. The default login is admin / admin. You will be prompted to change the password on first login.

Step 5: Configure Grafana with Prometheus Data Source

Now, let's connect Grafana to your Prometheus instance.

1. Log in to Grafana (default: admin / admin). Change the password when prompted.
2. On the left-hand menu, hover over the 'gear' icon (Configuration) and click 'Data Sources'.
3. Click 'Add data source'.
4. Select 'Prometheus' from the list.

5. Configure the data source:

   • Name: Prometheus (or any descriptive name).
   • URL: http://localhost:9090 (or http://YOUR_SERVER_IP:9090 if Prometheus is on a different machine).
   • Leave other settings as default for now.
6. Click 'Save & Test'. You should see a message like 'Data source is working'.

Step 6: Import/Create Grafana Dashboards

Grafana shines with its dashboards. You can create custom ones or import pre-built templates from the Grafana community.

Import a Pre-built Dashboard (Recommended for Node Exporter)

A popular dashboard for Node Exporter metrics is Node Exporter Full (ID: 1860).

1. In Grafana, hover over the 'plus' icon (Create) on the left-hand menu and click 'Import'.
2. In the 'Import via grafana.com' field, enter 1860 and click 'Load'.
3. Select your 'Prometheus' data source from the dropdown.
4. Click 'Import'.

You should now see a comprehensive dashboard displaying your bare metal server's CPU usage, memory, disk I/O, network traffic, and more!

Creating Custom Dashboards

For specific application monitoring or unique use cases (e.g., game server player count, database query rates), you can build custom dashboards:

1. Click the 'plus' icon (Create) and select 'Dashboard'.
2. Click 'Add new panel'.
3. Select your Prometheus data source.
4. Use PromQL (Prometheus Query Language) to query your metrics (e.g., node_cpu_seconds_total, node_memory_MemFree_bytes).
5. Choose your visualization type (Graph, Stat, Gauge, etc.).
6. Customize titles, units, and display options.

Advanced Configuration & Best Practices

Alerting with Alertmanager

While Grafana offers basic alerting, Prometheus's Alertmanager is designed for more sophisticated alert routing, grouping, and deduplication. For critical bare metal infrastructure, setting up Alertmanager to send notifications via email, Slack, PagerDuty, or other channels is highly recommended.

Security Considerations

• Firewall: Restrict access to Prometheus (9090), Node Exporter (9100), and Grafana (3000) ports to trusted IPs or your internal network where possible.
• TLS/SSL: Secure Grafana with HTTPS, especially if it's publicly accessible. You can use a reverse proxy like Nginx or Apache with Let's Encrypt.
• Strong Passwords: Change default Grafana credentials immediately.
• Dedicated Users: As implemented, running services under unprivileged users enhances security.

Storage Considerations

Prometheus stores data locally in its TSDB. Consider:

• Retention: The default retention is 15 days. Adjust --storage.tsdb.retention.time (e.g., --storage.tsdb.retention.time=30d) or --storage.tsdb.retention.size (e.g., --storage.tsdb.retention.size=50GB) in your prometheus.service file. Longer retention requires more disk space.
• Disk Type: SSDs are highly recommended for the Prometheus data directory (/var/lib/prometheus) due to their high I/O performance.

Monitoring Multiple Servers

To monitor additional bare metal servers, simply install Node Exporter on each server and configure your central Prometheus instance to scrape their respective :9100 endpoints. Update your prometheus.yml:

  - job_name: 'additional_servers'
    static_configs:
      - targets: ['server2_ip:9100', 'server3_ip:9100']

Real-World Use Cases for Monitoring

• Game Servers: Track CPU load, network latency, and memory usage to ensure a smooth gaming experience. Monitor specific game server metrics via custom exporters.
• Web Hosting: Observe web server (Nginx/Apache) metrics, database performance (MySQL/PostgreSQL), and application response times for high-traffic websites.
• Databases: Monitor query rates, connection counts, disk I/O, and buffer pool usage for optimal database health and performance.
• Mail Servers: Keep an eye on mail queue size, inbound/outbound traffic, and resource consumption to prevent delivery issues.
• Streaming Services: Monitor bandwidth usage, CPU utilization during encoding/transcoding, and storage performance.
• CI/CD Pipelines: Track build agent resource usage, job execution times, and overall pipeline health to identify bottlenecks.

Troubleshooting Common Issues

Prometheus Not Starting or Scraping

• Configuration Errors: Use promtool check config /etc/prometheus/prometheus.yml to validate your Prometheus configuration file.
• Port Conflicts: Ensure port 9090 isn't already in use by another application. Check logs with sudo journalctl -u prometheus.service.
• Permissions: Verify the prometheus user has read access to /etc/prometheus/prometheus.yml and write access to /var/lib/prometheus.
• Firewall: Ensure port 9090 is open.

Node Exporter Not Reachable

• Service Status: Check if Node Exporter is running: sudo systemctl status node_exporter.
• Firewall: Confirm port 9100 is open on the server running Node Exporter.
• Prometheus Configuration: Double-check the target IP/hostname and port in prometheus.yml.
• Logs: Review sudo journalctl -u node_exporter.service for errors.

Grafana Data Source Connection Issues

• Prometheus Status: Ensure Prometheus is running and accessible from the server hosting Grafana (if they are separate).
• Firewall: Verify port 9090 is open from Grafana's perspective.
• URL Correctness: Confirm the Prometheus URL in Grafana's data source configuration is correct (e.g., http://localhost:9090).

Missing Metrics in Grafana

• Prometheus Targets: Check Prometheus UI (Status -> Targets) to ensure all expected targets (Prometheus, Node Exporter) are up and healthy.
• PromQL Query: Verify your PromQL queries in Grafana are correct and target the metrics you expect. Use the Prometheus UI's 'Graph' interface to test queries first.
• Time Range: Ensure your Grafana dashboard's time range covers the period when data should be available.

Permission Errors

If you encounter permission denied errors during installation or when starting services, double-check that the files and directories have the correct ownership (e.g., prometheus:prometheus for Prometheus files, node_exporter:node_exporter for Node Exporter) and appropriate read/write permissions.