Prometheus & Grafana: Bare Metal Server Monitoring Setup

Why Choose Prometheus and Grafana for Bare Metal Monitoring?

Monitoring is critical for any server infrastructure, especially for dedicated or bare metal servers where performance and uptime are paramount. Prometheus is an open-source monitoring system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. Grafana, on the other hand, is an open-source analytics and interactive visualization web application that connects to various data sources, including Prometheus, to create powerful, customizable dashboards.

Choosing a bare metal server from Valebyte for your monitoring stack offers distinct advantages:

• Dedicated Resources: No resource contention, ensuring your monitoring system always has the CPU, RAM, and I/O it needs, even under heavy load.
• Performance: Superior disk I/O (especially with NVMe SSDs) for storing and querying large volumes of time-series data efficiently.
• Isolation: Complete control over the environment, enhancing security and stability for critical monitoring infrastructure.
• Scalability: Easier to upgrade hardware components directly as your monitoring needs grow.

For smaller deployments, such as monitoring a single application or a few personal services, a high-performance Valebyte VPS can serve as an excellent, cost-effective platform for Prometheus and Grafana. However, for monitoring multiple production servers, storing extensive historical data, or handling a high scrape rate, a Valebyte dedicated server is the recommended choice due to its superior, consistent performance.

Prerequisites and Minimum Server Requirements

Before you begin, ensure you have:

• A bare metal server or a robust VPS running Ubuntu 22.04 LTS (this tutorial uses Ubuntu).
• Root or sudo privileges on the server.
• Basic familiarity with the Linux command line.
• An internet connection to download packages.

Minimum Hardware Specifications for the Monitoring Server:

• CPU: 2 vCPUs / 2 cores (e.g., Intel Xeon E3-12xx equivalent or better)
• RAM: 4 GB
• Disk: 50 GB NVMe SSD (for optimal performance and data retention)
• Bandwidth: 100 Mbps (unmetered or high monthly allowance)

Step-by-Step Tutorial: Prometheus & Grafana Setup

Follow these steps to deploy Prometheus and Grafana on your Valebyte server.

1. Prepare Your Server

Update your system's package list and upgrade existing packages. Install necessary utilities like wget and curl if they aren't already present.

sudo apt update && sudo apt upgrade -y
sudo apt install -y wget curl libfontconfig1

Create dedicated system users for Prometheus and Node Exporter for security.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false node_exporter

2. Install Prometheus

Download the latest stable version of Prometheus. Check the Prometheus download page for the most current version. As of this writing, 2.49.1 is stable.

wget https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz
tar -xvf prometheus-2.49.1.linux-amd64.tar.gz
sudo mv prometheus-2.49.1.linux-amd64 /usr/local/prometheus

Create directories for Prometheus configuration and data, and set appropriate permissions.

sudo mkdir /etc/prometheus
sudo mv /usr/local/prometheus/prometheus.yml /etc/prometheus/
sudo mv /usr/local/prometheus/promtool /usr/local/bin/
sudo mv /usr/local/prometheus/prometheus /usr/local/bin/
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /usr/local/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Edit the Prometheus configuration file /etc/prometheus/prometheus.yml. For a basic setup, the default configuration is often sufficient, but we'll add a scrape target for Node Exporter later. For now, ensure it looks similar to this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Create a systemd service file for Prometheus to manage its lifecycle.

sudo nano /etc/systemd/system/prometheus.service

Paste the following content:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address=0.0.0.0:9090
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

Reload systemd, enable, and start Prometheus.

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Verify Prometheus is running by navigating to http://YOUR_SERVER_IP:9090 in your web browser. You should see the Prometheus UI.

3. Install Node Exporter

Node Exporter is a Prometheus exporter for hardware and OS metrics. Download the latest stable version. Check the Prometheus download page for the most current version. As of this writing, 1.7.0 is stable.

wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64 /usr/local/node_exporter

Move the executable and set permissions.

sudo mv /usr/local/node_exporter/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create a systemd service file for Node Exporter.

sudo nano /etc/systemd/system/node_exporter.service

Paste the following content:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Reload systemd, enable, and start Node Exporter.

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

Verify Node Exporter is running by navigating to http://YOUR_SERVER_IP:9100/metrics. You should see a page with various system metrics.

4. Configure Prometheus to Scrape Node Exporter

Edit the Prometheus configuration file /etc/prometheus/prometheus.yml to add Node Exporter as a target.

sudo nano /etc/prometheus/prometheus.yml

Add the following scrape configuration under the scrape_configs: section:

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Your prometheus.yml should now look like this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Reload Prometheus to apply the new configuration.

sudo systemctl reload prometheus

Go to the Prometheus UI (http://YOUR_SERVER_IP:9090), navigate to 'Status' -> 'Targets'. You should see both 'prometheus' and 'node_exporter' targets listed as 'UP'.

5. Install Grafana

Add Grafana's GPG key and repository to your system.

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Update your package list and install Grafana.

sudo apt update
sudo apt install -y grafana

Enable and start Grafana.

sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Verify Grafana is running by navigating to http://YOUR_SERVER_IP:3000. The default login is admin / admin. You will be prompted to change the password immediately.

6. Configure Firewall (UFW)

If you're using UFW (Uncomplicated Firewall), allow access to Prometheus and Grafana ports.

sudo ufw allow 9090/tcp comment 'Allow Prometheus'
sudo ufw allow 9999/tcp comment 'Allow Prometheus Alertmanager' # If you plan to use Alertmanager
sudo ufw allow 9100/tcp comment 'Allow Node Exporter'
sudo ufw allow 3000/tcp comment 'Allow Grafana'
sudo ufw enable
sudo ufw status

7. Add Prometheus as a Data Source in Grafana

Log into Grafana (http://YOUR_SERVER_IP:3000, default admin/admin, change password). Click on the gear icon (Configuration) -> 'Data Sources' -> 'Add data source'. Select 'Prometheus'.

• Name: Prometheus (or any descriptive name)
• URL: http://localhost:9090
• Click 'Save & test'. You should see 'Data source is working'.

8. Import a Grafana Dashboard

To visualize your server metrics, import a pre-built dashboard. A popular choice for Node Exporter is 'Node Exporter Full' (Dashboard ID: 1860).

• In Grafana, click the '+' icon -> 'Import'.
• Enter 1860 in the 'Import via grafana.com dashboard' field and click 'Load'.
• Select your 'Prometheus' data source from the dropdown.
• Click 'Import'.

You should now see a comprehensive dashboard displaying metrics from your Node Exporter.

Quick pick

Need a dedicated server?

Bare metal with NVMe in 70+ locations — configure and order in minutes.

Browse servers

Scaling Your Monitoring Infrastructure

As your infrastructure grows, so will the demands on your monitoring server. The following table provides guidance on hardware specifications for different scales of monitoring.

Choosing a bare metal server from Valebyte provides the flexibility to match these specifications precisely, ensuring your monitoring stack performs optimally without resource bottlenecks.

Troubleshooting Common Issues

• Service not starting: Check logs with sudo journalctl -u prometheus.service or sudo journalctl -u grafana-server.service for errors.
• Prometheus not scraping targets: Verify prometheus.yml syntax (use promtool check config /etc/prometheus/prometheus.yml) and ensure target services (like Node Exporter) are running and accessible on the specified port. Check 'Status -> Targets' in the Prometheus UI.
• Grafana not displaying data: Ensure the Prometheus data source URL is correct (http://localhost:9090 from Grafana's perspective on the same server) and that Prometheus is running and collecting data. Check Grafana logs for data source errors.
• Firewall blocking access: Double-check UFW rules or other firewall configurations to ensure ports 9090 (Prometheus), 9100 (Node Exporter), and 3000 (Grafana) are open.

Use Cases for Your Monitoring Stack

A properly configured Prometheus and Grafana setup is invaluable for a wide range of applications:

• Web Hosting: Monitor server load, HTTP request rates, response times, and database performance for Nginx, Apache, MySQL, PostgreSQL.
• Game Servers: Track CPU usage, RAM consumption, network latency, and player counts to ensure a smooth gaming experience.
• Databases: Collect metrics on query performance, connection counts, disk I/O, and replication status for critical data stores.
• Mail Servers: Observe queue sizes, message delivery rates, and resource usage for Postfix, Dovecot, and other mail services.
• Streaming Services: Observe bandwidth usage, concurrent streams, and server health to prevent interruptions.
• CI/CD Pipelines: Track build times, test execution, and resource consumption of Jenkins, GitLab CI, or other automation tools.
• Self-Hosted Applications: Gain insight into the performance of personal clouds (Nextcloud), media servers (Plex), VPNs (OpenVPN, WireGuard), and other self-hosted tools.