Prometheus & Grafana: Bare Metal Server Monitoring Guide

Why Dedicated Server Monitoring Matters

Dedicated servers from Valebyte offer unparalleled performance, security, and customization. However, with great power comes the need for proactive management. Without robust monitoring, you're operating in the dark, vulnerable to unexpected outages, performance bottlenecks, and resource exhaustion. Real-time insights into your CPU, memory, disk I/O, network traffic, and application metrics are crucial for:

• Preventing Downtime: Identify and address issues before they escalate into service disruptions.
• Optimizing Performance: Pinpoint resource hogs and bottlenecks to ensure your applications run at peak efficiency.
• Capacity Planning: Understand resource trends to make informed decisions about scaling your infrastructure.
• Troubleshooting: Quickly diagnose the root cause of problems with historical data and detailed metrics.
• Security: Monitor unusual activity or resource spikes that could indicate a security breach.

Whether you're running high-traffic web applications, a demanding game server, complex databases, streaming services, or CI/CD pipelines, a well-implemented monitoring solution is your first line of defense and a key to operational excellence.

Understanding Prometheus and Grafana

Prometheus and Grafana form a powerful, open-source monitoring stack that has become an industry standard. They complement each other perfectly, with Prometheus handling the data collection and storage, and Grafana providing the visualization layer.

Prometheus: The Metrics Collector

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It's designed for reliability and scalability, making it ideal for bare metal environments. Key features include:

• Multi-dimensional Data Model: Time series data is identified by metric name and key/value pairs.
• Flexible Query Language (PromQL): Allows for powerful and precise querying of collected metrics.
• Pull Model: Prometheus actively scrapes metrics from configured targets at specified intervals.
• Service Discovery: Integrates with various mechanisms to automatically discover new targets.
• Alerting: Integrates with Alertmanager to handle alerts based on PromQL expressions.

For bare metal monitoring, Prometheus typically scrapes metrics from 'exporters'. The most common exporter for host-level metrics is the Node Exporter.

Grafana: The Visualization Powerhouse

Grafana is an open-source analytics and interactive visualization web application. It connects to various data sources, including Prometheus, to create beautiful and informative dashboards. With Grafana, you can:

• Create Dynamic Dashboards: Build highly customizable dashboards with various panel types (graphs, single stats, tables, heatmaps, etc.).
• Explore Data: Drill down into metrics with powerful query builders.
• Set Alerts: Configure alerts directly from your dashboards based on metric thresholds.
• Collaborate: Share dashboards and allow team members to view or edit.

Prerequisites for Bare Metal Monitoring

Before diving into the installation, ensure you have the following:

• A Dedicated Server from Valebyte: With root or sudo privileges. This guide assumes an Ubuntu 22.04 LTS server, but the steps are largely similar for other Debian-based distributions.
• Basic Linux Command-Line Knowledge: Familiarity with commands like cd, mkdir, cp, wget, tar, systemctl.
• Internet Connectivity: To download necessary packages and binaries.
• Firewall Configuration: You'll need to open specific ports. We'll use UFW (Uncomplicated Firewall) as an example.
• Required Ports:
  • Prometheus: Default port 9090 (for web UI)
  • Node Exporter: Default port 9100 (for metrics)
  • Grafana: Default port 3000 (for web UI)

First, update your system's package list:

sudo apt update
sudo apt upgrade -y

Install curl and wget if not already present, as they are useful for downloading files:

sudo apt install -y curl wget

Step-by-Step Installation Guide

We'll install Prometheus, Node Exporter, and Grafana on the same dedicated server for simplicity. For larger deployments, you might run Node Exporter on multiple servers and Prometheus/Grafana on a dedicated monitoring server.

1. Setting Up Prometheus

Create a Prometheus User and Directories

For security best practices, run Prometheus under a dedicated user account.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Download and Install Prometheus

Visit the Prometheus download page to get the latest stable version. As of writing, we'll use a placeholder version.

# Check for the latest version on the Prometheus download page
PROMETHEUS_VERSION="2.48.0" # Replace with the latest stable version

wget https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz
cd prometheus-${PROMETHEUS_VERSION}.linux-amd64

sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

sudo cp -r consoles /etc/prometheus
sudo cp -r console_libraries /etc/prometheus

Configure Prometheus

Create the Prometheus configuration file /etc/prometheus/prometheus.yml. For now, we'll start with a basic configuration.

sudo nano /etc/prometheus/prometheus.yml

Add the following content:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Set appropriate ownership for the configuration file:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create a Systemd Service for Prometheus

This allows Prometheus to run as a background service and start automatically on boot.

sudo nano /etc/systemd/system/prometheus.service

Add the following content:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Reload systemd, start Prometheus, and enable it to start on boot:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Verify Prometheus Status

sudo systemctl status prometheus

You should see an 'active (running)' status. You can also check the logs:

sudo journalctl -u prometheus -f

2. Setting Up Node Exporter

Node Exporter exposes a wide range of hardware and OS metrics (CPU, memory, disk I/O, network stats, etc.) from the server it runs on.

Create a Node Exporter User and Directories

sudo useradd --no-create-home --shell /bin/false node_exporter

Download and Install Node Exporter

Visit the Prometheus download page for the latest Node Exporter version.

# Check for the latest version on the Prometheus download page
NODE_EXPORTER_VERSION="1.7.0" # Replace with the latest stable version

wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
tar xvf node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz
cd node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64

sudo mv node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create a Systemd Service for Node Exporter

sudo nano /etc/systemd/system/node_exporter.service

Add the following content:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Reload systemd, start Node Exporter, and enable it to start on boot:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Verify Node Exporter Status

sudo systemctl status node_exporter

You should see an 'active (running)' status. You can also check the metrics directly from your server:

curl http://localhost:9100/metrics

This will output a large amount of plain text metrics.

3. Setting Up Grafana

Install Grafana

Grafana provides official APT repositories for easy installation.

sudo apt install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt update
sudo apt install grafana -y

Start and Enable Grafana

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Verify Grafana Status

sudo systemctl status grafana-server

You should see an 'active (running)' status.

4. Configure Firewall (UFW)

To access the Prometheus and Grafana web UIs, you need to open their respective ports in your firewall. Assuming UFW is enabled:

sudo ufw allow 9090/tcp comment 'Allow Prometheus Web UI'
sudo ufw allow 9100/tcp comment 'Allow Node Exporter Metrics'
sudo ufw allow 3000/tcp comment 'Allow Grafana Web UI'
sudo ufw reload

If you have SSH enabled (which you should), ensure port 22 is also open: sudo ufw allow 22/tcp.

Configuring Prometheus to Scrape Node Exporter

Now that Node Exporter is running and exposing metrics, we need to tell Prometheus to scrape them. Edit the Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

Add a new job_name under scrape_configs for Node Exporter. Since Node Exporter is on the same server, we'll use localhost:9100.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100'] # Or replace localhost with the IP of your target server

Save and exit the file. Then, restart Prometheus to apply the changes:

sudo systemctl restart prometheus

Configuring Grafana Data Source and Dashboard

1. Access Grafana Web UI

Open your web browser and navigate to http://YOUR_SERVER_IP:3000. You should see the Grafana login page.

The default login credentials are:

• Username: admin
• Password: admin

You will be prompted to change the default password immediately. Choose a strong, secure password.

2. Add Prometheus as a Data Source

1. On the left-hand menu, hover over the gear icon (Configuration) and click Data sources.
2. Click Add data source.
3. Search for and select Prometheus.
4. In the 'HTTP' section, set the 'URL' to http://localhost:9090 (since Prometheus is on the same server).
5. Scroll down and click Save & test. You should see a message confirming "Data source is working."

3. Import a Node Exporter Dashboard

Grafana's strength lies in its vast library of community-contributed dashboards. A popular dashboard for Node Exporter metrics is Node Exporter Full (ID 1860).

1. On the left-hand menu, hover over the '+' icon (Create) and click Import.
2. In the 'Import via grafana.com' field, enter 1860 and click Load.
3. On the next screen, select your Prometheus data source from the dropdown.
4. Click Import.

You should now see a comprehensive dashboard displaying various metrics from your dedicated server, including CPU usage, memory, disk I/O, network traffic, and more!

Testing Your Monitoring Setup

To ensure everything is working as expected:

• Prometheus UI: Go to http://YOUR_SERVER_IP:9090. Click on 'Status' -> 'Targets'. You should see both 'prometheus' and 'node_exporter' targets listed with a 'UP' state. If not, check Prometheus logs and configuration.
• Node Exporter Metrics: Open http://YOUR_SERVER_IP:9100/metrics in your browser. You should see a page full of raw metrics.
• Grafana Dashboard: Interact with the Node Exporter Full dashboard. Change time ranges, observe real-time data, and ensure all panels are displaying data correctly.

Advanced Monitoring Considerations

Alerting with Alertmanager

While this guide focuses on basic setup, a complete monitoring solution includes alerting. Prometheus's Alertmanager handles alerts sent by client applications, grouping, deduplicating, and routing them to notification channels like email, Slack, PagerDuty, etc.

Monitoring Multiple Servers

To monitor multiple Valebyte dedicated servers, simply install Node Exporter on each server you wish to monitor. Then, in your central Prometheus server's prometheus.yml, add each server's IP address and Node Exporter port (e.g., 192.168.1.10:9100) to the node_exporter job's targets list. Remember to open port 9100 on the firewall of each monitored server.

Security Best Practices for Monitoring

• Firewall: Only expose monitoring ports (9090, 9100, 3000) to trusted networks or IPs. For Grafana, consider placing it behind a reverse proxy (like Nginx or Apache) with HTTPS for secure access.
• Authentication: Always change default Grafana passwords. Consider integrating Grafana with an external authentication provider (LDAP, OAuth) for production environments.
• Dedicated Users: Running Prometheus and Node Exporter under unprivileged users (as done in this guide) limits potential damage if a service is compromised.

Troubleshooting Common Issues

Even with careful steps, you might encounter issues. Here's how to diagnose them:

1. Service Not Starting (Prometheus, Node Exporter, Grafana)

• Check Service Status:

  sudo systemctl status prometheus
  sudo systemctl status node_exporter
  sudo systemctl status grafana-server

  Look for 'active (running)' or specific error messages.
• Examine Logs:

  sudo journalctl -u prometheus -f
  sudo journalctl -u node_exporter -f
  sudo journalctl -u grafana-server -f

  Logs are your best friend for understanding why a service failed to start. Look for permission errors, configuration file issues, or port conflicts.
• Configuration File Syntax: For Prometheus, you can check your prometheus.yml for syntax errors:

  /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml

  This will tell you if there are any parsing errors.

2. Prometheus Not Scraping Targets (Node Exporter Metrics Missing)

• Check Prometheus UI Targets: Navigate to http://YOUR_SERVER_IP:9090/targets. If Node Exporter is listed but 'DOWN', check the 'Error' column for clues.
• Firewall on Monitored Server: Ensure port 9100 (Node Exporter) is open on the server running Node Exporter. If Prometheus is on a different server, ensure its IP is allowed by the Node Exporter server's firewall.
• Node Exporter Running: Verify Node Exporter is active and accessible locally on the monitored server:

  sudo systemctl status node_exporter
  curl http://localhost:9100/metrics

  If curl fails, Node Exporter isn't running or isn't listening on the expected port/interface.
• Prometheus Configuration: Double-check the targets entry in /etc/prometheus/prometheus.yml for typos (e.g., localhost:9100 or the correct IP address). Remember to restart Prometheus after any config changes.

3. Grafana Not Displaying Data or "Data Source Error"

• Prometheus Data Source Connectivity: In Grafana, go to 'Configuration' -> 'Data sources', click on your Prometheus data source, and click 'Save & test'. If it fails, check the Prometheus server's IP and port in the Grafana data source configuration.
• Firewall on Prometheus Server: Ensure port 9090 (Prometheus) is open on the server running Prometheus, allowing Grafana to connect.
• Prometheus Running and Scraping: Verify Prometheus is running and actively scraping Node Exporter metrics (as per troubleshooting step 2). If Prometheus isn't collecting data, Grafana won't have anything to display.
• Dashboard Configuration: If some panels are empty, ensure the queries within those panels are correct and reference the right metrics. Sometimes community dashboards might need slight adjustments for specific Prometheus versions or metric names.

4. Permission Issues

If services fail to start or write data, check file and directory permissions. Ensure the dedicated users (prometheus, node_exporter) own their respective directories and files as configured in the systemd service files.

ls -la /usr/local/bin/prometheus
ls -la /etc/prometheus/prometheus.yml
ls -la /var/lib/prometheus

By systematically checking these points, you can resolve most common issues encountered during the setup of your Prometheus and Grafana monitoring stack on a Valebyte dedicated server.