Bare Metal Monitoring: Prometheus & Grafana Setup Guide

Why Dedicated Server Monitoring is Critical

Dedicated servers from Valebyte offer unparalleled performance, security, and customization. Unlike shared or virtualized environments, every resource—CPU, RAM, storage, and network bandwidth—is exclusively yours. This complete control, however, comes with the responsibility of comprehensive oversight. Effective monitoring allows you to:

• Prevent Downtime: Proactively identify and address issues before they impact your services.
• Optimize Performance: Pinpoint resource bottlenecks and fine-tune your applications and server configurations.
• Ensure Security: Detect unusual activity or resource spikes that could indicate a security breach.
• Plan for Growth: Understand resource utilization trends to make informed decisions about scaling your infrastructure.
• Maintain Uptime SLAs: Guarantee the reliability and availability your clients or users expect.

Prometheus and Grafana form a formidable duo for this task, providing a flexible, scalable, and highly customizable monitoring solution perfectly suited for the raw power of bare-metal infrastructure.

Understanding Prometheus and Grafana

Prometheus: The Metrics Powerhouse

Prometheus is an open-source monitoring system and time-series database. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. Key features include:

• Multi-dimensional Data Model: Time series data identified by metric name and key/value pairs.
• Flexible Query Language (PromQL): Allows for powerful ad-hoc querying and aggregation of time series data.
• Pull Model: Prometheus pulls metrics from instrumented jobs, rather than pushing them.
• Service Discovery: Integrates with various systems to automatically discover targets.
• Alerting: Sends notifications via Alertmanager.

Grafana: The Visualization Maestro

Grafana is an open-source analytics and interactive visualization web application. It connects to various data sources (including Prometheus) and allows you to create, explore, and share dashboards and alerts. With Grafana, you can:

• Build Dynamic Dashboards: Create beautiful and interactive dashboards with a wide range of visualization options.
• Explore Data: Drill down into your metrics with powerful query editors.
• Set Up Alerts: Configure alerts based on your metrics and receive notifications.
• Collaborate: Share dashboards and allow teams to monitor their applications effectively.

Prerequisites and Server Requirements

Before we begin, ensure you have the following:

• A Valebyte Dedicated Server: This tutorial assumes you're working with a bare-metal server running a fresh installation of Ubuntu 20.04+ or Debian 11+. The steps are largely similar for CentOS/RHEL, with minor adjustments for package management (yum/dnf instead of apt).
• Root or Sudo Access: You'll need administrative privileges to install software and configure services.
• Basic Linux Command-Line Familiarity: Comfort with navigating directories, editing files, and running commands.
• Open Ports: Ensure the following ports are open in your server's firewall (e.g., UFW or iptables) to allow external access (if desired) and internal communication:
  • 9090 (Prometheus): For Prometheus web UI and scraping targets.
  • 9100 (Node Exporter): For Node Exporter to expose host metrics.
  • 3000 (Grafana): For Grafana web UI.

Firewall Configuration Example (UFW on Ubuntu/Debian):

sudo ufw allow 9090/tcp
sudo ufw allow 9100/tcp
sudo ufw allow 3000/tcp
sudo ufw enable
sudo ufw status

Step-by-Step Installation Guide

Step 1: Prepare Your Bare Metal Server

First, update your system's package list and upgrade any existing packages. This ensures you're working with the latest security patches and software versions.

sudo apt update
sudo apt upgrade -y

Install necessary utilities:

sudo apt install -y wget curl gnupg2 software-properties-common apt-transport-https

Step 2: Install Prometheus

We'll install Prometheus from its official binaries. It's recommended to create a dedicated user for Prometheus for security reasons.

2.1 Create Prometheus User and Directories

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

2.2 Download and Extract Prometheus Binaries

Visit the Prometheus download page to get the latest stable version. Replace the version number below with the current one.

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar xvf prometheus-2.47.0.linux-amd64.tar.gz
cd prometheus-2.47.0.linux-amd64

2.3 Move Binaries and Set Permissions

sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo mv consoles /etc/prometheus
sudo mv console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

2.4 Configure Prometheus

Create the Prometheus configuration file /etc/prometheus/prometheus.yml. This initial configuration will scrape Prometheus itself.

sudo nano /etc/prometheus/prometheus.yml

Add the following content:

global:
  scrape_interval: 15s # How frequently to scrape targets

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Set appropriate permissions for the configuration file:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

2.5 Create Systemd Service for Prometheus

Create a systemd service file to manage the Prometheus process.

sudo nano /etc/systemd/system/prometheus.service

Add the following:

[Unit]
Description=Prometheus Monitoring System
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address=0.0.0.0:9090
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

2.6 Start and Enable Prometheus

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

2.7 Verify Prometheus Status

Check if Prometheus is running:

sudo systemctl status prometheus

You should see an 'active (running)' status. You can also access the Prometheus web UI by navigating to http://YOUR_SERVER_IP:9090 in your browser. Go to the 'Status' -> 'Targets' page to confirm that Prometheus is scraping itself.

Step 3: Install Node Exporter (for Host Metrics)

The Node Exporter collects host-level metrics (CPU, memory, disk I/O, network I/O, etc.) and exposes them in a format Prometheus can scrape.

3.1 Create Node Exporter User and Directories

sudo useradd --no-create-home --shell /bin/false node_exporter

3.2 Download and Extract Node Exporter Binaries

Check the Prometheus download page for the latest Node Exporter version.

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvf node_exporter-1.6.1.linux-amd64.tar.gz
cd node_exporter-1.6.1.linux-amd64

3.3 Move Binaries and Set Permissions

sudo mv node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

3.4 Create Systemd Service for Node Exporter

sudo nano /etc/systemd/system/node_exporter.service

Add the following:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --web.listen-address=0.0.0.0:9100

[Install]
WantedBy=multi-user.target

3.5 Start and Enable Node Exporter

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

3.6 Verify Node Exporter Status

sudo systemctl status node_exporter

You should see an 'active (running)' status. You can also verify by visiting http://YOUR_SERVER_IP:9100/metrics in your browser, where you should see a long list of metrics.

3.7 Add Node Exporter to Prometheus Configuration

Now, tell Prometheus to scrape metrics from the Node Exporter. Edit /etc/prometheus/prometheus.yml:

sudo nano /etc/prometheus/prometheus.yml

Add a new scrape_configs entry for node_exporter:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Save and exit. Then, reload Prometheus to apply the new configuration:

sudo systemctl reload prometheus

Go to the Prometheus web UI (http://YOUR_SERVER_IP:9090) and check 'Status' -> 'Targets'. You should now see both 'prometheus' and 'node_exporter' targets listed as 'UP'.

Step 4: Install Grafana

We'll install Grafana using its official APT repository for easy updates.

4.1 Add Grafana APT Repository

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update

4.2 Install Grafana

sudo apt install -y grafana

4.3 Start and Enable Grafana

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

4.4 Verify Grafana Status

sudo systemctl status grafana-server

You should see an 'active (running)' status.

4.5 Access Grafana Web UI

Open your web browser and navigate to http://YOUR_SERVER_IP:3000. The default login credentials are:

• Username: admin
• Password: admin

You will be prompted to change the default password upon your first login. Do this immediately for security.

Step 5: Configure Grafana with Prometheus

5.1 Add Prometheus as a Data Source

1. Log in to Grafana.
2. On the left-hand menu, hover over the gear icon (Configuration) and click 'Data sources'.
3. Click 'Add data source'.
4. Select 'Prometheus' from the list.

5. Configure the data source:

   • Name: Prometheus (or any descriptive name)
   • URL: http://localhost:9090 (since Grafana is on the same server as Prometheus)
   • Access: Server (default)
6. Click 'Save & test'. You should see a message confirming 'Data source is working'.

5.2 Import a Node Exporter Dashboard

Grafana's strength lies in its dashboards. A great way to start is by importing a pre-built dashboard from the Grafana community site.

1. Go to the Grafana left-hand menu, hover over the '+' icon (Create) and click 'Import'.
2. In the 'Import via grafana.com' field, enter 1860 (this is a popular Node Exporter Full dashboard ID).
3. Click 'Load'.

4. On the next screen:

   • Name: Node Exporter Full (or similar)
   • Folder: Choose a folder or leave as 'General'.
   • Prometheus: Select the Prometheus data source you just created.
5. Click 'Import'.

You should now see a comprehensive dashboard displaying your bare metal server's CPU, memory, disk I/O, network usage, and more, all collected by Node Exporter and visualized by Grafana. Explore the panels, change time ranges, and get a feel for your server's performance.

Practical Advice for Dedicated Server Users

Beyond the basic setup, here's how to maximize your monitoring on Valebyte dedicated servers:

• Customize Dashboards: While pre-built dashboards are great, tailor them to your specific applications and services running on your dedicated server. Focus on the metrics most critical for your workload (e.g., database query times, web server request rates, game server tick rates).
• Set Up Alerts: Configure alerts in Grafana (or Prometheus with Alertmanager) for critical thresholds. Examples include high CPU usage, low disk space, excessive memory consumption, or network saturation. Integrate alerts with communication channels like email, Slack, or PagerDuty.
• Monitor Applications: Install specific Prometheus exporters for the applications running on your server. Examples include:
  • Nginx/Apache Exporter: For web server metrics.
  • MySQL/PostgreSQL Exporter: For database performance.
  • Redis Exporter: For caching server metrics.
  • Custom Application Exporters: If your application exposes metrics in Prometheus format.
• Security Best Practices:
  • Keep Prometheus and Grafana updated.
  • Restrict access to Prometheus and Grafana UIs using a firewall. Consider putting them behind a reverse proxy (like Nginx) with basic authentication or client certificates for external access.
  • Change default Grafana passwords immediately.
• Disk Space Management: Prometheus stores metrics data on disk. Monitor /var/lib/prometheus and configure retention policies in prometheus.yml (--storage.tsdb.retention.time or --storage.tsdb.retention.size) to prevent it from consuming all available storage on your dedicated server.

Real-World Use Cases for Dedicated Server Monitoring

The insights gained from your Prometheus + Grafana setup are invaluable for various dedicated server applications:

• Game Servers: Monitor CPU utilization, RAM usage, network latency, and packet loss to ensure smooth gameplay. Detect DDoS attacks or resource hogs in real-time.
• High-Traffic Web Hosting: Track web server response times, error rates, concurrent connections, and resource consumption (CPU, memory, disk I/O) to maintain optimal website performance and availability.
• Database Servers: Keep an eye on query performance, connection counts, disk I/O, cache hit ratios, and replication status to prevent slowdowns and data inconsistencies.
• Mail Servers: Monitor queue sizes, inbound/outbound traffic, spam detection rates, and resource usage to ensure reliable email delivery and prevent blacklisting.
• Streaming Platforms: Observe bandwidth usage, concurrent viewer counts, server load, and transcoding performance to guarantee a seamless streaming experience.
• CI/CD Pipelines: Track build times, resource consumption during builds, test execution times, and deployment success rates to optimize development workflows and infrastructure.
• Big Data Processing: Monitor resource utilization for Hadoop, Spark, or other big data clusters to ensure efficient job execution and resource allocation.

Troubleshooting Common Issues

1. Service Not Starting (Prometheus/Node Exporter/Grafana)

Symptom: systemctl status <service_name> shows 'failed' or 'inactive'.

Solution:

• Check Logs: Use journalctl -u <service_name> -f to view real-time logs for errors.
• Configuration Errors: For Prometheus, run promtool check config /etc/prometheus/prometheus.yml. For Grafana, check /etc/grafana/grafana.ini for syntax issues.
• Permissions: Ensure the user (prometheus, node_exporter, grafana) has read/write permissions to its respective directories (e.g., /var/lib/prometheus, /etc/prometheus).
• Port Conflicts: Ensure no other service is already listening on the required ports (9090, 9100, 3000). Use sudo netstat -tulpn | grep <port>.

2. Metrics Not Appearing in Prometheus

Symptom: 'Status' -> 'Targets' in Prometheus UI shows a target as 'DOWN' or 'N/A'.

Solution:

• Firewall: Ensure the firewall on the Prometheus server and the target server (if different) allows traffic on the exporter's port (e.g., 9100 for Node Exporter).
• Exporter Running: Verify the exporter service (e.g., node_exporter) is running on the target server.
• Prometheus Config: Double-check /etc/prometheus/prometheus.yml for correct IP addresses/hostnames and port numbers. Ensure proper indentation (YAML is sensitive).
• Reachability: From the Prometheus server, try to curl http://<target_ip>:<exporter_port>/metrics. You should see metrics output.

3. Grafana Dashboard Not Displaying Data

Symptom: Dashboard panels show 'No Data' or errors.

Solution:

• Prometheus Data Source: In Grafana, go to 'Configuration' -> 'Data sources' and ensure your Prometheus data source is configured correctly and 'Save & test' passes.
• Prometheus Functioning: Verify Prometheus itself is running and successfully scraping targets (check its web UI at http://YOUR_SERVER_IP:9090).
• Query Syntax: If you're building custom panels, ensure your PromQL queries are correct. Use the Prometheus UI's 'Graph' tab to test queries first.
• Time Range: Ensure the time range selected in Grafana (top right) covers the period for which you expect data.

4. High Disk Usage by Prometheus

Symptom: The /var/lib/prometheus directory grows very large.

Solution:

• Configure Retention: Edit your prometheus.service file. Add --storage.tsdb.retention.time=<duration> (e.g., 30d for 30 days) or --storage.tsdb.retention.size=<size> (e.g., 100GB) to the ExecStart line. Reload and restart Prometheus.
• Reduce Scrape Interval: For less critical metrics, increase the scrape_interval in prometheus.yml.
• Filter Metrics: Use relabel_configs in prometheus.yml to drop unwanted metrics before ingestion, reducing storage needs.

By systematically checking these points, you can quickly diagnose and resolve most common issues encountered during your Prometheus and Grafana setup on your dedicated server.