Installing Netdata on VPS: Real

Configuration

After installation, Netdata works "out of the box", but for more fine-grained tuning, security, and alert integration, configuration is required. The main Netdata configuration file is located at /etc/netdata/netdata.conf. Additionally, /etc/netdata/health/health_alarm_notify.conf is used for alerts, and files in /etc/netdata/conf.d/ are used for modules.

1. Main Netdata Configuration File (netdata.conf)

Open the netdata.conf file for editing. Always make a backup before making changes.

sudo cp /etc/netdata/netdata.conf /etc/netdata/netdata.conf.bak
sudo nano /etc/netdata/netdata.conf

Important Settings:

• [web] section:
  • bind to = 127.0.0.1: By default, Netdata listens on all interfaces (0.0.0.0). For security, if you plan to use a reverse proxy, it's better to bind it to the local interface so that it is only accessible from within the server.
  • default port = 19999: If you want to change the port, do it here. Don't forget to update your firewall rules.
• [global] section:
  • hostname = my-vps-server: Set a more descriptive hostname that will be displayed in the web interface.
  • history = 3600: The number of seconds of data stored in RAM. Default is 1 hour. Increasing this value requires more RAM.
  • memory mode = ram: Data storage mode. ram (default) stores everything in RAM. For long-term storage, you can use dbengine, but this will increase disk load.

Example changes in netdata.conf:

[web]
    bind to = 127.0.0.1
    default port = 19999

[global]
    hostname = my-awesome-vps
    history = 7200  # Store 2 hours of history in RAM
    memory mode = ram

After saving the changes, restart Netdata:

sudo systemctl restart netdata

2. Configuring Alerts

Netdata has a powerful alerting system. The configuration is located in /etc/netdata/health/health_alarm_notify.conf.

sudo nano /etc/netdata/health/health_alarm_notify.conf

In this file, you will find many examples for various notification methods (email, Slack, Telegram, PagerDuty, etc.).

Example Email Alert Configuration:

Find the SEND_EMAIL section and uncomment or modify the following lines:

#------------------------------------------------------------------------------
# email global notification options

# enable global EMAIL notifications
# This is the only setting you need to enable to receive email notifications.
SEND_EMAIL="YES"

# The recipient email address.
# Multiple recipients can be given, separated by spaces
DEFAULT_RECIPIENT_EMAIL="[email protected]"

# The sender email address.
# A special keyword '$HOSTNAME' can be used, which Netdata will replace with the hostname of the machine.
DEFAULT_SENDER_EMAIL="netdata@$HOSTNAME"

# The name of the mail command to use.
# It can be mail, mailx, sendmail etc.
# Check man mail for more info.
MAIL_COMMAND="mail"

To send emails, your system needs a Mail Transfer Agent (MTA), such as Postfix or Sendmail. Let's install Postfix:

sudo apt install postfix -y

During Postfix installation, select "Internet Site" and specify your domain name (or VPS hostname). After that, restart Netdata.

Example Telegram Alert Configuration:

For Telegram, you will need to create a bot via @BotFather and obtain the bot token, as well as find out the Chat ID of your chat/group. In the health_alarm_notify.conf file, find the TELEGRAM section and fill in the details:

#------------------------------------------------------------------------------
# telegram global notification options

# enable global TELEGRAM notifications
SEND_TELEGRAM="YES"

# The telegram bot token
# get a bot from @BotFather on telegram
TELEGRAM_BOT_TOKEN="YOUR_TELEGRAM_BOT_TOKEN"

# The telegram chat id.
# It can be a user, group or channel chat id.
# To find your chat id, send a message to your bot and then open:
#   https://api.telegram.org/botYOUR_TELEGRAM_BOT_TOKEN/getUpdates
TELEGRAM_CHAT_ID="YOUR_TELEGRAM_CHAT_ID"

Secrets via Environment Variables: For production systems, it's better not to store tokens and passwords directly in configuration files. Netdata supports passing secrets via environment variables. For example, you can define variables in the file /etc/systemd/system/netdata.service.d/override.conf (create the directory and file if they don't exist):

# /etc/systemd/system/netdata.service.d/override.conf
[Service]
Environment="NETDATA_ALARM_TELEGRAM_BOT_TOKEN=YOUR_TELEGRAM_BOT_TOKEN"
Environment="NETDATA_ALARM_TELEGRAM_CHAT_ID=YOUR_TELEGRAM_CHAT_ID"

After creating the file, execute:

sudo systemctl daemon-reload
sudo systemctl restart netdata

Then, in health_alarm_notify.conf, you can use these variables:

TELEGRAM_BOT_TOKEN="${NETDATA_ALARM_TELEGRAM_BOT_TOKEN}"
TELEGRAM_CHAT_ID="${NETDATA_ALARM_TELEGRAM_CHAT_ID}"

3. Configuring HTTPS with Caddy (Recommended)

Accessing Netdata over HTTP (port 19999) is insecure. For production servers, HTTPS must be configured. Caddy is a modern web server that automatically manages Let's Encrypt certificates and is very easy to set up.

Caddy Installation:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy -y

Caddyfile Configuration:

Create or edit the /etc/caddy/Caddyfile file:

# /etc/caddy/Caddyfile
yourdomain.com {
    reverse_proxy 127.0.0.1:19999
    # Optional: basic authentication
    # basicauth {
    #     username your_username
    #     password your_hashed_password # Use caddy hash-password to generate
    # }
}

Replace yourdomain.com with your actual domain name that points to your VPS IP.

If you want to add basic authentication, uncomment the basicauth lines and generate a password hash:

caddy hash-password --plaintext your_password
# Copy the generated hash and paste it into the Caddyfile.

Firewall Configuration for Caddy:

Allow HTTP (port 80) and HTTPS (port 443) through UFW:

sudo ufw allow http
sudo ufw allow https
sudo ufw reload

Restarting Caddy:

sudo systemctl reload caddy
sudo systemctl enable caddy

Now you can access Netdata at https://yourdomain.com. Caddy will automatically obtain and renew the SSL certificate from Let's Encrypt.

4. Verifying Operation

Make sure everything is working as expected:

• Check Netdata service:

sudo systemctl status netdata

• Check Netdata accessibility via local interface (if bind to = 127.0.0.1 is configured):

curl http://127.0.0.1:19999

You should receive the HTML code of the Netdata page.

• Check accessibility via Caddy (HTTPS):

curl -I https://yourdomain.com

You should see HTTP/2 200 OK and headers indicating a successful connection.

• Check alerts: You can manually trigger an alert by changing the threshold of any alert in /etc/netdata/health.d/ to a very low value so that it triggers immediately (e.g., change the RAM usage threshold to 1%). After checking, revert the value.

Backups and Maintenance

Although Netdata primarily stores data in RAM and is relatively "stateless," it's important to back up its configuration files and, potentially, custom plugins. Real-time monitoring data is usually not backed up, as it is ephemeral and can be restored after a restart.

1. What to Back Up

Key elements that should be regularly backed up:

• Netdata Configuration Files: The /etc/netdata/ directory contains all settings, including netdata.conf, alert settings health_alarm_notify.conf, and custom rules in health.d/, as well as plugin configurations in conf.d/.
• Custom Plugins: If you have created your own Netdata plugins, they might be located in /usr/libexec/netdata/plugins.d/ or /opt/netdata/usr/libexec/netdata/plugins.d/.
• Reverse Proxy Configuration: For example, /etc/caddy/Caddyfile.
• Firewall Configuration: UFW rules can be saved with the command sudo ufw status verbose or by backing up files in /etc/ufw/.

2. Simple Auto-Backup Script

Let's create a simple script that will archive important files and save them to a specified directory.

# Create a directory for backups
sudo mkdir -p /var/backups/netdata_configs

# Create the backup script
sudo nano /usr/local/bin/backup_netdata.sh

Insert the following content into the backup_netdata.sh file:

#!/bin/bash

# Directory for storing backups
BACKUP_DIR="/var/backups/netdata_configs"
# Backup file name with timestamp
TIMESTAMP=$(date +%Y%m%d%H%M%S)
BACKUP_FILE="$BACKUP_DIR/netdata_config_backup_$TIMESTAMP.tar.gz"

# Directories and files to back up
CONFIG_DIRS="/etc/netdata /etc/caddy /etc/ufw"
CUSTOM_PLUGINS="/usr/libexec/netdata/plugins.d" # Check where your custom plugins are located

# Create the archive
tar -czf "$BACKUP_FILE" $CONFIG_DIRS $CUSTOM_PLUGINS 2>/dev/null

# Delete old backups (keep the last 7 days)
find "$BACKUP_DIR" -name "netdata_config_backup_*.tar.gz" -mtime +7 -delete

echo "Netdata configuration backup created: $BACKUP_FILE"

Make the script executable:

sudo chmod +x /usr/local/bin/backup_netdata.sh

3. Configuring Cron for Automatic Backups

Add the script to the Cron scheduler so that it runs daily.

sudo crontab -e

Add the following line to the end of the file to make the script run every day at 03:00 AM:

0 3 * * * /usr/local/bin/backup_netdata.sh >> /var/log/netdata_backup.log 2>&1

This command will run the script and redirect its output (including errors) to the /var/log/netdata_backup.log file.

4. Where to Store Backups

Storing backups on the same server you are backing up is unsafe. In case of a VPS failure, you will lose both the server and the backups. It is recommended to use external storage:

• Cloud S3-compatible storage: Services like Amazon S3, MinIO, Wasabi, Backblaze B2 offer reliable and inexpensive storage. Use utilities such as s3cmd, rclone, or restic to synchronize backups.
• Separate VPS: You can set up a second, small VPS to which you will synchronize backups via rsync over SSH.
• Local storage (for a home server): External hard drive or Network Attached Storage (NAS).

Example of S3 integration using rclone:

# Install rclone
curl https://rclone.org/install.sh | sudo bash

# Configure rclone (will require interactive S3 credential input)
rclone config

# Add to backup_netdata.sh script after archive creation:
# rclone copy "$BACKUP_FILE" "my_s3_remote:netdata-backups/"

5. Updates

Regular updates are important for security and new features.

• Netdata Update: Netdata provides its own script for updating. It downloads the latest version and recompiles Netdata.

sudo /opt/netdata/usr/sbin/netdata-updater.sh

It is recommended to run it once a month or after significant releases.

• Operating System Update: Regularly update system packages.

sudo apt update && sudo apt upgrade -y

For important servers, it's better to plan "maintenance windows" for updates to minimize the risk of downtime, although for Netdata this is rarely an issue.

Troubleshooting + FAQ

Netdata is not starting or is inaccessible via IP:19999. What should I do?

First, check the status of the Netdata service: sudo systemctl status netdata. If it's not active, check the logs: journalctl -u netdata --since "1 hour ago". Ensure that port 19999 is not occupied by another process (sudo lsof -i :19999) and that the firewall (UFW) allows incoming connections to this port: sudo ufw status. If you have configured Netdata to listen only on 127.0.0.1, then direct access via IP:19999 from outside will not be possible – use a reverse proxy.

How to add custom metrics to Netdata?

Netdata is very flexible in adding custom metrics. You can use Python plugins, bash scripts, or send data directly via the Netdata API. For simple cases, it's enough to create an executable script in the /etc/netdata/charts.d/ directory (for bash) or /etc/netdata/python.d/ (for Python) that outputs data in a specific format. Netdata will automatically detect and start collecting these metrics.

Alerts are not arriving. Where should I look for the problem?

Check the /etc/netdata/health/health_alarm_notify.conf file: ensure that SEND_EMAIL="YES" (or SEND_TELEGRAM="YES") and all necessary variables (addresses, tokens) are correctly specified. Check Netdata logs for errors related to sending alerts: journalctl -u netdata. If you are using email, ensure that your MTA (e.g., Postfix) is correctly configured and capable of sending emails (check Postfix logs: tail -f /var/log/mail.log).

Netdata consumes too many resources. How to optimize?

Netdata is usually very lightweight, but when monitoring hundreds of services or storing history for a long time, its consumption can increase. In /etc/netdata/netdata.conf, you can reduce the history value (number of seconds data is stored in RAM). You can also disable unnecessary plugins in /etc/netdata/conf.d/ by creating a .conf file with the content enabled: no. For example, to disable Docker monitoring, create /etc/netdata/conf.d/go.d/docker.conf with enabled: no.

What is the minimum suitable VPS configuration?

For Netdata itself, without considering the main workload, 1 vCPU, 512 MB RAM, and 10 GB SSD are sufficient. However, since Netdata usually monitors other services, it is recommended to use a VPS with 2-4 vCPUs, 4-8 GB RAM, and 100-200 GB SSD (NVMe preferred) for stable operation of both Netdata and your main application (e.g., GitLab or Bitcoin nodes).

What to choose — VPS or dedicated for this task?

For most tasks, such as running GitLab, Mattermost, a Minecraft server, or Bitcoin/Ethereum nodes, a powerful VPS will be sufficient and more economical. Dedicated servers should be considered if you require absolute performance without virtualization overhead, huge amounts of RAM (over 64 GB), very high disk subsystem performance (e.g., NVMe RAID for databases with extreme loads), or specific hardware not available on a VPS.

How to update Netdata to the latest version?

To update Netdata, use the official update script that was installed with Netdata. Execute the command: sudo /opt/netdata/usr/sbin/netdata-updater.sh. The script will automatically download and compile the latest stable version of Netdata, and then restart the service.

Is it possible to monitor multiple servers with a single Netdata instance?

Yes, Netdata supports monitoring multiple servers. You can use Netdata Cloud for centralized management and viewing dashboards from all your servers. Alternatively, you can configure a "parent-child" mode, where several child Netdata instances send their metrics to a single parent Netdata installation for aggregation and visualization. This will require additional configuration in netdata.conf on each server.

Conclusions and Next Steps

Congratulations! You have successfully installed and configured Netdata on your VPS, providing yourself with a powerful tool for real-time resource monitoring with automatic alerts. Now you have full transparency into your server's operation, allowing you to quickly react to problems and maintain the stability of your services.

Next steps to further improve your monitoring and server operations:

• Explore advanced Netdata features: Dive into the Netdata documentation to discover capabilities for creating custom dashboards, using machine learning for anomaly detection, and integrating with other systems.
• Integrate Netdata Cloud: If you have multiple servers, consider connecting them to Netdata Cloud for centralized management, metric aggregation, and more convenient dashboard viewing from a single point.
• Optimize based on data: Use the metrics collected by Netdata to analyze your application's performance, identify bottlenecks, and make informed decisions about optimizing server configuration or application code.
• Expand monitoring: Configure Netdata to monitor specific services, such as databases (PostgreSQL, MySQL), web servers (Nginx, Apache), Docker containers, message queues, and much more, using built-in and custom plugins.