VPS Security Hardening: fail2ban, SSH Keys, Closed Ports

To effectively harden your VPS security for a VPN server, you need just 15-20 minutes of basic configuration, which includes generating and using SSH keys instead of passwords, changing the default SSH port from 22 to a non-standard one, installing and configuring fail2ban for automatic attacker banning, and setting up a UFW firewall with a whitelist of allowed ports.

Why is VPS and VPN Server Security Hardening Essential?

Your VPS is the foundation for your VPN server, and like any foundation, it must be robust. Millions of server hacking attempts occur daily on the internet, and your new VPS with a public IP address becomes a target immediately after activation. Automated bots scan IP address ranges, searching for open standard ports (e.g., 22 for SSH, 80/443 for web servers) and attempting to guess passwords. If your VPS is used for a VPN, server compromise means not only the loss of your data but also its potential use for illicit activities, which can lead to IP address blocking by your hosting provider, and in the worst case, legal repercussions.

Therefore, VPS security hardening is not an option but a necessity. Even if you use your VPS exclusively for personal needs, its security is critically important. The simple yet effective measures we outline below will significantly enhance your server's protection without requiring deep cybersecurity knowledge. This is the fundamental vps security hardening that every user should implement.

Common Threats to VPS and VPN Servers

• Brute-force attacks: Attempts to guess passwords for SSH, VPN services, and control panels.
• Port scanning: Identifying open services and their vulnerabilities.
• Exploits: Leveraging known vulnerabilities in software (operating system, VPN server, web server, etc.).
• DDoS attacks: Overloading the server with traffic to cause a denial of service.
• Unauthorized access: Gaining control over the server through compromised credentials or vulnerabilities.

This is why we strongly recommend prioritizing security from the outset. If you're just planning to deploy your VPN, consult our complete guide to setting up a VPN on your VPS to incorporate security aspects from the start.

Essential VPS Security Hardening: SSH Keys and Changing the SSH Port

SSH (Secure Shell) is the primary tool for remote management of your VPS. By default, most VPS providers configure password-based access via the standard port 22. While convenient, this is highly insecure. Passwords can be easily guessed, especially if they are simple. SSH keys, conversely, offer a far more robust authentication method.

Generating and Using SSH Keys

SSH keys consist of two parts: a public key (stored on the server) and a private key (stored on your local computer). They work in tandem to provide cryptographically strong authentication.

Step 1: Generate Keys on Your Local Machine

Open your terminal (Linux/macOS) or Git Bash/WSL (Windows) and execute the command:

ssh-keygen -t rsa -b 4096 -C "ваша_почта@example.com"

You will be prompted to specify a path to save the keys (default is ~/.ssh/id_rsa) and to enter a passphrase. We strongly recommend using a passphrase — this adds an extra layer of protection for your private key. Even if someone gains access to your private key, they won't be able to use it without this passphrase.

Step 2: Copy the Public Key to Your VPS

After generation, you will have two files: `id_rsa` (the private key) and `id_rsa.pub` (the public key). Now, you need to copy the public key to your VPS. The simplest method is:

ssh-copy-id -i ~/.ssh/id_rsa.pub user@your_vps_ip

Replace `user` with your username (typically `root` or `admin`) and `your_vps_ip` with your VPS's IP address. You will be prompted to enter the user's password on the VPS. After successful copying, you will be able to log in to the server without a password, using only your SSH key.

Step 3: Disable Password Authentication

Once you've confirmed that you can log in using your SSH key, you must disable password-based login to enhance VPS security hardening. Edit the SSH server configuration file on your VPS:

sudo nano /etc/ssh/sshd_config

Find and modify the following lines (or add them if they don't exist):

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no

Save the changes and restart the SSH service:

sudo systemctl restart sshd

Now, access to your VPS is only possible using SSH keys, which significantly boosts security.

Changing the Default SSH Port

The standard SSH port — 22 — is the primary target for most automated scanners and bots. Changing the SSH port to a non-standard one (e.g., 2222, 22022, or any other in the 1024-65535 range not used by other services) will significantly reduce the number of automated attacks on your server.

Edit the same `sshd_config` configuration file:

sudo nano /etc/ssh/sshd_config

Find the line `Port 22` and change it to something like:

Port 22022

Save the changes. IMPORTANT: Before restarting SSH, ensure that the new port is allowed in your firewall (we will configure UFW later, but if you already have another firewall, add a rule for the new port). Otherwise, you will lose access to the server.

Restart the SSH server:

sudo systemctl restart sshd

Now, to connect to the server, you will need to specify the new port:

ssh -p 22022 user@your_vps_ip

Automated Brute-Force Protection: Fail2ban Configuration for VPS Security Hardening

Even after changing the SSH port and using keys, password guessing attempts can still occur if you haven't disabled password authentication for all users or if you're running other services vulnerable to brute-force attacks. This is where fail2ban comes in — a powerful tool that scans server logs for suspicious activities (e.g., multiple failed login attempts) and automatically blocks attacker IP addresses using firewall rules.

Installing and Basic Fail2ban Configuration

Installing fail2ban on most Linux distributions is straightforward:

# For Debian/Ubuntu
sudo apt update
sudo apt install fail2ban

# For CentOS/RHEL
sudo yum install epel-release
sudo yum install fail2ban

After installation, fail2ban works "out of the box" with basic protection for SSH. However, for optimal fail2ban VPS configuration, it's best to create a local configuration file.

Copy the default configuration file:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Edit `jail.local`:

sudo nano /etc/fail2ban/jail.local

In this file, locate the `[DEFAULT]` section. Here you can configure global parameters:

• `bantime`: The duration in seconds an IP address is blocked (default 10 minutes = 600 seconds). You can increase it to 1h (3600), 1d (86400), or even -1 for a permanent ban.
• `findtime`: The time window during which failed attempts must occur for an IP to be blocked (default 10 minutes = 600 seconds).
• `maxretry`: The number of failed attempts before blocking (default 5).

[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 3
destemail = [email protected] ; Optional: for receiving notifications
sendername = Fail2ban Alert
mta = sendmail

Configuring Protection for SSH and Other Services

To activate SSH protection (and other services), find the corresponding sections in `jail.local`. Ensure that the `enabled = true` option is set for `sshd`:

[sshd]
enabled = true
port = ssh,22022 ; Specify your new SSH port if you changed it
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 1d

If your VPN server (e.g., OpenVPN or WireGuard) also logs failed connection attempts, you can configure a separate "jail" for it. This might require creating a custom filter `filter.d/yourvpn.conf` and a corresponding section in `jail.local`.

After making changes, restart fail2ban:

sudo systemctl restart fail2ban

You can check the operational status of fail2ban and active "jails" with the command:

sudo fail2ban-client status

And the status of a specific "jail," for example, `sshd`:

sudo fail2ban-client status sshd

This will display blocked IP addresses and other useful information. Fail2ban VPS configuration is a key element in automated brute-force protection.

UFW Firewall: Whitelisting for Enhanced VPS Security

A firewall is your first line of defense. UFW (Uncomplicated Firewall) is a user-friendly wrapper for iptables that significantly simplifies firewall rule configuration on Linux systems. With it, you can easily implement a "whitelist" concept: deny all incoming traffic by default and only allow what you truly need.

Activating UFW and Default Rules

Installing UFW:

# For Debian/Ubuntu
sudo apt update
sudo apt install ufw

# For CentOS/RHEL (use firewalld or iptables)
# CentOS defaults to firewalld; UFW is not recommended. If UFW is still desired, install via Snap or compilation.
# For simplicity, this section focuses on Ubuntu/Debian.

After installation, you need to configure the default rules. Your UFW firewall server should be set to deny all incoming traffic and allow all outgoing traffic. This is standard and the most secure practice.

sudo ufw default deny incoming
sudo ufw default allow outgoing

Configuring Rules for VPN and SSH

Now, let's allow access to the necessary services. If you changed your SSH port to 22022, allow it:

sudo ufw allow 22022/tcp

If you're using OpenVPN, it typically operates on port 1194 (UDP) or 443 (TCP). WireGuard uses various ports, such as 51820 (UDP). Allow your VPN server's port(s):

# For OpenVPN (UDP)
sudo ufw allow 1194/udp

# For OpenVPN (TCP, if used)
sudo ufw allow 443/tcp

# For WireGuard (example port)
sudo ufw allow 51820/udp

If you're running a web server (e.g., for a VPN control panel) on ports 80 (HTTP) and 443 (HTTPS), allow those as well:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

After configuring all necessary rules, enable UFW. You will be asked if you want to proceed, as this might interrupt existing SSH connections:

sudo ufw enable

Check the firewall status:

sudo ufw status verbose

You will see a list of allowed rules. Now, your UFW firewall server reliably protects your VPS, permitting only essential traffic. This is a critically important component for VPS security hardening.

VPN Server Security: Advanced Hardening Measures

Beyond basic VPS hardening, you must also pay attention to the VPN server itself. VPN server security depends on many factors, including protocol choice, regular updates, and proper configuration.

Software Updates and Service Minimalism

Regular Updates: Always keep your operating system and all installed software up to date. Updates often include security patches. Configure automatic updates or perform them manually on a regular basis:

sudo apt update && sudo apt upgrade -y # Debian/Ubuntu
sudo yum update -y # CentOS/RHEL

Minimalism: Install only the services and packages absolutely necessary for your VPN to function. The less software installed, the fewer potential entry points for attackers. Remove any unnecessary packages.

Using Secure Protocols and Encryption

Choose modern and secure VPN protocols, such as OpenVPN with TLS or WireGuard. Avoid outdated protocols like PPTP. Ensure your VPN server uses strong encryption algorithms (e.g., AES-256) and hashing (e.g., SHA256).

• OpenVPN: Use TLS authentication, long keys (e.g., 2048-bit or 4096-bit for RSA), and strong encryption algorithms.
• WireGuard: The protocol is inherently more secure due to its simplified codebase and use of modern cryptographic primitives. Ensure private keys are stored securely and do not have excessive permissions.

Don't forget about regular automatic backups of VPN configs from your VPS. This will allow you to quickly restore server functionality in case of unforeseen issues or compromise.

Control Panels and Permissions: Limiting Access for VPS Security

Many users install control panels on their VPS (e.g., VestaCP, ISPManager, cPanel, or specialized VPN panels like Pritunl, OpenVPN Access Server, WireGuard UI). While these panels greatly simplify management, they can become a serious vulnerability if not properly secured.

Restricting Access to Control Panels and User Permissions

Never expose your control panel on a public port without restrictions! This is one of the most common mistakes. If your panel is accessible from anywhere in the world, it will immediately become a target for brute-force attacks and exploit attempts.

Solutions:

1. IP Address Restriction: Use UFW (or another firewall) to allow access to the control panel port only from your home or office IP address.

sudo ufw allow from your_home_ip to any port 8080 # Example for a panel on port 8080

2. VPN Tunnel: The best approach is to not expose the control panel to the internet at all. Connect to it via the VPN tunnel you've already set up on this VPS. That is, first connect to your VPN, and then access the control panel via its local IP address or loopback (127.0.0.1) if the panel is configured to listen only on it.
3. Two-Factor Authentication (2FA): If the panel supports it, be sure to enable 2FA.
4. Strong Passwords: Use complex, unique passwords for panel access.

User Permission Separation:

Never use the `root` user for daily tasks. Create a separate user with limited privileges and use `sudo` for administrative commands. This reduces risk if a regular user account is compromised.

sudo adduser newuser
sudo usermod -aG sudo newuser # To grant sudo privileges on Debian/Ubuntu

With these measures, VPS security hardening becomes more effective, especially when dealing with critical management interfaces.

Monitoring and Logging: Signs of Compromise and VPS Security

Even with the most stringent security measures, the risk of compromise is never zero. It's crucial to be able to recognize signs of a breach and have a logging system for incident investigation. Monitoring your VPN server on a VPS is not just about uptime but also about security.

What to Log and How to Detect Compromise

What to Log:

• Authentication Logs: `auth.log` (Debian/Ubuntu) or `secure` (CentOS/RHEL) — records all SSH login attempts, sudo, and other authentications.
• VPN Server Logs: Logs from OpenVPN, WireGuard, or other VPN services — track connections, authentication errors, and transmitted traffic.
• Firewall Logs: UFW or iptables can log dropped packets, which helps identify port scanning or attacks.
• System Event Logs: `syslog`, `kern.log` — general system messages, errors, warnings.

Signs of Compromise to Watch For:

1. Unusual SSH Activity: Numerous failed login attempts (fail2ban should handle this, but too many is a red flag), logins from unfamiliar IP addresses, unusual login times.
2. Unknown Processes: Running processes that you didn't install and whose purpose you don't know. Check with `ps aux` or `top`.
3. Unusual Network Traffic: Spikes in outbound traffic (especially to unknown IP addresses), unusual open ports (`netstat -tulnp`). A compromised server is often used for DDoS attacks, spamming, or cryptocurrency mining.
4. File System Changes: Unknown files or directories, changes in permissions for critical files, modification of system files. Use utilities like `aide` or `rkhunter` to check file integrity.
5. High CPU/RAM/Disk Usage: Unexplained high resource utilization, especially if your VPN isn't serving many users.
6. Strange VPN Behavior: Unexpected disconnects, reduced speed, inability to connect. This could also be a sign of compromise or that your VPN speed on VPS is dropping due to anomalous load.
7. Receiving Abuse Reports: If your hosting provider sends you notifications about spam, DDoS attacks, or other undesirable activity originating from your IP address, it's a clear sign that your server has been compromised and is being used for abuse.

For automated monitoring and alerts, you can use tools like Prometheus/Grafana, Zabbix, or simple scripts that send notifications via email or Telegram.

What to Do If Your VPS is Already Compromised?

If you detect signs of compromise, you must act quickly and decisively. Every minute counts in terms of potential damage.

1. Disconnect the Server from the Network: The very first step is to prevent further use of the server by attackers. You can usually do this through your hosting provider's control panel (e.g., Valebyte.com).
2. Create a Disk Snapshot: If possible, create a snapshot of the server's disk. This will allow you to preserve its current state for later analysis, if needed.
3. Change All Credentials: Change passwords for VPS access (if used), reset SSH keys, and change passwords for all services running on the server (VPN, control panels, databases, etc.).
4. Restore from a Trusted Backup: If you have an up-to-date and clean backup, this is the best option. Restore the server from it. If there's no backup, or it's outdated, you'll need to reinstall the operating system.
5. Conduct an Analysis (if possible and knowledgeable): If you have sufficient knowledge, try to analyze the logs and file system of the disk snapshot to understand how the breach occurred and what actions were taken.
6. Strengthen Security Measures: After restoration or reinstallation, be sure to apply all VPS security hardening measures described in this article: SSH keys, port change, fail2ban, UFW.
7. Notify Your Hosting Provider: Inform your hosting provider about the compromise. They may be able to assist with recovery or provide additional information.

Remember that restoring from a backup and subsequent hardening is the most reliable path. Attempts to "clean" a compromised server without a full reinstallation often result in re-compromise, as hidden backdoors can be missed.

VPS Specifications for a VPN Server: Scaling Resources

Choosing the right VPS for your VPN server also impacts its performance and ability to handle load, which indirectly affects security (e.g., during DDoS attacks). Below is a table with recommended VPS specifications based on the number of concurrent VPN users.

For 50 concurrent users, 4 vCPU, 8 GB RAM, and an 80 GB NVMe disk are sufficient.

Concurrent VPN Users	vCPU	RAM	Disk	Port	Estimated Price ($/month)
1-5 (Personal)	1	1 GB	20 GB SSD	1 Gbps	$3-5
5-20 (Small Team)	2	2-4 GB	40 GB NVMe	1 Gbps	$7-12
20-50 (Medium Team)	4	8 GB	80 GB NVMe	1 Gbps	$15-25
50-100 (Large Team/Small Business)	6-8	16 GB	160 GB NVMe	1-10 Gbps	$30-50
100+ (Enterprise)	8+	32+ GB	320+ GB NVMe	10 Gbps	$60+

Note that these are general recommendations. Actual requirements may vary depending on the chosen VPN protocol, usage intensity (constant streaming vs. occasional email checks), and additional software running on the VPS. For a more detailed selection, consider factors affecting performance, such as NVMe, RAM, or Network: What Really Matters for a VPS for Proxies and VPNs.

Frequently Asked Questions

How long does basic VPS security hardening take?

Basic VPS hardening, including setting up SSH keys, changing the SSH port, installing fail2ban, and configuring the UFW firewall, takes an average of 15 to 30 minutes. Most of this time is spent entering commands and waiting for package installations. These measures significantly enhance VPS security hardening and are critically important for any server.

Can fail2ban be used to protect services other than SSH?

Yes, fail2ban is highly flexible and can be configured to protect various services. Besides SSH, it is often used to protect web servers (Apache, Nginx), mail servers (Postfix, Dovecot), FTP servers (vsftpd), and even VPN servers, provided they log information about failed authentication attempts. This typically requires creating or using a pre-existing filter and activating the corresponding "jail" in the `jail.local` file.

What if I forget the new SSH port after changing it?

If you forget the new SSH port and cannot connect to your server, first check the `sshd_config` configuration via your hosting provider's control panel, if it offers file system access or a console. In an emergency, you can use the recovery or emergency console provided by Valebyte.com to gain access to the server and edit the `/etc/ssh/sshd_config` file, either reverting to the standard port 22 or setting a new one you'll remember. Ensure UFW allows this port.

Why is it not recommended to expose control panels on a public port?

Exposing a control panel on a public port without additional restrictions makes it an easy target for automated attacks. These panels often have web interfaces that can be vulnerable to brute-force attempts and known exploits. Opening such a port to the entire internet significantly increases the attack surface on your VPS. Restricting access by IP or using a VPN tunnel before connecting to the panel is the best practice for VPS security hardening.

Conclusion

Implementing basic VPS security hardening is a fundamental task for any server owner. Deploying SSH keys, a non-standard SSH port, fail2ban, and a UFW firewall with a whitelist takes minimal time but drastically improves the security of your VPN server. Do not neglect these simple yet effective measures to keep your VPS reliable and inaccessible to attackers, ensuring the stable operation of your VPN.