How to deploy a Kubernetes cluster on dedicated servers

Deploying a Kubernetes cluster on dedicated servers involves meticulous operating system preparation, installation of containerization components, and cluster initialization using the kubeadm utility on a minimum of three nodes, which provides full control over the infrastructure and significant cost savings compared to cloud solutions.

Amidst constantly growing demands for application scalability, reliability, and performance, Kubernetes has become the de facto standard for container orchestration. While cloud providers offer managed K8s services, deploying kubernetes on dedicated servers, or so-called kubernetes bare metal, opens the door to unparalleled performance, cost savings, and complete control over the infrastructure. This article will help you understand how to set up an efficient k8s dedicated server cluster and why it might be the optimal choice for your business.

Why Choose K8s on Dedicated Servers (Bare Metal Kubernetes)?

Choosing kubernetes bare metal for your K8s cluster offers several significant advantages that often outweigh the convenience of cloud solutions, especially for medium and large-scale projects:

• Cost-effectiveness: For significant resource volumes, dedicated servers become considerably cheaper than cloud alternatives. The absence of hourly charges for virtualization, network traffic, and I/O operations allows for reduced operational costs in the long run. You pay a fixed amount for hardware, while getting maximum return.
• Maximum Performance: Deploying Kubernetes directly on physical hardware eliminates virtualization overhead. This means your applications get full access to all CPU, RAM, and disk I/O resources, ensuring minimal latency and maximum throughput. This is critical for high-load services, databases, and computations.
• Full Control and Customization: By owning the physical server, you gain complete control over the operating system, network settings, drivers, and hardware. This allows you to precisely tune the environment to your applications' specific requirements, utilize specialized hardware (e.g., FPGA, GPU), and implement any necessary patches or modifications.
• Resource Predictability: Unlike cloud environments where resources can be "bursty" or shared with other clients (noisy neighbor effect), on a dedicated server, you always know the exact amount of resources available to your applications. This ensures stable and predictable performance.
• Security: Physical isolation from other provider clients reduces risks associated with common cloud infrastructure vulnerabilities. You control the entire stack from hardware to applications, which simplifies compliance with strict security requirements and regulations.

What Servers Are Needed for a Kubernetes Cluster on Dedicated Servers?

To create a stable and performant kubernetes cluster setup on dedicated servers, careful consideration must be given to hardware selection. While minimal requirements might be low, for a production environment and to ensure high availability (HA), it's advisable to aim for more powerful configurations.

Minimum Requirements for Each Node (Control Plane and Worker):

• Processor (CPU): Minimum 2 cores. For high-load control plane nodes or worker nodes performing resource-intensive tasks, 4+ cores are recommended.
• Random Access Memory (RAM): Minimum 2 GB for each node. For production worker nodes and control plane nodes, 4-8 GB RAM or more is recommended, depending on the number and requirements of your pods.
• Disk Subsystem: Minimum 20 GB free space. For the OS and core K8s components. It is recommended to use SSD or NVMe disks for all nodes, especially for the control plane, where etcd is stored. NVMe provides significantly higher I/O speeds, which is critical for etcd and application performance.
• Network: Each node must have a stable network connection. 1 Gbit/s Ethernet is recommended, and for high-load clusters — 10 Gbit/s. All nodes should be on the same flat network or configured to interact via VPN/tunneling.

Recommendations for a Production Environment:

• Control Plane Nodes (Master Nodes): A minimum of 3 nodes is recommended to ensure high availability of etcd (the distributed K8s database) and control components. Each such node should have at least 4 vCPU, 8-16 GB RAM, and fast NVMe SSD disks (100-200 GB).
• Worker Nodes: The number and configuration of these nodes depend on your workloads. To start, you can use 2-3 nodes with 4-8 vCPU, 16-32 GB RAM, and NVMe SSD disks (200-500 GB). As needs grow, you will be able to add new worker nodes.
• Network Hardware: For optimal performance and to minimize latency between nodes, especially in large clusters, consider using network cards supporting 10 Gbit/s or even 25/40 Gbit/s.

Why Does a K8s Cluster Require a Minimum of 3 Nodes?

To ensure high availability and fault tolerance of a kubernetes cluster setup, especially its control plane, it is critically important to use a minimum of three nodes. This requirement is related to the operation of the distributed etcd database, which stores all cluster state.

• etcd Quorum: etcd uses the Raft algorithm to ensure data consistency. To make a decision (e.g., about a new pod state), a quorum is required — a majority of votes from the total number of cluster participants.
  • In a 1-node cluster: If the node fails, quorum is lost, and the cluster becomes inoperable.
  • In a 2-node cluster: If one node fails, only one remains, which is 50% of the total. Quorum (a majority, i.e., 2 out of 2) is not achieved, and the cluster again loses functionality. This is a "split-brain" scenario where two nodes cannot agree on who is the leader.
  • In a 3-node cluster: If one node fails, two nodes remain (2 out of 3), which constitutes a majority and allows etcd to maintain quorum and continue functioning. The cluster remains operable.
• Fault Tolerance: With three control plane nodes, the cluster can survive the failure of one of them without losing functionality. This ensures the continuous operation of your applications.
• Reliability: A larger number of master nodes increases the overall reliability of the system, reducing the likelihood of a complete failure due to hardware malfunctions or software issues on one of the nodes.

While it is theoretically possible to run K8s on one or two nodes (e.g., for development), for any production environment, kubernetes on servers with dedicated resources strongly recommends using a minimum of three nodes for the control plane.

Preparing Dedicated Servers for Kubernetes Installation

Before proceeding with the kubernetes cluster setup using kubeadm, you need to perform a series of preparatory steps on each of your dedicated nodes (both control plane and worker). We will use Ubuntu Server as an example operating system.

1. System Update and Dependency Installation

sudo apt update
sudo apt upgrade -y
sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release

2. Disable Swap

Kubernetes requires Swap to be disabled for kubelet to function correctly.

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

3. Configure Kernel Parameters (sysctl)

You need to enable IPv4 traffic forwarding and allow the bridge interface to see incoming traffic.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

4. Install Container Runtime (Containerd)

Kubernetes uses the Container Runtime Interface (CRI) to interact with container runtimes. Containerd is the recommended choice.

# Add Docker GPG key repository
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y containerd.io

# Containerd configuration
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# Replace system cgroup driver with systemd
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

5. Install kubeadm, kubelet, and kubectl

These tools are necessary for deploying and managing kubernetes on servers.

# Add Kubernetes GPG key
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

# Add Kubernetes repository
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

6. Configure Firewall (UFW)

You need to open ports for Kubernetes components. On each node:

sudo ufw disable # Or configure UFW rules
# For control plane nodes:
# 6443/tcp (Kubernetes API server)
# 2379-2380/tcp (etcd server client API)
# 10250/tcp (Kubelet API)
# 10259/tcp (kube-scheduler)
# 10257/tcp (kube-controller-manager)

# For worker nodes:
# 10250/tcp (Kubelet API)
# 30000-32767/tcp (NodePort Services - default range)

Important: Ensure that all nodes can exchange traffic on these ports.

Deploying a Kubernetes Cluster with kubeadm: Step-by-Step Guide

After preparing all dedicated servers, you can proceed with the initialization of the kubernetes cluster setup using kubeadm.

On the Control Plane Node (First Master Node):

1. Initialize the Cluster

Execute this command on one of your chosen master nodes. Replace <MASTER_NODE_IP_ADDRESS> with the actual IP address of your master node. --pod-network-cidr specifies the IP address range for pods, which will be used by the network plugin (CNI), for example, Calico.

sudo kubeadm init --apiserver-advertise-address=<MASTER_NODE_IP_ADDRESS> --pod-network-cidr=192.168.0.0/16

After executing the command, you will see output containing commands for configuring kubectl and a command for joining worker nodes. Save these commands.

2. Configure kubectl for the Current User

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. Install Network Fabric (CNI)

For correct network operation between pods, you need to install a CNI plugin. We recommend Calico due to its reliability and flexibility.

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

Wait until all pods in the kube-system namespace transition to the Running state.

kubectl get pods --all-namespaces

On Worker Nodes:

1. Join the Cluster

On each worker node, execute the command you received after initializing the cluster on the control plane node. It will look approximately like this:

sudo kubeadm join <MASTER_NODE_IP_ADDRESS>:6443 --token <TOKEN> \
    --discovery-token-ca-cert-hash sha256:<HASH>

If you lose the token or hash, you can retrieve them on the control plane node:

# Get token
sudo kubeadm token create --print-join-command

# Get hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 | cut -d' ' -f2

Verify the Cluster

Return to the control plane node and check the status of all nodes:

kubectl get nodes

All nodes should be in the Ready status.

Optimal Valebyte Dedicated Server Configurations for K8s

Valebyte offers a wide selection of dedicated servers, ideally suited for deploying a kubernetes cluster setup on kubernetes bare metal. Below are several recommended configurations that will provide high performance and reliability for your K8s clusters, from entry-level to enterprise solutions.

*Prices are approximate and subject to change. For current information, please visit Valebyte.com.

When choosing a configuration for your k8s dedicated server cluster, we recommend:
For control plane nodes: aim for "K8s Start" or "K8s Pro" configurations with fast NVMe disks.
For worker nodes: choose "K8s Pro", "K8s Enterprise", or "K8s Scale" depending on your applications' CPU, RAM, and disk subsystem requirements.

Recommendations for Operating Kubernetes on Dedicated Servers

Successful deployment of kubernetes on dedicated servers is just the beginning. To ensure stable, secure, and efficient cluster operation, you need to follow a series of recommendations:

1. Resource Monitoring: Implement a comprehensive monitoring system (e.g., Prometheus + Grafana) to track node and pod status, CPU, RAM, disk usage, and network traffic. This will allow for timely identification of issues and optimization of resource utilization.
2. Centralized Logging: Set up a centralized log collection system (e.g., ELK Stack or Loki + Grafana) for all pods and system components. This will simplify debugging and analysis of application behavior.
3. Backup Strategy: Regularly create backups of the etcd state (using etcdctl snapshot save) and cluster configuration files. For persistent data, use Volume Snapshots if your CSI driver supports them.
4. Updates and Maintenance: Develop a strategy for regular updates of Kubernetes, the operating system, and the container runtime. Use tools like Kured for automatic node reboots after kernel updates.
5. Cluster Security:
   • Use RBAC (Role-Based Access Control) to restrict access to the Kubernetes API.
   • Implement Network Policies to control traffic between pods.
   • Regularly scan container images for vulnerabilities.
   • Restrict access to control plane nodes only from trusted IP addresses.
6. Persistent Storage Solutions: Since dedicated servers provide local storage, to ensure data persistence between pods and their migration, consider using solutions like Ceph (with Rook) or Longhorn. These allow you to create distributed storage on top of your nodes' local disks.
7. Deployment Automation: Use Ansible, Terraform, or other IaC (Infrastructure as Code) tools to automate node deployment and configuration. This will ensure repeatability and reduce the likelihood of errors.
8. Network Interaction Optimization: If your cluster has high network load, consider using 10 Gbit/s network cards and optimized CNI plugins.

Conclusion

Deploying kubernetes on dedicated servers provides a powerful, flexible, and cost-effective alternative to cloud solutions for container orchestration. By meticulously preparing the infrastructure, utilizing the kubeadm utility, and adhering to operational recommendations, you can create a high-performance and fault-tolerant k8s dedicated server cluster, fully adapted to your needs. Valebyte provides ideal dedicated servers for building such infrastructure, ensuring a reliable foundation for your ambitious projects.