Dedicated server for Elasticsearch: search and analytics

For optimal Elasticsearch performance, especially with large data volumes and high search and analytics loads, a dedicated server with a minimum of 32-64 GB RAM and fast NVMe or SSD disks is the preferred solution, ensuring stability and speed of query processing.

Elasticsearch is a powerful distributed search and analytics system capable of processing petabytes of data and executing complex queries in real-time. However, its performance directly depends on the resources on which it is deployed. Using a dedicated server for Elasticsearch not only ensures stability but also unlocks the full potential of this platform, especially when dealing with critical data and high operational intensity.

Why Elasticsearch Hosting Requires a Dedicated Server?

When it comes to deploying Elasticsearch, choosing the right hosting is crucial. Unlike shared or even many VPS solutions, an Elasticsearch dedicated server offers exclusive access to all physical server resources. This is critically important for Elasticsearch, as the system actively uses CPU for indexing and searching, RAM for caching and JVM heap, and the disk subsystem for continuous data writing and reading.

On a dedicated server, you avoid the "noisy neighbors" problem, where your system's performance degrades due to the activity of other users on the same physical hardware. Full control over hardware and software allows for fine-tuning the operating system and JVM for maximum Elasticsearch efficiency, which is not possible on standard cloud or VPS platforms. This is especially relevant for high-load systems where every millisecond matters for user experience and analytics.

Elasticsearch Server Requirements: RAM, CPU, Storage

Elasticsearch's efficiency directly depends on the server's hardware characteristics. Understanding these requirements is key to choosing the optimal solution.

RAM: The Foundation of Elasticsearch Performance

RAM is the most critically important resource for Elasticsearch. It is used for:

• JVM Heap: Elasticsearch is a Java application, and the JVM heap is used to store data that is actively processed, such as queries, aggregations, and index caches. It is recommended to allocate up to 50% of available RAM for the JVM heap, but no more than 30-32 GB, to avoid issues with compressed ordinary object pointers (OOPs).
• Filesystem Cache: The remaining RAM is used by the operating system to cache frequently used parts of indices on disk. This significantly speeds up read operations, as data is loaded from fast RAM rather than slow disk.
• Fielddata: For some types of aggregations and sorting, Elasticsearch may load field data into RAM. Although modern versions of Elasticsearch try to avoid this, Fielddata can still be a significant memory consumer.

The minimum RAM for an Elasticsearch production environment starts at 16 GB, but for most real-world scenarios, 32 GB, 64 GB, or even 128 GB and more is required per cluster node.

# Пример настройки JVM heap в jvm.options
-Xms32g
-Xmx32g

CPU: Balancing Cores and Frequency

Processor resources are important for indexing, query processing, and aggregation execution. Elasticsearch scales well with cores but also benefits from high clock speeds for single-threaded operations. It is generally recommended to choose processors with:

• High clock speed: For fast processing of individual queries.
• Multiple cores: For parallel indexing and executing multiple queries simultaneously.

For most workloads, modern Intel Xeon E3/E5/E-2xxx or AMD EPYC processors with 4-8 cores and a frequency of 3.0 GHz or higher per node provide a good balance. For very high loads, servers with 16+ cores may be required.

Storage: NVMe and SSD are Mandatory

The disk subsystem is the second most important resource after RAM. Elasticsearch actively writes and reads data from disk, especially during indexing, segment merging, and processing queries that do not fit into the cache. The use of traditional HDD disks for production Elasticsearch deployments is strongly discouraged.

• NVMe SSD: This is the best choice. NVMe drives provide maximum read/write speed and minimal latency, which is critically important for Elasticsearch performance.
• SATA SSD: A good alternative if NVMe is not available or the budget is limited. They are significantly faster than HDDs but inferior to NVMe in speed and IOPs.

The disk space must be sufficient to store all data, including indices, logs, and operational files. Always allow for extra space, as Elasticsearch can consume more than expected, especially with active indexing and replicas. You can read more about choosing disks in our article: NVMe vs SSD vs HDD: Which Disk to Choose for a Server.

Building an Elasticsearch Cluster: Shards and Replicas

To ensure high availability, fault tolerance, and scalability, Elasticsearch is almost always deployed as a cluster. Elasticsearch hosting on dedicated servers provides a reliable foundation for such a cluster.

• Shards: An index in Elasticsearch is divided into smaller, independent parts called shards. Each shard is a complete, autonomous Lucene index. Distributing shards across different cluster nodes allows Elasticsearch to scale horizontally, processing queries in parallel and distributing the load.
• Replicas: One or more replicas can be created for each primary shard. Replicas serve two purposes:
  • Fault tolerance: If a primary shard fails (e.g., due to a node crash), a replica automatically becomes the primary shard, ensuring continuous operation.
  • Read scaling: Read requests can be distributed between the primary shard and its replicas, which increases the cluster's throughput.

On a dedicated server, you can deploy individual cluster nodes (master, data, ingest nodes), each of which will have dedicated resources, which is critically important for the stability and performance of a distributed system.

Recommended Configurations for Elasticsearch Dedicated Server

The choice of Elasticsearch server configuration depends on the data volume, query intensity, and fault tolerance requirements. Here are a few typical scenarios:

Small Data Volume and Low Load (up to 50 GB of data)

• Usage: Prototyping, small applications, low-load analytical dashboards.
• Resources:
  • CPU: Intel Xeon E3/E-2xxx, 4 cores, 3.0+ GHz
  • RAM: 32 GB DDR4
  • Storage: 1x 500 GB NVMe SSD
  • Network interface: 1 Gbps
• Estimated cost: $70 - $120/month.

Medium Data Volume and Moderate Load (up to 500 GB of data)

• Usage: Enterprise applications, logging for medium projects (ELK server), internal search systems.
• Resources (per cluster node):
  • CPU: Intel Xeon E5/E-2xxx or AMD EPYC, 6-8 cores, 3.0+ GHz
  • RAM: 64 GB DDR4
  • Storage: 2x 1 TB NVMe SSD (RAID 1)
  • Network interface: 10 Gbps
• Estimated cost: $150 - $250/month.

High Load and Large Data Volumes (over 1 TB of data)

• Usage: Large-scale logging systems, full-text search for large web resources, real-time BI systems.
• Resources (per cluster node):
  • CPU: Intel Xeon E5/E-2xxx/Scalable or AMD EPYC, 8-16 cores, 2.5+ GHz
  • RAM: 128 GB DDR4 (or more)
  • Storage: 4x 2 TB NVMe SSD (RAID 10)
  • Network interface: 10 Gbps or 25 Gbps
• Estimated cost: From $300/month per node.

Comparison Table of Typical Elasticsearch Configurations

ELK Stack on a Dedicated Server: More Than Just Elasticsearch

The ELK Stack (Elasticsearch, Logstash, Kibana) is a powerful tool for collecting, processing, storing, and visualizing logs and metrics. Deploying the entire stack on an ELK server based on a dedicated server provides a synergistic effect:

• Logstash: Requires significant CPU and RAM resources for parsing and transforming data, especially with high volumes of incoming logs.
• Kibana: Although Kibana is less resource-intensive, it benefits from fast access to Elasticsearch and stable network connectivity.

A dedicated server allows you to allocate sufficient resources for each component of the ELK Stack, preventing bottlenecks and ensuring the smooth operation of the entire analytics platform. You can distribute components across different cluster nodes on dedicated servers, for example, dedicating one or more servers to Logstash, others to Elasticsearch data nodes, and a separate server to Kibana and Elasticsearch master nodes.

Valebyte.com: Your Reliable Partner for Elasticsearch Hosting

Valebyte.com offers a wide selection of dedicated servers, ideally suited for deploying Elasticsearch and ELK Stack of any complexity. We understand the specific requirements of this platform for RAM, CPU, and especially fast NVMe drives. Our servers can be configured to meet your unique needs, ensuring maximum performance and reliability.

We offer:

• Flexible configurations: Choose the number of CPU cores, RAM volume, and type/quantity of NVMe/SSD disks that precisely match your requirements.
• High-speed network: Access to 10 Gbps and 25 Gbps ports for fast data transfer between cluster nodes and external services.
• Reliable infrastructure: Modern data centers, stable power, and redundant communication channels.
• Expert support: Our team is ready to assist with server selection and configuration for your Elasticsearch project.

Practical Tips for Optimizing Elasticsearch on a Dedicated Server

After choosing a suitable Elasticsearch dedicated server, there are several key settings that will help maximize performance:

1. JVM Heap Size Configuration: Set -Xms and -Xmx in jvm.options to the same value, not exceeding 50% of total physical RAM and no more than 30-32 GB. This prevents memory overflow and allows the OS to use the remaining RAM for the filesystem cache.
2. Disabling Swapping: Elasticsearch is extremely sensitive to swapping. Ensure that swap is disabled or minimized. You can use sudo swapoff -a and configure vm.swappiness=1 in /etc/sysctl.conf.
3. Increasing File Descriptor Limits: Elasticsearch opens many files. Set ulimit -n 65536 or higher for the Elasticsearch user.
4. Filesystem Cache Configuration: Ensure that the operating system has enough free RAM for the filesystem cache (50% of total RAM). This is critically important for read speed.
5. Optimizing Shard Count: Avoid too many shards. The optimal number of shards per node depends on the data, but it is generally recommended not to exceed 20-30 shards per GB of RAM allocated for the filesystem cache.
6. Monitoring: Regularly monitor Elasticsearch performance metrics (CPU, RAM, disk I/O, JVM heap, query count). Use built-in Elasticsearch APIs (_cat, _nodes) or external monitoring systems such as Prometheus/Grafana. You can learn more about monitoring here: Server for Monitoring: Zabbix, Prometheus, Grafana.
7. Using Index Lifecycle Management (ILM): Automate index management by moving old data to slower disks or deleting it to optimize resource usage.

Conclusion

A dedicated server is the optimal choice for deploying Elasticsearch, providing the necessary performance, reliability, and control over resources, which is critically important for high-load search and analytics systems. The correct selection of hardware, with an emphasis on sufficient RAM and fast NVMe drives, along with fine-tuning the system, will fully unlock Elasticsearch's potential. Valebyte.com offers specialized dedicated servers ready for the most demanding Elasticsearch tasks.