Server for big data analytics: ClickHouse, Elasticsearch

For effective big data analytics using ClickHouse and Elasticsearch, a powerful big data server is essential, featuring high-performance NVMe drives, ample RAM (from 64 GB), and a multi-core CPU (from 8 cores), capable of processing petabytes of information. Such configurations of dedicated servers or specialized VPS start from $150/month, depending on data volume and query intensity.

Which big data server to choose for ClickHouse and Elasticsearch?

Choosing the optimal server for big data analytics using ClickHouse and Elasticsearch is key to fast processing and access to information. Both solutions are powerful tools for working with Big Data but have their own peculiarities and, consequently, different hardware requirements. Understanding these differences will help you select the most suitable analytics server.

ClickHouse is a high-performance columnar DBMS designed for online analytical processing (OLAP) queries. It is ideal for aggregating large volumes of data in real-time, for example, for web analytics, monitoring, or telemetry. A ClickHouse server makes maximum efficient use of CPU and RAM resources and also requires very fast disks for writing and reading.

Elasticsearch is a distributed search and analytics system based on Apache Lucene. It is excellent for full-text search, log analysis, infrastructure monitoring, and any tasks requiring fast access to unstructured or semi-structured data in real-time. Elasticsearch hosting implies high I/O intensity and active memory usage for caching indices.

Both of these data processing server solutions require significant resources, and compromises in hardware selection can lead to a substantial decrease in performance and increased latencies.

Hardware requirements for a ClickHouse server: RAM, CPU, NVMe

ClickHouse is renowned for its ability to process billions of rows of data in seconds. Achieving such performance requires a properly configured server.

RAM for ClickHouse

ClickHouse actively uses RAM for storing intermediate query results, dictionaries, data caching, and performing complex aggregations. The more RAM, the fewer disk accesses, which is critically important for the speed of OLAP queries. The recommended RAM size depends on the size of the "hot" data you want to keep in memory and the complexity of your queries.

• Minimum: 32-64 GB for small installations (up to 1-2 TB of data).
• Optimal: 128-256 GB for medium loads (up to 5-10 TB of data).
• High load: 512 GB and more for large clusters and petabyte-scale data volumes.

Example of RAM usage configuration in ClickHouse:

<yandex>
    <max_memory_usage>100000000000</max_memory_usage> <!-- 100 GB -->
    <max_bytes_before_external_group_by>50000000000</max_bytes_before_external_group_by> <!-- 50 GB -->
</yandex>

CPU for ClickHouse

ClickHouse very efficiently uses all available CPU cores for parallel data processing. For it, the total number of cores is more important than the high clock speed of a single core.

• Minimum: 4-8 cores (e.g., Intel Xeon E3-12xx v5/v6 or equivalent).
• Optimal: 8-16 cores (e.g., Intel Xeon E5-26xx v3/v4 or AMD EPYC 73xx).
• High load: 24-48+ cores (e.g., AMD EPYC 74xx/75xx/77xx or Intel Xeon Scalable Gold/Platinum).

AMD EPYC processors often show better performance in terms of price/core count for ClickHouse.

NVMe drives for ClickHouse

Disk subsystem speed is one of the most critical factors for ClickHouse. Columnar data storage means that only necessary columns are read during queries, but these columns can be very large. NVMe drives provide the necessary throughput and low latency.

• NVMe only: Using HDDs or SATA SSDs for ClickHouse data is not recommended, as it will become a bottleneck.
• Capacity: Depends on the volume of data you plan to store. From 2 TB to 10 TB and more per node.
• RAID: For NVMe, software RAID (mdadm) RAID 0 is typically used for maximum write and read performance if data is replicated between cluster nodes. For a standalone server or systems with less fault tolerance, RAID 1 can be considered.

Example of mounting an NVMe drive:

sudo mkfs.ext4 /dev/nvme0n1
sudo mkdir /var/lib/clickhouse
sudo mount /dev/nvme0n1 /var/lib/clickhouse
sudo chown clickhouse:clickhouse /var/lib/clickhouse

Optimal server for Elasticsearch hosting and analytics

For Elasticsearch, as with ClickHouse, performance is important, but the emphasis may shift slightly towards a balance between CPU, RAM, and I/O.

RAM for Elasticsearch

Elasticsearch uses JVM (Java Virtual Machine), and its memory settings are critical. It is recommended to allocate up to 50% of available RAM for the JVM Heap (but no more than 30-32 GB), and leave the rest for the OS cache, which Elasticsearch actively uses to store indices.

• Minimum: 32 GB RAM (16 GB for JVM, 16 GB for OS cache) for small clusters or test environments.
• Optimal: 64-128 GB RAM (30-32 GB for JVM, the rest for OS cache) for most production systems.
• High load: 256 GB and more for very large clusters with high indexing and search intensity.

Example of JVM Heap configuration in jvm.options:

-Xms30g
-Xmx30g

CPU for Elasticsearch

Indexing and search operations in Elasticsearch can be quite CPU-intensive, especially when working with complex queries or large volumes of incoming data. A good balance between the number of cores and their clock speed is important.

• Minimum: 4-8 cores (e.g., Intel Xeon E3/E5).
• Optimal: 8-16 cores (e.g., Intel Xeon E5-26xx or AMD EPYC 73xx).
• High load: 24+ cores (e.g., Intel Xeon Scalable Gold/Platinum or AMD EPYC 74xx/75xx).

NVMe drives for Elasticsearch

Disk subsystem speed is critically important for Elasticsearch, especially for indexing (write) and aggregation (read) operations. NVMe drives significantly reduce response time and increase throughput.

• NVMe only: As with ClickHouse, using HDDs or SATA SSDs will lead to performance degradation.
• Capacity: Depends on the volume of data being indexed. From 1 TB to 5 TB and more per node.
• RAID: RAID 1 or RAID 10 (for a balance between performance and fault tolerance) are often used for Elasticsearch on NVMe.

The importance of NVMe drives and fast RAM for a data processing server

In the world of Big Data, where information volumes are measured in terabytes and petabytes, and queries must be executed in milliseconds, traditional hard drives (HDDs) become a critical bottleneck. This is why high-speed components are vital for any serious data processing server based on ClickHouse or Elasticsearch.

NVMe drives: The main advantage of NVMe (Non-Volatile Memory Express) over SATA SSDs, and even more so HDDs, is significantly higher throughput (read/write speed) and much lower latency. NVMe drives connect directly to the PCIe bus, bypassing SATA controllers, allowing them to achieve speeds of several gigabytes per second and hundreds of thousands of IOPS (input/output operations per second). For ClickHouse, this means fast data loading and aggregation execution, and for Elasticsearch, instant indexing and searching. Without NVMe drives, even the most powerful CPU and large amount of RAM cannot compensate for a slow disk subsystem.

Fast RAM: RAM plays a role not only in storing temporary query data but also in caching frequently used data blocks, which significantly reduces the number of disk accesses. The faster the RAM (e.g., high-frequency DDR4), the faster the processor receives the necessary data. For Elasticsearch, where the JVM actively caches indices, and for ClickHouse, where complex in-memory aggregations are performed, a large volume and high speed of RAM minimize disk operations and accelerate query execution.

Proper design of the disk subsystem, taking into account dedicated server with large disk from 1 TB to 100 TB and RAM, is the foundation for building a high-performance analytics server.

Recommended Valebyte.com configurations for your analytics server

At Valebyte.com, we offer dedicated servers optimized for Big Data tasks, including ClickHouse and Elasticsearch. Our configurations are designed with CPU, RAM, and NVMe drive requirements in mind to ensure maximum performance for your analytics server.

For higher loads or specific performance requirements, we recommend considering a powerful dedicated server: AMD EPYC and Intel Xeon for enterprise, which can be additionally equipped with high-speed network cards up to 100 Gbps. It is also worth paying attention to a dedicated server with a 10 Gbps port to ensure maximum data transfer speed.

Practical tips for choosing and optimizing a big data server

Choosing and configuring a server for big data is not a one-time task but a continuous optimization process. Here are some recommendations:

1. Start small, scale as you grow: Don't overpay for excessive resources at the start. Begin with a configuration that meets your current needs and be prepared to scale your server as load increases.
2. Monitoring is your best friend: Implement a comprehensive monitoring system (Prometheus, Grafana) to track CPU, RAM, disk subsystem (IOPS, throughput, latency), network traffic, and specific ClickHouse/Elasticsearch metrics. This will help identify bottlenecks and plan upgrades.
3. Network bandwidth: For distributed Big Data systems and clusters, a fast network port is critical. 10 Gbps is the de facto standard, and for very large data volumes, consider 25 Gbps or 40 Gbps.
4. Partitioning and sharding strategy: Proper distribution of data into partitions in ClickHouse and shards in Elasticsearch significantly improves query performance and simplifies data management.
5. Backup and recovery: Develop a reliable backup strategy. For ClickHouse, this could be file system snapshots or tools like clickhouse-backup. For Elasticsearch, use the Snapshot API.
6. Data center location: Choose a data center that is geographically close to your users or data sources to minimize latency.
7. Query optimization: Even on powerful hardware, inefficient queries can be slow. Regularly analyze and optimize queries in ClickHouse and Elasticsearch.

Conclusion

Choosing and configuring a server for big data analytics with ClickHouse and Elasticsearch requires a careful approach to hardware resources. Key factors include high-performance NVMe drives, a large amount of fast RAM, and a multi-core CPU. Valebyte.com offers specialized dedicated servers that provide optimal performance and scalability for the most demanding Big Data tasks.