Renting a GPU H100: where is it cheaper for LLM training

Renting a GPU H100 for LLM training can range from $2.50 to $6.00 per hour for PCIe versions and from $3.50 to $10.00+ per hour for high-performance SXM modifications, depending on the provider, region, instance type (on-demand or reserved), and the availability of additional resources such as NVLink and high-speed networking.

When H100 is Needed: Superiority over A100 and RTX 4090 for LLM Training

In the world of Large Language Models (LLMs), GPU performance is a critically important factor, directly influencing training speed, the size of models that can be processed, and ultimately, project cost. NVIDIA H100, based on the Hopper architecture, represents a significant leap compared to previous generations like A100 (Ampere) and consumer cards such as RTX 4090 (Ada Lovelace). But when exactly does this power become a necessity, rather than just a desirable luxury?

H100 Architectural Advantages for LLM Training

The key difference of the H100, making it indispensable for scalable LLM training, lies in its Hopper architecture. Specifically, this includes:

• Transformer Engine: A specialized mechanism designed to accelerate the training of transformer models, which form the basis of most modern LLMs. The Transformer Engine dynamically adapts to data, using FP8 and FP16 formats, significantly increasing performance without sacrificing accuracy. This is critical for models with billions of parameters.
• Fourth-generation Tensor Cores: These cores provide unprecedented performance in matrix multiplication operations, which are the foundation of deep learning. Compared to the A100, the H100 demonstrates up to 6 times faster speed in FP8 and up to 3 times in FP16.
• Fourth-generation NVLink: For multi-gigabit communication between GPUs, NVLink in H100 provides bandwidth of up to 900 GB/s per GPU (1.5 times more than A100). This allows for the creation of arrays of tens and hundreds of H100s working as a single unit, which is absolutely essential for training the largest models, such as GPT-4 or LLaMA 3.
• HBM3 Memory: The H100 is equipped with up to 80 GB of high-speed HBM3 memory with over 3.35 TB/s bandwidth. This allows larger models and batches to be loaded into memory, reducing data transfer time and accelerating training iterations.

Performance in the Context of LLM Training: H100 vs. A100 vs. RTX 4090

For smaller models or fine-tuning, where the data volume and number of parameters do not exceed a certain threshold, an A100 or even several RTX 4090s can be quite effective. However, when it comes to pre-training LLMs from scratch, training models with hundreds of billions or trillions of parameters, or working with massive datasets, the H100 becomes the undisputed choice.

• RTX 4090: An excellent card for developers and small projects. It has 24 GB of GDDR6X memory and high FP32 performance. However, it lacks specialized instructions for FP8/FP16, HBM memory bandwidth, and most importantly, NVLink for effective scaling. Attempting to train large LLMs on multiple RTX 4090s will encounter bottlenecks in inter-card communication and limited memory.
• A100: For a long time, it was the standard for cloud computing and ML. The A100 80GB offers 80 GB of HBM2e memory and third-generation Tensor Cores. It scales well but falls short of the H100 in all key metrics: Tensor Core performance, NVLink bandwidth, and memory. For medium-sized models, the A100 is still relevant, but for cutting-edge research and production, the H100 offers a significant speed advantage. A more detailed comparison and A100 rental prices can be found in our separate article.
• H100: Reduces LLM training time by orders of magnitude. NVIDIA states that the H100 provides up to 9 times higher performance in LLM training compared to the A100. This means a task that would take weeks on an A100 can be completed in days on an H100. For companies aiming to iterate quickly and bring new models to market, this is a colossal advantage.

Thus, if your project involves:

• Training LLMs from scratch, where the model has billions or hundreds of billions of parameters.
• The need for rapid fine-tuning on large datasets.
• Using the most modern architectures requiring FP8/FP16 acceleration.
• Scaling training across tens and hundreds of GPUs.

Then renting an H100, despite its higher hourly price, will likely prove to be more economically viable due to reduced overall computation time.

H100 GPU Features: SXM and PCIe Models and Their Impact on Rental Cost

When choosing to rent H100 GPU, it's important to understand that there are two main versions of this video card: H100 SXM and H100 PCIe. While both are based on the Hopper architecture and offer outstanding performance, their form factor, connectivity options, and consequently, cost and usage scenarios differ significantly.

SXM vs. PCIe Comparison: Bandwidth, Form Factor

The differences between H100 SXM and PCIe are due to their intended purpose:

• NVIDIA H100 SXM (SXM5):
  • Form Factor: A module designed for direct installation on the motherboard, typically in specialized high-density GPU servers, such as the NVIDIA DGX-H100.
  • Connectivity: Uses fourth-generation NVLink for direct connection to other GPUs in the system. Each SXM module has 18 NVLink connections, providing a cumulative bandwidth of up to 900 GB/s per GPU. This allows for the creation of virtually monolithic clusters of 8, 16, 32, and more GPUs with minimal latency and maximum data exchange speed.
  • Cooling: Usually liquid or high-efficiency air cooling, integrated into the server rack, allowing the GPU to operate at maximum power without overheating.
  • Performance: Often slightly higher due to better cooling and more stable power delivery, allowing for higher clock speeds.
• NVIDIA H100 PCIe:
  • Form Factor: A standard PCIe Gen5 x16 expansion card, similar to regular consumer graphics cards, but significantly larger and more powerful.
  • Connectivity: Plugs into a PCIe slot on the motherboard. Although it also supports NVLink, its quantity is limited (typically 4 NVLink connections per card, providing up to 600 GB/s per GPU in an 8-card configuration). Scaling to a large number of GPUs is more challenging, as bandwidth between servers is limited by the bandwidth of network cards (InfiniBand or Ethernet).
  • Cooling: Usually air-cooled, with a massive heatsink and fans.
  • Performance: Very high, but when scaling to tens of GPUs, it may fall short of SXM systems due to limitations in inter-card communication and memory bandwidth.

Impact on H100 Price and Availability for Training

Differences in architecture and form factor directly affect where and at what price you can rent H100:

• H100 SXM:
  • Higher Price: Systems with H100 SXM (e.g., NVIDIA DGX H100) are the pinnacle of engineering and are significantly more expensive to purchase, which is reflected in a higher hourly rental cost. This is a premium segment.
  • Limited Availability: Such systems are primarily offered by large cloud providers (AWS, Azure, GCP) and specialized hosting providers focused on HPC and AI. Their quantity is limited.
  • Ideal for: Large-scale LLM pre-training, where maximum bandwidth between GPUs and minimal latency are required. If your model is distributed across many GPUs, SXM systems will perform much more efficiently.
• H100 PCIe:
  • Lower Price: The hourly rental cost of H100 PCIe is generally lower than that of SXM versions. This makes them more accessible to a wider range of users.
  • Wider Availability: Offered by a larger number of providers, including cloud giants, niche GPU hosting providers, and even some dedicated server providers who can provide a server with multiple H100 PCIe cards.
  • Ideal for: Training models that can fit into the memory of one or more GPUs without extreme need for inter-card communication, fine-tuning, inference, as well as for experiments and development. If you are working with several independent tasks, each using one or more GPUs, H100 PCIe can be a more economical choice.

When choosing to rent H100, always clarify which version of the GPU the provider offers and what networking capabilities are available for scaling. This will help avoid unpleasant surprises with performance and cost.

The Real Cost of LLM Training on H100: Beyond the Hourly H100 Price

When it comes to h100 price per hour, many focus solely on the cost of the GPU itself. However, the real cost of training a large language model (LLM) on an H100 is significantly broader and includes many other factors. Ignoring these aspects can lead to serious overspending and project delays.

Factors Affecting the Total Cost of LLM Training

In addition to the hourly GPU rate, here's what else you need to consider when budgeting for H100 rental:

1. Data Storage Cost: LLM projects operate with petabytes of data. This can include training datasets, model checkpoints, and logs. Storing this data in the cloud (S3-compatible storage, block storage) has its own price, which can quickly escalate.
2. Traffic and Data Transfer: Uploading data for training, downloading results, inter-regional traffic between the GPU cluster and storage, and outbound traffic (if you provide an API) can be significant cost items. For some providers, traffic between GPUs and storage in the same zone is free, but outbound traffic is always charged.
3. CPU and RAM: Although the GPU performs the main work, the CPU and server RAM (host RAM) are necessary for data preparation, process management, operating system operations, and various libraries. Insufficient CPU/RAM can lead to GPU "starvation," where it idles while waiting for data.
4. Network Infrastructure: Effective training on multiple H100s requires a high-speed, low-latency network (InfiniBand or high-speed Ethernet). Providers offering H100 SXM usually include this in the cost, but for PCIe versions or when building your own clusters, this can be a separate expense.
5. Software Licenses: While most ML frameworks are open source, some specialized tools or proprietary libraries may require licenses.
6. Engineering Time: The most expensive resource. The time spent by engineers on environment setup, debugging, code optimization, monitoring, and results analysis must be accounted for. A faster GPU, such as the H100, reduces iteration time, thereby saving engineering time.
7. Idle Time: If you rent GPUs on an on-demand model, and they sit idle due to code errors, data issues, or lack of tasks, you still pay. Efficient resource management and automation of instance startup/shutdown are critical.
8. Monitoring and Logging: Monitoring and logging systems (e.g., Prometheus, Grafana, ELK stack) also consume resources and can be paid services in the cloud.

Approximate Calculations for Different Models and Scenarios

Let's consider a hypothetical LLM training scenario to illustrate the real cost:

Scenario: Training an LLM with 70 billion parameters (analogous to LLaMA 2 70B) on a dataset of 2 trillion tokens.

Basic Assumptions:

• Efficiency: 150 TFLOPS (FP16) on H100 SXM.
• Total operations (FLOPs) for training a 70B model on 2T tokens (using Chinchilla scaling laws): ~1400 PFLOPS-days.
• 1 H100 SXM: ~150 TFLOPS FP16.
• Required: 1400 PFLOPS-days / (0.15 PFLOPS/H100) = ~9333 H100-days.

Option 1: Using 8x H100 SXM (at $5/hour per GPU)

• Total performance: 8 * 150 TFLOPS = 1.2 PFLOPS.
• Training time: 9333 H100-days / 8 H100 = ~1166 days (this is too much for one machine, but for example).
  *Note: In reality, for such a model, many more GPUs are used to reduce the time to weeks/months.
• GPU cost: 8 H100 * $5/hour * 24 hours/day * 1166 days = ~$1,119,360.
• Additional costs (storage, traffic, CPU/RAM, engineering time): May add 20-50% to the GPU cost, i.e., ~$220,000 - $550,000.
• Total estimated cost: ~$1,340,000 - $1,670,000.

Option 2: Using 64x H100 SXM (at $5/hour per GPU)

• Total performance: 64 * 150 TFLOPS = 9.6 PFLOPS.
• Training time: 9333 H100-days / 64 H100 = ~145 days.
• GPU cost: 64 H100 * $5/hour * 24 hours/day * 145 days = ~$1,113,600.
• Additional costs: May be slightly higher due to cluster complexity, but engineering time is reduced. Approximately ~$220,000 - $550,000.
• Total estimated cost: ~$1,330,000 - $1,660,000.

As you can see, while the number of GPUs and training time vary significantly, the total cost of GPU time remains roughly the same. This is because you pay for the total amount of computation. However, using more GPUs reduces the calendar time of the project, which saves engineering time and allows for faster results.

Important Conclusion: When planning your budget for H100 for training, always consider the full picture of expenses, not just the hourly GPU rate. Optimization at each stage can lead to significant savings.

Where to Rent H100: Overview of Providers and Their Pricing Policy for H100 Rental

The market for H100 rental is dynamic, offering solutions from cloud computing giants to specialized GPU hosting providers. The choice of provider depends on your scaling needs, budget, data localization requirements, and ease of use.

Major Cloud Providers (AWS, Azure, GCP)

These providers offer the most reliable and scalable solutions, integrated into extensive ecosystems. They are ideal for large enterprises and projects requiring high availability and global presence.

• Amazon Web Services (AWS):
  • Instances: Primarily p5.48xlarge instances, equipped with 8x H100 SXM.
  • Features: Deep integration with other AWS services (S3, SageMaker, EKS), global availability, high reliability.
  • Price: From $40-$50/hour for an instance with 8x H100 (which is $5-$6.25/hour per H100 SXM) in on-demand mode. Significant discounts are available with Reserved Instances or Savings Plans.
  • Pros: Ecosystem, scalability, reliability, support.
  • Cons: Can be expensive for small projects, complex pricing, requires deep AWS knowledge.
• Microsoft Azure:
  • Instances: ND H100 v5 series, typically with 8x H100 SXM.
  • Features: Integration with Azure ML, high-performance network (InfiniBand), enterprise support.
  • Price: Similar to AWS, from $40-$50/hour for an instance with 8x H100 ($5-$6.25/hour per H100 SXM) on-demand. Reserved VM Instances are available.
  • Pros: Enterprise solutions, integration with Microsoft ecosystem, good options for large clusters.
  • Cons: Similar to AWS, high cost, complexity for beginners.
• Google Cloud Platform (GCP):
  • Instances: A3 series, also with 8x H100 SXM.
  • Features: Integration with Google Kubernetes Engine (GKE), Vertex AI, high-performance network.
  • Price: Starts from $40-$50/hour for an instance with 8x H100 ($5-$6.25/hour per H100 SXM) on-demand. Discounts are provided for sustained use and Committed Use Discounts.
  • Pros: Excellent for Kubernetes, strong ML ecosystem.
  • Cons: Expensive, may be less common in some regions.

Specialized GPU Hosting Providers

These providers focus exclusively on offering GPU resources, often providing more flexible pricing and simplified access to rent H100 GPU.

• CoreWeave:
  • Features: Specializes in GPU cloud, offering both H100 SXM and PCIe. Known for their aggressive pricing and flexibility.
  • Price: Often offer lower prices than major clouds. For H100 SXM, prices can start from $3.50-$4.50/hour, for H100 PCIe – from $2.50-$3.50/hour. Long-term contracts are often required for the best prices.
  • Pros: Competitive prices, specialized support, flexible terms.
  • Cons: Less extensive ecosystem than AWS/Azure/GCP.
• Lambda Labs:
  • Features: Another specialized provider focused on ML/AI. Offers instances with H100 (both versions).
  • Price: Similar to CoreWeave, from $3.00-$5.00/hour per H100 depending on the version and rental period.
  • Pros: Ease of use, good prices, focused on the ML community.
  • Cons: Limited number of data centers.
• RunPod:
  • Features: Decentralized GPU network, offering H100 from various owners. Allows renting individual GPUs.
  • Price: Highly variable, depends on supply and demand. H100 PCIe can be found from $2.00-$3.00/hour, but availability may not be guaranteed, especially for large clusters.
  • Pros: Low prices, flexibility, pay-as-you-go.
  • Cons: Unpredictable availability, varying quality of hardware and network, suitable for less critical tasks.
• Paperspace (CoreWeave):
  • Features: Paperspace was recently acquired by CoreWeave, so their offerings are becoming increasingly similar. Offers GPU cloud with H100.
  • Price: Similar to CoreWeave.
  • Pros: User-friendly interface, good value for money.
  • Cons: Integration with CoreWeave is still ongoing.

Valebyte, as a provider of VPS and dedicated servers, is focused on delivering high-performance computing resources. While we do not specialize exclusively in H100s, our dedicated servers can be equipped with powerful GPUs (e.g., A100 or RTX 4090) and offer flexible solutions for those seeking full control over their infrastructure and the ability to deploy their own GPU clusters. For tasks requiring high CPU performance and the option to install specialized GPUs, our dedicated servers can be an excellent choice.

On-Demand vs. Reserved Instances: How to Save on H100 GPU Rental

Choosing between on-demand and reserved instances is a key decision that can significantly impact the total cost of H100 rental. Each approach has its advantages and disadvantages, and the optimal choice depends on the nature of your LLM training project.

On-Demand: Flexibility and Instant Access to H100 for Training

On-demand instances allow you to rent GPU resources without any long-term commitments. You pay only for the time the instance is running, usually billed by the hour or even minute.

• Advantages:
  • Maximum flexibility: Launch and stop instances when you need them. Ideal for experiments, prototyping, short tasks, or projects with unpredictable workloads.
  • No commitments: No need to plan usage in advance or make large upfront payments.
  • Latest technologies: On-demand instances usually get access to the newest GPUs, such as H100, first.
• Disadvantages:
  • High cost: The hourly rate for on-demand instances is significantly higher than for reserved instances.
  • Availability issues: During peak loads or for rare instances (especially with H100 SXM), it can be difficult to obtain the necessary resources in the desired region.
  • Risk of overspending: It's easy to forget to stop an instance, leading to unnecessary costs.

When to choose On-Demand:
Use on-demand if you are just starting a project, conducting small experiments, fine-tuning, or if your workflow is highly intermittent. For example, for testing a new model architecture that only takes a few hours.

Reserved Instances / Committed Use Discounts: Savings Through Commitments

Reserved Instances (or Committed Use Discounts, Savings Plans from different providers) imply that you commit to using a certain amount of resources (e.g., one H100) for a specific period (1 year, 3 years) in exchange for a substantial discount from the on-demand price.

• Advantages:
  • Significant savings: Discounts can reach 50-70% off on-demand prices, making H100 rental much more cost-effective for long-term projects.
  • Guaranteed availability: Providers usually guarantee the availability of reserved resources.
  • Easier budgeting: You know your main GPU expenses in advance.
• Disadvantages:
  • Commitments: You are bound by a 1 or 3-year contract, even if your needs change or the project is canceled.
  • Upfront payments: Partial or full upfront payment is often required, which can be a significant barrier for startups.
  • Less flexibility: Changing the instance type or region can be difficult or impossible.

When to choose Reserved Instances:
Choose reserved instances if you have a long-term H100 training project with predictable workload. For example, if you plan to train a large model for several months or constantly perform fine-tuning and inference. This is also a good option for production systems where stable resource availability is required.

Break-even Point: When Does Reserved Become More Cost-Effective Than On-Demand?

The break-even point is when the total cost of a reserved instance becomes lower than the total cost of an equivalent on-demand instance. This depends on the discount size and commitment period, but typically ranges from 6 to 12 months of continuous use. If you plan to use H100 for more than half a year, reserved instances will almost always be more cost-effective.

Example:
If an on-demand H100 costs $5/hour, and a reserved one for 1 year with a 50% discount costs $2.50/hour.
After 1 year:

• On-demand: $5/hour * 24 hours/day * 365 days = $43,800
• Reserved: $2.50/hour * 24 hours/day * 365 days = $21,900

The savings are obvious. Even if you don't use the GPU 24/7, but say, 12 hours a day, Reserved can still be more cost-effective if the total usage time exceeds a certain threshold.

Careful analysis of your needs and usage forecasting will help you make the right choice and significantly reduce the cost of H100 rental.

How to Reduce Costs When Renting H100 for LLM Training

Training large language models on H100 is an expensive process. However, there are many strategies to optimize costs without sacrificing performance. Efficient resource management and a smart development approach can significantly reduce the total cost of H100 rental.

Optimizing Code and Models for Efficient H100 Usage

1. Use Mixed Precision Training: H100 is specifically designed for FP8 and FP16 computations. Using mixed precision (e.g., with NVIDIA Apex or PyTorch Automatic Mixed Precision) can significantly increase training speed and reduce memory consumption without losing accuracy.

   import torch
   import torch.nn as nn
   from torch.cuda.amp import autocast, GradScaler
   
   # ... your model, optimizer, data loader
   
   scaler = GradScaler()
   
   for epoch in range(num_epochs):
       for data, target in dataloader:
           optimizer.zero_grad()
           with autocast():
               output = model(data)
               loss = criterion(output, target)
           scaler.scale(loss).backward()
           scaler.step(optimizer)
           scaler.update()

2. Quantization: After training, a model can be quantized to lower precision (e.g., int8) for inference, which significantly reduces memory requirements and speeds up operations. This is less applicable for training but can be useful for fine-tuning or distillation.
3. Gradient Accumulation: If your batch size is limited by GPU memory, you can use gradient accumulation to simulate a larger batch size without increasing memory consumption. This can help utilize the H100 more effectively.

   accumulation_steps = 4
   for i, (data, target) in enumerate(dataloader):
       with autocast():
           output = model(data)
           loss = criterion(output, target) / accumulation_steps
       scaler.scale(loss).backward()
   
       if (i + 1) % accumulation_steps == 0:
           scaler.step(optimizer)
           scaler.update()
           optimizer.zero_grad()

4. Optimize Data Pipeline: Ensure that the CPU and disk subsystem are not bottlenecks. Use efficient data loaders (e.g., DataLoader with num_workers > 0), data caching, and fast disks (NVMe SSD) to feed data to the H100 without delays.
5. Choose an Optimal Model Architecture: Sometimes a smaller but more efficient model can yield comparable results to a much larger but less optimized one. Explore different architectures and their efficiency.

Efficient Resource Utilization and Provider Selection

1. Use Spot Instances / Preemptible VMs: Major cloud providers offer instances with significant discounts (up to 90%), but with the possibility of forced shutdown (preemption). This is ideal for non-critical tasks, checkpointing, or training that can be resumed from the last save.
2. Stop Instances When Not in Use: This seems obvious but is often forgotten. Automate instance shutdown using scripts or cloud functions if they are idle.
3. Choose the Right Instance Size: You don't always need to rent an 8-H100 cluster if the task can be done on one or two. Assess your needs and choose the minimally sufficient configuration.
4. Use Reserved Instances for Long-Term Tasks: As discussed earlier, for projects lasting more than 6-12 months, reserved instances provide significant savings.
5. Optimize Data Storage and Traffic:
   • Store data in the same region as the GPU cluster to avoid inter-regional traffic charges.
   • Use cheaper cold storage for rarely used data.
   • Compress data before transfer and storage.
6. Monitoring and Usage Analysis: Regularly monitor GPU (utilization, memory) and CPU metrics to identify bottlenecks and inefficient resource usage. Use tools like NVIDIA-SMI for control:

   nvidia-smi

   or for more detailed monitoring:

   watch -n 1 nvidia-smi

7. Consider Specialized GPU Hosting Providers: Providers like CoreWeave, Lambda Labs, or RunPod can offer more competitive prices for rent H100 GPU compared to major clouds, especially if you only need GPU resources without a broad ecosystem.
8. CI/CD Automation: Integrate model training into continuous integration/continuous deployment pipelines to reduce manual operations and downtime.

By applying these strategies, you can significantly reduce the cost of H100 rental, making your LLM training projects more economical and efficient.

Comparative Price Table for H100 GPU Rental from Various Providers (On-Demand)

For ease of comparison, below is an approximate table of hourly prices for H100 rental in On-Demand mode from various providers. Prices may vary depending on region, availability, and current promotions. The price listed is for one H100 GPU.

Provider	H100 Type	Approximate Price per H100 (On-Demand, $/hour)	Minimum Rental Period	Features
AWS (p5.48xlarge)	H100 SXM (80GB)	$5.00 - $6.25 (per GPU, 8 GPU instance)	Hourly	Extensive ecosystem, global presence, high reliability.
Azure (ND H100 v5)	H100 SXM (80GB)	$5.00 - $6.25 (per GPU, 8 GPU instance)	Hourly	Integration with Azure ML, enterprise solutions.
GCP (A3)	H100 SXM (80GB)	$5.00 - $6.25 (per GPU, 8 GPU instance)	Hourly	Strong ML ecosystem, Kubernetes, sustained use discounts.
CoreWeave	H100 SXM (80GB)	$3.50 - $4.50	Hourly (best prices with long-term contracts)	Specialized GPU hosting, competitive prices, flexibility.
CoreWeave	H100 PCIe (80GB)	$2.50 - $3.50	Hourly (best prices with long-term contracts)	More affordable option for individual GPUs or small clusters.
Lambda Labs	H100 SXM (80GB)	$3.00 - $5.00	Hourly	Focused on ML/AI, ease of use.
Lambda Labs	H100 PCIe (80GB)	$2.50 - $4.00	Hourly	Good value for money.
RunPod	H100 PCIe (80GB)	$2.00 - $3.00 (highly variable)	Hourly (per-minute billing)	Decentralized network, lowest prices, but variable availability.

*Prices are approximate and current at the time of writing. Always check actual rates directly with providers. Prices for SXM H100 are often quoted per instance with multiple GPUs (e.g., 8x H100), so to get the price per 1 GPU, you need to divide the total instance cost by the number of GPUs.

Conclusions: Key Recommendations for H100 Rental

For the most economical H100 rental for LLM training, first determine the scope of your project: for large-scale pre-training, choose H100 SXM from specialized providers like CoreWeave or Lambda Labs with reserved instances, which can reduce the cost to $2.50-$4.50/hour per GPU; for fine-tuning or experiments, consider H100 PCIe on RunPod or CoreWeave at prices from $2.00-$3.50/hour in on-demand mode, actively using code optimization and stopping unused resources.