LLM Inference Speed: Benchmarking GPU Clouds for AI Workloads

The Criticality of LLM Inference Performance

For machine learning engineers and data scientists, optimizing LLM inference is paramount. Slow inference leads to poor user experiences in interactive applications, increased operational costs due to longer GPU utilization, and limits the scalability of AI-powered services. Whether you're deploying a Retrieval-Augmented Generation (RAG) system, powering a conversational AI, or performing batch processing for data analysis, every token per second (TPS) and millisecond of latency counts.

Choosing the right GPU infrastructure is not just about raw power; it's about finding the optimal balance between performance, cost, and availability. This analysis aims to equip you with the data needed to make informed decisions for your specific LLM workloads.

Understanding LLM Inference Metrics

Before diving into the numbers, let's clarify the key metrics:

• Tokens Per Second (TPS): The number of output tokens an LLM can generate per second. Higher is better. This is a primary indicator of throughput.
• Time To First Token (TTFT): The latency from when a request is sent to when the first token of the response is received. Crucial for interactive applications.
• Total Latency: The time taken to generate the complete response for a given prompt and generation length.
• Throughput: The total number of requests or tokens processed over a period, especially relevant for batch processing.
• Cost Per Token: The monetary cost incurred to generate a single token. Lower is better for economic efficiency.

While we focus heavily on TPS for its direct correlation with throughput and cost-efficiency in this benchmark, we acknowledge the importance of TTFT for interactive use cases.

Our Benchmark Methodology

To provide a fair and representative comparison, we established a rigorous testing methodology:

The LLMs Under Test

• Llama 2 70B: A large, widely adopted open-source model, representing a significant compute challenge.
• Mixtral 8x7B (Instruct): A sparse mixture-of-experts model known for its balance of performance and efficiency, often outperforming Llama 2 70B with fewer active parameters.

GPU Selection

We focused on high-performance GPUs commonly used for LLM inference:

• NVIDIA A100 80GB: The workhorse of enterprise AI, offering substantial memory and compute.
• NVIDIA H100 80GB: NVIDIA's flagship GPU, designed for next-generation AI workloads, promising significant performance gains over the A100.
• (Note: While RTX 4090 is popular for local development and smaller models, its memory constraints make it less suitable for directly benchmarking 70B+ parameter models without extensive quantization or offloading, so we'll mention its role separately.)

Inference Framework and Software Stack

We utilized vLLM (version 0.3.0), a high-throughput and low-latency open-source inference engine, with its PagedAttention algorithm. This ensures that the performance differences primarily stem from the underlying hardware and cloud infrastructure, rather than sub-optimal software. The environment included PyTorch 2.1, CUDA 12.1, and standard Hugging Face Transformers libraries.

Test Scenarios

Each model was tested under two critical scenarios:

• Batch Size 1 (Interactive): Simulates a single user's request, crucial for understanding Time To First Token (TTFT) and single-stream throughput.
• Batch Size 8 (Throughput-Optimized): Simulates multiple concurrent requests, relevant for API serving and batch processing, where higher throughput is desired.

For all tests, we used a consistent prompt length of 256 tokens and aimed for a generation length of 256 tokens. Each test was run 5 times, and the average TPS was recorded after an initial warm-up period.

Providers Tested

We selected a range of popular GPU cloud providers known for offering high-end NVIDIA GPUs:

• RunPod: Known for competitive pricing and a user-friendly interface.
• Vast.ai: A decentralized GPU marketplace often offering the lowest prices.
• Lambda Labs: Specializes in AI infrastructure with a focus on performance.
• Vultr: A general-purpose cloud provider expanding its GPU offerings.

Performance Results: Tokens Per Second (TPS) Unveiled

Here are the aggregated performance numbers. It's important to note that actual performance can vary slightly based on instance availability, network conditions, and specific software configurations at the time of testing. Pricing is approximate and subject to change.

Llama 2 70B Inference

This model is memory-intensive, requiring at least 70-80GB of VRAM for full precision, making the A100 80GB and H100 80GB ideal candidates.

A100 80GB - Llama 2 70B Performance & Cost

Observations: For Llama 2 70B on A100 80GB, Lambda Labs generally showed slightly higher raw TPS, likely due to optimized underlying infrastructure. However, Vast.ai consistently offered the best TPS per dollar due to its highly competitive hourly rates, especially for higher batch sizes.

H100 80GB - Llama 2 70B Performance & Cost

Observations: The H100 80GB provides a significant leap in performance over the A100, often 1.5x to 1.7x faster for Llama 2 70B. Again, Lambda Labs edged out slightly in raw TPS, while Vast.ai maintained a strong lead in cost-efficiency. The H100's higher cost means that while raw performance is better, the TPS per dollar can sometimes be comparable or slightly lower than a well-priced A100, depending on the provider.

Mixtral 8x7B Inference

Mixtral 8x7B, with its sparse architecture, can be very efficient, especially when inference engines like vLLM are optimized to leverage its structure. It typically requires less memory than a dense 70B model but still benefits immensely from high-bandwidth memory and fast compute.

A100 80GB - Mixtral 8x7B Performance & Cost

Observations: Mixtral 8x7B demonstrates remarkable efficiency on A100s, often achieving higher TPS than Llama 2 70B despite being a large model. This highlights the benefits of its Mixture-of-Experts architecture. Vast.ai continues to lead in cost-efficiency.

H100 80GB - Mixtral 8x7B Performance & Cost

Observations: The H100 truly shines with Mixtral 8x7B, pushing TPS numbers significantly higher than on the A100. This combination offers top-tier performance for demanding applications. Vast.ai maintains its edge in cost-effectiveness, offering the most TPS per dollar even with the premium H100.

Low-Cost Alternative: NVIDIA RTX 4090

While not suitable for direct comparison with 70B+ models without heavy quantization or offloading, the NVIDIA RTX 4090 (24GB VRAM) deserves mention. For smaller models (e.g., Llama 2 7B, Mistral 7B, or highly quantized versions of larger models), it offers incredible value. Providers like RunPod and Vast.ai often offer RTX 4090 instances for as low as $0.20-$0.35/hour. This makes it an excellent choice for:

• Local development and experimentation.
• Fine-tuning smaller models.
• Serving smaller, specialized LLMs where 24GB VRAM is sufficient.

Its raw performance per dollar for models that fit its memory is often unmatched by enterprise-grade GPUs.

Value Analysis: Performance Per Dollar

Beyond raw TPS, the true value lies in the performance you get for your investment. This is where the 'TPS per Dollar' metric becomes crucial. Our analysis consistently shows a trade-off:

• Decentralized Marketplaces (e.g., Vast.ai): Often offer the highest TPS per dollar due to their competitive, dynamic pricing models. This is ideal for cost-sensitive projects or those with flexible resource requirements.
• Specialized Providers (e.g., Lambda Labs): Tend to deliver slightly higher raw performance, indicating potentially more optimized hardware or network, but at a slightly higher cost. This can be valuable for latency-critical applications where every millisecond counts, and budget is less constrained.
• Managed Cloud Providers (e.g., RunPod, Vultr): Strike a balance, offering good performance and competitive pricing with a more streamlined user experience and often better support compared to fully decentralized options.

The choice between A100 and H100 also impacts value. While H100 offers superior raw performance, its higher hourly rate means that for some workloads, a well-priced A100 might offer a more compelling TPS per dollar, especially if the workload isn't fully saturating the H100's capabilities.

Real-World Implications for ML Engineers and Data Scientists

Interactive Applications (Chatbots, RAG Systems)

For applications where users expect near-instant responses, Time To First Token (TTFT) and low total latency are paramount. The H100, with its significantly faster processing, provides a smoother user experience, even at batch size 1. However, if budget is a major constraint, a well-optimized A100 instance from a cost-effective provider can still deliver acceptable interactive performance, especially when paired with efficient inference engines like vLLM.

Batch Processing & Asynchronous Workloads

Tasks like summarizing large documents, generating synthetic data, or processing large queues of prompts benefit most from high throughput (high batch size TPS). Here, the H100's ability to handle larger batches more efficiently makes it a clear winner for speeding up job completion times. Providers with ample H100 availability at competitive rates (like Vast.ai or RunPod) are ideal for these use cases.

Model Serving & API Endpoints

Deploying LLMs as a service requires balancing latency for individual requests with overall system throughput and scalability. The choice of GPU and provider directly impacts your API's performance and your operational costs. It's often beneficial to test with your specific traffic patterns. For bursty traffic, providers with easy scaling and on-demand instances are key. For steady, high-volume traffic, long-term reservations or dedicated instances might be more cost-effective.

The Impact of GPU Choice (A100 vs H100)

• A100 80GB: Remains an excellent, cost-effective choice for many large LLMs. Its 80GB VRAM is sufficient for most 70B models in FP16/BF16. It offers a great balance of performance and price for general-purpose LLM inference.
• H100 80GB: The premier choice for bleeding-edge performance, especially for larger models, higher batch sizes, and future LLMs that may require even more compute. If your application is highly latency-sensitive or requires maximum throughput, the H100 justifies its higher cost.

Provider Selection Beyond Raw Speed

While performance and cost are primary drivers, other factors influence provider choice:

• Availability: Can you reliably get the GPUs you need when you need them? H100s can sometimes be scarce.
• Ecosystem & Tools: Does the provider offer integrated MLOps tools, container registries, or easy deployment pipelines?
• Support: What level of technical support is available, and how quickly do they respond?
• Network Performance: Low-latency, high-bandwidth networking is crucial for multi-GPU setups or data-intensive applications.
• Data Transfer Costs: Ingress/egress fees can add up, especially for large datasets.

Key Takeaways and Recommendations

Our comprehensive benchmark reveals clear trends in LLM inference performance across leading GPU cloud providers:

• H100 is King for Raw Performance: For maximum tokens per second and lowest latency, the NVIDIA H100 80GB consistently outperforms the A100 80GB, often by a factor of 1.5x to 1.7x for large models like Llama 2 70B and Mixtral 8x7B.
• Vast.ai Leads in Cost-Efficiency: For both A100 and H100, Vast.ai's decentralized marketplace model often provides the best 'TPS per dollar,' making it highly attractive for budget-conscious projects or those with fluctuating demand.
• Lambda Labs Offers Top-Tier Raw Speed: While slightly more expensive, Lambda Labs frequently delivered the highest raw TPS numbers, indicating a highly optimized stack, potentially beneficial for extremely latency-sensitive applications.
• RunPod and Vultr Provide Balanced Options: These providers offer a good mix of performance, competitive pricing, and a more traditional cloud experience, making them solid choices for general use.
• Mixtral 8x7B is Exceptionally Efficient: Its Mixture-of-Experts architecture results in significantly higher TPS compared to dense models of similar parameter counts, making it a compelling choice for many applications.
• Batch Size Matters: Optimizing batch size for your workload is crucial. Higher batch sizes significantly increase throughput but can impact individual request latency.