Inference is the second phase of a generative AI model’s lifecycle :
Training : The process of learning model weights from data.
Inference : Serving generative AI models in production.
During the past decade’s machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle of training and inference for ML models.
Inference for classic ML models is relatively straightforward. In the early days of Baseten, we ran inference for models built with tools like XGBoost on lightweight CPUs with a simple software stack.
In contrast, inference for generative AI models is complex. You can’t simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. Doing inference well requires three layers :
Runtime : Optimizing the performance of a single model on a single GPU-backed instance.
Infrastructure : Scaling across clusters, regions, and clouds without creating silos, while maintaining excellent uptime.
Tooling : Providing engineers working on inference with the right level of abstraction to balance control with productivity.
These three layers must work together to create a system that can handle mission-critical inference at scale.
A complete inference stack includes runtime and infrastructure optimizations
The runtime layer is responsible for ensuring an individual model running on a GPU (or across several GPUs in a single instance) runs as performantly and efficiently as possible. This layer depends on a sophisticated software stack, from CUDA, to PyTorch, to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains.
The runtime layer relies on a number of model performance techniques that apply new research to the challenges of inference on generative AI models :
Batching : Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.
Caching : Reuse the KV cache – the cached results of the attention algorithm – between requests that share prefixes.
Quantization : Lower the precision of select pieces of the model to access more compute and reduce memory burden.
Speculation : Generate and validate draft tokens to produce more than one token per forward pass during decode.
Parallelism : Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks.
Disaggregation : Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers.
These model performance techniques are used for all modalities and not just LLMs, such as vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation, which extend the capabilities of AI systems and require their own inference optimizations. But these runtime optimizations are not enough : no matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle. This is not a CUDA problem or a PyTorch problem, it’s a systems problem that needs to be solved at the infrastructure layer.
The nature of infrastructure problems changes at each level of scale. At first, the problems are around autoscaling : knowing when to add and remove replicas, and figuring out how to do so quickly.
Past a certain scale – generally a few hundred GPUs – infrastructure problems are defined by capacity. To get access to enough GPUs, inference engineers begin spreading workloads across multiple regions and cloud providers. This quickly leads to silos, where models in one cluster may be starved for resources while other clusters have unused capacity. The final level of scale in infrastructure is a global system that treats all available resources as a single unified pool of compute.
Thoughtful multi-cloud infrastructure also improves reliability, protecting against downtime in any individual region or cloud provider. And for global applications, running inference near to end users improves end-to-end latency.
Once these runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide as the critical third layer in a complete inference platform.
Of course, developer experience is subjective. For inference, one extreme is the black box : give a platform model weights, and get back an API. At the other extreme is providing only basic constructs for compute, network, disk, and so forth.
The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, and enough abstraction to work productively.
In conclusion, according to the Pragmatic Engineer newsletter, inference is the most valuable category in the AI industry, but inference engineering, on the other hand, is still in its infancy. In their work, inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, and more reliable serving of generative AI models in production.
When ChatGPT launched in late 2022, there were perhaps a few hundred inference engineers in the world, and they didn’t call themselves that. These specialists mostly worked at frontier labs like OpenAI, Midjourney, and Anthropic, or at big tech companies like Google and NVIDIA.
