Enterprise LLM Serving: Scaling to Millions with vLLM and Ray

📅 December 29, 2025⏱️ 55 min read🏷️ MLOps & Infrastructure

It is easy to run a Large Language Model (LLM) on your laptop. You download Ollama, run `ollama run llama3`, and you're chatting. But serving that same model to 10,000 concurrent users with sub-200ms latency is a completely different beast.

In the enterprise, the bottleneck isn't usually compute—it's memory bandwidth. The naive approach of loading a model into a Flask app and processing one request at a time (sequential processing) is catastrophically inefficient. The GPU sits idle 90% of the time waiting for data to move from VRAM to the compute cores.

This guide is for the engineers tasked with building the "inference engine" for their company. We will move beyond Hugging Face Pipelines and dive deep into vLLM, Continuous Batching, and Ray Serve to build a production-grade inference cluster that rivals OpenAI's throughput.

The Problem: LLMs are Memory Bound

Understanding LLM performance requires understanding the KV Cache. When an LLM generates the 100th token, it needs to attend to (look at) all 99 previous tokens. We cache the Key and Value matrices for these tokens so we don't have to recompute them. This is the KV Cache.

This cache grows linearly with sequence length. For long contexts (e.g., 128k tokens), the KV Cache can consume gigabytes of VRAM per request. If you allocate memory statically (reserving a contiguous block for the max possible length), you waste massive amounts of VRAM due to fragmentation. This "internal fragmentation" was the primary killer of LLM throughput until 2023.

Theory: PagedAttention & KV Cache Management

The breakthrough that put vLLM on the map is PagedAttention. It borrows a 50-year-old idea from operating systems: Virtual Memory and Paging.

How OS Paging Works vs. PagedAttention

OS: Programs think they have a contiguous block of memory. In reality, the OS breaks memory into non-contiguous "pages" and maps them via a Page Table. This eliminates external fragmentation.

PagedAttention: Instead of reserving a contiguous VRAM block for a user's entire 4k context window, vLLM breaks the KV cache into blocks (e.g., 16 tokens per block). These blocks are stored in non-contiguous physical memory. When a user generates more tokens, vLLM simply allocates a new block from anywhere in memory and updates the block table.

This results in near-zero memory waste. If a request finishes early, its blocks are freed immediately. This allows vLLM to pack 20x more concurrent requests into the same GPU compared to naive Hugging Face implementations.

Memory Mapping Visualization

Logical KV Cache (Tokens)       Physical Block Table (VRAM)
-------------------------       ---------------------------
[Seq A: Token 0-15 ]  ----->    [ Block 7  (Addr 0x00A0) ]
[Seq A: Token 16-31]  ----->    [ Block 42 (Addr 0x0FE0) ]
[Seq B: Token 0-15 ]  ----->    [ Block 2  (Addr 0x0010) ]

Just like virtual memory allows your 16GB RAM laptop to run Chrome (which thinks it has 100GB), PagedAttention allows the GPU to "oversubscribe" memory safely.

Theory: Continuous Batching (Iteration-level Scheduling)

In traditional "Static Batching," if you batch 4 requests together, the batch is only finished when the longest request finishes. If Request A generates 5 tokens and Request B generates 500 tokens, the GPU slots for Request A sit idle for 495 steps.

Continuous Batching (or Orca-style scheduling) solves this. The scheduler works at the iteration level, not the request level.

Advanced Theory: Speculative Decoding

Even with batching, LLMs are limited by the speed of memory access (memory bandwidth). Generating one token requires reading all 70B parameters from VRAM.

Speculative Decoding uses a small "Draft Model" (e.g., a 120M parameter model) to quickly guess the next 5 tokens. Then, the big "Target Model" (70B) verifies all 5 tokens in a single forward pass.

Because checking 5 tokens in parallel costs about the same as generating 1 token (due to matrix math parallelism), if the Draft Model is accurate, we get a 2x-3x speedup in latency for free.

Python Implementation: Ray Serve Cluster with vLLM

We will use Ray Serve to orchestrate our inference. Ray allows us to scale across multiple nodes and handle autoscaling.

Why Ray?

Ray provides the "Control Plane" that vLLM lacks. vLLM is just an engine (like a V8 engine). Ray is the car. Ray handles:

from typing import Dict
from ray import serve
from vllm import AsyncLLMEngine, EngineArgs, SamplingParams
import uuid

# Define the Ray deployment
@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class VLLMDeployment:
    def __init__(self):
        # Initialize vLLM engine with PagedAttention
        args = EngineArgs(
            model="meta-llama/Meta-Llama-3-8B-Instruct",
            tensor_parallel_size=1, # 1 GPU per replica
            gpu_memory_utilization=0.90, # Use 90% VRAM
            max_num_batched_tokens=4096
        )
        self.engine = AsyncLLMEngine.from_engine_args(args)

    async def __call__(self, request: starlette.requests.Request):
        json_input = await request.json()
        prompt = json_input.get("prompt")
        stream = json_input.get("stream", False)

        request_id = str(uuid.uuid4())
        sampling_params = SamplingParams(temperature=0.7, max_tokens=500)

        results_generator = self.engine.generate(
            prompt, sampling_params, request_id
        )

        if stream:
            return StreamingResponse(self._stream_results(results_generator))

        # Non-streaming wait for final result
        final_output = None
        async for request_output in results_generator:
            final_output = request_output

        return {"text": final_output.outputs[0].text}

    async def _stream_results(self, results_generator):
        async for request_output in results_generator:
            text = request_output.outputs[0].text
            yield text.encode("utf-8")

# Deploy the application
deployment = VLLMDeployment.bind()

This Python code runs inside the Ray cluster. Ray handles the HTTP ingress, load balances requests between the 2 replicas, and restarts them if they crash.

Java Implementation: The Spring Boot Gateway

Your Ray cluster should not be exposed directly to the public internet. You need a Spring Boot Gateway to handle Authentication, Rate Limiting, and Logging before forwarding to Ray.

package com.devmetrix.llmgateway;

import org.springframework.cloud.gateway.route.RouteLocator;
import org.springframework.cloud.gateway.route.builder.RouteLocatorBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;

@Configuration
public class GatewayConfig {

    @Bean
    public RouteLocator customRouteLocator(RouteLocatorBuilder builder) {
        return builder.routes()
            .route("llm_route", r -> r.path("/v1/chat/completions")
                .filters(f -> f
                    .filter(new ApiKeyAuthFilter()) // Custom Auth
                    .requestRateLimiter(c -> c.setRateLimiter(redisRateLimiter())) // Redis Rate Limiting
                    .rewritePath("/v1/chat/completions", "/") // Rewrite to match Ray's endpoint
                )
                .uri("http://ray-cluster-service:8000")) // Internal K8s DNS
            .build();
    }
}

// Simple Auth Filter Example
class ApiKeyAuthFilter implements GatewayFilter, Ordered {
    @Override
    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        String apiKey = exchange.getRequest().getHeaders().getFirst("X-API-Key");
        if (apiKey == null || !isValid(apiKey)) {
            exchange.getResponse().setStatusCode(HttpStatus.UNAUTHORIZED);
            return exchange.getResponse().setComplete();
        }
        return chain.filter(exchange);
    }
}

Security: Model Weights Protection & DoS Prevention

Running inference at scale introduces unique security challenges.

The "Model Extraction" Attack

If an attacker can query your model unlimited times with carefully crafted inputs, they can train a copycat model on your outputs (Distillation Attack). They effectively steal your IP (the fine-tuned weights) for the cost of API calls.

Defense Strategies

🌌
Purple Dream
Active Theme