AI Performance Engineering: A GPU-to-Production Performance Evidence Ladder for Inference Bottlenecks

Why GPU utilization is not a performance diagnosis

A GPU inference service can look healthy on a dashboard and still feel slow to users. The device is busy. Requests are waiting. Tail latency is drifting. Retries are eating capacity. Some outputs never pass product acceptance. Those facts can all be true at the same time.

GPU utilization is often the least useful headline metric in an inference incident. It tells you the hardware did work. It does not tell you whether the right work completed inside the workload and quality rules the product needs.

The Wafer AI Performance Engineering V2 resource map helps because it collects the layers teams usually have to reason through, from CUDA execution and profiling to kernels, serving systems, and distributed communication. A resource map is still not an acceptance method. Teams need a path from symptom to evidence, then from evidence to a production change with a rollback plan.

For production inference, the practical unit is accepted tasks under a defined workload contract. That means p50, p95, p99, time-to-first-token, inter-token latency, queue time, prompt length, output length, concurrency, batching, error rate, retry rate, acceptance rate, and cost per accepted task should move through the analysis together. Raw tokens per second has its place in a lab run. In production, a faster route that causes more retries or fails acceptance checks is worse, not better. A lower average latency can also hide a broken tail.

This same evidence-first posture appears in related Optijara infrastructure guides, including the AI-in-RAN route acceptance test, the TensorRT Model Connect checkpoint-to-bundle acceptance test, and the Qwen3.8-27B QRAT guide. The principle is the same here. Define the evidence package before changing the system. For search-facing AI products, the discipline also pairs with Optijara's Google August spam update and GEO checklist, where source-backed claims matter more than a neat single score.

The Optijara Performance Evidence Ladder (PEL)

The Optijara Performance Evidence Ladder, or PEL, is a five-stage diagnosis method for GPU-to-production inference work. It exists to stop a common failure pattern: someone sees a symptom, picks a favorite fix, and then searches for metrics that make the fix look reasonable. PEL slows that down just enough to put evidence in the right order.

Stage 1: Request and workload trace

Start where the user feels the system. Capture request arrival rate, concurrency bands, prompt length distribution, output length distribution, batching behavior, queue time, time-to-first-token, inter-token latency, p50, p95, p99, failures, retries, and acceptance status. If the service handles different task types, split them. Retrieval calls, tool-using workflows, streaming chat, and long-context requests can hit different limits.

The output is a workload contract. It states what is being served, under which constraints, and what counts as accepted. Without it, later GPU evidence can make the wrong route faster.

Stage 2: GPU execution and memory evidence

Stage 2 asks what the device and host are actually doing. CUDA documentation gives the mental model for execution, thread hierarchy, memory hierarchy, and synchronization. Nsight Systems helps teams inspect CPU activity, CUDA API calls, GPU kernels, memory operations, and gaps between them on one timeline.

Do not chase a busier activity bar. Look for causes. A gap between kernels is not the same as memory bandwidth pressure. Synchronization delay is not the same as a slow kernel. Host-side request admission is not device-side execution. Stage 2 should name the class of evidence before anyone rewrites kernels or changes serving knobs.

Stage 3: Kernel and operator evidence

Stage 3 moves from timelines to operators and kernels. PyTorch profiler can show operator paths, CPU and CUDA time, memory behavior, and traces that reveal which model operations dominate a run. PyTorch compile diagnostics matter when graph capture, fusion, or fallback behavior changes the execution path. Triton enters the picture when a team needs to inspect custom kernels, tile shapes, memory movement, and the boundary between Python-level logic and device-level work.

This is where arithmetic intensity, memory bandwidth pressure, operator fusion, quantization paths, and kernel launch patterns belong. It is also where expensive mistakes happen. Quantization, compilation, and custom kernels can change output behavior, observability, numerical properties, and rollback cost. Test the candidate against task acceptance, not only one latency trace.

Stage 4: Inference-engine scheduling and cache evidence

Stage 4 looks at the serving engine. Systems such as vLLM expose metrics around requests, tokens, scheduler behavior, cache state, and queueing. That layer matters because LLM serving is not just model execution. It includes prefill, decode, continuous batching, request admission, KV cache pressure, prompt and output length mix, cold behavior, warm behavior, and scheduler decisions.

High time-to-first-token may come from queueing, prefill cost, cold starts, admission control, or cache state. High inter-token latency may come from decode behavior, memory pressure, kernel efficiency, scheduling, or communication. PEL keeps those hypotheses separate until the evidence points to one.

Stage 5: Distributed serving and capacity evidence

Stage 5 applies when one GPU is not the whole system. Distributed inference can involve tensor parallelism, pipeline parallelism, collectives, network movement, node-level capacity, placement, data transfer, and failure domains. NCCL documentation matters here because collectives and communication behavior can become part of the serving path.

At this point, the acceptance package should include canary scope, rollback trigger, stop conditions, and capacity evidence. A topology change is not accepted because one benchmark improves. It is accepted when the workload contract, tail latency, errors, retries, acceptance criteria, and operational risk stay inside the agreed bounds.

A bottleneck decision matrix for production inference

Stop profiling once four things are true: the bottleneck is reproduced, the responsible layer is identified, the acceptance impact is visible, and the rollback threshold is defined. Profiling can change workload behavior, so more tracing is not automatically better. The right stopping point is enough evidence to make a decision without pretending the trace is the product.

Implementation checklist: from benchmark run to accepted production change

Keep cold, warm, and canary comparisons separate. Cold runs expose startup, compilation, cache fill, or model load effects. Warm runs show steady-state behavior under the workload contract. Canary runs answer the production question: does this candidate behave well enough on real traffic without exposing the whole workload to avoidable risk?

Tool-specific evidence: where each source fits in PEL

CUDA documentation supports Stage 2 because it gives the model for threads, blocks, memory hierarchy, synchronization, and execution behavior. Nsight Systems supports the same stage from a timeline angle, especially when teams need to see CPU work, GPU kernels, CUDA APIs, memory operations, and gaps together.

PyTorch profiler supports Stage 3 by making operator-level behavior visible. PyTorch compile diagnostics can matter when a compiled route changes graph behavior, fusion, or fallback paths. Triton fits Stage 3 when standard operators are not enough or when a custom kernel path needs inspection. That does not mean every bottleneck deserves a custom kernel. PEL asks teams to prove the bottleneck lives at that layer first.

vLLM metrics fit Stage 4 because serving behavior often explains symptoms that GPU-level metrics cannot. Scheduler state, cache behavior, request metrics, token metrics, and queueing can show whether the issue is admission, batching, prefill, decode, or cache pressure. NCCL fits Stage 5 when distributed communication is part of the serving path.

What teams get wrong when optimizing GPU inference

Treating utilization as the answer

Utilization is easy to observe and easy to overread. It does not tell a team whether users are queued, whether p99 latency is acceptable, whether retries are rising, or whether the task was accepted. Use it as one signal in Stage 2, not as the headline metric.

Optimizing benchmark tokens instead of accepted tasks

Tokens per second can help inside a controlled experiment, but production systems complete tasks. A route that emits tokens quickly while failing acceptance checks, increasing errors, or requiring more retries is not better. Accepted-task throughput is the safer business-facing unit.

Changing batching without protecting tail latency

Batching and continuous batching can improve hardware use in some workloads. They also interact with queue time, TTFT, output length, and p99. Evaluate a batching change by task type and concurrency band, not only by aggregate throughput.

Ignoring cache staleness and workload drift

KV cache behavior, prompt distributions, output lengths, and retrieval patterns can drift. A run that looks good on an old workload may fail when the request mix changes. Keep workload contracts versioned and repeatable.

Profiling in ways that change the workload

Profiling adds overhead and can alter timing. That does not make profiling unusable. It means the overhead has to be labelled, profiled and unprofiled runs must be compared carefully, and trace behavior should not be treated as the system's natural state.

Caveats and trade-offs: performance work is an engineering system

Performance evidence reduces guesswork, but it does not remove implementation cost. Instrumentation takes time. Trace collection can raise privacy and security issues because prompts, outputs, identifiers, and operational metadata may be sensitive. Logs should be minimized, access-controlled, and retained only as long as needed.

Optimization can also affect quality. Quantization may change output behavior. Compilation may alter execution paths. Kernel changes may create correctness or portability issues. Scheduler and cache changes can shift latency between request types. Distributed topology changes can improve one path while adding communication cost or operational complexity elsewhere.

Provider, hardware, driver, framework, and model variance matter. Evidence from one environment should not be copied into another as a guaranteed result. PEL makes those assumptions explicit instead of hiding them behind a confident average. Cost per accepted task can be useful, but only when cost attribution is defined.

PEL in practice: a compact evidence flow and machine-readable summary

{
  "framework": "Optijara Performance Evidence Ladder",
  "stages": [
    {"stage": 1, "name": "request_trace", "signals": ["p50", "p95", "p99", "TTFT", "queue_time", "acceptance_rate"]},
    {"stage": 2, "name": "gpu_execution_memory", "signals": ["kernel_gaps", "memory_movement", "occupancy", "synchronization"]},
    {"stage": 3, "name": "kernel_operator", "signals": ["operator_time", "arithmetic_intensity", "bandwidth_pressure", "compile_path"]},
    {"stage": 4, "name": "serving_scheduler_cache", "signals": ["prefill", "decode", "KV_cache", "batching", "admission"]},
    {"stage": 5, "name": "distributed_capacity", "signals": ["collectives", "network_movement", "placement", "canary", "rollback"]}
  ],
  "acceptance_unit": "cost_per_accepted_task_under_workload_contract",
  "stop_conditions": ["reproduced_bottleneck", "owner_layer_identified", "acceptance_impact_visible", "rollback_threshold_defined"]
}

The working question for PEL is direct: which layer limits accepted tasks, what evidence proves it, and what production change can ship with a clear rollback plan? If the answer is still "the GPU is busy," the diagnosis is not finished.