Model Serving — Triton & NIM

Where this course picks up: the MLOps Pipelines course ends with a blessed model version in a registry and a GitOps promotion chain waiting to deploy it. This course is about the thing being deployed — the inference service — and why it is a genuinely different engineering problem from both training and ordinary web services.

Three ways to consume a model

Batch (offline) inference scores a large dataset on a schedule — it is just another pipeline (your Argo skills apply verbatim) and the cheapest option when freshness allows. Online (real-time) inference answers synchronous requests inside a latency budget — the hard case, and this course's subject. Streaming inference sits between: models consuming from a queue/topic with soft-real-time deadlines. Choosing batch where the product only needs daily scores is the single biggest serving cost optimization available — make it consciously.

Why not just Flask?

Every team's first serving stack is the framework in a web app: model.predict() behind Flask/FastAPI, containerized, done. It works — and it hits four walls in production. Wall 1, GPU economics: one request at a time uses a fraction of a GPU that costs real money per hour; without batching concurrent requests into one forward pass, utilization sits in single digits. Wall 2, framework bloat: shipping all of PyTorch to run inference means multi-GB images, slow cold starts, and a big attack surface. Wall 3, ops surface: you now own health semantics, metrics, model versioning, and zero-downtime model swaps — per model, per team, in application code. Wall 4, performance: Python request handling in front of GPU work leaves large latency and throughput gains (compiled runtimes, pinned memory, CUDA streams) on the table.

A model server — Triton is the canonical example — is the specialized piece of infrastructure that solves all four once: it loads models from a repository in optimized runtimes, batches dynamically, exposes standard inference/health/metrics APIs, and manages versions — so your artifact is just the model plus a config file.

The vocabulary of the course

Four terms recur on every page, so fix them now. Latency — per-request wall time, always discussed as percentiles (p50/p99), never means. Throughput — requests (or tokens) per second per replica. Utilization — the fraction of the GPU actually working; the bridge between the first two and your bill. SLO — the latency/availability target the service commits to; every tuning decision in this course is a trade of one of these against another, and naming the trade is what separates tuning from twiddling.