How to deploy a machine learning model for inference

What model serving is

Training produces a model artifact; serving makes it usable. Model serving exposes a trained model behind an interface, usually an HTTP API, so applications can send inputs and receive predictions. Production serving must be fast, reliable, and observable.

Prerequisites

• A saved model artifact
• Docker installed
• A target runtime such as Kubernetes or a serverless platform

Steps

1. Save and load the model artifact

Serialize the trained model and load it once at startup, not per request, to avoid repeated cold-load cost.

model = load_model("model.pkl")  # loaded at startup

2. Build an inference API

Expose a prediction endpoint. A lightweight framework such as FastAPI keeps it simple.

@app.post("/predict")
def predict(features: Features):
    return {"prediction": model.predict([features.values])[0]}

3. Validate inputs and handle batching

Reject malformed inputs with a clear error. For throughput, batch incoming requests so the model processes several at once.

4. Containerize the service

Package the app and its dependencies into an image so it runs identically everywhere.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

5. Deploy and autoscale

Run the container on Kubernetes with a Deployment and a horizontal autoscaler keyed to CPU or request rate, so capacity follows demand.

6. Monitor latency and drift

Track p50 and p99 latency, error rate, and throughput. Also watch input distributions; if live data drifts from training data, prediction quality degrades silently.

Verification

Send a known input to the running service and confirm the prediction matches what the model produced locally. Load-test the endpoint and confirm autoscaling adds replicas under pressure and latency stays within target.

Next Steps

Add a GPU runtime for large models, version models behind the API, run canary deployments, and connect drift alerts to retraining.