aimljourney.com · by Vinit Kumar

The MLOps Stack Cheatsheet

// End-to-end MLOps lifecycle — from problem framing to retraining in production. No fluff.

Problem Design Data Engineering Train & Track Package & Deploy Monitor & Retrain
The MLOps loop — every ML system runs this cycle
Problem Design
KPI + constraints
→
Data Engineering
DVC / Airflow
→
Experiment & Train
MLflow / W&B
→
Model Packaging
Docker / ONNX
→
Deployment
K8s / KServe
→
Monitoring
Evidently / Prom
→
Retraining
CronJob / trigger
↩
// 70% of ML projects never reach production. The loop above is why the other 30% succeed.
01

Problem Definition & Design

Business KPI first
Measure what matters
Define metrics that align with business — revenue uplift, not just accuracy. Establish a simple baseline (rule-based or simple model) for comparison before writing a single line of ML code.
ROC-AUC F1 score RMSE Revenue uplift
Data & constraints
Know your inputs
List input data sources (databases, files, APIs) and output requirements (batch vs real-time, latency, throughput). Clarify constraints — latency SLAs, batch windows, data privacy, budget.
MySQL / Kafka REST API S3 / BigQuery
Design doc
Write it down first
Draft a system architecture covering data flow and ML vs infra components. Ensure data availability and quality — bias audits, privacy constraints, schema checks — before committing to the approach.
Architecture diagram Problem statement
Anti-pattern — the most common ML failure
"Let's try ML and see what happens" — is a recipe for failure. You end up with a model no one knows how to evaluate, tied to no business metric, and impossible to justify for production resources.
Always design around clear goals.
Example problem spec — lead scoring for SaaS
# Problem statement template Goal: Score leads for sales prioritisation Business KPI: Increase conversion rate by 15% Baseline: Rule-based: recency + firmographics Data sources: - CRM database (MySQL) — contact history - User events (Kafka) — product engagement - Firmographic API — company size / ARR Output: Real-time API — score 0–100 per lead Latency SLA: <200ms P95 Retrain: Weekly batch retraining on new labels Privacy: GDPR — PII masked before ingestion
02

Data Engineering

Data versioning — DVC
# Initialize DVC inside a Git repo $ dvc init $ git add .dvc .gitignore && git commit -m "Init DVC" # Track a data file $ dvc add data/raw/data.csv $ git add data/raw/data.csv.dvc # Configure S3 remote and push $ dvc remote add -d origin s3://mybucket/mlops-data $ dvc push # Retrieve exact version on another clone $ dvc pull
Data validation — Great Expectations
# Set up a new expectation suite $ great_expectations init $ great_expectations suite new # Run validation checkpoint $ great_expectations checkpoint run \ --checkpoint-name default # Expectation YAML example expect_table_row_count_to_be_between: min_value: 1000 max_value: 100000 expect_column_values_to_not_be_null: column: user_id
Data pipeline — Apache Airflow DAG
from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime def extract(): # fetch data from source (DB, API, Kafka) pass def transform(): # clean and feature-engineer data pass with DAG('data_pipeline', start_date=datetime(2024,1,1), schedule_interval='@daily') as dag: t1 = PythonOperator(task_id='extract', python_callable=extract) t2 = PythonOperator(task_id='transform', python_callable=transform) t1 >> t2
Schema & lineage
  • Maintain schema files (JSON/Avro) to validate inputs
  • Record data lineage via Git tags and DVC commits
  • Log dataset version in every training run
  • Use lakeFS for git-like branching on object stores
  • Never train without knowing what data version was used
Common data failure modes
  • Schema drift — upstream changes column names/types
  • Label leakage — future data leaks into training features
  • Null explosion — nulls not caught, silently corrupt features
  • Distribution shift — train data no longer reflects prod
  • PII in training — GDPR violation risk
Data engineering checklist
  • Data versioned with DVC or lakeFS
  • Schema validated before every pipeline run
  • Row count and null rate checks as gates
  • Lineage tracked end-to-end
  • Pipeline is idempotent and retryable
  • PII masked before storage
03

Experimentation & Training

MLflow — experiment tracking
# Set tracking URI and start experiment $ export MLFLOW_TRACKING_URI=http://mlflow-server:5000 $ mlflow ui --port 5000 import mlflow with mlflow.start_run(): mlflow.log_param("alpha", alpha) mlflow.log_param("l1_ratio", l1_ratio) mlflow.log_metric("rmse", rmse) mlflow.log_metric("latency_p95_ms", 342) # log model to registry mlflow.sklearn.log_model(model, "model")
Weights & Biases — sweeps & tracking
# CLI setup $ wandb login $ wandb sweep sweep.yml $ wandb agent user/proj/sweepid import wandb wandb.init(project="my-project") wandb.config.update({"learning_rate": 0.01}) wandb.log({"accuracy": acc, "loss": loss}) # Always log these wandb.log({ "git_sha": git_sha, "dataset_version": dvc_tag, "random_seed": seed })
Hyperparameter tuning — Optuna
import optuna def objective(trial): lr = trial.suggest_loguniform('lr', 1e-5, 1e-1) n_estimators = trial.suggest_int('n_estimators', 50, 300) model = train_model(lr, n_estimators) return evaluate(model) study = optuna.create_study(direction='maximize') study.optimize(objective, n_trials=50) # Other libraries: Ray Tune, Hyperopt
What to always log
  • Random seed for reproducibility
  • Git SHA (code version)
  • Dataset version (DVC tag)
  • All hyperparameters
  • All evaluation metrics
  • Training duration + hardware
Model registry — promotion flow
Experiment
→
Registered
→
Staging
→
Production
client = mlflow.MlflowClient() client.transition_model_version_stage( name="my-model", version="3", stage="Production", archive_existing_versions=True )
Common training errors
  • Version mismatch — training vs serving framework differ
  • GPU/CPU inconsistency — results differ across hardware
  • Unstandardised env — use conda or Docker always
  • Seed not set — results not reproducible next run
  • No baseline — can't tell if model actually improves
04

Model Packaging

Serialization — pick your format
# scikit-learn import joblib joblib.dump(model, "model.pkl") # PyTorch — state dict or TorchScript torch.save(model.state_dict(), "model.pth") scripted = torch.jit.script(model) scripted.save("model_scripted.pt") # TensorFlow SavedModel model.save("./saved_model") # ONNX — cross-framework serving torch.onnx.export(model, example, "model.onnx", opset_version=13)
Containerization — Docker
# Dockerfile FROM python:3.9-slim COPY model.pkl /app/model.pkl COPY requirements.txt /app/ WORKDIR /app RUN pip install -r requirements.txt COPY serve.py /app/ CMD ["python", "serve.py"] # Build and run $ docker build -t my-ml-model:latest . $ docker run -p 8000:8000 my-ml-model:latest # Or use MLflow to build directly $ mlflow models build-docker \ -m runs:/<run_id>/model \ -n myuser/model-image --enable-mlserver
FastAPI inference server
from fastapi import FastAPI import joblib, numpy as np model = joblib.load("model.pkl") app = FastAPI() @app.post("/predict") def predict(features: dict): x = np.array(list(features.values())) y = model.predict([x]) return {"prediction": y.tolist()} # run: uvicorn serve:app --port 8000
Environment capture
  • Always pin exact package versions
  • Use pip freeze > requirements.txt
  • Or use conda env export
  • Test container loads model on fresh build
  • Serving env must match training env exactly
Packaging pitfalls
  • Unpickleable objects in sklearn pipeline
  • CUDA version mismatch — model crashes on serve
  • Missing preprocessing step — features not transformed
  • Model file not in image — silent load failure
  • Fix: fresh docker run test in every CI build
05

Deployment

Deployment types — pick based on latency requirement
Batch
nightly scoring
Scheduled jobs. High throughput. No latency constraint. Use Airflow + Spark.
Real-Time API
<200ms P95
REST/gRPC endpoint. FastAPI, TorchServe, TF Serving, KServe, Seldon.
Streaming
Kafka / Kinesis
Online predictions on data streams. Faust/Flink consumers call model API.
KServe InferenceService — Kubernetes-native
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "my-model"
  namespace: "mlops"
spec:
  predictor:
    containers:
    - name: "predictor"
      image: "myregistry/my-mlflow-model:latest"
      ports:
      - containerPort: 8080
      env:
      - name: "PROTOCOL"
        value: "v2"

$ kubectl apply -f inference_service.yaml
$ kubectl get inferenceservice my-model
Kubernetes Deployment + Service
apiVersion: apps/v1
kind: Deployment
metadata: { name: ml-model }
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: ml-model
        image: myregistry/my-ml-model:latest
        ports: [{ containerPort: 8000 }]
---
apiVersion: v1
kind: Service
metadata: { name: ml-model-svc }
spec:
  type: LoadBalancer
  ports: [{ port: 80, targetPort: 8000 }]
  selector: { app: ml-model }
Deployment strategies — when to use which
StrategyHow it worksBest for
Blue/GreenNew env in parallel; traffic switch after validationInstant full switch + rollback capability
CanaryRoll out to small % of users first; expand graduallyLarge-scale tech platforms, high-stakes APIs
ShadowReal traffic to new model, predictions logged not servedHigh-risk domains — fraud, finance, healthcare
A/B TestingSplit traffic; measure business KPI per variantE-commerce, ads, recommendations
Champion–ChallengerChallenger competes against champion on live trafficFraud scoring, risk modelling, finance
Common deployment errors
  • Version mismatch — serving env != training env
  • Resource undersize — OOM kills, slow inference
  • Cold start — container not ready, first request fails
  • Networking — load balancer / TLS misconfiguration
TorchServe quick reference
# Archive model $ torch-model-archiver \ --model-name mynet \ --version 1.0 \ --serialized-file model.pth # Start server $ torchserve --start \ --model-store model_store \ --models mynet=mynet.mar # Stop $ torchserve --stop
Cloud ML services
  • SageMaker — model registry + endpoints, AWS-native
  • Azure ML — managed endpoints, MLflow integration
  • Vertex AI — GCP pipelines + model monitoring
  • All abstract K8s complexity — useful for fast prototyping
  • Still understand the underlying commands
06

CI/CD & Continuous Training

GitHub Actions — ML pipeline with MLflow
name: CI with MLflow
on:
  push:
    branches: ["master"]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-python@v3
      with:
        python-version: '3.9'
    - name: Install dependencies
      run: |
        pip install -r requirements.txt
    - name: Run training
      env:
        MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }}
      run: |
        python train.py
    - name: Build and push Docker image
      run: |
        docker build -t myregistry/ml-model:$GITHUB_SHA .
        docker push myregistry/ml-model:$GITHUB_SHA
    - name: Deploy to Kubernetes
      run: |
        kubectl set image deployment/ml-model \
          ml-model=myregistry/ml-model:$GITHUB_SHA
Pipeline stages
CI
Continuous Integration
On code push: unit tests, data validation, training pipeline run on small sample. Gate: 100% tests pass.
GitHub Actions / Jenkins / GitLab CI
CD
Continuous Delivery
If CI passes: build Docker image, scan for CVEs, push to registry, deploy to staging, run smoke tests.
Docker + Trivy + Helm + kubectl
CT
Continuous Training
Trigger retraining on new data availability, schedule, or drift detection event. Feed new model back into CD.
Airflow / CronJob / EventBridge
Kubeflow / Argo Workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ml-pipeline-
spec:
  entrypoint: train-pipeline
  templates:
  - name: train-pipeline
    steps:
    - - name: train
        template: train-task
  - name: train-task
    container:
      image: myregistry/my-ml-image
      command: [python, train.py]
      env:
      - name: MLFLOW_TRACKING_URI
        value: "http://mlflow-server:5000"
Quality gates — never skip
  • Data schema validation before training
  • Model performance beats current prod by ≥2%
  • Latency SLA validated in staging
  • CVE scan — no critical vulnerabilities
  • Smoke test — 50 requests, <0.5% errors
Rollback — one command
# Helm rollback $ helm rollback ml-service 0 -n prod # kubectl pin to last good SHA $ kubectl set image deployment/ml-model \ ml-model=myregistry/ml-model:$LAST_GOOD_SHA # Always test rollback in staging monthly
Prompt version control
Treat prompts like code — version, test, never change prod prompts without a PR.
# track prompt in MLflow mlflow.log_artifact( "prompts/v2/system.txt") mlflow.log_param( "prompt_version", "v2.3.1")
07

Monitoring & Metrics

Golden signals — what to always monitor
Model Metrics
Accuracy / F1 / RMSE
AUC on incoming data
Confidence distribution
Hallucination rate (LLMs)
Data Metrics
Feature distribution drift
Schema changes
Missing value rate
Label delay / availability
System Metrics
Latency P50/P95/P99
Throughput (req/s)
5xx error rate
GPU/CPU utilisation
Business Metrics
Conversion rate
Revenue impact
CPC / ROAS
User retention
Monitoring stack layers
Drift Detection — Evidently AI
Compute PSI and KL divergence on feature distributions. Compare reference dataset vs current window. Generate HTML dashboard or push metrics to Prometheus via pushgateway.
Metrics — Prometheus + Grafana
Expose /metrics endpoint via prometheus_client. Scrape config in prometheus.yml. Set alerts on P95 latency, error rate, and model accuracy SLOs.
Alerting — Alertmanager
Alert on: model accuracy drop, data schema mismatch, 5xx spike. Route to PagerDuty (critical) or Slack (warning). Silence flapping with repeat_interval.
LLM Observability — LangSmith
Full chain tracing: prompt → retrieval → LLM call → response. Capture run_id, token counts, latency per step. Feed eval scores back into monitoring pipeline.
Evidently — drift report
from evidently.dashboard import Dashboard from evidently.dashboard.tabs import DataDriftTab dashboard = Dashboard( tabs=[DataDriftTab(verbose_level=0)]) dashboard.calculate( reference_df, current_df, column_mapping=col_map) dashboard.save("drift_report.html") # Push drift metrics to Prometheus # via pushgateway or custom exporter
Prometheus scrape config
# prometheus.yml scrape_configs: - job_name: 'ml_model' static_configs: - targets: - 'ml-model-service:8000' # Instrument your service from prometheus_client import Histogram REQUEST_LATENCY = Histogram( 'model_request_latency_seconds', 'Model inference latency')
08

Retraining Strategy

When to retrain — trigger conditions
Performance Drop
accuracy below threshold
Data Drift
PSI or KL divergence
Time Schedule
weekly / monthly
New Labels
active learning
Kubernetes CronJob — scheduled retraining
apiVersion: batch/v1
kind: CronJob
metadata: { name: retrain-cronjob }
spec:
  schedule: "0 0 * * SUN"  # every Sunday midnight
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: retrainer
            image: myregistry/ml-retrainer:latest
            command: ["python", "train.py"]
            env:
            - name: MLFLOW_TRACKING_URI
              value: "http://mlflow-server:5000"
          restartPolicy: OnFailure

$ kubectl apply -f retrain-cronjob.yaml
Airflow sensor-based trigger
An Airflow DAG with a sensor watches for new data arrival (S3, DB row count, Kafka offset) then triggers the retraining pipeline automatically.
from airflow.sensors.s3_key_sensor import S3KeySensor s3_sensor = S3KeySensor( task_id='wait_for_new_data', bucket_name='my-data-bucket', bucket_key='training/{{ds}}/data.parquet', aws_conn_id='aws_default' ) s3_sensor >> retrain_task
Model promotion after retrain
  • Compare new model vs current prod on holdout set
  • New model must beat prod by ≥2% on primary KPI
  • Check calibration and fairness gates
  • Only then promote via MLflow registry API
  • Shadow deploy before full production switch
Cloud event triggers
  • AWS EventBridge → Lambda → SageMaker training job
  • GCP Cloud Scheduler → Cloud Function → Vertex AI
  • Azure Event Grid → Function → Azure ML pipeline
  • Kafka consumer detects drift metric → retrain webhook
Reproducibility rules
  • Versioned data (DVC tag or lakeFS commit)
  • Versioned code (Git SHA logged in MLflow)
  • Pinned dependencies (requirements.txt locked)
  • Store new model in registry with version label
  • Any retrain run must be fully reproducible
Retraining pitfalls
  • Label leakage — future data included in training window
  • No holdout — evaluating on training data, false confidence
  • Silent failure — retrain job silently errors, no alert
  • Over-retrain — retraining too frequently on noise
09

Security & Governance

Never do this
  • Hardcode API keys or tokens in code
  • Use static AWS credentials in pods
  • Log raw user prompts (PII risk)
  • Skip input schema validation at inference
  • Deploy without CVE scan
  • Store model artifacts with public access
Always do this
  • Use AWS Secrets Manager or HashiCorp Vault
  • Use IRSA — pod identity, no static keys
  • Validate and sanitize all inference inputs
  • Sign model binaries or use checksums
  • Rate limit all model endpoints
  • Log all training runs for audit trail
Secrets management
# Kubernetes Secret — inject at runtime apiVersion: v1 kind: Secret metadata: name: model-secrets type: Opaque data: api_key: <base64-encoded> # Reference in pod spec env: - name: API_KEY valueFrom: secretKeyRef: name: model-secrets key: api_key
API security checklist
  • Authentication on every model endpoint (API key / OAuth)
  • Rate limiting — prevent abuse and cost blowout
  • Input validation — reject malformed schema before inference
  • WAF in front of public-facing model APIs
  • TLS everywhere — never HTTP for model endpoints
  • Audit log every inference request (anonymised)
OWASP AI/ML threats
  • Data poisoning — malicious data corrupts training
  • Model inversion — reconstruct training data from model
  • Prompt injection — override LLM system instructions
  • Adversarial inputs — crafted inputs fool the model
  • Mitigate: input validation + access control + adversarial testing
Compliance — regulated domains
Healthcare (HIPAA)
Anonymise patient data before training. Log all data access. Audit trail for model decisions. No PII in logs.
Finance (SOC 2)
Model explainability for credit decisions. Bias audits per demographic. Immutable audit trail. Encryption at rest + in transit.
GDPR (EU)
Right to erasure — delete training data on request. Pseudonymisation before processing. Data minimisation — collect only what's needed.
10

Tool Stack by Layer

MLOps ecosystem — tools by category
Category Tool License Platform Maturity
Data Versioning DVC Apache 2.0 Any (S3/GCS/Azure) Mature 2018
lakeFS Apache 2.0 S3/Azure/GCS Mature 2019
Data Validation Great Expectations Apache 2.0 Local/Cloud Mature 2019
Feature Store Feast Apache 2.0 K8s/AWS/GCP/Azure Growing 2021
Tecton Commercial AWS/Azure/Databricks Established
Experiment Tracking MLflow Apache 2.0 Any Mature 2018
Weights & Biases Proprietary SaaS Any Mature 2017
Orchestration Apache Airflow Apache 2.0 Cloud/Self-hosted Mature 2015
Kubeflow Pipelines Apache 2.0 Kubernetes Mature 2018
Model Serving TF Serving Apache 2.0 GPU/TPU / Cloud Mature 2017
TorchServe BSD GPU Cloud/on-prem 2019
KServe / Seldon Apache 2.0 Kubernetes Mature 2017
CI/CD GitHub Actions GitHub License Hosted (GitHub) Mature
Jenkins MIT Any server Very mature 2011
Monitoring Prometheus Apache 2.0 K8s/VMs Mature 2012
Grafana Apache 2.0 Any Mature 2014
Evidently AI BSD (OSS) Any (Python) Newer 2020
Executive rules — never break these
  • Version everything — code, data, model, config
  • Keep it DRY — automate repetitive pipeline steps
  • Test extensively — data schema tests + prediction sanity
  • Security by default — auth, encryption, input validation
  • MLOps is continuous — monitor and iterate always
Quick decision guide
Data versioning → DVC
Experiment tracking → MLflow
Hyperparameter tuning → Optuna
Data drift → Evidently AI
Model serving K8s → KServe
Feature store → Feast
Pipeline orchestration → Airflow
LLM tracing → LangSmith
Collaboration matters
MLOps succeeds when data scientists, ML engineers, and operations teams work together on shared pipelines with full visibility. Siloed work produces siloed failures.
Data Scientists ML Engineers DevOps / SRE Data Engineers Security