// End-to-end MLOps lifecycle — from problem framing to retraining in production. No fluff.
pip freeze > requirements.txtconda env exportdocker run test in every CI buildapiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "my-model" namespace: "mlops" spec: predictor: containers: - name: "predictor" image: "myregistry/my-mlflow-model:latest" ports: - containerPort: 8080 env: - name: "PROTOCOL" value: "v2" $ kubectl apply -f inference_service.yaml $ kubectl get inferenceservice my-model
apiVersion: apps/v1 kind: Deployment metadata: { name: ml-model } spec: replicas: 2 template: spec: containers: - name: ml-model image: myregistry/my-ml-model:latest ports: [{ containerPort: 8000 }] --- apiVersion: v1 kind: Service metadata: { name: ml-model-svc } spec: type: LoadBalancer ports: [{ port: 80, targetPort: 8000 }] selector: { app: ml-model }
| Strategy | How it works | Best for |
|---|---|---|
| Blue/Green | New env in parallel; traffic switch after validation | Instant full switch + rollback capability |
| Canary | Roll out to small % of users first; expand gradually | Large-scale tech platforms, high-stakes APIs |
| Shadow | Real traffic to new model, predictions logged not served | High-risk domains — fraud, finance, healthcare |
| A/B Testing | Split traffic; measure business KPI per variant | E-commerce, ads, recommendations |
| Champion–Challenger | Challenger competes against champion on live traffic | Fraud scoring, risk modelling, finance |
name: CI with MLflow on: push: branches: ["master"] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v3 with: python-version: '3.9' - name: Install dependencies run: | pip install -r requirements.txt - name: Run training env: MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }} run: | python train.py - name: Build and push Docker image run: | docker build -t myregistry/ml-model:$GITHUB_SHA . docker push myregistry/ml-model:$GITHUB_SHA - name: Deploy to Kubernetes run: | kubectl set image deployment/ml-model \ ml-model=myregistry/ml-model:$GITHUB_SHA
apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ml-pipeline- spec: entrypoint: train-pipeline templates: - name: train-pipeline steps: - - name: train template: train-task - name: train-task container: image: myregistry/my-ml-image command: [python, train.py] env: - name: MLFLOW_TRACKING_URI value: "http://mlflow-server:5000"
/metrics endpoint via prometheus_client. Scrape config in prometheus.yml. Set alerts on P95 latency, error rate, and model accuracy SLOs.
run_id, token counts, latency per step. Feed eval scores back into monitoring pipeline.
apiVersion: batch/v1 kind: CronJob metadata: { name: retrain-cronjob } spec: schedule: "0 0 * * SUN" # every Sunday midnight jobTemplate: spec: template: spec: containers: - name: retrainer image: myregistry/ml-retrainer:latest command: ["python", "train.py"] env: - name: MLFLOW_TRACKING_URI value: "http://mlflow-server:5000" restartPolicy: OnFailure $ kubectl apply -f retrain-cronjob.yaml
| Category | Tool | License | Platform | Maturity |
|---|---|---|---|---|
| Data Versioning | DVC | Apache 2.0 | Any (S3/GCS/Azure) | Mature 2018 |
| lakeFS | Apache 2.0 | S3/Azure/GCS | Mature 2019 | |
| Data Validation | Great Expectations | Apache 2.0 | Local/Cloud | Mature 2019 |
| Feature Store | Feast | Apache 2.0 | K8s/AWS/GCP/Azure | Growing 2021 |
| Tecton | Commercial | AWS/Azure/Databricks | Established | |
| Experiment Tracking | MLflow | Apache 2.0 | Any | Mature 2018 |
| Weights & Biases | Proprietary SaaS | Any | Mature 2017 | |
| Orchestration | Apache Airflow | Apache 2.0 | Cloud/Self-hosted | Mature 2015 |
| Kubeflow Pipelines | Apache 2.0 | Kubernetes | Mature 2018 | |
| Model Serving | TF Serving | Apache 2.0 | GPU/TPU / Cloud | Mature 2017 |
| TorchServe | BSD | GPU Cloud/on-prem | 2019 | |
| KServe / Seldon | Apache 2.0 | Kubernetes | Mature 2017 | |
| CI/CD | GitHub Actions | GitHub License | Hosted (GitHub) | Mature |
| Jenkins | MIT | Any server | Very mature 2011 | |
| Monitoring | Prometheus | Apache 2.0 | K8s/VMs | Mature 2012 |
| Grafana | Apache 2.0 | Any | Mature 2014 | |
| Evidently AI | BSD (OSS) | Any (Python) | Newer 2020 |