Why Most MLOps Break in Production and How to Fix It

Welcome to Part 5 of our Microservices, Architecture, and ML Engineering series.

If you missed earlier posts, here’s the journey so far:

Part 1: Microservices for ML engineers.
Part 2: Microservice for ML engineers: design and modelling.
Part 3: Microservice for ML engineers: How to communicate.
Part 4: MLOps that actually works in production

In this post, we’ll dive deeper into MLOps — drawing on real-world experience and returning to the streaming platform case study from Part 3. I’ll walk you through the strategies and concepts that help ML systems survive production without constant fire-fighting.

If you want to grow into an engineer and architect who builds resilient systems — rather than patching broken apps — follow along for updates.

A couple of years ago, I thought deployment was done. Rollout finished, dashboards green, I mentally checked out. Then the system tanked.

This was a multi-region ML system with users around the globe. Suddenly, APAC latency spiked to 620ms up from the usual 140ms. The “safe” model update was choking on cold TensorRT compilations while Redis frantically evicted carefully warmed cache keys.

The only good news? Canary guardrails caught it in three minutes and rolled back automatically. The better news? I’ll show you how to build that same safety net.

Where theory meets reality

Here’s the context. StreamFlow, a global streaming platform, runs four mission-critical ML systems:

1. Real-time recommendations: the revenue maker that is core for business.
2. Content understanding: CV+NLP pipeline to classify and understand the content for internal usage.
3. Fraud detection: the silent guardian.
4. Content moderation: keeping things clean.

Each has its own area of responsibility, its own failure modes, and its own way of ruining your weekend 😂

The Numbers that keep us in good fit

Here’s what I was held to every single day:

These aren’t aspirational. Miss any of these consistently, and you’re updating your resume.

The Architecture: from wishful thinking to production reality

What we told to investors (the logical view)

Simple, right? Yeah, about that…

What Actually Runs (physical level)

Three hard truths shaped this architecture:

1. No single point of failure (learned this after a 2 AM outage).
2. Sub-100ms globally (or users leave).
3. Handle 40% traffic spikes (new season drops are brutal).

The five principles that will save your sanity

After countless production fires, these principles emerged. Violate them at your peril:

1. Isolation is your best friend

GPU inference and CPU feature computation should never, ever fight for resources. We learned this the hard way when feature extraction starved our GPUs, causing cascading timeouts.

The fix was just to separate node pools, separate services, separate failure domains.

2. Automation isn’t optional

If a developer has to do it twice, automate it. This includes:

• Deployments and rollbacks
• Canary progression
• Drift detection
• Cost alerts
• Performance regression checks
• Infrastrucutre alerts and logging

3. Infrastructure as code (IaC) is law

Everything goes through version control:

Kubernetes manifests, Istio configs, GPU pools, even alerting rules.

No clicking in UIs. Ever.

4. Zero-Downtime is the minimum bar

Your users don’t care about your deployment schedule. Ship these patterns or suffer:

• Warm model pools
• Preloaded caches
• Weighted traffic shifting
• ML-aware health checks

5. Observe everything, alert on what matters

Three lanes of metrics, one dashboard:

• System: CPU, GPU, memory, latency
• Model: Accuracy, drift, staleness
• Business: CTR, conversion, revenue/session

When something breaks, you need all three to diagnose.

The deployment patterns that actually Work

Pattern 1: The real readiness probe

Kubernetes has three probe types, and most people use them wrong. Let me explain what each actually does:

Startup Probe: “Is the application starting up?”

• Runs first, disables other probes
• Give it generous time for cold starts
• Fails → Pod is restarted

Readiness Probe: “Can this pod handle traffic?”

• Determines if pod receives traffic
• Fails → Pod removed from service endpoints
• This is where ML-specific checks belong

Liveness Probe: “Is this pod dead?”

• Last resort health check
• Fails → Pod is killed and restarted
• Keep this simple — just process health

Most teams check if the container is up. That’s not enough. Here’s what production-grade looks like or should look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rec-model-v3
spec:
  template:
    spec:
      containers:
      - name: model-server
        
        # Give model loading plenty of time
        startupProbe:
          httpGet:
            path: /startup  # Just checks process is running
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 30  # 10 + 10*30 = 310 seconds max startup
        
        # This is the critical one for ML
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3
          successThreshold: 2  # Must pass twice to be considered ready
          failureThreshold: 3   # Fails three times → remove from traffic
        
        # Keep this dead simple
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60  # Don't check until startup is done
          periodSeconds: 10
          failureThreshold: 3

Why this matters: Without proper readiness, Kubernetes happily routes traffic to pods that can’t actually serve it. Result? Timeouts and angry users and your headache.

Pattern 2: Smart autoscaling that understands ML

Traditional autoscaling uses CPU/memory. ML systems need smarter metrics because:

1. GPU utilization doesn’t correlate linearly with throughput (batch processing effects)
2. Latency is more important than utilization (SLA > efficiency)
3. Cold starts are expensive (keep warm pool)

CPU metrics lie for ML workloads. Here’s what actually works:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rec-model-v3
  
  # Don't scale too aggressively
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
      policies:
      - type: Percent
        value: 50  # Scale down by max 50% at once
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60  # Scale up faster than down
      policies:
      - type: Percent
        value: 100  # Can double capacity if needed
        periodSeconds: 60
  
  minReplicas: 6
  maxReplicas: 40
  
  metrics:
  # Primary: Request latency (most important for UX)
  - type: Pods
    pods:
      metric:
        name: http_request_duration_seconds_p95
      target:
        type: AverageValue
        averageValue: "0.12"  # 120ms p95
  
  # Secondary: Throughput per pod
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"  # 100 RPS per pod
  
  # Tertiary: GPU utilization (efficiency)
  - type: Pods
    pods:
      metric:
        name: gpu_utilization_percent
      target:
        type: AverageValue
        averageValue: "75"  # Sweet spot for efficiency

Keep GPU utilization around 70–80%. Above that, latency spikes exponentially. Also remember that GPU ML services are completely different:

• Process starts → Code loads → Model loads to CPU memory → Model transfers to GPU memory → GPU kernel compilation → Optimization passes → Ready to serve
• Each step can fail independently
• Time to readiness: 30 seconds to 5 minutes

When you use TensorRT (NVIDIA’s inference optimization library), it doesn’t just load your model. It analyzes your specific model architecture and compiles optimized CUDA kernels specifically for your GPU type. This compilation:

• Happens on first inference if not pre-compiled
• Takes 30–90 seconds for complex models
• Is different for each GPU architecture (T4 vs V100 vs A100)
• Can’t be shared across different batch sizes without additional config

This is why our pods showed “ready” but couldn’t actually serve traffic. Kubernetes thought the container was healthy, but the model wasn’t actually usable yet.

Pattern 3: Canary deployments with ML-aware gates

This is where most teams fail. They check system metrics but ignore model performance. Here’s the complete picture:

# Istio traffic management
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: rec-model
spec:
  hosts:
  - rec.api.svc.cluster.local
  http:
  - route:
    - destination:
        host: rec-model
        subset: stable  # v2
      weight: 90
    - destination:
        host: rec-model
        subset: canary  # v3
      weight: 10

But traffic splitting is just the start. Here’s the Python controller that makes it smart:

class MLCanaryController:
    def __init__(self):
        self.gates = [
            SystemGate(),      # Latency, errors
            ModelGate(),       # Accuracy, drift
            BusinessGate()     # CTR, revenue
        ]
    
    def evaluate_canary(self, canary_metrics, baseline_metrics):
        """
        Run every 5 minutes during rollout
        """
        for gate in self.gates:
            if not gate.passes(canary_metrics, baseline_metrics):
                self.trigger_rollback()
                self.page_on_call(f"Gate failed: {gate.name}")
                return False
        
        return True
    
    def progress_canary(self, current_weight):
        """
        Gradual rollout with validation
        """
        stages = [10, 25, 50, 100]
        next_weight = stages[stages.index(current_weight) + 1]
        
        # Wait for statistical significance
        if self.has_enough_samples(current_weight):
            if self.evaluate_canary():
                self.update_weight(next_weight)
            else:
                self.rollback()

class ModelGate:
    def passes(self, canary, baseline):
        # Check accuracy drop or other metrics that you have
        if canary.ndcg < baseline.ndcg * 0.99:  # 1% tolerance
            return False
        
        # Check for data drift (or using your tests)
        for feature in TOP_10_FEATURES:
            psi = calculate_psi(
                baseline.feature_dist[feature],
                canary.feature_dist[feature]
            )
            if psi > 0.25:  # Significant drift
                return False
        
        return True

Pattern 4: The drift detection that actually catches problems

Data drift is silent and deadly. Here’s how we catch it before users notice:

from scipy import stats
import numpy as np

class DriftMonitor:
    def __init__(self, training_stats):
        self.training_stats = training_stats
        self.alert_threshold = 0.25  # PSI threshold
        self.critical_features = [
            'user_watch_time_7d',
            'genre_preferences',
            'device_type',
            'hour_of_day',
            # ... your critical features
        ]
    
    def check_drift(self, current_batch):
        """
        Run this every hour in production
        """
        alerts = []
        
        for feature in self.critical_features:
            # Population Stability Index
            psi = self.calculate_psi(
                self.training_stats[feature],
                current_batch[feature]
            )
            
            # Kolmogorov-Smirnov test
            ks_statistic, ks_pvalue = stats.ks_2samp(
                self.training_stats[feature]['values'],
                current_batch[feature]
            )
            
            if psi > self.alert_threshold:
                alerts.append({
                    'feature': feature,
                    'psi': psi,
                    'severity': 'HIGH' if psi > 0.5 else 'MEDIUM'
                })
            
            # Log for monitoring
            self.metrics.record('drift_psi', psi, 
                              tags={'feature': feature})
        
        if alerts:
            self.trigger_retraining_evaluation(alerts)
        
        return alerts
    
    def calculate_psi(self, expected, actual):
        """
        PSI = sum (actual - expected) * ln(actual / expected)
        """
        # ......
        # PSI measures how much a distribution has shifted
        # PSI < 0.1: No shift
        # PSI 0.1-0.25: Slight shift, monitor closely  
        # PSI > 0.25: Significant shift, investigate immediately
        pass

Also as a suggestion you can combine it with KS test (Kolmogorov-Smirnov test):

# KS test measures the maximum distance between two cumulative distributions
# p-value < 0.01: Distributions are significantly different
# p-value ≥ 0.01: Cannot reject that distributions are the same

from scipy import stats

def detect_distribution_shift(training_data, production_data):
    """
    The KS test is non-parametric (doesn't assume normal distribution)
    making it perfect for feature drift detection.
    """
    ks_statistic, p_value = stats.ks_2samp(training_data, production_data)
    
    # KS statistic: Maximum vertical distance between CDFs
    # Ranges from 0 (identical) to 1 (completely different)
    
    # P-value: Probability of seeing this difference by chance
    # Low p-value = distributions are truly different
    
    return {
        'shifted': p_value < 0.01,
        'ks_statistic': ks_statistic,
        'p_value': p_value,
        'interpretation': 'significant shift' if p_value < 0.01 else 'no shift detected'
    }

Why use both PSI and KS? They catch different types of shifts:

• PSI is sensitive to shifts in the shape of the distribution
• KS catches shifts in location (mean) and scale (variance)

Together, they form a robust drift detection system.

The Observability stack that shows everything

You need three types of metrics on ONE dashboard. Here’s our Prometheus/Grafana setup:

# What to emit from your model service
from prometheus_client import Histogram, Counter, Gauge

# System metrics
inference_latency = Histogram(
    'inference_latency_ms',
    'Model inference latency',
    buckets=[10, 25, 50, 75, 100, 150, 200, 300, 500]
)

# Model metrics
model_accuracy = Gauge(
    'model_online_accuracy',
    'Rolling window accuracy vs ground truth'
)

drift_score = Gauge(
    'feature_drift_psi',
    'Population stability index by feature',
    ['feature_name']
)

# Business metrics
business_ctr = Gauge(
    'recommendation_ctr',
    'Click-through rate for recommendations',
    ['model_version', 'experiment_id']
)

# Use them
@inference_latency.time()
def predict(self, features):
    # Your inference code
    pass

The One-Page Runbook That Actually Gets Used

Print this. Laminate it. Live by it.

When Alerts Fire: The 5-Minute Response

1. STOP (0–30 seconds)

• Freeze any ongoing rollout
• Acknowledge the page

2. SHIFT (30–90 seconds)

• Route 100% traffic to stable version
• Command: kubectl apply -f configs/stable-only.yaml

4. DIAGNOSE (90 seconds — 5 minutes)

• Check the dashboard (you have one, right?)
• Look for:
• GPU memory spikes → model too large
• Feature latency → cache miss or data pipeline issue
• Accuracy drop → data drift or bad model
• Error rate → code bug or infra issue

5. MITIGATE (If needed)

• Scale up stable version: kubectl scale deployment rec-model-stable --replicas=20
• Warm backup region: ./scripts/warm-backup-region.sh eu-west
• Enable fallback model: kubectl set env deployment/rec-model FALLBACK_ENABLED=true

Real-World optimizations that made a difference

Optimization 1: batch everything

GPUs hate single predictions. Here’s the pattern that 3x’d our throughput:

class BatchingModelServer:
    def __init__(self, model, batch_size=32, timeout_ms=10):
        self.model = model
        self.batch_size = batch_size
        self.timeout_ms = timeout_ms
        self.pending = []
        self.futures = []
    
    async def predict(self, features):
        future = asyncio.Future()
        self.pending.append((features, future))
        
        if len(self.pending) >= self.batch_size:
            await self._process_batch()
        else:
            # Wait up to timeout_ms for more requests
            asyncio.create_task(self._timeout_trigger())
        
        return await future
    
    async def _process_batch(self):
        if not self.pending:
            return
        
        batch = self.pending[:self.batch_size]
        self.pending = self.pending[self.batch_size:]
        
        # Batch inference - this is where GPU shines
        inputs = np.stack([f for f, _ in batch])
        predictions = self.model.predict(inputs)
        
        # Resolve futures
        for (_, future), pred in zip(batch, predictions):
            future.set_result(pred)

Optimization 2: cache aggressively (but smartly)

class SmartFeatureCache:
    def __init__(self, redis_client):
        self.redis = redis_client
        self.local_cache = {}  # L1 cache
        
    async def get_features(self, user_id):
        # L1: Local memory (microseconds)
        if user_id in self.local_cache:
            age = time.time() - self.local_cache[user_id]['timestamp']
            if age < 60:  # 1 minute local cache
                return self.local_cache[user_id]['features']
        
        # L2: Redis (milliseconds)
        features = await self.redis.get(f"features:{user_id}")
        if features:
            self.local_cache[user_id] = {
                'features': features,
                'timestamp': time.time()
            }
            return features
        
        # L3: Compute (hundreds of milliseconds)
        features = await self.compute_features(user_id)
        
        # Write through to cache
        await self.redis.setex(
            f"features:{user_id}",
            ttl=300,  # 5 minutes
            value=features
        )
        
        return features

The Mistakes I Made (So You Don’t Have To)

Mistake 1: Trusting container readiness

What happened: Pods marked ready, but models weren’t loaded. Traffic → timeouts. The fix was real readiness probes that actually check model state.

Mistake 2: Ignoring cold starts

What happened: Scale-up during traffic spike → new pods → cold TensorRT → cascade failure. The fix is to init containers + warm standby pools.

Mistake 3: Deploying on friday

What happened: Do I need to explain this one?

Mistake 4: Single metric autoscaling

What happened: Scaled on CPU, but GPU was the bottleneck. The fix is multi-metric HPA with business logic.

Mistake 5: No drift monitoring

What happened: Model accuracy degraded over 3 weeks. Nobody noticed until revenue dropped. The fix is automated drift detection with alerting.

Production ML isn’t about having the best model. It’s about having a model that stays up, performs consistently, and doesn’t wake you up at 3 AM.

The patterns I’ve shared aren’t theoretical they’re scarred into our infrastructure after dozens of production incidents. They’re the difference between a system that serves millions users reliably and one that’s constantly on fire.

Remember:

• Your readiness probe is your first line of defense
• Canaries without gates are just slow failures
• Drift is silent until it’s catastrophic
• Automation isn’t optional when you’re at scale

These patterns have saved us countless hours and probably millions in prevented downtime. If you’re fighting similar battles, I’d love to hear your war stories.

Found this useful? Hit the clap button and share it with your team. Your on-call engineers will thank you.

Have questions or war stories? Drop them in the comments.

P.S. — Yes, we still occasionally have incidents. The difference is now they last minutes, not hours, and nobody panics. That’s the real victory.

If this helped you avoid communication disasters, consider ☕buying me a coffee☕, more detailed architecture breakdowns are coming!