Using Latency SLOs Instead of Error Rate in Argo Rollouts Canary Analysis

In my environment, relying solely on error rates for canary promotions often misses subtle degradations that hurt user experience. A service can return 200 OK while slowly degrading response times — something users notice immediately, but error-rate-based analysis ignores. That’s why I switched to latency SLOs in Argo Rollouts for canary analysis. It aligns promotion decisions with what users actually feel.

The core idea is simple: instead of measuring HTTP 5xx or failed requests, we track the percentage of requests under a latency threshold (e.g., p95 < 200ms). If the canary meets this SLO, we promote; if not, we rollback or pause. This approach catches performance regressions early, even when error rates stay flat.

Here’s how I configured it in a recent rollout for a customer-facing API.

Defining Latency-Based Metrics in Argo Rollouts

Argo Rollouts supports custom metrics via Prometheus, making latency SLO integration straightforward. I defined a metric that calculates the ratio of requests under 200ms over a 2-minute window.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-slo
spec:
  args:
  - name: service-name
  metrics:
  - name: latency-slo
    interval: 30s
    count: 4
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          (
            sum(rate(http_request_duration_seconds_bucket{job="api", le="0.2"}[2m]))
            /
            sum(rate(http_request_duration_seconds_count{job="api"}[2m]))
          ) * 100
    successCondition: result >= 95
    failureCondition: result < 90
    limit: 3
    failFast: true

This query calculates the percentage of requests under 200ms (le="0.2") over the last 2 minutes. If at least 95% of requests meet this threshold, the metric passes. Below 90%, it fails. I use a 30s interval with 4 readings to smooth out noise.

Note: Adjust the latency threshold (0.2s) and SLO target (95%) based on your service’s performance baseline. I usually derive these from historical p95 data.

Integrating the Template into a Rollout

Next, I referenced this template in the rollout spec under analysis. This ensures the metric runs during the canary step.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 20
      - analysis:
          templates:
          - templateName: latency-slo
            args:
            - name: service-name
              value: api
      - setWeight: 50
      - analysis:
          templates:
          - templateName: latency-slo
            args:
            - name: service-name
              value: api
      - setWeight: 100

The analysis runs at 20% and 50% traffic shifts. If latency SLO isn’t met, Argo Rollouts halts the rollout and begins rollback. This prevents bad-performing versions from scaling to full traffic.

Why Latency SLOs Beat Error Rates for User Experience

Error rates are binary: a request either fails or it doesn’t. But latency affects every user interaction. A slow API makes pages feel unresponsive, increases bounce rates, and degrades trust — even if every request returns 200.

In one case, a deploy introduced a slow database query that increased p95 latency from 180ms to 420ms. Error rate stayed at 0.1%, so traditional analysis would have promoted it. But the latency SLO dropped from 98% to 65%, triggering an automatic rollback. Users never felt the degradation.

This approach also aligns with SRE practices: we’re measuring what we promised in our SLOs, not just avoiding failure. It shifts the mindset from "is it broken?" to "is it fast enough?"

Common Pitfalls and How I Avoid Them

• Misaligned thresholds: Copying someone else’s latency target without baselining your own service leads to false positives or missed regressions. I always review 4 weeks of p95/p99 data before setting SLOs.

• Prometheus query mistakes: A missing rate() or incorrect bucket label breaks the metric. I test queries in Prometheus UI first, then copy them into the AnalysisTemplate.

• Too-short windows: Using 10s intervals causes flapping due to natural variance. I stick to 30s-1m intervals with multiple reads.

• Ignoring network jitter: Service mesh or ingress latency can skew results. I isolate the metric to the application layer (job="api") to avoid blaming the rollout for infrastructure issues.

Warning: Don’t use raw latency averages — they’re sensitive to outliers. Always use percentile-based SLIs (like p95) for SLO comparisons.

When to Still Use Error Rate

Latency SLOs don’t replace error rate analysis — they complement it. I still include error-rate checks for critical failure modes (e.g., 5xx spikes, timeout cascades). A combined approach works best:

• Latency SLO → protects user experience under normal failure modes
• Error rate → catches hard failures, cascades, or downtime

This gives me layered protection: fast detection of both performance regressions and outages.

Final Thoughts

Switching to latency SLOs in Argo Rollouts changed how I judge release quality. It’s not just about whether the service works — it’s about whether it works well enough for users. By measuring what users actually experience, I catch issues earlier, reduce blame games, and ship with more confidence.

If you’re using Argo Rollouts and still relying only on error rates for canary decisions, try adding latency-based analysis. Start with one service, baseline your latency, and observe how promotion behavior changes. You might be surprised how often latency catches what errors miss.

Cover image: Mikecligon · CC0 (Wikimedia Commons / kamu malı) · https://commons.wikimedia.org/wiki/File:Michael_Ligon_Stock_Market_Conference_2023.png