Latency SLO-Based Canary Analysis and Auto Rollback in Argo Rollouts

In my environment, I’ve shifted from error-rate-only canary checks to latency SLOs as the primary gate for promotions in Argo Rollouts. Latency directly impacts user experience, and waiting for 5xx errors to spike means you’ve already degraded service. By the time error rates rise, users have already suffered. Using latency SLOs lets you catch performance regressions earlier and automate rollbacks before users notice.

The core idea is simple: define a latency SLO (e.g., 95th percentile request duration < 200ms), expose it as a Prometheus metric, and configure Argo Rollouts to analyze that metric during canary steps. If the SLO is violated, the rollout pauses and can auto-rollback based on failure limits.

Here’s how I set it up. First, ensure your application exposes latency histograms via Prometheus. I use the Prometheus client library in my Go services to record request duration:

httpDurationsVec := prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "http_request_duration_seconds",
    Help: "Duration of HTTP requests",
    Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
}, []string{"method", "endpoint", "status"})

Then, in your Prometheus rule or recording rule, compute the 95th percentile:

- record: job:http_request_duration_seconds:quantile95
  expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

This gives you a clean metric: job:http_request_duration_seconds:quantile95. I label it by job so I can target specific services.

In Argo Rollouts, you reference this metric in an AnalysisTemplate. I created one called latency-slo-analysis.yaml:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-slo
spec:
  args:
  - name: service-name
  metrics:
  - name: latency-slo
    interval: 30s
    count: 5
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          job:http_request_duration_seconds:quantile95{job="{{args.service-name}}"} < 0.2
    successCondition: result == "true"
    failureLimit: 3
    # Optional: add failureLimitSteps to auto-rollback after N failures

The query returns 1 (true) if the 95th percentile latency is under 200ms, 0 otherwise. Argo Rollouts treats successCondition: result == "true" as a pass. After 5 intervals (2.5 minutes), if 3 or more failures occur, the analysis fails.

I then use this template in a Canary step:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-service
spec:
  strategy:
    canary:
      steps:
      - setWeight: 20
      - analysis:
          templateName: latency-slo
          args:
          - name: service-name
            value: my-service
      - setWeight: 50
      - analysis:
          templateName: latency-slo
          args:
          - name: service-name
            value: my-service

At 20% and 50% traffic, Argo runs the latency analysis. If latency exceeds 200ms for 3 out of 5 checks, the step fails. By default, Argo pauses the rollout. To enable auto-rollback, I set:

spec:
  strategy:
    canary:
      autoRollbackOnFailure: true

Now, if the latency SLO is violated at any canary step, Argo automatically rolls back to the stable version.

One gotcha: make sure your Prometheus query is resilient to missing data. If the target service has no traffic during a scrape, the query might return no rows, causing Argo to treat it as an error. I wrap the query in absent() or use or to default to a safe value:

query: |
  (job:http_request_duration_seconds:quantile95{job="{{args.service-name}}"} < 0.2) or on() vector(0)

This ensures that if the metric is absent (no requests), it defaults to 0 (false), triggering a failure — which is safe: no traffic means something’s wrong.

I’ve found this approach catches slow rollouts (e.g., due to resource limits, inefficient DB queries, or external API latency) that error-rate analysis would miss. As I mentioned before in my post on using latency SLOs instead of error rate, this shifts focus from "is it broken?" to "is it fast enough?" — a better user-centric gate.

Warning: Don’t rely solely on latency. Combine it with error rate and business metrics (e.g., conversion drop) for layered safety. Latency SLOs are excellent for catching performance regressions, but they won’t detect logic errors that don’t affect speed.

Cover image: TSGT L. HERNANDEZ, USAF · PDM (Wikimedia Commons / kamu malı) · https://commons.wikimedia.org/wiki/File:Cobra_Dane_radar_system_operation_status_control_console_room.jpg