Perform a Canary Rollout

A canary rollout lets you test configuration changes on a small subset of your fleet before rolling them out to everyone. If something goes wrong, you can abort and rollback instantly.

What You'll Learn

• How to deploy a policy change to 10% of your fleet first
• How to monitor canary health metrics
• How to promote to 100% or abort and rollback

Time: ~10 minutes

Prerequisites: At least 10 connected agents (or be okay with 1 agent being your canary)

Step 1: Prepare Your Change

1. Go to Fleet → Policies
2. Click on the policy you want to update
3. Make your config change in the editor

Example change — switch from debug exporter to a real backend:

# BEFORE
exporters:
  debug:
    verbosity: detailed

# AFTER  
exporters:
  otlp:
    endpoint: https://otel-backend.internal:4317
    tls:
      ca_file: /etc/certs/ca.pem

Step 2: Set Canary Percentage

Before clicking Publish, set the rollout strategy:

Rollout Type: Canary
Canary Percentage: 10%

CollectorCtrl will:

1. Randomly select 10% of agents matching the policy's target selector
2. Push the new config only to those agents
3. Keep the remaining 90% on the old config

Step 3: Monitor the Canary

After publishing, watch the Rollout Status panel:

Status Indicators

Key Metrics to Watch

• Config Status — should show "Applied" (green) on canary agents
• Health Status — should remain "Healthy"
• Remote Config Status — should not show "Error"

Check Canary Agent Logs

$sudo journalctl -u collectorctrl-supervisor -n 30 --no-pager

Look for:

[OpAMP] Received remote config (hash: abc123...)
[Config] Validation passed
[HotReload] Config applied successfully

If you see:

[Config] Validation FAILED: unknown exporter "otlp"

...the canary caught a problem before it hit your full fleet.

Step 4: Decide — Promote or Abort

After monitoring for a few minutes (or longer, depending on your comfort level):

Option A: Promote to 100%

Click Promote to 100% in the policy editor.

CollectorCtrl will:

1. Push the new config to all remaining agents
2. Track the rollout progress in real-time
3. Report final sync status

Option B: Abort and Rollback

Click Abort Rollout.

CollectorCtrl will:

1. Immediately revert canary agents to the previous config version
2. Halt any in-progress deployments
3. Restore the previous policy snapshot as the active version

How It Works Under the Hood

The Canary Selection Algorithm

CollectorCtrl selects canary agents using:

1. Filter — only agents matching the policy's target selector
2. Random Shuffle — agents are randomly ordered
3. Percentage Cut — top N% are selected as canary

This means:

• The same 10% of agents are not always chosen (unless your fleet is tiny)
• Selection is deterministic for a given policy version (same agents get the same canary)

Health Check Monitoring

During a canary rollout, CollectorCtrl monitors:

• Config parsing errors reported by the Supervisor
• Agent health status transitions (Healthy → Unhealthy)
• Connection drops (agents disappearing from the fleet)

If the error rate exceeds thresholds, the rollout automatically pauses.

Best Practices

Troubleshooting

Next Steps

• Upgrade agents fleet-wide
• Set up SSO for your team
• Read about config versioning and rollback