Upgrade Agents Fleet-Wide

Upgrade your Supervisor agents across your entire fleet without logging into a single server. CollectorCtrl handles download, verification, swap, and automatic rollback if anything goes wrong.

---

What You'll Learn

• How to upload a new Supervisor binary to the Package Repository
• How to trigger a fleet-wide upgrade
• How the Upgrade Governor protects your fleet
• How automatic rollback works

Time: ~15 minutes

---

Prerequisites

• A running CollectorCtrl Management Server
• Multiple Supervisor agents connected and healthy
• The new Supervisor binary file ready (Windows .exe or Linux tarball)

---

Step 1: Upload the New Binary

Navigate to Package Management

1. Log into the CollectorCtrl dashboard
2. Click Artifacts → Packages in the sidebar
3. Click Upload Package

Fill in Package Details

Field	Value
Name	collectorctrl-supervisor
Version	1.1.10 (or your target version)
OS	windows or linux
Architecture	amd64 or arm64
Type	Primary
File	Select your binary file

Click Upload. The server will:

1. Store the file in the package directory
2. Calculate the SHA-256 hash
3. Register it in the Package Repository database

---

Step 2: Create an Upgrade Policy

1. Go to Fleet → Policies
2. Click New Policy or edit an existing policy
3. In the Upgrade section, select:
   • Package: Your newly uploaded package
   • Target OS/Arch: Match your fleet's platforms

Alternatively, trigger an upgrade directly from the Packages page:

1. Find your uploaded package
2. Click Deploy Upgrade
3. Select target agents (or use a label selector)

---

Step 3: Monitor the Upgrade Progress

The Upgrade Governor manages the rollout with these safety mechanisms:

Concurrency Throttling

• Maximum 3 concurrent downloads at any time
• Remaining agents are placed in a Queued state
• As active upgrades complete, the next agents are automatically dequeued

Automatic Expiration

• If an active upgrade takes longer than 90 seconds, it's marked as Failed
• The slot is freed for the next queued agent

Bulk Fail-Safe

• If the failure rate exceeds 20% (minimum 3 completed), the Governor halts all remaining queued upgrades
• This prevents a bad binary from taking down your entire fleet

Upgrade Status Indicators

Status	Meaning
Queued	Waiting for a download slot
Downloading	Agent is fetching the package (45% progress)
Installing	Binary swap in progress (75% progress)
Installed	Upgrade complete, agent running new version
Rolled Back	Connection watchdog triggered, reverted to old version
Failed	Download failed, checksum mismatch, or OS error

---

Step 4: Automatic Rollback (If Needed)

If a new Supervisor binary fails to connect to the server within 45 seconds, the automatic rollback triggers:

1. The watchdog process detects the connection failure
2. It terminates the new Supervisor process
3. It restores the backup binary (supervisor.exe.bak → supervisor.exe)
4. It writes a .bad file marker
5. The OS service manager restarts the old version
6. The rolled-back agent reports: "Upgrade failed: Connection watchdog timed out. Rolled back to previous version."

Manual Rollback

If you want to revert the entire fleet:

1. Go to Fleet → Policies
2. Find the upgrade policy
3. Click Rollback
4. All upgraded agents will revert to their previous Supervisor version

---

How the Upgrade Flow Works

Server: "New package available: v1.1.10"
    ↓ OpAMP WebSocket
Supervisor: "My version is v1.1.5, I'll upgrade"
    ↓
Supervisor: Downloads package from server URL
    ↓
Supervisor: Verifies SHA-256 hash matches
    ↓
Supervisor: Renames current binary → .bak
    ↓
Supervisor: Writes new binary
    ↓
Supervisor: Spawns watchdog, exits
    ↓
OS Service Manager: Restarts service
    ↓
New Supervisor: Starts, connects to server
    ↓ (within 45s)
Watchdog: Cancels timer, upgrade successful

If connection fails:

New Supervisor: Fails to connect
    ↓
Watchdog (45s): "Connection timeout!"
    ↓
Watchdog: Kills new process
    ↓
Watchdog: Restores .bak → original
    ↓
OS: Restarts old version
    ↓
Old Supervisor: Connects successfully, reports rollback

---

Troubleshooting

Problem	Solution
Agents stuck in "Queued"	Governor is throttling; wait or check if 3 agents are actively downloading
"SHA-256 mismatch" error	The uploaded file was corrupted; re-upload and verify
Agent shows "Rolled Back"	The new binary couldn't connect within 45s; check server reachability
Upgrade Governor halted	Failure rate exceeded 20%; investigate failed agents before retrying
Windows service won't start	Check Event Viewer; may be missing VC++ redistributables

---

Next Steps

• Create a Fleet Policy
• Set up SSO
• Read the full Supervisor Lifecycle docs