Config Changes Not Applying

You published a policy update, but agents still show the old config or report errors. This guide helps you diagnose why.

Quick Diagnostic Flow

Published policy → Agent shows "Reconciling"? → Wait 10-30 seconds
                                    ↓
                            Agent shows "Error"? → Check validation
                                    ↓
                            Agent shows "Applied" but old behavior? → Check drift
                                    ↓
                            Agent not in target selector? → Check labels

1. Check the Agent's Target Selector Match

The most common reason: the agent doesn't match the policy's target selector.

Verify in the Dashboard

1. Go to Fleet → Overview
2. Click on the agent's name to open the detail view
3. Check the Labels section
4. Compare with your policy's matchLabels / matchExpressions

Example Mismatch

Policy selector:

{"matchLabels": {"env": "production"}}

Agent labels:

{"env": "staging", "host.os": "linux"}

→ No match. The agent won't receive this policy.

Fix

Either:

• Update the policy selector to include the agent's labels
• Or add the expected label to the agent's resource attributes

2. Check Config Validation Errors

Before applying any config, the Supervisor runs a validation check:

otelcol --config /tmp/staged-config.yaml validate

If this fails, the config is rejected and the agent reports Config Error.

Read the Validation Error

Windows:

PS >Get-Content 'C:\ProgramData\CollectorCtrlSupervisor\supervisor.log' -Tail 30 | Select-String 'validation|error|failed'

Linux:

$sudo journalctl -u collectorctrl-supervisor -n 30 --no-pager | grep -i 'validation\|error\|failed'

Common Validation Errors

3. Check for Config Drift

If the agent shows Applied but the collector behavior hasn't changed, the Supervisor may be fighting a manual edit.

How Drift Detection Works

1. Supervisor receives new config from server
2. Writes it to the local config file
3. Continuously monitors the file hash
4. If someone edits the file manually → hash changes → Supervisor overwrites it
5. This creates a loop that appears as constant "Reconciling"

Check for Drift

1. Go to Fleet → Overview
2. Look at the Remote Config column
3. If it cycles between "Applied" and "Reconciling", drift is occurring

Fix

Stop manually editing config files on agent machines. All config changes should go through CollectorCtrl policies.

If you need node-specific tweaks, use the Override Config feature:

1. Click on the agent in the Fleet Overview
2. Go to the Config tab
3. Edit the Override Config
4. Click Save

The Supervisor will merge the override with the policy and apply the result.

4. Check the Hot-Reload Mechanism

On Linux, the Supervisor uses SIGHUP for hot-reload. On Windows, it uses the collector's admin API.

Linux: Verify SIGHUP Works

# Check if the collector process is running
$ps aux | grep otelcol
# Manually send SIGHUP to test
$sudo kill -HUP $(pgrep otelcol)
# Check if collector reloaded without restarting
$sudo journalctl -u collectorctrl-supervisor -n 10

Windows: Verify Admin API

The collector must have the health_check extension on port 13133:

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]

Test locally:

PS >Invoke-RestMethod -Uri 'http://localhost:13133/schema/reload' -Method POST

If this returns an error, the collector doesn't support admin API reload. The Supervisor will fall back to a fast process restart.

5. Check Policy Priority Conflicts

An agent can match multiple policies. CollectorCtrl resolves this by priority — higher priority wins.

Check for Overlapping Policies

1. Go to Fleet → Policies
2. Check the Priority column for each policy
3. If two policies target the same agent, the higher-numbered one takes precedence

Example Conflict

If your new policy has lower priority than an existing one, it will be overridden.

Fix

Increase your policy's priority number so it takes precedence.

6. Check the Effective Config

The Effective Config is what actually runs on the collector. It's the result of:

Policy Template + Override Config = Effective Config

View the Effective Config

1. Click on an agent in the Fleet Overview
2. Go to the Config tab
3. View the Effective Config panel

Compare this with what you expected. If the merge didn't produce what you wanted, check:

• YAML list overwrite behavior (lists are replaced, not merged)
• Indentation in the override config
• Matching keys between policy and override

7. Policy Not Published

Sounds obvious, but it's easy to miss: did you click Publish?

• Saving a policy as a draft does not push it to agents
• Only clicking Publish triggers the OpAMP broadcast
• Check the policy status — it should show "Active", not "Draft"

Summary Table

Next Steps

• Read about Fleet Policies
• Read about config merge logic
• Agent won't connect?