Emergency Collector Rollback

Incident Report for Auvik Networks Inc.

Postmortem

Service Disruption - SNMPv3 Monitoring Loss Following Collector Upgrade

Root Cause Analysis

Duration of the incident

Discovered: Apr 13, 2026 20:00 - UTC
Resolved: Apr 13, 2026 22:32 - UTC

Customer impact

Customers experienced a loss of monitoring data from only devices using specific SNMPv3 configurations. While devices remained online and reachable, monitoring data was not collected, resulting in reduced visibility across environments.

Cause

A recent collector upgrade introduced changes to encryption handling that affected support for certain legacy SNMPv3 configurations. This resulted in failures when attempting to collect data from devices configured that way.

Effect

Monitoring data collection failed for affected devices across multiple clusters. This led to a noticeable drop in available device metrics and visibility, despite no loss of connectivity to the devices themselves.

Future consideation(s)

Expand test coverage to include a broader range of SNMP configurations
Improve monitoring to detect drops in data collection more proactively
Strengthen validation processes for major upgrades and dependency changes
Implement additional safeguards to identify compatibility issues prior to release

Posted Apr 16, 2026 - 18:14 EDT

Resolved

The incident has been fully resolved. Regular service has been restored, and all systems are operating as expected.

Impact:
Users should no longer experience any issues related to this incident.
If you are still experiencing issues, please do not hesitate to reach out to the support team and update your ticket or report any problems you haven't reported yet.

Service has been fully restored. We apologize for the degradation in services. We thank you for your understanding. If you continue to experience issues, please don't hesitate to contact our support team.
We will post an RCA after an internal investigation.

Posted Apr 13, 2026 - 22:33 EDT

Update

We are continuing to rollback collector versions on each cluster.

Impact:
Customers may continue to experience collector SNMP monitoring issues
We will be having cluster maintenance windows as we roll back the collector upgrade on each cluster.
We are rolling back the remaining clusters currently.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are implementing mitigation measures and will provide progress updates.

Posted Apr 13, 2026 - 21:55 EDT

Identified

Our team has identified a suspected cause of the Collector SNMP monitoring issue and is taking steps to remediate.

Impact:
Customers may continue to experience collector SNMP monitoring issues
We will be having cluster maintenance windows as we roll back the collector upgrade on each cluster.
We are rolling back the EU2 cluster currently.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are implementing mitigation measures and will provide progress updates.

Posted Apr 13, 2026 - 21:10 EDT

Investigating

We are currently investigating reports of a collector affecting monitoring.

Impact:
Customers may experience issues with the SNMP monitoring protocol.
Other services are not affected.
Auvik will be performing an emergency rollback of the collector upgrade this weekend.

Next Steps:
Our team is working to identify contributing factors. Updates will follow as more information becomes available.

Posted Apr 13, 2026 - 20:32 EDT

This incident affected: Network Mgmt (us1.my.auvik.com, us2.my.auvik.com, us3.my.auvik.com, us4.my.auvik.com, us5.my.auvik.com, us6.my.auvik.com, eu1.my.auvik.com, eu2.my.auvik.com, au1.my.auvik.com, ca1.my.auvik.com, lnx.my.auvik.com).