Resolved -
The incident has been fully resolved, and all services are operating normally.
Customers should no longer experience any related issues. If you continue to experience problems, please don't hesitate to contact Auvik Support.
We will provide a Root Cause Analysis (RCA) once it is available.
Aug 20, 10:11 EDT
Update -
We have continued to observe stable alerting behavior across all systems. Our engineering team will continue active monitoring and address any reported issues.
Current Impact:
All services are operating normally.
Critical Reminder:
If you experience any unexpected alerting behavior or identify other outliers, please report them immediately to Auvik Support for investigation.
Aug 20, 07:18 EDT
Update -
We have continued to observe stable alerting behavior across all systems. Our engineering team will continue active monitoring through the day to ensure continued performance.
Current Impact:
All services are operating normally.
Critical Reminder:
If you experience any unexpected alerting behavior or identify other outliers, please report them immediately to Auvik Support for investigation.
Aug 19, 15:23 EDT
Update -
We have continued to see stable alerting behavior.
We are continuing to monitor closely to ensure ongoing stability.
Impact:
Services are operating normally. If you experience any unexpected alerting behavior or other outliers, please report them to Auvik Support so our team can investigate.
Aug 19, 10:33 EDT
Monitoring -
We have continued to see stable alerting behavior overnight, and alerting services remain fully recovered. We are continuing to monitor closely to ensure ongoing stability.
Impact:
Services are operating normally. If you experience any unexpected alerting behavior or other outliers, please report them to Auvik Support so our team can investigate.
Aug 19, 05:31 EDT
Update -
We have continued to see stable alerting behaviour following the changes made earlier, and alerting services remain fully recovered. We are continuing to monitor closely to ensure ongoing stability.
Impact:
Services are operating normally. If you experience any unexpected alerting behaviour or other outliers, please report them to Auvik Support so our team can investigate.
Next Steps:
We will continue monitoring services and will provide a final update once we are confident the issue remains fully resolved.
Aug 18, 17:30 EDT
Update -
We continue investigating the legacy alerting issue causing unexpected email notifications.
We are continuing to validate our findings and work through the remaining steps.
Next Steps:
We will provide another update as we confirm the resolution or as additional information becomes available
Aug 18, 14:46 EDT
Update -
We continue investigating the legacy alerting issue causing unexpected email notifications.
Our investigation has made progress, and we believe we are moving toward a resolution. We are continuing to validate our findings and work through the remaining steps.
Next Steps:
We will provide another update as we confirm the resolution or as additional information becomes available
Aug 18, 11:35 EDT
Identified -
We continue to receive reports of unexpected emails being sent by legacy alerting.
Our team is actively investigating the issue.
Next Steps:
We will provide another update as more information becomes available.
Aug 18, 10:09 EDT
Update -
We are continuing to monitor for any further issues.
Aug 18, 04:31 EDT
Update -
We have observed a full recovery of the affected services, and alerts are now appearing in the UI as expected across all clusters. We are continuing to closely monitor services to ensure stability.
Aug 17, 21:47 EDT
Update -
We have applied additional changes to address the issue causing alerts to not appear in the UI. We are seeing alerts gradually recover and are continuing to closely monitor services to ensure a full recovery.
Impact:
Alert notifications should no longer be sent to unintended users. Alerts may still be delayed in appearing in the UI as services continue to recover. If you continue to encounter problems, please report them to Auvik Support.
Next Steps:
We will continue monitoring alert recovery and service stability and will provide a final update once we confirm the issue is fully resolved.
Aug 17, 19:15 EDT
Monitoring -
We have applied changes to address the issue. Alert notifications should no longer be sent to unintended users
We are closely monitoring services for stability.
Impact:
Services should be operating normally, though new alerts may be delayed in the UI until services catch up; if you continue to encounter problems, please report them to Auvik Support.
Next Steps:
We will post a final update once we confirm the resolution.
Aug 17, 16:31 EDT
Identified -
Our team has identified a suspected cause of the incorrect notifications and is taking steps to fix the issue.
Impact:
Customers may continue to experience false paused notifications
The following services are not affected: V2 alerting
Please report any related issues to Auvik Support so we can track and assist further.
Next Steps:
We are applying mitigation measures and will provide progress updates.
Aug 17, 15:45 EDT
Update -
We are continuing to investigate the issues with incorrect notifications being sent.
Impact:
Customers may experience improper alerts and notifications
The following services are not affected: V2 alert notifications.
Next Steps:
Our team is working to identify contributing factors. We will share updates as more information becomes available.
Aug 17, 15:04 EDT
Investigating -
We are investigating reports of a high volume of inaccurate notifications.
Impact:
Customers may experience improper alerts and notifications
The following services are not affected: V2 alert notifications.
Next Steps:
Our team is working to identify contributing factors. We will share updates as more information becomes available.
Aug 17, 13:53 EDT
