Service Degraded - Auvik Dashboard in AU1

Incident Report for Auvik Networks Inc.

Postmortem

Service Disruption - AU1 Cluster Performance Degradation

Root Cause Analysis

Duration of incident

Discovered: Jun 3, 2025 – 00:10 UTC
Resolved: Jun 3, 2025 – 02:16 UTC

Cause

The performance degradation was caused by resource limitations in the database system supporting the AU1 cluster. These limitations temporarily prevented the system from efficiently cleaning up and processing data. This led to slower load times until the underlying resources were increased.

Effect

Customers connected to the AU1 cluster experienced slow performance when accessing the Auvik Web UI. Pages were taking longer than usual to load, and in some cases, data within the interface appeared delayed or incomplete. While monitoring systems remained unaffected, the responsiveness was degraded during the incident window.

Action taken

All times are in UTC

06/03/2025

00:10 Customer Support escalated the slowness to Engineering. Investigation began immediately.

00:25 Signs of performance issues in the backend database were detected.

01:00 Resource contention was identified as the source of the slowdown.

02:07 Resources allocated to the database were increased.

02:13 UI responsiveness returned to normal.

02:16 Incident declared resolved and public status page updated.

Future consideration(s)

  • Implement database health monitoring to identify issues proactively.
  • Tune database cleanup and optimization settings to match usage patterns better.
Posted Jun 11, 2025 - 10:23 EDT

Resolved

This incident has been resolved.
Posted Jun 03, 2025 - 22:16 EDT

Update

We are continuing to investigate this issue.
Posted Jun 03, 2025 - 21:43 EDT

Investigating

Affected Services: Auvik Dashboard
Cluster(s): AU1

Description:
We are currently experiencing degraded performance loading the Auvik Dashboard. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible.

Impact:
Users may experience slower load times of the Auvik Dashboard.
Monitoring services are not impacted.

Next Steps:
We will update as more information becomes available.

Thank you for your patience as we work to restore full functionality.
Posted Jun 03, 2025 - 21:42 EDT
This incident affected: Network Mgmt (au1.my.auvik.com).