Service Disruption - AU1 Cluster Performance Degradation

Root Cause Analysis

Duration of incident

Discovered: Jun 3, 2025 – 00:10 UTC
Resolved: Jun 3, 2025 – 02:16 UTC

Cause

The performance degradation was caused by resource limitations in the database system supporting the AU1 cluster. These limitations temporarily prevented the system from efficiently cleaning up and processing data. This led to slower load times until the underlying resources were increased.

Effect

Customers connected to the AU1 cluster experienced slow performance when accessing the Auvik Web UI. Pages were taking longer than usual to load, and in some cases, data within the interface appeared delayed or incomplete. While monitoring systems remained unaffected, the responsiveness was degraded during the incident window.

Action taken

All times are in UTC

06/03/2025

00:10 Customer Support escalated the slowness to Engineering. Investigation began immediately.

00:25 Signs of performance issues in the backend database were detected.

01:00 Resource contention was identified as the source of the slowdown.

02:07 Resources allocated to the database were increased.

02:13 UI responsiveness returned to normal.

02:16 Incident declared resolved and public status page updated.

Future consideration(s)

• Implement database health monitoring to identify issues proactively.
• Tune database cleanup and optimization settings to match usage patterns better.