Service Disruption - Cloud Ping Check Not Responding

Root Cause Analysis

Duration of incident

Discovered: Jan 16, 2024, 21:40 - UTC
Resolved: Jan 16, 2024, 22:33 - UTC

Cause

There was a significant spike in CPU/memory resources for the ping services in the product.

Effect

Auvik clients with Internet connection checks enabled received a large volume of connection alert failures.

Action taken

All times in UTC

01/16/2024

21:17 - Auvik Support alerted Auvik Engineering of a sudden influx of tickets concerning failed Internet connection checks

21:27 - Engineering confirms there was no disruption to the number of connected agents

21:31 - Engineering confirms there has been an escalation in CPU/memory for the ping server

21:48 - A broken backend connection was deleted and recreated.

21:56 - Engineering confirms that resource demands start to decrease and manually confirms clients that reported connection alerts are now responding

22:08 - Engineering confirms with its alerting team that there’s no manual intervention needed for the alerts that were fired; they will resolve themselves

22:33 - Incident has been resolved - alerts resolved themselves, and resources decreased to expected values for the affected service.

Future consideration(s)

• Auvik will create internal alerting for the Ping services.
• Auvik will create a failover instance of the Ping service to prevent a single point of failure situation in the future.