Service Disruption - Traffic Insights (TI) Data Stopped Processing on US4 Cluster

Root Cause Analysis

Duration of incident

Discovered: Nov 6, 2023, 13:04 - UTC
Resolved: Nov 7, 2023, 06:32 - UTC

Cause

Updates to code caused an unexpected restart of services that affected TI data flow on the US4 cluster.

Effect

The restart of services began a restart loop that prevented TI data in the US4 cluster from flowing into the user interface as expected.

Action taken

All times in UTC

11/06/2023

13:04 - After approval code is released into production.

13:05 - Services unexpectedly restart. The restart loop of services begins, which causes a delay in updating TI data in the US4 cluster.

13:34 - An internal alert is fired, notifying Auvik Engineering that TI data on the US4 cluster was delayed.

15:52 - Engineering begins its investigation.

16:45 - Engineering adjusts the TI data flow for clients on the US4 cluster to bypass the restart issue.

16:48 - Engineering can confirm TI data flow back into the US4 cluster client is working. Engineering monitors the reduction of TI data lag in the US4 cluster.

18:00 - Engineering continues to monitor the reduction in TI data lag from being current for clients in the US4 cluster. Additional resources are allocated to speed up the lag reduction. Engineering continues to monitor.

11/07/2023

02:38 - All TI data lag is confirmed to have caught up.

06:32 - All data processes are confirmed as up-to-date and working correctly. The incident is closed.

Future consideration(s)

• Auvik will update CPU limits on services related to Traffic insights to prevent resource bottlenecks.
• Auvik will investigate and determine service dependencies and better document possible conflicts.
• Auvik will update older code to take advantage of new services to prevent this type of incident with these services in the future.