Service Disruption - Hierarchical Data Display Issues

Root Cause Analysis

Duration of incident

Discovered: July 26, 2025 14:00 UTC
Resolved: July 31, 2025 12:45 UTC

Cause

Following a core system upgrade on July 26, 2025, multiple clusters experienced degraded performance due to unexpected issues during service initialization. This led to corruption in certain hierarchical data structures, which in turn impacted various user experiences.

A core infrastructure failure caused an overload in internal system processes, preventing certain backend services from initializing properly. As a result, critical hierarchical user-role data and related settings failed to load or loaded incorrectly across several environments.

Effect

The disruption impacted customers across multiple regions. Effects included:

• Missing custom settings such as alert preferences and interface configurations.
• Alert notifications being sent to incorrect recipients.
• Inability to access dashboards or view accurate site data.
• Site maps failing to render correctly.
• Login issues for end-users in some environments.
• Inconsistent or missing hierarchical relationships in site selectors.
• Temporary loss of monitoring due to disassociated shared collectors.
• These issues collectively degraded service functionality, limited access for internal support teams, and disrupted monitoring workflows.

Action taken

All times are in UTC

07/26/2025

11:00 – Core upgrade initiated.

13:27 – Initial service failures observed. Engineering begins recovery processes.

14:19 – Backend query failures reported. Engineering continues to recover and stabilize services.

16:24 – Incident response team mobilized.

07/27/2025
(Services restored throughout the day)

Hierarchy services replayed and clusters rebooted to stabilize services.

Alerting system functionality restored.

7/28/2025
(Services restored throughout the day)

Shared monitoring agents reassociated.

Back-end service migrations initiated for affected tenants.

7/28/2025
(Services restored throughout the day)

Repair scripts run to reset affected data and restore processing pipelines.

7/30/2025-07/31/2025
(Services restored throughout the day)

Corrupted tenants identified and corrected.
All affected clusters verified for service integrity.

07/31/2025
12:45 - Incident considered resolved.

Future consideration(s)

• Improve validation checks during post-upgrade procedures to avoid cascading service impacts.
• Temporarily pause specific background services (e.g., data cleaners and processors) during upgrades until core services are stable.
• Implement automated detection for corrupted tenant hierarchies or missing role-based configurations.
• Revisit default alert notification behavior to avoid unintended mass-notifications.