Performance Degraded - Site dropdown not working for several clusters

Root Cause Analysis

Duration of the incident

Discovered: Aug 25, 2025 – 15:40 UTC
Resolved: Aug 25, 2025 – 16:45 UTC

Cause

A recent system update introduced a change that depended on data not yet available in our production environment. As a result, the site selector (drop-down) was unable to load correctly, which prevented users from switching between sites until the update was rolled back.

Effect

Customers were unable to switch sites in the platform, which limited access to some account information and historical data.

Action taken

All times are in UTC

08/25/2025

15:40 – Update deployed across production clusters.

16:17 – Customer reports received that the site drop-down was not loading.

16:26 – Engineering identified the change that introduced the issue.

16:27 – Incident declared; response team assembled.

16:40 – Deployment was rolled back.

16:45 – Service restored; site drop-down working as expected. Incident resolved.

Future consideration(s)

• Add extra checks before updates to ensure all required data is available in production.
• Improve safeguards so that unrelated changes are not deployed together.
• Build fallback mechanisms to ensure that new features do not impact existing functionality.
• Strengthen monitoring to detect and alert on similar issues in the future quickly.