Service Degraded - Some Customers Experiencing Login Issues After Maintenance

Root Cause Analysis

Duration of the incident

Discovered: Mar 7, 2026 18:42 - UTC
Resolved: Mar 9, 2026 21:10 - UTC

Cause

Following a platform upgrade, an internal reconciliation process incorrectly identified a subset of user records as deleted. This occurred due to an error condition in a synchronization component that failed to process all expected user data during reconciliation.

Effect

Affected users were able to log in to the platform but were unable to access their expected sites or tenants due to missing user authorizations. In some cases, API access using previously generated API keys was also affected.

Monitoring, alerting, and data collection services continued to function normally throughout the incident.

Action taken

Engineering paused the process that spreads the changes across the platform to prevent further impact.

User access was then restored using backup data taken prior to the upgrade. This allowed the team to recover user access, permissions, and related access keys.

After restoration was completed and access was verified across all environments, the incident was declared resolved.

Future consideration(s)

• Implement additional monitoring to detect unexpected changes in user or authorization records.
• Add safeguards within reconciliation processes to prevent large-scale unintended deletions.
• Improve post-upgrade validation checks to identify abnormal record changes earlier.
• Enhance disaster recovery tooling and backup retrieval processes to speed up restoration efforts.