Login issues
Incident Report for Auvik Networks Inc.
Postmortem

Service Disruption - Users unable to log in to Auvik

Duration of incident

Discovered: Oct 22, 2021 14:42 UTC
Resolved: Oct 22, 2021 18:12 UTC

Cause

A certificate used for authentication against multi factor authentication lost its chain authority.

Effect

Auvik was unable to accept any new logins.

Action taken

10/22/2021

14:42 UTC - Issues reported by internal users that logins to Auvik were failing.
14:43 UTC - Auvik engineering begins investigation.
14:59 UTC - Auvik posts the outage to its Status page.
15:14 UTC - Auvik attempts to roll the certificate back to the last known working configuration.
15:57 UTC - Auvik tests pointing authentication to another server.
16:30 UTC - Auvik’s roll back to the old certificate did not resolve the issue.
17:03 UTC - The changes to use a different server URL for authentication also failed to resolve the issue.
17::29 UTC - The changes to the different URL for authentication are reverted back.
17:44 UTC - Auvik discovers incomplete certificate chain authority information was applied with the certificate. The complete information is uploaded.
18:00 UTC - Authentication is verified to be working and users are now able to log in to Auvik. Engineering begins the validation of all certificate connections
18:12 UTC - Auvik posts that the issue is resolved on its Status page.

Future consideration(s)

  • Auvik will update certificate renewal and installation documentation to reflect issues discovered from this incident.
  • Auvik will time the application of certificate renewals during maintenance windows.
Posted Nov 02, 2021 - 12:18 EDT

Resolved
The issue is resolved. Logins are working again.
Posted Oct 22, 2021 - 14:12 EDT
Monitoring
A fix has been implemented. We are starting to see successful logins, but expect that some logins will fail until changes have fully propagated. We are monitoring and reviewing systems for other impact.
Posted Oct 22, 2021 - 13:52 EDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 22, 2021 - 13:02 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 22, 2021 - 11:34 EDT
Update
We are continuing to investigate this issue.
Posted Oct 22, 2021 - 11:00 EDT
Investigating
We are currently investigating this issue.
Posted Oct 22, 2021 - 10:59 EDT
This incident affected: Network Mgmt (my.auvik.com, us1.my.auvik.com, us2.my.auvik.com, us3.my.auvik.com, us4.my.auvik.com, eu1.my.auvik.com, eu2.my.auvik.com, au1.my.auvik.com, ca1.my.auvik.com) and Auvik TrafficInsights, Auvik Website (www.auvik.com).