Affected Services: Internet Connection Service Cluster(s):US3
Description: The issue affecting Internet Connection Ping Checks has been fully resolved. Regular service has been restored, and all systems are operating as expected.
Impact: Users should no longer experience any issues related to this incident.
Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.
Posted Apr 01, 2025 - 10:52 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: Our team has implemented a fix for the issue affecting the Internet connection ping check for the tenants on the US3 cluster, and performance is returning to normal. We monitor the situation to ensure stability and confirm that the service remains fully functional.
Impact: Services are operating normally for most sites. We do continue monitoring for irregularities with a few sites that have been contacted
Next Steps: Tenants on the US3 cluster are still recovering and look healthy. We are attending to a few sites to regain full functionality.
Thank you for your patience, and we apologize for any inconvenience caused.
Posted Apr 01, 2025 - 09:54 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: Our team has implemented a fix for the issue affecting the Internet connection ping check for the tenants on the US3 cluster, and performance is returning to normal. We are monitoring the situation to ensure stability and confirm that the service remains fully functional.
Impact: Services should be operating normally; however, we continue monitoring for irregularities.
Next Steps: Tenants on the US3 cluster are still recovering and look healthy. We will continue to monitor the status of the tenants on US3 overnight and report back in the morning.
Thank you for your patience, and we apologize for any inconvenience caused.
Posted Mar 31, 2025 - 21:25 EDT
Monitoring
Affected Services: Internet Connection Service Cluster(s):US3
Description: Our team has implemented a fix for the issue affecting the Internet connection ping check for the tenants on the US3 cluster, and performance is returning to normal. We are currently monitoring the situation to ensure stability and confirm that the service remains fully functional.
Impact: Services should be operating normally; however, we continue monitoring for irregularities.
Next Steps: Tenants on the US3 cluster are still recovering and look healthy.
Thank you for your patience, and we apologize for any inconvenience caused.
Posted Mar 31, 2025 - 21:11 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: The 20-minute maintenance window for the internet connection service for all clusters has been completed. Services, including other monitoring services, are not impacted.
Next Steps: The US3 cluster is still going through its restart process.
We sincerely apologize for the extended window for this action.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 20:22 EDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 31, 2025 - 19:49 EDT
Update
Affected Services: Internet Connection Service Cluster(s):All Clusters
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: Auvik will perform an emergency cluster restart on US3 tenants at 00:00, which will take approximately 1.5 hours to complete.
At this time, Auvik will also perform a 20-minute maintenance window to allow for a restart of the Internet connection service for all of Auvik.
We sincerely apologize for the extended window for this action.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 19:48 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: Auvik has disabled alerts for clients on the US3 cluster. This action will continue for an additional hour until 23:00 UTC. This is a preventative measure as we work through false alerts for the internet connection checks. Clients may experience a slowed UI response time during this work. Any UI slowness should be very short, if noticeable at all.
We apologize for the extended window for this action.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 18:59 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: Auvik will disable alerts for clients on the US3 cluster for up to 1 hour starting at 22:00 UTC. This is a preventative measure as we work through false alerts for the internet connection checks. Clients may experience a slowed UI response time during this work. This UI slowness should be very short if it is noticeable at all.
We apologize for the late notice.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 18:19 EDT
Identified
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: Auvik will disable alerts for clients on the US3 cluster for up to 1 hour starting at 22:00 UTC. This is a preventative measure as we work through false alerts for the internet connection checks.
We apologize for the late notice.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 17:57 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: We will update you as more information becomes available or by 22:00 UTC.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 17:02 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: We will update you as more information becomes available or by 21:00 UTC.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 16:00 EDT
Update
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: We will update you as more information becomes available or by 20:00 UTC.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 14:54 EDT
Investigating
Affected Services: Internet Connection Service Cluster(s):US3
Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible.
Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted.
Next Steps: We will update you as more information becomes available or within the next hour.
Thank you for your patience as we work to restore full functionality.
Posted Mar 31, 2025 - 14:15 EDT
This incident affected: Network Mgmt (us3.my.auvik.com).