NTS and OIT System Incident History

1. resolvedAug 28 · 21:27 UTC

   Google reports this issue has been resolved. Per their status dashboard: "The issue with Google Chat has been resolved for all affected users as of Friday, 2026-08-28 13:45 PDT. From preliminary analysis the issue was triggered due to a recent change which was rolled back to mitigate the impact. We thank you for your patience while we worked on resolving the issue." Thank you for your time as we followed this resolution!

Reported by NTS and OIT System on their status page.

1. resolvedAug 28 · 17:44 UTC

   Canvas courses remain able to be cross-listed by Canvas admins and instructors without issue, and we are considering this incident resolved at this time. If you are still experiencing problems or require your course content to be recovered and have not already submitted a help ticket, please do so by emailing helpdesk@alaska.edu with a description of your issue/request and include the CRNs for the affected sections. Thank you so much for your patience as this issue was resolved!

Reported by NTS and OIT System on their status page.

1. postmortemAug 26 · 20:11 UTC

   IRU Authentication Outage Problem Impact Analysis Event Occurrence: 20260803 # Background The IRU authentication service provides user authentication capabilities for managed macOS devices and web-based access through Kandji Passport. The service relies on integration between Kandji Passport, Microsoft Entra ID, Conditional Access policies, and Duo MFA to provide secure authentication for users. # Break Down of the Problem On August 3, 2026, users experienced authentication issues when attempting to access IRU services. The issue affected users attempting to authenticate through Kandji Passport Web Login and impacted the ability to complete the login process. The issue was investigated through Entra ID sign-in logs, Conditional Access policies, Duo MFA authentication flow, and Kandji Passport authentication services. # Target State / Goal  A functional authentication into IRU-managed macOS devices. # Root Cause Analysis  ‌ Changes to Microsoft’s Conditional Access methods, effective as of Jun 15, 2026: * [https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-enforcement-resource-exclusions](https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-enforcement-resource-exclusions) * [https://techcommunity.microsoft.com/blog/microsoft-entra-blog/upcoming-conditional-access-change-improved-enforcement-for-policies-with-resour/4488925](https://techcommunity.microsoft.com/blog/microsoft-entra-blog/upcoming-conditional-access-change-improved-enforcement-for-policies-with-resour/4488925) The changes outlined above now require DUO to be set as the default login method on a user’s account in M365.  # Develop Countermeasures  Immediate countermeasure developed by adding users to Entra group that enabled the newer DUO authentication method. # Implementation of Countermeasures IT staff completed a change that added all users to receive this fix.  # Follow Up / Review Regularly review progress of long term solution implementation.

Reported by NTS and OIT System on their status page.

1. postmortemAug 26 · 18:27 UTC

   Banner Employee Service Unresponsive Problem Impact Analysis Event Occurrence: August 10, 2026 10:30am to 10:53am  # Background The Banner Employee Self Service application is the main service that employees submit timesheets, view their leave balances, and similar items related to their employment across the University of Alaska System. The vendor Ellucian hosts the University Banner and Banner Self Service instances and is responsible for ensuring their availability through monitoring and corrective actions.  # Break Down of the Problem At approximately 10:35am the UAF NTS Service Desk reached out to Banner team in Slack to inform them about reports of extreme slowness in Banner. A brief review revealed that only the Employee Self Service module was encountering issues. The Service Desk recommended a work around of trying an incognito browser, which worked for some. The EAS team reached out to Ellucian and created a P1 ticket. The service began functioning again as expected at approximately 10:53am.  # Target State / Goal  All components of Banner and Banner Self Service should be available with the expectations of the daily maintenance window and other scheduled outage times.  # Root Cause Analysis  The Ellucian cloud team which was engaged during restoration shared the following related to this event. Review of their monitoring showed that a performance degradation started at 10:42am and became fully unavailable at 10:50am. The automation related to the containers completed a successful start up at 10:53am and by 10:57am Employee Self Service fully stabilized. Ellucian did not evaluate any further related to cause of the outage. The EAS team is aware of a product defect in the Banner Employee Self Service software that was shown to cause this sort of event, first experienced in March and April of this year related to a product defect that has yet to be corrected in production, but is scheduled for application.  # Develop Countermeasures  To resolve this for the future, we expect that applying the recent defect resolution fix will correct this issue.  # Implementation of Countermeasures September 12-13, 2026: Deploy Banner related upgrades that resolve the identified defects.  # Follow Up / Review September 14, 2026: Validate upgrades for resolution were deployed.

Reported by NTS and OIT System on their status page.

1. resolvedAug 22 · 00:16 UTC

   We've had no further reports of issues calling the University, and the Fairbanks North Star Borough School District confirmed that their similar issues have been resolved. We are considering this outage resolved at this time; please call your campus IT service desk if you continue to have any issues contacting any UA campus by phone! Thanks so much for your time.

Reported by NTS and OIT System on their status page.

1. postmortemAug 19 · 21:26 UTC

   [vdi.alaska.edu](http://vdi.alaska.edu) access disruption. ‌ Expired Unified Access Gateway Certificates. Date Time:07/23/2026 4:00 PM.  # Background [vdi.alaska.edu](http://vdi.alaska.edu) provides user access to the VMware/Omnissa Horizon VDI environment.  ‌ Both Unified Access Gateway appliances \(exist on the VXrail platform\) provide the external access layer for users who want to access their VDI both on campus or remotely.  ‌ # Break Down of the Problem Both certificates on each UAG expired on July 23, 3:59 PM, 2026.  ‌ The problem was noticed Friday morning July 24th around 8:30 am,  when users reported they could not access their “My Desktop” VDI environments when resolving to [vdi.alaska.edu](http://vdi.alaska.edu) ‌ ‌ When a user tried to resolve to [vdi.alaska.edu](http://vdi.alaska.edu) through their browser, they received a “Your connection is not private” notice and then would be blocked from reaching the [vdi.alaska.edu](http://vdi.alaska.edu) Horizon page at all.   ‌ Actions taken to restore service: Requested renewal certificate from PAWS. This cert contained the primary domain name, [vdi.alaska.edu](http://vdi.alaska.edu) but also included the alternate names \(both UAG names\)    [fbk-uag1.apps.ad.alaska.edu](http://fbk-uag1.apps.ad.alaska.edu) [fbk-uag2.apps.ad.alaska.edu](http://fbk-uag2.apps.ad.alaska.edu) [fbk-uag1.io.apps.ad.alaska.edu](http://fbk-uaf1.io.alaska.edu) [fbk-uag2.io.alaska.edu](http://fbk-uag2.io.alaska.edu) ‌ ‌ Duration of outage and time that service was restored:  Thursday July 23rd 4:00 pm - Friday the 24th 2:00 pm. ‌ ‌ ‌ # Target State / Goal  ‌ Both UAG’s should continuously present valid, trusted certificates and external and internal VDI access  ‌ # Root Cause Analysis # Both UAG appliances were presenting expired TLS certificates. TSS received a list of VXrail server appliances that had estimated times of when their certificates were expiring. Both UAG appliances were not in this list so we were unable to dictate when their certificates were going to expire.  # Develop Countermeasures  I created a master certificate spreadsheet that now contains both UAG’s certificate information to include expiration dates. I’m sharing it with our TSS team leads so we can continue to monitor expected certificate expirations in the future. Requested access to Certinext service for future certificate renewals.  ‌ Schedule for the implementation: The above should allow us to request and implement the certs before they expire on both UAG’s before they expire.  # Follow Up / Review Verify certificate expiration dates on both UAG’s by monitoring the master certificate spreadsheet. Put in a request to PAWS to renew the certificate, plan a brief CAB outage, apply the certificate to both UAG’s and then confirm renewal was successful and access occurs

Reported by NTS and OIT System on their status page.

1. postmortemAug 19 · 21:25 UTC

   # **Root Cause Analysis – Argus Server Incident** ## **Executive Summary** The Argus server experienced a rapid loss of available space on the Windows C: drive. During investigation, PAWS identified significant disk activity occurring within the Lenel application log directory. Available disk space continued to decrease and eventually reached approximately 37 MB. PAWS cleanly shut down the server to prevent the operating system from reaching 0 available disk space and potentially causing additional system issues. PAWS expanded the C: drive and successfully returned the server to service. Following recovery, disk utilization stabilized and approximately 20 GB of free space remained available. PAWS is responsible for the underlying Windows server infrastructure but does not administer or support the Lenel application. The cause of the increased application-generated disk activity has not been determined and requires investigation by the application owner/vendor. ‌ ‌ ## **Incident Impact** The Argus server became unavailable during the incident and infrastructure recovery activities, resulting in an interruption to services dependent on the server. Reported impacts included on-campus door card reader functionality. ‌ ‌ ## **Timeline** **All times are Alaska Time.** **Approximately 11:04 AM** – PAWS was notified of an issue involving the Argus server and reported impacts to on-campus door card readers. The associated incident was TDX 991008. **Approximately 11:23 AM** – PAWS identified rapidly decreasing available space on the Windows C: drive. Resource Monitor showed significant write activity occurring within: `C:\ProgramData\Ln\logs` PAWS attempted to recover available disk capacity; however, C: utilization continued to increase. The C: drive eventually reached approximately **37 MB of available space**. PAWS cleanly shut down the VM to prevent the Windows system drive from reaching 0 available space and potentially causing additional operating system issues. PAWS performed the infrastructure work necessary to increase the capacity of the system drive. **Approximately 3:34 PM** – Infrastructure work was completed and the C: drive was expanded. The server was brought back online and PAWS verified normal server and disk operation. **Approximately 3:49 PM** – The server was reported operational. The following day, C: remained stable at approximately **19.9 GB free**, with no recurrence of the rapid disk consumption. ‌ ‌ ## **Root Cause** The immediate cause of the server outage was exhaustion of available capacity on the Windows C: drive. During the incident, PAWS observed significant disk write activity within the Lenel application log directory: `C:\ProgramData\Ln\logs` This activity rapidly consumed the remaining available capacity on the system drive. The underlying reason for the increased application-generated disk activity has **not been determined**. PAWS is responsible for administration of the underlying server infrastructure but does not administer, configure, or support the Lenel application. Determining why the application generated the increased disk activity requires investigation by the application owner/vendor and is outside the scope of this infrastructure RCA. ‌ ‌ ## **Contributing Factor** ### **Limited Available System Disk Capacity** The server did not have sufficient remaining C: drive capacity to absorb the unexpected increase in application-generated disk utilization. Available capacity eventually decreased to approximately **37 MB**, requiring PAWS to shut down the server to prevent Windows from reaching 0 available disk space. ‌ ‌ ## **Resolution** PAWS completed the following infrastructure recovery actions: * Investigated the rapidly decreasing C: drive capacity. * Identified the source directory associated with the increased disk activity. * Cleanly shut down the server before the system drive reached 0 available space. * Increased the capacity available to the Windows C: drive. * Returned the server to service. * Verified normal server and disk operation. * Continued monitoring disk utilization following recovery. Following recovery, the C: drive remained stable at approximately **19.9 GB free**. ‌ ‌ ## **Corrective and Preventative Actions** ### **Disk Capacity Monitoring** PAWS will review disk capacity monitoring and alert thresholds for the Argus server to provide sufficient notification when system drive capacity approaches critical levels. ### **Server Capacity** PAWS will continue monitoring C: drive utilization to verify that the additional capacity provides sufficient operating headroom under normal server conditions. ### **Application Investigation** The application owner/vendor will need to determine the cause of the unexpected increase in Lenel-generated disk activity and identify any application-level corrective actions that may be required. Application configuration, logging, retention, and application-level remediation are outside the scope of PAWS server administration. ‌ ‌ ## **Current Status** The Argus server is online and operational. The C: drive remained stable at approximately **19.9 GB free** during follow-up monitoring, with no recurrence of the rapid disk consumption observed during the incident. PAWS considers the **server infrastructure portion of the incident resolved**. The underlying cause of the Lenel application behavior remains undetermined and requires follow-up by the application owner/vendor.

Reported by NTS and OIT System on their status page.