Is 12twenty Down Right Now? Check if there is a current outage ongoing.

Access to the platform has stabilised, and users are no longer experiencing slow wait times or pages that fail to load.

We are continuing to work on a fix for this issue.

Some users are reporting that pages are slow to load or fail to load initially. We are investigating the issue.

Our team has resolved the degraded performance.

Our team is actively investigating the issue and working on a resolution. We will provide updates as more information becomes available. Thank you for your patience.

This incident has been resolved.

We are currently investigating an issue where admin users are reporting that they are unable to update or save changes to student profiles. Our team is actively working to identify the root cause and will provide updates as soon as possible.

The issue impacting our email delivery service has now been fully resolved. Our provider, SendGrid, has restored normal operations, and all queued emails have been successfully sent. We apologize for any disruption this may have caused and appreciate your patience throughout the incident.

We are currently awaiting an update from our email service provider, SendGrid, as they work to resolve the issue impacting our email service. No emails have been lost during this incident, and all queued emails will be sent once the issue is fully resolved by SendGrid.

.

We are currently experiencing an issue with our email delivery system. Emails are not being delivered as expected. Our team has been made aware of the problem and is actively investigating the root cause.

This incident has been resolved.

A fix has been implemented. The team is monitoring the resolution.

Some users have reported experiencing slow load times or 500 error pages. The team is aware of the issue and currently working on a resolution.

A fix has been implemented and has been successful. Please note that your emails, calendar sync activities and document packets will still be delivered over the course of the day, and no data has been lost.

A fix has been implemented and we are monitoring the results. Please note that your emails, calendar sync activities and document packets will still be delivered over the course of the day, and no data has been lost.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating reports of email deliverability issues

We have received reports that some users are unable to access the platform and are experiencing a "gateway" error. The team is aware of the issue and is working with our team to restore access. Access was restored at approximately 18:45UTC

The team has released a fix that restores access to the platform.

We have received reports that some users are unable to access the platform or are experiencing persistent "Oops" errors. The team is aware of the issue and is working to restore access to the platform.

Sites that were affected by the SAML Certificate Rollover can now access the platform successfully. If you are still receiving errors, please reach out to your IT team to request a refreshed of the 12twenty Metadata coming from InCommon.

Sites that use a Shibboleth InCommon SSO integration experienced issues where students or admins were not able to login using SSO between 2023-10-23 9:13pm PT and 2023-10-24 10:00am PT. This issue likely did not affect sites with custom SAML integrations or sites that updated the 12twenty metadata manually. This issue was a result of the InCommon metadata changes being approved but not yet published at the time of 12twenty's Shibboleth configuration change. InCommon's publication of the updated metadata is scheduled for 2023-10-24 12pm PT. The change was reverted but will need to be reapplied once the old certificate expires. Internal testing of the SAML certificate rollover process did not fully cover all use cases across our various school SSO implementations. We understand how disruptive these issues can be and will ensure that future changes will include more comprehensive testing across all use cases.

We have resolved the degraded performance.

The team has implemented a fix and we are actively monitoring site performance.

We have identified the source causing degraded performance. The team will be implementing a fix.

Some users are reporting degraded performance while accessing the site. Our teams are actively investigating this issue.

The issue with site access has been fully resolved and all systems are operational.

We are continuing to monitor for any further issues.

Site access has been fully restored. We are continuing to monitor.

A fix has been implemented and the teams will continue to monitor all sites.

The team has identified the issue and is actively working on implementing a resolution. An incremental improvement to site access is expected.

Users are reporting that sites cannot be accessed. The 12twenty team is actively investigating this issue.

The issue where an "Oops!" error appears when attempting to schedule an appointment has been resolved.

A fix has been implemented and we are monitoring the results.

The team has identified the issue and is actively working on a fix.

Users have reported that an "Oops!" error appears when attempting to book an appointment. The team has begun their investigation.

Our teams have resolved the ongoing issue for users attempting to update or save new records.

Our engineering team reports improved performance for users saving records. We are continuing to triage this issue and appreciate your patience. We will provide further updates when we have additional information to share.

We are continuing to monitor for any further issues.

Our engineering team is investigating an issue that is preventing users from saving some records. We will provide further updates when we have additional information to share.

12twenty partners may have been experiencing issues exporting schedules and application packets.

Some partners may have experienced an error while logging in via Single Sign-On (SSO). There was a change that caused a misconfiguration with the SSL certificate. This was identified and resolved shortly after the change occurred. A platform update was released and no further disruptions are expected.

One of our web server instances was accepting requests but responding extremely slowly to many requests. This issue lasted for about two hours and affected between 15%-30% of incoming requests before the offending server was restarted.

We believe a code issue caused the web server to hold onto too many existing requests given a certain load profile, preventing it from being able to accept new requests. This issue lasted for about 10 minutes and affected the majority of requests before the offending servers were restarted. We have disabled the feature we believe to be causing the issue and are working on a solution.

A recent deployment stalled while executing database migrations, perhaps due to database deadlocks. Many requests to the database timed out during this time, leading to errors. Lower environments did not experience this; we are still investigating the cause of the issue. The increase in timeouts/errors lasted between 7:35pm PT and 8:15pm PT.

Between 5:55pm PT and 6:05pm PT, users may have received an error while using the 12twenty platform. The errors were a result of a new version of the application being deployed that was not backwards compatible with the prior version of the application. Errors resulted during the deployment of this release. We apologize for the issues this may have caused and we will ensure that all future releases can be deployed without adversely affecting users.

Summary of impact: between 1/5/22 and 1/14/22, a small percentage of users on a handful of occasions (about 7 during this period of time) were faced with 502 errors when trying to access the site. These errors were ephemeral, but experienced by .5%-1.5% of users at the moment in time when one of the web application's instances was being automatically replaced due to hardware issues. Root Cause: working with the Azure support team, we determined that there was an issue with one of the instances serving the web application. Moving the web application to new instances has resolved the issue. Next steps: the replacement of instances should be completely transparent to the user and we are continuing to work with the Azure team to identify why this was not the case in our situation.

This incident has been resolved.

We made some configuration changes on the evening of 1/12/22 that we believe may resolve this issue. We are continuing to monitor.

We are still seeing some users experience this issue and are still working with Azure to investigate. Please let our support team know if a refresh or retrying the request does not result in a successful response.

No errors in the past 12 hours. We are continuing to monitor the situation and are continuing to work with Azure to fully resolve.

We are currently seeing up to 1% of requests for brief periods of time during the day receiving a 502 Bad Gateway error. We are currently working with Cloudflare and Azure to determine the root cause of the issue. Refreshing the page, retrying the request, or using a different browser will often result in a successful request. If you are experiencing this behavior or if you are consistently unable to access the site, please email support with how often you are experiencing this issue and how quickly the error message appears after executing your request. Thank you and we apologize for any inconvenience this has caused.

From Microsoft: DNS issue impacting multiple Microsoft services - Mitigated (Tracking ID GVY5-TZZ) Summary of Impact: Between approximately 21:30 UTC and 22:30 UTC on 01 Apr 2021, customers may have experienced intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live. Preliminary Root Cause: We are continuing to investigate the underlying cause for the DNS outage but we have observed that Microsoft DNS servers saw a spike in DNS traffic. Mitigation: Microsoft engaged resilient DNS capabilities to absorb the spike in DNS traffic. Next Steps: We apologize for the impact caused by this outage. We are continuing to investigate to establish the full root cause. Stay informed about Azure service issues by creating custom service health alerts: https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation.

See https://status.azure.com/en-us/status for more info SUMMARY OF IMPACT: Starting at approximately 21:30 UTC on 01 Apr 2021, a subset of customers may experience intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live. CURRENT STATUS: Microsoft engineering teams have been engaged and are currently investigating a potential underlying DNS issue. The next update will be provided in 60 minutes or as events warrant.

The fix that was implemented late night 1/1 resolved the issue. There was a recent code change that caused an issue where web server worker threads were not properly released. Once a certain load for a particular process was reached, the web server began denying new requests, which resulted in intermittent failed requests for various users. This issue was not immediately apparent in our non-prod environments because of the specific type of request and load required to cause the issue. The alerting that did occur was misdiagnosed and subsequently ignored. To ensure this issue does not occur in the future, we will be improving our load-based automated testing and internal release processes.

A fix has been implemented and we are monitoring the results

An issue is causing memory strain on the web application servers, leading to intermittent failed requests. This issue was likely caused by a code deploy late night on 12/31. We are currently investigating the issue.

We have identified the issue and reverted the change that caused the issue. Web app is operating normally now. Portmortem to follow.

Currently investigating an issue with the web servers responding to requests very slowly or timing out

## Summary of Impact On 2020-09-30, between 9:00am PT and 12:30pm PT, background jobs processed through 12Twenty's platform were delayed. Background jobs include the sending of emails \(including email verification codes\), document packet generation, data uploads, etc. As an example, when generating document packets, the user saw a ‘please wait' until our background job infrastructure was able to finish processing the jobs queued before the user’s job. ## Root Cause A number of personalized mass emails with large attachments were sent by numerous schools within a short timeframe through our platform. Due to the attachments, the processing of each email took ~30x longer than emails normally take to process. 12Twenty did not sufficiently prioritize the various types of background jobs that run through the platform. ## Mitigation The background job infrastructure self-recovered after a little over 3 hours of delays. ## Next Steps The below highlights some of the steps we are taking to ensure this type of incident does not occur in the future: * Ability to easily increase the number of background job worker processes * Improve the prioritization of background job types * Faster scale out of background job infrastructure * Work with our email sending provider to look into the excessive performance degradation when emails with large attachments are sent

Background jobs processed through the 12Twenty platform are experiencing delays. Background jobs include the sending of emails (including email verification codes), document packet generation, data uploads, etc. As an example, when generating document packets, the user will see a ‘please wait' until the background job infrastructure is able to finish processing the jobs queued before the user's job.

## **Summary of Impact** On 2020-10-07, between 11:20am PT and 1:30pm PT, background jobs processed through 12Twenty's platform were delayed. Background jobs include the sending of emails \(including email verification codes\), document packet generation, data uploads, etc. As an example, when generating document packets, the user saw a ‘please wait' until our background job infrastructure was able to finish processing the jobs queued before the user’s job. ## **Root Cause** An Azure service issue caused Office 365 calendar sync jobs to freeze up. These jobs stayed in a ‘processing’ state for over 2 hours and prevented other background jobs from being queued. From Azure: > Summary of Impact: Between 18:20 UTC and 18:42 UTC on 07 Oct 2020, a subset of customers may have encountered increased latency, packet loss, failed connections and authentication failures across multiple Azure services. Retries may have succeeded during this time and users who had authenticated prior to the impact start time were less likely to experience authentication issues. ## **Mitigation** Background job services were restarted around 1:00pm PT. The background job infrastructure took about 30 minutes to finish processing the delayed jobs in the queue. ## **Next Steps** The below highlights some of the steps we are taking to ensure this type of incident does not occur in the future: * Improve monitoring of background job infrastructure to ensure we are alerted in a more timely manner if delays are occurring * Determine why connectivity to Azure caused the jobs to hang and ensure that similar issues do not prevent other jobs from processing in the future

Background jobs processed through the 12Twenty platform are experiencing delays. Background jobs include the sending of emails (including email verification codes), document packet generation, data uploads, etc. As an example, when generating document packets, the user will see a ‘please wait' until the background job infrastructure is able to finish processing the jobs queued before the user's job.

## **Summary of Impact** On 2020-10-13, between 9:00am PT and 1:00pm PT, background jobs processed through 12Twenty's platform were delayed. Background jobs include the sending of emails \(including email verification codes\), document packet generation, data uploads, etc. As an example, when generating document packets, the user saw a ‘please wait' until our background job infrastructure was able to finish processing the jobs queued before the user’s job. ## **Root Cause** Similar to the experienced on 9/30: A number of personalized mass emails with large attachments were sent by numerous schools within a short timeframe through our platform. A few intermediary patches were deployed in the last two weeks, but they were not sufficient to fully address the problem. ## **Mitigation** The background job infrastructure self-recovered after about 4 hours of delays. ## **Next Steps** We know how frustrating these issues can be and we apologize for the delay in procuring an adequate resolution. A patch to specifically address the issue mentioned above was deployed on 10/13 - mass emails will no longer affect the processing of other background jobs like document packet generation and verification code emails. The below highlights some of the additional steps we are taking to ensure this type of incident does not occur in the future: * Further improve the prioritization of background job types: expecting to deploy within two weeks * Faster scale out of background job infrastructure: expecting to have a preliminary solution within the next week and a longer term solution is also in progress. * Work with our email sending provider to look into the excessive performance degradation when emails with large attachments are sent: still in progress * Improved monitoring: expecting to deploy within the next week.

Background jobs processed through the 12Twenty platform are experiencing delays. Background jobs include the sending of emails (including email verification codes), document packet generation, data uploads, etc. As an example, when generating document packets, the user will see a ‘please wait' until the background job infrastructure is able to finish processing the jobs queued before the user's job.