Learnosity Incident History

1. resolvedOct 26 · 20:10 UTC

   As of 20:10 UTC, we have seen no further issues with Item Pool or Local Device Assessment package creation or update. We're resolving this incident and will continue to investigate the root cause of the degraded performance. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

Reported by Learnosity on their status page.

1. postmortemOct 27 · 13:31 UTC

   Affected Systems and Regions

   On 21 October 2026, Learnosity temporarily took the Learnosity Demos site offline as a precautionary measure following the disclosure of two critical PHP vulnerabilities, CVE-2026-109618 and CVE-2026-109619, commonly known as PHP2Shell.
   The maintenance affected the Learnosity Demos site only. Learnosity's production APIs and services remained operational, with no customer-facing service disruption identified.

   Investigation

   The vulnerabilities affected PHP, a widely used server-side programming language used by Learnosity services, as well as an estimated 70% of internet applications. Both vulnerabilities were rated critical (CVSS 9.2) and could potentially allow unauthorized remote code execution.
   Learnosity's Platform and Security teams reviewed potential exposure across the platform and implemented additional monitoring and protective measures while awaiting official security updates.
   Our investigation and subsequent monitoring found no evidence of exploitation or compromise associated with these vulnerabilities.

   Resolution

   The PHP Project released security updates at 14:00 UTC on 21 October 2026. Learnosity immediately began deploying these updates, rebuilding all PHP services using the official PHP 8.3.36 release.
   Following completion and verification of the updates, the Learnosity Demos site was restored to service on 22 October 2026.
   All Learnosity Build servers have been patched, and no action is required for customers using Learnosity's hosted APIs and services.

   Prevention

   Learnosity continues to monitor its services for potential exploitation and maintains security monitoring and protective measures to identify and respond to emerging threats.
   Customers operating their own PHP environments, including those using the Learnosity PHP SDK in server-side integrations, should ensure their PHP installations are updated to a patched version following their hosting provider's or operating system's guidance.

Reported by Learnosity on their status page.

1. postmortemOct 27 · 13:27 UTC

   Affected Systems and Regions

   On 20 October 2026, Learnosity experienced scoring delays and intermittent request failures affecting Enhanced Math scoring in US-East-1. Customer impact began at approximately 13:06 UTC and ended at 13:47 UTC, lasting approximately 41 minutes.

   Affected scoring requests experienced longer than normal processing times, and some direct requests to the Data API scoring endpoint returned errors. For sessions saved through the Items API, affected scoring requests were queued and retried, resulting in delayed score availability rather than scoring failures.

   Assessments without math questions and services in other regions were unaffected.

   Investigation

   Our investigation identified a specific pattern of scoring requests that resulted in unexpectedly high processing times. This reduced the capacity available to process other scoring requests, causing delays and intermittent timeouts.

   The issue was associated with how certain scoring inputs were processed, rather than an unexpected increase in overall request volume.

   Resolution

   Scoring performance returned to normal by 13:47 UTC.

   At 15:12 UTC, Learnosity engineers implemented additional traffic isolation to prevent the affected scoring workload from impacting scoring operations for other customers.

   Prevention

   Learnosity has implemented changes to reduce unnecessary resource consumption and improve the recovery of affected scoring requests.

   Further improvements are underway to strengthen scoring request validation, improve processing safeguards and prevent individual scoring workloads from affecting the wider service.

   Additional isolation measures will remain in place while these improvements are completed and their effectiveness is verified.

Reported by Learnosity on their status page.

1. postmortemOct 28 · 22:14 UTC

   Affected Systems and Regions

   On 8 October 2026, Learnosity experienced intermittent API errors only affecting requests that specified the retired API version v2024.1.LTS in the US Virginia, US Oregon, Australia, and Ireland regions. This version reached End of Life in March of 2026

   After sunset, affected requests should have been automatically redirected to the latest long-term-support (LTS) release. Instead, some returned server errors.

   Customer impact began at approximately 00:14 UTC and continued intermittently until 15:08 UTC, varying by region. Only a portion of production servers were affected, so not all requests specifying this version failed.

   Investigation

   The incident was caused by an incomplete configuration update during the October 8 platform release, which retired v2024.1.LTS.

   Although the updated version mapping was correct in staging, a required release step to update production configuration servers was not completed. Consequently, some production servers retained an outdated mapping that directed requests to API versions that had been removed.

   Our existing release validation verified that deployed API versions were available but did not verify that retired versions redirected correctly in each production region. As a result, the configuration inconsistency was not detected during deployment.

   Resolution

   Learnosity engineers identified the outdated version mapping and restored service by regenerating configuration on affected production servers, region by region.

   By 15:08 UTC, verification confirmed that the correct version mapping was in place across all checked regions and requests were being handled correctly. The incident was formally resolved at 15:48 UTC.

   Prevention

   Learnosity is implementing improvements to its release and validation procedures to prevent similar configuration inconsistencies.

   These include automated verification that all production configuration servers are running the correct release version, integrating configuration updates directly into the deployment process so they cannot be inadvertently skipped, and expanding release validation to confirm that retired API versions redirect correctly in every region.

   We are also improving alerting to detect this specific class of error earlier and developing tooling to identify and refresh servers retaining outdated configurations.

   These measures are intended to ensure consistent deployments across regions, detect configuration discrepancies before they affect customers, and accelerate recovery should similar issues occur.

Reported by Learnosity on their status page.

1. postmortemOct 27 · 01:24 UTC

   Affected Systems and Regions

   On 2 October 2026, Learnosity experienced intermittent scoring errors and timeouts affecting math question types in US-East-1. Customer impact began at approximately 16:34 UTC and ended at 18:47 UTC, lasting approximately 2 hours and 13 minutes.

   Affected requests could experience delays of up to 40 seconds before failing. Other regions were unaffected.

   Investigation

   The incident occurred when demand on the math scoring service exceeded available capacity. Although capacity and scaling improvements had been introduced following an earlier incident, the service encountered an additional limit in its network configuration that prevented further instances from being provisioned.

   The impact was compounded by slow calculations that occupied instances longer than intended because calculation time limits were not reliably enforced under heavy load. Together, these conditions prevented the service from scaling sufficiently to handle demand.

   Resolution

   Learnosity engineers restored service by moving the scoring service to a network configuration with additional scaling capacity and increasing the compute resources allocated to each instance.

   These changes allowed the service to recover, with scoring errors clearing by 18:47 UTC.

   Prevention

   Learnosity has updated the calculation time limit enforcement mechanism to reliably terminate slow calculations under heavy load. Health checks have also been separated from calculation processing to prevent unnecessary instance replacement.

   The service now operates with additional compute and network capacity. Further architectural improvements are underway to prevent network capacity from limiting future scaling, and we are evaluating shorter calculation time limits to reduce the impact of unusually slow calculations.

Reported by Learnosity on their status page.

1. ResolvedOct 2 · 19:52 UTC

   As of 19:45 UTC, we have resolved the issue affecting Enhanced Math Scoring in the US-East-1 region. All APIs and services are functioning normally.Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required.Please reach out if you have any questions or concerns.

Reported by Learnosity on their status page.

1. postmortemOct 27 · 01:23 UTC

   Affected Systems and Regions

   On 11 September 2026, Learnosity experienced intermittent scoring errors and timeouts affecting math question types in US-East-1. Customer impact began at approximately 17:00 UTC and ended at 21:45 UTC, lasting approximately 4 hours and 45 minutes.

   Other question types continued to score normally, and other regions were unaffected. Most failed requests succeeded when retried.

   Investigation

   The incident was caused by insufficient capacity in the math scoring service following an increase in demand over the preceding weeks. The service reached its configured maximum number of instances and could not scale further to accommodate the load.

   This was compounded by limited compute resources per instance and slow calculations that occupied instances longer than intended. Under heavy load, the calculation time limit was not reliably enforced, and busy instances could fail health checks and be replaced, further reducing available capacity.

   Resolution

   Learnosity engineers restored service by increasing the number of running instances and raising the maximum instance limit. Normal scoring operations resumed by 21:45 UTC, and all remaining queued scoring requests were processed by 23:17 UTC.

   On 14 September, additional configuration changes were deployed to increase compute resources, improve automatic scaling, and reduce unnecessary instance replacement.

   Prevention

   Learnosity has increased scoring capacity, raised instance limits, and improved automatic scaling based on processor utilization. Health checks have been separated from calculation processing to prevent busy instances from being incorrectly identified as unhealthy.

   We have also improved calculation time limit enforcement and expanded monitoring and alerting for resource utilization, response times, and scoring errors. Further work is underway to strengthen network scaling capacity and evaluate shorter calculation time limits.

Reported by Learnosity on their status page.