Historical record of incidents for Asana
Report: "MCP Maintenance"
Last updateAsana's MCP server is currently undergoing maintenance.
Report: "Planned Maintenance"
Last updateOn Saturday, May 10th, 2025, between 8:00 am to 10:00 am pacific time, users may experience interruptions or limited availability while we upgrade our databases. During this maintenance window, Asana web and mobile applications and API may be affected. Thank you for your understanding and patience. Please contact support for any questions.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Transient application errors"
Last updateWe've paused upgrades for the remaining customer databases, and will complete the remaining upgrades after looking into why we saw more errors than expected.
We started upgrade processes for databases; these are proceeding, but some restarts are taking longer than expected. As a result some customers are encountering transient errors during this process. We expect each customer database upgrade to complete within a few minutes; and no data will be lost.
Report: "Transient application errors"
Last updateWe started upgrade processes for databases; these are proceeding, but some restarts are taking longer than expected. As a result some customers are encountering transient errors during this process. We expect each customer database upgrade to complete within a few minutes; and no data will be lost.
Report: "SAML Login Failures"
Last update**Incident:** From April 4 to April 8, 2025, a subset of users were blocked from logging in with SAML, despite no changes to their identity provider setup. We believe the root cause to be a combination of recent code changes upgrading our infrastructure. These changes inadvertently affected how SAML responses were processed. **Impact:** Approximately 0.5% of users who authenticate through SAML were unable to log in to their Asana accounts during this period. No customer data was lost or compromised, and users who do not use SAML were unaffected. Our Customer Support team worked with affected domains to restore access while we resolved the issue. **Moving forward:** Our existing monitors did not trigger for this incident, which led to an extended period of downtime. We are improving existing monitoring and increasing monitor coverage to allow us to detect and respond to authentication issues more quickly in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
From April 4th to April 8th, approximately 0.5% of users using SAML for authentication experienced errors during login.
Report: "SAML Login Failures"
Last updateFrom April 4th to April 8th, approximately 0.5% of users using SAML for authentication experienced errors during login.
Report: "App and API downtime"
Last update**Incident**: A configuration change triggered a bug affecting how internal systems process configuration data, leading to application downtime. The issue impacted internal server uptime, resulting in approximately 28 minutes of degraded service on March 25, 2025, from 22:32 UTC to 23:00 UTC \(3:32 PM to 4:00 PM PDT\). **Impact**: During the incident, customers experienced difficulties accessing Asana on web and mobile apps, including login failures and errors while using the application. Internal services gradually recovered after the problematic change was reverted. **Moving forward**: We are enhancing our configuration validation processes to prevent similar issues. Additional monitoring and safeguards are being introduced to ensure rapid identification and recovery from similar incidents. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
Our systems have recovered, and we are continuing to monitor.
Asana App and API had about 9 minutes of downtime due to a configuration error; a fix has been implemented, and we are observing for recovery.
Report: "App and API downtime"
Last updateThis incident has been resolved.
Our systems have recovered, and we are continuing to monitor.
Asana App and API had about 9 minutes of downtime due to a configuration error; a fix has been implemented, and we are observing for recovery.
Report: "App and API errors"
Last updateThis incident has been resolved. We are doing root cause analysis and will post an update within 72 hours.
We have scaled up affected infrastructure are continuing to monitor application health and metrics.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We saw a high rate of errors due to overload of internal infrastructure. We see partial recovery, and are continuing to investigate.
Report: "App and API errors"
Last updateThis incident has been resolved. We are doing root cause analysis and will post an update within 72 hours.
We have scaled up affected infrastructure are continuing to monitor application health and metrics.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We saw a high rate of errors due to overload of internal infrastructure. We see partial recovery, and are continuing to investigate.
Report: "Asana down"
Last updateIncident: At 2025-02-06 15:37 UTC, a configuration change caused some of our servers to crash, leading to complete downtime for 20 minutes. We rolled back the configuration, but this event triggered a cascade failure in network routing components; our systems recovered over the next 30 minutes, with full recovery by 2025-02-06 16:30. Impact: Users were unable to use Asana during this event in all regions, with downtime including App, API, and Mobile. No customer data was lost. Moving forward: We have added validation to prevent a repeat of this issue, and are updating our networking components to avoid this form of cascade failure. We will be moving to staged configuration rollouts to identify this class of issue without causing downtime for customer traffic. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified a cause and are implemented a fix.
We are seeing elevated errors for App and API, and are investigating.
Report: "Asana down in US"
Last updateIncident: Starting at 2025-02-05 21:05 UTC, a configuration change caused a large increase in server logs, overloading logging infrastructure and causing server restarts. This caused cascade failure as other servers were overloaded. The configuration change was reverted, and our application recovered after the servers restarted, with recovery by around 2025-02-05 21:30. Impact: Users saw errors loading our application and loading data within the application. No customer data was lost. Moving forward: We have removed the source of logs which resulted in overload, and are improving monitoring and resilience of the system which triggered failure. We will also be moving to staged configuration rollouts to help reduce or avoid customer impact for similar issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have made configuration changes and are monitoring for recovery.
We are continuing to investigate this issue.
We are seeing a high rate of errors for the Asana application in the US region.
Report: "Partial Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers. Our Development Team is currently investigating this issue and we're hoping to get it fixed as soon as possible. Sincere apologies for the inconvenience; if you're impacted by this issue, please keep an eye on this page for the latest updates.
Report: "Asana unavailable"
Last updateIncident: During an automated deployment of new code, an error in an internal networking service caused several system components to retain stale network connections. This caused requests to fail across a portion of our European data center. Impact: Users in our European datacenter were unable to load the Asana web app between 2024-12-02 2340UTC and 2024-12-03 0050UTC. The mobile app, our developer API, and app integrations saw degraded performance and an inability to modify or add data to Asana. Existing sessions on the Asana web app were mostly unaffected by this incident. Moving forward: We are adjusting our configuration of the library responsible for incorrectly retaining connections to avoid similar issues in the future. Additionally, Asana is in the process of upgrading and simplifying our internal routing infrastructure. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Intermittent issues with SAML login"
Last updateIncident: A gradual rollout changing Asana’s network routing resulted in certain internal requests for our non-North American clusters being misrouted. This caused sporadic errors loading information for users whose Asana domains are hosted in non-US clusters. Impact: Users with SAML SSO configured saw intermittent errors when attempting to login to Asana between November 18, 11:02 UTC and November 19, 01:27 UTC. Moving forward: We are improving our internal monitoring to promptly identify issues with future networking changes and working to make our login flow more resilient to data loading errors.Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We are currently investigating this issue.
Report: "Issues in some regions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are experiencing issues in regions other than the US due to a network issue. We've identified the problem and are working on a fix.
Report: "Issues with signing up"
Last updateFrom 7 AM - 3 PM UTC time, there was a problem with signing up to Asana. This issue has now been resolved, and users should be able to sign up again.
Report: "Major outage"
Last updateOur engineers have fixed the issue and Asana which cause global downtime for ~10 minutes and connectivity issues for a small subset of users for around an hour after that. Asana is now running normally for all our customers.
Report: "Errors loading Asana"
Last update**Incident:** On 2024-10-10, between 01:23 and 02:31 UTC a software change caused overload to a shared resource, leading to application errors. The change was rolled back, with recovery for most users around 02:31 and full recovery by 02:39 UTC. **Impact:** Users were unable to start new sessions on the web and desktop apps, though mobile and existing sessions were unaffected. No data was lost. **Next Steps:** We will conduct a retrospective using the 5 Whys method to improve monitoring, tooling, and playbooks. We will also update our software to manage load more efficiently and reduce reliance on globally shared resources to prevent similar issues in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We have seen full resolution. We add a postmortem within 72 hours.
We are continuing to monitor for any further issues.
We have reverted the change, see recovery, and are monitoring.
Users who are attempting to open Asana are seeing errors. We believe this is due to a deployment, and we are reverting this change.
Report: "Errors accessing Asana"
Last updateIncident: An experimental flag was enabled that changed initial page loading behavior. With the flag enabled, users were inadvertently directed to load a code version that was not servable by all groups of servers. Subsequent requests would fail when requesting this incompatible version, including requests to report the page load errors. Impact: From 16:52 to 18:59 UTC, 10% of users were unable to load the Asana application in the desktop app or web browser. Existing sessions, as well as mobile and API traffic, were not impacted. Moving forward: We reverted the configuration change and identified gaps in our monitoring. We are adding monitoring and alerting for version mismatches during page load in order to more quickly identify similar regressions in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've received reports of users unable to access the Asana web and desktop application and are currently investigating.
Report: "Webapp degraded performance and crashes"
Last updateWe expect that all user-visible impact has been resolved.
We are investigating issues which appear to include poor application performance followed by session crashes, affecting groups of users for approximately 15 minutes.
Report: "Application and API errors"
Last update**Incident:** On 2024-08-01 at 14:09 UTC, an increased connection rate on a central database saturated the database’s ability to accept new connections. As a result, Asana became unavailable to users in all regions. In response, Asana engineers throttled traffic for automations, webhooks, and API, and were able to restore primary functionality of the Asana application at 15:01 UTC. The application continued to run with sporadic unavailability until 18:22 UTC, when the underlying cause of the increased connections was determined and disabled. At this point the application was fully operational, but asynchronous workloads such as automations and notifications were delayed due to the work enqueued while traffic was throttled. The incident was resolved when asynchronous work queues fully caught up at 19:39 UTC. **Impact:** From 14:09 UTC to 15:01 UTC, users were unable to access Asana through the web, mobile, or API. From 15:01 UTC to 18:22 UTC users could access Asana through the web and mobile, but experienced sporadic unavailability. Access to the API was restored gradually between 15:15 UTC and 15:58 UTC. From 15:01 UTC to 19:39 UTC users experienced delayed automations, background actions, webhooks, and event streams. No customer data was lost. **Moving forward:** We have reverted the change that caused the increased connections. We are adding monitoring to detect changes in database connection rates, so we can identify potential problems earlier. Beyond this, we will be replicating data from the central database to other databases to reduce the impact of any single database being overloaded.
This incident has been resolved.
We've identified the root cause and are monitoring our systems
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are investigating application issues related to overload, and will provide updates.
Report: "Automations and Background Actions Delayed"
Last updateOn 2024-07-26, a deployment of new code introduced a bug which caused delays for automations and background actions for a subset of our customers. Jobs were delayed between 00:55 and 12:30 UTC. We reverted this deployment, which allowed our system to catch up. Impact: During the incident, a subset of customers experienced delayed background actions, such as rules, email notifications, imports, and exports. For a small fraction of customers, during a 7-hour period, automations may not have executed as intended. Moving forward: We have identified the root cause of this particular bug and will improve our systems and processes to prevent it from happening in the future.
Report: "Asana unreachable for some users"
Last updateIncident: On 2024-07-01 22:30 UTC, changes related to ongoing work towards smaller, independent deployment environments resulted in an invalid configuration affecting approximately 18% of our customers in our US datacenter. Resolution was delayed due to an unrelated issue with our infrastructure configuration management tools, but the impact was resolved for half of the affected users at 23:35 and for all affected users by 23:42. Impact: Affected customers were unable to open our application \(either with a web browser or with our desktop application\). Existing sessions worked, and no data was lost. Mobile and API were unaffected. Users in other data centers \(and ~82% of users in the US datacenter\) were unaffected by this event. Moving forward: We have done a retrospective analysis using the [5 Whys process](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/), and have identified bugs, process improvements, and monitoring improvements to avoid future downtime while rolling out more fine-grained deployment environments. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Some users were unable to connect to Asana; we investigated and have fixed the issue. We will post a postmortem with additional information.
Report: "Page load failures"
Last updateIncident: An off-peak routine update of our service handling new page loads caused us to deploy code that depended on an artifact that wasn't yet available. The net effect was that all traffic attempting to do a fresh page load was broken for 48 minutes from June 28 5:52 pm PT to June 28 6:39 pm PT. Impact: Asana was unavailable to new page loads, but existing sessions continued to work as normal. Moving forward: This incident involved interaction across teams and systems. Our postmortem of this event identified operational and tooling changes to reduce the likelihood of incidents -- and decrease the time to resolution. We have an ongoing [5 Whys](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/) analysis for this incident.
We've resolved the issue where some users were unable to load the Asana application. We're investigating the root cause. We apologize for the inconvenience.
Report: "Slow loads and degraded mobile app usability"
Last update**Incident**: On June 28, 2024 between 0334 and 0356 UTC, network maintenance was performed to remap internal endpoints, with unforecasted negative effects on communication between internal services. This caused a large amount of requests to Asana during the maintenance period to either drop or time out. **Impact**: App usability was degraded for US-based users, resulting in load times of up to a minute. Users of the mobile app in our US, European, Australian, and Japanese data centers were also affected to a lesser extent by elevated error rates. **Moving forward**: We've identified process gaps during internal retrospective. Moving forward, we'll perform internal endpoint remappings more incrementally to avoid app disruption, and will apply a more thorough approval process for similar maintenance procedures. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Incident: On June 28, 2024 between 0334 and 0356 UTC, network maintenance was performed to remap internal endpoints, with unforecasted negative effects on communication between internal services. This caused a large amount of requests to Asana during the maintenance period to either drop or time out. Impact: App usability was degraded for US-based users, resulting in load times of up to a minute. Users of the mobile app in our US, European, Australian, and Japanese data centers were also affected to a lesser extent by elevated error rates. Moving forward: We've identified process gaps during internal retrospective. Moving forward, we'll perform internal endpoint remappings more incrementally to avoid app disruption, and will apply a more thorough approval process for similar maintenance procedures. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Report: "Elevated error rates"
Last updateIncident: Asana uses several load balancers to route incoming requests to the correct backend service. As our volume of traffic increased over time, these load balancers consumed more resources to handle the additional requests. High traffic pushed computing resource utilization past a critical threshold leading to queueing of slow requests and eventual failures. This incident was made worse by certain error reporting mechanisms sending additional requests through the same load balancing infrastructure. Impact: Between 15:08 and 16:50 UTC on June 17, 2024, requests to load the Asana application, use of the Asana mobile applications, and calls to our API were delayed or failed at an elevated rate. Users in our European, Australian, and Japanese data centers may have experienced additional crashes while using Asana. Moving forward: In the short term, we have increased the capacity for our load balancing infrastructure to handle our current traffic as well as anticipated growth, and improved the monitoring to alert us before failure. In the longer term, we have efforts underway to replace this component entirely with routing and load balancing infrastructure that automatically scales in response to increases in traffic. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're currently experiencing some difficulties; as a result, a majority of our users are unable to access Asana. Our Development Team is currently working to resolve this issue as soon as possible. Sincere apologies for the inconvenience caused, please keep an eye on this page for the latest updates.
Report: "Asana Webhooks [Partial Outage - Fixed]"
Last updateWebhook subscriptions filtered to a specific story type (for example, comment_added) or a specific task type (approval, milestone, default_task) were dropped and not delivered between Friday June 7th, 2024, 1:16 AM (UTC) and Mon, Jun 10, 2024, 19:37 PM (UTC). A change in the data pipeline introduced this issue and has now been rolled back. We will share our plans to prevent similar issues after completing an analysis of the incident. Our engineers have fixed the issue and Asana is now running normally. Our sincere apologies for any inconvenience caused.
Report: "Australia, Japan, and EU downtime"
Last update**Incident:** Around 2024-06-06 21:41 UTC a software deployment started sending traffic to 3 regions \(AU, JP, EU\) which triggered a version incompatibility bug for subsequent new sessions. The rate of errors increased after a few minutes, and our engineers were alerted due these errors at 22:07 UTC. After investigation we rolled back the new version, with rollback completing around 23:08 UTC. **Impact:** Affected users who initiated new sessions during the timeframe saw an elevated rate of crashes. Existing sessions from before this timeframe were not affected. **Moving forward:** In the short term, we are changing how we deploy software so that all global regions will get software deployments at the same time, with validation; that could have prevented this incident. We will also update our incident response training and monitoring to improve response time. In the long term, we have a project to ensure that client and server versions are consistent for the software which encountered this incident’s error, which would allow us to avoid this class of issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Asana users whose data is hosted in Australia, Japan, and the EU were unable to start new sessions, seeing application crashes for both web browser and desktop applications.
Report: "Outage"
Last update**Incident**: On 2024-04-08 at 19:40 UTC, an operator ran a routine database maintenance script on a recently-upgraded database containing data required for user sessions. At 19:50 UTC, the script experienced unexpected behavior in MySQL which prevented the database from creating new connections. As a result of this issue, starting at 19:53 UTC, the database was unable to respond to requests and Asana became unavailable to users. At 20:11 UTC, an engineer rebooted the database. The incident was resolved after the database reboot completed at 20:13 UTC. **Impact**: From 19:53 UTC to 20:13 UTC, users of Asana in all regions experienced application crashes, making Asana fully inaccessible. No customer data was lost. **Moving forward**: We are rewriting the database maintenance script to avoid hitting the code path that triggered the issue, and we have warned operators not to run the script until it can be modified to address this behavior. Based on the nature of the issue and the load patterns of our databases, we are confident it will not be triggered in our other databases or via other code paths. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Asana users experienced errors and app crashes due to a database problem.
Report: "Partial Outage"
Last update**Incident:** A change to how we load data from our databases led to increased server memory usage. During a period of heavy traffic the increased memory pressure exceeded a critical threshold. This caused some servers to become overloaded, which resulted in slow or unresponsive request handling and retries. We reverted to a prior deployment and observed system health recover. **Impact:** Between 13:47 and 15:20 UTC on 2024-03-07, attempts to create or edit data in Asana were delayed up to a few minutes, about 1% of web sessions crashed, and some API requests failed or were delayed. Background actions such as automations and email notifications were also delayed. **Moving forward:** We have identified and reverted the problematic change and are improving our memory usage monitoring to identify regressions more quickly before causing user impact. Additionally, we discovered an issue with a safety measure to prevent excessive memory usage that we will address to prevent similar memory pressure issues in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers and some changes to e.g. tasks may not be saving correctly. Our Development Team is currently investigating this issue and we aim to have this resolved as soon as possible. Our sincere apologies for the inconvenience; if you're impacted by this issue, please check this page for the latest updates.
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers. Our Development Team is currently investigating this issue and we're hoping to get it fixed as soon as possible. Sincere apologies for the inconvenience; if you're impacted by this issue, please keep an eye on this page for the latest updates.
Report: "Delayed Webhooks and Event Streams"
Last updateWe have continued to observe successful new event delivery. Further updates on any delayed or missed events from this incident will be shared on our forum thread https://forum.asana.com/t/incident-affecting-events-and-webhooks-on-march-7th/734268 going forward.
Due to a bug we were unable to process some webhook and event stream events between 2024-03-07 00:20 and 2024-03-07 04:10 UTC. We are processing all events after this timeframe and are working on changes to deliver events from the timeframe above. We will follow up with more detailed information on our developer forum (https://forum.asana.com/c/forum-en/api/24) once we better understand what events may have been delayed or missed.
Report: "Errors connecting to Asana"
Last update**Incident**: A bug caused stale routing configuration for 49 hours, primarily in the US data center. We became aware of this through user reports, and cycling the affected servers restored functionality. **Impact**: From 2024-03-02 11:40 UTC - 2024-03-04 12:00 UTC, up to 0.5% of users were unable to establish new sessions connecting to Asana. **Moving forward**: We are adding monitoring for this failure mode, and have identified further improvements to detect problems like this. We have identified improvements in our routing layer configuration. We will be conducting a retrospective to explore how we can prevent similar issues in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We investigated reports of errors connecting to Asana. Some customers experienced persistent errors while loading the application.
Report: "Investigating connect/load errors"
Last updateFrom approximately 2024-03-05 19:43 - 21:30 UTC (11:43 AM PST - 01:30 PM PST), some customers experienced connectivity issues from an external provider to AWS, resulting in errors loading the Asana application.
Analysis of errors and reports indicates this was limited to customers in California using a specific ISP; we are seeing recovery, and expect affected users should be able to connect now.
We are currently investigating reports that some users are unable to load the Asana application. This is affecting a subset of users. We will have an update by 2024-03-05 22:00 UTC. If you are reporting issues for this incident, please provide your location and ISP as part of the report.
Report: "OAuth App degradation"
Last updateOur engineers have fixed the issue and Asana is now running normally. Again sincere apologies for any inconvenience this caused!
Update: We've identified affected traffic as API requests using OAuth when making batch requests and are working on a solution. Future updates can be found at our forum post: https://forum.asana.com/t/ongoing-oauth-app-degradation-when-making-batch-requests/715557
We've traced down the issue to a code change and are in the process of integrating a fix.
We're currently looking into an issue affecting the API related to OAuth apps.
Report: "Delayed automations and background actions"
Last update**Incident**: On 2024-02-07, a deployment of new code introduced a performance regression for automations and background actions. In our European data center during peak traffic from 03:56 to 04:54 UTC, our background job system hit autoscaling safety limits and did not keep up with incoming work due to the performance regression. We reverted this deployment and increased our scaling limits to speed time to recovery. **Impact**: During the incident, customers in our European data center experienced delayed background actions, such as rules, email notifications, imports and exports. No customer data was lost. **Moving forward**: We will improve our monitoring and playbooks to more quickly identify regressions like this and improve our ability to promptly respond. We have identified the root cause of this particular performance regression and are resolving the issue. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
Report: "Partial Outage (OAuth Apps unavailable for 1 hour from 19:18-20:16 UTC)"
Last update**Incident:** On 2024-02-08 from 19:19 UTC to 20:16 UTC one of our API authentication mechanisms experienced an outage, resulting in a 44% failure rate for API requests during the incident across all regions. The outage was caused by a backwards-incompatible dependency upgrade that was not caught before being deployed to production. [Asana's status page](https://status.asana.com/) was updated at 19:30 UTC to reflect a partial outage and we finished rolling back the deployment at 20:15 UTC. **Impact:** The outage impacted all customers accessing the API with an OAuth bearer token, directly or through the use of an app integration. API access using personal access tokens and via service accounts was not impacted by this outage. No data loss occurred during the incident. **Moving forward:** We will enhance our monitoring with test cases for the specific failure we observed. We are also investing in changes to our deployment process to better catch issues like this prior to deployment to production. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Our engineers have fixed the issue and Asana is now running normally. Again sincere apologies for any inconvenience this caused! This issue only impacted the API component.
Report: "Partial Outage"
Last updateOur engineers have fixed the issue and Asana is now running normally. Again sincere apologies for any inconvenience this caused!
We are currently experiencing some difficulties; as a result, Asana might not be available for some of our customers. Our Development Team is currently investigating this issue and we're hoping to get it fixed as soon as possible. Sincere apologies for the inconvenience; if you're impacted by this issue, please keep an eye on this page for the latest updates.
Report: "Partial Outage"
Last updateThis incident has been resolved.
Report: "Delayed writes and API performance issues"
Last update**Incident** On 2023-12-20, an engineer made a configuration change that caused a large amount of unexpected logging. The following day during peak hours, at 2023-12-21 at 14:13 UTC, the excess logging caused resource saturation that led to slow requests and timeouts. The configuration change was reverted, and related infrastructure was scaled up to aid recovery. After initial recovery, some users experienced a second period of degraded performance due to a retry mechanism that confirms event logs are recorded. **Impact** On 2023-12-21, between 14:13 UTC and 14:18 UTC, about 20% of Asana users in the US region experienced forced reloads, and existing sessions experienced delayed writes and API performance issues. Between 15:35 UTC and 15:45 UTC, about 5% of Asana users experienced delayed writes and API performance issues. Application uptime and performance was fully restored by 2023-12-21 15:46 UTC. No data was lost. **Moving Forward** The incident uncovered a gap in how we monitor logging volume and its effects on server performance during high traffic times. Moving forward, we have action items to improve the resiliency of our infrastructure to unexpected increases in logging volume, as well as to ensure our monitoring will enable us to proactively detect and prevent similar issues in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We are currently investigating this issue.
Report: "Application and API downtime"
Last update**Incident**: At 2023-12-14 23:40 UTC, an engineer made a configuration change that caused downtime. We reverted, but determined that some application servers got into an unrecoverable bad state. Affected servers were identified and restarted; to avoid overload, traffic was allowed to return incrementally as the servers became ready. Background actions were delayed starting at 2023-12-15 00:22 to aid recovery. **Impact**: This resulted in full application and API downtime across all regions. For the US region, application downtime was resolved for most users between 2023-12-15 00:13 and 00:30. For EU, Japan, and Australia, resolution started around 00:40. API traffic was restored between 00:36 and 00:43. Application and API availability was completely restored for all users by 2023-12-15 00:49. Background actions were delayed from 2023-12-15 00:22 through 01:08, with complete resolution by 01:30. **Moving forward**: This incident uncovered a bug preventing automatic restarts of our backend servers. We have action items to fix this bug, improve validation and config handling, and enhance observability and operational tools. In the medium term, we are working on creating more independent failure domains to limit the scope of unexpected downtime. We will follow our [5 Whys process](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/) to identify other potential action items. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved
We're currently experiencing some difficulties; as a result, a majority of our users are unable to access Asana. Our Development Team is currently working to resolve this issue as soon as possible. Sincere apologies for the inconvenience caused, please keep an eye on this page for the latest updates.
Report: "Outage"
Last update**Incident:** Databases containing authentication information experienced increased load and reduced throughput due to a database migration. **Impact:** On 2023-11-28 from 18:00-18:25 PST, the Asana API was operating in a degraded state and/or unavailable. Asana Mobile and API users experienced increased response times and errors during this incident. **Moving forward:** To prevent similar incidents, Asana has increased database compute provisioning and has developed monitoring around the root cause. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Incident: Databases containing authentication information experienced increased load and reduced throughput due to a database migration. Impact: On 2023-11-28 from 18:00-18:25 PST, the Asana API was operating in a degraded state and/or unavailable. Asana Mobile and API users experienced increased response times and errors during this incident. Moving forward: To prevent similar incidents, Asana has increased database compute provisioning and has developed monitoring around the root cause. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Report: "Some Asana features have been affected by an Amazon Web Services outage"
Last update**Incident:** On June 13, 2023, AWS had an outage for the AWS Lambda service for about 3.5 hours. This outage led certain Asana features, most notably webhooks, to not be available to Asana users. **Impact:** Asana users experienced an outage in webhooks functionality for about 3 hours, with a degradation for approximately another 40 mins. During the outage, users of webhooks were not able to receive webhook events. The outage led to a delay in delivery of these events and there was no loss of data. **Moving forward:** We are enhancing our operational tools in order to be more resilient to this type of outage and to minimize any potential impact of AWS unavailability.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We currently are experiencing an issue with webhooks and remote jobs being in a degraded state (Ex: attachments functionality is affected) jobs and are working on this issue
Some Asana features have been affected by an Amazon Web Services outage. We're are currently working on this issue and we apologise for the inconvenience. For the latest updates, please check https://status.aws.amazon.com/ The Asana app is functional, except for the following: - Webhooks are down - Remote jobs are in a degraded state. Ex: attachments functionality is affected - OneDrive attachments - Incoming emails not working
We are currently experiencing some difficulties; Our Development Team is currently investigating this issue and we're hoping to get it fixed as soon as possible. Sincere apologies for the inconvenience; if you're impacted by this issue, please keep an eye on this page for the latest updates. The Asana app is functional, except for the following: - Webhooks are down - Remote jobs are in a degraded state. Ex: attachments functionality is affected - OneDrive attachments - Incoming emails not working
Report: "Major Outage"
Last updateIncident: Several internal services became undiscoverable as part of a routine rollback operation. Many customer-facing services that depend on these were unable to serve traffic as a result. Impact: For 19 minutes from 2023-05-10 01:52-02:11 UTC, almost all API calls timed out. New pages in the web app failed to load, but most existing pages continued to function. Moving forward: We have fixed the bug in our rollback process which caused the outage.
We are currently investigating this issue.
Report: "asana.com is down."
Last updateOur engineers fixed the issue affecting webpage links, and they’re now working. Sorry for the trouble!
We’re experiencing a system issue that’s impacting some webpage links. We hope to have it fixed very soon. Sorry for the trouble!
Report: "Major outage"
Last updateIncident: From 2023-03-15 14:38 UTC to 2023-03-15 15:18 UTC there was a webhook outage for about 40 minutes in the US region. A bad deployment caused reduced capacity and delays in scheduling some asynchronous work, including webhook delivery. Impact: Webhook users experienced a considerable amount of delay, max ~one hour difference between events generated and events delivered. There was no data loss due to this delay and backlog was recovered in ~20 minutes Moving forward: We have added better tooling that would help us investigate this issue faster and have increased the priority of the job responsible for streaming webhook events.
Report: "Major Outage"
Last update**Incident**: From 2023-03-23 13:30 UTC until 2023-03-23 15:10 UTC all databases responsible for storing data related to Asana users were periodically failing over to their standby instance. We run these databases using AWS RDS Multi-AZ, and AWS triggers these failovers based on health signals they are monitoring. We are still working with AWS to investigate and determine the full root cause but our current best theory is that this was caused by elevated DNS lookup latency resulting in database connection saturation. The increase in DNS lookup latency caused connections to the database to pile-up eventually overloading the databases triggering the automated RDS failover system. We recovered after deploying a configuration change to disable DNS hostname lookups. **Impact**: During the incident, users of the web app were unable to load or use Asana for 1 to 5 minutes while the database was unavailable during failover. Users may have experienced these periodic outages multiple times during the 100 minute event. API users experienced a similar downtime impact. **Moving forward**: We’re working with the AWS RDS team to fully understand the root cause of the issue as well as making some configuration changes such as disabling DNS hostname lookups on our databases so that we will no longer be as susceptible to increased DNS latency. We believe our database use case does not need to resolve client hostnames. In the long-term, we’ll be revisiting our database connection strategy to make more use of connection pooling, which will reduce our dependence on establishing and managing quick and short-lived database connections.
This incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We're currently experiencing some difficulties; as a result, a majority of our users are unable to access Asana. Our Development Team is currently working to resolve this issue as soon as possible. Sincere apologies for the inconvenience caused, please keep an eye on this page for the latest updates.
Report: "Partial Outage"
Last update**Incident**: Around 2023-02-28 20:31 UTC servers responsible for reactivity started to fail due to unexpected input. After reverting the change triggering this, most recovered between 21:30 and 22:00. In some cases stale data was displayed until caches were cleared, which finished at 23:05. Approximately 1% of users continued to see reactivity failures and stale data until around 2023-03-01 00:04 UTC. **Impact**: While reactivity servers were down, API writes failed and changes were not reflected to other tabs. After recovery of reactivity servers, in some cases stale data was displayed within our applications until the caches were fully cleared. No customer data was lost. **Moving forward**: We are making changes to the application servers which crashed to make them more resilient against unexpected input, and making tooling changes to reduce time to resolution for this class of incident. Architectural changes which are in progress will provide smaller failure domains, which would reduce impact and provide faster resolution for this class of failure. We use the [5 Whys](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/) approach to identify technical, operational, and organizational changes to reduce the likelihood and severity of incidents. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This has been resolved but we'll continue to monitor any further issues.
The majority of users should be able to see their changes again, but some data will not yet show up. We expect to see a full recovery in ~ 1 hour, at which point all users will see all of their changes again
We are continuing to investigate this issue.
We are currently experiencing some difficulties. Some percentage of API requests are failing, and changes you make will be slow to show up. Our Development Team is currently investigating this issue and we're hoping to get it fixed as soon as possible. Sincere apologies for the inconvenience; if you're impacted by this issue, please keep an eye on this page for the latest updates.
Report: "App down"
Last update**Incident**: Around 15:50 UTC, a system used for caching entered an overload state due to network connection tracking within our hosting environment. Connections from servers to this cache timed out, and request queuing created a cascading failure. Reducing load via application changes allowed recovery, and we then made configuration changes to provide additional capacity. **Impact**: Asana was unavailable in all regions for as long as 16 minutes, with a partial outage for 6 additional minutes. **Moving forward**: We've updated configuration to avoid overload, and are adding monitoring to detect this type of saturation before failure. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We are currently investigating this issue.
Report: "Automations and Background Actions Delayed"
Last updateOn January 23, 2023, we experienced a 50 minute Automations and Background Actions outage. Impact was delayed execution of automations and background actions for two data centers. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Report: "Automations and Background Actions Delayed"
Last updateOn January 16, 2023, we experienced a 15 minute Automations and Background Actions outage. Impact was delayed execution of automations and background actions in one data center. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Report: "Web page load failures"
Last updateIncident: A bug in our cleanup scripts caused some active container repositories to become unavailable. At approximately 13:33 UTC, a memory problem caused servers which handle pageload \(initial load of our web application\) to crash, and because of the bug in the cleanup script they were unable to restart without operator intervention. Customers were unable to start new sessions for about 28 minutes, but existing sessions were unaffected. Impact: Asana was largely unavailable to new sessions for 28 minutes on Asana Desktop. Existing sessions, mobile and API were unaffected. Moving Forward: We will be conducting a 5 Whys analysis to identify process changes that can prevent similar incidents in the future. We will also work on detecting this type of issue before they impact customer traffic. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We are currently investigating this issue.
Report: "Downtime"
Last update**Incident**: A scheduled, manual deployment led to abnormally high load on our reactive data loading layer, overloading it. Application level caches were slow to update, and a subset of data returned to customers was stale. **Impact**: Although application writes continued to be successful, a subset of customers experienced delays in viewing their changes to the Asana webapp from 2022-12-21 00:27 - 01:52 UTC. By 02:20 UTC, tabs that were loaded or refreshed no longer contained stale data. By 03:10 UTC, all users were no longer seeing stale data. About 2% of customer API requests also experienced timeouts until 01:52 UTC. Affected API customers would have seen persistently failing API read requests. **Moving forward**: We have identified operational changes to reduce the likelihood of incidents and decrease the time to resolution. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "Partial outage"
Last updateIncident: To ensure our systems can reliably recover when systems fail, Asana triggers an individual node failure at a time of low traffic when engineers are available to address any problems. A software bug in an internal application which replaces failed nodes prevented recovery when a node was terminated in this manner. Impact: Until engineers intervened to manually replace the failed node, about 12.5% of users experienced application crashes and about 1% of API requests failed. Moving forward: Planned work includes improved monitoring and resilience for node failures. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
We are currently investigating this issue.
Report: "Major Production Outage"
Last updateIncident: Between 20:00 UTC and 23:30 UTC, Asana experienced a major production outage resulting in limited web application functionality and elevated error rates for API traffic. Two principal application services that serve Asana's web traffic were impacted: LunaDb, which performs data loading and handles communication with web clients, and [Worldstore](https://blog.asana.com/2020/09/worldstore-distributed-caching-reactivity-part-1/), which functions as a database caching layer and allows users to see the changes they’ve made. Typically these services deploy independently to reduce the load on either system. However, during this incident we saw updates to both services overlap which placed stress on a shared service, causing it to fail which then cascaded to other services. Engineers responded to automated alerts within minutes of the start of the incident, but stabilizing the Worldstore cluster took several hours and several different attempts. Impact: For the duration of the incident, web-app users saw a loss of reactivity, i.e. they perceived their own changes not being saved or did not receive collaborative edits made by other users. Users of Asana’s API and mobile may have been unable to make changes to Asana at all. At around 23:30 UTC, full application functionality across webapp and API was restored. Moving forward: We are changing the configurations of our LunaDb and Worldstore services to prevent overload under similar circumstances, and adjusting deployment times of these services to avoid updating both simultaneously. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
This incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
Changes made in the past few hours on Asana might not be visible, but any new changes made should be visible. We are working on redeploying our servers, which should make all previous writes to Asana visible.
We are continuing to work on a fix for this issue. Changes made in the app will be persisted but may not be reflected immediately.
We've identified some potential issues and are restarting affected servers.
Our team has identified the problem and is currently working on a fix. We appreciate your patience!
Report: "Partial Outage - EUDC"
Last updateIncident: A server version change resulted in some servers in the European and Australian data centers becoming incapable of serving new sessions, leading to users being unable to load Asana. Impact: From 3:19 UTC to 8:36 UTC, a significant portion of attempts to start new sessions failed for customers with data residency in Europe or Australia. ~20% of requests failed in Europe and ~80% failed in Australia. Existing sessions were unaffected, and there was no loss of customer data. Moving forward: We have added monitoring to alert us of this failure case, and have also added safeguards to the deployment process to prevent servers from reaching this state again. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Our engineers have fixed the issue and Asana is now running normally. Again sincere apologies for any inconvenience this caused!
A fix has been implemented and we are monitoring the results.
We are currently experiencing some difficulties; as a result, some customers who are hosted in our EU Data Center may be experiencing issues accessing Asana. We are currently investigating this issue and hope to have it fixed as soon as possible. Our sincere apologies for the inconvenience; if you’re impacted by this, please keep an eye on this page for the latest updates.