Is Jira Product Discovery Down Right Now? Check if there is a current outage ongoing.

We were experiencing cases of degraded performance for outgoing emails from Confluence, Jira Work Management, Jira Service Management, Jira, Opsgenie, Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Compass, and Loom Cloud customers. The system is recovering and mail is being processed normally as of 16:45 UTC. We will continue to monitor system performance and will provide more details within the next hour.

### Summary On May 20, 2025, between 13:10 UTC and 22:12 UTC, a small subset of Atlassian customers using Jira Software and Jira Service Management products were unable to access certain functionality, such as creating, viewing, or editing boards, issues, filters, and Jira Service Management requests. The root cause was data duplication in our software related to upgrades of \(non-faulty\) third party apps. The incident was detected within 4 minutes by Automated Monitoring, and mitigated through the subsequent 8 hours and 58 mins. Factors that prolonged the impact of the incident included: * An attempt to mitigate the issue by disabling the \(non-faulty\) apps was ineffective because the disablement mechanism was not integrated with the affected data source * Attempts to manually rectify the data in the database were hampered by the binary storage format used for the data \(necessary for performance reasons\) ### **IMPACT** During the impact period stated above, Jira Software and Jira Service Desk sites that had a specific version of the third party apps installed were impacted. For the duration of the incident, functionality such as creating, viewing, or editing of boards, issues, filters, and Jira Service Management requests was unavailable on those sites. ### **ROOT CAUSE** During the process of version upgrade for the third party apps with dynamic modules, duplicated data was left behind due to a bug in Atlassian systems triggered during a prior upgrade. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified during development because of the specific sequence of events required to trigger the latent duplicate data to impact the product’s functioning. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Detecting and repairing duplicated data at the source * Updating the faulty “write” subsystem to prevent further duplication of data Furthermore, we are taking steps to enable swifter mitigation should such an incident occur in the future. We are: * Enhancing app disablement tooling \(and its documentation\) to make it simple and effective to deploy against this class of issue * Enhancing validation of the app upgrade process * Developing an affordance for viewing, repairing, and modifying specific app data We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Deployment of the fix for this issue has now completed rolling out and this should be resolved for all users.

We are continuing to investigate an issue that is disrupting access to core Jira functionality for a limited number of Jira customers. We will provide an update when we have identified a fix.

We are continuing to investigate issues with the Jira degradation and will provide updates as we learn more.

We are investigating cases of degraded performance for all Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We will provide more details within the next hour.

We are investigating cases of degraded performance for all Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We will provide more details within the next hour.

Between 10:55 AM UTC to 13:15 PM UTC on April 28, 2025, we experienced partial unavailability in managing Forge apps; Such as lifecycle developer console usage, and the creation and deployment of Forge apps for Atlassian Partners, Jira Work Management, Jira, Atlassian Developer, Jira Product Discovery, Compass, and Atlassian Migrations. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The conditions that cause the bug have been addressed and we're actively working on a permanent fix. The issue has been resolved and the service is operating normally.

The issues causing the unavailability of related Forge apps capabilities on multiple Atlassian platforms have been resolved, and services are now operating normally for all impacted customers. We will continue to closely monitor for stability.

We are currently experiencing partial unavailability in managing Forge apps on different Atlassian platforms, such as apps lifecycle (installation, removals), developer console usage (including tunneling), and the creation and deployment of Forge apps. We have identified the root cause and are in the process of mitigating the impact. We'll keep you posted with further updates. Next updates in 1 hour.

We are currently experiencing partial unavailability in managing Forge apps on different Atlassian platforms. Such as apps lifecycle (installation, removals), developer console usage (including tunnelling), and the creation and deployment of Forge apps. Our team is working diligently to resolve this issue and restore services quickly. We are in the process of identifying the root cause and implementing the necessary fixes. Next updates in 1 hour.

We are currently experiencing partial unavailability in managing Forge apps on different Atlassian platforms. Such as apps lifecycle (installation, removals), developer console usage (including tunnelling), and the creation and deployment of Forge apps. Our team is working diligently to resolve this issue and restore services quickly. We are in the process of identifying the root cause and implementing the necessary fixes.Next updates in 1 hour.

Between 13:00 UTC to 23:00 UTC on April 23, 2025, we experienced automation rule execution delays for Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery. The issue has been resolved and the service is operating normally. For some high volume customers who regularly have rule throttling, there may be an extended period to clear full backlogs. For those high volume customers we expect any extended period rule throttling to be resolved by 07:00 UTC on April 24, 2025.

New executions are continuing to run without delay. Remaining delayed executions will complete in approximately one hour. We will continue to monitor the progress and provide an update within the next hour.

New executions are running without delay. Remaining delayed executions will complete in the next 2 hours. We will continue to monitor the progress and provide an update within the next hour.

A majority of new automation executions are running without delay. Remaining delayed automation executions should be completed within 3 hours. We will continue to monitor the progress and provide an update within the next hour.

We have applied significant mitigations and the majority of delayed automation executions should be completed within 2 hours. Some customers may experience some delay in some executions during the 2 hours following. Rule execution delays should be mitigated in 4 hours. We will continue to monitor the progress and provide an update within the next hour.

We are still investigating automation rule delays that is impacting some Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. The backlog of previous mitigations has decreased and continues to decrease, however some customers may be still be experiencing delays. The backlog of automations is being processed as we continue to apply mitigations and will provide more details within the next hour.

We are still investigating automation rule delays that is impacting some Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. Previous mitigations have decreased the average delays across executions, however some customers may be still be experiencing delays. We are applying more mitigations and will provide more details within the next hour.

We are investigating automation rule execution delays that are impacting some Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We have applied a temporary mitigation and are continuing to investigate for root cause. We will provide more details within the next hour.

We are investigating cases of degraded performance for automation rules for Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We will provide more details within the next hour.

We are investigating cases of degraded performance for automation rules for Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We will provide more details within the next hour.

Between 3:30 UTC to 23:00 UTC April 16, 2025, we experienced an issue with loading certain Jira Dashboard widgets for Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery. The issue has been resolved and the service is operating normally.

We continue to work on resolving the issue with loading certain Jira Dashboard widgets for for Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery. We have identified the root cause and are currently deploying fixes. We expect to complete the deployments by 23:00 UTC.

We continue to work on resolving the issue with loading certain Jira Dashboard widgets for for Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery. We have identified the root cause and expect recovery shortly. We will provide another update at 22:00 UTC.

We continue to work on resolving the issue with loading certain Jira Dashboard widgets for Jira Work Management, Jira Service Management, and Jira Product Discovery. We have identified the root cause and expect recovery shortly. We will provide another update at 19:00 UTC.

We are aware of users experiencing issues attempting to load Jira Dashboard widgets. Our team is investigating this issue with priority.

We are aware of users experiencing issues attempting to load Jira Dashboard widgets. Our team is investigating this issue with priority.

Until the end of May 2025, Atlassian plans to perform maintenance on the infrastructure supporting the Jira and Confluence platforms.This maintenance will result in a temporary service outage for Free edition users located in Singapore and Northern California, with a total duration of no more than 120 minutes per customer. The maintenance will be performed during regular business hours in the specified regions.To learn more, please refer to this community post https://community.atlassian.com/forums/Announcement-articles/Upcoming-Maintenance-Notification-for-Jira-and-Confluence/ba-p/2973795#M6870

Scheduled maintenance is currently in progress. We will provide updates as necessary.

On March 14th, 2025, between 09:00 and 19:14 UTC, some customers experienced issues creating new sites. The errors occurring for some users during new site creation should now be resolved.

We are investigating cases of errors for some users creating new sites. We will provide more details within the next hour.

Our team has been monitoring this issue and can confirm that the errors that were previously occurring have now been resolved.

Our team has implemented a fix for the connection errors that were being experienced across Jira Work Management, Jira Service Management, Jira and Jira Product Discovery Cloud products. We are continuing to monitor this change to ensure that the issue is mitigated.

Our team is continuing to investigate the issues relating to partial outage being experienced by some customers across Jira Work Management, Jira Service Management, Jira and Jira Product Discovery Cloud products. We will provide an update when more information relating to mitigating the issue is available.

We are continuing to investigate the root cause of the partial outage with that is impacting some Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We are applying mitigation to affected customers. We will provide more details within the next hour.

We are investigating reports of intermittent errors for some Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. We will provide more details once we identify the root cause.

Between 05/Feb/25 2:30 PM UTC and 05/Feb/25 3:28 PM UTC some customers for Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud experienced intermittent errors. We have identified the root cause of the intermittent errors and have mitigated the problem. The issue has been resolved and the service is operating normally.

We continue to monitor the mitigation measures implemented to address the intermittent errors reported.

We have identified the root cause of the intermittent errors and have mitigated the problem. We are now monitoring this closely.

We are investigating reports of intermittent errors for some Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery Cloud customers.

Between 3:16pm PST and 7:38pm PST, we experienced an outage for Rovo and all AI features across Atlassian Products. The issue has been resolved and the service is operating normally

We are witnessing significant recovery in the services. We will continue to monitor the situation closely.

We are actively communicating with the affected AI Service provider and closely monitoring the ongoing outage situation. While we have observed some signs of recovery, the majority of API calls are still resulting in errors. We have successfully restored the Smart Answers in Search feature by rerouting requests to an alternative model. All other Atlassian Intelligence capabilities and Rovo remain unavailable at this time due to the ongoing AI Service provider outage.

Between 07:50 UTC to 11:31 UTC, some customers experienced previews of attachments unavailable for Confluence, Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The conditions that cause the bug have been addressed and we're actively working on a permanent fix. The issue has been resolved and the service is operating normally.

Between 06/Nov/24 12:00 AM UTC to 06/Nov/24 9:22 AM UTC, we experienced Automation Audit Logs not loading for customers intermittently for Confluence, Jira Service Management, Jira, and Jira Product Discovery. The issue has been resolved and the service is operating normally.

We are investigating cases of degraded performance with the Automation logs for Confluence, Jira Service Management, Jira, and Jira Product Discovery Cloud customers. However, automation rule runs are not impacted. We will provide more details within the next hour.

Between 08:00 UTC to 11:54 UTC, we experienced problems with the invitation of new users for Cloud customers on admin.atlassian.com. The issue has been resolved and the service is operating normally.

We continue to work on resolving the invitation workflow in admin.atlassian.com We have identified the root cause and performed changes in the environment to mitigate the issue.

We are investigating reports of intermittent errors for some Atlassian customers when they are trying to invite users using their admin panels (admin.atlassian.com) We will provide more details once we identify the root cause.

Between 2024-10-23, 15:09 UTC and 2024-10-23, 17:06 UTC, we identified an issue that affected tenant provisioning and user management actions at Atlassian Cloud, causing a delay in performing actions related to these two topics. We have identified the root cause and the issue has been mitigated.

Between 22-Oct 09:00 AM UTC to 22-Oct 14:18 PM UTC, we experienced intermittent errors for Jira Cloud products. The issue has been resolved and the service is operating normally.

We continue to work on resolving the intermittent errors for Jira Cloud products. We have identified the root cause and expect full recovery shortly.

We are investigating reports of intermittent 'Too many requests or errors' messages showing up for some Jira Cloud customers. Once we identify the root cause, we will provide more details.

We experienced availability issues limited to a specific instance in AMER, resulting in Jira being temporarily unavailable. The issue began at 18:58 CET and was restored at 19:59 CET. All services are now fully operational

Between 2300 14 October to 0200 16 October, we experienced an outage affecting Team Managed Project creation for Jira Work Management, Jira Service Management, Jira, and Jira Product Discovery. The issue has been resolved and the service is operating normally.

Some customers are getting an error when trying to create team-managed Jira projects (Jira, Jira Work Management, Jira Service Management, Jira Product Discovery). We are currently investigating this and we will continue to share further updates.

We have identified that the issue is related to a recent update on the Brave browser (https://brave.com/latest/), which is causing some cookie-related problems for users. As a temporary workaround, we recommend using one of our supported browsers: https://support.atlassian.com/atlassian-account/docs/supported-browsers-for-atlassian-cloud-products/. We understand that many users prefer Brave, and our engineering team will continue to investigate this issue and work towards a resolution on a best-effort basis.

We are investigating an issue with accessing Jira content that is impacting some Jira Cloud customers. We will provide more details within the next hour.

The issue has been mitigated and JPD is operational now.

We are continuing to investigate this issue.

We have identified an issue wherein JPD is not loading or is loading very slow. We have started to investigate the issue and will provide further updates shortly.

Between 12:00am 9th July to 08:00am 10th July, we experienced email deliverability issues for some recipient domains for Confluence, Jira Work Management, Jira Service Management, Jira, Trello, Atlassian Bitbucket, and Jira Product Discovery. The issue has been resolved and future emails will flow normally.

We continue to work on resolving the Email Notifications for Confluence, Jira Work Management, Jira Service Management, Jira, Trello, Atlassian Bitbucket, and Jira Product Discovery. We have identified the root cause.

We are investigating reports of intermittent errors whilst sending Email Notifications for Confluence, Jira Work Management, Jira Service Management, Jira, Trello, and Jira Product Discovery Cloud customers. We will provide more details once we identify the root cause.

Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Jira Product Discovery. The issue has been resolved and the service is operating normally.

We have mitigated the problem and continue looking into the root cause. The outage was between 8:08pm 03/07 UTC - 08:31pm 03/07 UTC We are now monitoring closely.

We are investigating an issue with <FUNCTIONALITY IMPACTED> that is impacting <SOME/ALL> Atlassian, Atlassian Partners, Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira, Opsgenie, Atlassian Developer, Atlassian (deprecated), Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Atlassian Analytics, and Rovo Cloud customers. We will provide more details within the next hour.

Between 22:18 UTC to 22:56 UTC, we experienced errors for multiple Cloud products. The issue has been resolved and the service is operating normally.

We are investigating an issue with error responses for some Cloud customers across multiple products. We have identified the root cause and expect recovery shortly.

Between 6:30 AM UTC to 9:50 AM UTC, we experienced failures in accessing some features from the Admin Portal. The issue has been resolved and the service is operating normally.

We are investigating an issue causing failures in accessing some features from the Admin Portal, which is impacting some of our Cloud customers. We have identified the root cause and anticipate recovery shortly.

All the impacted customers have been contacted and all the impacted rules have been fixed. We are now marking this issue as resolved.

We have successfully restored the majority of automation rules that were affected, but we are currently in the process of manually fixing the remaining ones. Our support team will be contacting the customers impacted by this issue to proceed with resolving their cases.

This issue has been mitigated, and we have restored the majority of automation rules that were affected. Those rules are working as normal, including the rules that had not been modified in the last 48 hours and the rules created after the incident. The remaining rules that haven't yet been restored are cases where customers have manually updated the rules after the incident happened. We are proceeding with caution to avoid overwriting these changes.

We are continuously working on restoring the affected automation rules configuration data. At the moment the majority of the impact has already been mitigated and the team is currently focusing on a few specific outstanding scenarios. We will continue to update you as we progress.

We are currently working on restoring the lost automation rules configuration data. The email notifications related to the automation will be addressed upon restoring the lost data. Customers can update the configurations with the Automation rules themselves, and our fix will not override those changes. We will share further updates as we progress.

We have identified the root cause and we are currently working on deploying fix to restore any lost configuration/rules. Next update will be provided in an hour.

We continue to investigate the issue with the Automation rules losing configuration data that is impacting certain cloud products. We are actively working to resolve this issue as quickly as possible.

We are investigating an issue with the Automation rules losing configuration data that is impacting some of the Cloud products. We are actively working to resolve this issue as quickly as possible.

On 28th Feb between 12:15 UTC and 15:20 UTC Connect apps were unable to perform user-impersonation operations and failed to execute. This had a widespread impact on all product extensions based on the Connect platform resulting in failure when performing actions on behalf of a user. Jira Cloud Automation rules that triggered during the outage may have failed to run. Those rules will not be able to be replayed by customers or Atlassian. Our support engineers have details regarding which customers and automation rules that were impacted. However, support engineers will not be able to replay the failed automation rules. We have identified the root cause of the fault and have resolved it by scaling up the infrastructure. The incident is now resolved. If you have questions or require support - please reach out at https://support.atlassian.com/contact/. We apologise to all of our developers, customers and users who were impacted during this time. We will release a public incident review (PIR) here in the upcoming weeks. We will detail all that we can about what caused the issue, and what we are doing to prevent it from happening again.

Starting from 13:15 UTC, Automation and Connect Apps affecting certain cloud products We have scaled up the underlying services and we're seeing an improvement in response times and success rates. We continue to investigate the root cause and will provide the next update by 18:00 UTC.

We continue to investigate the issue with the Automation and Connect Apps affecting certain cloud products. We are actively working to resolve this issue as quickly as possible.

We are investigating an issue with Automation and Connect Apps that is impacting some Cloud products. We will provide more details within the next hour.

Between 28th Feb 2024 23:15 UTC to 29th Feb 2024 00:05 UTC, we experienced issue with new product purchasing for all products. All new sign up products have been successfully provision and confirmed issue has been resolved and the service is operating normally.

We are investigating an issue with new product purchasing that is impacting for all products. Customers adding new cloud products may have experienced a long waiting page or an error page after attempting to add a product. We have mitigated the root cause and are working to resolve impact for customers who attempted to add a product during the impact period. We will provide more details within the next hour.

### **Summary** On February 14, 2024, between 20:05 UTC and 23:03 UTC, Atlassian customers on the following cloud products encountered a service disruption: Access, Atlas, Atlassian Analytics, Bitbucket, Compass, Confluence, Ecosystem apps, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery, Opsgenie, StatusPage, and Trello. As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names used for internal service-to-service connections. Active domain names were incorrectly deleted during this event. This impacted all cloud customers across all regions. The issue was identified and resolved through the rollback of the faulty deployment to restore the domain names and Atlassian systems to a stable state. The time to resolution was two hours and 58 minutes. ### **IMPACT** External customers started reporting issues with Atlassian cloud products at 20:52 UTC. The impact of the failed change led to performance degradation or in some cases, complete service disruption. Symptoms experienced by end-users were unsuccessful page loads and/or failed interactions with our cloud products. ### **ROOT CAUSE** As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names that were being used for internal service-to-service connections. Active domain names were incorrectly deleted during this operation. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The detection was delayed because existing testing & monitoring focused on service health rather than the entire system’s availability. To prevent a recurrence of this type of incident, we are implementing the following improvement measures: * Canary checks to monitor the entire system availability. * Faster rollback procedures for this type of service impact. * Stricter change control procedures for infrastructure modifications. * Migration of all DNS records to centralised management and stricter access controls on modification to DNS records. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. ‌ Thanks, Atlassian Customer Support

We experienced increased errors on Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Align, Jira Product Discovery, Atlas, Compass, and Atlassian Analytics. The issue has been resolved and the services are operating normally.

We have identified the root cause of the Service Disruptions affecting all Atlassian products and have mitigated the problem. We are now monitoring this closely.

We have identified the root cause of the increased errors and have mitigated the problem. We continue to work on resolving the issue and monitoring this closely.

We are investigating reports of intermittent errors for all Cloud Customers across all Atlassian products. We will provide more details once we identify the root cause.

Between 15:40 UTC to 15:57 UTC customers experienced intermittent failures when searching for users in Atlassian cloud services: Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, and Compass. The issue has been resolved and the service is operating normally.

We have mitigated the problem. We are now monitoring closely.

We are investigating an issue with our user search service that is impacting Atlassian cloud service: Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, and Compass. We will provide more details within the next hour.

Between 02:00 AM UTC to 04:13 AM UTC, we experienced email address handling failures, for email fields and smart values, which resulted in rule executions failing for some customers using Automation for Jira - JSW, JSM, JWM, JPD. The issue has been resolved and the service is operating normally.

Between 14:45 UTC to 18:25 UTC, we experienced degraded performance for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery for a small subset of instances. The situation has stabilized, and the service is operating normally.

We are investigating reports of intermittent degraded performance for Jira Service Management, Jira Software, Jira Work Management, and Jira Product Discovery Cloud, affecting a small cohort of customers in a US shard. We will provide more details shortly.

Between 06:52 UTC to 10:29 UTC, we experienced an outage for Jira Work Management, Jira Software, and Jira Product Discovery. The issue has been resolved and the service is operating normally.

Our engineering team has implemented fixes for Jira and should be recovered again. We will continue to monitor all systems.

We have identified the root cause of the issue and working on the fix.

We are investigating an issue with outage that is impacting Jira Work management, Jira Software, Jira Service Management and Jira Product Discovery. We will provide more details within the next hour.

Between 23:45 UTC to 00:30 UTC, we experienced an outage in some Atlassian Intelligence features for Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, Atlas, and Compass. The issue has been resolved and the service is operating normally.

We have identified the root cause of the increased errors and have mitigated the problem. We are now monitoring closely.

We are investigating an issue with Atlassian Intelligence that is impacting some Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details within the next hour.

### **SUMMARY** On Dec 18, 2023, between 12:29 p.m. and 3:35 p.m. UTC, Dec 18, 2023, Atlassian's cloud customers using Atlas, Bitbucket Cloud, Compass, Confluence Cloud, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery products were unable to search for users or use the "@mention" functionality. Customers' user search results failed or were delayed as Atlassian's service returning user search results was degraded in several regions. The incident originated from a computationally intensive operation that was triggered multiple times in rapid succession, resulting in degraded performance of Atlassian's user search service across several regions. Notably, customers in the EU west region were most affected. The incident was detected within 2 minutes by automated monitoring, and our team promptly took action by recovering unhealthy systems and scaling up the service's infrastructure temporarily. The resolution process concluded in 3 hours and 06 minutes. ### **IMPACT** The overall impact was between Dec 18, 2023, between 12:29 p.m. UTC and Dec 18, 2023, 3:35 p.m. UTC. The Incident caused service disruption to cloud customers worldwide. Customers experienced delayed or failed user searches when using the following Atlassian cloud products: * Atlas * Bitbucket Cloud * Compass * Confluence Cloud * Jira Service Management * Jira Software * Jira Work Management * Jira Product Discovery ### **ROOT CAUSE** The incident stemmed from Atlassian's user search service receiving commands to process multiple computationally intensive operations in rapid succession. These operations were directed at the same customer data set, and therefore overloaded resources within a clustered database system, leading to memory exhaustion and subsequent unresponsiveness to user search requests.  ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** To prevent a recurrence of such incidents, we are implementing the following measures: * Implement a mechanism to queue computationally intensive operations in order to avoid overloading the resources within the systems and process them without impact on customer experience. * Fine-tune our clustered database settings to mitigate the impact of resource exhaustion on the overall system.  We apologize to customers whose services were affected during this incident; we are taking immediate steps to improve the service’s resiliency. Thanks, Atlassian Customer Support

It has been resolved. Atlassian's cross product user search is working.

Atlassian's cross product user search service is currently healthy. Searches for users within Atlassian products are working as expected. We are in the process of investigating the root cause of this incident.

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

We are investigating reports of intermittent errors for <SOME/ALL> Atlassian, Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details once we identify the root cause.

We have not observed any further complaints of this issue and believe the matter is resolved.

We have identified some delays in the delivery of emails which should be resolved now. Customers who retried are expected to receive the exports. We are currently monitoring the situation.

Some end users have confirmed that they are now able to receive the email to download the export file. The issue might still persist for a small number of customers and we will keep monitoring and investigating further. However, there are indications that the frequency for the issue has been reduced. We will provide further updates early next week.

Some end users confirmed that they are now able to receive the email to download the export file. The issue still persists for some customers and we are working with them to identify and investigate this issue further.

We are investigating cases where some customers aren't receiving email notification to download the file when exporting users from User Management. We will provide more details within the next hour.

The systems are stable after the fix and monitoring for a specified duration

The issue was identified and a fix implemented. We are monitoring currently.

We are currently investigating an incident that result in outbound connections from Atlassian cloud in us-east-1 intermittently timing out. This affects Jira, Trello, Confluence, Ecosystem products. The features affected for these products are those that require opening a connection from Atlassian Cloud to public endpoints on the Internet

Including Atlassian Developer

We are currently investigating an incident that result in connection time outs on service egress proxy. This affects Jira, JSM, Confluence, BitBucket, Trello, Ecosystem products. The features affected for these products are those that require a connection to service egress.

Forge Invocations had an 8 minute outage between 2023-11-29 03:05:13 UTC to 2023-11-29 03:13:27 UTC resulting in Smart Links failing. This service has recovered post this time period.

Between 23:25 UTC and 23:46 UTC on 2nd November, we encountered a problem where automation rules in Jira that send out emails failed for certain customers. This was caused by spam-protection limits being incorrectly applied to the legitimate sending of emails. The issue has been resolved and the service is operating normally.

### **SUMMARY** From 9pm UTC on October 23, 2023, until 8pm UTC on October 24, 2023, Atlassian customers using the Jira family products that have configured notifications from custom email domains experienced a degradation as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of a custom domain. The incident was detected by Atlassian customers as they started to receive notifications from the [atlassian.net](http://atlassian.net) domain instead of previously configured custom domains. It was mitigated by excluding redundant validations from the custom domain verification job which put Atlassian systems into a known good state. The total time to resolution was about 23 hours. ### **IMPACT** The overall impact was between October 23, 2023 9pm UTC and October 24 8pm UTC on Jira family products, including Jira Service Management, Jira Software and Jira Work Management. The incident caused service disruption to customers who have notifications from custom email domains enabled for their Jira products as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of the custom domain configured by the customer. ### **ROOT CAUSE** The issue was caused by a change to the service responsible for custom domain email notifications lifecycle that inadvertently enabled SPF record validation as part of a daily validation job. As a result, previously working custom domain configuration transitioned to an unverified state for some Atlassian customers, resulting in notifications from Jira products being sent from [atlassian.net](http://atlassian.net) domain instead of a previously configured custom domain. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Improve monitoring and alerting for custom domains lifecycle to detect such regressions before they reach production. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Between 10-23-2023 21:00 UTC and 10-26-2023 03:00 UTC, some customers received emails stating that the domain verification was failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. The issue has been resolved and the service is operating normally. Additional details about the issue will be shared in our Post Incident Review

The fix for the issue has been rolled out and is expected to be complete by 3 am UTC, 26 October. In the meantime, we have refreshed all affected domains and the service is expected to be working normally. We will continue to monitor this issue for 24 hours to ensure it has been resolved and will share an update once confirmed.

We have identified the root cause of the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. A fix is currently being implemented, and we expect this issue to be resolved shortly.

We continue to work on resolving the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. We have identified the root cause and expect recovery shortly.

We are investigating an issue with emails that have been sent out to admins stating domain verification is failing. Affected users may see emails sent from Atlassian's default domain instead of the custom domain they have configured We will provide more details within the next hour.

### **SUMMARY** Between September 18 2023 5:47 PM UTC and September 19 2023 04:15 AM UTC, customers using Confluence Cloud, Jira Software, Jira Service Management, Jira Work Management, Jira Product Discovery and Trello products with services hosted in the AWS us-east-1 and us-west-2 region experienced slow performance and/or page load failures as a result of an AWS issue that began on September 18 2023, 4:00 PM UTC. This was triggered by an underlying networking fault in our cloud provider AWS, which affected multiple AWS services in their us-west-2 and us-east-1 regions, used by Atlassian. The incident was detected within one minute by our monitoring systems. Recovery of affected Atlassian services occurred on a product-by-product basis with full recovery for all products completed by September 19 2023 04:15 AM UTC. ‌ ### **IMPACT** Product impact varied based on which regions and availability zones services are using, with services hosted in us-west-2 being affected more than services hosted in us-east-1.   Product-specific impacts are listed below: * **Jira Software -** A number of Jira nodes were affected with highly elevated error rates due to Jira databases in us-east-1 and us-west-2 being impacted.  The impact was varied with some Jira nodes being unusable whilst others were in a usable but degraded state. * **Jira Service Management -** Some users hosted in us-east-1 and us-west-2 experienced problems when creating issues through the Help Center, viewing issues, transitioning issues, posting comments and using queues * **Jira Work Management -** Users based in us-west-2 experienced minor service degradation. * **Jira Product Discovery -** Users experienced some issues when loading insights. * **Confluence Cloud -** Impact was limited to customers hosted in the us-west-2 region. During this time, users attempting to load confluence pages experienced sporadic product degradation, including brief periods where Confluence was inaccessible, complete and partial page load failures, page timeouts, increased request latency. * **Trello -** Users had minimal service degradation - only 0.1% of Trello users had automation rules impacted. ### **ROOT CAUSE** The root cause was an issue with subsystem responsible for network mapping propagation within the Amazon Virtual Private Cloud in the us-east-1 \(use1-az1\) and us-west-2 \(usw2-az1 and usw2-az2\) regions, which impacted network connectivity for multiple AWS services which Atlassian products rely upon. There was a delay between the AWS incident and Atlassian being affected as existing compute instances and resources were not affected by the issue. However any changes to networking state - such as scaling-up with additional compute nodes - experienced delays in the propagation of network mappings. This led to network connectivity issues until these network mappings had been fully propagated. Other AWS services that create or modify networking resources also saw impact as a result of this issue. There were no relevant Atlassian-driven events in the lead-up that have been identified to cause or contribute to this incident. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** To avoid repeating this type of incident, we are prioritizing documenting and evaluating ways to improve Availability Zone failure resiliency. Thanks, Atlassian Customer Support

Between 09/18 10:47 UTC to 09/19 04:15 UTC, we experienced degraded performance for some Confluence, Jira Work Management, Jira Service Management, Jira Software, Trello, Atlassian Access, and Jira Product Discovery customers. The issue has been resolved and the service is operating normally.

Services are confirmed to be stable. We are performing final validation checks before confirming the incident's resolution.

We have identified the root cause of the downgraded performance and have mitigated the problem. We are monitoring this closely.

We are investigating cases of degraded performance for some Confluence, Jira Work Management, Jira Service Management, Jira Software, Trello, and Jira Product Discovery Cloud customers. We will provide more details within the next hour.

### **SUMMARY** On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes. ‌ ### **IMPACT** The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases. ‌ ### **ROOT CAUSE** The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service. ‌ ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Audit and improve service rate limits and client retry and backoff behavior. * Improve scale and load test automation for complex service interactions. * Audit cross-service dependencies and minimize them where possible related to sign-in flows. ‌ Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to: * Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available. * Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled. ‌ We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability. Thanks, Atlassian Customer Support

Between 12:45 UTC to 15:30 UTC, we experienced login and signup issues for Atlassian Accounts. The issue has been resolved and the service is operating normally. We will publish a post-incident review with the details of the incident and the actions we are taking to prevent similar problem in the future.

We are no longer seeing occurrences of the Atlassian Accounts login errors, all clients should be able to successfully login now. We will continue to monitor.

We can see a reduction in the Atlassian Accounts login issues after the mitigation actions were taken. We are still monitoring closely and will continue to provide updates.

We have identified the root cause of the Atlassian Accounts login issues impacting Cloud Customers and have mitigated the problem. We are now monitoring this closely.

We are investigating an issue with Atlassian Accounts login that is impacting some Cloud customers. We will provide more details within the next hour.

### **SUMMARY** On August 30, 2023, between 4:07 and 5:30 UTC, some customers were unable to login to Atlassian's Cloud products using [id.atlassian.com](http://id.atlassian.com).  Logged-in users were also unable to switch accounts, change passwords, or log out. Users with existing sessions were not impacted. Between 5:32 and 6:00 UTC, traffic was incrementally restored to a previous build, mitigating the impact for users. The total time to resolution was one hour and 53 minutes. ### **IMPACT** Users were not able to login using Atlassian's shared account management system \([id.atlassian.com](http://id.atlassian.com)\). This affected users who were trying to login to the following products: Jira, Confluence, Trello, Opsgenie, mobile apps and ecosystem apps. Aside from the inability to login, there was no impact on other Atlassian products or features. ### **ROOT CAUSE** Multiple Set-Cookie headers were unintentionally modified so that only the last Set-Cookie header remained in the response to user's browsers.  The issue was caused by a change to Network Extensions within the Edge Network. As a result, users that needed a new session could not login.  Upon login, the users were redirected to login again and no session was created for them. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue was not detected in Atlassian's staging environment.  End-to-end tests did not cover the use case of multiple Set-Cookie headers in the single response and therefore this bug went unnoticed. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Automated tests to be put in place to validate that cookies are not being removed from responses. * Configuration of networking extensions will be guaranteed to be identical in staging and production to ensure errors are picked up earlier. Furthermore, we typically deploy our changes progressively by cloud region to avoid broad impact, but in this case, the change was not deemed risky and was deployed to all regions. To minimize the impact of breaking changes to our environments, we will implement additional preventative measures: * Changes to network extensions in the future will use progressive rollouts. * With staging being properly utilized, errors similar to this one will not be deployed to any production environments. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Between 4:30AM UTC to 6:00AM UTC, we experienced issues for users attempting to login for Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Jira Product Discovery, Compass, and Atlassian Analytics. The issue has been resolved and the service is operating normally.

We are investigating reports of intermittent errors for login to Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Jira Product Discovery, Compass, and Atlassian Analytics Cloud customers. We will provide more details once we identify the root cause.

Between 04:00 UTC to 05:00 UTC, we experienced a partial outage for Bitbucket Pipelines, issues viewing in-product attachments; and degraded performance across all products. The issue has been resolved and all services are operating normally.

We have identified the root cause, initiated mitigation actions, and are seeing a reduction in errors when scaling up in response to customer traffic. We are still monitoring closely and will continue to provide updates.

A major infrastructure outage is affecting our ability to scale up in response to customer traffic. We are pre-emptively disabling any downscaling to minimize future impact. This is primarily affecting Bitbucket Pipelines, and attachment storage for all products. All regions are affected. We will provide more details in the next hour.

A major infrastructure outage is affecting our ability to scale up in response to customer traffic. We are pre-emptively disabling any downscaling to minimize future impact. This is primarily affecting Bitbucket Pipelines, and attachment storage for all products. All regions are affected. We will provide more details in the next hour.

Between 06:22 UTC to 7:20 UTC, we experienced a problem with the service responsible for file and media services across our cloud services including Confluence, Jira Work Management, Jira Service Management, Jira Software, Bitbucket and Jira Product Discovery. The issue has been resolved by reverting a network change and service has now been restored.

We are investigating an issue with the attachments or media service that is impacting some Atlassian Cloud customers. We will provide more details within the next hour.

This incident has been resolved.

We're experiencing an elevated level of API errors and are currently looking into the issue.

The small cohort of Jira Software, Jira Work Management, Jira Service Management, and Jira Product Discovery customers no longer face any issues in accessing their Atlassian Sites. We are implementing measures to prevent similar issues from occurring in the future.

The small cohort of Jira Software, Jira Work Management, Jira Service Management and Jira Product Discovery customers no longer face any issues in accessing their Atlassian Sites.

We are investigating reports of Jira Software, Jira Work Management, Jira Service Management and Jira Product Discovery being unavailable for a small cohort of customers.

We have resolved a case of degraded performance for Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Atlassian Access, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details within the next hour.

We are pleased to report that the issues concerning slowness and intermittent errors while creating new Cloud instances for Jira Service Management, Jira Software, Jira Work Management, and Jira Product Discovery in the Europe region have now been resolved. We have identified and rectified the root cause, ensuring smooth operations hereafter.

We are investigating reports of slowness and intermittent errors while creating new Cloud instances for Jira Service Management, Jira Software, Jira Work Management, and Jira Product Discovery in Europe region. No impact to existing Cloud instances was observed. We will provide more details as we identify the root cause and fix the issue

Between 04/08 to 07/08, we experienced Issues for Billing preview for Confluence, Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. The issue has been resolved and the service is operating normally. Users can now correctly browse their billing preview and take action

We are investigating reports of Billing preview not being displayed for some products for some customers. We will provide more details once we identify the root cause.

We mitigated the issue with Sign-ups, Product Activation, and Billing, and the systems are back to BAU, and all functionality is restored.

We have identified the root cause of the Sign-ups, Product Activation, and Billing not working and have mitigated the problem. We are now monitoring closely.

We are investigating an issue with Sign-ups, Product Activation, and Billing that is impacting all of our Cloud Customers. We will provide more details within the next hour.

### **SUMMARY** We understand the importance of providing reliable and consistent service to our valued customers. On July 6, 2023, from 03:52 to 15:11 UTC, we experienced an issue with an upgraded version of a third-party tool that functions as our internal artifact management system. Despite our monitoring system identifying the incident within two minutes, this issue led to the degradation of the scaling capabilities of our internal hosting platform, resulting in service degradation or outages for customers of Atlassian cloud. In response to this situation, we are taking immediate measures to enhance the stability of our system and prevent similar issues from re-occurring. ### **IMPACT** This incident affected multiple regions and products due to the diminished scaling capabilities of our internal hosting platform. In most products and offerings, customers faced reduced functionality, slower response times, and limited access to specific features. ### **ROOT CAUSE** The root cause of the incident was the introduction of new functionality in a third-party tool that functions as our internal artifact management system. It led to an unexpected increase in the load on the primary database of the artifact system. Upon identifying and localizing the problem, we promptly adjusted the system configuration to regain stability. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** Over the next months, we will enact a temporary freeze on non-critical upgrades of the artifact management system, and we will focus our efforts on three high-priority initiatives: 1. **Enhancing system scaling:** We prioritized work ensuring that downtime in a critical infrastructure component does not affect the scaling of other components. We expect to complete this initiative within the next two months. 2. **Reducing interdependencies:** We are working to mitigate the risk of potential cascading failures by ensuring that significant system components are able to operate independently in the case of issues. Initiatives 1 and 2 are already in progress but have been given priority to be completed as soon as possible. 3. **Strengthening testing procedures:** Alongside these initiatives, we are addressing the need for even more stringent testing procedures than we already have in place to prevent potential issues in future updates. We are committed to collaborating closely with our technology partners to ensure the most optimal experience for our customers. We apologize for any inconvenience caused by this incident and appreciate your understanding. Our team is dedicated to continually improving our systems and processes to provide you with the exceptional service you deserve. Thank you for your continued support and trust in us. Sincerely, Atlassian Customer Support

We experienced performance issues and outages for several Atlassian Cloud Products. The issue has been resolved and the service is operating normally.

We have identified the root cause of an issue with an internal infrastructure component that has been impacting multiple Cloud products, including Jira Software, Jira Service Management and Confluence, and customers. This issue had lead to a performance impact and, in some cases, outages. We have implemented a fix to resolve the issue and recovery is in progress.

We are investigating an issue with an internal infrastructure component that is impacting multiple Cloud products, including Jira Software, Bitbucket, Jira Service Management and Confluence, and customers. These issues include performance impact and, in some cases, outages. Users may experience slow loading and uploading of attachments, login issues or inability for new customers to sign up. We have identified the root cause and are actively working on the service recovery.

Between 07:31 UTC to 12:32 UTC, we experienced errors during login for Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, Compass, and Atlassian Analytics. The issue has been resolved and the service is operating normally.

We have identified the root cause of the errors during login and have mitigated the problem. We are now monitoring closely.

We are investigating reports of errors during login that is impacting some Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, Compass, and Atlassian Analytics. We have identified the root cause and expect recovery shortly.

We are investigating reports of errors during login for some customers that is impacting some Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, and Atlassian Analytics Cloud customers. We will provide more details within the next hour.

We are investigating reports of errors during login for some customers that is impacting some Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, and Atlassian Analytics Cloud customers. We will provide more details within the next hour.

We are investigating reports of errors during login for some customers that is impacting some Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, and Atlassian Analytics Cloud customers. We will provide more details within the next hour.

We are investigating reports of intermittent errors during login for some customers using Confluence, Jira Work Management, Jira Service Management, Jira Software, Trello, Atlassian Bitbucket, Atlassian Access, and Jira Product Discovery Cloud customers. We will provide more details once we identify the root cause.