Compass

Is Compass Down Right Now? Check if there is a current outage ongoing.

Compass is currently Operational

Last checked from Compass's official status page

Historical record of incidents for Compass

Report: "Customers may experience delays or failures receiving emails"

Last update
investigating

We were experiencing cases of degraded performance for outgoing emails from Confluence, Jira Work Management, Jira Service Management, Jira, Opsgenie, Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Compass, and Loom Cloud customers. The system is recovering and mail is being processed normally as of 16:45 UTC. We will continue to monitor system performance and will provide more details within the next hour.

Report: "Both Compass UI and APIs are not responding"

Last update
resolved

We have verified that the incident has been resolved.

identified

The issue has been identified and a fix is being implemented. We have put some mitigation mechanisms in place and our systems are starting to recover.

investigating

We are currently investigating an issue impacting both the Compass UI and APIs. We are investigating this issue and will share an update once we have additional information.

Report: "Both Compass UI and APIs are not responding"

Last update
Investigating

We are currently investigating an issue impacting both the Compass UI and APIs. We are investigating this issue and will share an update once we have additional information.

Report: "Issues with data pipelines for customers using GitHub for Jira Connector."

Last update
resolved

Existing events are continuing to be processed. Any updates to existing entities such as pull requests will also correct and update the data here as required. If there are any concerns about impact to your site with this connector please reach out to support and the team will address your concerns as soon as possible.

identified

Processing of previously impacted events is now underway. An update will be provided when these operations have been completed and historical processing has been completed.

identified

Our team has identified the root cause of this issue and have reverted the change that was causing these data processing errors. All new updates from GitHub should now be flowing correctly. We are actively working to address a potential data gap for previously impacted events, and will provide an update when available.

investigating

We are aware of an issue where users of GitHub for Jira connector may be missing pull requests and experiencing other data pipeline issues. Our team is looking into this with urgency and an update will be provided when available.

Report: "Issues with data pipelines for customers using GitHub for Jira Connector."

Last update
Investigating

We are aware of an issue where users of GitHub for Jira connector may be missing pull requests and experiencing other data pipeline issues. Our team is looking into this with urgency and an update will be provided when available.

Report: "New components not visible when performing searches"

Last update
resolved

This incident has been resolved.

monitoring

We've implemented a fix for this issue and are seeing systems recover. We will keep monitoring to make sure we don't see a recurrence of this issue.

investigating

We're seeing a recurrence of the earlier issue around components not showing up in our search indexing. We're working on restoring the service.

Report: "New components not visible when performing searches"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We're investigating an issue around new components not being visible in search for a period of time after creation.

Report: "Management of Apps is partially unavailable on different platforms"

Last update
resolved

Between 10:55 AM UTC to 13:15 PM UTC on April 28, 2025, we experienced partial unavailability in managing Forge apps; Such as lifecycle developer console usage, and the creation and deployment of Forge apps for Atlassian Partners, Jira Work Management, Jira, Atlassian Developer, Jira Product Discovery, Compass, and Atlassian Migrations. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The conditions that cause the bug have been addressed and we're actively working on a permanent fix. The issue has been resolved and the service is operating normally.

monitoring

The issues causing the unavailability of related Forge apps capabilities on multiple Atlassian platforms have been resolved, and services are now operating normally for all impacted customers. We will continue to closely monitor for stability.

identified

We are currently experiencing partial unavailability in managing Forge apps on different Atlassian platforms, such as apps lifecycle (installation, removals), developer console usage (including tunneling), and the creation and deployment of Forge apps. We have identified the root cause and are in the process of mitigating the impact. We'll keep you posted with further updates. Next updates in 1 hour.

investigating

We are currently experiencing partial unavailability in managing Forge apps on different Atlassian platforms. Such as apps lifecycle (installation, removals), developer console usage (including tunnelling), and the creation and deployment of Forge apps. Our team is working diligently to resolve this issue and restore services quickly. We are in the process of identifying the root cause and implementing the necessary fixes. Next updates in 1 hour.

Report: "New components not visible when performing searches"

Last update
Investigating

We're investigating an issue around new components not being visible in search for a period of time after creation.

Report: "Management of Apps is partially unavailable on different platforms"

Last update
Investigating

We are currently experiencing partial unavailability in managing Forge apps on different Atlassian platforms. Such as apps lifecycle (installation, removals), developer console usage (including tunnelling), and the creation and deployment of Forge apps. Our team is working diligently to resolve this issue and restore services quickly. We are in the process of identifying the root cause and implementing the necessary fixes.Next updates in 1 hour.

Report: "[Scheduled Maintenance] Database Upgrade"

Last update
Scheduled

Compass will be conducting a required database upgrade that will impact the ability to perform any write operations to Compass for all users on April 13, 2025 between 5:00 UTC and 6:00 UTC.For more information about this maintenance, please refer to the community post - https://community.atlassian.com/forums/Compass-articles/Scheduled-maintenance-for-Compass/ba-p/2991224.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Report: "Some users are experiencing errors while creating new Jira and Confluence sites"

Last update
resolved

On March 14th, 2025, between 09:00 and 19:14 UTC, some customers experienced issues creating new sites. The errors occurring for some users during new site creation should now be resolved.

investigating

We are investigating cases of errors for some users creating new sites. We will provide more details within the next hour.

Report: "Issues with search within Confluence and Compass"

Last update
resolved

Searches within Confluence and Compass should now be operating as normal. This issue will now be resolved.

monitoring

Issues causing error messages when users were attempting to search within Confluence and Compass should now be resolved. Our team is monitoring ongoing performance at this time.

investigating

We are aware that some users of Confluence and Compass are currently experiencing errors when attempting to search. Our team is investigating this issue with urgency and will provide an update when available.

Report: "Unable to load the Operations page"

Last update
resolved

The issue has been resolved and the service is operating normally.

identified

We are investigating reports of the Operations page not loading for Compass Cloud customers. There is no impact to the API. We have identified the issue and are in the process of fixing it.

Report: "Github integration not working"

Last update
resolved

Compass customers using Github may have experienced intermittent problems in receiving events or leveraging features such as "config-as-code", this was due to an outage on Github

Report: "Gitlab integration not working"

Last update
resolved

Compass customers using Gitlab may have experienced intermittent problems in receiving events or leveraging features such as "config-as-code", this was due to an outage on Gitlab.

Report: "Bitbucket integration is experiencing a partial outage related to an ongoing incident with Bitbucket Cloud"

Last update
resolved

We have verified that the Bitbucket integration is now working as expected. Marking this incident as Resolved.

investigating

Bitbucket Cloud is experiencing a major outage: https://bitbucket.status.atlassian.com/ . Compass cannot setup new integrations with BBC and is not able to process updates. Events and metrics that have already been synced from Bitbucket to Compass are not impacted.

Report: "Gitlab integration not operational"

Last update
resolved

Gitlab has mitigated the issue and we are seeing recovery and no errors during ingestion.

investigating

We're seeing issues with the Gitlab integration, given Gitlab's ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32

investigating

We're seeing issues with the Gitlab integration, given Gitlab's ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32

Report: "Issues accessing Compass"

Last update
resolved

This incident has been resolved.

identified

We're seeing recovery across all of our systems. The team is still working on ensuring that impact is fully mitigated.

investigating

We're seeing issues accessing the Compass application. The team is working on identifying the underlying cause.

Report: "Compass service inaccessible"

Last update
resolved

This incident has been resolved.

monitoring

We've identified the issue and reverted the faulty change. We're monitoring the change propagation and are seeing systems recover across the board.

investigating

We're seeing issues accessing Compass. The team has identified the issue and is currently rolling out a fix.

Report: "Compass Github app unavailable"

Last update
resolved

This issue has now been fixed and the app is now available for ise.

identified

We have identified the issue and are working on a fix at the moment.

investigating

We are currently investigating an issue that is causing the Github app to be unavailable for some tenants.

Report: "Compass services not accessible"

Last update
resolved

This incident has been resolved.

identified

We've identified the cause of the issue and are seeing service quality recover.

investigating

We are currently investigating this issue.

Report: "Github integration not operational"

Last update
resolved

This incident has been resolved.

identified

We're seeing issues with the Github integration, given Github's ongoing issues: https://www.githubstatus.com/incidents/kz4khcgdsfdv

Report: "Event ingestion delayed"

Last update
resolved

Event ingestion and metric derivation are both working at normal capacity. This incident is resolved.

monitoring

Event ingestion times are back to normal, and we're seeing the delay between event ingestion and metric derivation go down. We're still monitoring the fix and will update once things are back to nominal.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The team has identified the cause for the delays and is rolling out a fix.

investigating

We're seeing event ingestion and metric derivation delayed. The team is working on resolving the issue.

Report: "Issues logging in to Compass"

Last update
resolved

Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Compass. The issue has been resolved and the service is operating normally.

monitoring

The team has implemented a mitigation and we're seeing service quality hold. We're continuing to look into the root cause of the issue. We've also been able to isolate the time of impact to a 20-minute timeframe between 8:08pm 03/07 UTC and 08:31pm 03/07 UTC.

identified

We're seeing recovery but are still working on identifying the root cause of the issue.

investigating

We're seeing customers experience issues logging into Compass and using the service. The team has been notified and is investigating the cause.

Report: "The Developer Experience and Pull Requests dashboard (in the Teams page) include incorrect PR data"

Last update
resolved

The incident has been resolved.

monitoring

We've resolved the issue and are monitoring to verify changes.

identified

The Developer Experience and Pull Requests dashboard in the Teams page include incorrect PR data from other repositories (in the same workspace) connected to the Compass instance. The root cause has been identified and a fix is in progress.

Report: "Users experiencing errors using when using the product"

Last update
resolved

This incident has been resolved.

monitoring

We've rolled back the change that was identified as having caused the issues and we're seeing systems recover. We will continue monitoring recovery and will post further updates as needed.

identified

The issue has been identified and a fix is being implemented.

investigating

We're seeing errors around authentication across the platform. The team has been alerted and is working on identifying the underlying issue.

Report: "Navigating to Apps in Component Details is not working"

Last update
resolved

This incident has been resolved.

investigating

We have released and verified the fix. Compass is operational again. Users are able to navigate to apps.

investigating

Users are not able to navigate to "apps" in the App menu on the Component details view. We are currently rolling out the fix. We will report back once the fix has been verified.

Report: "Admin Portal Feature Access Issue"

Last update
resolved

Between 6:30 AM UTC to 9:50 AM UTC, we experienced failures in accessing some features from the Admin Portal. The issue has been resolved and the service is operating normally.

identified

We are investigating an issue causing failures in accessing some features from the Admin Portal, which is impacting some of our Cloud customers. We have identified the root cause and anticipate recovery shortly.

Report: "Some paginated queries in Forge hosted storage kept repeating the last page"

Last update
resolved

We have identified and resolved a problem with Forge hosted storage, where some paginated queries kept repeating the last page. The incident was detected by our internal monitoring and was resolved quickly after detection by reverting the deployment. Activating changes recently made to the query cursors for paginated queries introduced a bug that impacted some apps. A small number of requests were impacted over a 16-minute window, while the incident lasted. Timeline: - 25/Mar/24 10:08 p.m. UTC - Impact started, when the changes were deployed to production - 25/Mar/24 10:09 p.m. UTC - Incident was detected - 25/Mar/24 10:24 p.m. UTC - Incident was resolved and impact ended The impact of this incident has been completely mitigated and our monitoring tools confirm that query operations are back to the pre-incident behaviour. We have also resolved the underlying bug and deployed the fix to production, completely eliminating the cause of this incident. We apologise for any inconvenience this may have caused to our customers and our developer community.

Report: "Investigating new product purchasing"

Last update
resolved

Between 28th Feb 2024 23:15 UTC to 29th Feb 2024 00:05 UTC, we experienced issue with new product purchasing for all products. All new sign up products have been successfully provision and confirmed issue has been resolved and the service is operating normally.

investigating

We are investigating an issue with new product purchasing that is impacting for all products. Customers adding new cloud products may have experienced a long waiting page or an error page after attempting to add a product. We have mitigated the root cause and are working to resolve impact for customers who attempted to add a product during the impact period. We will provide more details within the next hour.

Report: "Service Disruptions Affecting Atlassian Products"

Last update
postmortem

### **Summary** On February 14, 2024, between 20:05 UTC and 23:03 UTC, Atlassian customers on the following cloud products encountered a service disruption: Access, Atlas, Atlassian Analytics, Bitbucket, Compass, Confluence, Ecosystem apps, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery, Opsgenie, StatusPage, and Trello. As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names used for internal service-to-service connections. Active domain names were incorrectly deleted during this event. This impacted all cloud customers across all regions. The issue was identified and resolved through the rollback of the faulty deployment to restore the domain names and Atlassian systems to a stable state. The time to resolution was two hours and 58 minutes. ### **IMPACT** External customers started reporting issues with Atlassian cloud products at 20:52 UTC. The impact of the failed change led to performance degradation or in some cases, complete service disruption. Symptoms experienced by end-users were unsuccessful page loads and/or failed interactions with our cloud products. ### **ROOT CAUSE** As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names that were being used for internal service-to-service connections. Active domain names were incorrectly deleted during this operation. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The detection was delayed because existing testing & monitoring focused on service health rather than the entire system’s availability. To prevent a recurrence of this type of incident, we are implementing the following improvement measures: * Canary checks to monitor the entire system availability. * Faster rollback procedures for this type of service impact. * Stricter change control procedures for infrastructure modifications. * Migration of all DNS records to centralised management and stricter access controls on modification to DNS records. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. ‌ Thanks, Atlassian Customer Support

resolved

We experienced increased errors on Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Align, Jira Product Discovery, Atlas, Compass, and Atlassian Analytics. The issue has been resolved and the services are operating normally.

monitoring

We have identified the root cause of the Service Disruptions affecting all Atlassian products and have mitigated the problem. We are now monitoring this closely.

identified

We have identified the root cause of the increased errors and have mitigated the problem. We continue to work on resolving the issue and monitoring this closely.

investigating

We are investigating reports of intermittent errors for all Cloud Customers across all Atlassian products. We will provide more details once we identify the root cause.

Report: "Newly created/updated Compass components are not visible while searching via the GraphQL API or UI"

Last update
resolved

This incident has been resolved.

investigating

The issue has been resolved and impact has been mitigated.

investigating

This issue is under investigation.

Report: "User searches failing"

Last update
resolved

Between 15:40 UTC to 15:57 UTC customers experienced intermittent failures when searching for users in Atlassian cloud services: Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, and Compass. The issue has been resolved and the service is operating normally.

monitoring

We have mitigated the problem. We are now monitoring closely.

investigating

We are investigating an issue with our user search service that is impacting Atlassian cloud service: Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, and Compass. We will provide more details within the next hour.

Report: "Drop in the success rate of Forge hosted storage API calls"

Last update
resolved

We identified a problem with the Forge hosted storage API calls, which resulted in a drop in invocation success rates in the developer console. The impact of this incident has been mitigated and our monitoring tools confirm that the success-rate is back to the pre-incident behaviour. It impacted 16 apps according to our logs, where these apps saw a reduced success rate of storage.get API calls, as listed in https://developer.atlassian.com/platform/forge/runtime-reference/storage-api-basic. As part of Forge's preparation to support Data Residency, Forge hosted storage has been undergoing a platform and data migration for storing app data. As part of this migration we do comparison checks for data consistency between the old and new platform. The previous incident earlier, https://developer.status.atlassian.com/incidents/9q71ytpjhbtl, had put the data on the new platform out of sync and so comparisons of the data from the old and new platform started showing failures and the migration logic retries on failures to test for consistency issues. This retry behaviour increased latency of these requests which led to 16 apps receiving an increased number of 504 timeout errors. Checking synchronously was identified by the team as a bug and should have been async. Once the root cause was identified we moved our backing platform rollout to a previous stage. The rollout is split into several stages. The issues we were having were on our blocking stage where we make calls to both the old and new platform and wait for both to complete so we can test any performance issues in the new platform before using it as our source of truth. It was in this blocking stage where we had a bug that included waiting on comparisons when it should've been async. To recover, we reverted back to our shadow mode stage. In this stage, all operations to the new platform are asynchronous, including comparisons that were blocking in the other stage and resulted in timeout issues and 504 errors being sent to apps. This is the state that Forge hosted storage has been in for several months without any problems. Here is the timeline of the impact: - On 2024-02-05 at 06:42 PM UTC, impact started with comparisons start happening on out of sync data in blocking mode - On 2024-02-05 at 08:57 PM UTC, impact was detected to API by our monitoring systems - On 2024-02-05 at 11:34 PM UTC, rollout to new platform was reverted to known stable state and impact ended We will release a public incident review, PIR, here in the upcoming weeks for this and the incident that happened earlier, https://developer.status.atlassian.com/incidents/9q71ytpjhbtl. We will detail all that we can about what caused the issue, and what we are doing to prevent it from happening again. We apologise for any inconveniences this may have caused our customers and the developer community and committed to preventing further issues with our hosted storage capability.

Report: "Performance degradation for Forge app invocations"

Last update
resolved

At around 4am UTC, about 40 percent of Forge app invocations experienced high latency, with a portion of the requests failing, during a 15 minute time window. The scaling of the instances was misconfigured following a new deployment of the service, which needed manual intervention which took a few minutes to resolve the issue. Timeline: - 2024-01-31 04:00 UTC: impact started - 2024-01-31 04:03 UTC: incident detected - 2024-01-31 04:15 UTC: the incident was resolved and the impact ended This issue is now resolved and Forge is fully operational. We apologize for any inconveniences this may have caused to our customers, partners, and our developer community.

Report: "Outage in Atlassian Intelligence functionality in multiple products"

Last update
resolved

Between 23:45 UTC to 00:30 UTC, we experienced an outage in some Atlassian Intelligence features for Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, Atlas, and Compass. The issue has been resolved and the service is operating normally.

monitoring

We have identified the root cause of the increased errors and have mitigated the problem. We are now monitoring closely.

investigating

We are investigating an issue with Atlassian Intelligence that is impacting some Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details within the next hour.

Report: "Intermittent errors when trying to access Compass"

Last update
resolved

Between 11:00 UTC to 12:00 UTC, we experienced degradation in our services. This impacted our catalog, search and provisioning operations. The issue has been stabilized and services are operating normally.

monitoring

Our services are stable now. We are now monitoring closely.

investigating

We are investigating reports of intermittent errors for some Compass Cloud customers. We will provide more details once we identify the root cause.

Report: "Atlassian's cross product user search service is currently degraded."

Last update
postmortem

### **SUMMARY** On Dec 18, 2023, between 12:29 p.m. and 3:35 p.m. UTC, Dec 18, 2023, Atlassian's cloud customers using Atlas, Bitbucket Cloud, Compass, Confluence Cloud, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery products were unable to search for users or use the "@mention" functionality. Customers' user search results failed or were delayed as Atlassian's service returning user search results was degraded in several regions. The incident originated from a computationally intensive operation that was triggered multiple times in rapid succession, resulting in degraded performance of Atlassian's user search service across several regions. Notably, customers in the EU west region were most affected. The incident was detected within 2 minutes by automated monitoring, and our team promptly took action by recovering unhealthy systems and scaling up the service's infrastructure temporarily. The resolution process concluded in 3 hours and 06 minutes. ### **IMPACT** The overall impact was between Dec 18, 2023, between 12:29 p.m. UTC and Dec 18, 2023, 3:35 p.m. UTC. The Incident caused service disruption to cloud customers worldwide. Customers experienced delayed or failed user searches when using the following Atlassian cloud products: * Atlas * Bitbucket Cloud * Compass * Confluence Cloud * Jira Service Management * Jira Software * Jira Work Management * Jira Product Discovery ### **ROOT CAUSE** The incident stemmed from Atlassian's user search service receiving commands to process multiple computationally intensive operations in rapid succession. These operations were directed at the same customer data set, and therefore overloaded resources within a clustered database system, leading to memory exhaustion and subsequent unresponsiveness to user search requests.  ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** To prevent a recurrence of such incidents, we are implementing the following measures: * Implement a mechanism to queue computationally intensive operations in order to avoid overloading the resources within the systems and process them without impact on customer experience. * Fine-tune our clustered database settings to mitigate the impact of resource exhaustion on the overall system.  We apologize to customers whose services were affected during this incident; we are taking immediate steps to improve the service’s resiliency. Thanks, Atlassian Customer Support

resolved

It has been resolved. Atlassian's cross product user search is working.

investigating

Atlassian's cross product user search service is currently healthy. Searches for users within Atlassian products are working as expected. We are in the process of investigating the root cause of this incident.

investigating

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

investigating

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

investigating

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

investigating

We are investigating reports of intermittent errors for <SOME/ALL> Atlassian, Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details once we identify the root cause.

Report: "Search indexing delays"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We're seeing degraded performance with our indexing pipeline which may result in components not showing up in search for several minutes after creation.

Report: "Gitlab Forge Integration Degraded"

Last update
resolved

Gitlab has resumed processing all background jobs. The Gitlab Forge integration is now fully operational.

identified

We have identified an issue with our Gitlab integration. Gitlab is currently experiencing delays in background processing that is impacting the Gitlab Forge integration.

Report: "Degraded performance for Forge integrations"

Last update
resolved

This incident has been resolved.

monitoring

We're seeing error rates subside and app behavior return to normal, but will continue monitoring impact.

identified

We are continuing to work on a fix for this issue.

identified

We're seeing an increase in errors in API calls to the Atlassian Forge platform across all our integrations. The team has identified the cause and is working on mitigating impact.

Report: "Atlassian Account login issues"

Last update
postmortem

### **SUMMARY** On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes. ‌ ### **IMPACT** The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases. ‌ ### **ROOT CAUSE** The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service. ‌ ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Audit and improve service rate limits and client retry and backoff behavior. * Improve scale and load test automation for complex service interactions. * Audit cross-service dependencies and minimize them where possible related to sign-in flows. ‌ Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to: * Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available. * Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled. ‌ We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability. Thanks, Atlassian Customer Support

resolved

Between 12:45 UTC to 15:30 UTC, we experienced login and signup issues for Atlassian Accounts. The issue has been resolved and the service is operating normally. We will publish a post-incident review with the details of the incident and the actions we are taking to prevent similar problem in the future.

monitoring

We are no longer seeing occurrences of the Atlassian Accounts login errors, all clients should be able to successfully login now. We will continue to monitor.

monitoring

We can see a reduction in the Atlassian Accounts login issues after the mitigation actions were taken. We are still monitoring closely and will continue to provide updates.

monitoring

We have identified the root cause of the Atlassian Accounts login issues impacting Cloud Customers and have mitigated the problem. We are now monitoring this closely.

investigating

We are investigating an issue with Atlassian Accounts login that is impacting some Cloud customers. We will provide more details within the next hour.

Report: "Multiple product logins"

Last update
resolved

Between 4:30AM UTC to 6:00AM UTC, we experienced issues for users attempting to login for Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Jira Product Discovery, Compass, and Atlassian Analytics. The issue has been resolved and the service is operating normally.

investigating

We are investigating reports of intermittent errors for login to Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Jira Product Discovery, Compass, and Atlassian Analytics Cloud customers. We will provide more details once we identify the root cause.

Report: "Atlassian services unable to scale"

Last update
resolved

We have resolved a case of degraded performance for Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Atlassian Access, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details within the next hour.

Report: "Degraded performance for Github Forge App"

Last update
resolved

This incident has been resolved.

investigating

We are noticing degraded performance for customers using Github Forge App. This is due to external dependency on Github's webhook delivery We will provide more details within the next hour.

Report: "Sign-ups, Product Activation, and Billing not working"

Last update
resolved

We mitigated the issue with Sign-ups, Product Activation, and Billing, and the systems are back to BAU, and all functionality is restored.

monitoring

We have identified the root cause of the Sign-ups, Product Activation, and Billing not working and have mitigated the problem. We are now monitoring closely.

investigating

We are investigating an issue with Sign-ups, Product Activation, and Billing that is impacting all of our Cloud Customers. We will provide more details within the next hour.

Report: "Github App installation unavailable"

Last update
resolved

This incident has been resolved.

identified

We have identified and fixed an issue with installing Compass Github App. The issue was related to a bug introduced in Github's OAuth API as documented here https://github.com/orgs/community/discussions/61291.

identified

Due to a recent change, users are unable to complete the installation flow in the Compass Github App. All other functionality in the Compass Github App remains unaffected and is working as intended. The team has identified the issue and is working on deploying a fix.

Report: "Performance issues and outages with Cloud products"

Last update
postmortem

### **SUMMARY** We understand the importance of providing reliable and consistent service to our valued customers. On July 6, 2023, from 03:52 to 15:11 UTC, we experienced an issue with an upgraded version of a third-party tool that functions as our internal artifact management system. Despite our monitoring system identifying the incident within two minutes, this issue led to the degradation of the scaling capabilities of our internal hosting platform, resulting in service degradation or outages for customers of Atlassian cloud. In response to this situation, we are taking immediate measures to enhance the stability of our system and prevent similar issues from re-occurring. ### **IMPACT** This incident affected multiple regions and products due to the diminished scaling capabilities of our internal hosting platform. In most products and offerings, customers faced reduced functionality, slower response times, and limited access to specific features. ### **ROOT CAUSE** The root cause of the incident was the introduction of new functionality in a third-party tool that functions as our internal artifact management system. It led to an unexpected increase in the load on the primary database of the artifact system. Upon identifying and localizing the problem, we promptly adjusted the system configuration to regain stability. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** Over the next months, we will enact a temporary freeze on non-critical upgrades of the artifact management system, and we will focus our efforts on three high-priority initiatives: 1. **Enhancing system scaling:** We prioritized work ensuring that downtime in a critical infrastructure component does not affect the scaling of other components. We expect to complete this initiative within the next two months. 2. **Reducing interdependencies:** We are working to mitigate the risk of potential cascading failures by ensuring that significant system components are able to operate independently in the case of issues. Initiatives 1 and 2 are already in progress but have been given priority to be completed as soon as possible. 3. **Strengthening testing procedures:** Alongside these initiatives, we are addressing the need for even more stringent testing procedures than we already have in place to prevent potential issues in future updates. We are committed to collaborating closely with our technology partners to ensure the most optimal experience for our customers. We apologize for any inconvenience caused by this incident and appreciate your understanding. Our team is dedicated to continually improving our systems and processes to provide you with the exceptional service you deserve. Thank you for your continued support and trust in us. Sincerely, Atlassian Customer Support

resolved

We experienced performance issues and outages for several Atlassian Cloud Products. The issue has been resolved and the service is operating normally.

monitoring

We have identified the root cause of an issue with an internal infrastructure component that has been impacting multiple Cloud products, including Jira Software, Jira Service Management and Confluence, and customers. This issue had lead to a performance impact and, in some cases, outages. We have implemented a fix to resolve the issue and recovery is in progress.

identified

We are investigating an issue with an internal infrastructure component that is impacting multiple Cloud products, including Jira Software, Bitbucket, Jira Service Management and Confluence, and customers. These issues include performance impact and, in some cases, outages. Users may experience slow loading and uploading of attachments, login issues or inability for new customers to sign up. We have identified the root cause and are actively working on the service recovery.

Report: "Intermittent errors during login for some customers"

Last update
resolved

Between 07:31 UTC to 12:32 UTC, we experienced errors during login for Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, Compass, and Atlassian Analytics. The issue has been resolved and the service is operating normally.

monitoring

We have identified the root cause of the errors during login and have mitigated the problem. We are now monitoring closely.

identified

We are investigating reports of errors during login that is impacting some Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Product Discovery, Compass, and Atlassian Analytics. We have identified the root cause and expect recovery shortly.

Report: "Compass Bitbucket Event Integration Degraded"

Last update
resolved

Between 17:30 to 22:20 UTC, we experienced issues with the Bitbucket App for Compass resulting in missing events and failed Config-As-Code syncs. The issue has been resolved and the service is operating normally.

investigating

We are investigating reports of intermittent errors for some Compass Cloud customers using the Bitbucket cloud event integration. We will provide more details once we identify the root cause.

Report: "Degraded performance impacting Search capabilities and signups"

Last update
resolved

Between 01:38 UTC to 02:46 UTC, we experienced degraded performance of Confluence, Jira Work Management, Jira Service Management, Jira Software, Jira Product Discovery, and Compass impacting Search capabilities, creation of new product instances and joining existing ones. The issue has been resolved and the service is operating normally.

identified

We are investigating reports of intermittent errors for some Confluence, Jira Work Management, Jira Service Management, Jira Software, Jira Product Discovery, and Compass Cloud customers impacting Search capabilities, creation of new product instances or joining existing ones. We will provide more details once we identify the root cause.

Report: "Intermittent Search and Page Load Failures"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are currently working to resolve intermittent search and page load failures. We have identified the root cause and are working on a fix.

Report: "Atlassian cloud product signup timeout"

Last update
resolved

Between 08:20 UTC to 09:30 UTC, we experienced degraded performance on new account signups for Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Atlassian Access, Atlas, and Compass. The issue has been resolved and the service is operating normally.

identified

We are investigating reports of intermittent errors for Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Atlassian Access, Atlas, and Compass Cloud customers. We will provide more details once we identify the root cause.

Report: "Errors navigating products, logging in, and logging out"

Last update
resolved

Between 03/Aug/22 15:20 UTC to 03/Aug/22 17:47 UTC, some customers experienced errors using Atlassian products, including errors while logging in or being forcibly logged out. The root cause was a DNS service deployment in US East region, which caused widespread DNS lookup errors for a variety of Atlassian services including authentication services. We have rolled back the change to mitigate the issue and have verified that the authentication services have recovered. The issue has been resolved and Atlassian services are operating normally.