Is Exalate Down Right Now? Check if there is a current outage ongoing.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

This upgrade for ServiceNow and Salesforce includes important bug fixes and improvements.For detailed information, please refer to the release notes:- https://docs.exalate.com/docs/salesforce-version-5250- https://docs.exalate.com/docs/servicenow-version-5250- Maintenance Window: The maintenance is scheduled to take place between 10 AM CET and 2 PM CET. During this time, updates will be applied to all customer nodes using the ServiceNow and Salesforce.- Service Impact for Your Instance: For your specific instance, the service will experience a brief interruption of approximately 3-5 minutes during a scheduled restart.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

This upgrade for Jira Cloud includes important bug fixes and improvements.For detailed information, please refer to the release notes:- https://docs.exalate.com/docs/jira-cloud-version-5250- Maintenance Window: The maintenance is scheduled to take place between 10 AM CET and 6 PM CET. During this time, updates will be applied to all customer nodes using the Jira Cloud.- Service Impact for Your Instance: For your specific instance, the service will experience a brief interruption of approximately 3-5 minutes during a scheduled restart.

All affected services are now back online, and no further impacts have been observed. We apologize for any inconvenience this may have caused.

All Cloud nodes are working as expected. We will continue to monitor the situation.

We've successfully restored 90% of the nodes, and they're now fully operational. We're actively working on the remaining systems and closely monitoring performance.

We are continuing to work on a fix for this issue.

We have successfully identified the root cause of the recent node downtime and communication disruptions. A mitigation is now being deployed, and we will maintain continuous monitoring of the system.

As part of our continued investigation into the recent service disruption, a number of nodes have been identified as temporarily unreachable. Our engineering teams are prioritizing node recovery and service restoration.

We are currently tracking an issue where cloud nodes are getting errors communication with the destination side and blocking the synchronization. Resolving the errors will make the synchronization work again but another error could reappear.

This incident has been resolved. We will be providing an RCA next week.

incident has been mitigated, we are currently monitoring.

Some nodes are currently down due. We are currently investigating and working to restore the service as soon as possible.

incident has been mitigated, we are currently monitoring.

Some nodes are currently down due. We are currently investigating and working to restore the service as soon as possible.

We are continuing to monitor for any further issues.

All nodes are back online, we will keep monitoring and will be providing further updates.

We have identified and we currently working on restoring the nodes.

We're currently experiencing an outage impacting some nodes due to an issue on a cluster. Our team is actively investigating and working to restore service as quickly as possible. We apologize for any inconvenience and appreciate your patience.

The incident is resolved.

We are continuing to monitor for any further issues.

A fix has been rolled out, and we are monitoring.

The website is currently not available and we are investigating with the hoster the reason why.

We consider the case closed. The root cause was a degraded performance at the salesforce side. We are looking how this can be notified in the Exalate product.

All nodes are back to normal. We continue monitoring

All the nodes are back operationally, but currently we don't see any activity on these nodes. We are moving the incident into monitoring (and will be doing some additional testing, lowering the severity to 'degraded performance'

Integration services to Salesforce are affected by a major service interruption on the Salesforce side Check here - https://status.salesforce.com/products/Salesforce_Services We are monitoring the situation and will be handling it as new information from Salesforce is provided.

The incident has been resolved, we will be providing a postmortem soon.

All nodes are back online, we will continue monitoring and will provide further updates.

We are continuing to work on resolving the outage. Currently, 80% of the affected nodes have been restored and are fully operational.

We are currently experiencing an incident affecting a limited number of nodes in our system. Current situation: We've identified the root cause and a mitigation has started, some nodes are back online.

The issue affecting Exalate Cloud has been resolved. We are currently preparing a post-mortem report, which will be shared early next week. Thank you for your patience and understanding.

All nodes are now back online. We will continue monitoring and further updates will be provided.

We are still addressing the issue affecting some of our nodes. As of now, 50% of the affected nodes are back online. Our team is working diligently to restore full functionality as soon as possible.

A fix is now being implemented.

We have identified an issue where some nodes are unreachable due to a routing problem. We have started to investigate the issue and will provide further updates shortly.

The fix has been implemented, we'll monitoring the environment for an additional day.

A containment has been agreed and is now being implemented.

Due to a routing problem, a subset of nodes (currently about 80%) are unreachable. Root causing is ongoing and status updates will be provided every 2 hours.

The incident has been resolved. All nodes are operational.

We are continuing to monitor for any further issues.

The incident has been successfully mitigated. Our team has taken all necessary actions to address the issue, and the situation is now under control. However, we are committed to ensuring that the resolution is sustained, and as such, we will continue to monitor the situation closely. Should any further action be required, we will take it promptly and keep you informed. Please feel free to reach out if you have any questions or need further clarification.

We have Identified the issue and we are currently working on a solution. If you are being affected by this outage, please create a Support ticket.

We are currently experiencing an incident affecting a limited number of ZenDesk nodes. Current Situation: Affected Nodes: Approximately 30% of ZenDesk nodes are currently experiencing downtime. Actions Taken: This is currently been investigated by our Eng department. Impact on Users: Some Zendesk nodes are not accessible. Ongoing Work: We are currently testing a solution for the customer affected by this outage, once proven effective, the solution will be roll out to all affected customer. At this time there is no action needed from the customer side. We appreciate your patience and understanding during this time.

The incident has been resolved. All nodes are operational.

Update: A fix has been deployed, and affected nodes are now gradually coming back online. We're continuing to monitor closely to ensure full recovery. Thank you for your patience and understanding.

We're currently experiencing an outage impacting some nodes due to a database issue in one cluster. Our team is actively investigating and working to restore service as quickly as possible. We apologize for any inconvenience and appreciate your patience.

Dear customers We are pleased to confirm that the issue affecting the Exalate Mapper function has been fully resolved. Following the successful system restart and continuous monitoring, we have confirmed that all functionalities are performing optimally without any discrepancies. We appreciate your patience and understanding throughout this process and are grateful for your continued trust in our services. Should you need any further assistance or have questions, please feel free to contact our support team. Warm regards, The Exalate team

Dear Exalate customers We are pleased to inform you that the scheduled restart of our Exalate systems on Jira Cloud has been successfully completed. We appreciate your patience and understanding. Current Status: Our systems are now fully operational, and all functionalities have been restored. We are actively monitoring the performance to ensure that everything is working as expected and to quickly address any potential issues. Next Steps: While we are confident in the stability and enhanced performance following the restart, we will continue to monitor the system closely over the next few days. This is part of our commitment to maintaining the highest standards of service reliability and user experience. No Action Required: At this time, there is no further action required on your part. However, we encourage you to report any unusual behavior or issues you encounter, as your feedback is invaluable in helping us maintain and improve our service. Thank You: Once again, we thank you for your cooperation and understanding as we continue to enhance our systems. Should you have any questions or require assistance, please feel free to reach out to our support team through the Exalate Support Portal. Best Regards, The Exalate Team

Hi In our ongoing efforts to ensure optimal performance and reliability, we are scheduling a restart of our Exalate systems on Jira Cloud. This action is part of our commitment to address and rectify the issues we recently communicated about, specifically related to the Mapper function. Schedule and Impact: The restart is scheduled to start in a few minutes. We anticipate a brief downtime during which the Exalate service will not be available. We aim to minimize this disruption and expect to restore full functionality promptly. Why This Is Necessary: This scheduled restart is a crucial step in implementing the fixes needed to enhance the stability and performance of Exalate. It will ensure that all recent updates are fully integrated and functioning as intended. Action Required: None from your end, the restart of each pod takes in average less than 2 minutes. A brief service pause is expected during said time but all sync events will resume afterwards. Looking Ahead: We are committed to continuously improving our service and preventing such issues in the future. Additional updates on our progress and any further actions that might be required on your part will be communicated. We apologize for any inconvenience this may cause and appreciate your understanding and cooperation. Should you have any questions or need further assistance, please do not hesitate to reach out to our support team via the Exalate Support Portal. Thank you for your patience and trust in us as we enhance our services. Best Regards, The Exalate Team

The Incident has been resolved. Thank you for your patience and understanding.

The problem is addressed, we are monitoring the results.

Status: In Progress Description: We are currently experiencing an outage affecting some of our customer nodes. This outage is a result of issues with our third-party hosting provider. Our team is actively investigating the issue and working closely with our provider to resolve it as quickly as possible. Impact: Customers may experience disruptions in service or connectivity. Next Update: We will provide another update within next 2 hours or as soon as more information becomes available. Thank you for your patience and understanding as we work to resolve this issue. If you have any urgent concerns or require immediate assistance, please don’t hesitate to reach out to our support team.

The solution has been applied successfully, the incident has been resolved. Please contact our support if you have any questions or comments.

The fix has been deployed on all affected instances, we are now monitoring for any errors.

Fix version is scheduled for deployment. Update: The fix for the reported issue is going to be deployed at 14:00 CET on all Github nodes. Action Required: No actions are required from users at this time. Thank you for your patience.

Fix version is undergoing final tests. Update: The fix for the reported issue is in testing. Current Status: The deployment of the fix is scheduled for later today. Action Required: No actions are required from users at this time. Thank you for your patience. Further updates are expected at 13:00 CET

Fix Development in Progress Update: The fix for the reported issue is ready and undergoing final preparations before deployment on all affected instances. Current Status: We're ensuring a smooth fix process across all affected nodes. Action Required: For urgent help, please create a ticket on our Support Portal. Our Support team is ready to assist you. Thank you for your patience. Further updates are expected at Feb 27, 9AM CET

Issue: Exalate for Github users may encounter errors that block outgoing synchronisation. Status: We have identified the root cause (there's recent change in the Github Webhook format affecting Exalate sync) and we are actively working on a solution to restore normal sync operations. Action Required: No actions are required from users at this time. We apologize for any inconvenience caused and appreciate your patience as we work to resolve this issue promptly. We will provide further updates as soon as the situation progresses. Thank you for your understanding.

This incident has been resolved. The Root Cause will be communicated later.

The Github Marketplace team has fixed the installation for all the apps. We're observing if there are any further issues.

Posted On: 27/02/2024 Severity: Under Investigation Dear Customers, We have identified an issue impacting the installation of Exalate for GitHub, which is currently preventing successful setup via the GitHub Marketplace. Our team is actively investigating this matter as it appears to be linked with processes within the GitHub Marketplace itself. What We Know: Incident Discovery Date: 27/02/2024 Issue Description: Installation failures for Exalate for GitHub through the GitHub Marketplace. Current Status: Our technical team is in the midst of diagnosing the root cause and contacting GitHub support to ensure a swift resolution. Immediate Recommendations: Patience and Monitoring: We kindly ask for your patience as we work towards resolving this issue. Updates will be provided as more information becomes available. Alternative Actions: Should you need urgent assistance or an interim solution, please reach out to our support team. We recognize the critical role Exalate plays in your workflow and are committed to restoring access as quickly as possible. Your understanding and patience during this time are greatly appreciated. Stay Updated: For real-time updates and further details, please keep an eye on this space or contact our customer support directly. We're here to help and ensure your experience remains as smooth as possible despite this hiccup. Your Feedback Matters: Should you have any concerns, suggestions, or need further assistance, do not hesitate to reach out. Your input is invaluable as we navigate this challenge together. Thank you for your understanding and cooperation as we work diligently to resolve this issue. We are committed to providing the high-quality service you expect and appreciate your patience and support. Warm Regards, The Exalate Team

Due to an intermittent issue on Atlassian, Exalate suffered from authentication issues when trying to access the Console or update issues. The issue has now been resolved and Exalate is operating normally.

Between 09:50 UTC to 10:04 CET, Atlassian experienced intermittent 503 errors for Jira software and Jira service management. It has now been resolved.

Incident Update: Inaccessible Exalate Console Date and Time: 20th of Feb 2024 Severity: High Dear Clients, We're pleased to announce the successful restart of our Exalate JCloud nodes. The required maintenance, prompted by the recent Atlassian incident, is now complete, and full service to the Exalate console has been restored. Quick Health Status: ✅ Service Normalization: Synchronization and other features are realigned back to operational standards. 🔄 Ongoing Observance: We're keenly monitoring the restored operations to ensure consistent service results. What Does This Mean for You? Your console and all features are now accessible and resuming as routinely managed. Here’s what we suggest: Initial Re-login: A recommendation to re-log into your dashboard and review your tasks and knowledge for a summarized recurence and flow. Verification of Duties: Properly ensure that there aren't any complexities or suspensions to your open or standby board of integrations. Should you encounter any problems reach out to our support at support@exalate.com. The Exalate team

Incident Report: Inaccessible Exalate Console Date and Time: 20th of Feb 2024 Severity: High Update: We are currently addressing a service disruption impacting the Exalate console, rendering it inaccessible to users. This issue stems from a token expiration problem, which is a direct result of an outage reported by Atlassian earlier today (Atlassian Status Page). This outage affects various services, including authentication mechanisms, leading to difficulties in accessing the Exalate console due to expired tokens. Impact: Users may encounter issues accessing their Exalate consoles. To mitigate these issues and ensure the stability and reliability of our synchronization services post-Atlassian downtime, we are required to perform a crucial restart of our Exalate JCloud nodes. This maintenance is expected to last approximately 2 hours, during which temporary disruptions in Exalate synchronization services may occur. Recommendations: Clear Cookies and Cache: If you are unable to access your Exalate console, please clear your browser's cookies and cache, specifically for Atlassian and Exalate sites. Raise a Support Ticket: If issues persist after following the above step, please raise a support ticket in our Support Portal, and our team will assist you further. Why is this necessary? This maintenance and recommended actions address the token expiration issue caused by the Atlassian outage, ensuring the security, reliability, and performance of our services. What we're doing: Our team is actively performing the restart of the nodes to ensure the refresh tokens are reloaded, this fix has been confirmed and is currently being applied to all pods affected. Next Update: We will provide continuous updates through our Status Page as we work towards resolving this incident. A notification will be sent out once the maintenance is complete and services are fully operational. We apologize for the inconvenience and appreciate your patience and understanding during this time. Thank you for your cooperation as we work diligently to maintain and improve the quality of our services. The Exalate Team

Incident Report: Inaccessible Exalate Console Date and Time: 20th of Feb 2024 Severity: High Description: We are currently experiencing a service disruption in the Exalate console, resulting in its inaccessibility to users. The root cause of this issue is attributed to a token expiration problem, directly related to an outage reported by Atlassian, earlier today (https://jira-software.status.atlassian.com/). This outage impacts various services, including authentication mechanisms. As a consequence, Exalate console users may encounter difficulties accessing their platforms due to expired tokens. Recommendation: 1. If you are unable to access your Exalate console, we recommend taking the following steps: Clear Cookies and Cache: Clear your browser’s cookies and cache, specifically for Atlassian and Exalate. 2. If you still experience issues after following these steps please raise a support ticket in our Support Portal and our team will assist you further. We apologize for any inconvenience this may cause and appreciate your patience as we work towards resolving this incident. The Exalate team

This incident has been resolved and the service is operating normally.

A fix has been implemented, Jira should have recovered. The Atlassian team is currently monitoring.

We are currently facing a critical outage that is impacting Jira Work Management, Jira Software, and Jira Service Management for customers using Exalate on Jira Cloud or connecting to a Jira Cloud instance. Users may experience synchronization issues with their data. The Atlassian team is actively investigating the issue to identify the root cause and working diligently to implement a resolution. Timeline: Incident Detected: 8:35 UTC Ongoing Mitigation: 09:49 UTC Current Status: Root cause identified and working on the fix Users using Jira Cloud or connecting to a Jira Cloud may encounter synchronization issues. The Exalate Service portal is currently unreachable, affecting the creation of support tickets. Please refer to the Atlassian status page for real-time updates. We understand the impact this may have on your operations and sincerely apologize for any inconvenience caused. We will continue to keep you informed of any developments. Thank you for your understanding.

Resolved: Database Incident Current Status: All nodes are now up and running smoothly. The database incident has been successfully resolved. Resolution Details: Our cloud team has diligently addressed the root cause of the database outage. Critical measures have been taken to ensure the stability and performance of the entire database infrastructure. Next Steps: Continuous monitoring and proactive measures will be in place to maintain optimal database performance. A comprehensive post-incident report will be prepared and shared with our users shortly. We appreciate your patience and understanding during the resolution of this incident. If you experience any lingering issues or have concerns, please reach out to our support team. Thank you for your cooperation. The Exalate team

All nodes are stable now - we'll be monitoring until tomorrow

The majority of the nodes is back to active. We are bringing the remaining nodes up.

Database recovery is in progress. We are bringing nodes up in a controlled way. Next update is expected at 16:00 CET

The solution is still under investigation. Next update is expected at 14:00 CET

Issue: Database Overload Current Status: We are currently undergoing operational changes to bring all nodes back online. Certain nodes are experiencing downtime . Operational Changes Details: In an effort to improve database performance and scalability, we are implementing critical operational changes. The changes are designed to optimize the infrastructure and enhance overall system reliability Next Steps: Following the completion of the changes, nodes will be brought back online progressively to minimize any service impact. Impact: SOME NODES DOWN: Temporary unavailability of services on affected nodes during the operational changes. Users may experience disruptions until the changes are successfully implemented. Contact: For urgent inquiries or additional information, please contact our support team. Next update is expected at 12:00 CET We apologize for any inconvenience caused and thank you for your patience and cooperation.

Brief downtime expected. Our team has identified database issues that require immediate attention to ensure the continued reliability and performance of our services. No action required from the customers.

No additional side effects have been detected during the day. We'll proceed with the postmortem and inform once that we reached conclusions.

All the affected nodes are up and running again. We are monitoring the situaiton

Nodes might be affected due to a misconfiguration in the database of one of the exalate clusters. Expect some downtime to recover from this situation

This incident has been resolved. Exalate version 5.7.4 containing the fix has been deployed on all customer instances.

Dear customers, We have successfully upgraded GitHub and Zendesk to the latest version, 5.7.4. Additionally, Jira nodes have also been updated to ensure compatibility. Tomorrow, we will proceed with upgrading the rest of the connectors. This update is crucial to maintaining seamless connections, as both sides need to be on the latest version for optimal performance. We appreciate your understanding and cooperation during this upgrade process. If you experience any issues, please don’t hesitate to reach out to our support team. Thank you for your continued trust in our services. The Exalate team

We are pleased to inform you that the resource overallocation issue has been successfully resolved. However, please be advised that some users may experience slowness in the processing of sync. Our dedicated team is actively investigating this matter to ensure a swift resolution. We appreciate your patience and understanding as we work towards restoring normal processing speeds.

We would like to extend our sincere apologies for any inconvenience caused by the recent service interruption experienced by some of our users. Outage Information: Our team has been actively working on resolving the issue that impacted a number of our customers. This issue led to the service interruption you may have experienced. Issue Resolution in Version 5.7.3: A solution has been implemented in our latest release, Exalate Version 5.7.3. This update not only addresses the resource overallocation issue but also includes key improvements to ensure the smooth and reliable functioning of our service. Next Steps: No Action is required from your end as the update has been deployed automatically, the consequences of the issue still can impact performance of the application which we are addressing as required. Your Patience is Appreciated: We understand the inconvenience this delay may have caused and appreciate your patience. We can assure you that nodes are actively being monitored to ensure uninterrupted transaction flow. Thank you for your understanding, and we will keep you updated on the progress. If you have any further questions or concerns, please don’t hesitate to reach out to our support and customer success teams.

We are currently experiencing an incident affecting a limited number of nodes in our system. The issue revolves around the Resource overallocation, resulting in downtime for the affected nodes. Current Situation: Affected Nodes: Approximately 5% of nodes are currently experiencing downtime. Recovery Attempts: Our team has been actively working on resolving the issue and has implemented various recovery measures. Actions Taken: Hot Fixes Released: We have deployed hotfixes on multiple platforms to address the Resource overallocation. The fixes have been applied to ServiceNow, Azure DevOps, Zendesk, Salesforce, and Jira Cloud (ongoing). Impact on Users: While the upgrades are in progress, customers may notice a temporary slowness in the processing of their sync. Ongoing Work: Node Restoration: Despite our efforts, 5% of nodes are still affected and remain offline. We are diligently working to bring these nodes back online. Sync Processing Investigation: Once all nodes are operational, we will conduct a thorough investigation into the processing of synchronization to ensure optimal performance. Github Hot Fix: A hotfix for Github is currently under testing. We aim to release it as soon as it passes the necessary testing procedures. We appreciate your patience and understanding during this time.

This incident has been resolved.

Deployment Status Update: Exalate for JCloud 5.7.2 The deployment of Exalate for JCloud 5.7.2 has been successfully completed. However, certain nodes are currently experiencing downtime as our team conducts essential duplicate trace clean-up. We are actively working to restore full functionality and appreciate your patience. Additionally, during this process, we have identified additional risks. To address these, a release (5.7.3) is planned and it is being tested, with the first deployment targeted for Jira connectors. We will keep you updated on our progress.

The fix is on validation stage and will start deploying on the nodes shortly. Your node will be updated during the day. We are committed to keeping you informed about the progress with developments. Stay tuned for the resolution.

We are currently working on a build, that encompasses the filtering of duplicates in sync events/requests and twin traces, along with the removal of duplicates during the upgrade. The build is currently undergoing internal testing for all cloud nodes.

The scheduled hotfix deployment for Zendesk and Github connectors has been postponed until further notice due to a database instability issue. We will keep you updated on any developments. Thank you for your comprehension.

Scheduled Hotfix: Zendesk & Github Connectors The hotfix for Zendesk and Github connectors is scheduled for 17:00 pm CET. We appreciate your understanding and patience as we work swiftly to enhance system stability.

Hotfix release completed for Jira Cloud based nodes. Other nodes will follow shortly.

Hotfix release started for Jira Cloud based nodes, ETA 6-8 hours from now. Other nodes will follow suit afterwards.

The deployment of the hotfix for the Java Heap error is expected to happen tonight at 22:00 CET for Jira cloud based nodes, other nodes will follow suit afterwards. Expect more updates shortly.

Service Status Update: A fix version is on validation stage and we expect a hotfix to address the Java Heap Space error release anytime now. No action is required from your end, we will keep you updated as soon as this is deployed. We apologize for any inconvenience caused and appreciate your patience.

We’ve detected an issue where a limited number of nodes are experiencing Java Heap Space errors, leading to node downtime. Despite various recovery attempts, some nodes remain affected. Impact on Service This issue mainly impacts nodes running version 5.7. Not all nodes are affected, but those with the issue may experience downtime. Current Actions Our engineering team is actively developing and testing a custom solution. We plan to first deploy this fix on a selected number of nodes to ensure stability. Updates and Communication We are committed to keeping you informed about the progress with developments. Stay tuned for further updates.

This incident has been resolved.

The fix has been applied and our Engineering and Cloud teams are monitoring the results. There are still a number of nodes where additional actions need to be taken from our side. We will keep you updated on any new developments.

A strategy has been formulated to address the repercussions of the database instability. Validation of this approach will be conducted throughout the morning, with node recovery scheduled for later today.

Current Status: Partial Service Disruption Issue Description: -Database services on Exalate cloud got into resourcing constraints as a spike in usage resulted in instabilities leading to a partial service disruption. Actions Taken: -Our team is actively investigating the root cause of the issue. -Immediate steps are being taken to mitigate the impact on possible affected nodes. Ongoing Mitigation: -We are working to extend the resources of the Exalate Cloud to alleviate the usage constraint. -Continuous monitoring is in place to ensure the stability of the affected nodes. We understand the inconvenience this may cause and appreciate your patience as we work to resolve the issue promptly. Further updates will be provided as we make progress towards a complete resolution. Thank you for your understanding

We are closing this incident although it is not fully resolved, a workaround is available. The incident is cluttering the statuspage and there is no additional value of having it here.

For those who want to have the full functionality of the bridge app available on their salesforce org, can follow the steps documented at https://docs.exalate.com/docs/known-issue-salesforce-installation-problems The security review is still ongoing. Salesforce is in the middle of reworking the way that apps can be submitted, so this can be a reason for slowdown.

For those attending this incident. Everything is ready for starting the security review of the package. Salesforce provided us a skip ahead pass, but the (unmodified) package still needs to get a stamp of approval.

Salesforce cannot fix the problem, and is asking us to resubmit the app.

We are still pushing Salesforce to take action - but this seems to be much more challenging than expected. The workaround has been validated in a customer environment - and the plan is to apply it in production. We'll provide a request type on our support portal for users who would like to apply the workaround.

A confirmation is received that the salesforce issue will be escalated internally. The workaround documentation will be ready by COB.

The request to Salesforce to relink the app is still outstanding, awaiting for replies. For the workaround, the upgrade path towards the final solution needs to be defined.

The workaround has been validated and we are now looking at how it can be deployed / requested for. We are still awaiting reaction from Salesforce. The next update can be expected later today, and then communication will resume on Monday.

We are still trying to get this issue be addressed by Salesforce at a higher priority. Next to that, we are working on a workaround which is being validated atm. More information will be provided in a next post.

The issue has been escalated to Salesforce - the support team reported that the request to relink the app is currently being reviewed by the internal team Salesforce partner team has been engaged to bump the priority. We can use help to increase the priority - if you can raise a salesforce case, we can make the point that this is not an isolated defect by referring to the case. If affected by the installation problem, please raise a ticket on our support portal https://support.idalko.com/servicedesk/customer/portal/8 With a reference to the salesforce case you created. Thanks

Issue identified and root cause known.

Reported problems with installation of Exalate for Salesforce bridge. The problem has been identified as an infrastructure issue on Salesforce side. The root cause has been located, we expect Salesforce to take action. We will inform when the solution is ready by updating the Status Page and via support tickets. Thank you for your patience and understanding.

\(All times UTC\+1\) # **Lead-up** On Tuesday, May 23rd at 9:21 A.M., a series of database restart events occurred, leading to the outage of all nodes in the system.  ‌ # **Fault** The failure generated a connectivity problem between the nodes and the database. Nodes couldn’t read and write the data. This root cause has been identified and backlogged as a critical bug Whenever a node tries to read from or write to the database - the operation got stuck. The consequence was that the HTTP request related to the read/write operation also got stuck, building up HTTP request queues on the reverse proxy infrastructure. # **Impact** The impact is large and affected the whole infrastructure * The nodes could not read/write transactions anymore missing on sync events and change events * The reverse proxy infrastructure got stuck affecting the monitoring capabilities and the automatic escalation * Because the escalation failed to work, support engineers on the call were not notified # **Detection** May 23rd at 9:21 AM: * The database unexpectedly restarted, causing all nodes to go down simultaneously. May 23rd at 10:35 AM: * A critical ticket was raised with Google Cloud to report the issue and seek assistance. May 23rd at 11:00 AM: * A call was initiated with the Google Cloud team to provide real-time updates and discuss the ongoing issue. May 23rd at 11:25 AM: * To address the situation, a decision was made to double the allocated resources, increasing the capacity of the affected system. May 23rd at 12:02 PM: * Google Cloud confirmed that the root cause of the database restarts was due to over-usage of allocated resources during the pick load May 23rd at 12:19 PM: * After increasing resources and implementing the necessary adjustments, all nodes were brought back online, resolving the outage. # **Response** All individual support tickets are answered. A status page opened on May 23rd at 10:09 AM # **Recovery** To solve the problem, the amount of allocated resources was increased and all necessary adjustments were done # **Root cause** There are multiple root causes. * The product must be able to survive flaky database connections. * Automated resource scaling protocol should be adjusted to cover such cases ‌ # **Lessons Learned** * Resource monitoring: Implement robust resource monitoring and alerting mechanisms to proactively identify potential over-usage scenarios * A big portion of our customer base is not aware of the status page ‌ # **Corrective Actions** * Fix the product bug and ensure that failing database actions can not clog up the HTTP request processing engine * Make the monitoring paths more robust, and start to monitor the monitoring services * Adjust the automated resource scaling protocol ‌ ‌ If you have any questions on this postmortem - please raise a request on our support portal \([here](https://support.idalko.com/servicedesk/customer/portal/8)\)

RCA and Postmortem to follow

During the outage, some sync events could be missed. To recover the missed sync events, please follow the next steps: 1. Create a search query that would contain the existing trigger query (if available) + the update date being between "2023-05-23 9:21" and time of the outage resolution "2023-05-23 12:23" 2. Find how many issues were affected. 3. Split the results into batches of 100 (500 for EPSO customers) by limiting the query results by issue key numbers being between given bounds or ticket creation date being between given bounds. 4. For every batch create a disabled trigger. 5. Run Bulk Exalate on one of the triggers, as soon as the sync queue is empty, repeat the step for other disabled triggers from step 4.

The fix has been deployed and the infrastructure has been restarted. We are monitoring the infrastructure.

The database is restarted.

A critical issue Has been raised with Google cloud.

Exalate infrastructure experienced a DB restart. The reason of the restart is under investigation. We are applying a fix on all Exalate instances. All Exalate nodes will be down during this process.

This incident has been resolved.

The documentation site is sporadically unavailable, we are currently monitoring the issue

This incident has been resolved.

We are currently investigating this issue.

The Exalate Doc site is back up.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

The Exalate Docs website is down due to third-party hosting platform error. We are monitoring the situation and will provide updates accordingly.

This incident is resolved. The fix has been deployed to all customer nodes. No further action required.

The development team has a positive progress in the resolution of the incident. We are working on platform stabilisation before deploying the permanent fix.

The problem has been identified and the fix is in testing. After the tests are passed the solution will be applied to all the unreachable Exalate instances. No action required from the users.

Exalate service can be unavailable for some cloud-based environments. We identified errors related to memory usage and currently working on the resolution with the highest priority. In case you are experiencing issues accessing Exalate on your instance, feel free to contact Exalate Support.

The new version has been deployed on all customer nodes.

The fix is currently under testing.

The issue has been identified, the fix is developed and is going through code review.

A number of Salesforce nodes are offline due to update error. The development team is working on a permanent fix. If you experience outage, please contact Exalate Support.

The update with fix has been deployed to all cloud nodes.

The permanent fix solution passed the QA tests and is going to be released in Exalate v5.4.8. The deployment of the new version on the customers' environments will start on Tuesday December 6 and end on Wednesday December 7 CET end of day. During the deployment time all nodes will be operational. It is expected that the node will restart and the sync will interrupt for about 1 minute. After the restart the sync will resume from the point it stopped. No actions required from the customers side.

Workaround has been applied. Permanent solution under testing.

The fix is implemented and currently under testing

The issue has been identified, workarounds provided to the customers upon Support request. The permanent solution is under development.

Customers are reporting problems with Access Token, such as 401 errors and update Token in general settings.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

change_request updates can be missed on entries which have been exalated manually (with the entity sync status) after July 11. A workaround and fix is being prepared.

Certificate resolved

Awaiting validation of the csr

SSL certificate expired for the main production endpoint

This incident has been resolved.

We are continuing to investigate this issue.

WORKAROUND: If you experience problems accessing Exalate from Azure DevOps, please try to access the console using Exalate Node URL (you can find it in the Verification email)

The number of requests in the queue for reinstallation didn't appear anymore

This needs to be fixed on the Atlassian side, in the mean while - when encountering the problem - please raise an issue in our support portal https://support.idalko.com/servicedesk/customer/portal/8

The problem is identified, however, only a work-around may be applied, since there's no automated way of triggering a re-installation. Please, contact our support https://support.idalko.com/plugins/servlet/desk/portal/8 if you see the following symptoms: * can't access Exalate interface in Jira Cloud * receive error notifications suggesting, that Exalate "Could not authenticate to this Jira Cloud ..."

We can see a number of instancing experienced a problem in the installation process, and were unable to update their security credentials. Please, contact our support https://support.idalko.com/plugins/servlet/desk/portal/8 if you see the following symptoms: * can't access Exalate interface in Jira Cloud * receive error notifications suggesting, that Exalate "Could not authenticate to this Jira Cloud ..."

This incident has been resolved.

The fix is available as a release candidate for Jira On Premise (server and data center)

We are continuing to investigate this issue.

Synchronisation queues are building up in case of private/public connection On the public side - many transactions are stuck in 'WAITING_REQUEST_RECEIVED' On the private side - many transactions are stuck in 'NOTIFY_REQUEST_RECEIVED' The impact is that the sync is stuck.

This incident has been resolved.

The root cause was found, no product change is necessary.

Setup a connection to a single repository. See that any issue gets synchronized.

We could not reproduce the case until now. Putting it as resolved for now.

Existing connections started to update random fields when processing incoming syncs.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

A fix is ready and can be deployed on demand (just reach out to our support) In the mean time it is getting ready for a patch release.

We are receiving reports of user synchronization failing in jira cloud. We believe the search by email does not give the expected results. Expect an update soon.

No reports coming in anymore.

The issue is resolved on it's own. The validations started to pass. We're monitoring it.

We are investigating the missing security context.

\(All times UTC\+1\) # Lead-up On Saturday Nov 13, 10:00 PM - the central database - hosting all the data of all the nodes - was upgraded to a new version. This is an automated process and performed regularly and unattended. During this failover - services are being failed over to another instance to guarantee continuity. # Fault The fail over to the new instance generated a connectivity problem between the nodes and the database. Nodes could still read the data but not write data. This root cause has been identified and backlogged as a critical bug Whenever a node tries to write to the database - the write operation got stuck. The consequence was that the http request related to the write operation also got stuck, building up http request queues on the reverse proxy infrastructure. The monitoring which is using the same proxy infrastructure to check on the health of the nodes also got stuck, leading to a failure of the monitoring infrastructure itself. # Impact The impact is large and affected the whole infrastructure * The nodes could not write transactions anymore missing on sync events and change events * The reverse proxy infrastructure got stuck affecting the monitoring capabilities and the automatic escalation * Because the escalation failed to work, support engineers on call were not notified. # Detection Customers started to notify about stuck syncs and availability problems from Sunday 3:00 am \(by email\), which got amplified during the day. It was not possible to interprete the individual incident notifications from an infrastructure wide incident. Early Monday morning - support tickets started to be raised. # Response All individual support tickets are answered. A status page opened at 8:00 am CET. # Recovery To solve the problem, all the nodes needed to be restarted, which was scheduled and took around 8 hours to complete. During the restart additional infrastructure problems were detected and resolved - leading to additional restarts. # Root cause There are multiple root cause. * The product must be able to survive flaky database connections. * The escalation path was not robust enough to ensure that the team was aware about the impact and severity of the problem ‌ # Lessons Learned * Whenever there are infrastructure upgrades, support team needs to be aware and validate that everything is still working correctly * A big portion of our customer base is not aware of status page ‌ # Corrective Actions * Fix the product bug and ensure that failing database actions can not clog up the http request processing engine * Make the monitoring paths more robust, and start to monitor the monitoring services * Ensure an on call service during the infrastructure component upgrades ‌ ‌ ‌ If you have any questions on this postmortem - please raise a request on our support portal \([here](https://support.idalko.com/servicedesk/customer/portal/8)\)

This incident has been resolved.

All exalate nodes are back online. We will continue to monitor the situation to make sure everything is ok.

Internal connectivity has been restored, exalate nodes are being restarted individually to take into account the new configuration. We'll keep you posted around the progress ...

This issue is taken at the highest priority - internal connectivity issues are degrading the functionality of a number of exalate nodes. We'll increase our reporting on this channel to keep everyone informed

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results

On latest release we have opt-in for assymetric authentication during installation and our assumption is that atlassian-connect has not been updated yet on customers instances. We opened an atlassian support ticket and as a counter measurement we will be releasing a hot-fix where in case assymetric authentication fails we still try with symmetric authentication using shared secret until issue gets resolved

We are currently investigating this issue

This incident has been resolved.

The service is back.

The nodes get auto restarted automatically

We'd released HOTFIX releases for ZENDESK, JIRA CLOUD, AZURE DEV OPS and SERVICENOW in the course of Yesterday to remedy that. The problem turned out to be logging tapping into attachment download requests effectively consuming the entire request stream.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

Working on the fix

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

The problem has been root caused, and the fix is being deployed

We are currently investigating this issue.

This incident has been resolved.

The nodes are slower starting up - the upgrade will take more time then scheduled

All nodes have been stable for more than 24 hours. Closing the incident

There are still nodes having database access problems.

All nodes are up and running - monitoring the situation.

We are continuing to work on a fix for this issue.

Google cloud (the underlying infrastructure) had a major outage https://status.cloud.google.com/incident/cloud-networking/21002 It is resolved but the exalate cloud service is now recovering.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

Exalate and Table Grid documentation portals are down. The issue has been identified and is being worked on.

Database restored.

We are investigating the issue

servcie up again

Host is down ... recovery on its way

This incident has been resolved.

Documentation portal (https://docs.idalko.com/) is currently down. The issue is being investigated.

Nodes have been stable for 12 hrs

We are continuing to investigate this issue.

We are currently investigating the issue

Upgrade completed

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.