Templafy

Is Templafy Down Right Now? Check if there is a current outage ongoing.

Templafy is currently Operational

Last checked from Templafy's official status page

Historical record of incidents for Templafy

Report: "Service degradation - Unable to Save and Share"

Last update
investigating

We are currently investigating this issue.

Report: "Service degradation - Issues accessing Templafy One"

Last update
postmortem

Incident Initiation At 8:29 PM CET on April 21, 2025, an issue was identified impacting availability of our Templafy One application hosted on Azure App Services in the North Europe region. The issue resulted in connectivity drops for end users, causing temporary disruptions in service. The incident was attributed to unplanned maintenance initiated by Azure without prior notice. Investigation Upon noticing the service disruptions, the engineering team began monitoring the application and associated Azure resources. Logs and metrics indicated availability drops aligning with the start time of the incident. Further investigation confirmed that the root cause was an unplanned hotfix applied by Azure App Services, as communicated in an official Microsoft Azure service notification \(Tracking ID 1VZN-PBG\). The maintenance activity began shortly after 7:00 PM CET on April 21, 2025 and was performed during non-business hours. Mitigation and Resolution No action was required from our side as the issue was caused by Azure infrastructure maintenance. The event concluded at 10:58 PM CET on April 21, 2025, with all services stabilizing post-maintenance. The application automatically recovered once the underlying Azure App Services instances resumed normal operation. Impact and Scope The incident was limited to Templafy One. User Experience: Users may have experienced temporary unavailability or connectivity interruptions during the maintenance window between 8:29 PM and 10:58 PM CET.

resolved

The incident has been resolved, and all Azure services are now operating normally. A detailed postmortem will be shared shortly. We apologize for the disruption and appreciate your patience.

monitoring

The incident has been successfully mitigated as a result of the Azure services coming back online, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects all Templafy One customers. The first investigation points to specific Azure Services being degraded and this is causing Templafy One to be down. We are working with Microsoft towards a resolution. Further updates will be posted here soon.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service Degradation: Dynamic Slide Insertion Issues in PowerPoint in West Europe (Production 0)"

Last update
postmortem

Incident Initiation At 10:22 AM CET on April 4, 2025, an issue was identified affecting the dynamic slide insertion feature in PowerPoint. Certain dynamic assets could not be inserted into documents, obstructing users in their daily work. The issue was officially detected at 3:00 PM CET. It was determined that the problem impacted a subset of customers within the West Europe \(Production 0\) environment. Investigation On April 7, 2025, at 11:11 AM CET, engineers began investigating the incident. The initial focus was to assess whether other production clusters were impacted. By 12:02 PM CET, engineers confirmed that the issue was limited to the West Europe \(Production 0\) environment. Investigation identified that allowing users to access the preview page by passing the asset ID in the URL inadvertently caused some dynamic assets to fail to be inserted into documents. Mitigation and Resolution As a mitigation effort, engineers reverted the behavior that allowed accessing the preview page by passing the asset ID in the URL. The change was deployed to the QA environment at 12:13 PM CET. Following successful validation, the fix was deployed to the West Europe \(Production 0\) environment at 12:22 PM CET, restoring dynamic slide insertion functionality. Impact and Scope Scope: The incident was limited to a subset of customers on West Europe \(Production 0\) cluster who were using the dynamic slide insertion feature in PowerPoint. User Experience: Users were unable to insert dynamic slides into documents, impacting their daily workflows. Post-Incident Actions To prevent similar incidents in the future, the engineering team will enhance validation procedures for URL-based asset access features and improve regression testing coverage for dynamic asset insertion functionalities.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The engineering team has deployed a fix and confirmed that the issue, which impacted only a subset of customers in West Europe (Production0), has been successfully mitigated. We are actively monitoring the situation to ensure ongoing stability and performance and are closely observing the systems to prevent any further disruptions.

identified

We have identified an issue in the Dynamic slide insertion that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Feature flag issues"

Last update
postmortem

Incident Initiation At 9:26 AM CET on March 31, 2025, engineers detected an issue affecting global feature flag functionality. Certain features controlled by feature flags, such as the collapsible task pane, became temporarily unavailable for a small subset of users in West Europe \(Production 0\). Investigation Between 9:30 AM and 10:10 AM, engineers conducted an investigation to identify the root cause. It was determined that the newly deployed Foundation package version contained a bug that disrupted global feature flag functionality. The impact was confined to West Europe \(Production 0\) systems that had upgraded to this version. Mitigation and Resolution At 10:10 AM, engineers initiated mitigation efforts by rolling back the faulty Foundation package version to the previous stable release. This rollback was applied across all affected services. At 11:21 AM, the final fix was successfully deployed to production, restoring feature flag functionality. To prevent further impact, production releases were temporarily blocked until a full fix was delivered across all systems. Impact and Scope Scope: The incident was limited to a subset of customer on our West Europe \(Production 0\) cluster who were using certain feature flags. User Experience: Feature flag-controlled functionalities, including the collapsible task pane, were temporarily unavailable. Post-Incident Actions To prevent similar incidents in the future, the engineering team will review the Foundation package update process and strengthen automated testing for feature flag dependencies before deployment.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

A limited number of customers in our West Europe (Production 0) cluster are experiencing issues where feature flags are not enabling the intended functionality. Further updates will be posted here soon.

Report: "Service degradation - Users encounter issues when accessing West Europe (Production 1)"

Last update
postmortem

**Incident Initiation** On February 13, 2025, at 1:27 AM CET, an issue was introduced which lead to excessive CPU utilization due to heavy database usage in West Europe \(Production 1\). The issue remained undetected until February 13, 2025, at 1:18 PM CET, when monitoring systems flagged abnormal resource consumption. The impact was significant, affecting multiple tenants and a large number of users. The degraded performance resulted in a high rate of exceptions/errors in system logs and hindered application functionality. **Investigation** The engineering team initiated an immediate investigation on February 13, 2025, at 1:36 PM CET. They hypothesized that the issue was related to an inefficient database query or workload increase. Further analysis confirmed that the heavy database usage was the root cause of CPU maxing out, leading to performance degradation. **Mitigation and Resolution** To mitigate the incident, the engineering team promptly scaled up the database resources at 1:40 PM CET to stabilize the application. Continuous monitoring was implemented to track system performance and ensure stability. By 2:29 PM CET, the application was stabilized, and error rates significantly reduced. The team planned to revert the database resources to normal levels by the following morning to ensure optimal operation. **Impact and Scope** The incident impacted multiple tenants across various clusters, leading to performance degradation for affected users. The issue was widespread, affecting application responsiveness and generating increased system errors. **Post-Incident Actions** In response to this incident, the engineering team will implement several post-incident actions, including a detailed review of database query efficiency and workload distribution. Additional procedural improvements will be made to monitor resource consumption more closely and introduce alerting mechanisms for early anomaly detection. We sincerely apologize for the disruption caused by this incident. Our engineering team is committed to ensuring service reliability and stability. We appreciate our customers' patience and understanding as we continue to enhance our monitoring and mitigation processes.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Email signatures are not being inserted or are taking a very long time to insert via signature Web add-in"

Last update
postmortem

**Incident Initiation:** Starting from 17th February, we slowly started to receive increasing reports from customers regarding an issue related to the slowness and inability to automatically insert email signatures through the Email Signature Outlook Web Add-in in Outlook Desktop, particularly in the Classic or Old version. As a result, an incident was formally created on February 19, 2025, at 10:21 AM CET. This issue disrupted the daily operations of multiple customers, preventing the proper functionality of the signature insertion feature. **Investigation:** The engineering team initiated an investigation into the issue. Initial analysis revealed that an upgrade to the Office JS version had been applied to the Classic or Old version of Outlook Desktop. This upgrade led to an increased number of exceptions, which sometimes resulted in slower signature insertion or, in some cases, prevented the insertion process from completing successfully. Since the exception is triggered by an external system \(Office JS\) to which we do not have direct access, further root cause analysis and resolution are limited. **Mitigation and Resolution:** To mitigate the impact of the issue, a workaround was provided to affected customers on February 19, 2025, at 04:48 PM CET. The workaround recommended switching to the New Outlook or Outlook Online, which resolved the problem. Additionally, to further assist with support and troubleshooting, a [dedicated guide](https://support.templafy.com/hc/en-us/articles/25268161740317-Automatic-signature-insertion-fails-in-Classic-Outlook#h_01JMF4Q7JJ2R0EGVK3RBN6C9BJ) for using the Email Signature Web Add-in in the Classic version of Outlook Desktop was created. Given that the issue originates from an external system, we are unable to fully prevent its occurrence. The product's functionality remains dependent on the external Office JS system. **Impact and Scope:** The incident affected multiple customers, only those using the Classic or Old version of Outlook. Users experienced delays in signature insertion or the failure of signature insertion altogether. This issue impacted a significant number of users across all production environments. **Post-Incident Actions:** Moving forward, we will be investigating on how we can collaborate more closely with Microsoft’s Outlook or Office JS teams to expedite the identification of the root cause. We sincerely apologize for the disruption caused by this incident and want to reassure our customers that we remain committed to providing reliable and stable services. Our team will continue to monitor the situation and explore ways to prevent similar issues in the future.

resolved

Email signatures are not loading for users of `Old Outlook` Desktop version. The issue is happening due to a bug in a Microsoft component. As a workaround, users can switch to `New Outlook` or `Outlook Online`. More details will be provided in Postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Template Designer fails to open"

Last update
postmortem

**Incident Initiation:** On December 3, 2024, at 2:03 PM CET, a defect was introduced in the West Europe \(Production 0\) environment of our Hive platform, specifically affecting the Content Library functionality. After users started reporting that templates from the Content Library were not opening, and an incident was created at 5:22 PM CET. This issue obstructed users' daily work, affecting multiple customers. **Investigation:** The engineering team initiated an investigation at 5:30 PM CET on December 3, 2024. Initial analysis indicated an intermittent error loading the VSTO Add-in when creating a document. Further investigation revealed that the root cause was a race condition during the loading of the VSTO Add-in. The VSTO Add-in was failing to load because it could not establish a bridge with the Web-App, the connection needed for the VSTO Add-in to communicate with the Web-App. This issue was linked to a change in the order of loading dependencies following an upgrade of a frontend component responsible for managing dependencies during the application startup. The VSTO Add-in incorrectly assumed that shared dependencies were being preloaded in a specific order. **Mitigation and Resolution:** To mitigate the incident, a workaround was provided to customers at December 3, at 8:31 PM CET, and further deployments were halted to limit additional impact. On December 4, at 8:15 AM CET, the engineering team continued working on a permanent solution. The engineering team started identifying the proper sequence for loading dependencies, ensuring that the function in the Web-App required by VSTO Add-in was correctly registered. Since at the time of halting the deployment, it was not clear which change was the root cause of the issue, a deployment of another system spread the issue to all production clusters, West Europe \(Production 1\), East US \(Production 2\), Australia East \(Production 3\), Canada East \(Production 4\) and West Europe \(Production 5\). At 2:07 PM CET, the engineering team had implemented the permanent solution and deployed this, restoring the full functionality to the Content Library.  At 8:39 PM CET, we discovered that the change in the loading behavior for dependencies also caused an issue in another component, the Template Designer. There had already been made a solution for this that was deployed to West Europe \(Production 0\). The engineering team quickly deployed the same solution to all other environments as well, fully resolving the issue at 9:34 PM CET. **Impact and Scope:** The incident affected multiple customers, obstructing users from opening templates from the Content Library and at a later time obstructing admins with using the Template Designer, which caused disruptions to their daily activities. The incident impacted a significant number of users in all Production environments. **Post-Incident Actions:** The engineering team will conduct a thorough review of the dependency loading processes and implement stricter testing and monitoring to detect issues in the future. The upgrade procedures for the frontend component for dependency management will be revised to prevent similar issues from occurring, ensuring that updates to our frontend component maintain compatibility with the VSTO Add-in. Additionally, changes to our internal processes and tools will be made to ensure halting faulty deployments will be more consistent, reducing the chance of similar errors occurring in the future. This incident was related to a previous incident, which can be reviewed here: [Incident Details](https://status.templafy.com/incidents/l738krky8h62). For the complete timeline, please refer to the original incident. We sincerely apologize for the disruption this incident caused and reaffirm our commitment to providing reliable and stable services for our users.

resolved

The issue was identified, and a fix was deployed to all environments. Further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

investigating

Template Designer won't open, and this is likely affecting admins for East US (Production 2).

Report: "Content Library Asset Insertion and Document Creation Actions Periodically Fail"

Last update
postmortem

**Incident Initiation:** On December 3, 2024, at 2:03 PM CET, a defect was introduced in the West Europe \(Production 0\) environment of our Hive platform, specifically affecting the Content Library functionality. After users started reporting that templates from the Content Library were not opening, and an incident was created at 5:22 PM CET. This issue obstructed users' daily work, affecting multiple customers. **Investigation:** The engineering team initiated an investigation at 5:30 PM CET on December 3, 2024. Initial analysis indicated an intermittent error loading the VSTO Add-in when creating a document. Further investigation revealed that the root cause was a race condition during the loading of the VSTO Add-in. The VSTO Add-in was failing to load because it could not establish a bridge with the Web-App, the connection needed for the VSTO Add-in to communicate with the Web-App. This issue was linked to a change in the order of loading dependencies following an upgrade of a frontend component responsible for managing dependencies during the application startup. The VSTO Add-in incorrectly assumed that shared dependencies were being preloaded in a specific order. **Mitigation and Resolution:** To mitigate the incident, a workaround was provided to customers at December 3, at 8:31 PM CET, and further deployments were halted to limit additional impact. On December 4, at 8:15 AM CET, the engineering team continued working on a permanent solution. The engineering team started identifying the proper sequence for loading dependencies, ensuring that the function in the Web-App required by VSTO Add-in was correctly registered. Since at the time of halting the deployment, it was not clear which change was the root cause of the issue, a deployment of another system spread the issue to all production clusters, West Europe \(Production 1\), East US \(Production 2\), Australia East \(Production 3\), Canada East \(Production 4\) and West Europe \(Production 5\). At 2:07 PM CET, the engineering team had implemented the permanent solution and deployed this, restoring the full functionality to the Content Library.  At 8:39 PM CET, we discovered that the change in the loading behavior for dependencies also caused an issue in another component, the Template Designer. There had already been made a solution for this that was deployed to West Europe \(Production 0\). The engineering team quickly deployed the same solution to all other environments as well, fully resolving the issue at 9:34 PM CET. **Impact and Scope:** The incident affected multiple customers, obstructing users from opening templates from the Content Library and at a later time obstructing admins with using the Template Designer, which caused disruptions to their daily activities. The incident impacted a significant number of users in all Production environments. **Post-Incident Actions:** The engineering team will conduct a thorough review of the dependency loading processes and implement stricter testing and monitoring to detect issues in the future. The upgrade procedures for the frontend component for dependency management will be revised to prevent similar issues from occurring, ensuring that updates to our frontend component maintain compatibility with the VSTO Add-in. Additionally, changes to our internal processes and tools will be made to ensure halting faulty deployments will be more consistent, reducing the chance of similar errors occurring in the future. We sincerely apologize for the disruption this incident caused and reaffirm our commitment to providing reliable and stable services for our users.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We are actively deploying a potential fix and continuing to investigate the root cause of the issue. While we anticipate this may provide partial mitigation, our team is committed to achieving a full resolution as quickly as possible.

identified

Last night, we identified an issue affecting our West Europe (Production 0) environment and believed it was resolved using a workaround, moving the status to monitoring. However, as of this morning, we have observed that the issue has propagated to additional production environments starting 7:48AM CET. As a result, we have updated the status back to identified and are actively working on resolving the broader impact. Thank you for your patience.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Some users cannot access Templafy for East US (Production 2) region"

Last update
postmortem

After investigation, we determined that a misconfiguration of authentication methods for two tenants caused issues for these tenants.

resolved

After investigation, we determined that a misconfiguration of authentication methods for two tenants caused issues for these tenants.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Web Add-ins login ends up in error page"

Last update
postmortem

**Incident Initiation:** On October 30, 2024, at 11:30 AM CET, an issue was introduced in the Hive environment, specifically affecting the login process for web add-ins. The issue was detected on October 31, 2024, at 10:00 PM CET, when users reported being redirected to an error page during login attempts. This disruption hindered users from accessing essential functionality, obstructing their daily work and resulting in a surge of support queries and a notable increase in error rates in system logs. **Investigation:** The engineering team initiated an immediate investigation at November 1, 2024, at 1:30 PM CET. They hypothesized that the issue was related to a recent deployment of a library update, which had potentially caused the authentication failure in web add-ins. Further investigation confirmed this hypothesis, identifying the update as the root cause of the login issues. **Mitigation and Resolution:** To mitigate the incident, the engineering team promptly reversed the library update that triggered the authentication failure. After deploying the fix on the testing environment, rigorous testing was conducted, which successfully resolved the issue. The team then proceeded to deploy the fix across all production clusters. By reverting the library update, the login process was restored, enabling users to access web add-ins normally. **Impact and Scope:** The incident impacted multiple tenants across various clusters, leading to significant workflow disruptions for affected users. The error affected all users attempting to log in to web add-ins during the incident window and increased support requests as a direct consequence. **Post-Incident Actions:** In response to this incident, the engineering team will implement several post-incident actions, including a detailed review of update processes to prevent future authentication-related disruptions. Additional procedural improvements will be made to monitor critical dependencies more closely, particularly within the authentication workflows.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating the issue. If you experience the issue, you can close Templafy task pane and reopen to be able using Templafy.

Report: "Service degradation - Third party Add-ins packages failed to download"

Last update
postmortem

**Incident Initiation** To improve the automation and security of package management, the engineering team implemented a series of changes. These updates included expanding the status options for each package and involved a database migration. However, the changes unintentionally caused the endpoint responsible for retrieving current packages for users to incorrectly exclude third-party packages. Consequently, these packages were removed from users' computers during the next Templafy Desktop run, making the associated Add-ins unavailable in Office applications. This change was deployed across all environments at 8:20 AM CET on October 23, 2024, though it only started impacting users after the Templafy Desktop component triggered a package update. The issue was first reported at 11:44 AM CET. **Investigation** The incident was placed at 12:10 PM CET, when the software engineering team was notified. The root cause was quickly identified and the engineering team immediately started implementing a fix to ensure the endpoint responsible for retrieving Add-ins correctly included third-party packages. **Impact and Scope** The incident impacted all regions but affected only specific tenants with third-party Add-ins installed. Among these, only users whose Templafy Desktop triggered a package update within the incident timeframe were affected. **Resolution** By 3:14 PM CET, the code fix was successfully deployed across all production environments. This update allowed the Templafy Desktop component to retrieve and reinstall third-party Add-ins, enabling users to access them again following a Templafy Desktop synchronization. **Post-Incident Actions** The engineering team has commenced an extensive review to expand test cases for package delivery, activation, and security assertion, ensuring future enhancements to package functionality do not impact the integrity and stability of existing configurations.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. The third-party Add-ins will be reactivated during the next synchronization of our Templafy Desktop component.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Manually configured claims in SAML2 appear to no longer be coming to User Profiles in Templafy"

Last update
postmortem

Incident Initiation: The issue started on September 20, 2024, at 10:54 AM CET but was detected on September 23, 2024, at 6:02 PM CET. On September 24, 2024, at 2:37 PM CET, the incident response process was initiated after users reported that manually configured User claims in the SAML2 authentication protocol within the Hive environment were no longer being recognized. The issue caused users to lose their User Claims when logging in via SAML2 authentication, due to a mismatch between customer configurations and the system's case-sensitive claims checks. Investigation: The issue affected only a subset of customers, making it difficult to reproduce and investigate at first. Engineers closely examined the system and identified that the problem was caused by an external package update that implicitly upgraded the "Microsoft.IdentityModel.\*" packages from version 7.x to 8.x. This update introduced case sensitivity in claims checking, which conflicted with how some customers had configured their SAML2 claims. Once the root cause was pinpointed, the team began working on a solution to mitigate the issue. Resolution: After identifying the root cause, the configuration was adjusted to account for the case-sensitive claims checks. Normal functionality was restored, and users could resume all affected operations, resolving the incident on September 24, 2024, at 10:31 PM CET. Post-Incident Actions: Moving forward, we will focus on improving our monitoring and testing processes for package updates. Closer collaboration with customers and better detection mechanisms will help us identify and resolve similar issues more efficiently.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Campaign images are deleted when uploaded in Edit mode"

Last update
postmortem

Incident Initiation: On September 4, 2024, at 11:59 AM CET, an anomaly was detected in the internal component of Hive, which initiated an incident. The issue specifically affected the campaign designer, causing newly uploaded images to disappear for users. Investigation: The engineering team immediately began investigating the issue. It was discovered that the newly uploaded images were not being transferred to long-term storage due to a malfunction in the storage service. The root cause was identified as a misconfiguration introduced during a recent update to the image handling system. Mitigation and Resolution: A temporary mitigation was put in place to ensure that images uploaded after the detection of the issue were properly stored. Following this, a correction was developed and deployed at 01:44 PM CET to restore the application's functionality. This fix addressed the underlying configuration error, and the system returned to normal operation. Impact and Scope: The incident impact was on across all regions and affected only certain tenants with a specific configuration attempting to upload images during the incident window. Post-Incident Actions: Procedural improvements will be implemented, including enhanced monitoring of storage services. The engineering team is also reviewing the update process to prevent similar misconfigurations.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Previews not generating in Templafy One"

Last update
postmortem

On August 23, 2024, at 09:37 AM CET an incident was started due to previews not generating in the Templafy One environment. Engineers started investigating the issue and discovered that the number of messages exceeded the capacity of the preview processing system that was initially configured. After adjusting the configuration, the messages began processing correctly again. This action restored normal function to the application allowing users to generate previews, resolving the incident at 10:07 AM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Email signature synchronizations taking longer then usual"

Last update
postmortem

On August 14, 2024, at 1:56 PM CET, an incident was initiated when the email signature synchronization process took longer than usual to complete, causing delays in distributing email signature updates to end users. At 4:02 PM CET, the team reverted to a previous version of email signature synchronization process, confirmed that the synchronization was running within the desired parameters, and deployed it to all production environments. The team continued to monitor system performance and, after ensuring everything was operating at the desired performance metrics, marked the incident as resolved at 4:42 PM CET. The engineering team will investigate to identify the root cause of the issues in the email signature synchronization process. Additionally, they are focusing on further quality automation and performance checks to prevent similar incidents in the future.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - A subset of customers can potentially experience issues while changing space ownership"

Last update
postmortem

On July 25, 2024, at 10:31 AM CET, an incident was initiated due to a problem where admins were having problems using the space administration functionality. This issue was specifically related to the assignment of space owner access flow, which could potentially remove all space owners if a user attempted to remove a subset of the owners. The engineering team initiated an investigation upon the incident creation and identified that the issue was caused by a newly created UI component. After isolating the problem the team then moved into the mitigation phase and worked diligently to integrate a solution across all impacted subsystems. At 12:56 CET they managed to deploy the fix to all the production environments. The issue was confirmed to be fixed and management operations for spaces was restored. We apologize for the disruption and remain committed to ensuring the reliability and stability of our services.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. The issue occurs when admins attempt to change space ownership. The memembership change could potentially have unexpected results. Further updates will be posted here soon.

Report: "Service degradation - Email signature synchronizations taking longer then usual"

Last update
postmortem

On July 22, 2024, at 11:04 AM CET, an incident was initiated when the email signature synchronization process took longer than usual to complete, causing delays in distributing email signature updates to end users. The engineering team quickly initiated an investigation and identified that the issue was caused by message processing. The problem arose due to a recent migration to a new messaging framework, which began to exhibit performance issues. To mitigate the problem, the team reverted to the previous messaging framework, confirmed that the synchronization was running within the desired parameters, and by 1:07 PM CET, managed to deploy the fixed version to all production environments. The team continued to monitor system performance and, after ensuring everything was operating at the desired performance metrics, marked the incident as resolved at 6:27 PM CET. The engineering team is currently conducting extensive tests to identify the root cause of the performance issues in the new messaging framework. Additionally, they are focusing on further quality automation and performance checks to prevent similar incidents in the future.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. Signature processing is working within the desired parameters now, and prior synchronization requests are being picked up and processed. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation: Cannot create a presentation template on West Europe (Production 0)"

Last update
postmortem

On June 20, 2024, at 7:58 PM CET, an incident was initiated because uploading of presentation templates were not fully processed due to the failure of slide preview generation. The issue was introduced at 2:05 PM CET and detected by a user at 4:50 PM CET while using document creation public API. The engineering team began the investigating at 8:05 PM CET. The incident was confirmed to be affecting more than one tenant and a large number of users, causing a high rate of exceptions/errors in system logs. At 8:15 PM CET, the engineering team identified that the recent release changes, specifically a minor version increase in an external library, as the potential cause and began reverting these changes. By 9:00 PM CET, the mitigation changes has been validated in our testing environment and prepared for production release. At 9:49 PM CET, the issue was resolved on West Europe \(Production 0\). This incident serves as a reminder of the importance of thorough testing and monitoring, especially after release changes. The engineering team's prompt response and effective mitigation strategies helped to minimize the impact and ensure a swift resolution.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Service degradation: Slow Access on West Europe (Production 1)"

Last update
postmortem

On May 28, 2024, at 10:45 AM CET, an incident impacting all users utilizing the Dynamics & Library system within the West Europe \(Production 1\) environment was detected. The issue caused the system to have degraded performance, causing the users to experience slow responses or even timeouts. The engineering team quickly discovered that the degraded performance was caused by the SQL server being under a heavy load due to a reindexing operation. The reindexing operation was part of a migration process that the engineering team was rolling out. At 11:00 CET, as an immediate mitigation, the engineering team initiated the capacity increase of the SQL server. By 11:05 CET, the extra resources to the SQL server were successfully allocated. At this time, the application performance restored to normal parameters, and the application users were no longer impacted. By 12:37 CET, the incident was resolved after the engineering team successfully applied the migration and confirmed it was working as expected. We are reviewing and enhancing our internal procedures for migrations to ensure that similar issues are prevented in the future.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation : Delayed access to West Europe (Production 0) for a subset of customers"

Last update
postmortem

On May 27, 2024, at 10:06 CET, an issue impacting users in the West Europe \(Production 0\) environment was identified. The issue caused users of West Europe \(Production 0\) to experience errors and slow responses when accessing the application. The engineering team quickly discovered that the problem was caused by a faulty database migration and immediately started to implement a fix. By 10:57 CET, the fix was implemented and deployed to our West Europe \(Production 0\) environment. This resulted in the application working correctly again. After the engineering team closely monitored the application and confirmed that it was working with the desired parameters, the issue was resolved at 16:19 CET. We apologize for the disruption and remain committed to ensuring the reliability and stability of our services.

resolved

After continuous monitoring, we have confirmed the system is operating normally. The incident has been resolved, and further details will be shared in an upcoming postmortem. We apologize for the impact on affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified the issue in regards to the delayed access and have resolved this delay. As a consequence of the delayed access, another issue occurred which resulted in errors when browsing the asset library. Engineers are currently investigating this issue.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation: In case Productivity Add-in is active, tables inside powerpoint are unusable"

Last update
postmortem

On May 22, 2024, at 4:58 PM CET, an incident impacting all customers using the Productivity Add-in within the Insider track using the Hive environment was triggered by a code change introducing a bug in PowerPoint, causing errors and numerous popup messages when users attempted to work with tables. Upon detection, the engineering team swiftly initiated an investigation and identified the issue by 5:03 PM CET on May 22. To mitigate the impact, the team promptly executed a plan to downgrade affected users to a previously stable version, completing the action by 5:26 PM CET on May 22. We apologize for the disruption and remain committed to ensuring the reliability and stability of our services.

resolved

The faulty Productivity Add-in version has been unlisted, affected users can update to the to the previous working version by checking for update inside Templafy Desktop or by restarting their pc. Only customers on the insider version were affected by this issue. Further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation: Templafy Desktop shutdows after trying to startup"

Last update
postmortem

`On May 8, 2024, at 4:00 PM CET, an incident began with the Templafy Desktop upgrade, which resulted in broken AddIns for a limited set of customers. This issue was detected, on May 8, 2024, at 12:00 PM CET. The engineering team promptly began investigating the incident at 4:07 PM CET on May 8, 2024. They worked to determine the scope of affected customers by testing various versions of the Templafy Desktop Host. By 4:38 PM CET, the team had identified and deprecated the problematic Templafy Desktop Client version to prevent further customer upgrades. By 4:52 PM CET, the incident status had moved into the mitigation phase. For customers already impacted, the team identified two solutions: reinstalling Templafy Desktop or removing the content of Templafy Desktop Client versions and restarting Templafy Desktop. These solutions were communicated to the relevant individual customers. To prevent the issue from happening again, we will fix the root cause and implement further testing between version upgrades.`

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation: Customer Experiencing Issues Logging in West Europe (Production 1)"

Last update
postmortem

On the 24-th of April at 9:05 CET an incident occured that affected users using our West Europe \(Production 1\) cluster. The incident resulted from an execution plan update of the SQL Server, causing queries to execute without utilizing the appropriate database index. Consequently, queries on the UserManagement database experienced high resource consumption, leading to severe slowdowns and eventual inaccessibility, causing the application on the West Europe \(Production 1\) cluster to become inaccessible. At 9:41 CET mitigation was achieved by scaling up SQL Server resources. Currently, the engineering team is actively investigating solutions to prevent similar incidents in the future.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Some users are receiving errors during document creation"

Last update
postmortem

On April 22, 2024, at 10:46 AM CET, an issue was reported affecting all users across every cluster, where the dialog for creating or updating documents, containing TextElement or SlideCollection insertions, would not advance beyond the loading screen on subsequent attempts. Initial investigations quickly linked the problem to a recent update in the UserManagement Microfrontend, specifically a defect introduced in the UserProfile component. This issue was rectified by reverting the changes, with normal functionality restored by 2:05 PM CET on the same day. In response to this incident, we have implemented more rigorous testing procedures for frontend components and expanded end-to-end testing to better simulate user interactions. We've also revised the integration and usage of the UserProfile component to prevent recurrence and introduced new tests that run asynchronously to detect issues early without delaying other processes. We apologize for the inconvenience caused and are committed to continuous improvements to ensure the reliability of our services.

resolved

The Engineering Team has finished implementing the solution and it has been deployed into all regions. The incident has now been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

The engineering team has identified the root cause of the issue and is actively working on implementing a solution.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. The initial investigation shows that document creation fails for documents with certain Dynamic elements. Further updates will be posted here soon.

Report: "Service degradation - A subset of users belonging to partner tenants are facing challenges while attempting to log in to the application."

Last update
postmortem

On February 1st, starting at 7:45 AM, users belonging to partner tenants encountered authentication issues. The problem surfaced only when users from partner tenants attempted to access those tenants other than their own, situated on different clusters. The engineering team identified the issue at 9:06 AM CET, attributing it to recent changes. Swiftly responding, a fix was implemented and deployed across all clusters by 10:40 AM CET, successfully restoring normal system functionality for the affected users. To prevent future occurrences, we are enhancing our automated testing procedures, ensuring the robustness of cross-tenant functionality and averting similar issues.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Service degradation - Infinite reload issue encountered by users"

Last update
postmortem

On January 23, 2024, starting at 16:04 CET, users in our West Europe \(Production 0\) region were affected by an issue caused by the migration to partitioned cookies. The problem impacted a subset of users who had their browsers or office applications open during the update process. The issue manifested as infinite web page reloading for the affected users. To address the problem, users were advised to restart both browsers and Microsoft Office. Following a detailed investigation, the engineering team identified the root cause and successfully implemented a solution that resolved the infinite reloading without requiring user intervention. The solution was deployed, and by 22:04 CET, the problem was mitigated. Subsequently, a comprehensive review led to the definition of actions and procedures aimed at enhancing the resiliency of our system against similar changes.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The engineering team has implemented a fix and has started rolling it out to all systems. We will post a notification once this is completed.

identified

The engineering team has identified the root cause of the issue and is actively working on implementing a fix.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. A potential workaround for users is to close both their web browser and Microsoft Office completely, and then reopen them. We are working on a fix for all users that will not require any user intervention. Further updates will be posted here soon.

Report: "Service degradation in the Australia East (Production 3) region"

Last update
postmortem

On November 24th, 2023, starting at 6:39 PM CET, users of Templafy Hive in the Australia East \(Production 3\) region started to experience degraded application performance when authenticating to the application. The root cause of the problem was an external dependency, which started to have availability issues, this propagated to our application, because the current implementation logic was reacting in a way that caused threads to not get enough resources to work properly in our application for authentication requests. By 6:42 PM CET, our engineers were notified by our automatic alert system about the slow performance. The root cause was quickly identified, and the issue was mitigated by the engineering team. This action restored the proper functionality to the system, successfully resolving the incident by November 24th at 10:15 PM CET. Currently, the engineering team is working on improving the system making it more resilient for similar occurrences.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers in the Australia East (Production 3) region. Customers could face delays or failures when trying to access the application. We are investigating. Further updates will be posted here soon.

Report: "The users cannot use "Search" in Library App on West Europe (Production 0)"

Last update
postmortem

On October 3rd, 2023, starting at 10:35 AM CET, a significant number of users of Templafy Hive in the West Europe \(Production 0\) region encountered difficulties while trying to use the search field within our platform. As we delved into the situation, it became apparent that some challenges related to an internal memory allocation setting had arisen, directly impacting the search functionality and hindering user experience. Upon detecting the issue, our engineers immediately identified the underlying memory allocation constraint and swiftly took actions to rectify the situation. By 1:25 PM CET, the system's performance and functionality, including the search field, were fully restored to ensure seamless operations for our Templafy Hive users in Production0.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

investigating

We are currently investigating this issue.

Report: "Outlook Web Add-ins not functioning correctly in Outlook Desktop"

Last update
postmortem

On October 30th, 2023, users of the Templafy Email Signature Outlook Web Add-in from the Prod0 West Europe region were not able to use the product on Outlook Desktop for Windows, email signatures were not being inserted automatically and manual insertion took a long time. The issue propagated on October 31st to all regions. Because an Outlook restart was needed to experience the problem, the first reports of it were created with a one day delay. After running investigations, we identified the problem on an internal library update that introduced two unsupported features in the Outlook Desktop environment, causing the Web Add-in to not load. Upon detecting the issue, our engineers immediately downgraded the internal library, an action taken to rectify the situation. By 13:15 CET, November 1st, the Web Add-in's functionality was fully restored in all regions.

resolved

The incident has been resolved by downgrading our UI Library. Root cause analysis is under way and it will be provided in the upcoming days. We apologize for the impact to affected customers.

identified

Upon further investigation we have identified an issue with recent update to our UI Library. We are testing a downgrade of the solution

identified

We have created an issue on Microsoft Office JS GitHub page and we are waiting for them to acknowledge the issue

identified

We have identified the following issues with Outlook Web Add-ins in Outlook Desktop for Windows : Signatures: Outlook Web Add-ins are failing to insert signatures automatically in outgoing emails. Task Panes: Task panes are experiencing slow loading times when accessing the Content Library and Email Signatures features. Actions Taken: The issue has been reported to the Microsoft Office JS team for further investigation and resolution.

Report: "Users and admins not able to access some assets in library"

Last update
postmortem

On September 14th, 2023, starting at 08:00 AM CET, users of Templafy Hive encountered an issue where they were unable to access some asset folders in library. The root cause of this problem was that the navigation paths for the library assets were out of sync at the database level. This issue was caused by a migration problem that occurred during recent refactoring. Upon receiving notifications, our engineers fixed the migration problem. This action restored the proper functionality of the application, successfully resolving the incident by September 15th at 03:27 PM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

All regions are affected by this bug at the moment.

identified

Root cause has been identified and the team is working towards a validating a fix.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Unable to access asset folders"

Last update
postmortem

On September 14th, 2023, starting at 08:00 AM CET, users of Templafy Hive encountered an issue where they were unable to access asset folders. The root cause of this problem was traced back to a recent change in the way navigation paths were handled. Specifically, the handling of certain characters had been altered, resulting in unforeseen complications in asset folder routing. Upon receiving notifications, our engineers swiftly identified the issue and initiated corrective measures to revert the system to its previous, fully functional state. This action restored the proper functionality of the application, successfully resolving the incident by 02:55 PM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

Our engineering team has successfully identified the root cause and authored a fix for the reported issue.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Text elements not available in Outlook Desktop on Templafy One"

Last update
postmortem

On August 11, 2023, starting at 09:14 AM CET, users of Templafy One encountered an issue wherein they were unable to view the "text elements" link within Outlook Desktop. The underlying cause of this problem was traced back to an updated dependency, which included some invalid logic in case of Templafy One. The logic was implemented for the Hive platform, and for Templafy One, it inadvertently resulted in the "text elements" link becoming hidden. Upon receiving notifications, the engineers promptly identified the problem and took steps to revert the changes back to the previously operational version. This corrective action effectively reinstated the proper functionality of the application and successfully resolved the incident at 1:51 PM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have found the source of the problem and are releasing a new version of our software that will address it.

identified

We are continuing to work on a fix for this issue.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Email Signature Web add-in Issue - "Unable to fetch Signatures""

Last update
postmortem

On August 2, 2023, from 16:13 to 18:56 CET, a subset of users encountered problems with the automatic insertion of email signatures through the Email Signature Outlook Web Add-in in Outlook Desktop. The root cause was traced back to the incorporation of a method call that proved incompatible with certain users' environments, specifically affecting customers with an older version of Outlook that does not support the specific method call. Upon identification of the issue, the engineering team promptly devised a solution by substituting the problematic method call with a compatible alternative. This swift intervention effectively restored the functionality for signature insertion, catering to users utilizing the older version of Outlook. While the resolution for the incident was devised on August 2 at 18:56, its comprehensive implementation faced a minor delay owing to caching challenges encountered by select customers. The definitive confirmation of complete resolution from customers was received on the 3rd of August, toward the close of the day.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Email Signature Web add-in Issue - "Unable to fetch Signatures""

Last update
postmortem

On August 2, 2023, from 16:13 to 18:56 CET, a subset of users encountered problems with the automatic insertion of email signatures through the Email Signature Outlook Web Add-in in Outlook Desktop. The root cause was traced back to the incorporation of a method call that proved incompatible with certain users' environments, specifically affecting customers with an older version of Outlook that does not support the specific method call. Upon identification of the issue, the engineering team promptly devised a solution by substituting the problematic method call with a compatible alternative. This swift intervention effectively restored the functionality for signature insertion, catering to users utilizing the older version of Outlook.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified the issue and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating the issue.

Report: "Service degradation - Slowness in WebApp"

Last update
postmortem

On August 3, 2023, starting 8:40 AM, 5% of the users started to have high response times which lead to slowness in the WebApp and customer cases started to appear. At 10:49 AM CET the engineers started to investigate the issue. The engineers identified the issue immedietly and started mitigating the issue by slowing down some of the large internal requests for background tasks to prevent slowness. At 11:40 AM CET the slowness disappeared and daily work could be processed normally. We monitored the behavior until 2:21 PM CET and could not see any slowness, both from our side and customers reported that the issue is not affecting them anymore. The incident was mitigated. To prevent this issue in future, we aligned the incoming request amount with our infrastructure scaling rules to be more reactive when large requests are being handled by our background tasks.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

investigating

We are currently investigating this issue.

Report: "Service degradation - Previews are not generated"

Last update
postmortem

As part of our security protocol, we conducted a regeneration of the access credentials for our main storage account. Unfortunately, this action resulted in the loss of access for our preview engine, which relies on accessing the storage to store generated previews. Consequently, the preview engine was unable to save the generated previews during this period. The incident came to light on the 27th of July at 16:29 PM when users reported the absence of saved previews. Our engineering team swiftly responded to the issue and resolved it by releasing a new version of the component responsible for generating previews. This updated version included the new security credentials and was deployed at 17:59 PM, successfully restoring the preview generation functionality. In the future, we will take extra precautionary measures to avoid similar incidents. These measures include changing the security method used by the preview component to access our storage account, adding comprehensive documentation covering all steps of the process, implementing pre-change validation procedures, and deploying robust monitoring systems to detect any potential access-related anomalies promptly.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Cannot insert Dynamics assets in Web-Addins"

Last update
postmortem

On the 26th of July 2023 starting at 9:55 AM CET, an incident impacting Templafy customers using the Web Add-ins was detected. The problem prevented users from inserting slides containing dynamic content into their documents. The engineering team identified the root cause and successfully applied a fix by 11:45 PM CET, resolving the problem for all affected customers. To prevent such incidents in the future, we have implemented more stringent code review and testing protocols for our Web Add-in code base.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified the issue and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating the issue.

Report: "Service Degradation: Navigation in the explorer is not working correctly"

Last update
postmortem

On July 25, 2023, starting at 9:41 AM CET, a small subset of our customers hosted on Prod0 in West Europe with the Spaces module enabled encountered a critical issue that prevented them from navigating the asset explorer folder structure. This issue was first detected via internal usage at 10:35 AM CET. The code causing the bug was identified, reverted and a fix was deployed at 12:12 PM CET

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Service degradation - Signatures are not automatically inserted in Outlook Web Add-in, in Outlook Desktop"

Last update
postmortem

On July 25, 2023, starting at 2:28 PM CET, users from all regions using the Email Signature Outlook Web Add-in in Outlook Desktop encountered a critical issue that prevented them from getting email signatures automatically inserted. This incident was attributed to an internal misconfiguration within the system. Following engineering investigation and troubleshooting, the internal misconfiguration was identified and fixed. This resolution action effectively restored the automatic signature insertion functionality for all affected users. The incident was successfully resolved at 8:00 PM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Service degradation : Users of Australia Email Signature Server are unable to send emails"

Last update
postmortem

On July 10, 2023, starting at 10:14 PM CET, we experienced a service interruption that affected some customers in Australia due to a misconfiguration in our Email Signature Server release pipeline. This resulted in temporary unavailability for a subset of customers. We have taken immediate action to address the issue and prevent similar incidents in the future. Our measures include enhancing configuration validation, improving testing procedures and strengthening monitoring capabilities. The incident was successfully mitigated at 11:42 PM CET. We sincerely apologize for any inconvenience caused and appreciate your patience.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated for the affected customers. Our team is actively monitoring the situation to ensure ongoing stability. We are observing the systems to prevent any further disruptions.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Report: "Users from West Europe located in Prod0 unable to create document(s)"

Last update
postmortem

On July 4, 2023, starting at 9:27 AM CET, users provisioned in WestEurope Prod 0 encountered a critical issue that prevented them from creating documents. This incident was attributed to an internal misconfiguration within the system. Following engineering investigation and troubleshooting, the internal misconfiguration was identified and subsequently reverted. This resolution action effectively restored the document creation functionality for all affected users. The incident was successfully resolved at 11:15 AM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Service degradation - Stuf DCR Connector issues"

Last update
postmortem

On May 6th, at 9:41 CEST, a subset of Templafy customers began experiencing difficulties with Stuf DCR, our service. We sincerely apologize for any inconvenience caused by this issue. The problem originated from the loss of STUF DCR certificate configuration, which occurred due to an unintended out-of-order execution of data and schema migrations. This unexpected sequence of events resulted in the configuration being overwritten or disrupted, leading to service disruptions and errors for our affected customers. As soon as we became aware of the issue at 9:41 CEST, our team promptly initiated an investigation to identify the root cause and find a resolution. By 03:43 CEST, we were able to successfully restore the lost data and rectify the underlying problem. During the restoration process, we closely monitored the situation to ensure that requests were being processed correctly. We are pleased to inform you that normal operations have resumed, and we are now observing successful requests being handled without any issues. We deeply regret any inconvenience or disruption this incident may have caused for our valued customers. We understand the importance of reliable and uninterrupted service, and we are committed to implementing measures to prevent similar occurrences in the future. Thank you for your patience and understanding throughout this situation.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

identified

Engineers have implemented the fix and currently are working on validating it. Further updates will be posted here soon.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating the issue.

Report: "Service degradation - Connector issues"

Last update
postmortem

A deployment at 8:02 CEST on the 1st of June caused data inconsistencies resulting in an outage for data connectors relying on certificate authentication. This affected only customers in West Europe \(Prod0\) region. The problem was first noticed by Templafy at 8:04 and reported by customers at 8:25 CEST. The incident was resolved at 11:30 CEST. Moving forward, steps will be taken to ensure that the data remains consistent, identifiers are unique, and better monitoring is in place to prevent similar incidents from happening again.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue related to a subset of our connectors and are working towards a resolution. The incident affects only customers in West Europe (Prod0) region. Further updates will be posted here soon.

Report: "Service degradation - Slow response time when using Templafy One"

Last update
postmortem

Starting at 23-th of May at 11:00 CEST a subset of Templafy One custtomers started experiencing slow response times when accessing the application. The issue was caused by an unexpected situation in our offline component, which resulted in an excessive number of operations on our storage account, causing it to have a slower response. After identifying the issue we started to apply mitigation steps to stabilize the system and restore performance to acceptable levels. At 12:10 CEST we could already confirm that the system was working within the desired performance metrics. Following this the engineering team maintained continuous monitoring of the system to ensure that all operational metrics pertaining to performance stayed within optimal parameters.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

investigating

We are seeing a subset of customers are experiencing slow response times. The engineering team is working on identifying the issue.

Report: "Service degradation - Slow response time when using Templafy One"

Last update
postmortem

Starting at 24-th of May at 10:27 CEST a subset of Templafy One custtomers started experiencing slow response times when accessing the application. The issue was caused by an unexpected situation in our offline component, that resulted in an excesive number of operations on our storage account, causing it to have a slower response. Upon identifying the issue, at 11:35 CEST, we promptly implemented short-term preemptive measures to stabilize the system and restore performance to acceptable levels. These measures, including immediate mitigation steps and intensified monitoring, proved effective in maintaining system stability and ensuring ongoing performance within our standards. In parallel, we have initiated a comprehensive review of our system to implement long-term changes aimed at enhancing efficiency and performance. Our team is diligently working on optimizing the codebase to identify and implement improvements that will ensure optimized performance in future scenarios.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

The system is stable and operating within our performance standards. We will continue to closely monitor the situation to maintain stability.

monitoring

We have applied mitigation steps and performance is improving. We are continuing to monitor the system to ensure ongoing stability and performance. In parallel we are also continuing investigating the root cause so that we are able to prevent similar issues in the future. Details will be provided in the postmortem.

identified

We have identified an issue that affects a subset of customers. Because of the issue, customers are experiencing slow response times. Further updates will be posted here soon.

Report: "Service degradation - Trusted publisher notification"

Last update
postmortem

**Summary:** Early this week, we identified an issue with Templafy VSTO Add-ins in Microsoft Office for PC. A subset of users received a “Publisher cannot be verified” notification when starting Office desktop applications that had Templafy VSTO Add-ins installed. Our service was not down, and users who chose to install the Office Add-in from the unknown publisher could continue using our product via Office desktop. As an immediate response, we assembled a team from Engineering, Product, Support, and InfoSec to scope and address the issue. Templafy identified that the issue was only impacting customers using a particular subset of VSTO add-ins and versions. Subsequently, on Tuesday, Templafy implemented and then deployed a fix for the impacted customers, without requiring any further action from the clients, to ensure that the issue was resolved. **Issue overview:** On Sunday, 30th of April, we received support tickets indicating users were being prompted with a Microsoft Office Customization Installer notification, stating “Publisher cannot be verified.” Upon investigation, we determined that the issue was limited to Windows users who opened Microsoft Word, PowerPoint, Excel, or Outlook using Templafy One and Hive with the following VSTO Add-ins and their versions: * Library version 6.1.36 - 6.1.73 * Productivity version 5.0.701 - 7.1.205 * Email Signature version 6.1.36 - 6.1.65 Templafy Web Add-ins \(For MS Office on-prem, Online & Mac\) were not impacted. Customers on the latest stable version of Templafy Hive were also not impacted. **Root cause:** Upon thorough investigation, we determined that the root cause was that the signing certificate, used to sign the VSTO Add-ins, could not be verified by Microsoft Office anymore due to a change in the Time Stamp Authority requirements. The signing certificate we used was valid until the 28th of April. To ensure that Add-ins are valid after the certificate expires, we use a Time Stamp Authority \(TSA\) when signing add-ins. As long as the Time Stamp Authority is trusted, add-ins remain valid after the certificate expires. To enhance global security, on the 1st of June 2021, [the CA/B Forum](https://cabforum.org/wp-content/uploads/Baseline-Requirements-for-the-Issuance-and-Management-of-Code-Signing.v2.8.pdf#page=40) increased the required key length for Time Stamp Authority certificates to a minimum of 3072[.](https://bit.ly/3LAEoXW).) The Time Stamp Authority \(TSA\) we used when signing our add-ins was using a key length of 2048. Therefore, it was no longer trusted, resulting in the add-ins becoming invalid after the certificate expiration date. We had anticipated this problem and had switched to a new Time Stamp Authority \(TSA\) when signing executable files in March 2021, but we had not correctly switched to the new TSA when signing manifest files. Because of this, after the certificate expiration date on the 28th of April, users with impacted VSTO Add-in versions who opened an Office application received the "Publisher cannot be verified" notification. **Resolution:** To resolve the issue, we manually re-signed Library, Email Signature, and Productivity VSTO Add-ins using a new certificate with a valid Time Stamp Authority without changing any behavior of the software. We manually validated the re-signed VSTO Add-ins with both Templafy One and Hive customers. After confirmation, we re-signed all affected add-ins and programmatically updated Templafy Desktop packages for all affected Templafy One customers. Similarly, we programmatically updated all affected Hive customers to the re-signed add-ins. This process was completed by the end of Tuesday, 2nd of May. The packages with the re-signed VSTO Add-ins were automatically installed by Templafy Desktop on end users' PCs after midnight Tuesday \(local time\). Going forward, we will take extra precautions to ensure that our signed packages remain valid and adhere to the necessary key length standards in order to prevent similar problems in the future. **Timeline:** * Sunday, 30th of April: We identified the incident and received the first support tickets. * Monday, 1st of May: We began investigating and escalated to involve our Engineering, Product, Support, and InfoSec departments. We identified the scope, cause, and solution. At the end of Monday, we got confirmation from a few customers that the fix was working. * Tuesday, 2nd of May: We automatically re-signed all add-ins and updated all affected customers. * Wednesday, 3rd of May: Customers received the corrected VSTO Add-ins and the issue was officially resolved for all impacted customers. Lastly, we would like to thank you for your patience and understanding during this incident. We know that disruptions to your business operations are costly and frustrating, and as outlined above, we are committed to doing everything in our power to prevent similar incidents from occurring in the future. If you have any further questions or concerns, please do not hesitate to reach out to [support@templafy.com](mailto:support@templafy.com). We are always here to assist you and help you get the most out of our product. We value your business and appreciate your ongoing trust in our service.

resolved

The incident has been resolved, and further information will be provided in a postmortem. We apologize for the impact to affected customers.

monitoring

We have mitigated the incident for all affected customers. All affected customers have been updated to the re-signed versions of the add-ins. To deploy this update, the Templafy Desktop app will install the re-signed add-ins the next time it checks for updates, which will automatically happen after midnight local time. It can also be triggered manually by restarting the machine or through the Templafy Desktop application itself. No additional action will be needed from the admin or end-user. For any customers using server mode (e.g. Citrix), an update would have to be triggered to fetch the newest update. If the end-user has disabled the add-in in Office in the past 24 hours, they might need to manually activate it again. Read more here: https://support.templafy.com/hc/en-us/articles/115005420025 While this issue is resolved, we will not close this incident until we have confirmed updates have been deployed for all customers. Please expect a detailed post-mortem within the next 24 business hours. To report any further issues, please reach out to support@templafy.com. Thank you for your patience as we resolved this issue.

identified

We have updated the majority of affected customers on Templafy One to the re-signed versions of the add-ins. Templafy Desktop will automatically install the re-signed add-ins the next time it checks for updates. This will automatically happen after midnight (local time) or can be triggered manually by the end-user by restarting the machine or through the Templafy Desktop application. No additional action will be needed from either the admin or end-user side. In a few cases, we have not yet been able to update the add-ins. This is in two situations: If you are using Server Mode (e.g. Citrix) or have special characters in your existing package. If an end-user has disabled the add-in in Office in the past 24 hours, they might need to manually activate it again. Read more here: https://support.templafy.com/hc/en-us/articles/115005420025 With this update, we have mitigated the incident for the majority of the affected customers. We are continuing work to create an update that also applies to customers on Templafy Hive as well as customers using Server Mode add-ins. If you’re on Templafy One (and the two cases above don’t apply to you) and still experience any issues, please reach out to support@templafy.com

identified

We now have a full overview of affected versions and customers. If you are a Hive customer on an automatic release track (like most tenants), you are not affected. All affected versions of Library, Email Signature, ProductivityPlus, and Check have now been re-signed with the new certificate, and the re-signed add-in versions have been tested and confirmed working with multiple customers. For customers on Templafy Hive: New versions of add-ins have been uploaded to the admin center, and we are currently working on automatically updating all affected customers to the version containing the re-signed add-ins. The only change to the new add-in version will be the re-signing of add-in files. For customers on Templafy One: New versions of existing Desktop packages are being uploaded to the admin center. These new packages contain re-signed add-in files but are otherwise the same as the active desktop package. New versions of ProductivityPlus, Check, and Email Signature have been uploaded to the admin center and are available from the dropdowns with an x.1 version extension. Like the other add-ins, no other changes than re-signing have happened to these add-ins. Next Steps We are actively working on an automatic update of the add-in version in the admin center for both One and Hive customers. In this automated update, versions will be changed to the re-signed version of what is already active on the tenant. When this has been done, no additional action will be needed from either the admin or end-user side. We expect this to happen throughout today (European time). Templafy Desktop will automatically install the re-signed add-ins the next time it checks for updates. This will automatically happen after midnight (local time) or can be triggered manually by the end-user by restarting the machine or through the Templafy Desktop application. Customers using server mode (e.g. Citrix) For customers using server mode such as a Citrix environment, an update of Templafy Desktop will need to be triggered on the customer side after the changes above have been rolled out.

identified

Templafy has identified a solution for this issue and is in the process of coordinating automatic deployment to all customers to ensure service is returned without customer involvement. Details of the fix: - We have already re-signed all impacted Library add-ins and are in the process of re-signing all impacted e-mail signature and Productivity add-ins - This is being validated for both One and Hive customers Additional details for Hive customers: - All Hive customers on either the stable or insider tracks have not been impacted - After all impacted add-ins have been validated, we will deploy the fix for Hive customers immediately Additional details for One customers: - We are developing a method for programmatically updating all Templafy Desktop packages with re-signed add-ins - We will provide an update ASAP as to when this deployment can be expected

identified

The latest stable Templafy Hive add-in versions are not impacted by this issue. This issue impacts the following VSTO add-ins and their versions: Library add-in version 6.1.36 - 6.1.73 Productivity version 5.0.701 - 7.1.205 Email Signature version 6.1.36 - 6.1.65 Templafy Hive customers experiencing this issue can upgrade to the latest stable version in order to resolve it.

identified

Cause: In April 2021, Templafy updated the timestamp server that we used to sign our Templafy VSTO certificates due to the timestamp server being purchased by another company. The certificates are used by Templafy to ensure that our software has not been tampered with and can be installed safely and silently by end users. Due to this change in timestamp server, Templafy started using a timestamp server that was flagged invalid at a later stage, therefore signing VSTO packages with the invalid timestamp server. Templafy is still identifying all affected VSTO versions. Potential resolution: Templafy is in the process of signing all new VSTO packages with the correct timestamp server as well as working diligently to identify a programmatic / centralized solution to make sure all previous VSTO packages are signed with the correct timestamp server. Please continue to follow the status page for more updates as they become available. Thank you.

identified

The engineering team is continuing to work on the incident and we will provide more information once available.

identified

We are continuing to investigate the issue as well as scope the affected versions of Templafy add-ins. We will continue to provide status updates every hour.

identified

We have identified the cause of "Publisher cannot be verified" messages. The recent routine certificate changes performed by Templafy have exposed a configuration change that was made on our root certificate timestamp servers. This issue has been escalated to the highest degree and all technical teams are involved to resolve the issue as soon as possible.

investigating

We have identified an issue where customers receive a notification popup when starting an Office application where Templafy VSTO Add-ins are installed. This only affects Windows users. For more information about it and how to proceed please see our knowledge base article: https://support.templafy.com/hc/en-us/articles/10485903891485-Trusted-publisher-notification Currently, our engineering team is actively working on implementing a solution.

Report: "Service degradation - Trusted publisher notification"

Last update
postmortem

On 19-th of April 2023 9:53 AM CEST, a software incident occurred during the rollout of a new certificate. The previous certificate was inadvertently removed from the Trusted Publisher store, causing a Certification prompt to be displayed for Add-Ins that still utilized the old certificate. To resolve the incident, the engineering team developed and rolled out a fix, which added both the old and new certificates to the Trusted Publisher store. The fix was deployed at 12:57 CEST. This prevented the Certification prompt from being displayed and restored normal operation. To avoid similar incidents in the future, the team is implementing several measures, such as improving the deployment process to ensure the proper configuration of Trusted Publisher store and conducting additional testing to catch potential issues before rollout. The incident caused inconvenience to users and required immediate action from the engineering team to restore normal operation. However, through a swift and effective response, the team was able to resolve the incident and put measures in place to prevent similar incidents in the future.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified an issue where customers receive a notification popup when starting an Office application where Templafy VSTO Add-ins are installed. This only affects Windows users. For more information about it and how to proceed please see our knowledge base article: https://support.templafy.com/hc/en-us/articles/10485903891485-Trusted-publisher-notification Currently, our engineering team is actively working on implementing a solution.

Report: "Templafy One customer unable to upload slides and presentations"

Last update
postmortem

Starting at 6th April 2023 11:50 AM CET, Templafy One customers started to get an exception when trying to upload slides and/or presentations. This behaviour was introduced by system misconfiguration. We noticed the issue at 11th April 8:23 PM CET from a customer report. The engineering team started working on this issue at 11th April 2023 9:00 PM CET and the issue was mitigated at 11th April 2023 10:15 PM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

identified

We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

investigating

We are currently investigating this issue.

Report: "Users are unable to create / download documents"

Last update
postmortem

Starting at 10th April 2023 07:45 AM CET customers stopped being able to create/download documents. This behaviour was introduced by a misalignment between component versions. We noticed the issue at 10th April 11:03 AM CET from a customer report. The engineering team started working on this issue at 10th April 2023 02:57 PM CET and the issue was mitigated at 10th April 2023 03:57 PM CET.

resolved

The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

investigating

We are currently investigating this issue.

Report: "Issues with Email Signaure Outlook Web Add-in"

Last update
postmortem

Since the incident was caused by a Microsoft issue details about it can be found at the Office JS Github [link](https://github.com/OfficeDev/office-js/issues/3240).

resolved

Microsoft has acknowledged that the issue is in the Outlook Online component, please follow up directly on https://github.com/OfficeDev/office-js/issues/3240 to see the progress.

identified

The Microsoft Outlook Add-ins team has responded and acknowledged the issue. These are the details they have communicated: "Thank you for reporting the issue. We are able to repro it. We also want to highlight that this issue is only impacting users in Targeted Release, not Standard Release (learn more here: Set up the Standard or Targeted release options - Microsoft 365 admin | Microsoft Learn). We will update the thread once we have more updates on the fix." The response can be seen on the above-mentioned GitHub issue.

identified

Microsoft is looking into the issue and we are waiting for them to provide a fix.

identified

Due to an update from Microsoft regarding the Outlook Online component some customers may experience issues related to the Email Signature Outlook Web Add-in. This can affect the loading of the Web Add-in or the automatic insertion of the signature. This only affects Outlook Online, Outlook Web Add-in loads correctly in Outlook for Desktop. An issue has been raised with Microsoft and we are waiting for an update on it. Here is the link to the mentioned issue: https://github.com/OfficeDev/office-js/issues/3240

Report: "Azure Network issues impacting connectivity worldwide causing service degradation in Templafy One and Templafy Hive"

Last update
postmortem

Preliminary Post Incident Review for Azure Networking Issue can be found here: [https://status.azure.com/en-us/status/history/#:~:text=Preliminary%20Post%20Incident%20Review%20\(PIR\)%20%E2%80%93%20Azure%20Networking%20%E2%80%93%20Global%20WAN%20issues%20\(Tracking%20ID%20VSG1%2DB90\)](https://status.azure.com/en-us/status/history/#:~:text=Preliminary%20Post%20Incident%20Review%20(PIR)%20%E2%80%93%20Azure%20Networking%20%E2%80%93%20Global%20WAN%20issues%20(Tracking%20ID%20VSG1%2DB90))

resolved

The incident has been resolved by Microsoft and both Templafy One and Hive are working as expected. Preliminary Root Cause provided by Microsoft: "We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, connectivity between services within regions, as well as ExpressRoute connections." Post-mortem to be provided by Microsoft in the next 3 days at the following link : https://status.azure.com/en-us/status/history/#:~:text=Azure%20Networking%20%2D%20Multiple%20regions%20%2D%20Mitigated%20(Tracking%20ID%20VSG1%2DB90)

monitoring

Microsoft reported this issue to be mitigated. Both Templafy One and Hive are working as expected. We will continue to monitor Templafy One and Hive status

identified

MIcrosoft has identified the issue and started to mitigate it. More information can be found at: https://status.azure.com/en-us/status.

identified

Service degradation in Templafy One and Hive caused by Azure Network issues in several regions Further updates can be seen here - https://status.azure.com/en-us/status