Is Exclaimer Cloud Down Right Now? Check if there is a current outage ongoing.

Our engineering team have been monitoring the issue reported by Microsoft and have noted a complete reduction in the number of messages that are being rejected or deferred by Microsoft relays at this time, indicating that the issue has now been resolved by Microsoft. If you are continuing to see mail flow issues, please reach out to our Support Team who will be able to assist further

Update: Exclaimer engineers have confirmed that the reported issue is linked to the ongoing Microsoft service alert EX1079935, which is currently impacting the AU region only at this time. You can view the Microsoft service alert here: Microsoft Service Health Dashboard – EX1079935 http://admin.microsoft.com/Adminportal/Home?ref=/servicehealth/:/alerts/EX1079935 Next Update: To be confirmed (TBC)

We are investigating a possible service alert. Issue: Server-side signatures are causing email delivery delays Symptoms: Users using server-side signature solutions may experience email delay NDR Next Update In: TBC

Our engineering team have been monitoring the issue reported by Microsoft and have noted a complete reduction in the number of messages that are being rejected or deferred by Microsoft relays at this time, indicating that the issue has now been resolved by Microsoft. If you are continuing to see mail flow issues, please reach out to our Support Team who will be able to assist further

Update: Exclaimer engineers have confirmed that the reported issue is linked to the ongoing Microsoft service alert EX1079935, which is currently impacting the AU region only at this time.You can view the Microsoft service alert here:Microsoft Service Health Dashboard – EX1079935http://admin.microsoft.com/Adminportal/Home?ref=/servicehealth/:/alerts/EX1079935Next Update: To be confirmed (TBC)

We are investigating a possible service alert. Issue: Server-side signatures are causing email delivery delaysSymptoms: Users using server-side signature solutions may experience email delay NDR Next Update In: TBC

This incident has now been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

Although we are still awaiting final confirmation from our certificate issuance partner, our engineers have noted that certificates are now beginning to be processed as normal again. We will continue to monitor

We have identified an issue with our certificate provider that is causing certificate requests to fail. This will impact new sign-ups to our server-side service, prevennting them from completing setup.

Issue: Customers unable to complete setup of Exclaimer or synchronize User Details manually.  Incident Length: 17 hours and 21 minutes  Incident Date: 30/04/2025, UTC 16:30 – 09:51 01/05/2025\]  Incident Status: Resolved    **Summary**  A subset of customers that attempted run the setup for Exclaimer experienced an error advising that ‘An unexpected error has happened.’ within the portal causing the Configuration to fail.  Existing subscriptions that were configured, and emails that were being routed would not have been impacted by this issue.    **Root Cause**  As part of standard maintenance of the Exclaimer Cloud service within all regions, the Secret keys used to verify the certifications used by Enterprise Applications for Exclaimer Cloud did not return the expected values to the configuration wizard. This resulted in subscriptions not being able to authenticate the application consent grant requested by Exclaimer Cloud’s setup.    **Mitigation**  To address the issue, the Secret Keys that are applied to the applications were reapplied to the Exclaimer Cloud applications in all regions, ensuring that the setup wizard was able to authenticate as expected.    Additional reviews and investigation have been undertaken internally to determine why the cause of the incident was not detected earlier and ensure that proper alerting and monitoring has been configured to prevent similar issues with other Enterprise applications from occurring.     **Incident Timeline**    01/05/2025   08:29 - Reports of configuration errors when attempting to configure Exclaimer Cloud from the Customer Support team prompted an investigation to begin with the Engineering team.  08:32 - A probable cause of the failure based on the failure reason being provided was determined and Engineers began to work on a resolution.  08:53 - A new set of Secret keys was generated and applied to the Enterprise Applications for AU region.  09:14 - Engineering confirmed the Secret keys had been applied and work began to deploy the Applications   09:22 - Engineers identified that the keys within the DE region were also impacted and began work to address this.  09:29 – Engineers confirmed that the Secret keys for AU applications had been applied successfully.  09:51 – Engineers confirmed success in resolving the fault for Germany and Australia regions through reapplying the Secret keys.  10:00 – Investigation began to determine why alerting did not advise of the error.   10:05 – Engineering was able to confirm the start of the failure rate within regional logging for Australia.  10:08 – Engineers identified that additional regions with pending Secret key changes may experience the issue; preventative work begins to address the issue within these regions.  10:16 – Engineers confirm success in reapplying the keys within Canada region  10:23 – Engineers confirm success in reapplying the keys within Europe region.  10:29 – Engineers confirm success in reapplying the keys within US region.  10:37 – Engineers confirm that keys in all regions have been reapplied successfully and the failure rate for this error has dropped to zero.

After continued monitoring by our Engineering team, we can confirm this incident has now been resolved. If you are continuing to experience issues within Exclaimer Cloud, please reach out to our Support team who will be able to assist in investigating the issue

A fix has now been applied to the service to restore full functionality. Our Engineering team are continuing to monitor the services for any further issues. If you are continuing to experience issues, please look to log out and log back into the Exclaimer portal, or raise a ticket with our support team.

Engineers have identified another area of impact and have been able to identify a possible cause for the issue. The team are currently working through remediation processes

We are continuing to investigate this issue.

We are currently investigating reports of an issue when attempting to run the Exclaimer Cloud Setup wizard for customers using Microsoft 365 subscriptions. This is not impacting already configured Exclaimer Cloud accounts and signatures are being applied as expected

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

We are investigating a possible service alert. We have received reports from customers about configuration error on server side. Customers are getting error "Configuration failed. The configuration failed with the following error: An unexpected error has happened". We are currently monitoring the issue.

This incident has been resolved.

The previously triggered alert has been mitigated and Engineers have confirmed that the system is stable, though we are continuing to monitor. Customers may have experienced an inability to access the UI for a brief period during this time.

We are investigating a possible service alert with our services in the UAE region

Issue: Degraded Service in both Client and Server-side signatures for UK and EU regions.  Incident Length: 1 hour and 2 minutes  Incident Date: 27/03/2025, UTC 11:23 – 12:25 27/03/2025  Incident Status: Resolved    **Summary**  Customers within the EU and UK experienced failures when trying to retrieve their signatures from the Exclaimer Add-in and would encounter delays in message delivery.    **Root Cause**  Initial implementation work of a new feature within Exclaimer branding system resulted in an unhandled exception whilst the system looked to generate signature templates. The coinciding exception that occurred resulted in the message being queued for retry in applying the signature. This caused a buildup of requests for signatures to be applied to messages.    **Mitigation**  Increased resources were provided to impacted regions to reduce the buildup of requests, and the changes were rolled back to resolve the issue.    **Incident Timeline**  11:17 – Implementation work for a new branding feature was completed and enabled within Exclaimer Cloud.  11:17 - Initial warnings were generated at this time but investigation indicated none of these were customer impacting.  11:22 – Additional alerting advised to an increase in wait time for Exclaimer Add-in applied signatures.  11:30 – SMTP requests to Exclaimer services begin to respond to connection attempts with a deferral due to a lack of signature server processing availability, mail flow alerts are raised due to increased processing time of messages.  11:37 – Exclaimer engineers begin to investigate the issue.  11:47 – A recent deployment to the signature template analytic services is isolated as a potential cause of the issue.  11:50 – The deployment is rolled back within the UK region. Engineers identify that the error rate is unchanged.` `11:51 – Implementation work was disabled but left present within Exclaimer systems.   11:59 – Resourcing within UK and EU regions are increased to process the buildup of signature requests.  12:12 – Exclaimer Engineers confirm that the service is now recovering. Services are continued to be monitored to ensure that the issue does not reoccur.  12:17 – Mail flow duration returned to normal levels.  12:25 – Exclaimer Add-in requests no longer reported any related errors.  12:49 – Exclaimer Engineers identify the cause of the issue and rollback the feature.

After extended monitoring, Exclaimer Engineers have confirmed that signatures are now being applied to messages when sent or whilst composing. A full postmortem will be provided shortly. If you are continuing to see issues, please raise a ticket with our Support Team who will be able to assist.

We are now seeing signatures being applied through our Add-in and routed through Exclaimer services as expected now. Engineering are continuing to monitor and investigate the cause.

We are continuing to investigate this issue.

Exclaimer Engineers are currently investigating a possible service alert that may be causing an impact with signature application at this time. Once we have more information from our Engineers this will be provided

After extended monitoring, Exclaimer can confirm that the Add-in is now appearing within Outlook and Outlook on the Web as expected for our customers. The cause of the Add-in not appearing within Outlook looks to have been due to an issue within the AppSource deployment system provided by Microsoft that the Exclaimer Add-in is recommended to be deployed through. We also note that other service providers reported issues with Add-ins not appearing as expected for end users within Microsoft 365. If you are continuing to experience issues with your Add-in, please reach out to our Support team for assistance.

The Exclaimer Add-in now appears to be displaying within supported Outlook versions and applying signatures as expected. If you are continuing to see issues with the Add-in not appearing within Outlook please check your deployment settings within the Microsoft 365 admin centre, or contact our Support team for assistance.

We are currently investigating an issue where end users are unable to see the Exclaimer Add-in within Outlook at this time. No updates to the add-in have been released. The Add-in will also show as deployed within the Integrated Apps menu in the 365 Admin Center If you not are able to see the Add-in within this menu, or experience any other issues with the Add-in, please raise a ticket to our Support team.

We have had confirmation that Sectigo have resolved the incident, and can conform all services are running as normal.

Certificates are provisioning as normal once again. Failed renewals will be retried by our automated system, and our records show no new sign-ups were affected. We will continue to monitor the situation

We are continuing to work on a fix for this issue.

We have identified an issue with our SSL certificate provider Sectigo that is affecting issuance of SSL certificates. This is mostly affecting certificate renewals, however it may also affect new sign-ups attempting to setup the server-side product. This is not having any impact on mail flow, and regular product function remains working as normal.

Issue: Exclaimer Add-in for Outlook returned the error message ‘domain’ is not an Exclaimer Customer and did not provide Client-side signatures. Incident Length:  7 hours and 20 minutes Incident Date: 03/03/2025, UTC 11:40 – 19:00  Incident Status: Resolved    **Summary**  Customers within the US may have experienced an issue where the Exclaimer Add-in would report that their domain was not a valid customer of Exclaimer, preventing any client-side signature from being applied whilst composing emails. There was no impact to server-side signature application during the time.    **Root Cause**  A Point of Presence \(POP\) machine used as part of Exclaimer’s Content Delivery Network within Microsoft Azure infrastructure within the US failed to correctly handle requests, returning a HTTP 404 response to some requests made within the US region. After contacting Microsoft, they were able to identify the impacted machine and resolve the issue with its routing.    **Mitigation**  Due to the timing of the incident, an update to the Exclaimer Cloud Add-in had been released earlier in the day, resulting in a rollback of the update being performed to rule out correlation to the update and the issue being experienced by customers.     Once the fault had been identified as located within the content delivery network within the US, an urgent request was raised with Microsoft to investigate the issue. Once the POP machine that was experiencing an issue with the rule sets was repaired and confirmed to be functioning correctly, errors advising ‘is not an Exclaimer Customer’ reduced dramatically to expected levels.    **Incident Timeline**  12:53 – Customer reports of ‘Domain is not a customer’ were received by Support personnel, initial troubleshooting began to investigate individual causes.  13:30 – Increased number of reports suggested a large issue occurring relating to the Add-in. Exclaimer Engineers were contacted to investigate.  13:57 An incident was raised internally to investigate the cause of the error due to a large, continued level of reports from customers  14:01 – Exclaimer Engineers identified start time of initial increase of the error.  14:08 – The error was determined to primarily impact US region customers. Additional engineers were contacted to review for any ongoing issues within the US region.  14:31 – Exclaimer Engineers verified that no configuration changes had taken place within the US region.  15:39 – Roll back of the Exclaimer Add-in was performed to verify that no changes within the Add-in caused the issue being seen.  16:00 – Rollback of the Add-in was completed by Engineers, but the error rate was unchanged.  16:10 – Engineering verified around 3% of domains were failing to return a result.  16:16 – A review of Content Delivery Network was completed and confirmed there was no issue with the data stored within, requests for data through the CDN were also successful.  16:55 – A high priority case was raised with Microsoft to investigate for possible issues.  19:00 – Exclaimer Engineers confirmed that the error rate had substantially dropped within our monitoring.  04/03 08:51 – Following a Microsoft internal investigation, Microsoft advised that requests from a specific POP located within the US were being incorrectly routed, resulting in the requests failing to find the CDN as expected.   10:15 – Monitoring of the service continued until Microsoft confirmed full resolution of the POP system   13:35 – Exclaimer Add-in version 1.23.200 was redeployed after confirming that the issue was resolved.

Engineers from Microsoft have been able to identify the root cause of the issue and implement a permanent fix. Exclaimer has continued to monitor the event for any additional reports and can confidently confirm there are no further instances of the error at this time. Additional information will be provided once the RCA from Microsoft has been completed. If you are experiencing issues with your signatures, please contact our support team for more assistance.

Our Engineering team has been monitoring the system's error rate since a significant drop in the number of failures was recorded at 18:50 GMT on March 3rd. We are continuing to monitor in the event the error rate increases again and maintain communication with Microsoft as well.

Customers using the Exclaimer Outlook Add-in may experience an error: "[domain] is not an Exclaimer Customer". Engineers from Exclaimer and Microsoft are currently working to resolve this issue and we will provide updates as we know more. We appreciate your patience in the meantime.

Issue: Exclaimer Add-In Preview Error Incident Length: 1 hour and 18 minutes Incident Date: Jan 15, 2025 01:18 am UTC to Jan 15, 2025 02:19 am UTC Incident Status: Resolved **Summary** Customers reported encountering error “Page Not Found” with Exclaimer Add-In Preview pane. Customers are still able to utilise Add In auto apply. This issue has impacted all regions' subscriptions. Going forwards, an action group rule in Azure service health has been created to send alerts internally when there are service issues, planned maintenance, health or security advisory alerts to prevent similar issues in the future. ‌ ‌**Root Cause** Due to the domain validation on the CDN failing during the migration, the add-in failed to connect to the endpoint. This resulted in the preview pane not displaying any signatures, and displaying the 'Page Not Found' error. ‌ **Incident Timeline** 01:18am UTC – Alerting advised to a failure to preview signatures in Exclaimer Add In preview pane. 01:30am UTC – Investigation indicated that the domain validation on the CDN failed during the migration. 01:50am UTC – Engineering confirmed that the preview pane is back up. 02:19am UTC – Several customers confirmed that issue is resolved status changed to resolved.

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

Exclaimer Add-In Preview pane is displaying "Page Not Found". Customers are still able to utilise Add In auto apply.

This incident has been resolved.

Exclaimer Add-in is experiencing delays in loading and intermittently signatures are not added by default.

Issue: TLS Negotiation failed, Certification Invalid for US subscriptions. Incident Length: 6 hours and 43 minutes Incident Date: 04/12/2024, UTC 00:52 – 07:35 04/12/2024 Incident Status: Resolved ‌ **Summary** Customers reported encountering the error ‘TLS Negotiation failed, Certification Invalid’ when routing messages through Exclaimer’s Server-side system. This issue only impacted subscriptions located within US regions.  Not all servers were affected by this issue, so this did not impact all messages being sent through Exclaimer during the incident. ‌ Going forwards, a new process for reviewing and confirming certificate updates has been introduced to prevent similar issues in the future. ‌ **Root Cause** Due to an unexpected oversight, the certificate for US relays was not updated on all routing servers. Resulting in the previous certificate expiring and no longer being valid for mail routing. ‌ **Mitigation** Once the expired certificate had been identified within the infrastructure as not correctly updated and applied. The certificate was replaced with the renewed certification to resolve the issue. All other instances of the certificate were then reviewed and verified to have been correctly updated ‌ **Incident Timeline** 00:52 – Alerting advised to a failure to obtain a message response at times within the US region 01:29 – Initial investigation indicated that all endpoints were responding to requests and were accessible to engineering staff, suggesting full operation of the Exclaimer system. 01:45 – Investigation continued to confirm that traffic routing also did not advise of an issue. However, traffic flow to other US servers remained higher. 02:13 – Another full review indicated that the system was operating as expected with no errors being reported in the run up to the alert being generated. 02:25 – Alert was documented and prepared for pickup during main operational hours 03:40 – Support alerted team to customer facing reports of messages being rejected with ‘Certification Invalid’ 03:52 – Identified that the active certificate on US2 was showing as having recently expired. 04:06 – Engineering confirmed that no other services were also attempting to use the expired certificate. 04:21 – A full review of all servers and an update of US2 to ensure only the latest certificate was applied was completed. 04:31 – Engineering confirmed system recovery of the original alert, and an improvement of traffic flow between all servers. Incident moved into Monitoring status. 07:35 – Support confirmed that new reports of the issue had ceased, and existing reports confirmed the issue was no longer occurring.

This incident has been resolved.

We are continuing to investigate this issue.

We are investigating a possible service alert. Issue: TLS Negotiation failed, certificate invalid: SSL [Leaf certificate is expired] Next Update In: TBA

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

We are continuing to investigate this issue.

We are investigating a possible service alert. Exclaimer Cloud - Add-in is not applying signatures as expected on Outlook 365 Desktop(Classic), we are currently investigating the issue.

Issue: Crash Error log generated by the Add-in within Outlook for Mac Incident Length: 1 Hour 29 minutes Incident Date: 12/11/2024, UTC 16:29 – 17:58 Incident Status: Resolved ‌ **Summary** An update to the Exclaimer Add-in resulted in MacOS Outlook installations running the Add-in to report a crash when composing a message. There was no reported impact for customers using Server-side Signatures or Client-side on Windows OS’, or Outlook on the Web, or Mobile devices. ‌ **Root Cause** An update to the Exclaimer Add-in, to apply Client-side signatures, to add support for Nested App Authentication \(NAA\) resulted in MacOS 15 users running the Add-in to report a crash when composing a message in Outlook for Mac. Initial investigation has indicated that this looks to potentially be an issue within the Outlook for Mac application at this time, and this has been raised for further investigation with Microsoft.   **Mitigation** To resolve the issue, a roll back process to revert the update to support NAA began was enacted. This roll back was completed quickly and without further issues. Due to caching of Add-in data within local Outlook clients, continued monitoring of reports was done by our Support Team to confirm that the disruption to service was resolved for our impacted customers. An internal review of the testing and deployment has been undertaken in response to this, to determine improvements that may be made in these processes to prevent this from happening again. ‌ **Incident Timeline** 14:36 – v1.22.100.219 of the Exclaimer Add-in is deployed 16:30 – Reports from customers of Crashing being seen within Mac for Outlook when composing emails. 16:51 – Exclaimer Support confirm replication of the issue and the scope of the fault. 17:23 – Rollback of Add-in globally begins. 17:37 – Rollback is Completed for all regions 17:58 – Support confirm that impacted customers are no longer experiencing the issue.

On the 12th November at 16:29 UTC, an update for the Exclaimer Add-in was released that resulted in Mac devices to report a Crash when attempting to apply a signature. This issue was resolved by 17:37 UTC with a full roll back to the previous version of the Add-in. Customer using the Add-in on Windows OS', Outlook on the Web, or Mobile Devices where not impacted. Additionally Server-side signatures continued to apply as expected.

We can confirm our Exclaimer support phone lines are operational and accepting calls. We apologize for the inconvenience caused and have appreciated your patience and understanding throughout. Thank you

We have pin-pointed the issue and are working hard with our provider to resolve this issue, resolution is imminent. We appreciate your patience and understanding in the meantime, and encourage you to please raise a support ticket if you are in need of technical support via our online form found on our website. Our support team will respond as soon as possible to resolve. Thank you

Unfortunately we are currently experiencing issues with our third party phone service resulting in customers being unable to make calls into Exclaimer Support team. We are working hard with our provider to fix this issue, however is likely to be ongoing until 9:30 GMT. We appreciate your patience and understanding in the meantime, and encourage you to please raise a support ticket if you are in need of technical support via our online form found on our website. Our support team will respond as soon as possible to resolve. Thank you

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

Exclaimer engineers have confirmed that the reported issue has been fixed and the service is returning to normal status. Engineers continue to monitor the alert to ensure service.

We are investigating a possible service alert. Issue: User data syncs to Exclaimer subscriptions are failing with a 000000 error. Symptoms: Error 00000 encountered when trying to sync user data to Exclaimer Next Update In: TBA

This incident has been resolved.

We are monitoring the issue

The issue has been identified: Sectigo is currently down, and according to their status page, the problem is being worked on.

# Exclaimer Post Incident Report: Issue: Signatures not applying via server-side Incident Length: 7 Hours Incident Date: 17/09/2024, UTC 15:30 – 21:12 Incident Status: Resolved ### **Summary** Customers would find that signature templates, that have been applied through the groups option on the Senders tab would not apply.  The signature tester would advise that the user is not a member of the group. This issue impacted signature application for both Client-side and Server-side application. Cached configurations on local clients would ensure that for some customers, signatures would continue to be available based on the last review of the signature rules on the subscription. No mail flow was impacted due to the issue, all messages were successfully routed through Exclaimer’s infrastructure and received by their intended recipient\(s\). ### **Root Cause** Planned Improvement work was carried out across all regions to the Exclaimer service.  This improvement aims to remove ambiguity and prevent signature errors for renamed groups that are applied to signature templates. This was arranged to take place on the 17th of this month. An unexpected oversight resulted in the new identifier for groups not being adopted by signature rules automatically. ### **Mitigation** Once identified that the planned work was not having the desired result, the original task was halted, and a planned rollback process was initiated to return all environments to their original configuration at 16:13 UTC. All regions were rolled back to the original configuration at 21:12 UTC. An extended monitoring period confirmed no further reports being received after this time. ### **Incident Timeline** 15:15 – Initial reports of signatures failing to apply started to come through to the support team 15:30 – Alerted to high quantity of cases being raised through support channels. Identified possible issues relating to the planned improvement work. 15:38 – Investigation work begins across Engineering and Support Teams. 15:58 – Confirmed impacted customers across multiple regions. 16:00 – Status page published advising of issues. 16:13 – Remediation action of rolling back agreed. 16:44 – Rollback begins in all regions 16:45 – DE Region is roll back. 17:00 – Cause of issue identified within original work. Rollback continues. 17:44 – UAE Region is rolled back. 17:59 – CA Region is rolled back. 19:31 – EU region is rolled back. 19:54 – AU region is rolled back. 20:16 – US region is rolled back. 21:12 – UK region is rolled back. Incident Status moved to Monitoring. 18/09 09:24 – Incident Status moved to resolved

After a period of monitoring, engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

Exclaimer engineers have confirmed that the reported issue has been fixed and the service is returning to normal status. Engineers continue to monitor the alert to ensure service.

The issue has been identified and remediation is in progress.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Customers are experiencing issues with signatures not applying at this time. We have identified a possible cause and our Engineering Team are investigating. Mailflow is currently not impacted. We understand the inconvenience this may cause and assure you that the matter will be addressed as soon as possible. We will provide an update shortly for affected customers. Thank you for your patience and understanding.

All services are now back to normal levels, and messages are being imprinted by the server-side feature.

We have identified the issue with the server-side feature and have implemented a fix. We will continue to monitor.

We are investigating an issue with the signature application on our server-side feature. We will provide an update shortly with the next steps for affected customers. This issue is not affecting mail flow.

This incident is now confirmed to be fully resolved.

The database is now back online, and all services have returned to normal operation. Engineers continue to monitor.

We have identified the current issue as having been caused by an Azure database outage. Engineers have implemented a partial fix, that has resolved mail routing, analytics and feedback services. Imprinting and client side signatures remain degraded for customers using Salesforce integration. Data sync and onboarding remain unavailable until the database outage is resolved. We will continue to monitor and update.

We are currently investigating an issue caused by a database outage in Azure. More details to follow.

Between Jun 13, 9:16 pm and Jun 13, 10:29 pm and Jun 14, 11:29 pm – Jun 15, 12:09 am BST. A subset of customer mailflow limited to the CA region (approximately 5%), may have received a Non-Delivery Report (NDR) with the following information when their message was routed through one of Exclaimers mail-processing data centers: '550 5.7.1 Service unavailable, Client host [52.233.37.155] blocked using Spamhaus [...]' The IP in question was subsequently de-listed and Exclaimer engineers are engaging with Spamhaus to determine the root cause and understand the specific event that triggered the listing.

We have confirmed that the issue was resolved yesterday and will now close this alert.

Exclaimer engineers have confirmed that the reported issue has been fixed and the service is returning to normal status. Engineers continue to monitor the alert to ensure service. Next Update In: 3 hrs

We are investigating a possible service alert. Summary of Impact: We are currently receiving reports of customers not receiving signatures via the Outlook Add-In with the following message visible “Please Check your network connection” This issue is present across all Outlook client types (desktop, mobile and web) Our Teams are actively investigating the issue at hand, we appreciate your patience whilst we do so and will update further here. Next Update In: 3 hrs

Between 13:53 and 15:53 BST on 22/05/2024, we received reports from customers about emails containing unexpected characters appearing within the message body or around the Exclaimer signature. This issue only affected UK regional subscriptions. Our teams resolved this matter on the same day it was reported, and we have not experienced any further incidents since then. Currently, our teams are working on compiling a Root Cause Analysis (RCA) with more information about the cause, resolution, and prevention of this issue.

Between 08:27 UTC and 09:05 UTC customers may have experienced delays in message flow. The Exclaimer Cloud UI was inaccessible for a period during this time and was recovered by Exclaimer Engineers by 09:15 UTC The affected services have now returned to normal. Engineers continue to monitor the system to ensure service availability.

Service Status: Operational The fix has been deployed and the service is operational. Engineers are continuing to monitor the service. A full root cause analysis will be shared after the investigation. We appreciate your patience.

Our engineering team have identified a fix for this issue and are working hard to deploy this change to the service.

We are receiving reports of connection issues when onboarding Exclaimer Cloud with the Microsoft 365 environment. Our engineering team are actively looking into this issue and hopes to have this resolved quickly. Users may experience a failure to connect to Microsoft when running the Setup Wizard within Exclaimer Cloud. Please be aware that this should not impact currently configured subscriptions, and the imprinting of your emails will continue as normal. We apologise for any inconvenience this may cause you and thank you for your patience.

Engineers have confirmed that the previous service alert has been resolved. The maintenance window has completed and certificates have been successfully provisioning again. The service is now back to full health and is working as normal. Thank you for your patience.

We are continuing to monitor for any further issues.

Our certificate provider Sectigo is currently carrying out scheduled maintenance. This means that until complete, they are unable to provision new certificates. This is expected to last until 01:00 UTC 28/04. As a result of this, new office 365 tenants attempting to onboard will be unable to complete the mail routing wizard. The status of this maintenance event can be viewed on the sectigo status page here: https://sectigo.status.io/pages/maintenance/5938a0dbef3e6af26b001921/66229e8ef5286d053643a58f Once the maintenance window has cleared, Engineers will verify the status of certificate provisioning and update accordingly.

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

We are continuing to investigate this issue.

Customers may experience slowness or delays when attempting to access their subscription from the Exclaimer web portal. Signatures that are configured to be applied Client-side or Server-side will continue to be applied as expected. The cause of the delay is currently being investigated by our Engineering team. Further updates will be provided shortly.

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

Exclaimer engineers have rolled out a successful fix across all infrastructure by 14:40 UTC and have been monitoring the service. We will continue to monitor the alert to ensure service stability.

Behaviour: When attempting to send a message whilst using the Exclaimer Outlook Add-in, you may receive the following message: "Unable to download default signature. Status 401" Investigation: We have identified a cause of this issue and are in the process of rolling out changes to address it. Next Steps: Once the changes are fully rolled out across our infrastructure we will monitor for further reports and be working on a root cause analysis.

Engineers have confirmed that the previous service alert has been resolved. The maintenance window has completed and certificates have been successfully provisioning again. The service is now back to full health and is working as normal. Thank you for your patience.

Our certificate provider Sectigo is currently carrying out scheduled maintenance. This means that until complete, they are unable to provision new certificates. This is expected to last until 03:00 UTC 24/03. As a result of this, new office 365 tenants attempting to onboard will be unable to complete the mail routing wizard. The status of this maintenance event can be viewed on the sectigo status page here: https://sectigo.status.io/pages/maintenance/5938a0dbef3e6af26b001921/65f48a9351261305827a8ad2 Once the maintenance window has cleared, Engineers will verify the status of certificate provisioning and update accordingly.

Engineers have confirmed that the previous service alert has been resolved. The maintenance window has completed and certificates have been successfully provisioning again since 01:16 UTC The service is now back to full health and is working as normal. Thank you for your patience.

Our certificate provider Sectigo is currently carrying out scheduled maintenance. This means that until complete, they are unable to provision new certificates. This is expected to last until 0200 UTC 10/03. As a result of this, new office 365 tenants attempting to onboard will be unable to complete the mail routing wizard. The status of this maintenance event can be viewed on the sectigo status page here: https://sectigo.status.io/pages/maintenance/5938a0dbef3e6af26b001921/65e1d118413cbc0dce8df282 Once the maintenance window has cleared, Engineers will verify the status of certificate provisioning and update accordingly.

Issue: Between 20:58 UTC and 23:27 UTC a subset of Exclaimer infrastructure that processes messages received higher than usual levels of traffic Symptoms: This may have resulted in delays to some messages for customers if messages were routed to the affected hosts during the incident Engineers have identified and resolved the issue and message processing has now returned to normal

Certificates are being provisioned and collected. Whilst there is still the occasional small delay in retrieving the certificate, this is well within normal operations.

A fix for this issue has now been tested and deployed to all regions. Certificates should now be provisioned and collected, however there may be a delay between the request and collection. We will continue to monitor the situation.

Engineers have identified the issue and are currently working on a fix. Next update will be given once fix has been tested and approved.

We are currently investigating this issue.

The Sectigo maintenance has now ended, however this has surfaced an issue with retrieving certificates. Engineers are investigating the issue, and we will update at 2100 UTC. Please not this will not affect existing O365, Google workspace or client side only users.

Our certificate provider Sectigo is currently carrying out a datacenter switchover test. This means that until complete, they are unable to provision new certificates. This is expected to last until 2100 02/03. As a result of this, new office 365 tenants attempting to onboard will be unable to complete the mail routing wizard.

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

Exclaimer engineers have confirmed that the reported issue has been fixed and the service is returning to normal status. Engineers continue to monitor the alert to ensure service.

Reports of issues saving changes to signature rules have been reported, we are investigating this issue and are now rolling out a fix to restore full usability to the portal.

### **Outlook Add-In TImeout Increase** ### **Resolved**  #### **Summary of Impact:**   Between 14th January and 25th January, a trend appeared where users of the Exclaimer Outlook Addin experienced a delay when loading the Add-in when sending an email.  The following message was displayed to a subset of the Outlook Add-in users. This message is consistent with the Add-in taking longer than 5 seconds to complete the operation.   Other experienced behaviours may have been the loss of the ability to send mail and other various related pop-ups.   #### **Root cause:**  On the 19th of December, a new piece of code was introduced to the add-in for the ‘OnMessageSend’ event as part of the work to collect analytics around which signature was being applied to support the collection of feature usage metrics.  The first instance of this was reported on the 24th of December through our Customer Support Team; however, as the volume of reports was very low, this did not trigger our alerting.  #### **Analysis:**   Post-incident analysis has concluded the cause of the initial reports was due to issues identified with calls to the Microsoft authentication provider which have been observed to sometimes hang and fall to return when attempting to confirm the user's identity, causing the operation to timeout.  Analysis does not show any delays caused by calling into Exclaimer services, delays and subsequent notification appear to be caused by client-side issues before calling into Exclaimer services in the Cloud. This supports the conclusion above that authentication is the initial root cause. #### **Mitigation:**   We released a new version of the add-in on the 16th of January which removed the code suspected of causing the initial incident. Our Engineering Team after this point saw a drop in calls to the data endpoint.  Unfortunately, our Support Teams continued to see incidents appearing and therefore, were once again escalated. An attempt to further mitigate this issue was released on the 24th of January, this removed the event hook from the manifest file and the handler from the backing JavaScript files. Version 1.12.300.30.  It took a while for customers to unfortunately adopt the latest version due to Microsoft’s rollout as the latest version can take 12-24 hours to fully propagate through a Microsoft tenant to all its end users. Once end users were confirmed on the correct version, some were still experiencing the same issues. This was due to the backing JavaScript files having different caching, consequently, users on the latest version were still experiencing the same behaviours until clearing their add-in's cache.   Once manifest and backing JavaScript files aligned on customers' systems the issue was no longer reported.  #### **Next Steps:**  The following corrective actions have been identified.  * Status page ownership has been transferred to achieve smoother and faster communications.  * The authentication method used should be switched to Exchange Access Token, to match the method used by the Add-in on mobile to avoid hanging during authentication.  * Backing JavaScript files should be ensured to be compatible with previously released manifests.  * JavaScript version number to be added into the task pane of the Add-in to allow visual identification of when the latest version is not being run by an individual user.  * A long-term stable version of the Add-in to be provided, which can be side-loaded to allow any future issues to be worked around by the customer in conjunction with support.  * Improved alerting and automation have been created to ensure a faster reaction and allow more proactive outreach

Engineers have confirmed that the latest version of the add-in has resolved the issue. The service is now back to full health and is working as normal. Thank you for your patience.

An update to the add-in has been released, and we have had confirmation from the field that this has resolved the issue. We are continuing to monitor reports to ensure that we are happy the issue is fully resolved.

Behaviour: When attempting to send a message whilst using the Exclaimer Outlook Add-in, you may receive the following message: "Exclaimer Add-in is taking longer than expected" Clicking "Send Anyway" will allow the message to be sent. Investigation: We have released a new version of the Add-In (1.12.300.3) that removes the code that, we believe, is contributing to this behaviour. This new version will be available through the Microsoft Add-In Store and will automatically update in your environment. This can then take up to 24 hrs to propagate to all users. Next Steps: We will be working on an root cause analysis for this issue, and monitoring for further reports once the update has reached the affected customers

Engineers have confirmed that the previous service alert has been resolved. The service is now back to full health and is working as normal. Thank you for your patience.

The incident reported by our certificate provider has now been resolved, and certificates are being provisioned and Exclaimer engineers have confirmed that the reported issue has been fixed and the service is returning to normal status. Engineers continue to monitor the alert to ensure service. Next Update In: As events warrant

We are continuing to work on a fix for this issue.

We have identified an issue caused by our SSL certificate provider where new certificates are not currently able to be provisioned. This will prevent new signups to the server-side product. Client-side and existing customers are unaffected.

<b>Summary of Impact:</b> Between 16:30 UTC and 22:00 UTC on 30th of November 2023 customers with Google Workspace subscriptions may have experienced emails being deferred or rejected by Exclaimer's servers.</br></br> <b>Root Cause:</b> Exclaimer's servers were unable to return processed emails to Google Workspace due to a Google incident affecting Gmail. Further details on the incident can be found on the <a href="https://www.google.com/appsstatus/dashboard/incidents/Hjq8CpkhWroicw7R1wSh">Google Workspace Status Dashboard</a></br></br> <b>Mitigation:</b> Due to the cause of the delayed or rejected emails being an incident affecting Google’s servers, there was no action Exclaimer engineers could have taken to resolve the issue. However, as a precautionary measure, Exclaimer engineers enabled additional Google Workspace specific IPs to reduce the likelihood of throttling by Google servers once the incident was resolved.</br></br> <b>Next Steps:</b> We are working on improving internal processes to reduce the time taken to communicate to customers when third party incidents are affecting Exclaimer services.

<b>Summary of Impact</b>: Between Wednesday 18th October 22:00 UTC and Thursday 19th October 02:10 UTC customers in AU would have been intermittently unable to reach the Exclaimer Cloud UI and may also have experienced delays in Exclaimer's Outlook Add-in and mail flow. </br></br> <b>Root Cause</b>: Exclaimer engineers were carrying out on-going internal maintenance to standardise some configurations across our infrastructure. This began outside of operational hours as a precaution but took longer than expected. This possibility originally wasn't a concern as re-routing traffic even during operational hours is done routinely. During the maintenance operation however the SQL instances associated became overloaded. This resulted in a delay in both mitigation efforts and communicating the issue to Exclaimer customers as the triggering alerts were incorrectly attributed to the ongoing maintenance as false-positives. </br></br> <b>Mitigation</b>: Engineers progressed through the maintenance work as it would allow the infrastructure undergoing maintenance to be bought back into the available pool of resource. At 01:30 UTC the SQL instances were scaled significantly to match the demand in load and at 01:35 UTC the incident went live on the status page. Whilst scaling the SQL instances started the recovery, there was a backlog of data to process and so this recovery was not immediate. </br></br> At 01:55 UTC the maintenance had been completed and the updated infrastructure began accepting connections. </br> By 02:10 UTC the service load and operation had recovered to normal levels.</br></br> <b>Next Steps</b>: We take any incident seriously and shall be taking a close look at our maintenance and failover procedures to identify improvements to our current processes and technology to better cope with dynamic changes in our infrastructure scaling. We will also be re-evaluating our change management, alerting and escalation procedures in line with this incident.</br>

Engineers have confirmed that the previous service alert has been resolved. Whilst Microsoft engineers work on mitigations for the remaining affected hosts Exclaimer engineers have confirmed that stability has remained since the previous update. We are continuing to monitor the situation regarding the Azure SQL outage, but the service is now back to full health and is working as normal. Thank you for your patience.

Exclaimer engineers have confirmed that the affected resources have been fixed, and the service is returning to normal status. Engineers continue to monitor the alert to ensure service. Next Update In: As events warrant.

We are investigating a possible service alert. Exclaimer engineers have taken steps to ensure the continued availability of mailflow. Microsoft are still working to mitigate the issue and we are ensuring all other services remain functional Issue: The Exclaimer Cloud UI in US is presently being affected by an Azure SQL incident in East US (2LZ0-3DG). Symptoms: Customers will be unable to launch into the UI to manage their email signatures. Next Update In: 60 minutes or sooner

We are investigating a possible service alert. Exclaimer engineers have taken steps to ensure the continued availability of mailflow. Microsoft are working to mitigate the issue and we are ensuring all other services remain functional Issue: The Exclaimer Cloud UI in US is presently being affected by an Azure SQL incident in East US (2LZ0-3DG). Symptoms: Customers will be unable to launch into the UI to manage their email signatures. Next Update In: 60 minutes or sooner

We are investigating a possible service alert. Issue: The Exclaimer Cloud UI in US is presently being affected by an Azure SQL incident in East US (2LZ0-3DG). Exclaimer engineers have taken steps to ensure the continued availability of mailflow. Microsoft are working to mitigate the issue and we are ensuring all other services remain functional Symptoms: Customers will be unable to launch into the UI to manage their email signatures. Next Update In: 60 minutes or sooner.

We are investigating a possible service alert. Issue: The Exclaimer Cloud UI in US is presently being affected by an Azure SQL incident in East US (2LZ0-3DG). Exclaimer engineers have taken steps to ensure the continued availability of mailflow. Microsoft are working to mitigate the issue and we are ensuring all other services remain funcitional Symptoms: Customers will be unable to launch into the UI to manage their email signatures. Next Update In: 60 minutes or sooner.

We are investigating a possible service alert. Issue: The Exclaimer Cloud UI in US is presently being affected by an Azure SQL incident in East US (2LZ0-3DG). Exclaimer engineers have taken steps to ensure the continued availability of mailflow. Symptoms: Customers will be unable to launch into the UI to manage their email signatures. Next Update In: 60 minutes or sooner.

We are investigating a possible service alert. Issue: The Exclaimer Cloud UI in US is presently being affected by an Azure SQL incident in East US (2LZ0-3DG). Symptoms: Customers will be unable to launch into the UI to manage their email signatures. Next Update In: 60 minutes or sooner

<b>Summary of Impact</b>: Between 10:45 and 21:30 UTC customers would have been unable to reach the Exclaimer Cloud UI in AU due to an issue impacting multiple Microsoft Azure services in East and Southeast Australia. <b>Root Cause</b>: Microsoft encountered an issue with multiple services (issue tracking ID VVTQ-J98) which meant Exclaimer was unable to make connections to the SQL database which stores tenant data and other storage services that compliment various components of the UI. <b>Mitigation</b>: Exclaimer failed over all application traffic to the active-active datacenter pair within. The impact to SQL availability from this issue however meant that failing over was not possible for the UI. <b>Next Steps</b>: We are continuing to work to reduce the failover time window for the UI where possible to make it consistent with the rest of the application services.

<b>Summary of Impact</b>: Between the morning of Tuesday 27th 04:00 UTC and Wednesday 28th 13:40 UTC a small subset of customers may have experienced message processing delays due to an issue with an update to some of Exclaimer's infrastructure. <b>Root Cause</b>: A scheduled update caused an issue in how certificates used to handle tenant attribution with Microsoft 365 were loaded. This resulted in potential delays only for customers whose messages were handled by the same node in a short period of time. <b>Mitigation</b>: After being made aware of the issue, Exclaimer engineers identified suitable mitigation for the update at 12:28 UTC and had rolled out the mitigation to all affected infrastructure by 13:40 UTC. <b>Next Steps</b>: Updates to Exclaimer infrastructure are staggered to allow for easier mitigation of issues. This issue affected a small number of messages (0.12% of total message volume). This low number meant that our existing monitoring processes did not automatically detect the issue before the update was rolled out further. The incident was instead bought to our attention via customer reports. We are reviewing our monitoring processes to determine how we can improve our detection of sensitive changes in mailflow.

<b>Summary of Impact</b>: Between 08:50 and 12:25 UTC a subset of customers may have experienced message processing delays or errors in the Outlook Signatures Add-in <b>Root Cause</b>: Out metrics provider Datadog experienced an issue affecting their entire network, their platform was unable to process metrics and log information from their customers, including Exclaimer. Exclaimer utilise Datadog for all of our metrics/logs/apm, health checking, and performance reporting. These service level indicators (SLIs) are also our metrics of scale, with our auto-scaling infrastructure pulling these SLIs from Datadog, and deciding the correct scale to run at to meet customer demand. As we were missing these SLIs, the app was unable to scale automatically and therefore stayed at the scale when the incident began at approx 6am UTC. When users began to come online the limited resources were swamped and began to throw errors. <b>Mitigation</b>: Exclaimer SREs manually scaled the infrastructure and managed the health manually until Datadog resolved their issue. <b>Next Steps</b>: We are reviewing our response and how we utilise Datadog and wether this should be supplemented with additional SLIs obtained from other sources.

<b>Summary of Impact</b>: Between 07:12 and 08:36 UTC a subset of customers may have experienced message processing delays due to a Microsoft Azure Issue: VSG1-B90. <b>Root Cause</b>: Microsoft experienced a Global WAN networking issue - more information can be found <a href="https://status.azure.com/en-gb/status/history/">here</a>. This limited our ability to send and receive messages to/front Microsoft O365. <b>Mitigation</b>: Once Microsoft resolved the incident we were able to communicate with EXO correctly once more. <b>Next Steps</b>: We are reviewing our response and scaling metrics in order to more quickly respond to similar issues in the future.

<b>Summary of Impact</b>: Between 12:58 and 14:06 UTC customers would have been unable to reach the Exclaimer Cloud UI. <b>Root Cause</b>: A change was made to update the certificate used by the Cloud UI as it was due to expire. A deployment issue meant that the certificate deployed was mismatched with the hostname of the site. This meant that users would receive an SSL error when attempting to launch the site. The deployment operation cleaned up the old certificate thus increasing the incident duration. <b>Mitigation</b>: Exclaimer engineers updated the certificate SAN and redeployed the affected service. <b>Next Steps</b>: We have made changes to the deployment mechanism so that we check the connection status of the site prior to any clean up operations taking place.

<b>Summary of Impact</b>: Between 15:29 and 16:36 UTC customers would have not been able to launch a subscription from the portal due to a backend database issue</a><br> <b>Root Cause</b>: An Exclaimer platform issue affecting our database resulted in an inability for the Portal Data Service to pull lists of active subscriptions for signed in users. This issue was caused by a long running query triggered by our Ops team.<br> <b>Mitigation</b>: The long running query was cancelled and the service was brought back <br> <b>Next Steps</b>: We are reviewing our request for change process to better catch a potential issue such as this in the future, the query was approved to be run but it was categorised as low risk. We have made changes to our monitoring to ensure that an issue with database performance is more rapidly detected. We have also made changes to convert our failure notifications from the services to more rapidly detect higher percentages of failing requests.

<b>Summary of Impact</b>: Between 04:15 and 06:12 UTC customers would have been unable to reach the Exclaimer Cloud UI in the EU due to an issue with Microsoft SQL in West EU. <b>Root Cause</b>: Microsoft encountered an issue with Azure SQL (Issue tracking ID 3TBL-PD8) which meant Exclaimer was unable to make connections to the SQL database which stores tenant data. <b>Mitigation</b>: Exclaimer failed over all application traffic to the active-active datacenter pair within , however the UI does not currently support the same immediate failover RTO, the issue was mitigated by Microsoft prior to failover being completed. <b>Next Steps</b>: We are working to reduce the failover time window for the UI to make it consistent with the rest of the application services.

<b>Summary of Impact</b>: Between 08:00 and 12:00 UTC a subset of customers may have experienced issues with messages being queued for delivery due to an Exchange Online Incident: EX397744<br> <b>Root Cause</b>: A Microsoft M365 DNS resolution issue resulted in significantly increased errors during communication between Exclaimer Cloud and EXO as we were not able to resolve the MX records of the M365 customers we had to route back to. This resulted in messages being rejected and retried by Exchange Online. <b>Mitigation</b>: Once Microsoft resolved the incident we were able to communicate with EXO correctly once more. Exclaimer Cloud struggled to process the backlog of messages created by the Microsoft incident which extended the issue period by approx. 60mins. <br> <b>Next Steps</b>: We are reviewing our response and scaling metrics in order to more quickly respond to similar issues in the future.

<b>Summary of Impact</b>: Between 07:00 and 11:30 UTC a subset of customers may have experienced issues with messages being queued for delivery due to an Exchange Online Incident: EX394347<br> <b>Root Cause</b>: A Microsoft M365 platform issue (caused by a power outage and failover issue, for full info see EX394347) resulted in significantly increased errors during communication between Exclaimer Cloud and EXO. This resulted in messages being rejected and retried by Exchange Online. <b>Mitigation</b>: Exclaimer Cloud scaled dramatically and within one hour we were able to scale enough to process every message (and the queue), however average message duration was still ~8x higher than usual for the remainder of the issue window. The incident was only fully mitigated once Microsoft resolved their underlying issue as described within Microsoft Incident: EX394347<br> <b>Next Steps</b>: We are reviewing our response and scaling metrics in order to more quickly respond to similar issues in the future.

<b>Summary of Impact</b>: Between 06:30 and 08:00 UTC customers would have been unable to reach all public Exclaimer Cloud endpoints which proxy their traffic through Cloudflare. This affected, but is not limited to, Portal, Cloud UI, Details Editor, Outlook Signatures Service, Analytics - it did not affect mail flow and server side imprinting. <b>Root Cause</b>: Exclaimer utilise Cloudflare for all external traffic management, this service provides Exclaimer with increased reliability, performance and security through their global CDN and associated services. During this time window Cloudflare suffered a serious outage, documented in full <a href="https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/">here</a>, this issue affected a large percentage of global internet traffic. <b>Mitigation</b>: The issue was only mitigated once Cloudflare resolved their issue <b>Next Steps</b>: We have reviewed our use of Cloudflare, and believe that on balance, their service still ensures that we can provide our customers with a higher uptime SLO, better reliability, and security guarantees than we would be able to offer without it.

<b>Summary of Impact</b>: Between 21:30 and 22:35 UTC a subset of Google Workspace customers may have experienced issues with messages being queued for delivery due to a Google Workspace <a href="https://www.google.com/appsstatus/dashboard/incidents/8C9piNvBvjPKodwHU9H2">Incident</a><br> <b>Root Cause</b>: A Gmail platform issue (described briefly by Google on their <a href="https://www.google.com/appsstatus/dashboard/incidents/8C9piNvBvjPKodwHU9H2">status page</a>) resulted in both significantly increased errors during communication between Exclaimer Cloud and Gmail, but also significantly reduced traffic sent out from Google to Exclaimer. This resulted in messages being rejected and retried by Google Workspace. The increased error rate during the incident, as well as the sharp drop off in messages sent (and then the proceeding sharp increase in messages sent to us - this is the 'wave' affect as a result of fewer emails being sent in the proceeding 60mins being retried and sent successfully) can be seen in Fig.1 <br> Fig.1 <img src="https://cdn.exclaimer.com/IncidentReports/220428rca.png"/><br> <b>Mitigation</b>: The incident was only fully mitigated once Google resolved their underlying issue. <br> <b>Next Steps</b>: We are reviewing our response and scaling metrics in order to quickly respond to similar issues in the future.

<b>Summary of Impact</b>: Between 13:00 and 15:45 UTC a subset of customers who sent messages through the AU1 datacenter may have experienced an issue relating to missing user data within signatures. Messages/requests sent through our AU2 datacenter would have been unaffected.<br> <b>Root Cause</b>: A change was made to the data reading service in our AU1 datacenter. This change inadvertently caused the service to reference an alternate table storage, used for different system services within the AU1 datacenter. This was not enough to cause the built-in readiness checks to fail (as the storage account was valid, but it did not contain user data) so the pods started. Requests made to the data service were not then able to find the data requested and therefore no data was passed back to the imprinting agent. This type of failure meant that the canary deployment checks which trigger roll back did not fail, as there was no specific failed requests/performance impact. <b>Mitigation</b>: Our system wide end-to-end monitoring alerted the SRE team to the issue who discovered the root cause and manually rolled back the deployment <br> <b>Next Steps</b>: We are reviewing both the readiness check and rollback mechanisms within our deployment pipelines to fully understand why this issue did not trigger the rollback and what more we can do to guarantee the success of the deployment.