Historical record of incidents for SYNAQ
Report: "Investigating Issues with Sending Mail from Outlook"
Last updateWe are currently aware of and investigating an issue affecting the ability to send emails via Microsoft Outlook on the CloudMail platform. Our engineering team is actively working to identify the root cause and implement a resolution as quickly as possible. Please note: Webmail access remains fully operational and unaffected at this time. We will provide further updates as we make progress on the investigation. Thank you for your patience and understanding.
Report: "CloudMail Integrated Archives Delayed Delivery"
Last updateDear SYNAQ Clients, We are pleased to inform you that the issue affecting the SYNAQ CloudMail Integrated Archives service has been successfully resolved. The service is now operating normally, and all mail queues have been cleared. Our engineers will continue to monitor the system closely to ensure ongoing stability and optimal performance. We sincerely appreciate your understanding and patience throughout this time. If you have any further questions or need assistance, please don’t hesitate to reach out to our support team. Kind regards, SYNAQ Technical Team
Dear SYNAQ Clients, Our engineers have implemented a fix for the SYNAQ CloudMail Integrated Archives service. The affected mail queues are now being processed, and the backlog is being cleared. Our team is actively monitoring the archive mail delivery performance to ensure the backlog clears as quickly as possible. We appreciate your patience and will continue to provide updates as the situation progresses. Kind regards, SYNAQ Technical Team
Dear SYNAQ Clients, The SYNAQ CloudMail Integrated Archives service is still experiencing delays. Our engineers have identified the root cause of the issue and are actively working to resolve it as quickly as possible. Again, this incident has no impact on mails to or from users mailboxes. We sincerely appreciate your patience and understanding as we work to restore normal service. Our next update will follow in 2 hours. Kind regards, SYNAQ Technical Team
Dear Clients, SYNAQ CloudMail Integrated Archives are currently experiencing an incident where by mail destined for for related archives ( CloudMail Archive Mailboxes ) are queued and delayed for delivery. This is identified by the "Warning: message delayed for 24/48 hours" Notification Mail. Engineers are investigating this as a matter of urgency. This incident has no impact on mails to or from users mailboxes. We will send our next update in 60 minutes.
Report: "Cloud Mail Network Error"
Last updateDear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service will soon return to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
We are continuing to investigate this issue.
Dear SYNAQ Client , We have identified a major Cloud Mail incident where clients are experiencing intermittent failures on Zimbra Cloudmail. Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise for the inconvenience caused. Sincerely, SYNAQ Technical Team
Report: "Securemail UI - Incident - Database isssue"
Last updateDear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our engineers have both identified and resolved the Securemail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our engineers are still working on the incident as a matter of urgency and will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our engineers have identified the cause of the incident and are working on resolving the matter. We will continue to provide updates as they become available. We apologise for the inconvenience caused and would like to assure you that this is being treated as a matter of importance and urgency. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client We have identified a major Securemail UI incident where clients are experiencing a database issue - impacting message listing data. Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
Report: "Service Disruption - SYNAQ Cloud Mail"
Last updateDear SYNAQ client, We have identified and resolved the current network accessibility issue, which was due to a data centre incident being experienced by our upstream provider. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Services to major Data centres have been restored and operating normally. We apologise for the service disruption and remain committed to providing you with the exceptional service you have come to expect from SYNAQ. We will continue to monitor the service. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, We are experiencing a service issue on the SYNAQ Cloud Mail platform. We apologise for this inconvenience. Our engineers are looking into the problem and should have us back to normal in the shortest time possible! Sincerely, SYNAQ Technical Team
Report: "CloudMail error_Mobile App access"
Last updateDear Valued Client, We are pleased to announce that the Cloud Mail mobile app performance issue has been resolved and the service is recovering. We will continue to monitor performance. We sincerely apologise for any inconvenience this may have caused and appreciate your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the root cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on Desktop, Webmail and phone browser. We are currently deploying a fix to restore impacted services. We will provide the next update once we recieve feedback from the upstream vendor. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the root cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on Desktop, Webmail and phone browser. We are currently deploying a fix to restore impacted services. We will keep you updated on the progress on an hourly basis. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the root cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on Desktop, Webmail and phone browser. We are pleased to inform you that we have received a potential solution (fix) from our upstream vendor. We are currently deploying and thoroughly evaluating this fix to assess its effectiveness in restoring impacted services. We will keep you updated on the progress and anticipated timeframe for complete service restoration on an hourly basis. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on desktop, Webmail and phone browser. We sincerely apologise for any inconvenience this may cause while our engineering team continue to resolve the issue. Follow up updates will be provided hourly. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on desktop, Webmail and phone browser. We sincerely apologise for any inconvenience this may cause while our engineering team continue to resolve the issue. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the root cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on Desktop, Webmail and phone browser. We sincerely apologise for any inconvenience this may cause while our engineering team continue to work on a fix for this issue. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified the cause of the Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on desktop, Webmail and phone browser. We sincerely apologise for any inconvenience this may cause while our engineering team resolve the issue. Thank you for your patience and understanding. Sincerely, SYNAQ Team
We are continuing to investigate this issue.
Dear Valued Client, We have identified a minor Cloud Mail incident where clients are experiencing degraded performance on the mobile application. Email access is still operational on Desktop, Webmail and phone browser. We sincerely apologise for any inconvenience this may cause while our engineering team investigate. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Report: "Intermittent Cloud Mail error, blank email issue on MS Outlook"
Last update**Subject: Cloud Mail Update: Intermittent Cloud Mail sending Issue, Resolved** Dear Valued Client, We're pleased to inform you that the temporary issue affecting email sending in Microsoft Outlook for some Cloud Mail users has been resolved! As you know, a recent MS Office update caused a small number of emails to be impacted. We understand the importance of seamless communication, and we're happy to announce that Zimbra and Microsoft have collaborated to release a patch fixing this issue. We appreciate your patience and understanding throughout this process. **For technical teams:** The implementation guide is available in **Appendix A** of this communication. **Monitoring the Fix:** While we're confident the patch will address the issue effectively, we kindly request your assistance in monitoring its performance when you return from the festive season break. Should you encounter any further issues with sending emails from Outlook, please don't hesitate to contact us for further assistance. **Thank you for your cooperation.** We hope you have a relaxing and enjoyable holiday season! Sincerely, SYNAQ Team ----------------------------------------------------------------------------------------------------------- **Appendix A: A guide for installing the new ZCO build:** **Before You Begin:** * Outlook Version: Make sure you know whether your Outlook installation is 32-bit or 64-bit. This information is essential for downloading the correct ZCO version. * Administrative Permissions: You'll need administrative permissions on your computer to install software. * Close Outlook: If Outlook is running, close it before proceeding. **Steps:** 1. **Uninstall Old ZCO \(if applicable\):** * If you have a previous version of ZCO installed, uninstall it first from your computer's Control Panel. 2. **Download the New ZCO Build:** * Download the ZCO installer file that matches your Outlook version \(32-bit or 64-bit\) from the designated download location. * The Zimbra Connector Outlook download can be found here: [https://www.zimbra.com/product/addons/zimbra-connector-for-outlook-download/](https://www.zimbra.com/product/addons/zimbra-connector-for-outlook-download/) 3. **Install ZCO:** * Double-click the downloaded ZCO .msi file. * Click Run to start the installation. * Click Next on the initial screen. * Read the license agreement and click I accept the terms in the License Agreement. * Click Next. * Click Next again to begin the installation. * When the installation completes, click Close. 4. **Create a New Outlook Profile:** * Open Mail from the Control Panel. * Click Show Profiles. * Click Add to create a new profile. * Select Manual Setup and then Other. * In the Other dialog box, ensure Zimbra Collaboration Server is selected and click Next. 5. **Enter Account Information:** * Server Name: cloudmail.synaq.com * Use Secure Connection: Check this box. * Email Address: Enter your full email address. * Password: Enter your mailbox password. 6. **Complete Installation:** * Click Apply to finish setting up the new profile. **Now you're ready to use Outlook with your Zimbra account!** Should you require additional assistance please contact Support on [011 262 3632]() or email [support@synaq.com](mailto:support@synaq.com) .
Dear Valued Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing intermittent errors on outgoing mails. Due to a recent MS Office update, some emails may be delivered as blank. We sincerely apologise for any inconvenience this may cause. Temporary Solutions: a) To ensure uninterrupted communication while continuing to use MS Outlook, we recommend, composing an email, saving it and then proceed to send, while our teams work on a permanent fix. b) Or should you prefer a web-based option, we recommend using SYNAQ Webmail while our teams work on a permanent fix. c) Alternatively, downgrade your MS Outlook to version “16924.20150”, ensuring that “auto updates” is not selected. The step-by-step guide on how to downgrade can be found here; https://kb.4team.biz/how-to-downgrade-outlook-version/. Permanent Solution: Zimbra are actively collaborating with Microsoft to release a patch that will fix this issue. The teams are diligently testing to ensure stability and effectiveness. We will closely monitor progress and communicate a confirmed release date as soon as it becomes available. We understand this disruption may be frustrating, especially during this busy period. If you have any questions or need assistance transitioning, please don't hesitate to contact us. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Dear Valued Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing intermittent errors on outgoing mails. Due to a recent MS Office update, some emails may be delivered as blank. We sincerely apologise for any inconvenience this may cause. Temporary Solutions: a) To ensure uninterrupted communication while continuing to use MS Outlook, we recommend, composing an email, saving it and then proceed to send, while our teams work on a permanent fix. b) Alternatively, should you prefer a web-based option, we recommend using SYNAQ Webmail while our teams work on a permanent fix. Permanent Solution: We are actively collaborating with Zimbra to release a patch that will fix this issue. While the patch is being rigorously tested, we don't have an exact release date yet. We will keep you updated on the progress. We understand this disruption may be frustrating, especially during this busy period. If you have any questions or need assistance transitioning, please don't hesitate to contact us. Thank you for your patience and understanding. Sincerely, SYNAQ Team
Report: "Connectivity errors on Webmail"
Last updateOn Tuesday, 29 August 2023, 07:05, Cloud Mail experienced network instability for a subset of clients trying to access the SYNAQ Webmail platform, affecting incoming and outgoing mail. The following concerns were raised. * Slow response on incoming and outgoing mail, and; * Server error when accessing Webmail. The resultant impact of the event was that certain users who were using Webmail experienced a slow response or a server error message on incoming and outgoing mail. The root cause of this event was due to degraded connectivity performance across the Cloud Mail platform impacting Webmail services. To resolve this incident, our engineers repaired the connectivity problems and restarted selected Cloud Mail services. This immediately resolved the degraded state and restored the platform to optimal functionality. As a result of the incident, SYNAQ will implement the below remedial actions: · The engineers will continue to monitor network connectivity to the Webmail platform.
Dear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely SYNAQ technical team
Dear SYNAQ Client, Our engineers have identified the cause of the error and are resolving as a matter of urgency. We will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ technical team
Dear SYNAQ Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing intermittent failures on Webmail. Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise for the inconvenience caused. Sincerly, SYNAQ technical team
Report: "Slow response - SYNAQ Cloud Mail"
Last updateOn Monday, 31 July 2023, 08:25, Cloud Mail experienced a slow response on SYNAQ Webmail platform, with incoming and outgoing mail. The following concerns were raised. * Slow response on incoming and outgoing mail, and; * Server error when accessing Webmail. The resultant impact of the event was that a subset of users who were using Webmail experienced a slow response on incoming and outgoing mail. The root cause of this event was due to degraded memory performance across the Cloud Mail Proxy stack. This resulted in slow connections on email when trying to login via Webmail. To resolve this incident, our engineers removed one proxy at a time from the cluster, restarted the relevant proxy services on it and added it back to the cluster. This immediately resolved the degraded state and restored the platform to normal operations. Upon investigation of the root cause of the incident the team at SYNAQ will implement the below remedial actions: · Updating the existing monitoring for memory swapping to perform remediation actions at a much earlier time.
Dear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Please ensure to log out and back into your account to resume services. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client Our engineers are still investigating as a matter of urgency and will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
We are continuing to investigate this issue.
Dear SYNAQ Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing slow email responses on webmail. Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise for the inconvenience caused. Sincerely, SYNAQ Technical Team
Report: "SYNAQ Cloud Mail - Intermittent SMTP (Outgoing Mail) Sending Failures"
Last updateOn Thursday, 6 July 2023, 15:25, Cloud Mail experienced intermittent sending failures on the SYNAQ Webmail platform, with outgoing mail. The following concerns were raised: * Slow response when sending email which contains attachments, and; * Emails stuck in Outbox. The resultant impact of the event was that a subset of users who were using Zimbra connector for Outlook were experiencing either a slow response on webmail or ActiveSync and/or timeout errors when sending emails containing large attachments. The root cause of this event was due to a degraded connection state between the load balancer and the mail cluster. This resulted in slow connections on email when trying to login via HTTPS, POP3/S, IMAP/S, SMTP/S, Zimbra connector for Outlook. To resolve this incident, we moved all sessions to the secondary load balancer. This immediately resolved the degraded state and restored the platform to optimal operations. Upon investigation of the root cause of the incident, the SYNAQ team will implement the below remedial actions: · New Monitoring to be implemented to detect any failed states on the Load-balancer, and; · Further streamlining the internal incident management process.
Dear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our engineers are working together with our upstream providers to further investigate the incident. We will continue to provide updates as they become available. We apologise for the inconvenience caused and would like to assure you that this is being treated as a matter of importance and urgency. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our engineers are still investigating as a matter of urgency and will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing intermittent failures on Sending Outgoing Mails. (Mails stuck in outbox, mail sending failures, cannot upload attachments and send). Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
Report: "Cloud Mail - Slow Webmail and Incoming Mail Delivery"
Last update**Summary and Impact to Clients** From 09:34am until 19:49pm on the 15th of December 2022, the SYNAQ Cloud Mail platform experienced an incident resulting in slow incoming mail delivery and webmail access for a subset of clients. More specifically, the affected users experienced intermittent slow access to webmail services, as well as intermittent delays that impacted incoming mail delivery to inboxes. **Root Cause and Solution** The root cause of the incident was a failed write/read cache in one of the redundant RAID array controllers in the Cloud Mail storage network. This resulted in a disk I/O bottleneck affecting the mail store servers attached to that particular RAID array. This bottleneck then also caused mail delivery attempts to queue and as a result mail delivery delays were experienced for some users. During the incident and before the final solution was implemented, SYNAQ engineers re-balanced certain disks to use alternative I/O access channels as an interim solution to optimise for the degraded performance. To completely resolve the issue, the faulty cache component was replaced on the Monday following the incident. This returned the affected RAID storage array to optimum performance, negating the need for the interim solution originally put in place. **Remediation Actions** _**Short Term Actions**_ SYNAQ will ensure that we keep raid cache components in stock to ensure speedier component swap out \(Due End January 2023\). _**Medium Term Actions**_ SYNAQ has already begun the process of moving mail store data from current storage technology to next-generation storage, which features much higher I/O capacity and fault tolerance. This project is ongoing and we estimate completion in the next six months.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality. SYNAQ Technical Team
Dear Clients, Our engineers have implemented a fix for the SYNAQ Cloud Mail platform and are monitoring the mail delivery performance in order to ensure mail backlogs clear as soon as possible. SYNAQ Technical Team
Dear Clients, Our engineers are still working on the backlog of delayed mail related the ongoing SYNAQ Cloud Mail issue. Mail is still flowing but intermittent delays stills persist. SYNAQ Engineers will be working continuously to ensure optimal Cloud Mail performance in the coming hours SYNAQ Technical Team
Dear Clients, Our engineers are still working on the backlog of delayed mail related the ongoing SYNAQ Cloud Mail issue. Mail is flowing but intermittent delays stills persist. SYNAQ Technical Team
Dear Clients, Our engineers are still working on the backlog of delayed mail related the ongoing SYNAQ Cloud Mail issue. SYNAQ Technical Team
Dear Clients, Our engineers are still working on the resolution of the SYNAQ Cloud Mail issue. This is being treated as a matter of urgency. SYNAQ Technical Team
Dear Clients, Our engineers have identified the problem causing the SYNAQ Cloud Mail performance issues and are working on a resolution. SYNAQ Technical Team
Dear Clients, SYNAQ Cloud Mail is currently experiencing a degradation in performance where webmail access (reading emails) may be slow and incoming mail delivery may be delayed - for some users. Engineers are investigating this as a matter of urgency. SYNAQ Technical Team
Report: "SYNAQ Cloud Mail - Intermittent Authentication Failures"
Last updateDear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality. Sincerely, SYNAQ Technical Team
Dear Clients, Our engineers have identified the problem on the SYNAQ Cloud platform and have implemented a fix. Engineers are monitoring the service stability for the next one hour We will send our next update in 60 minutes. Sincerely, SYNAQ Technical Team
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we are treating this as a top priority. We will send our next update in 60 minutes. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, We are experiencing an intermittent Send/Receive authentication issue on the SYNAQ Cloud Mail platform. Users may experience errors appearing on their mail clients when some authentication attempts time out. Our engineers are investigating the problem. Sincerely, SYNAQ Technical Team
Report: "General Services Incident - Intermittent Network Access"
Last updateDear SYNAQ Client, The network accessibility incident has been confirmed as resolved by our upstream provider. All SYNAQ platforms are thus operating optimally. We appreciate your understanding and continued support. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our upstream provider has implemented a fix for the network accessibility incident and we are monitoring the service stability. We will provide further feedback on this status in the next 60 minutes. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, Our upstream provider is still working on the resolution of the network accessibility incident as a matter of urgency. We would like to assure you that this is our top priority and will continue to provide feedback as it becomes available. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, We have identified that the current network accessibility issue is due to a data centre incident being experienced by our upstream provider. Whilst no ETA has been made available, we are managing this issue as a matter of urgency with our provider and will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
Dear SYNAQ Client, We are experiencing a network accessibility issue on all SYNAQ platforms, due to an upstream provider incident. We apologise for the inconvenience caused and would like to assure you that our engineers are working with the upstream provider to resolve this issue as a matter of urgency. Sincerely, SYNAQ Technical Team
Report: "Service Disruption - SYNAQ Cloud Mail"
Last update# Post Mortem Report The purpose of this post-mortem is to detail the investigations and subsequent findings of the impact of Cloud Mail incident of May 26 2022. ## **Background** * On Zimbra, we perform regular proactive maintenance on our LDAP servers. * This involves reloading of the LDAP Databases on all masters and replica servers using a backup of one of the master databases. * This is done quarterly to maintain optimal performance of the LDAP service. * This process has been performed over the last 3 years in conjunction with our vendor without issue. **On Wednesday 25th May 2022:** * An LDAP maintenance change was performed on the evening of the 25th at 20:00. * A SYNAQ Engineer was working with our vendor engineer for this change and a miscommunication occurred between them, which resulted in the incorrect backup being applied \(it was a legacy backup from January 18 2022\), instead of current backup taken from a master that evening. **On Thursday 26th May 2022: Authentication Failures Experienced** * \[07:00am\]: We received reports of a subset of clients that were experiencing authentication issues \(i.e. could not login to their mailbox\). * Our engineers investigated the issue and we uncovered that we were working on incorrect version of the LDAP database * \[8:03am – 9:31am\]: SYNAQ engineers took the LDAP system down for the purposes of restoring the most current version of the backups to all LDAP servers and this reloading to approximately 90 minutes. * Authentication services were restored at 09:31 AM. ## Remedial Actions **Immediate – 0-3 months:** * Improve and verify standard operating procedure for LDAP optimisation process with additional test cases. * Build an additional monitoring alert that allows us to detect anomalous changes in expected data found in LDAP after a reload process is performed. **Long term: 6 – 12 months:** * Zimbra 9 – LDAP fixes and improvements.
Dear Clients, The SYNAQ Cloud Mail service is restored. We apologise once again for any inconvenience caused. Sincerely, SYNAQ Technical Team
Dear Clients, Our engineers have implemented a fix for the SYNAQ Cloud Mail incident and are monitoring the service stability.
Dear Clients, Our engineers are still working on the resolution of the SYNAQ Cloud Mail Incident. This is being treated as a matter of urgency. We will send our next update in 30 minutes and we are still on track for the ETA for resolution in 10 minutes.
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail incident and are working on a resolution. We will send our next update in 30 minutes and the ETA for resolution will be 45 minutes.
Dear SYNAQ Client, We are experiencing an authentication issue on the SYNAQ Cloud Mail platform. We apologise for this inconvenience. Our engineers are looking into the problem and should have us back to normal in the shortest time possible! Sincerely, SYNAQ Technical Team
Report: "SYNAQ Cloud Mail Incident - 12/04/2022"
Last updateSummary and Impact to Customers On Tuesday 12th April 2022 from 12:06 to 13:22, SYNAQ Cloud Mail experienced an authentication incident affecting a subset of users. The resultant impact of the event was that this subset of users were unable to authenticate and access their mailboxes. Root Cause and Solution The root cause of this event was due to a Cloud Mail mailbox store becoming unresponsive. Due to the unresponsive server, it was not able to process authentication requests for any of the users on this store. To resolve the issue initially, we rebooted the affected store, but it did not boot up, because the server was unable to load the operating system as it could not find the boot information. We identified and resolved a firmware issue related to VMware SCSI drivers which prevented the VM from booting the operating system disk. Remediation Actions * We have identified other VMs which could potentially be exposed to the same risk, and will be scheduling time to address them.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident where one of our stores are not responding. This will cause certain POP and IMAP users to not be able to access their mail. Engineers are investigating this as a matter of urgency. We will send our next update in 60 minutes.
Report: "Cloud Mail Incident - 25/01/2022"
Last updateSummary and Impact to Customers On Tuesday 25th January 2022 from 11:30 to 16:38, SYNAQ Cloud Mail experienced an authentication incident affecting a subset of users. The resultant impact of the event was that certain users were unable to authenticate and access their mailboxes. Root Cause and Solution The root cause of this event was due to Virtual Machine \(VM\) backup snapshots on two Cloud Mail mailbox stores not being removed as part of a maintenance task. As part of our routine nightly backup processes, or when server maintenance is being performed, backups snapshots are created for each VM to ensure quick roll back and minimal impact to clients should there be an issue during maintenance or otherwise. Once a maintenance task has been completed the snapshots are then removed manually. If snapshots are not removed the performance of the VM slowly degrades over time. Human error resulted in snapshots not being removed from two of the stores after maintenance was performed last week Thursday. Whilst the removal of snapshots is not normally a problem in isolation, a further error occurred with our monitoring scripts that look for this issue – they did not detect that these VM’s had old out-of-date snapshots and therefore no alerts were raised. As such, these snapshots started to impact the performance of the stores on Monday. To resolve this degradation, the snapshots were removed on Tuesday morning. Five of the seven snapshots removed successfully, however, two of the snapshots did not complete their consolidation step and this degraded the VM’s performance even more forcing them to enter an unresponsive state. To resolve this issue, a manual consolidation of both VM’s needed to be run. Due to the business day load and performance of these VM’s this took a couple of hours to complete. Remediation Actions • Process improvements will be made to the existing maintenance process to ensure that the snapshot removal task is triple checked for completion. • Snapshot monitoring checks to be repaired and updated to alert for old snapshots. • Snapshot removal to only take place outside of business hours despite possible degradation of VM performance.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and all services are running optimally.
Dear Clients, Our engineers are still working on the resolution of the SYNAQ Cloud Mail Incident. This is being treated as a matter of urgency. Half of affected users have already recovered. We will send our next update in 60 minutes
Dear Clients, Our engineers are still working on the resolution of the SYNAQ Cloud Mail Incident. This is being treated as a matter of urgency. Half of affected users have already recovered. We will send our next update in 60 minutes
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail incident and are working on a resolution. We will send our next update in 60 minutes
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident where a subset of users are unable to log into their mailboxes. Engineers are investigating this as a matter of urgency. We will send our next update in 60 minutes.
Report: "Cloud Mail Incident - 13-10-2021"
Last updateSummary and Impact to Customers On Wednesday 13th October 2021 from 12:03 to 14:06, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that users received an authentication pop-up message when trying to login and experienced slow access to mail. Root Cause and Solution The root cause of this event was due to an abnormal amount of modification requests being received from the SYNAQ API. The increased writes and load on the master LDAP server prevented the LDAP replicas from being able to replicate the LDAP changes from the master. Since LDAP replicas are responsible for servicing requests for authentication and mail delivery, and were unable to sync correctly from the master LDAP server in this instance, authentication and mail delivery requests were not processed. To resolve this issue, the primary server and all replicas had to be restarted to re-establish the syncing between LDAP servers. Once all the LDAP servers were in sync, authentication requests were once again being processed and users could access their mail. Remediation Actions • More stringent limits on the API are to be put in place to limit the number of requests that can come in from a single IP address to prevent increased load on the LDAP servers. • The SYNAQ infrastructure teams are working in collaboration with the 3rd Party vendor to improve and harden our LDAP configuration.
Dear Clients, The SYNAQ Cloud Mail issue has been resolved and the service has returned to optimal functionality.
Dear Clients, The SYNAQ Cloud Mail issue has recovered. Our engineers are monitoring the service stability. We will provide a further update in 30 minutes
Dear Clients, Engineers have identified the Cloud Mail issue and are working on the root cause as a matter of urgency. Users are currently able to access their mailboxes. We will provide a further update in 30 minutes
Dear Clients, SYNAQ Cloud Mail is still experiencing an issue where users are unable to access their mailboxes. Engineers are investigating the issue as a matter of urgency. We will provide an update in 30 minutes
Dear Clients, SYNAQ Cloud Mail is currently experiencing an issue where users are unable to access their mailboxes. Engineers are investigating the issue as a matter of urgency. We will provide an update in 30 minutes
Report: "Cloud Mail Incident - 22/06/2021"
Last update**Summary and Impact to Customers** On Tuesday 22nd June 2021 from 14:48 to 28th June 2021 14:55, SYNAQ Cloud Mail experienced an intermittent, degraded mail authentication incident. The resultant impact of the event was that certain users would receive authentication pop-up messages when trying to login via HTTPS, POP3/S, IMAP/S, SMTP/S, as well as slow access to webmail. **Root Cause and Solution** On the 22nd of June 2021 at 14:48 SYNAQ Cloud Mail began to experience incoming mail delays. This delay occurred at the Zimbra MTA \(Mail Transfer Agent\) layer. Once identified, we attempted a series of fixes to resolve the incident. At 15:30 a change was made to increase the processing threads of the MTA servers from 100 to 150 to increase the amount of mail dealt with at any one time by an MTA when trying to deliver mail. The change was done in an attempt to increase the processing of the mail building up in the queue. However, this did not have the desired effect. At 16:30 a new Exim server was added to the MTA cluster \(a project that was going to take place within the next couple of weeks – as detailed below in the root cause section\), and this server was able to process mails with no delays. We ceased new mail delivery to the MTA’s, one at a time, until they cleared their queues. Thereafter, normal mail flow was restored by 16:51. On the 23rd of June 2021 at 09:09, mail delays re-occurred, coupled with a select group of users receiving authentication failure “pop-up” messages when trying to login to their mailboxes. At 09:52 debugging was performed on the Anti-Virus functions on the MTA servers, as this appeared to be where the delay was taking place. Configuration changes then were made to timeout settings and processing times to try and resolve this issue. Unfortunately, this did not have the desired effect. At 10:15 the replacement of the rest of the current Cloud Mail MTAs with new Exim servers commenced. This replacement was decided upon because the one server currently in production was processing mail without delay. This was fully completed by 20:30. At 17:14 mail delays and authentication had recovered. On the 24th of June 2021 at 09:41 a select group of users were receiving authentication pop -up messages when trying to login and experienced slow access to mail. As a result, at 10:17 we moved our focus from the MTAs to the LDAP servers as mail flow was no longer affected since the change to the new Exim servers. A data dump of the master LDAP database was performed and reloaded on all the replicas to rule out any memory page fragmentation \(performance inhibiting side-affect\) across these LDAP servers. At 13:00 file system errors were discovered on the master LDAP server. All Cloud Mail servers were adjusted to point to the secondary master. A file system check and repair was run on the primary LDAP master. While the sever was down memory and CPU resources were increased. At 13:10 mail authentication and slow access had recovered. On the 25th of June 2021 at 09:22, a select group of users were receiving authentication pop-up messages when trying to login and experiencing slow access to mail. At 10:35, TCP and connection and timeout settings adjusted on all the LDAP servers. At 12:35, connection tracking was disabled on the load balancer. This was done to ensure that if there was a particular problem with an individual LDAP replica, then the connections move seamlessly to another replica. Mail authentication and slow access then recovered. On the 26th and 27th of June we experienced no further recurrences of the issues. On the 28th of June 09:30, a select group of users were receiving authentication pop-up messages when trying to login and experienced slow access to mail. At 10:00, two new LDAP replicas were built to be added to the cluster. At 11:03, the global address list feature was turned off \(for classes of service with large domains that did not need this feature\) to try and reduce the traffic to the LDAP servers. At 13:05, we deleted 30 data sources \(external account configurations\) that were stored in LDAP but were showing errors during LDAP replication. At 14:30, the two new LDAP servers and all unique components of Cloud Mail, stores, MTAs, and proxies were pointed to their own unique set of LDAP replicas. At 14:55, mail authentication and slow access had recovered. The root cause of this event was due to a project that we initiated last year to replace standard Zimbra MTAs with custom built Exim MTAs. The purpose of this project was to vastly increase the security and delivery of clients’ mail. The initial project phase \(last year\) was to replace the outbound servers and then to do the inbound servers in July. A test inbound server was added, and this resulted in the start of the experienced issues. In addition, the replacement of all of the remaining MTAs with the new inbound servers in an attempt to resolve the issue, only exacerbated the problem. The problem that was introduced was that all native servers to Zimbra establish a persistent connection through to the LDAP servers. These new MTAs, introduced to reduce load and traffic to the LDAP servers, established short-term connections. The load balancer tried to deal with the different ways to establish connections in the same way and would overload a single LDAP server and would then proceed to affect the rest in cascading manner as the load was redistributed. To resolve this issue, two different load balancer IP addresses were configured with their own separate LDAP servers behind them. One, to manage persistent connections and the other, to manage short-term connections. Thereafter, the relevant servers were pointed to the load balancer IP that suits how they communicate and connect to LDAP. **Remediation Actions** • Two additional LDAP replicas have been built and added to the LDAP cluster. • Two different load balancer IP addresses have been configured with their own separate LDAP servers behind them. One to manage persistent connections and one to manage short-term connections. Thereafter, the relevant servers were pointed to the load balancer IP that suits how they communicate through to LDAP. • A third load balancer IP will be added to improve LDAP redundancy. This will allow store servers to attempt a new connection rather than remaining connected to an LDAP server that is no longer responding.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. We will send our next update in 60 minutes.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident where mail delivery is delayed. Engineers are investigating this as a matter of urgency. We will send our next update in 60 minutes.
Report: "Cloud Mail Incident - 23/06/2021"
Last update**Summary and Impact to Customers** On Tuesday 22nd June 2021 from 14:48 to 28th June 2021 14:55, SYNAQ Cloud Mail experienced an intermittent, degraded mail authentication incident. The resultant impact of the event was that certain users would receive authentication pop-up messages when trying to login via HTTPS, POP3/S, IMAP/S, SMTP/S, as well as slow access to webmail. **Root Cause and Solution** On the 22nd of June 2021 at 14:48 SYNAQ Cloud Mail began to experience incoming mail delays. This delay occurred at the Zimbra MTA \(Mail Transfer Agent\) layer. Once identified, we attempted a series of fixes to resolve the incident. At 15:30 a change was made to increase the processing threads of the MTA servers from 100 to 150 to increase the amount of mail dealt with at any one time by an MTA when trying to deliver mail. The change was done in an attempt to increase the processing of the mail building up in the queue. However, this did not have the desired effect. At 16:30 a new Exim server was added to the MTA cluster \(a project that was going to take place within the next couple of weeks – as detailed below in the root cause section\), and this server was able to process mails with no delays. We ceased new mail delivery to the MTA’s, one at a time, until they cleared their queues. Thereafter, normal mail flow was restored by 16:51. On the 23rd of June 2021 at 09:09, mail delays re-occurred, coupled with a select group of users receiving authentication failure “pop-up” messages when trying to login to their mailboxes. At 09:52 debugging was performed on the Anti-Virus functions on the MTA servers, as this appeared to be where the delay was taking place. Configuration changes then were made to timeout settings and processing times to try and resolve this issue. Unfortunately, this did not have the desired effect. At 10:15 the replacement of the rest of the current Cloud Mail MTAs with new Exim servers commenced. This replacement was decided upon because the one server currently in production was processing mail without delay. This was fully completed by 20:30. At 17:14 mail delays and authentication had recovered. On the 24th of June 2021 at 09:41 a select group of users were receiving authentication pop -up messages when trying to login and experienced slow access to mail. As a result, at 10:17 we moved our focus from the MTAs to the LDAP servers as mail flow was no longer affected since the change to the new Exim servers. A data dump of the master LDAP database was performed and reloaded on all the replicas to rule out any memory page fragmentation \(performance inhibiting side-affect\) across these LDAP servers. At 13:00 file system errors were discovered on the master LDAP server. All Cloud Mail servers were adjusted to point to the secondary master. A file system check and repair was run on the primary LDAP master. While the sever was down memory and CPU resources were increased. At 13:10 mail authentication and slow access had recovered. On the 25th of June 2021 at 09:22, a select group of users were receiving authentication pop-up messages when trying to login and experiencing slow access to mail. At 10:35, TCP and connection and timeout settings adjusted on all the LDAP servers. At 12:35, connection tracking was disabled on the load balancer. This was done to ensure that if there was a particular problem with an individual LDAP replica, then the connections move seamlessly to another replica. Mail authentication and slow access then recovered. On the 26th and 27th of June we experienced no further recurrences of the issues. On the 28th of June 09:30, a select group of users were receiving authentication pop-up messages when trying to login and experienced slow access to mail. At 10:00, two new LDAP replicas were built to be added to the cluster. At 11:03, the global address list feature was turned off \(for classes of service with large domains that did not need this feature\) to try and reduce the traffic to the LDAP servers. At 13:05, we deleted 30 data sources \(external account configurations\) that were stored in LDAP but were showing errors during LDAP replication. At 14:30, the two new LDAP servers and all unique components of Cloud Mail, stores, MTAs, and proxies were pointed to their own unique set of LDAP replicas. At 14:55, mail authentication and slow access had recovered. The root cause of this event was due to a project that we initiated last year to replace standard Zimbra MTAs with custom built Exim MTAs. The purpose of this project was to vastly increase the security and delivery of clients’ mail. The initial project phase \(last year\) was to replace the outbound servers and then to do the inbound servers in July. A test inbound server was added, and this resulted in the start of the experienced issues. In addition, the replacement of all of the remaining MTAs with the new inbound servers in an attempt to resolve the issue, only exacerbated the problem. The problem that was introduced was that all native servers to Zimbra establish a persistent connection through to the LDAP servers. These new MTAs, introduced to reduce load and traffic to the LDAP servers, established short-term connections. The load balancer tried to deal with the different ways to establish connections in the same way and would overload a single LDAP server and would then proceed to affect the rest in cascading manner as the load was redistributed. To resolve this issue, two different load balancer IP addresses were configured with their own separate LDAP servers behind them. One, to manage persistent connections and the other, to manage short-term connections. Thereafter, the relevant servers were pointed to the load balancer IP that suits how they communicate and connect to LDAP. **Remediation Actions** • Two additional LDAP replicas have been built and added to the LDAP cluster. • Two different load balancer IP addresses have been configured with their own separate LDAP servers behind them. One to manage persistent connections and one to manage short-term connections. Thereafter, the relevant servers were pointed to the load balancer IP that suits how they communicate through to LDAP. • A third load balancer IP will be added to improve LDAP redundancy. This will allow store servers to attempt a new connection rather than remaining connected to an LDAP server that is no longer responding.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality. We once again would like to apologise for the lengthy incident resolution, and want to assure you that we are doing a very intensive root cause analysis to understand how we can prevent this from happening in the future.
Dear Clients, Our platform is currently 100% stable and performant. Our engineers are continuing to monitor .
Dear Clients, The SYNAQ Cloud Mail incident is still being monitored. Improvements were made to the environment last night and need to be monitored through the peak morning period. We have partners worldwide working with us, please be assured that returning the service to full stability is our top priority.
Dear Clients, SYNAQ Cloud Mail is still currently experiencing an incident where users are receiving pop-ups to login and slow access to email intermittently. Engineers are still investigating this as a matter of urgency and working on several different resolution options.
Dear Clients, SYNAQ Cloud Mail is still currently experiencing an incident where users are receiving pop-ups to login and slow access to email intermittently. Engineers are still investigating this as a matter of urgency and working on several different resolution options.
Dear Clients, SYNAQ Cloud Mail is currently experiencing a recurrence of the incident where a subset of users are getting pop-ups or slow mail access. Engineers are investigating this as a matter of urgency.
Dear Clients, The SYNAQ Cloud Mail platform is now operational. All clients should be able to access services without any issues. Our Engineers will continue to monitor the platform closely for the next 72 hours before resolving this incident. Please Note: Closing and re-opening your mail application may resolve any remaining issues should they still exist. We sincerely apologise for this lengthy disruption to your email services, we are absolutely committed to excellence and will ensure world-class service moving forward.
Dear Clients, SYNAQ Cloud Mail platform is still experiencing an incident where a subset of users may experience slow access to their mailbox and/or receive intermittent login pop-up requests. Our Engineers are investigating this as a matter of urgency. Please note: Restarting your mail client or webmail should resolve the issue.
Dear Clients, SYNAQ Cloud Mail platform is currently experiencing an incident where a subset may experience slow access to their mailbox and/or receive intermittent login pop-up requests. Our Engineers are investigating this as a matter of urgency. Please note: Restarting your mail client or webmail should resolve the issue. We will send our next update in 60 minutes.
Dear Clients, Our engineers have implemented a fix for the SYNAQ Cloud Mail incident and all services are operational. We will continue to monitor the service stability for the next 24 hours.
Dear Clients, Our engineers are still working on the resolution of the SYNAQ Cloud Mail issue. This is being treated as a matter of urgency. We will send our next update in 60 minutes.
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail intermittent access issue and are working on a resolution. Mail flow and delays have been fully restored. The current intermittent issue regarding mail authentication pop-ups and slow access is being addressed. Please note restarting your mail client or webmail should resolve the issue. We will send our next update in 60 minutes
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident where certain users are getting password prompts and slow access to mail. Engineers are investigating this as a matter of urgency. We will send our next update in 60 minutes.
Dear Clients, Mail flow has been restored. SYNAQ Engineers will continue to monitor throughout the day and will send a full restore once confirmed.
Dear Clients, Our engineers have implemented a fix for the SYNAQ Cloud Mail incident and are monitoring the service stability for the next 12 hours.
Dear Clients, Our engineers have implemented a fix for the SYNAQ Cloud Mail incident and are monitoring the service recovery and processing of email backlog
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail incident and are working on a resolution. We have approximately 60% of new mail delivering and we will have 100% of new mail delivering within 30 minutes. The backlog of mail will need to be worked through. We will send our next update in 30 minutes
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail incident and are working on a resolution. We have approximately 30% of new mail delivering and we will have 100% of new mail delivering within 30 minutes. The backlog of mail will need to be worked through. We will send our next update in 30 minutes
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. We will send our next update in 30 minutes.
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. We will send our next update in 30 minutes.
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. Engineers have restored partial mail flow and are working on restoring the remaining servers. We will send our next update in 60 minutes.
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. We will send our next update in 60 minutes.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident where mail is being delayed. Engineers are investigating this as a matter of urgency. We will send our next update in 60 minutes.
Report: "SYNAQ Incident - 08/05/2021"
Last updateSummary and Impact to Customers On Wednesday 8th May 2021 from 06:00 to 12:28, SYNAQ Cloud Mail experienced a mail availability incident. The resultant impact of the event was that users were unable to authenticate and could not access the platform. Root Cause and Solution The root cause of this event was due to a scheduled change on the morning of the 8th of May from 00:00 to 06:00. The change was to improve the overall switch redundancy in the storage network. The change was unsuccessful and there was a requirement to roll back. Once the change was rolled back not all the hosts could see all the relevant storage paths. This prevented many of the VM’s from being able to start up correctly so the environment could not come up completely. To resolve this issue, the port channels between the two core switches had to be brought back up. Once this was done all communication was restored from all servers to all the relevant storage. Once this occurred all VM’s were successfully restored, and users were able to authenticate and access their mail once again. Upon further investigation, it was determined that the change to stack the Core switches to increase switch redundancy took down the existing port channels between the Core switches. Once these were brought back up the switch configuration was returned to its original state and all Cloud Mail functionality was restored. Remediation Actions • A full audit of change has been conducted and all the reasons for the change being a failure have been identified. • Changes have been made to improve our rollback plan process to ensure crucial steps are not missed going forward.
Dear Clients, The SYNAQ incident has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers have implemented a fix for the SYNAQ incident and are monitoring the service stability for the next 90 minutes.
Dear Clients, Our engineers are still working on the resolution of the SYNAQ outage. This is being treated as a matter of urgency. We will send our next update in 60 minutes
Dear Clients, Our engineers have identified the SYNAQ incident and are working on a resolution. We will send our next update in 30 minutes
Dear Clients, SYNAQ is currently experiencing an incident Cloud Mail and SecureArchive. These services are currently unavailable, engineers are investigating this as a matter of urgency. We will send our next update in 30 minutes.
Report: "SYNAQ Securemail Issue - 14/05/2021"
Last updateDear Clients, The SYNAQ Securemail issue has been resolved and the service has returned to optimal functionality.
Dear Clients, SYNAQ Securemail is currently experiencing an issue where certain mails sent to O365 are experiencing delays in delivery. Engineers are investigating this as a matter of urgency.
Report: "Cloud Mail Incident - 28/04/2021"
Last updateSummary and Impact to Customers On Wednesday 28th April 2021 from 13:38 to 14:56, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that users were unable to authenticate could not access the platform. Root Cause and Solution The root cause of this event was due to a failed route on our primary “High Availability” server and the Secondary server not taking over. With the route being down, the authentication requests were unable to reach the Cloud Mail proxy servers and authentication was unable to complete. To resolve this issue, the primary server had to be shutdown to bring the routes back up again. Once the routes had been restored to the Cloud Mail proxy server, authentication requests were once again being processed and users could access their mail. Upon further investigation, it was determined that the Secondary server had a configuration error which prevented it from being able to detect there was a problem with the primary server, which would have allowed it to take over. This configuration has been corrected and failover has since been tested and this moved across seamlessly. Remediation Actions • A full audit of all HA servers to ensure all their rules are identical. • Scheduled failover testing of HA servers to be implemented.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident were certain users are struggling to authenticate and download mail. Engineers are investigating this as a matter of urgency. We will send our next update in 30 minutes.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident were certain users are struggling to authenticate and download mail. Engineers are investigating this as a matter of urgency. We will send our next update in 30 minutes.
Report: "SYNAQ Securemail - 26/03/2021"
Last updateDear Clients, The SYNAQ Securemail incident has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers have implemented a fix for the SYNAQ Securemail incident and are monitoring the service stability for the next hour. There will be mail delays until the queues are back to normal.
Dear Clients, SYNAQ Securemail is currently experiencing an incident where no mail is being accepted inbound or outbound. Engineers are investigating this as a matter of urgency. We will send our next update in 30 minutes.
Report: "Cloud Mail Incident - 2021/03/12"
Last updateSummary and Impact to Customers On Friday 12th March 2021 from 07:52 to 14:45, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that users using ActiveSync to authenticate could not access the platform. Root Cause and Solution The root cause of this event was due to an error with the license for the ‘Next Generation’ mobile module, deployed at the beginning of February. The license failed to validate against the upstream licensing server and went into an automatic grace period. This grace period only lasts for 20 days and once the 20-day period had expired, the mobile module was no longer responding to authentication requests from ActiveSync. This was tested in staging and pre-production and passed tests successfully as this was not left to run past the 20-day grace period. In order to resolve this issue, a license that had been validated against the license server was deployed. Once deployed, authentication requests via ActiveSync resumed. Remediation Actions • A permanent fix to correct these validation issues will be released in the next patch which will be rolled out to our servers following our extensive testing process. • New monitoring alerts are being reviewed to be able to detect if these licenses move into the grace period again.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality.
Dear Clients, The SYNAQ Cloud Mail incident has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers are still working on the resolution of the SYNAQ Cloud Mail issue. This is being treated as a matter of urgency. We will send our next update in 60 minutes.
We have Identified the cause of the incident with our upstream vendor, and a fix is underway. Please note this incident affects only a subset of users using ActiveSync for mobile device access. You may still access email using webmail, IMAP and POP3.
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. We will send our next update in 60 minutes.
Dear Clients, Our engineers are still investigating the SYNAQ Cloud Mail incident. Please be assured that we have this under control and are treating this as a top priority. We will send our next update in 60 minutes.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an incident where users are unable to access their mail via their mobile devices. Engineers are investigating this as a matter of urgency. We will send our next update in 60 minutes.
Report: "Cloud Mail Incident - 11/12/2020"
Last updateSYNAQ Cloud Mail services are now fully restored and running optimally.
Dear Clients, SYNAQ Cloud Mail is current experiencing an issue where users are unable to authenticate and send mail. Engineers are investigating as a matter of urgency.
Report: "SYNAQ Cloud Mail Incident - 11/12/2020"
Last updateDear Clients, SYNAQ Cloud Mail services are now fully restored and running optimally.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an issue where some users are unable to login and send mail. Webmail is unaffected. Engineers are investigating the issue as a matter of urgency.
Report: "SYNAQ Cloud Mail Incident - 03/12/2020"
Last updateDear Clients, The SYNAQ Cloud Mail issue has been resolved and the service has returned to optimal functionality.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an partial issue where some users are unable to log into their mailboxes. Engineers are investigating the issue as a matter of urgency.
Report: "SYNAQ Cloud Mail Incident - 10/11/2020"
Last updateSummary and Impact to Customers On Tuesday 10th November 2020 from 15:40 to 16:30, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that users using IMAP, POP3 and SMTP to auth could not access the platform. Root cause and Solution The root cause of this event was due to a error with the deployment of a new SSL cert into the environment. This was tested in staging and pre-prod and passed tests successfully. Staging and Pre-Prod only make use of one authentication server and this configuration was deployed to production. This caused all production auth requests to attempt to authenticate against a single auth server and this server could not handle all the requests. This process has been performed many times without issue in the past, and human error caused the pre-prod configuration to incorrectly be applied to production. In order to resolve this issue, the production configuration was reapplied and authentication was served all auth servers in the cluster. Remediation Actions • Staging and Pre-Prod configuration has been updated to match Production and include all auth servers in its configuration. • New monitoring alerts have been built and deployed to detect if this scenario ever re-occurs.
SYNAQ Cloud Mail services are fully restored and operational
SYNAQ Cloud Mail is currently experiencing an incident where users are unable to login and are being prompted for a password. Engineers are currently investigating the issue.
Report: "SYNAQ Cloud Mail Incident"
Last updateDear Clients, The SYNAQ Cloud Mail Incident has been resolved and the service has returned to optimal functionality.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an issue affecting a subset of users from logging into the platform. Engineers are investigating as a matter of urgency.
Report: "SYNAQ Securemail Incident - 25/05/2020"
Last updateSummary and Impact to Customers On Monday 25th May 2020 from 02:53 to 11:27, SYNAQ Securemail experienced a mail timeout incident. The resultant impact of the event was intermittent connectivity to and from our platform when trying to send outbound mail. This caused the delay of outbound mail. Root cause and Solution The root cause of this event was due to the change logged on Sunday the 24th to move the Securemail environment to a new location within the Data Centre. The Securemail outbound servers all have multiple IP addresses assigned to them in order to combat blacklisting. The outbound servers utilise a script that only allows for one IP address to be brought up per server at a time. Unfortunately, this script failed, and multiple IP addresses were brought up at the same time, thus causing a conflict. This caused timeouts when trying to connect to and from these servers on port 25, causing the delay of sending mail. In order to resolve this issue, the additional IP addresses were removed so that only a single IP address remained on each server and mail returned to normal functionality and began to flow. Remediation Actions • The automated scripts that failed have been removed and replaced with improved technology. • An improved monitoring check is being built to detect this scenario earlier, should it reoccur.
Dear Clients, The SYNAQ Securemail Incident has been resolved and the service has returned to optimal functionality.
SYNAQ Securemail is currently experiencing an incident where emails are intermittently delayed when sending or receiving. Engineers are investigating the incident as a matter of urgency.
Report: "Cloud Mail Incident - 04/03/2020"
Last updateSummary and Impact to Customers On Wednesday 4th March 2020 from 11:01 to 13:05, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that a subset of users could not access the platform. Root cause and Solution The root cause of this event was due to a corruption of certain files in the Cloud Mail stores MYSQL databases. This caused the services on that particular store to stop and then users on that particular store were unable to authenticate and access their mailboxes. In order to resolve this issue, we restored the database from backups which allowed the services to be restarted and users to once again access their mailboxes. Remediation Actions • A call has been logged with our 3rd party provider to do some analysis of the MYSQL database information to and try and determine why this corruption occurred and what can be done in future to mitigate this. • A new monitoring alert has been built to try and detect early any corruptions of this nature.
Dear Clients, The Cloud Mail issue, affecting a subset of clients, has been partially resolved. Affected clients can now access their mailboxes. Engineers will continue to work on mailboxes to restore any mail that may be missing for the last few hours.
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail issue and are working on the resolution as a matter of priority.
Dear Clients, SYNAQ Cloud Mail is experiencing an issue where a subset of users cannot access their mailbox. Engineers are investigating this as a matter of urgency.
Report: "Securemail Incidet - 09/03/20202"
Last updateDear Clients, The processing of the message listing data backlog is complete. All data is now live.
Dear Clients, We experienced an issue with the SYNAQ Securemail Admin Portal, affecting users' ability to release and review recent mail logs*. Our engineers have identified and resolved the cause of issue and we will continue to monitor whilst the message listing data backlog is being processed. *Please note that mail delivery is not affected at all.
Report: "Securemail - Email Delivery Delays"
Last updateSummary and Impact to Customers On Thursday 13 th February from 00:14am – Sunday 16 th February 23:14pm, SYNAQ Securemail experienced a service incident which affected the delivery of mail. The impact of the event was an intermittent delay of emails being sent out of the Securemail platform that were then being delivered back inbound to domains using the Securemail platform. Cause and Solution The root cause of this event was due the failure of a core switch, belonging to our hosting provider. The effect of the switch failure caused a variety of different issues within our Securemail environment, resulting in intermittent connection timeouts between the outbound and inbound Securemail servers. This meant that, for some emails, numerous delivery retries were required to ultimately complete delivery. In order to resolve this issue, the failed switch was replaced and SYNAQ engineers worked with Cisco to resolve the timeouts. After many hours of troubleshooting and numerous attempted solutions, the issue was resolved. Whilst we are aware of what initiated this issue, we are continuing to work with our hosting provider and Cisco to ascertain why this switch failure affected the Securemail environment in this way. Actions * We are continuing our investigations with our hosting provider and Cisco to establish why this switch failure affected the Securemail environment in this way. * New monitoring alerts have been built and implemented for early detection of this issue should it recur.
Dear Clients, The SYNAQ Securemail issue has been resolved and the service has returned to optimal functionality.
Dear Clients, The SYNAQ Securemail outbound mail delays are no longer occurring and all mail is flowing optimally. We will continue to monitor the mail flow before a final resolution is sent.
Dear Clients, Our engineers have identified the cause of the intermittent outbound email delays and are working with the upstream provider to resolve this issue as a matter of priority.
Dear Clients, Our engineers are still investigating the SYNAQ Securemail intermittent outbound email delays. Please be assured that this matter is being treated as a matter of urgency and priority.
Dear Client, SYNAQ Securemail is experiencing intermittent email delays when sending outbound through our platform. Engineers are attending and treating this as a matter of urgency.
Report: "SYNAQ Securemail Incident - 11/02/2020"
Last updateDear Clients, The SYNAQ Securemail issue has been resolved and the service has returned to optimal functionality
Dear Clients, SYNAQ Securemaili is currently experiencing an issue where inbound mails are delayed. Engineers are investigating the issue as a matter of urgency.
Report: "Securemail outage"
Last updateSecuremail services have been restored and are working as normal again.
Our platform has recovered and we currently monitoring.
Dear Clients, SYNAQ Securemail is currently experiencing an outage on our platform. Engineers are investigating the outage as a matter of urgency.
Report: "SYNAQ Cloud Mail Incident - 21/01/2020"
Last updateDear Clients, The SYNAQ Cloud Mail issue has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail issue and are continuing to work on a resolution. Please take note that this does not affect mail flow and is only affecting a subset of users archive access.
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail issue and are working on a resolution.
Dear Clients, SYNAQ Cloud Mail is currently experiencing an issue where certain users will be unable to access their archives. Engineers are investigating the issue as a matter of urgency.
Report: "SYNAQ Branding and Securemail User Interface issue - 15/01/2020"
Last updateSummary and Impact to Customers On Wednesday 15th January from 09:21am – 01:29pm, SYNAQ Securemail and Branding experienced a service incident which affected the branding of mail and access of the relevant user interfaces. The resultant impact of the event was that Clients were unable access both the Branding and Securemail user interfaces and mails were not being branded. Root Cause and Solution The root cause of this event was due to a hardware failure in user interface cluster and failover not taking place correctly. As a result, users were unable to access the user interfaces for Branding and Securemail to perform any administrative actions. In conjunction with this all outbound mails were not being branded. In order to resolve this issue, the affected hardware was replaced and all services were restored. There was a delay in information in the Securemail user interface quarantine as the backlog was processed. This backlog took about 3 hours to be fully caught up. Remediation Actions • A project is underway to improve the failover mechanisms should a hardware failover of this magnitude occur again. The work on this project will be completed before the end of January 2020.
Dear Clients, The SYNAQ Securemail and branding issue has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers have implemented a fix for the SYNAQ Securemail and branding issue and are monitoring the service stability. All User interfaces are available and mails are branding. Securemail quarantine backlog busy catching up.
Dear Clients, Our engineers have identified the SYNAQ Securemail and branding issue and are working on a resolution. Please take note that Emails are not currently being branded. Email flow is not affected.
Dear Clients, Our engineers are still investigating the SYNAQ Securemail and branding issue. Please be assured that this matter is being treated as a top priority.
Dear Clients, SYNAQ Securemail and branding is currently experiencing an issue where their user interfaces are not available. Engineers are investigating the issue as a matter of urgency.
Report: "SYNAQ Securemail Incident - 25/10/2019"
Last updateDear Clients, The SYNAQ Securemail issue has been resolved and the service has returned to optimal functionality.
Dear Clients, Our engineers have implemented a fix for the SYNAQ Securemail issue and are monitoring the service stability.
Dear Clients, SYNAQ Securemail is currently experiencing an outage/issue users are unable to send and receive mail or able to connect to the UI. Engineers are investigating the outage/issue as a matter of urgency.
Report: "SYNAQ Branding - 09/10/2019"
Last updateSYNAQ Branding services have fully restored
SYNAQ engineers are still currently investigating the issue. This is being treated with the highest priority.
SYNAQ engineers are still currently investigating the issue. This is being treated with the highest priority.
SYNAQ engineers are still currently investigating the issue. This is being treated with the highest priority.
SYNAQ branding is currently experiencing an incident. The WebUI is unavailable and branding of mails is intermittent. Engineers are currently investigating.
Report: "SYNAQ Branding"
Last updateSYNAQ Branding incident has been resolved. Web UI is once again accessible and all mails are being branded
We are continuing to investigate this issue.
SYNAQ Branding is currently experiencing a major outage. The web UI is inaccessible and certain mails are not being branded. Engineers are busy investigating the issue as a matter of urgency.
Report: "SYNAQ Cloud Mail Incident - 09/10/2019"
Last updateSummary and Impact to Customers On Monday 9 th September 2019 from 10:00am – 18:35 pm and Tuesday 10 th September 2019 from 08:48am – 11:09 am, SYNAQ Cloud Mail experienced a degraded service incident which caused mail delays and interrupted mailbox use for a subset of clients. The resultant impact of the event was that certain users experienced a delay in mails and their mailboxes were very slow to respond. Root Cause and Solution The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused delayed mail delivery and interrupted mail access for a subset of users. In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery and access performance was restored to the affected users. Remediation Actions SYNAQ engineers are implementing a storage plan to increase the single data path capacity to handle failover load without impacting users.
SYNAQ Cloud Mail services have been fully restored
SYNAQ engineers have identified the issue and are currently working on a resolution
Dear Clients. SYNAQ Cloud Mail is currently experiencing degraded performance affecting a subset of mailboxes and delays sending and receiving certain mails. Engineers are currently investigating the issue.
Report: "SYNAQ Securemail Incident - 05/09/2019"
Last updateSummary and Impact to Customers On Thursday 5 th September from 07:48am – 09:52am, SYNAQ Securemail experienced a service incident which affected the delivery of mails for all .za domains. The resultant impact of the event was that Clients were unable to send and receive mail to and from .za domains. Root Cause and Solution The root cause of this event was due to an issue experienced by the ZA root servers where they were no longer allowing secure queries from DNS servers. As a result, when our DNS cache servers were trying to query the affected root servers, it was returning an invalid response, which resulted in the inability to send and receive mails to and from .za domains. In order to resolve this issue, the enforced secure queries capability were disabled to the ZA root servers and we were once again able to resolve all .za domain information. Remediation Actions Custom monitoring checks are going to be built to proactively identify these forms of DNS issues.
SYNAQ Securemail Incident has been resolved. All mail flow has returned to normal.
There is currently a 15 minute mail delay. The queues are currently clearing and engineers will continue to monitor.
SYNAQ Engineers have identified the issue. Mail flow has resumed but there is a delay in mail while working through the backlog
SYNAQ Securemail is current experience delays when sending and receiving mail. Engineers are currently investigating
Report: "SYNAQ Cloud Mail Incident - 10/10/2019"
Last updateSYNAQ Cloud Mail services have been fully restored.
Dear Clients. SYNAQ Cloud Mail is currently experiencing degraded performance affecting a subset of mailboxes and delays sending and receiving certain mails. Engineers are currently investigating the issue.
Report: "SYNAQ Cloud Mail Incident - 10/06/2019"
Last updateSummary and Impact to Customers On Monday 10th June from 1:38pm to Tuesday 11th June 4:55pm, SYNAQ Cloud Mail experienced a minor service incident which caused mail delays for a subset of clients. The resultant impact of the event was that certain users experienced mail delays of up to 2 hours for some of their mail. Root Cause and Solution The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused mail delivery to users mailboxes to be delayed. In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery performance was restored to the affected users. Remediation Actions • Whilst the built-in redundancy on the storage array worked to prevent a complete loss of access to mailbox data, SYNAQ engineers are working with our storage vendor to increase the single data path capacity to handle failover load without impacting users.
Dear Clients. The SYNAQ Cloud Mail delays' issue has been resolved. Engineers will continue to monitor.
Dear Clients, SYNAQ Cloud Mail is currently experiencing a delay in mail delivery for certain users. Please note the issue has been identified and is currently being worked on by engineers.
Report: "SYNAQ Cloud Mail Incident - 11/06/2019"
Last updateSummary and Impact to Customers On Monday 10th June from 1:38pm to Tuesday 11th June 4:55pm, SYNAQ Cloud Mail experienced a minor service incident which caused mail delays for a subset of clients. The resultant impact of the event was that certain users experienced mail delays of up to 2 hours for some of their mail. Root Cause and Solution The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused mail delivery to users mailboxes to be delayed. In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery performance was restored to the affected users. Remediation Actions • Whilst the built-in redundancy on the storage array worked to prevent a complete loss of access to mailbox data, SYNAQ engineers are working with our storage vendor to increase the single data path capacity to handle failover load without impacting users.
Dear Clients Mail flow is optimal again
Dear Clients, The Cloud Mail incident has been resolved. Mail queues are currently working through the backlog and engineers are monitoring
Dear Clients, SYNAQ Cloud Mail is experiencing mail delays. We are aware of the issue and we are working on the resolution as a matter of urgency.
Report: "SYNAQ Branding Incident - 05/06/2019"
Last updateSummary and Impact to Customers On Wednesday 5th June from 10:55am to 7:27pm, SYNAQ Branding experienced a major service incident which affected all elements of the email branding service. The resultant impact of the event was that a Branding Clients experienced malformed mails or mails with the body being removed and eventually no branding at all. Root Cause and Solution The root cause of this event was due to a code release to correct a line break format issue for mails destined for a certain type of ticket management system, where the branding was not showing correctly. We enforced line breaks on every mail to be a CRLF format type and these in turn caused issues with certain outbound MTA’s that were expecting the line breaks to be in an LF format. As a result of the above, formatting issues occurred with the message body of numerous mails going through the platform. In order to resolve this issue, the Branding service was disabled to allow mail to be delivered in its original format. Thereafter, the new code was rolled back to the previous stable version and Branding was then re-enabled for all clients and mail began to be correctly branded. Remediation Actions • Additional end-to-end testing methods were introduced to proactively identify this issue for any further releases.
Dear Clients, All branding services are fully restored and all mails are once again being correctly branded.
Dear Clients, The issue has been identified and engineers are currently working on the resolution.
Dear Clients, Please note that SYNAQ Branding has been temporarily disabled to alleviate the mail corruption issues identified at 11:00 this morning. As a result, no branding will be added to mails whilst this issue is being investigated. Engineers are currently investigating the cause of these issues and are working on a speedy resolution.
Report: "SYNAQ Securemail Admin Portal Incident - 21/05/2019"
Last updateDear Clients, All admin UI functionality has been fully restored.
Dear Clients, We that we are still working on a swift resolution to the incident. Please note that mail still remains unaffected.
Dear Clients, Please note we have identified the issue and the engineers are working on the resolution.
Dear Clients, The Securemail admin portal is currently experiencing degraded performance and mail release function will be slower than usual. Please note mail flow is not affected by this incident.
Report: "SYNAQ Securemail Incident - 09/05/2019"
Last updateMail flow is normal and the processing of the mail backlog is completed. Securemail is fully restored.
New mail flow is now optimal with a full catch up of backlog mail in 45 minutes.
We are continuing to monitor for any further issues.
Dear Clients Latent Notification - Email delays were experienced from 2am - 6am for incoming mails during that period. The issue has been resolved and clients can expect a mail delay of up to 2 hours whilst the system works through the backlog of mails.
Report: "SYNAQ Cloud Mail Incident 03-04-2019"
Last updateSummary and Impact to Customers On Wednesday the 3rd April from 12:49pm to 16:58, SYNAQ Cloud Mail experienced a service incident. The resultant impact of the event was the intermittent mailbox access and mail sending and receiving delays. Root Cause and Solution The root cause of this event was due to a lack of communication between a cluster VMHost and its backing storage. This caused the inability for users to authenticate against stores and archive servers that were operating on the affected host. As a result, Clients experienced intermittent mailbox access and mail delivery issues. In order to resolve this incident, the affected VMHost was rebooted to re-establish communication with the backing storage, so that mailbox access and delivery could function as per normal. Remediation Actions • Improve our monitoring process to include this unique scenario, to prevent it from recurring in the future.
The Cloud Mail incident has been fully restored and all mail and archive services are fully functional.
Dear Clients. Whilst mail flow and mail access is completely restored, engineers are continuing to work on the archive access resolution as a matter of urgency.
Dear Clients. The Cloud Mail platform authentication and mail delay issue has been restored. Certain users, however, may still experience an issue accessing their archives. Engineers are continuing to work on the archive access resolution.
Dear Clients. The Cloud Mail platform issue has been identified and engineers are urgently working on the issue resolution.
Dear Clients, we are currently experiencing a major outage on the Cloud Mail platform with intermittent login capability and mail sending delays. Engineers are investigating this issue as a matter of urgency.
Report: "SYNAQ Securemail Incident - 13/03/2019"
Last updateSummary and Impact to Customers On Wednesday 13th March 2019 from 07:40am – 20:04pm SYNAQ Securemail experienced a major incident. The resultant impact of the event was Clients were initially unable to send and receive mail and thereafter experienced mail delays. Root Cause and Solution The root cause of this event was due to exhausted DNS connections on our DNS servers as a result of two authoritive name servers that were not responding to DNS queries for the domains that they were the authoritive for. DNS is important to the operation of the Securemail service because it is used to perform multiple security checks, resolve MX records and to allow connectivity to our platform via the relevant host names. This caused connections from our DNS server to the two authoritive name servers to remain open as the affected servers did not respond to our queries. This caused the SYNAQ DNS servers to reach their connection limit and as such, prevented all further DNS queries from taking place. As a result, no mail could be sent or received. In order to solve this issue, and whilst we were waiting for the name server provider to resolve their incident, we implemented a temporary work-around where we redirected our DNS queries away from the root servers to other DNS servers that still had cached results and responded with the required domain information. This allowed mail to start flowing again and to work through the existing backlog of mails. At approximately 15:40pm, the name server provider resolved their incident and we rerouted our DNS queries back to the root servers. This allowed our mail delivery to resume at a normal rate and the backlog of mails was completed at 20:04pm. Remediation Actions We are investigating new methods to mitigate our connections limits from being reached should a third party DNS name server provider experience any incidents.
The issue has been resolved and no further mail delays are being experienced
Dear Clients. The incident has been resolved. Engineers will continue to monitor the situation whilst the mail queues continue to reduce.
Mail is flowing and the mail queues are reducing whilst our engineers continue to work on the resolution.
Our engineers are still working on the complete resolution as a matter of urgency
Dear Clients, Status Update: Mail is flowing and we are currently processing a backlog of approximately 40 minutes. Our engineers will continue to monitor closely whilst working on the resolution.
Mail is flowing, processing backlog of ±40 min, engineers monitoring closely
Engineers have identified the problem and are implementing a fix. Mail is starting to flow again and access to the API slowly being restored
Engineers are still working on the issue. This is being treated with the highest priority.
There is currently a connectivity problem through to SYNAQ Securemail services. This will affect all sending and receiving of mail and provisioning to the API
Report: "Cloud Mail Outage 12-12-2018"
Last updateAll services fully restored
Engineers still investigating the issue.
Admin console access has been resolved. Sending of mail still being investigated
We are currently experiencing an incident on the SYNAQ Cloud Mail platform. The admin console is down and users are not able to send mail