Historical record of incidents for Mailosaur
Report: "Investigating issues with authentication and API services"
Last updateWe are experiencing issues with authentication and core services related to an upstream issue with Google Cloud services (https://status.cloud.google.com/) We will provide updates as soon as we have them, but hope to have normal service resumed shortly.
Report: "Mailosaur UI is experiencing degraded performance"
Last updateThis issue was resolved.
We are aware of an issue, related to our support chat functionality, which is causing the Mailosaur UI to be slow to load. Our team is working on a fix for this right now and we hope to have a resolution very quickly.
Report: "Intermittent connectivity issues for SMTP and POP3"
Last updateThis incident is now closed following a solution implemented by Google.
Services are returning to normal for all customers. We will continue to monitor for the next few hours.
Mitigation work is currently underway by our Google's engineering team. We do not have an ETA for mitigation at this point. We will provide an update within the next hour.
Google Cloud Networking have confirmed that there is a known issue with increased network latency which is causing issues with SMTP/POP3 connections from outside Europe. Google have confirmed their networking team is actively investigating this issue and attempting to fix ASAP.
The issues impacting SMTP and POP3 for some customers only relate to connections outside of Europe. Our team believes this issues is related to Google Cloud Platform network load balancing, but we are still investigating at this time. The nature of SMTP means that emails will automatically be retried and arrive in most cases, but you may see delays in arrival.
We are aware of an issue impacting come clients connecting via SMTP and POP3. This is being actively investigated, we will update as soon as possible.
Report: "SMS messages not being delivered to some numbers"
Last updateWe are satisfied that normal SMS service has now resumed for some time and so are now closing this incident
A fix has been implemented and we are starting to see SMS services resume to normal. We are continuing to monitor this process.
Our supplier has identified the cause of the issue affecting some SMS numbers and is working to resolve this as soon as possible. We will update when further information is available.
We are continuing to work with our supplier on this matter, we are looking to resolve this issue as soon as possible.
We are aware that some customers are not receiving SMS messages to their Mailosaur phone numbers. This is being caused by an issue with one of our service suppliers, which we are actively investigating now. We will provide an update as soon as possible.
Report: "Some users unable to log into the Dashboard"
Last updateAn update was applied today which causes several customers to be logged out and unable to log back into the Mailosaur Dashboard, this has now been resolved by our team and you should no longer see any issues.
Report: "Issue with anti-virus scanning of attachments"
Last updateThe earlier issue we had with slow anti-virus scanning has now been resolved.
We have identified the cause and are pushing out a fix now, we should see files become available as normal shortly as the AV scanning catches up.
We're currently investigating an issue affecting some customers who are trying to access attachments on emails. The attachments are not being virus scanned in time, resulting in them being inaccessible. We are actively investigating this issue and will update as soon as possible.
Report: "Degraded performance"
Last updateThis issue has now been resolved, effected customers should no longer experience any issues
Customers are still reporting performance issues so our team will continue investigating
Some customers are finding services are unavailable to them. We are investigating this issue at the moment.
Report: "Service interruption for some customers"
Last updateThis incident has been resolved.
Service for affected customers has been restored. As mentioned in our last update, we do not expect further interruptions.
We became aware of an issue that was affecting some customers today, which meant we had to restart one of our database nodes at short notice. The downtime will be short-lived (a matter of minutes) and we do not expect this issue to repeat itself.
Report: "Services interrupted"
Last updateWe had a temporary internal connection issue which caused our service to temporarily go offline for some customers. Connectivity was quickly restored (within 15 minutes) and we're actively monitoring this now.
Report: "Deploying new Mailosaur Dashboard"
Last updateAll deployed and back to normal operations now
We're deploying a new version of our Dashboard will may result in up to 5 minutes of downtime of the UI only
Report: "Degraded performance"
Last updateThis incident has been resolved.
We have identified and rectified that issue that was causing degraded performance for some customers. All systems should be back to normal, but we continue to monitor them right now.
We are currently investigating an issue causing degraded performance to some customers. We'll update you as soon as possible.
Report: "Potential issues with Google Cloud Platform"
Last updateThings have remained stable since returning recently. We are marking this one as resolved.
It appears that services are operating again as normal. Our team are actively monitoring this at present.
We are currently aware of an issue effecting our testing services. We believe this is the same issue effecting many other sites and apps around the world, potentially related to Google Cloud Platform. We will update further, as soon as we have more information.
Report: "API - false positive"
Last updateDue to a false positive in one of our monitoring tests, the API was incorrectly marked as offline. However, in fact, no services or customers were impacted.
Report: "Degraded performance of SMTP"
Last updateThis incident has been resolved.
Due to an error with a searching index, we had degraded performance across some SMTP servers this evening. We have now implemented a fix, which we are actively monitoring.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Problems with emails containing no text body"
Last updateThis incident has been resolved.
We have identified an issue with emails received that have no text body within them. We are fixing this right now and should be back to normal very soon.
Report: "Errors when creating and editing servers"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are aware of an issue affecting API endpoints that create and update servers. Customers are currently receiving a 401 - Unauthorized error when calling these endpoints. Our team are aware of the cause and will be deploying a fix very shortly.
Report: "Partial outage of services"
Last updateWe have now fully resolved the early issue. This was caused by a reference to a redundant container image, we do not expect any degradation in performance now, and all systems should be back to full operation. We apologise for any inconvenience.
We have identified the cause of the issue, and are deploying a fix which will return full stability within the next few minutes.
We are currently investigating an issue impacting our API and SMTP endpoints. We will keep you updated.
Report: "Database issue"
Last updateAn abnormal query was executed on a database instance which caused unusually poor performance and required a full restart on our part. Systems came back online as expected and all tests passed fine before putting the system back online to users. We will be reviewing our procedures to ensure this cannot happen again.
We are investigating database downtime. Please bear with us
Report: "Downtime"
Last updatePlease accept our apologies for the downtime. Our services should be back to normal again. Thanks.
We are currently experiencing performance issues and are working hard on resuming services as before. Thanks for your patience.
Report: "SMTP service"
Last updatePerformance has restored to normal. We will continue monitoring the situation.
We have some reports of SMTP connections timing out, we are investigating this issue now.
Report: "Unplanned downtime and delays"
Last updateTo improve our ability to process millions of emails per month reliably, we migrated to Google Cloud last Saturday during planned downtime. However, some issues remained and became clearer over the last few days. Customers were seeing delays in checking their emails and in some cases could not connect anymore. We have identified the underlying issues and as part of the solution also doubled our database capacity. Our performance graphs look a lot healthier now. Please contact us if you are still experiencing issues. We are continually monitoring our systems for bottlenecks.
Report: "Website offline"
Last updateOur website is back to normal operation.
Our website is currently offline - the Mailosaur service itself is unaffected. We are aware of the cause (a private networking issue) and will be back online within the next 1-2 hours
Report: "Dropped connections to some sessions"
Last updateThis incident has been resolved.
Normal activity has been restored, we will continue to monitor
We're currently restarting a node that has stopped accepting connections
Report: "Website offline"
Last updateOur web site is back working as normal - this issue appears to have related to a breakage in private networking between a load balancer and web site server.
We are continuing to investigate this problem (please note that Mailosaur functionality is unaffected by this issue, only our web site is down during this time). As a temporary measure we are automatically redirecting to https://mailosaur.com/app
The website server is currently offline - we have raised a ticket with our hosting provider and are looking to bring up a replacement server now.
Report: "Unexpected SMTP downtime"
Last updateThis incident has been resolved.
We had reports of downtime between 02:45 and 05:00 this morning. This appears to be due to a crashed SMTP server node that did not properly resolve itself. We have resolved this issue, but are now looking at ways to avoid this in incident in future.
Report: "Database improvements deployed"
Last updateWe switched database servers between 10:40 and 11:30 GMT today. Customers should not have experienced any downtime in this period.