Historical record of incidents for Capsule
Report: "Dropbox Processing Delay"
Last updateThis incident has been resolved.
We have identified the cause of the delay and a mitigation has been put in place. We expect most delayed emails to be processed within 30 minutes.
We're currently experiencing delays storing emails sent to Capsule's dropbox addresses. Our team is investigating and working on a fix.
Report: "Capsule is currently degraded"
Last updateThis incident has been resolved.
We're continue to monitor application performance.
We are continuing to work on a fix for this issue.
We have put some mitigations in place. However you may still experience some poor performance. We continue to work on restoring the service to normal.
Capsule is still working but availability is patchy some requests taking longer than normal to serve. We're are continuing to work on mitigating the outage. We'll keep you updated.
We're continuing to work on the issue. Capsule is up, but running slowly. We're working to fully mitigate and get the performance back to normal.
We're working on mitigating a Denial of Service attack against our site. We'll keep you updated.
We're are continuing to investigate.
We're investigating the issue.
Report: "Connectivity issues"
Last updateWe believe this third party routing issue affecting a number of UK based ISPs has now been resolved.
A small number of users are experiencing problems accessing Capsule. We are currently investigating and will post any updates here.
Report: "Capsule is unavailable or slow"
Last updateWe're very sorry if you suffered from the periods of slow responses and elevated error rates with Capsule yesterday. We’ve tracked the issue down the a bug in our data stores query optimiser that resulted in what should have been a fast responding query taking several minutes to complete. This only happens in rare situations with an unusual configuration of data. While we fully understand the exact nature of the optimiser bug we have applied a fix that stops the bug from occurring. Once again we’re sorry for this disruption this caused.
This incident has been resolved.
We understand the underlying problem and are working on a permanent fix
Performance has returned to normal.
We are currently investigating this issue.
Report: "BT Broadband Connectivity Issues"
Last updateBT are reporting that the issue has been resolved. If affected please reboot your router and try connecting to Capsule again. If you're still having issues please contact us.
We're currently aware of an issue affecting some BT Broadband customer's access to Capsule. BT appear to be working on a fix.
Report: "Capsule is unavailable because of network connectivity"
Last updateThis incident has been resolved. The outage was the result of a failure on one of our load balancer servers, we were able to switch it over to a replacement but unfortunately only after a few minutes of downtime.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "Capsule is unavailable for some customers"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The 3rd party ISP has recovered. We'll keep monitoring for a little longer.
We've identified connectivity errors and performance issues with a 3rd Party ISP. We'll keep you updated as we know more.
Capsule is unavailable for some users for short periods of time. We're seeing intermittent connectivity errors and performance issues with a 3rd Party ISP. We'll keep you updated as we know more.
Report: "Gmail Add-on Delayed Storage of Messages"
Last updateThe backlog of messages has now cleared and new messages should be store without delay. Apologies for this disruption in your day. We'll be seeing what mechanisms we can put in place to avoid this in the future.
We have identified the cause of the delayed message storage and mitigated it. No messages have been lost during this incident but a backlog has accumulated. Messages are now processing at normal speed and we expect to be caught up soon. We'll continue to monitor and provide an update once it has cleared.
Some users of our Gmail Add-on may be experiencing delays between messages being received in Gmail and stored in Capsule. We are currently investigating and will provide an update shortly.
Report: "Drop Box - Delayed Email Processing"
Last updateThe backlog of queued messages has now been fully processed and the time taken to store new messages using the Drop Box should now be back to normal. Although these incidents are rare we are actively working on a solution to make the Drop Box more resilient. Apologies for the disruption in your day.
Some users may be experiencing a delay in the storage of emails using our Drop Box feature. This was due to a server failure. All messages sent to the Drop Box should have been queued for processing during this time but may take some time to appear in Capsule due to a backlog that has accumulated. We are currently monitoring and will update once the backlog has been processed.
Report: "Gmail Add-on Delayed Storage of Messages"
Last update_From 02:16 UTC April 2nd to 13:52 UTC April 14th around 10% of our Gmail Add-on users will have experienced increasingly long delays between messages being received on existing stored conversations in Gmail and them appearing in Capsule. This was due to a processing issue which we have now resolved._ We know a large number of our users have come to rely on our new Gmail Add-on and we’re extremely sorry it hasn’t been working to the usual high standard you would expect from a Capsule feature. Therefore we feel it’s only right to provide a more detailed explanation and assurances that we’re working to prevent this from happening again. First a little background - Over the past two weeks we were receiving sporadic reports from our users that our Gmail Add-on wasn’t storing new messages that were part of an existing stored conversation. We immediately began an investigation into the issue and identified a potential cause. This related to a timeout inside our code that communicates with Gmail. A fix for the timeout and some additional logging to assist with the investigation was rolled out. We then asked some of the users affected if the issue was resolved for them. It quickly became apparent it was not. At that point we started analysing the additional logging we had put in place to see if there was any pattern to the reports we were receiving. The analysis was still ongoing on Friday \(April 12th\) when we noticed a sudden unusual size increase in one of a number of queues which forms part of the processing for our Gmail Add-on. The queue is a stream of events we receive from Google for a subset of user mailboxes which has our Gmail Add-on enabled. We consume these events so we know when something has happened on a stored conversation i.e. a new message is sent or received. When we receive an event for a stored conversation we request all the updates since we last fetched the history of it and store any new messages that have occurred since. The increasing queue size suggested we had an issue processing the event stream which could easily have accounted for the missing message reports so we focused our investigation on that. After a short time we identified an issue fetching conversation histories from a single user’s mailbox. The fetch from Gmail was taking an abnormally long time and hitting a transaction timeout we have in our code. In an effort to recover automatically, our code re-attempted the fetch. This caused an infinite loop that we didn’t recover from. Unfortunately this effectively stalled the processing of the event stream from Google for the users sharing the same queue as that user. Once we found this loop, we disabled the Gmail Add-on for that single user and restarted the process which consumes the event stream. This kicked off a recovery processing a backlog events. This recovery took a number of hours due to the volume of events which had built up. After the backlog was processed we confirmed that new messages that were part of an existing stored conversation were being processed and stored in near real time once again. We’re now in the process of introducing additional monitoring and alerting so we are made aware of any delays in processing. We’re also introducing additional safeguards to ensure an issue with a single user doesn’t impact other users sharing the same queue. And finally we’re looking at how we can re-architect the solution to ensure that if another backlog of events ever does form we can process through it as quickly as possible. This was quite a challenging incident for us and we’ve learnt a lot from it. Once again we’re extremely sorry for the impact this had on our users and please be assured that we’re working hard to prevent it happening again.
From 02:16 UTC April 2nd to 13:52 UTC April 14th around 10% of our Gmail Add-on users will have experienced increasingly long delays between messages being received on existing stored conversations in Gmail and them appearing in Capsule. This was due to a processing issue which we have now resolved.
Report: "Some users are unable to access Capsule"
Last updateThis incident has been resolved.
We have identified an issue with one of our load balancers. This has now been replaced and we are continuing to monitor.
Our systems monitoring has spotted a problem that is affecting access to Capsule. Our team has been alerted and are investigating. We'll post a further update once we have more information.
Report: "Some users are unable to access Capsule"
Last updateThis incident was caused by scheduled maintenance on one of our database instances. Users may have experienced a brief period (less than 2 minutes) where they were unable to login to Capsule. The maintenance was completed successfully.
Our systems monitoring has spotted a problem that is affecting access to Capsule. Our team has been alerted and are investigating. We'll post a further update once we have more information.
Report: "Configuration error with monitoring system"
Last updateThis incident has been resolved.
There is a configuration fault on our monitoring system which is incorrectly showing that Capsule is down. Capsule is operating normally.
Our systems monitoring has spotted a problem that is affecting access to Capsule. Our team has been alerted and are investigating. We'll post a further update once we have more information.
Report: "Some users are unable to access Capsule"
Last updateThis incident has been resolved.
Sorry for the brief outage while database updates were applied.
Our systems monitoring has spotted a problem that is affecting access to Capsule. Our team has been alerted and are investigating. We'll post a further update once we have more information.
Report: "Degraded Performance - Attachments"
Last updateOur hosting provider has given the all clear and Capsule is back to fully operational. File upload is back and the drop box is humming along nicely. Thanks everyone for your patience during all this. We hope it hasn't caused too much trouble.
Our hosting provider is reporting that they are fully recovered and that the file storage service is operating normally. Access to files, pictures and icons in Capsule is back and so is the ability to upload files. The Capsule drop box is also back in action and it's currently going through the backlog. It won't be long before everything is back to normal.
We’re starting to see some improvements around viewing and uploading attached files and icons & profile pictures are slowing coming back as well. We’re still waiting for everything to be 100% resolved before we can get the Capsule drop box up and running again. As always we’ll let you know.
Our hosting provider believe they understand the root cause of the problem and they're working hard to get it fixed. We're still monitoring their progress closely and will keep you posted on any more updates.
We are aware of an issue for which may mean you are temporarily unable to view or add files in Capsule. This will affect emails sent to the Capsule drop box as well. In addition, profile pictures might also not be displaying for you. This appears to be affecting other services hosted by our provider. We are continuing to monitor the situation and we'll post an update once we have more information.
Report: "LastPass Chrome Extension"
Last updateWe have not been able to get satisfactory assurances that LastPass will be able to resolve this issue with their extension. However in the meantime we have been able to create a workaround to resolve the issue by making changes to Capsule. If you're currently logged into Capsule, you'll need to log out then back in again. You'll then be able to use Capsule as normal with the LastPass extension enabled.
Over the weekend an update to the LastPass Chrome extension has introduced a bug that is causing unexpected logouts when attempting to complete forms or perform actions in Capsule. While we work with LastPass to help them fix the extension we suggest disabling the LastPass extension or switching to an alternative browser like Edge, Firefox or Safari.
Report: "Capsule Interruption"
Last updateCapsule was unavailable for 10 minutes between 10:31pm and 10:41pm GMT as a result of a server outage. We’ve replaced the server and things are all back to normal. We’re sorry for the disruption.
Report: "Capsule Interruption"
Last updateSome users may have experienced degraded performance between 2:30pm and 2:54pm UTC as a result of an issue with a server. Very sorry for the interruption. We’ve resolved the issue and everything is back to normal again.
Report: "Internet connectivity issues affecting a small number of users"
Last updateWe’ve had confirmation from users of the affected ISPs that the connectivity trouble within their networks was resolved over the weekend.
We have received reports from a very small number of users who are having trouble using Capsule recently - reports are that pages are slow to load or that actions on a page appear to hang or fail to complete on occasion. We can see from our monitoring that all Capsule services are running as expected from our data center side. Having investigated this further with the affected customers we can confirm the performance issue is limited to a small group of users on specific Internet Service Providers (ISPs), specifically their connection to our hosting provider. We have engaged our hosting provider to see if there is anything they can do to resolve this with the affected ISPs. In the meantime, if you are having trouble with performance of Capsule, please try switching to an alternative internet connection if possible.
Report: "Network connectivity issues intermittently affecting some users"
Last updateNetwork connectivity issues have been resolved. AWS confirmed that between 11:37 AM and 12:13 PM UTC they experienced an Internet connectivity issue which affected traffic for some customers using AWS services in the US-EAST-1 Region (from http://status.aws.amazon.com).
We are seeing intermittent connectivity issues for some users which may mean you are temporarily unable to connect to Capsule. This appears to be affecting other services hosted by our provider (AWS), not just Capsule. We’re continuing to monitor this and will update this post as we have more information.
Report: "Some customers having trouble accessing Capsule"
Last updateA fix has been applied and our monitoring is showing that connection times are now recovering. We’re very sorry for the disruption if you were affected.
We are aware of some slow app servers affecting connectivity to Capsule for a small number of users today.
Report: "Unable to access Capsule"
Last updateAnd we’re back... We’re really sorry for the disruption today. The problem was an accidental misconfiguration of our load balancing setup which has been corrected. We understand what went wrong and we’ll improve our processes and scripts to make sure this doesn’t happen again.
We are aware of an issue preventing users from accessing Capsule. We are working hard on getting this fixed and expect to be back online very soon.
Report: "Capsule Interruption"
Last updateCapsule is back to normal for everyone now and we’re monitoring the situation closely.
We’re seeing reports that a small number of users are having trouble logging in to Capsule or are getting a 504 error. We’re in the process of getting this resolved and will let you know as soon as everything is back to normal.