Historical record of incidents for Superhuman
Report: "Degraded Service for Gmail Users"
Last updateThere is an ongoing outage with Google services, and we are currently investigating the impact. At the moment, there is degraded performance for sending emails and syncing mailboxes.
Report: "Elevated failure rates for AskAI, Auto Labels and Auto Summaries"
Last updateWe are currently investigating an issue affecting the AskAI, Auto Labels and Auto Summaries feature, resulting in an elevated rate of request failures. The root cause has been identified as an outage with a third-party service that AskAI depends on. We are actively working with the third-party provider to resolve the issue and will continue to monitor the situation closely. We will provide updates as soon as we have more information or when the issue is resolved. Thank you for your patience and understanding.
Report: "Elevated Failure Rates for AskAI Feature"
Last updateWe are currently investigating an issue affecting the AskAI feature, resulting in an elevated rate of request failures. The root cause has been identified as an outage with a third-party service that AskAI depends on. We are actively working with the third-party provider to resolve the issue and will continue to monitor the situation closely. We will provide updates as soon as we have more information or when the issue is resolved. Thank you for your patience and understanding.
Report: "Degraded Service for Personal Outlook Calendars"
Last updateWe have identified a error in MSGraph responses impacting Calendars for Personal Outlook accounts. The underlying error is also reported here. https://learn.microsoft.com/en-us/answers/questions/2279133/getting-500-response-errors-to-get-me-calendars-al This error is causing calendars to fail to sync in Superhuman for users with Personal Outlook Calendars/Accounts. Gmail calendars are un-impacted. We are monitoring the bug from MSGraph and reviewing potential workarounds.
Report: "Degraded Service for Personal Outlook Calendars"
Last updateWe have identified a error in MSGraph responses impacting Calendars for Personal Outlook accounts.The underlying error is also reported here. https://learn.microsoft.com/en-us/answers/questions/2279133/getting-500-response-errors-to-get-me-calendars-al This error is causing calendars to fail to sync in Superhuman for users with Personal Outlook Calendars/Accounts.Gmail calendars are un-impacted. We are monitoring the bug from MSGraph and reviewing potential workarounds.
Report: "Slow response times for Ask AI and delay in indexing new messages"
Last updateThis incident has been resolved.
Our service provider has added additional resources to the cluster, and the indexing backlog and performance should be largely back to normal.
We continue to investigate the issue. AskAI queries are still slower than expected, and may not reflect all data from today. We will be working on upgrading the underlying hardware over the weekend.
We're continuing to investigate the root cause of slow queries.
We are currently investigating this issue.
Report: "Degraded Send operations for Microsoft users"
Last updateMicrosoft has deployed a fix for the authorization issues that were causing send failures for some Outlook users when sending with large attachments. The issue was resolved late last week, and sending functionality has since returned to normal. We appreciate your patience while this was being addressed.
We're seeing a 401 Unauthorized error affecting a small subset of Microsoft users when sending messages with attachments. After a day of investigation, we've determined the impact is limited and are moving to monitoring for the time being. We'll continue to watch the issue closely to ensure it does not escalate.
We're investigating an issue with sending messages with attachments for Microsoft users.
Report: "Unstable Send Behavior from Android Devices"
Last updateThis incident has been resolved. If you haven't updated to the latest version of Superhuman for Android, please do so by visiting the Google Play Store: https://play.google.com/store/apps/details?id=com.superhuman.mail
We have released a new version of the app (version 1.12.12, build 6411) to the Google Play Store that includes a fix for the issue. You can check your current version at the bottom of your settings screen. We're keeping a close eye on things and expect everything to be fully resolved soon.
We are continuing to work on a fix for this issue.
We identified the issue and are pushing out a fix soon. This unstable send behavior effects Android users when they forward a thread with an attachment. We will continue to update our progress here as more information becomes available.
We are currently investigating an issue where emails fail to send from Android devices. This appears to be isolated to Android platforms, and all other platforms are functioning normally. Our team is actively working to identify the root cause and implement a fix. We will provide updates as soon as more information becomes available.
Report: "Microsoft Outlook service disruption"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Outlook is currently experiencing a service disruption that may cause delays in email processing. The impact is limited to a small subset of users.
Report: "Ask AI Service Outage"
Last updateThe Ask AI service outage has been resolved. Full functionality has been restored, and the service is now operating normally. Thank you for your patience and understanding during the disruption.
We're currently experiencing a service disruption with Ask AI. Our team is actively investigating the issue and working to restore functionality as quickly as possible. We appreciate your patience and will provide updates as they become available.
Report: "Delayed sends"
Last updateThe issue causing email send delays has been resolved, and the system is now operational. We are continuing to ensure all emails affected during the delay are fully ingested.
We are currently investigating an issue causing email sends to be delayed
Report: "Degraded Ask AI Performance"
Last updateOur third-party provider has completed their rollback, and all services, including Ask AI and AI Compose, are now operating normally. We will continue to monitor, but no further impact is expected. Thank you for your patience.
Our third-party provider has identified the issue and is rolling back changes to restore stability. Our AskAI and AI Compose endpoints are functioning well, but customers could still see some increased delays. We are monitoring the rollback and will provide further updates as the situation progresses.
We are continuing to investigate this issue.
We are currently experiencing degraded performance with our Ask AI feature due to an ongoing issue with a third-party provider. This may result in slower response times or intermittent failures. We are actively monitoring the situation and working on mitigations where possible. We will provide updates as we learn more.
Report: "Delay on indexing new emails to Ask AI"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Superhuman inbox sometimes does not update on its own"
Last updateThis incident has been resolved - if you're experiencing this issue, please update your app.
A fix has been implemented - updating the app should remove the issue. We will continue to monitor to ensure this incident is resolved.
We believe we've identified the issue and are working on a fix. We will continue to update our progress here as we know more.
Report: "OpenAI outage impacting TL;DR and Semantic Search"
Last updateOpen AI's incident has been resolved and all systems are back to normal.
There is an OpenAI incident impacting TL;DR and Semantic Search https://status.openai.com/incidents/ctrsv3lwd797
Report: "Delays in Ask AI email indexing"
Last updateThe issue causing email indexing delays has been addressed, and the system is now operational. We are continuing to ensure all emails affected during the delay are fully ingested.
We have implemented a fix for the email indexing delay. While some lag in indexing persists, it is progressively improving. Our team is actively monitoring the situation to ensure indexing resumes at normal speed.
We have identified the root cause of the issue causing delays in email indexing for Ask AI. Our team is actively working on a fix to restore normal indexing functionality.
Ask AI: Email indexing temporarily delayed. Ask AI itself remains functional, but newly received emails may experience delayed indexing. We're monitoring the situation and will update as indexing resumes.
Report: "Microsoft Outlook service disruption"
Last updateThis incident has been resolved.
Microsoft has implemented a fix, and service is expected to be fully recovered by 7PM PST.
We are currently experiencing a service disruption affecting Outlook. Users may notice delays with reminders and inbox synchronization. Microsoft has identified the problem and are working on a resolution, but have not yet provided a timeline to resolve the issue.
Report: "Salesforce Performance Degradation"
Last updateSalesforce solved the incident. https://status.salesforce.com/incidents/13231
The second phase of the emergency release rollout is in progress.
Report: "AI Features Currently Down"
Last updateOnce our provider put in the fix, our semantic search and summary features returned to normal operation. All AI features are now fully operational.
The issue has been identified - one of our providers had an outage and is implementing a fix now. We will continue to monitor and communicate updates through this channel.
We are currently investigating an issue we're having with our AI services. Please follow here for further information as we learn more.
Report: "HubSpot Outage"
Last updateHubSpot has resolved the outage. Superhuman users are no longer facing errors and the HS integration should be stable now.
HubSpot is currently having a major outage. Superhuman users will not be able to - Access HS Social profile contact/company/deal info - Change HS contact/company/deal info - Connect/Disconnect from HS
Report: "Gmail Outage"
Last updateThis incident has been resolved.
The volume of errors sending and receiving emails for Gmail users is reducing and we're continuing to monitor the situation.
A Gmail/G Suite outage is currently affecting some Superhuman customers. The outage is affecting sending emails and emails sent to gmail accounts.
Report: "Delays in Ask AI email indexing"
Last updateAsk AI: Email indexing is working as expected. This incident has been resolved.
A fix has been implemented and we are monitoring the results.
Ask AI: Email indexing temporarily delayed. Ask AI itself remains functional, but newly received emails may experience delayed indexing. We're monitoring the situation and will update as indexing resumes.
Report: "Partial outage for Microsoft accounts."
Last updateThis incident has been resolved.
The number of failures has decreased and we are currently monitoring the issue.
We are seeing a number of send failures for Microsoft users. We are investigating the issue and will continue to update here as we know more.
Report: "Email sends are failing for some Gmail users"
Last updateThis incident has been resolved.
A fix has been implemented, and we are monitoring the results.
Gmail has updated their status page to show degraded service to sending emails and saving drafts with attachments. This is likely related, but we'll continue to investigate to confirm, and update here as we know more.
We are continuing to investigate this issue.
We're seeing a number of sends fail for some gmail users. We're investigating this issue and will continue to update here as we know more.
Report: "Delay in notifications"
Last updateThis incident was resolved. It was caused by slow responses from Gmail's API.
Notifications are back to normal; we are still monitoring the system
We are currently investigating this issue.
Report: "Backend slowness and disruptions"
Last updateThis incident has been resolved. All delayed sends should now be processed and delivered. Any outliers will be contacted shortly.
We have identified the issue and deployed a fix. The backend now appears stable and returning to normal. We are continuing to monitor to ensure we are fully stable. We will report here once we've confirmed we are fully stable.
We are experiencing an issue with backend slowness and disruptions. Some percentage of API calls are failing and sends are delayed. We are investigating the issue and will continue to report our findings here.
Report: "Attachment Upload Issue in Superhuman Browser Client"
Last updateFrom our investigation it appears some users were experiencing attachment upload issues. Possibly related to their antivirus software interfering with the Superhuman app's locally stored files. We've put in an initial fix and updated the client error reporting when encountering the error. A small number of users may see an upload error while adding an attachment in compose. If you encounter this don't hesitate to reach out to hello@superhuman.com. We're continuing our investigation to resolve the final remaining upload errors.
We are still actively investigating the issue and we will continue to keep you updated on our progress.
We are currently addressing an issue with the attachment upload feature on the Superhuman browser client. Some users may have encountered difficulties when attempting to upload attachments, resulting in files appearing with a size of 0kb and only a file name being present. Please be assured that our team is actively investigating this issue, and we are committed to resolving it as quickly as possible. We understand the importance of this feature in your workflow and will keep you updated on our progress.
Report: "Small percentage of sends are failing due to Microsoft API errors"
Last updateThis incident has been resolved. The Microsoft team has deployed a fix last week and we haven't seen any new occurrences since Thursday afternoon.
We're not seeing any new send failures and some previously failing sends have been retried successfully. We continue monitoring the situation to confirm that the problem has resolved itself.
We do not have an update on a fix, but we continue to monitor the situation. A handful of sends containing large attachments for Microsoft accounts continue to be unsuccessful. We do not have a way to circumvent at this time. We continue troubleshooting with Microsoft.
We are seeing a small number of sends fail due to errors with the Microsoft API. Most users are not affected. This only affects a small portion of emails with attachments over 3MB sent with Microsoft accounts. If one of your emails fails to send you will see it at the top of your inbox with a red dot. Our current recommendation is to find an alternative way to send the attachment, e.g. Microsoft OneDrive or Dropbox. We are in communication with Microsoft about this, and hope to have more information to share soon.
Report: "Elevated Google Calendar API Errors"
Last updateThis incident has been resolved.
Google has released a fix for the Google Calendar APIs. We haven't received any new errors and will continue monitoring.
Google Calendar APIs are currently experiencing an outage. Some Google Users may encounter issues with their calendar.
Report: "Some Microsoft Users are getting logged out"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue where some Microsoft Users are getting logged out
Report: "Some notifications are not being delivered"
Last updateThis incident has been resolved.
Our internal monitoring systems are no longer detecting any notification delivery errors. We are continuing to monitor for any further issues.
Beginning at approx 11:57 am PST, an issue with one of our service providers has caused an outage in delivering some user notifications. More information about the underlying outage is available here https://status.firebase.google.com/incidents/NowVgaeZF7G8Fbdh9jzU We are continuing to monitor the issue.
We have identified that some notifications are failing to be delivered. We are investigating this issue.
Report: "Notifications are not being delivered"
Last updateThis incident has been resolved.
Our internal monitoring systems are no longer detecting any notification delivery errors. We are continuing to monitor for any further issues.
Beginning at approx 6:03 am PST, an issue with one of our service providers has caused an outage in delivering most user notifications. More information about the underlying outage is available here https://status.firebase.google.com/incidents/9ZPv9faHLen8bzLVSaft We are continuing to monitor the issue.
Report: "Microsoft Outlook service disruption"
Last updateMicrosoft has confirmed the issue is resolved after an extended period of monitoring. All systems are back to normal.
We are continuing to monitor for any further issues.
Microsoft is reporting the majority of impact has been repaired, with service availability at 99.9%. We'll continue to monitor until Outlook is back to 100%.
Microsoft has implemented a fix and we are starting to see a gradual recovery.
Microsoft has identified the problem and are working on a resolution, but have not yet provided a timeline to resolve the issue. Email sending and receiving is still degraded for some Outlook accounts.
Outlook is experiencing a service disruption and email sending might be delayed.
Report: "Network Issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Beginning at approx 9:46 am PST, an issue with one of our service providers caused an outage in multiple Superhuman features. During this outage: - some users were unable to load mail.superhuman.com. - all users were unable to send new emails. - most users were able to read and receive new emails. This issue began to clear up at 10:08 am PST. Between 10:08 am PST and ~10:22 am PST, users were able to read, send, schedule, and receive new emails, with some delays. As of ~10:22 am PST, Superhuman users are able to read, send, schedule, and receive new emails as usual. More information about the underlying outage is available here https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh. We are continuing to monitor the issue.
Issues with our service provider appear to be clearing up. We are continuing to monitor the issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Gmail Outage"
Last updateGoogle has resolved the relevant incident on their side. Emails are now sending successfully to gmail users again.
We are at this point sure that the issue is on the Gmail receiving side, as emails sent from any email client to a gmail address are bouncing. We have raised the issue with Google, and will let you know when we know more
We are continuing to investigate this issue.
We are continuing to investigate this issue.
A Gmail/G Suite outage is currently affecting some Superhuman customers. The outage is sending emails and emails sent to gmail accounts.
Report: "Gmail outage"
Last updateGoogle has resolved the relevant incident on their side, and emails are now sending successfully from Superhuman again.
We’re no longer seeing a high error rate from the Gmail API. Superhuman should be working again for everyone, and we will resolve this incident when all the emails sent during the outage have successfully retried.
We're seeing signs this is improving for everyone. In the meantime, any emails you send within Superhuman will be queued up and sent when Gmail is working again.
A Gmail/G Suite outage is currently affecting some Superhuman customers. The outage is affecting logging in and sending emails
Report: "Gmail outage"
Last updateThis incident has been resolved.
Everything seems back to normal. We continue to monitor the situation.
A fix has been implemented and we are monitoring the results.
A Gmail/G Suite outage is currently affecting some Superhuman customers. The outage is affecting logging in and sending emails
Report: "Elevated send failure rates"
Last updateAnd we're officially back! We stopped seeing elevated error rates at around 3am PST, and Google officially resolved the incident at 4:19am. If you had emails that failed to send, they'll show up in Superhuman and you can tap on "Send Failed" to retry. A small subset of reminders that should have triggered between 10pm and 2am PST were delayed until 2am PST.
The error rate we're observing on the Gmail API has been trending down significantly, and for the last 30 minutes our send failure rate has stayed below 1%. Although we're not ready to close this issue yet (we're still seeing many slow sends, and a much-higher-than-normal failure rate) most people should see everything working again. If you had emails that failed to send, they'll show up in Superhuman and you can tap on "Send Failed" to retry. Any reminders that should have returned to your inbox during the time that Gmail's API was down will have returned just now. Google continues to work on the underlying issue, and they post updates here: https://status.cloud.google.com/incident/zall/20008
We're now seeing close to 40% of emails fail to send, and a similar number of reminders are not returning on time (as they rely on a similar Gmail API). By design Superhuman will retry about 10 times over the course of several minutes until sending succeeds. While that's happening we optimistically include the emails in your Sent Items (usually this makes everything feel blazingly fast!). The retry logic means that during an outage like this some emails will send more slowly than usual (when it retried many times before working), while others will not send and re-appear in your Inbox as failed after several minutes. (Pro-tip: to tell if an email in your sent items is really sent, or still being tried, try to add a label to it cmd+k "Label" will not show up until an email is sent successfully). Google's incident coverage is here: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a45de3b26d6c5872f4cfe8e3424d7a82.
Currently about 10% of emails sent via Superhuman are failing due to errors communicating with the Gmail API. If this happens to you it will show up in the client with a red dot that says "send failed". Tapping on the "send failed" will let you re-send your email if you'd still like to.
A small portion of emails sent from Superhuman are failing at the moment due to an upstream outage on Gmail's API. We're investigating and will update when we have a better idea of the impact and potential resolution steps.
Report: "Network connectivity problems"
Last updateThis incident has been resolved.
Our provider has resolve the underlying outage and everything should be back to normal.
It looks like our DNS provider is currently down. We will continue to monitor the situation, and investigate possible short-term mitigations.
We are continuing to investigate this issue.
We are seeing connectivity issues connecting to Superhuman, and are investigating.
Report: "Connection issues and delayed sends"
Last updateThe database is back up and running in a stable configuration, so we're considering this incident resolved for now. We will be making further changes around midnight PST this evening to re-enable high availability on the instance. This will require approximately 10 minutes of downtime.
We are about to revert the workaround, this will cause under 2 minutes of further downtime; but will restore Superhuman to a stable state. Your client may appear offline for a few minutes, but will reconnect and re-send any outstanding emails when the service returns to normal.
A fix has been implemented and we are monitoring the results.
We have implemented a workaround, which we hope will restore Superhuman access temporarily; and we continue to stay in touch with Google's team as they work on fixing the root cause https://status.cloud.google.com/incident/compute/19012. Queued up emails should begin to send when your client next reconnects
Report: "Emails sending slowly"
Last updateWe've fixed the slow-down and ensured that all delayed emails have been sent! 😊
We've noticed some instances of emails sending slowly. We're investigating this as our top priority now and will share updates here and at twitter.com/superhuman as we learn more.
Report: "Problems signing into Superhuman"
Last updateGmail has fixed the underlying issue and logging into Superhuman should be working now for all users. If you see any further issues, or have any questions, don't hesitate to reach out at hello@superhuman.com or twitter.com/superhuman.
Gmail's team has identified the problem and logging in is now working for most people.
We are experiencing some issues with Gmail right now, which may affect logging into Superhuman. We'll be sharing updates here and on twitter.com/superhuman as we learn more.
Report: "Emails are slow to send"
Last updateThe incident has been resolved. Thanks for your patience!
Emails take a longer time than usual to send. We're investigating the issue. No data has been lost.
Report: "Delayed email sending"
Last updateGoogle have resolved the relevant incident on their side, and emails are now sending successfully from Superhuman again! If any of your emails failed to send during the last few hours, they will show up in your inbox in Superhuman and can be retried or skipped as appropriate.
We're seeing an increase in delayed sends as the issues with Google Cloud and Gmail continue, affecting about 15% of emails sent. We'll continue monitoring the situation and let you know as things change. https://www.google.com/appsstatus https://status.cloud.google.com/
A fix has been implemented and we are monitoring the results.
We're monitoring an issue that causes delays to send emails in some instances. This is due to networking issues with our hosting provider, Google Cloud. Google Cloud is aware of the issue and is actively working on fixing this issue. Your emails are safely stored on your device, and will send when the issue is resolved
Report: "Redis connectivity"
Last updateThe issue is fully resolved, and all emails that were delayed have now sent. Please don't hesitate to reach out to hello@superhuman.com if you have any questions.
We are experiencing some issues with Redis connectivity right now which may affect sending emails. We are working hard to resolve as quickly as possible.
Report: "<0.5% of sends are failing due to Gmail API errors"
Last updateThis incident seems resolved for all Superhuman users. Gmail has not yet closed their issue yet, but the failure rate that we observed has dropped to 0.
Update from Google here: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=e287a05e9ee515a2ad24a58eaa5b2e33.
We are seeing a small number of sends fail due to errors with the Gmail API. This is not affecting most users. If one of your emails fails to send you will see it at the top of your inbox with a red dot, and you can either resend it or discard it by clicking on the red dot. We think it's likely that retrying the email manually will successfully send it. We are in communication with Google about this, and hope to have more information to share soon.
Report: "Google Account Login Outage"
Last updateIt appears these login issues are fixed! We'll stay on the lookout just in case anything else crops up, but you should be able to return to your regularly scheduled emailing now. :-)
Google login seems to be getting better, so you may be able to log in now if you couldn't earlier. We're still keeping an eye on things and will share any more progress we see.
We’re seeing problems with signing into Superhuman. The cause is an outage of Google’s login system that also prevents signing into Gmail. We’re in touch with Google and will share updates here as we have them. If you like, you can also follow along here on Google's status page: https://status.cloud.google.com/incident/cloud-iam/19001
Report: "Gmail attachment failures"
Last updateAccording to Google (and our metrics) everything should be back to normal. Thank you for bearing with us. Emails that were sent during the outage, and which you didn't cancel or retry yourself, have now been retried and have sent successfully. If you have further questions, please get in touch conrad@superhuman.com.
It looks like the error rates have returned to normal. We're going to keep an eye on the situation and ensure your emails are sent correctly. We'll resolve this completely when we hear good news from Google.
We've been in touch with the Gmail team, and they are working on the issue. It is affecting users of mail.google.com too, so they expect to have a resolution quickly. We'll keep this incident open until we hear back from them that the issue is resolved.
We are investigating an issue with Gmail attachments. This may cause sends to fail when you reply or forward an email that contains an attachment, and will mean that you cannot download attachments from Gmail. We've escalated this to the Gmail support team, as it seems to be an outage with that API endpoint.
Report: "Planned Maintenance"
Last updateThis maintenance was completed successfully. We're monitoring things closely, but everything appears to be back to normal. Please reach out to me if you have any questions: conrad@superhuman.com Conrad
We are upgrading our database configuration after the outage yesterday. We estimate that this will require ~10 minutes of downtime starting at 11:50 pm PST (our lowest traffic point of the week). During this window, emails will receive as normal but they will not send. Any emails that you do send will be safely stored within Superhuman, and will deliver just as soon as connectivity is restored. Please reach out to me if you have any questions: conrad@superhuman.com Conrad CTO
Report: "Database outage"
Last updateHello all, Yesterday, Superhuman was down for nearly two hours due to a failure with our database. I am deeply sorry for this. We know that email is mission critical, and that this much downtime is unacceptable. During the downtime, emails could not be sent and it was not possible to log into Superhuman. If you were already logged in, you could still receive emails. If you were not logged in, then it was not possible to log in. The failure was due to two simultaneous issues: 1\. Our database was running low on disk space. 2\. One of the availability zones that our database runs in was unable to provision more disk space. For our database, we use Google Cloud SQL in High Availability mode. We also use the built-in feature to “automatically increase disk space”. We failed to realize two important things about this setup: 1\. The automatic disk space increase is very conservative. Based on current load, it would only allocate enough space for a few additional hours at peak traffic. 2\. Increasing disk space is an operation that requires both availability zones to be active. We spoke with Google Cloud Support who explained all off this in detail, and then we took the decision to temporarily disable high availability so that we could resize the primary database. This is the timeline of events: 09:40. The auto-scaler detected we had less than 25GB of free space and started to increase capacity, but this failed. 12:03. Our database ran out of disk space. 12:03-12:11. We tried to manually increase disk space and failover to another zone, but these both failed. 12:11. We opened a ticket with Google Cloud Support. 12:59. We were on the phone with Google, who provided a detailed explanation of the issue. 13:30. We disabled high availability on our database, and resized it in the working zone. 13:34. Our database was back up again. 13:34-13:50. Clients began to reconnect and send email. 13:59. Normal operations resume, though our database is not high availability for the time being. As a result of this incident, we are going to make several changes: 1\. Tonight, we are going to re-enable high availability on our database. This will cause ~10m of downtime, but we will do it when we have our lowest traffic: 11:50 pm PST. 2\. We have built our own database auto-scaler that will trigger much before the built-in auto-scaler. 3\. We have added alerting on database disk-utilization metrics so that we can pre-empt any similar failures. 4\. We will fix the client so that if the backend is unexpectedly down, it will not log you out so that you can continue to read and process email. 5\. We are going to practice failing over to our secondary read replica. This will be helpful if we are ever again in a situation where both our primary-replica pairs are not functioning. Again, I am truly sorry that this happened. These steps will ensure that we do not have a similar incident in the future. If you have any questions, please just ask: [conrad@superhuman.com](mailto:conrad@superhuman.com). Conrad CTO
The issue is fully resolved, and all emails that were delayed have now sent. Please don't hesitate to reach out to hello@superhuman.com if you have any questions.
Our database is now back up, and all systems are functioning normally. All emails that were sent during the outage should go out shortly. We will now conduct a thorough investigation into what happened, and we will ensure it cannot happen again.
We are continuing to investigate this issue.
We're seeing elevated errors with our database. Investigating…
Report: "Inaccessible from some locations"
Last updateThis incident has been resolved.
This turned out to be a mistake on our end, which is now resolved. Access should be restored everywhere over the next few minutes
We are currently investigating an issue where Superhuman is inaccessible from some locales, we're working with our infrastructure provider to debug.
Report: "Delayed sending and signing in"
Last updateWe’ve now fixed the underlying issue, and notified all users who’s emails did not send.
A fix has been implemented and we are monitoring the results.
Message sending and signing in should now be working normally. We're still investigating the underlying root causes internally.
We're investigating an issue that causes sending and signing in to Superhuman to slow, and sometimes fail. Your emails are safely stored on your device, and will send when the issue is resolved.
Report: "Sending delayed"
Last updateWe have resolved the issue, and all emails have sent.
We're investigating an issue causing emails to send slowly.
Report: "Sending delayed"
Last updateWe've confirmed that all emails sent during the outage have successfully sent.
A fix has been deployed. Emails will send successfully the next time Superhuman retries (usually 1-10 minutes). We will continue monitoring to ensure that all emails go out.
We investigating an issue that is causing emails to send slowly. All emails are safely stored on your device, and will automatically send when the issue is resolved.