Is SafeBase Down Right Now? Check if there is a current outage ongoing.

We are experiencing this issue due to the multi-regional GCP outage. Check their status page here: https://status.cloud.google.com/

We are currently investigating this issue.

Our dashboard provider is having an outage, which is causing the dashboards in the app to fail. We are closely monitoring the situation. More information can be seen at https://status.explo.co/

This was a downstream GCP issue which has since been resolved.

We are currently investigating this issue.

This was a downstream GCP issue which has since been resolved.

We are currently investigating this issue.

A connection with third-party service that was interrupted due to scheduled maintenance. This caused our settings page to become briefly unavailable. The issue has been resolved, and new monitoring tasks have been added to prevent similar issues in the future.

We are continuing to investigate this issue.

We are currently investigating this issue.

The SafeBase team identified the problem and released a hotfix.

The issue has been identified and a fix is being implemented.

All free TCs proxied through Fly.io should be back online. We will continue to still monitoring to ensure things remain stable.

Our Fly.io reverse proxy is currently down due to a Fly.io issue. We are working with them to restore access as soon as possible. This only affects free customers whose public Trust Centers are proxied through Fly.

The incident is resolved.

A fix has been implemented, we are continuing to monitor the issue.

We are currently experiencing an outage with our email and Slack messages. This means that all emails and Slack messages are not sent at this point in time.

This has been fixed.

Our Fly.io reverse proxy is currently down due to a Fly.io issue. We are working with them to restore access as soon as possible. This only affects customers whose public Trust Centers are proxied through Fly.

This incident has been resolved.

A hotfix has been released and we are monitoring for continued issues.

We are currently investigating an issue with Tray.io, our middleware provider that allows for HubSpot, Salesforce, and Jira integrations. We are engaged with the provider and are working with them to restore service ASAP.

The DDoS attack has been stopped and we have added additional mitigations to reduce the risk of future DDoS attacks.

We are experiencing a DDOS attack, we are handling it now.

* On August 29, 2023 at 10:38 PM ET we sent an email to a select group of customers using our [Trust Center Updates](https://help.safebase.io/en/articles/6082911-how-to-post-a-trust-center-update) custom audience feature. * Within a few minutes, several customers alerted us to their inboxes receiving multiples copies of the same TCU email. We began investigating, and we determined that this was a bug with our email notification management system, Courier. * Within 20 minutes we were able to write a script to cancel all emails that were queued up and waiting to be delivered. However, we noticed that the queue continued to be rebuilt, so we contacted Courier support. To mitigate the number of emails being, we continuously ran this script to reduce the likelihood of additional duplicate emails. * Courier support joined us on a Zoom call at 12:01 PM ET and advised that we halt all Courier related emails until the issue was sorted out. * At 1:00PM ET the Courier team notes that they discovered the issue and would have a hotfix available within an hour or two. The root cause of this bug was related to logic that attempted to continuously resent this email due to an issue related to s3 timeouts/socket connections and an improper way of marking the emails as delivered when this happened. * At 3:56 PM ET the Courier team informed us that the fixes were in place, and that emails could safely be delivered again. * At 4:00 PM ET we re-enabled the Courier integration to allow for emails to flow once again. The Courier team began to gather any emails that were not sent during this outage period. * At 7:23 PM ET the Courier team confirms that all emails that were stuck during the outage period were delivered.

All emails in the backlog have been processed.

Emails that were not delivered during this outage are now being sent out. We appreciate your patience as we ramp back up to full operational capacity.

A fix has been deployed and we are monitoring. Emails that were backed up during the outage will be processed shortly.

Courier has noted another fix is on the way that will require another 60 minutes. We appreciate your patience.

The downstream vendor, Courier, has identified the bug on their end and are working on deploying the fix now. The ETA is 90 minutes. We will be working with them to send out any queued emails that were generated during this outage.

We are continuing to investigate this issue.

We have temporarily stopped email notifications until our email provider has a fix in place.

The issue is with our email infrastructure provider. We are working with them to resolve the issue urgently. Some customers might be receiving duplicate emails.

We are investigating an issue where certain trust center updates are sending repeatedly to recipients if uploaded via a CSV.

This was determined to be due to a downstream issue from Cloudflare, who have since issued a fix.

We have identified the root cause and are working with our infrastructure provider to resolve those issues. Customers experiencing a hangup issue can refresh the browser, or go directly to app.safebase.io again in order to work around this in the meantime.

The issues are still not completely over.

We have deployed a fix and we're monitoring to make sure the issue isn't happening anymore.

The app sometimes hands up for some users. We are investigating the root cause

This issue has been resolved and the app should be functioning as normal again.

We have identified the issue as a DDoS attack against our app. We have blocked the attack and we are monitoring the situation.

We have received reports that users are currently seeing periodic 500 errors for our main application. We are investigating and will provide an update as soon as we can.

This incident has been resolved and all emails have been sent

The issue has been resolved. We are verifying that all emails that had failed get resent properly

Emails should be returning to normal. Our engineering team will continue to monitor.

We are currently having an issue with our email infrastructure service and emails are not being sent out.

The issue has been resolved

The Portal management page (/portal) in app.safebase.io is not working for some customers. Public portals are working as normal. We are working on the issue and it should be resolved within an hour.

The error seems to have just been a slight delay, no user impact has occurred.

We have noticed an error causing emails to not be sent in some cases. We are currently investigating.

Postmark service issue seems to have been resolved and all emails in the queue have been delivered.

Postmark, SafeBase's email delivery provider, is having an outage at the moment. This is causing emails to be delayed. Once service is restored, all emails in the queue will be delivered.

Auth0 has resolved the issue on their end.

Auth0 is having global downtime, causing our main app login flow to randomly fail. If you are running into this issue, refreshing the page can help. We're monitoring the situation and will update when the service is back up. You can follow Auth0 here: https://status.auth0.com/

This has been resolved.

We are continuing to investigate this issue.

We are currently investigating an issue with our email provider and emails not being sent out.

AWS outage seems to have been resolved and our third party vendors are functioning properly again

Our email provider (Courier) is having a partial outage (https://status.courier.com/) due to an ongoing AWS incident (https://status.aws.amazon.com/). We are monitoring the situation and will update when these systems are back online.

This downtime appears to be resolved.

We are continuing to investigate this issue.

We are currently waiting for updates from Google, which is having a major outage with GCP at the moment.

We have discovered a fix and have deployed it.

We are currently investigating this issue.

We have resolved the issue.

We are currently investigating this issue.

When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.

Empathize with those affected and let them know everything is operating as normal.

As you continue to work through the incident, update your customers frequently.

Let your users know once a fix is in place, and keep communication clear and precise.