Historical record of incidents for Airbrake
Report: "Reduced performance in API and error dashboard"
Last updateUpdate: A subset of customers experienced service disruptions due to an unresponsive backend API instance. This was triggered by a log storage issue that correlated with a surge in error ingestion traffic. We are actively investigating automated solutions to prevent similar occurrences in the future. We apologize for any inconvenience this caused, please reach out with any questions to support@airbrake.io.
We have resolved the backend issue where a subset of Airbrake projects' experienced blank error dashboards. The backlog in error events have been processed. Please reach out to support@airbrake.io if you have further questions.
We have identified an issue affecting a certain subset of customers and are currently working on remediation. We'll keep this issue updated. Please reach out to support@airbrake.io if you have further questions.
Report: "Reduced performance in API and error dashboard"
Last updateUpdate: A subset of customers experienced service disruptions due to an unresponsive backend API instance. This was triggered by a log storage issue that correlated with a surge in error ingestion traffic. We are actively investigating automated solutions to prevent similar occurrences in the future.We apologize for any inconvenience this caused, please reach out with any questions to support@airbrake.io.
We have resolved the backend issue where a subset of Airbrake projects' experienced blank error dashboards. The backlog in error events have been processed. Please reach out to support@airbrake.io if you have further questions.
We have identified an issue affecting a certain subset of customers and are currently working on remediation. We'll keep this issue updated. Please reach out to support@airbrake.io if you have further questions.
Report: "Email notifications for alerts have been disabled"
Last updateThis issue has been resolved. Our emails have been flowing for the past day and our sign-up page is live again. We apologize for any inconvenience. If you have any questions, please reach us at support@airbrake.io.
Updating the status to "monitoring" as alert emails are fixed. We will update the status to "resolved" when the new sign-up page is live.
Our email provider has reenabled our alerts domain. All email notifications should be operational again. We still have our sign-up page disabled to implement an additional verification measure and will close this incident when the new sign-up page is live. If you have any questions, please reach us at support@airbrake.io.
Our alerts notifications have been disabled. Digests and other notification types (like Slack) are unaffected. We are working with our email service provider to reenable our alerts domain and will follow up as there is more information. Reach us at support@airbrake.io if you have any questions.
Report: "Email notifications not delivering"
Last updateThis issue has been resolved. Emails are being delivered again. Reach out to support@airbrake.io if you have any questions.
We have identified an issue with delivering email alerts. We are investigating the cause of the disruption and will report more as we gather more info. Errors are still being recorded and are being delivered to your error dashboards. Integration alerts are also unaffected.
Report: "Processing delay for Errors and APM for some customers"
Last updateWe apologize for any inconvenience this caused. The incident has been resolved and normal operations have resumed.
We have identified an issue affecting a certain subset of customers and are currently working on remediation. You may notice a delay in Error processing. We'll keep this issue updated. Please reach out to support@airbrake.io if you have further questions.
Report: "Delay in error processing"
Last updateThe backlog in error notifications has been processed. Users should get error emails and see errors show up on their dashboards without delays again. We apologize for any inconvenience. If you have any questions, please reach us at support@airbrake.io.
We have identified the issue and are currently working to process the backlog of notifications. We will continue to monitor the situation and provide updates.
We are currently investigating a delay in error processing. Errors on your error dashboard and error email notifications might take longer to appear than usual. We will share updates as we make progress in addressing this issue. If you have any questions or concerns, please reach us at support@airbrake.io. We apologize for any inconvenience.
Report: "Increased Errors"
Last updateThis incident has been resolved. Services should be back to normal.
We are currently investigating increased errors.
Report: "Increased queue size, Delayed notifications"
Last updateThis incident has been resolved.
Queues are back to normal and notifications are once against being delivered.
We are continuing to work on a fix for this issue.
We've identified an issue with our queue. Notification delivery is currently delayed. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email support@airbrake.io
Report: "Airbrake web app temporarily unavailable"
Last updateThis incident has been resolved, the Airbrake web app is now working as expected. If you have any issues or questions please email support@airbrake.io
Some users may not be able to access the Airbrake web app. We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email support@airbrake.io
Report: "Degraded dashboard experience"
Last updateThe cause of the incident has been identified and resolved. Systems have returned to their operational states.
Some users may not be able to load error, performance, or deploy data in their dashboard. The API is accepting errors as usual and you should be able to continue troubleshooting errors as soon as the dashboard has been restored. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email support@airbrake.io.
Report: "Degraded performance"
Last updateThe backlog of notice and apm data has been processed and shard performance has stabilized. Please reach out to support (support@airbrake.io) with further questions.
A fix has been implemented and we are monitoring while the backlog of notice data / APM events are being processed.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating degraded performance on a database shard. Some customers may experience delays processing error and APM data.
Report: "Error dashboard temporarily unavailable"
Last updateThis incident has been resolved.
Some users may not be able to access the error dashboard. The API is accepting errors as usual and you should be able to continue troubleshooting your errors as soon as the dashboard has been restored. We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email support@airbrake.io
Report: "Notice and APM processing delay"
Last updateWe've applied a change that relieved the load on our production database that was causing a delay in processing performance data. Delays in application performance should be resolving. If you continue to experience any issues, please let us know at support@airbrake.io.
We are continuing to work on a fix for this issue.
We are continuing to investigate the issue causing a delay in processing performance data. Customer's project dashboards and email notifications should be returning to normal. If you experience any issues or have any questions, reach us at support@airbrake.io
We have identified an issue where for some requests, our API is failing due to data caching errors. Affected users might experience their error dashboards failing to load and a delay in email notifications. If you have any questions or concerns, reach out to support@airbrake.io.
We are continuing to investigate an issue.
We are currently investigating API errors which are affecting some projects.
Report: "Inbound API errors"
Last updateThe overloaded ingress server which caused issues with deploy and error tracking was scaled up and the issues have been resolved. If you have any questions, please reach out to us at support@airbrake.io.
All systems are back to normal, we will continue to monitor the situation. If you experience any issues sending errors or tracking deploys for your project, please let us know support@airbrake.io
We've received reports of issues with tracking deploys, we are investigating this and believe it to be a related symptom. If you are experiencing this issue please let us know support@airbrake.io, we apologize for any inconvenience this causes.
We are continuing to investigate this issue.
We are currently investigating API errors which are affecting some projects.
Report: "Processing delays and API errors affecting some projects"
Last updateThe issue causing the API errors and delay in error notice processing has been resolved. The backlog has been processed and systems are operating normally. We apologize for the inconvenience. Please reach us at support@airbrake.io if you have any questions.
A fix has been implemented and we are monitoring the results. Our APIs are back to normal and there are no more processing delays. Please contact the support at support@airbrake.io if you have any questions.
We have identified the issue causing API errors and processing delays which is affecting some projects. We are currently working to mitigate the outage in processing and are restarting core processing components. We apologize for the inconvenience. If you have any questions or concerns, please reach us at support@airbrake.io.
We are currently investigating API errors and processing delays which are affecting some projects. Some errors will be delayed but not lost. Sorry for the inconvenience.
Report: "Delay in error processing and dashboard timeouts"
Last updateThis incident has been resolved, you should no longer see any delay in error processing or dashboard loading. No error data was lost during this incident, only delayed. We apologize for any inconvenience this has caused! If you have issues or questions, please email support@airbrake.io
We are continuing to investigate this issue.
We’re experiencing delays in error processing and some customers may experience timeouts when loading dashboards. No error data is being lost and we are actively working to resolve this issue. If you have any questions or issues, please contact us at support@airbrake.io.
Report: "Delay in error processing"
Last updateAll systems are back to normal and the backlog of data has been fully processed. Customers impacted by the delay should see errors again in their dashboard. If you have any questions or issues, please contact us at support@airbrake.io.
A fix has been implemented and we are monitoring the results. For customers experiencing a delay the back log of errors is currently being processed.
We’re experiencing periodic delays in error processing. Some customers may also experience an empty error dashboard. No error data is being lost and we are actively working to resolve this issue. If you have any questions or issues, please contact us at support@airbrake.io.
Report: "Delay in error processing"
Last updateAll systems are back to normal and the backlog of data has been fully processed. Some errors that occurred during the last (approximately) 12 hours will have been late to appear on your dashboard and to raise notifications. During this incident, we had some large delays in processing errors resulting in out-of-date dashboard data and late error notifications. This is not acceptable service and for this, we're very sorry. We're working hard to learn and improve from what happened.
The fix was successful and is complete, we expect to be fully restored to normal operation very soon. We are processing a large backlog of error data and as a result, errors that occurred in your projects during the last ~12 hours are likely to start appearing and raising notifications. We are extremely sorry for this delay and unsatisfactory service. We'll close with a more detailed postmortem, soon.
A fix is being implemented. We hope to be fully operational in approximately 1 hour, but it's difficult to predict at this time. We are very sorry for this major disruption. We are working hard to resolve it quickly. If you have any questions or concerns, you can reach us at support@airbrake.io.
Some customers may also experience an empty error dashboard. We're working hard to fix this as soon as possible.
We are continuing to investigate this issue.
Some accounts may be experiencing delays in error processing. We’re actively investigating and will post an update once we learn more. If you have any questions, please contact us at support@airbrake.io.
Report: "Delay in error processing"
Last updateThis incident has been resolved.
We’re experiencing some delays in error processing and we’re actively working on a fix. If you have any questions, please contact us at support@airbrake.io.
Report: "Intermittent redirect loops"
Last updateWe've identified the issue and pushed a fix. Please email us at support@airbrake.io if you have any further issues or any questions.
We are currently investigating an intermittent issue causing redirect loops when trying to view a project. We apologize for the inconvenience, this issue is not affecting error processing. Please email any questions or concerns to support@airbrake.io
Report: "Most email notifications not being sent."
Last updateEmails are back to full flow but. Our email provider is monitoring their fix. Many email notifications are lost. We're very sorry for this inconvenience. We'll continue to monitor for any changes.
Many (but not all) email notifications are flowing again. Our email provider has identified the problem and is working on a fix.
Most email notifications are not being sent. Our email provider has problems. We're monitoring and will provide updates as soon as possible. More details here http://status.mailgun.com/incidents/5jz5w33sx2qv
Report: "Delay in notifications"
Last updateThe issue with notifications has been resolved. No errors were lost and they continue to be processed normally, however some notifications were undelivered. If you have any questions, please reach us at support@airbrake.io.
We're investigating an issue causing a delay in notification delivery. Errors are being processed as normal. If you have any questions or concerns, please reach us at support@airbrake.io.
Report: "Delay in error processing"
Last updateThis incident has been resolved.
We’re experiencing periodic delays in error processing. No error data is being lost and we’re actively working to speed up the error processing. If you have any questions or issues, please contact us at support@airbrake.io.
Report: "Delay in error processing and notifications"
Last updateThe delay in processing errors and notifications has been resolved. The backlog in errors is being processed. If you have any questions or issues, please contact us at support@airbrake.io.
We are currently investigating a delay in error processing and notifications. Please reach out to us at support@airbrake.io if you have any questions or concerns. We will post updates as we address the issue.
Report: "Delay in error notifications"
Last updateThe delay in error notifications has been resolved and the backlog is being processed. If you have any questions, let us know at support@airbrake.io.
We're currently investigating a delay in error notifications. The error dashboard and web app are unaffected. Reach us at support@airbrake.io if you have any questions.
Report: "Dashboard & website unavailable, intermittently"
Last updateThis is resolved, sorry for the bumps. We're working on preventing future occurrences.
The dashboard and website are unavailable intermittently.
Report: "Dashboard unavailable"
Last updateAll previously affected users should be able to successfully access their error dashboards again. Please let us know if you have any pending question or concerns at support@airbrake.io.
We are investigating an issue with users' dashboards not loading. We will update you here once we learn more. Please reach us at support@airbrake.io if you have any questions or concerns. We apologize for the inconvenience.
Report: "Error dashboard unavailable"
Last updateWe have resolved the issue and access to error dashboards has been restored. Please reach us at support@airbrake.io If you have any concerns or questions.
Users may experience issues loading their projects' error dashboards. We're working on resolving the issue and will post an update soon. Please reach us at support@airbrake.io if you have any questions.
Report: "Issue with DNS resolution for European users"
Last updateOur DNS provider is no longer experiencing issues. The subset of European Airbrake users affected should have access to their error dashboards again. Airbrake error processing was unaffected. More information: https://dnsimple.statuspage.io/incidents/d667pmrhw34v If you have any questions, please reach us at support@airbrake.io.
Our primary DNS provider is experiencing DNS resolution issues in some parts of Europe. Some users may be unable to access the dashboard and other services. More details: https://dnsimple.statuspage.io/incidents/d667pmrhw34v
Report: "Issues viewing errors"
Last updateAll systems are back to normal. Sorry for any inconvenience caused.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue viewing errors. Error processing is temporarily delayed.
Report: "Error processing delay"
Last updateAll systems are recovered and operational. We apologize for any inconvenience this caused. If you have any questions, please contact us at support@airbrake.io.
We are currently investigating a delay in error processing. We apologize for any inconvenience. If you have any questions or concerns, please contact us at support@airbrake.io.
Report: "Error dashboard temporarily unavailable"
Last updateAll systems are recovered and operational. Sorry for the inconvenience. If you have any questions, please contact us at support@airbrake.io.
Some users may not be able to access the error dashboard. We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email support@airbrake.io
Report: "Delay in queueing service"
Last updateAll systems are recovered and operational. Sorry for the inconvenience. If you have any questions, please contact us at support@airbrake.io.
As of 12:37 pm PST we are fully operational. We are continuing to monitor and harden the fix. More information will follow.
We're currently investigating an issue with our queueing service. Error emails may experience delays. Please contact us if you have any questions or concerns at support@airbrake.io.
Report: "Web app unavailable and delay in error processing"
Last updateThis issue has been resolved. Queued errors have been processed so there should be no delay in error email or integration notifications. Please let us know if you have any questions at support@airbrake.io. We apologize for any inconvenience.
You may notice a delay in error email and integration notifications, as errors are processed from the queue this delay will decrease. Your project's error dashboard will have the most up to date error information for your debugging needs.
The web app is back and the error dashboard is available again. Queued errors are being processed and should arrive soon in your error dashboard. We apologize for any inconvenience, please let us know if you are experiencing any further issues.
Our team is actively working to restore access to the web app and resume normal error processing. We are currently queueing incoming errors, sorry for any inconvenience this has caused. If you have any questions or concerns please email support@airbrake.io
Report: "GitHub required logins"
Last updateGitHub's service outage has been resolved. Airbrake accounts using the GitHub integration to log in, create issues, or create projects should have full, uninterrupted functionality again. If you have questions or problems, please contact us at support@airbrake.io.
Due to GitHub's ongoing service outage, Airbrake accounts using the 'require GitHub logins' security feature may experience intermittent failures when logging in: https://airbrake.io/docs/integrations/github-logins/#security-options GitHub issue creation and automatic project importing may also experience issues. If you have any problems logging in, questions, or concerns, please contact us at support@airbrake.io.
Report: "Delay in error processing"
Last updateThe delay in error processing for the subset of affected accounts has been resolved. The backlog of errors has been processed. We apologize for any inconvenience. If you have any questions, please reach us at support@airbrake.io.
We are currently investigating a delay in error processing for a subset of accounts. We apologize for any inconvenience. If you have any questions or concerns, please contact us at support@airbrake.io.
Report: "Error charts loading intermittently"
Last updateError charts are operational. No other services affected. Thank you for your patience and sorry for the inconvenience. Please email any concerns or questions to support@airbrake.io
Error charts are loading intermittently. We are currently investigating this issue. No other services are affected. We apologize for the inconvenience.
Report: "Error trend charts failing to load"
Last updateError trend charts are back online. No other services were affected. Sorry for inconvenience.
Error trend charts failing to load. No other services are affected.
Report: "Error processing delay"
Last updateThis incident has been resolved.
Error processing has returned to normal and errors should once again be arriving in your dashboard. We continue to monitor the error processing backlog, please email any questions/concerns to support@airbrake.io Thank you for your patience!
We have identified the cause of a delay in processing incoming errors. Errors pending processing will arrive once the backlog has been digested. We apologize for any inconvenience this has caused. If you have any further questions please email support@airbrake.io
Report: "Error dashboard and API unavailable"
Last updateThe issue with our API has been resolved. The error dashboard is available again. Please email any questions or concerns to us at support@airbrake.io.
We're currently investigating an outage with our API resulting in the error dashboard being unavailable. We apologize for the inconvenience. Our team is actively working on this and will post an update once we learn more. Please email any questions or concerns to us at support@airbrake.io.
Report: "Maintenance: Error charts unavailable"
Last updateCharts are back online. During the maintenance window (0850 to 1225 UTC) error stats were not collected due to complications, charts will be flat for this brief period. There was no impact on error reporting, notifications or any other services. We are sorry for any inconvenience.
Error charts will be unavailable for a short time while maintenance is performed. All other services are unaffected.
Report: "Error processing delay"
Last updateThere was a delay in processing errors that began at 1620 UTC. We have resolved the problem and the delay is currently dropping. No errors have been lost and any delayed errors should be available in the UI soon. If you have concerns, please email support@airbrake.io, thank you for your patience.
Report: "Error processing backlog"
Last updateError processing is back to normal, the backlog is clear and no errors were lost.
We expect to have processed the backlog of errors soon. Twiddling thumbs, watching graphs :)
Delays processing new errors, they are being safely queued and will be processed asap.
Report: "Web-app outage"
Last updateAll back to normal, no errors were lost. Sorry for the inconvenience.
We are currently investigating this issue.
Report: "Delay sending notifications"
Last updateThis incident has been resolved.
We've identified a delay sending notifications and are working on a fix. If you have any issues or questions please email support@airbrake.io
Report: "Settings dashboard unavailable"
Last updateWell that _was_ quick! The settings dashboard is back. Sorry for any inconvenience.
The settings dashboard is unavailable, we know the problem and expect a quick fix
Report: "Delay in error processing"
Last updateA small hardware problem caused some delays. No errors were lost
We're investigating an issue with delays in error processing. If you have any issues or questions please email support@airbrake.io.
Report: "Error dashboard delayed performance"
Last updateWe've resolved the issue with delayed performance when accessing the error dashboard. If you have any questions or concerns about this incident, please contact us at support@airbrake.io.
We're currently investigating an issue with delayed performance when accessing the error dashboard. If you have any questions or concerns, please contact us at support@airbrake.io.
Report: "Degraded performance for notification processing"
Last updateThe issue with degraded performance for notification processing has been resolved. If you have questions, please email support@airbrake.io.
We're investigating an issue with degraded performance for notification processing. If you have any issues or questions please email support@airbrake.io.
Report: "Intermittent timeouts with error dashboard"
Last updateThe issue has been resolved and the dashboard is fully operational.
We're investigating an issue with intermittent timeouts when loading the error dashboard. If you have any issues or questions please email support@airbrake.io.
Report: "Delay processing notifications"
Last updateThank you for your patience, notifications processing have returned to normal. No errors were lost, if you have any questions or concerns please email support@airbrake.io
We're investigating an issue with notifications. There may be some delay in delivering notifications. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email support@airbrake.io