Historical record of incidents for Autobooks
Report: "Customers experiencing issues loading the bill pay application."
Last updateSome users may encounter an error when loading the Bill Pay application. Our team is investigating and will provide additional details as they become available.
Report: "Some users may encounter an error when trying to enroll in Autobooks."
Last updateThis incident has been resolved.
A fix has been implemented and the team is monitoring
Our team is actively investigating and will provide an update as soon as possible.
Report: "Intermittent outage"
Last updateWe have resolved the issue and are continuing to monitor.
We are experiencing an intermittent outage of our application.
Report: "Service Disruption to Autobooks Platform"
Last updateThis incident has been resolved.
At this time, we are no longer observing the issue that was affecting customers' ability to use Autobooks. The Autobooks team is continuing to monitor the issue.
We're currently investigating an issue impacting our customer’s ability to use Autobooks. Engineers are engaged and working to resolve the issue as quickly as possible.
Report: "Some customers may be experiencing connection issues with the Autobooks Platform"
Last updateThe incident has been resolved by our network infrastructure provider.
As of 4:06 PM EDT, our network infrastructure provider has implemented an additional mitigation for this issue and are in the process of validating full recovery. The Autobooks Platform is fully available at this time.
At 10:10 AM EDT, our network infrastructure provider rolled out a mitigation for the issue. We are continuing to monitor our network infrastructure providers' updates on the situation.
We’re experiencing an issue due to a problem identified by one of our network infrastructure providers, which is affecting some of our customers. We're working with them to resolve the issue as quickly as possible.
Report: "Degraded performance of Autobooks Platform"
Last updateThis incident has been resolved.
A fix has been implemented and the team is monitoring
Users may experience some slow operations while using the Autobooks web application, for example while creating invoices. We have identified the issue and are working on a resolution.
Report: "Degraded Web App Performance"
Last updateThe incident has been resolved.
The remediation has successfully been deployed. We are continuing to monitor the issue.
Our engineers have identified the cause and are currently deploying a remediation.
We have identified an issue causing intermittent degraded performance on our web application. Our engineers are engaged and currently investigating.
Report: "Autobooks platform is unvailable"
Last updateThis incident has been resolved.
The Autobooks platform is available again and our team is monitoring for further issues
We are currently investigating the issue
Report: "Web application outage"
Last updateThe incident has been resolved and the Autobooks web application is fully available
A fix has been implemented and we are monitoring the situation.
We have identified the issues impacting the Autobooks web application and are currently working on a resolution.
Report: "Service Disruption to Autobooks Platform"
Last updateThis incident has been resolved.
A fix has been implemented and payments are now processing successfully
One of our payment processing partners has identified an issue that is impacting online credit card payments and is working on a fix.
Report: "We are investigating an issue causing the Hub to be unavailable to some users."
Last updateThe issue impacting Hub availability has been addressed.
We are currently investigating this issue. This issue is currently not impacting the core Autobooks application.
Report: "Availability"
Last updateWe have observed no further issues and the Autobooks Platform is functioning normally at this time.
We are currently monitoring the platform for further issues, and are no longer seeing issues at this time.
We are investigating an issue where some functionality of the Autobooks platform is currently unavailable.
Report: "Incident affecting Platform"
Last updateThis incident has been resolved.
Starting at 2:50 PM EST through 3:03 PM, an incident with an external vendor interrupted connectivity to the Autobooks platform. As of now we are no longer seeing errors, and users are able to use Autobooks as usual.
Report: "Intermittent Onboarding Issues"
Last updateThe issues have been resolved. We apologize for any inconvenience this caused and appreciate your patience.
Our partner has implemented a fix and we are no longer seeing any impact to our users. We will continue to update our status page with the latest information.
We are currently experiencing intermittent failures with one of our third-party integrations. Our team is aware of the issue and actively working with the provider to resolve it as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience as we work to restore full functionality. We will continue to update our status page with the latest information.
Report: "Degraded Performance on Web Application"
Last updateThe incident has been resolved.
The core functionality of the Autobooks platform is operating normally. We are continuing to monitor the incident. Some users may see issues with images loading.
A fix has been implemented and we are monitoring traffic
We are currently investigating an issue with the Webapp being unavailable for users.
Report: "ACH Payment Processing"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
The issue has been resolved.
We are continuing to investigate this issue.
We have identified an issue with our ACH provider which is resulting in payment errors in the Autobooks application. As a result, users may receive an error when attempting to submit an ACH payment at this time.
Report: "ACH Payment Processing"
Last updateWe have not seen any errors with payment processing since our ACH provider indicated the issue is resolved.
Our ACH provider has implemented a fix for their systems and we're now seeing payments process successfully. We'll continue to monitor this situation to ensure everything is back up and running correctly. We apologize for any inconvenience this has caused you or your customers.
We are continuing to work with our provider to restore ACH payment processing capabilities. Thank you for your patience.
We have identified an issue with our ACH provider which is resulting in payment errors in the Autobooks application. As a result, users may receive an error when attempting to submit an ACH payment at this time. We are actively working with our ACH provider until this incident is resolved.
Report: "Credit Card Payment Processing"
Last updateWe have completed monitoring and confirmed no further errors have been logged.
We have identified the root cause of the intermittent payment failures and applied a fix. We will continue to actively monitor our logs to ensure no further payment errors occur.
The Autobooks Team continues to investigate the root cause of the intermittent payment failures.
We are continuing to investigate this issue.
We are continuing to investigate this issue and will provide an update within the next hour.
We have identified an issue with our credit card provider which is resulting in intermittent payment errors in the Autobooks application. As a result, users may receive an error when attempting to submit a credit card payment at this time. We are actively working with our credit card provider until this incident is resolved.
Report: "Platform - Partial Outage"
Last updateOur cloud service provider has resolved the incident. We've completed monitoring and confirmed all system components are operational.
We are seeing improved performance in the Autobooks application but will continue to monitor until the incident is fully resolved with our cloud service provider.
We are continuing to monitor this incident until a full resolution has been completed with our cloud service provider.
Customers may be experiencing some connection issues to the Autobooks platform.
Report: "Service Disruption to Autobooks Platform"
Last updateThis incident has been resolved.
We have stopped seeing connection issues to Autobooks and are continuing to monitor the situation
Customers of Autobooks may be experiencing some connection issues to the Autobooks platform.
Report: "Platform - Outage"
Last updateThis incident has now been resolved. Our teams have completed monitoring and confirmed all system components are operational.
A third party vendor of Autobooks has resolved the cause of the outage. The Autobooks Team will continue to monitor the platform to ensure the incident is fully resolved.
We are currently investigating an issue which affects the Autobook's platform. We are working to resolve the issue as quickly as possible, and we apologize for any disruption this may cause.
Report: "Degraded System Performance"
Last updateNo further errors have been recorded to our logs since 3:00 PM EST. We'll be working with our partner to understand the root cause of this degraded performance. We apologize for any inconvenience.
We are currently investigating degraded system performance with the Autobooks application. The Incident Response Team is currently investigating and will have an update within the next hour.
Report: "Intermittent Impact / Degraded Performance"
Last updateWe were notified of an expired certificate for our CDN and were able to reinstate the current certificate within approximately 30 minutes. Due to the time of the incident and browser caching, we believe the impact to users would have been minimal. We apologize for any inconvenience.
Report: "Platform - Outage"
Last updateThis incident has now been resolved. Our teams have completed monitoring and confirmed all system components are operational.
We have applied fixes to bring the system back up. We'll be monitoring the system to make sure everything is functioning as expected.
We're continuing to work with our 3rd party vendor to resolve the issue. We apologize for any disruption this may cause.
We're continuing to work with our 3rd party vendor to resolve the issue. We apologize for any disruption this may cause.
We're continuing to work with our 3rd party vendor to resolve the issue. We apologize for any disruption this may cause.
We're continuing to work with our 3rd party vendor to resolve the issue. We apologize for any disruption this may cause.
We're continuing to work with our 3rd party vendor to resolve the issue. We apologize for any disruption this may cause.
We have currently identified an issue with a 3rd party vendor. We are in contact with the vendor and working to resolve the issue as quickly as possible.
We are currently investigating an issue which affects the Autobook's platform. We are working to resolve the issue as quickly as possible, and we apologize for any disruption this may cause.
Report: "Intermittent Impact / Degraded Performance"
Last updateAfter further monitoring, we have confirmed transaction processing delays no longer persists.
A fix has been implemented and we are currently monitoring transactional processing. All delayed transactions have been fully processed at this time.
The issue with delayed processing of transactions has been identified. We are actively working on a resolution and have confirmed the transactions are beginning to complete processing.
We are currently investigating system issues resulting in delayed transaction processing.
Report: "Credit Card Payment Processing"
Last updateAfter additional monitoring, we've confirmed this incident with our partner has been fully resolved.
A fix has been implemented by our partner. We will continue to actively monitor our logs to ensure no further errors occur.
The root cause has been identified by our partner. We are actively working with this partner until payment processing errors have been fully resolved.
We are continuing to investigate this issue with our provider and will provide further updates as they become available.
We are continuing to investigate this issue with our provider and will provide further updates as they become available.
We are continuing to investigate this issue with our provider and will provide an update within the next hour.
We are continuing to investigate this issue and will provide an update within the next hour.
We have identified an issue with our credit card provider which is resulting in intermittent payment errors in the Autobooks application. As a result, users may receive an error when attempting to submit an credit card payment at this time. We are actively working with our credit card provider until this incident is resolved.
Report: "Credit Card Payment Processing"
Last updateThis incident has now been resolved by our credit card provider. Our teams have completed monitoring and have confirmed the last credit card payment error occurred at 10:33 AM EST.
A fix has been implemented by our credit card provider. We will continue to actively monitor our logs to ensure no further errors occur.
We have identified an issue with our credit card provider which is resulting in intermittent payment errors in the Autobooks application. As a result, users may receive an error when attempting to submit an credit card payment at this time. We are actively working with our credit card provider until this incident is resolved.
Report: "Intermittent Impact / Degraded Performance"
Last updateThis incident has now been resolved. Our teams have finished monitoring and confirmed no further latency issues.
The performance of the Autobooks application has remained steady since 3:00 PM EST. We are continuing to actively monitor our application's latency and work with Microsoft to ensure the incident is fully resolved.
We are continuing to work with Microsoft and are beginning to see improvements in our application's latency. Our next update to this incident will be posted by 4:00 PM EST or sooner.
We are continuing to work with Microsoft Support on a resolution to this incident. Our next update to this incident will be posted by 3:00 PM EST or sooner.
Microsoft Azure is currently experiencing degraded performance resulting in latency within our application. We are in contact with Microsoft Support for a resolution.
We are currently investigating intermittent issues impacting page latency and failures.
Report: "Service Disruption to Autobooks Platform"
Last updateThere have been no further connectivity errors. The API and Web App was degraded from 10:30 PM to 11:20 PM EST.
Services are returning to normal and we are continuing to monitor the situation.
We are continuing to work on a fix for this issue.
We are currently affected by an issue with our global cloud provider's network, leading to connectivity issues with our web application and the API.
Report: "Platform - Outage"
Last update**Summary:** On March 28th, 2021 from 2:38ET to 13:30ET Autobooks experienced an outage with our primary cloud service provider that resulted in error pages and/or request timeouts for all customers interacting with the Autobooks platform. **Details:** Autobooks utilizes many premier cloud partners as part of providing our SaaS platform, including Microsoft Azure. Microsoft Azure is used to securely host the Autobooks platform in redundant datacenters across separate geographic regions to ensure the platform is available in the event of any sort of internet outage affecting a significant portion of the internet. At 2:05ET on March 28th our team began to perform scheduled maintenance as part of routinely testing the BC/DR \(business continuity / disaster recovery\) capabilities of the Autobooks cloud platform. Many of these capabilities are exercised as part of our standard bi-weekly software deployment cycle, but we routinely like to exercise the entirety of these capabilities in a controlled environment as part of standard operating procedure. At 2:08ET, our team verified that the cross-region database failover that was being tested as part of this exercise finished as planned. The team then spent the next thirty minutes evaluating logs and monitoring services to ensure everything worked as expected. Once verified successfully, our team initiated a failover to return the platform back to its pre-test configuration. At 2:38ET, our team determined that the cross-region database failover back to its original configuration was stuck in an undesirable state, leaving both the primary and secondary instances read-only. This read-only scenario is initiated to ensure the integrity of the underlying data, but ultimately leaves the Autobooks platform in a similarly read-only or otherwise unusable state. At this point our team engaged Microsoft support directly for assistance in remediating the issue. From 2:38ET to 13:30ET, our team worked closely with the Microsoft support team to identify the root cause of this unexpected failure state, which was ultimately related to underlying security certificates that Microsoft uses as part of geographic failover to ensure the privacy and integrity of the underlying data. These certificates were in an inconsistent state due to unrelated maintenance that Microsoft was doing as part of their own standard security certificate rotation procedures. Unfortunately, this root cause was specific to the underlying Microsoft Azure infrastructure in a way that was invisible to the Autobooks team and was not obvious to the Microsoft support team at that time which led to the longer than acceptable issue duration. Since that time, we’ve worked with Microsoft to better understand what caused the issue, and what we can jointly do to ensure it doesn’t happen again. Microsoft is working to improve this certificate rotation process to prevent similar issues going forward. The failover to the secondary region and subsequent failover back in short order is what triggered the issue on the Microsoft side, so in the interim we’ll avoid that scenario as part of our testing. We apologize for any inconvenience this may have caused. Your business is very important to us, and we strive to provide you with exceptional service.
This incident has now been resolved. Our teams have completed monitoring and we have confirmation from our Cloud Provider, Microsoft Azure, that all systems have been fully restored.
We are continuing to monitor for any further issues.
The incident has been resolved with our cloud provider, Microsoft. We will continue to actively monitor the incident.
During routine failover testing, we identified an issue in which we have engaged our cloud provider, Microsoft. We have an open severity A request with our provider in which we are actively working to resolve.
We are continuing to investigate this issue and will provide further updates as they become available.
We are continuing to investigate this issue and will provide updates as they become available.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating an issue which affects the Autobook's platform. We are working to resolve the issue as quickly as possible, and we apologize for any disruption this may cause.
Report: "ACH Payment Processing"
Last update**Summary:** On April 6th, 2021 from 12:16 ET to 14:05ET the Autobooks platform experienced an outage with a third-party ACH provider that prevented Administrators and SMB customers from submitting ACH payments during this window. **Details:** Autobooks utilizes many premier cloud partners as part of providing our SaaS platform, including partners that provide capabilities like ACH \(Automated Clearing House\) that we then provide to our customers. At 12:16 ET on April 6th our automated monitoring began to alert the team to potential issues affecting ACH payments via the Autobooks platform. Our team quickly determined that the issue was related to failing calls to a third-party ACH provider. At 12:26 ET, our team was able to confirm this issue with our ACH provider directly and worked to monitor the situation while this third-party provider worked to resolve the issue. At 14:00 ET, our team determined that ACH payments were again starting to succeed in the Autobooks platform. Continued testing reported no further issues our end and we ultimately gave the all-clear at 14:05 ET. Our team has used data collected as part of this incident to identify a number of improvements that will minimize the impact of similar issues going forward and will work to prioritize these improvements as part of our standard development cycle. These improvements include technology to queue up requests in the event of a similar situation in the future, as well as tools to put ACH functionality in “maintenance mode” to gracefully signal to SMB customers that this feature is currently unavailable in the event of an extended outage or maintenance scenario. We apologize for any inconvenience this may have caused. Your business is very important to us, and we strive to provide you with exceptional service.
This incident has now been resolved. Our teams have completed monitoring and have confirmed the last ACH payment error occurred at 2:05 PM EST.
A fix has been implemented by our ACH provider. We will continue to actively monitor our logs to ensure no further errors occur.
As our ACH provider continues to make progress on resolving this incident, we are beginning to see some successful ACH payments post within the Autobooks application. Our next update to this incident will be posted by 3:00 PM EST or sooner.
We have identified an issue with our ACH provider which is resulting in payment errors in the Autobooks application. As a result, users may receive an error when attempting to submit an ACH payment at this time. We are actively working with our ACH provider until this incident is resolved.
Report: "Platform Outage"
Last update**Summary:** On April 1st, 2021 from 17:26 ET to 18:45 ET the Autobooks platform experienced an outage with our primary cloud service provider that resulted in service failures for our customers. **Details:** Autobooks utilizes many premier cloud partners as part of providing our SaaS platform, including Microsoft Azure. Microsoft Azure is used to securely host the Autobooks platform in redundant datacenters across separate geographic regions to ensure the platform is available in the event of any sort of internet outage affecting a significant portion of the internet. At 17:26 ET on April 1st our automated monitoring began to alert the team to potential issues affecting the Autobooks platform. Our team quickly determined that the issue was related to failing DNS \(Domain Name Systems\) queries that are both used by external parties to connect to our system as well as for internal components within the platform to connect to each other. At 17:35 ET, our team determined that the issue was likely an Azure-wide DNS outage as multiple public components related to Microsoft Azure were also unavailable, which included the Azure public status page that would generally be used to confirm this theory. Shortly thereafter Microsoft’s Twitter account confirmed this as a global DNS outage and that they were working to remediate the issue. From this point forward our team was engaged to monitor the situation and prepare to test our services as soon as Microsoft resolved the issue on their end. At 18:45 ET, Microsoft stated that the issue was resolved, and our team verified there were no further issues affecting the Autobooks platform. Microsoft communicated the following as the official root cause: “Azure DNS servers experienced an anomalous surge in DNS queries from across the globe targeting a set of domains hosted on Azure. Normally, Azure’s layers of caches and traffic shaping would mitigate this surge. In this incident, one specific sequence of events exposed a code defect in our DNS service that reduced the efficiency of our DNS Edge caches. As our DNS service became overloaded, DNS clients began frequent retries of their requests which added workload to the DNS service. Since client retries are considered legitimate DNS traffic, this traffic was not dropped by our volumetric spike mitigation systems. This increase in traffic led to decreased availability of our DNS service.” Since that time, Microsoft has worked to repair the code defect that prevented these requests from being cached and improved the automatic detection and mitigation technology that is used to minimize the impact of these sort of issues. We apologize for any inconvenience this may have caused. Your business is very important to us, and we strive to provide you with exceptional service.
This incident has now been resolved. Our teams have completed monitoring and have confirmed that all systems are fully restored.
We have confirmed that Microsoft has resolved the issue and all Autobooks systems are 100% operational at this time. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence.
We have identified an issue with our global cloud provider Microsoft Azure and are working with Microsoft support to resolve the issue.
We are currently investigating an issue which affects the Autobook's platform. We are working to resolve the issue as quickly as possible, and we apologize for any disruption this may cause.
Report: "Transaction Processing"
Last updateAfter further monitoring, we have confirmed the intermittent transaction processing issue no longer persists.
There have been no transaction processing errors since yesterday, 12/9 at 9:07 PM EST. We are continuing to investigate the root cause of this issue.
We are continuing to monitor and investigate this issue. We will continue to provide updates as they become available.
No additional errors have occurred since last update. We are continuing to monitor this issue.
No additional transaction processing errors have occurred since 9:07 PM EST. We are continuing to investigate and monitor this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating intermittent issues with transaction processing.
Report: "Autobooks Login"
Last updateThis incident has been resolved.
A fix has been implemented and we are actively monitoring.
The issue has been identified and a fix is being implemented.
We are currently investigating issues impacting login into the Autobooks platform for some users.
Report: "Platform - Outage"
Last updateOn the evening of July 17th, some Autobooks customers received a degraded experience from 5:14 PM to 5:51 PM EST. This incident was triggered from a misconfiguration by an upstream provider. As a result, some Autobooks customers experienced intermittent access issues with the Autobooks site for approximately 37 minutes. The outage was immediately detected by our internal monitoring which engaged our Incident Response Team. The team quickly identified the root cause with our upstream provider and engaged with that provider. The team continued to monitor and work with our provider through the deployment of the resolution. We apologize for any inconvenience this may have caused. Your business is very important to us, and we strive to provide you with exceptional service.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the issue with our Third-Party Provider and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating an issue which affects the Autobook's platform. We are working to resolve the issue as quickly as possible, and we apologize for any disruption this may cause.
Report: "Service Disruption to Autobooks Support - Customer Care"
Last updateAll issues have been remediated, monitoring has been completed, and all systems have been restored to full health. All support channels are now fully operational.
We have identified the issue and deployed a solution to remediate the issue. We are beginning to see services recover. We will continue to monitor the issue until resolved.
We have currently identified an issue with a 3rd party provider within our Customer Care team. Users may experience longer than normal wait times when contacting our Customer Care team. Currently impacted by this incident: - Phone Support - Email Support - Chat Support - Knowledge Base
We're currently investigating an issue impacting our Customer Support channels.
Report: "Service Disruption to Autobooks Support - Customer Care"
Last updateAll issues have been remediated, monitoring has been completed, and all systems have been restored to full health. All support channels are now fully operational.
Our 3rd party provider has identified the issue and deployed a solution to remediate the issue. We are beginning to see services recover. We will continue to monitor the issue until resolved.
We have currently identified the issue is with a 3rd party provider within our Customer Care team. Users may experience longer than normal wait times when contacting our Customer Care team. Currently impacted by this incident: - Phone support - Email support - Chat support - Knowledge Base
We're investigating an issue impacting our customer support.
Report: "Network Performance Issues"
Last updateThis incident has been resolved.
We are experiencing networking performance issues with one of our service provider partners that is impacting the Autobooks platform and public facing internet sites. We are actively investigating the issue