Historical record of incidents for Codat
Report: "Datasets getting stuck in ValidationQueued"
Last updateWe are actively investigating an issue causing datasets to be stuck in the ValidationQueued state. This incident will be updated as soon as we have identified the issue, and no later than 21:45 BST.
Report: "QBO Push operations completing slowly"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to push new data via the Codat API for their QBO companies without disruption.If you are still affected by this issue, please contact support@codat.io
Affected users should now notice that push requests to QBO process in a timely manner. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing some users from successfully pushing data to QBO . Affected users will notice that pushes are sitting in a pending state for a long time before completing successfully.
Report: "QBO Push operations completing slowly"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to push new data via the Codat API for their QBO companies without disruption. If you are still affected by this issue, please contact support@codat.io
Affected users should now notice that push requests to QBO process in a timely manner. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing some users from successfully pushing data to QBO . Affected users will notice that pushes are sitting in a pending state for a long time before completing successfully.
Report: "API Intermittent Timeouts"
Last updateOur monitoring flagged an issue with our API intermittently timing out between 02:16 UTC and 02:28 UTC.Affected users may have noticed that their API calls were yielding 504 error codes. The cause of the issue has been identified as an issue with our cloud infrastructure provider. Service has now been restored and we are working with the third party to ensure there are no further regressions. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "API Intermittent Timeouts"
Last updateOur monitoring flagged an issue with our API intermittently timing out between 02:16 UTC and 02:28 UTC. Affected users may have noticed that their API calls were yielding 504 error codes. The cause of the issue has been identified as an issue with our cloud infrastructure provider. Service has now been restored and we are working with the third party to ensure there are no further regressions. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Intermittent API timeouts"
Last updateThe incident has now been resolved as ongoing monitoring has shown the platform has returned to normal stability. API responses should have now returned to being performant. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix here. Affected users should now notice that the response times of our API have returned to normal. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue which is causing intermittent timeouts for our API Affected users may notice that certain API calls are timing out or taking longer than usual to complete. This incident will be updated as soon as we have identified the issue, and no later than 17:45
Report: "Codat Link Flow: Intermittent Errors"
Last updateThis incident has now been resolved. Link Flow has been fully restored. If you are still affected by this issue, please contact support@codat.io for further assistance.
A fix was implemented at 5:30PM GMT and we are monitoring the results. If you are still affected by this issue, please contact support@codat.io for further assistance.
The cause of the issue has been identified and a fix is being implemented.
Our monitoring flagged that intermittent errors started occurring in the link flow on 1:20PM GMT on 12th May. Affected users may have noticed a "Something went wrong" error message occurred when attempting to link a company, preventing completion of the link flow.
Report: "Sage 50 Connected Companies unable to fetch data"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to fetch data from Sage 50 without error. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix to the problems we had been experiencing with our Sage50 integration. Affected users should now notice that there fetch operations for Sage50 are able to complete without error. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are continuing to implement a fix.
We have identified the cause of the incident and are continuing to implement a fix.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix, and no later than 13:30
We are actively investigating an issue preventing Sage 50 connected companies from fetching data. Sage50 connected companies should notice that all datasets fail with a FetchError This incident will be updated as soon as we have identified the issue, and no later than 12:57
Report: "Production API timeouts"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to make requests to the API and use the portal without disruption. If you are still affected by this issue, please contact support@codat.io
We are no longer seeing the timeouts in production and affected users should not experience further requests timing out or the portal failing to load. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue resulting in timeouts in the portal and making API requests. Affected users will notice that some requests are timing out resulting in intermittent page loading and failed requests. This incident will be updated as soon as we have identified the issue, and no later than 15.30 BST.
Report: "Codat Portal logins intermittently failing"
Last updateOur monitoring alerted us to the fact that between 11:48 and 12:01 GMT, some logins to the Codat portal may have been failing with an error. The cause of the issue has been identified as an problem with our cloud infrastructure provider, which has now been resolved. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "NetSuite bills MapError"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to sync bills from NetSuite without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that NetSuite bills datasets now complete successfully. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix, and no later than 10AM BST.
We are actively investigating an issue preventing users from syncing bills from NetSuite. Affected users will notice that any bill syncs from NetSuite will fail with a MapError. This incident will be updated as soon as we have identified the issue, and no later than 10AM BST.
Report: "Push operations for Netsuite Bills and BillPayments failing in Production"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to push bills and billPayments to NetSuite without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that they're able to push bills and billPayments to NetSuite successfully . Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have started seeing 500 responses again on bill and billPayments pushes to NetSuite. We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix, and no later than 10.30AM BST.
We have implemented a fix in Production. Affected users should now notice that Bill and BillPayment push operations no longer return a 500 response. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in 10:10PM GMT
We are actively investigating an issue preventing users from pushing Bills and BillPayments to Netsuite. Affected users will notice that a 500 error response is returned. This incident will be updated as soon as we have identified the issue, and no later than 10:10 GMT
Report: "Categorized Bank Statement, Credit Review and Spend Analysis reports unavailable in Production"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to generate reports without disruption. If you are still affected by this issue, please contact support@codat.io
We have identified the cause of the incident and have implemented a fix in Production. Affected users should notice that they are now able to download reports via the Codat Portal and API as usual. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from downloading reports in Production for some of our products. Affected users will notice that they are not able to download new reports for their companies using our Categorized Bank Statement, Credit Review or Spend Analysis products. This incident will be updated as soon as we have identified the issue, and no later than 5pm UK time.
Report: "Suppliers for some Netsuite connections may be deleted"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that Netsuite Suppliers previously marked as deleted have been restored. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in production, and no later than 9:10pm GMT.
We are continuing to investigate this issue.
We are actively investigating an issue resulting in removal of previously synced Suppliers for some Netsuite connections. Affected users will notice that a large number of Suppliers are marked as deleted. This incident will be updated as soon as we have identified the issue, and no later than 9:10pm GMT.
Report: "Some QuickBooks Online datasets syncs may be failing in Production"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to queue and complete syncs for QBO without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in Production. Affected users should now notice that QBO syncs should complete successfully. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from syncing QuickBooks Online (QBO) companies successfully in Production. Affected users will notice that QBO companies may be failing to sync new datasets. This may be affecting the syncing of any QBO companies linked. This should not prevent users from connecting new companies. This incident will be updated as soon as we have identified the issue, and no later than 10:30am GMT.
Report: "Netsuite Link Flow errors"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to link their Oracle Netsuite entities without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in Production. Affected users should now notice that they are able to link their NetSuite connections successfully. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified an issue preventing some users from linking subsidiaries to Netsuite. This issue started at approximately 16:30GMT. Affected users would have noticed that they either experienced an error attempting to link or were unable to select a Netsuite subsidiary during the linking process.
Report: "Webhook Consumer endpoints not currently accessible."
Last updateA fix has been deployed and verified. The webhook consumer API endpoints are now accessible.
The root cause of the issue has been identified and we are deploying a fix.
We are actively investigating an issue preventing users from making calls to the Webhook Consumer endpoints. Affected users will notice that they cannot make API calls to manage Webhook consumers. This incident will be updated as soon as we have identified the issue, and no later than 15:01
Report: "Webhooks being sent with a delay"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. Webhooks which would have been triggered in approximately the last hour will have recently been resent. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production which has resolved the issue. Any web hook which have yet to reach you will be resent shortly. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue where we are observing webhook delivery taking longer than usual. Affected clients may notice that they have yet to receive webhook alerts they were expecting. This incident will be updated as soon as we have identified the issue, and no later than 17:20
Report: "Password resets not working for a very small number of users"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should have received a password reset email. If you are still affected by this issue, please contact support@codat.io
We have identified the cause of the incident and are currently implementing a fix to remediate the issue.
We are actively investigating an issue preventing a small subset of our users from resetting their password to the Codat portal. Affected users will notice that once they click the 'Reset Password' button, they do not receive an email. This incident will be updated as soon as we have identified the issue, and no later than 11:16
Report: "Linking to FreeAgent Platform currently throwing errors"
Last updateWe have deployed and verified a fix for this problem. Freeagent connections can now be completed
The issue has been identified and we are deploying a fix.
We are continuing to investigate this issue.
We are actively investigating an issue preventing users from connecting to FreeAgent Affected users will notice that they are unable to create new connections to FreeAgent. This incident will be updated as soon as we have identified the issue, and no later than 14:28
Report: "QBO Push operations completing slowly"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to create new data via the Codat API for their QBO companies without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that push requests to QBO process in a timely manner. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
The issue has been identified and a fix is being implemented. You may have noticed more QBO push operations completing successfully, but we are still implementing a fix to work through the backlog more quickly.
We are actively investigating an issue preventing some users from successfully pushing data to QBO . Affected users will notice that pushes are sitting in a pending state for a long time.
Report: "Microsoft Dynamics 365 Business Central extension maintenance"
Last updateWe have completed the upgrade of our Microsoft Dynamics 365 Business Central extension to support the upcoming changes announced by Microsoft. All Microsoft Dynamics 365 Business Central should be able to link, sync and install our extension without further disruption.
We have been made aware of an upcoming change to the packages provided by Microsoft Dynamics 365 Business Central which may prevent some clients upgrading to the latest release of v25 to link or sync new data. We are actively working on updating our extension to minimise the impact on end-users attempting to link or sync new data using our Microsoft Dynamics 365 Business Central integration in the coming days. This information notice will be updated as soon as we have successfully upgraded our Microsoft Dynamics 365 Business Central extension to support the upcoming changes announced by Microsoft.
Report: "NetSuite link flow errors"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to link to NetSuite without disruption. If you are still affected by this issue, please contact support@codat.io
We are continuing to monitor for any further issues.
We have identified an issue preventing some new users from linking to Netsuite. Affected users would have noticed that they either experienced an error attempting to link or the linking process indicated that the Netsuite bundle had not been installed, though it had. We have implemented a fix in Production. Affected users should now notice that they are able to link their NetSuite connections successfully. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
Report: "Linking to Wave throws an Error"
Last updateThe incident has now been resolved. We have restored the functionality which allows for companies to Link to Wave.
The root cause of the incident has been identified and we are in the progress of deploying a fix.
We are actively investigating an issue preventing users from linking their Wave Accounts to the Codat system Affected users will notice that when they attempt to progress through the link flow, after selecting Wave, a message is displayed 'Unknown Error Occurred' This incident will be updated as soon as we have identified the issue, and no later than 17:04
Report: "Delay in account categorisation"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should see account categorisation processing in a timely fashion. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that account categorisation is processing in a timely manner. There may be some delay as our services work through the backlog of requests. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified an issue in production causing a delay in the processing of our account categorisation required for the enhanced reports. We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in production, and no later than 12.40GMT.
Report: "Some requests resulting in 524 response"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to make requests to our API without disruption. If you are still affected by this issue, please contact support@codat.io
We are continuing to monitor for any further issues.
We have implemented a fix in production. Affected users should now notice that they're able to make requests as expected. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue causing 524 responses from our API. Affected users will notice that some requests are failing with a 524 response. This incident will be updated as soon as we have identified the issue, and no later than 15.45 BST.
Report: "Banking transactions enhancements on lending API temporarily unavailable"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to retrieve banking transaction enhancements without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that requested categorizations do complete. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from retrieving banking transaction enhancements through lending API Affected users will notice that requested categorizations do not complete. This incident will be updated as soon as we have identified the issue, and no later than 20:20
Report: "NetSuite Integration intermittent failures fetching and pushing data"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to sync billCreditNotes, invoices, creditNotes and directIncomes datatypes from NetSuite and push bills to Netsuite without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. The most severely impacted users should now notice that billCreditNotes, invoices, creditNotes and directIncomes datasets complete successfully for NetSuite. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in production.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in production.
We believe that we have identified the root cause of this issue and working on deploying a fix asap.
We are continuing to investigate this issue.
We are actively investigating an issue whereby we are observing a higher than normal proportion of data fetches from NetSuite failing. This is also affecting some push operations. Affected users will notice that some billCreditNotes, invoices, creditNotes and directIncomes datasets for NetSuite connected companies may fail with a Fetch Error as well as Bill push operations. This incident will be updated as soon as we have identified the issue, and no later than 16:40
Report: "QBO - Data fetch/push failing for some Bill Payments and Payments"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to push and pull billPayments and payments datatypes for QuickBooks Online without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in Production. Affected users should now notice that QBO bill payments & payments datasets are syncing successfully. Datasets affected will be retried in line with your instance's sync settings. Push operations affected may have been returned with a 'Failed' status but are likely to have completed successfully, Affected clients will be contacted with more information in the next 24h. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are actively investigating an issue preventing some users from successfully fetching or pushing Payments and Bill Payments to QBO . Affected users will notice that Bill Payments or Payment pushes fail with a 500 error or data fetches for either dataset fail with a MapError. This incident will be updated as soon as we have identified the issue, and no later than 10:21GMT.
Report: "Portal Aged debtors report unavailable error."
Last updateA fix has been deployed and we have verified that the Aged Debtors Report generation is now working in the Codat Portal. If you continue to have any issues or concerns, please feel free to raise with support@codat.io
We are continuing to investigate this issue.
We are actively investigating an issue preventing users from generating the Aged Debtors Report in the Codat portal. Affected users will notice that when they hit the 'Load Aged Debtors' button, a 'Aged debt report unavailable' error will be shown on screen. This incident will be updated as soon as we have identified the issue, and no later than 10:03
Report: "Unable to access Codat API and Portal"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to access the Codat API and Portal without disruption. If you are still affected by this issue, please contact support@codat.io
We are continuing to monitor for any further issues.
We have identified an issue preventing users from accessing the Codat API and Codat Portal. We have implemented a fix in production. Affected users should now notice that they're able to access the API and portal without issue.. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
Report: "Unable to access commerce data"
Last updateOur monitoring flagged that commerce data was not accessible between 16:28 BST and 16:54. Affected users would have noticed that they were not able to access any commerce data. The cause of the issue has been identified as an incorrect connection string being used by some services. Service has now been restored. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Unable to access production services"
Last updateOur monitoring flagged that that between 12.28 and 12.40. Affected users would have noticed that they were unable to access Codat Services. The cause of the issue has been identified as an update causing unexpected downtime. Service has now been restored. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Amazon Seller Central: Datasets failing with permissions errors"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to fetch data from Amazon without unexpected Permissions errors. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix to the Amazon Seller Central issue. Affected users should now be able to fetch data from the Amazon integration without unexpected permission errors Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently working on delivering a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix, and no later than 12:37
We are continuing to investigate this issue.
We are actively investigating an issue preventing users from fetching data from our Amazon Seller Central integration. Affected users will notice that some datasets are currently failing with Permissions Errors. Our engineers are actively investigating the root cause. This incident will be updated as soon as we have identified the issue, and no later than 12:08
Report: "High number of Xero deauthorizations."
Last updateWe have been made aware that at about 11:10 PM GMT a higher than normal number of Xero connected companies deauthorized. The cause of the issue has been identified as an incident on the Xero side. You can read more about this on their status page here: https://status.developer.xero.com/incidents/66sf5nwnr14f . Xero's Services have now been restored and we have relinked any incorrectly disconnected companies. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Delay in processing some datasets"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to successfully sync datasets without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production. Affected users should now notice that datasets are processing again. There may still a delay in processing while the backlog of datasets is processed. This should complete in the next 20 minutes. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in Production, and no later than 11.30AM BST.
We are actively investigating an issue causing a delay in the processing of some datasets. Affected users will notice that some datasets will remain in the fetching status and a message in the portal will be shown `Paused due to concurrency limitations`. This incident will be updated as soon as we have identified the issue, and no later than 11.30AM BST.
Report: "Some Xero connections deauthorized in Production"
Last updateWe noticed that some client were unable to link or sync new data for their Xero companies between midnight UTC and 00:30 UTC. Affected users would have noticed that their Xero data connections had become unexpectedly deauthorized. The cause of the issue has been identified as a Xero outage (https://status.xero.com/incidents/85j714f5ctkw), now resolved. We have started to relink affected connections and the service should recover fully by 10am UK time. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "500 errors accessing Codat Portal."
Last updateThe incident has now been resolved, ongoing monitoring has shown the portal UI has returned to normal stability.
We have implemented a fix to the Codat portal. Affected users should now notice that they are able to access app.codat.io again without any errors being displayed. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from accessing the Codat Portal Affected users will notice that when they attempt to visit app.codat.io, a 500 category error will be displayed to them in their browser. This incident will be updated as soon as we have identified the issue, and no later than 13:38
Report: "Webhooks failing to send"
Last updateThis incident has been resolved. Our monitoring flagged that some webhooks were failing to send between 19:20 BST - 20:19 BST. Affected users would have noticed that some webhooks failed to be sent. These missing webhooks have now been requeued and sent successfully.
Our monitoring flagged that some webhooks were failing to send between 19:20 BST - 20:19 BST. A fix has been implemented and we are now monitoring further. Affected users would have noticed that some webhooks failed to be sent. These missing webhooks are now being requeued and sending successfully. If you are still affected by this issue, please contact support@codat.io for further assistance.
We have identified an issue which is preventing some webhooks from sending. This incident will be updated as soon as we have successfully tested and rolled out a fix, and no later than 21:00 BST.
Report: "Microsoft Social Login Throwing Errors"
Last updateThis incident has been resolved.
We have implemented a fix. Affected users should now notice that they are again able to sign into the Codat portal using Microsoft login. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from signing in using the Microsoft social login button. Affected users will notice the system return to them text detailing a 500 category error, shortly after logging in. Google login, as well as the username/ password form are unaffected. This incident will be updated as soon as we have identified the issue, and no later than 16:55
Report: "Codat API: Intermittent timeout errors retrieving commerce data"
Last updateOur monitoring flagged that some requests to access commerce data from our API were failing between 02:58 and 11:32 GMT. Affected users would have noticed that API requests to retrieve commerce data were intermittently failing with timeout errors. The cause of the issue has been identified as faulty infrastructure within our internal service which serves this data. Service has now been restored. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Degraded Portal performance"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to access the Portal without disruption. If you are still affected by this issue, please contact support@codat.io
The services previously affected have recovered. Affected users should now notice that the Portal loads successfully. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
Our Portal remains affected by an Azure outage (https://azure.status.microsoft/en-us/status/). We are investigating mitigation steps meanwhile and will update our status page at 3:30pm UK time latest.
We are actively investigating an issue preventing some users from logging into the Codat portal as result of the Azure outage (https://azure.status.microsoft/en-gb/status). Affected users will notice that logging into the Portal takes longer than expected or may get a blank page when accessing app.codat.io. This incident will be updated as soon as we have identified the issue, and no later than 2:15pm UK.
Report: "Some Xero connections deauthorized in Production"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to pull and push data for connections affected previously. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in Production. Affected users should now notice that their affected Xero companies have been relinked successfully. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of an incident that has resulted in a multiple Xero data connections becoming deauthorized. We are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in Production, and no later than 12:30pm.
Report: "Codat API/Sync for Commerce: 504 errors requesting commerce data for all users, and failing syncs for Sync for Commerce"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to retrieve commerce data and perform syncs without disruption. If you are still affected by this issue, please contact support@codat.io
We are actively investigating an issue preventing users from retrieving commerce data, and causing syncs to fail for Sync for Commerce. Affected users will notice 504 errors when requesting commerce data from the Codat API, or failing syncs. This incident will be updated as soon as we have identified the issue, and no later than 17:00 GMT.
Report: "Codat Link Flow: Intermittent 500 Errors in Production"
Last updateOur monitoring flagged that 500 errors were occurring in the link flow between 00:10AM and 12:50PM GMT on 28th July. Affected users would have noticed that 500 errors occurred when attempting to link a company, preventing completion of the link flow. The cause of the issue has been identified as an infrastructure failure in the backend service which supports the link flow. Service has now been restored. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Intermittent 524 API Responses"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to make requests to our API without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in Production. Affected users should now notice that they're able to make API requests as normal. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix in production and no later than 14.30 BST.
We are actively investigating an issue causing our API to return intermittent 524 responses. Affected users will notice that a portion of requests result in a 524 response. This incident will be updated as soon as we have identified the issue, and no later than 14.25 BST.
Report: "Crowdstrike Outage"
Last updateWe are aware of the Crowdstrike outage which has affected internet infrastructure globally. Our systems remain unaffected but some upstream integrations may have experienced difficulties within the past few days. We are monitoring the situation, but if you have any concerns, please reach out to support@codat.io
We are aware of the Crowdstrike outage which has affected internet infrastructure globally. Our systems remain unaffected but some upstream integrations may have experience difficulties within the past 24 hours. We are monitoring the situation, but if you have any concerns, please reach out to support@codat.io
Report: "Degraded sync performance for some QBD connectors"
Last updateOur monitoring flagged that QBD syncs were delayed for some users in Production between ~7am UK time and 12pm UK time on July 11th 2024. Affected users would have noticed that QBD syncs were much slower to complete; this issue has not resulted in a higher error rate visible to your end-users. Service has now been fully restored. If you are still affected by this issue, please contact support@codat.io for further assistance.
Report: "Netsuite syncs stuck in a queued state"
Last updateThe incident has now been resolved, ongoing monitoring has shown that Netsuite has returned to normal stability. All users previously affected should see successfully synced datasets for their Netsuite connections. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in production at 21:55 BST. Affected users should notice that the previously stuck syncs are now completing. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing a subset of users from pulling data from Netsuite. Affected users will notice that recent syncs are stuck in a queued state indefinitely. This was first identified on July 3rd @ 18:24 BST. This incident will be updated as soon as we have identified the issue.
Report: "Codat Portal: Login Unavailable."
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to login to the Portal without disruption. If you are still affected by this issue, please contact support@codat.io
We have successfully restored the availability of the Portal Affected users should now notice that they are able to login. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from logging into the Codat portal. We now believe that there is an underlying issue with our cloud service provider, to which we have escalated the matter. We are still actively attempting to assuage the problem on our side. This incident will be updated as soon as we have identified the issue, and no later than 14:30
We are actively investigating an issue preventing users from logging into the Codat portal. This incident will be updated as soon as we have identified the issue, and no later than 14:15
Report: "Codat API and Portal down in Production"
Last updateThe incident has now been resolved, ongoing monitoring has shown the platform has returned to normal stability. All users previously affected should be able to query the API or access the Portal without disruption. If you are still affected by this issue, please contact support@codat.io
We have implemented a fix in Production. Affected users should now notice that the Codat portal and API are up. Users affected should now be able to queue syncs, link new companies or query our API without disruption. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We are actively investigating an issue preventing users from using the Codat Portal or the API in Production. Affected users will notice that they are unable to queue syncs for existing or new companies, link new companies or query data using our API. We began noticing issues at approximately 20:37 BST, and may have experienced degraded performance until 21:05 BST. This incident will be updated as soon as we have identified the issue.
Report: "Users Unable To Login To The Codat Portal"
Last updateThis incident has now been resolved. We are confident that the changes we implemented fixed the underlying issue.
We have implemented a fix to the Codat portal Affected users should now be able to Login without issue. Our teams continue to monitor the platform to ensure continued stability. If you experience any issues please contact support@codat.io for further assistance.
We have identified the root cause of the incident and are currently implementing a fix to remediate the issue. This incident will be updated as soon as we have successfully tested and rolled out a fix to our Portal, and no later than 10:30 AM
We are actively investigating an issue preventing users from logging into the Codat portal. Affected users will notice that when they attempt to login to the Codat portal they will instead see an Error page displayed. This incident will be updated as soon as we have identified the issue, and no later than 11:19 AM
Report: "Intermittent 500 errors in the portal and API"
Last updateFor approx 8.5 mins on June 25th, starting at midday, some visits to the portal, as well as requests to our API, were failing with a 500 category error being displayed to the end user. Our monitoring was able to immediately picked this up and our engineers rolled back a code change which had introduced the issue.