Historical record of incidents for Check
Report: "Increased Errors on Payroll Endpoint"
Last updateWe have identified an issue causing elevated rates of 500 errors on the payroll endpoint. A fix for this issue is currently being deployed.
Report: "500s on payroll preview"
Last updateWe have identified an issue causing elevated rates of 500 errors on payroll preview. A fix for this issue is currently being deployed.
Report: "Elevated Payroll Preview 500s"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring the results. We'll continue to monitor over the course of the next business day and then resolve this incident.
The issue has been identified and a fix is expected within one hour.
We are continuing to investigate this issue.
The Check team is investigating an issue for elevated 500s on the payroll preview endpoint. This is impacting a subset of reimbursement only payroll items.
Report: "Some Employee Payments Landing at 1pm ET"
Last updateIncident is now resolved. All impacted employees have been paid. Root cause has been identified.
This incident only impacts employee payments made using Instant Payments. The Check team will follow up with a list of all impacted companies and workers.
We are aware of several employee payment failures the morning of February 28th. You may experience employees reporting that payments have not yet landed in their accounts. Everyone will be paid today. Employee payments are being processed via Same Day ACH and these will land by 1pm ET.
Report: "Check API Degraded Performance"
Last updateWe have confirmed the applied fix has resolved this issue. API interactions for Payroll and Employee endpoints have been restored to normal activity.
We have identified and applied a fix to address the increase in 502 errors across the Payroll and Employee endpoints, and are now monitoring its effectiveness.
We are currently investigating elevated errors stemming from resource contention (known to be impacting Payroll and Employee endpoints).
Report: "API unavailability from 1:30-1:37pm ET"
Last updateFrom 1:30 - 1:37pm ET, Check's API was unavailable, affecting Console and Components. As of 1:37pm ET, service has been restored.
Report: "Check Component Links Failing"
Last updateThe Employer Component Link generation issue is now resolved.
Check identified an issue with certain Employer Component Links failing (known to be impacting Run Payroll Component and Filing Authorization). A fix to mitigate this issue is deploying now. Estimating a resolution within 30 minutes.
Report: "Check API Degraded Performance"
Last updateAPI, Console, and Onboard requests from 4:18–4:22pm and 4:55–4:59pm ET saw elevated errors stemming from database migrations creating resource contention. We have resolved the issue.
Report: "Degraded API response latency"
Last updateFor 25 minutes from 6:05–6:30pm ET, Check's API latency was elevated beyond normal levels across endpoints. We've identified the root cause and are implementing immediate mitigations to prevent similar issues going forward.
Report: "Potential Payments Delays"
Last update9:30am ET Update: We have received multiple confirmations of successful funding for today’s payrolls. We are confident that everything was paid on time this morning and will be paid on time for future payrolls. We are now considering this incident resolved. We plan to follow up with a Root Cause Analysis next week that will outline in greater detail what happened and the steps Check is taking to avoid any delayed payments going forward. In the meantime, if you have any questions or should any issues arise, please reach out to support@checkhq.com or via our Support Portal. Please know that even with this immediate incident considered resolved, we remain diligent in our efforts to ensure complete & ongoing payments reliability. We understand that paying people on time is critical to our collective success. We simply did not meet the mark with yesterday’s payment delays. We appreciate your patience and understanding as we work to ensure payments reliability, and we will follow up with more details on our go-forward approach next week.
6:00pm ET Update: We have continued to work closely with JP Morgan Chase to remedy the situation and ensure all transactions going forward are paid out on time. To recap, all credits that were delayed today were successfully processed at the 10:30am ET ACH window and were settled with receiving banks at 1:00pm ET. As a reminder, it may take until midnight local time for funds to be made available to affected employees & contractors, depending upon the bank. Regarding the debits that were delayed, we have confirmed with JPMC that these debits will process in the overnight 8:30pm ET ACH window tonight and will settle at 8:30am ET tomorrow morning. Regardless of when Check receives those funds, there will be no impact or delay for any future credits to employees & contractors for whom those debits were meant. Check will float the risk for all of the delayed debits to ensure everyone impacted by the delayed debits is paid on time tomorrow morning. Looking forward, we have spent several hours collaborating with the JPMC team to gain comfort in tomorrow’s payments landing on time. Specifically, we have pre-cleared the amounts for tomorrow’s debits and credits with their risk team, ensuring that no limits will be tripped that could cause a manual review. We have also confirmed that our account balance is more than sufficient to process transactions tomorrow. We will continue to coordinate closely with the JPMC team throughout the evening until all payments have been released to the Fed. This will be our final update this evening. Once we have confirmed that tomorrow’s payments have been successful, we will provide one more response and close out this incident. We will also plan to provide a full Root Cause Analysis next week. Thank you all for your continued patience and understanding in this matter.
3:30pm ET Update: We continue to work closely with JP Morgan Chase to remedy the situation. We have now identified all of the employee & contractor credits and employer debits that were delayed this morning. We have updated your existing impact lists with this data, inclusive of when we expect these transactions to settle. Please see these in the “Actually Impacted Debits” and “Actually Impacted Credits” sheets of the file [2023-07-20] Delayed Payments by going to the Incidents folder of our Shared Google Drive. We are now focusing exclusively on ensuring that all payments for tomorrow will land on time. We will follow up by 6:00pm ET with our latest status.
2:15pm ET Update: We continue to work closely with JP Morgan Chase to remedy the situation. All credits that were delayed today were successfully processed at the 10:30am ET ACH window and were settled with receiving banks at 1:00pm ET. As a reminder, it may take until midnight local time for funds to be made available to affected employees & contractors, depending upon the bank. Regarding the debits that were delayed, we have confirmed with JPMC that these debits will process in the overnight 8:30pm ET ACH window tonight and will settle at 8:30am ET tomorrow morning. Regardless of when Check receives those funds, there will be no impact or delay for any future credits to employees & contractors for whom those debits were meant. Check will float the risk for all of the delayed debits to ensure everyone impacted by the delayed debits is paid on time tomorrow morning. We continue to work with JPMC to gather up a list of all impacted credits to employees & contractors and debits from employers and will share that information with you ASAP. We will follow up as soon as impact lists are prepared, or latest at 5:00pm ET with our latest status.
11:30am ET Update: We have successfully worked with JPMC to hit the 10:30 am ET Same Day ACH window and have now confirmed that all credits have been sent out in that window. This means that funds would settle with receiving banks by latest 1:00pm ET. Please note that we have already confirmed via our canary payments and specific employees that these payments have landed. However, it may take until midnight local time for funds to be made available to affected employees & contractors, depending upon the bank. For any debits that have not yet completed, we are still investigating their status. Regardless of processing timeline, we are committed that all employees & contractors will be paid on time tomorrow. We will follow up at 2:00pm ET with our latest status.
11:00am ET Update: We are continuing to work with JP Morgan Chase to unblock the subset of payments that were held up in a manual risk review. At this point, the payments have passed any risk checks and files are currently undergoing JPMC’s release process. Our efforts currently are to release affected payments at the next Same Day ACH window at 2:45pm ET. This would mean that funds would settle with receiving banks by 5:00pm ET. Please note that it may take until midnight local time for funds to be made available to affected employees & contractors. We will follow up again as soon as we confirm these payments will hit the 2:45pm ET window, or otherwise by 1pm ET, with our latest status.
Update: We have established contact with JP Morgan Chase and have determined that roughly one-third of our payment transactions have been held up due to a manual review on JPMorgan’s end. At this current time, we are not certain which of these transactions have been paid out and which are delayed. JPMorgan is actively working to release the currently-held funds and have everyone paid today via one of today’s Same Day ACH windows. We will follow-up by 11:00am ET with another update.
Several of our test payments through JP Morgan Chase have not landed on time for this morning’s 8:30am ET ACH window. Based on this information, it is likely that payments will be delayed for some of your customers. What we have identified: Check sends several debit + credit canary test payments out during all money movement transfer windows to ensure that payments land on time. This morning’s payments did not settle at their usual time and are now beyond the 8:30am ET ACH settlement window, which may indicate that some customer payments due to be paid out this morning are delayed. Note however that we have seen our aggregate bank balance in JP Morgan Chase increase overnight, indicating that some money movement has been successful. As a reminder, please note that the statuses in Check’s system assume payment success, and are currently not reliable indicators of whether a payroll has successfully been paid. Customer impact: All customers who were meant to be debited and credited today are potentially impacted. For more detail on affected customers, please see the file [2023-07-20] Delayed Payments by going to the Incidents folder of our Shared Google Drive. Note that these delays do not impact payments to tax agencies. What action is required: At this time, we do not recommend that you reach out proactively to customers. Rather, we suggest you brief your Support teams to be prepared to speak to any delay and planned resolution. Please let us know if you hear any reports of delayed payments from your customers. When we plan to next follow up: We will follow up by no later than 10:00am ET with additional information. Check takes paying people on time seriously. We apologize for the delay and are working hard to rectify the situation and get all impacted persons paid out today. Please reach out to support@checkhq.com or via our Support Portal if you have any questions.
Report: "Partial production outage"
Last updateFor 2.5 minutes, API, Console, and Onboard requests from 6:59:30 to 7:02:00 ET saw elevated errors stemming from a rate limiting change. We have resolved the issue.
Report: "Delayed Employee / Contractor Payments"
Last updateFollowing up with great news. We have received several notices of successful payments to employees & contractors. We have also seen a successful test payment that Check sent come through as part of the same ACH file. Together, these factors give us strong confidence that the payments we sent at the 4:45pm ET same-day ACH window have settled with receiving banks and all affected employees & contractors will be paid today. As a reminder, while the funds have settled, it is up to the discretion of the bank itself on when they make those funds available to the employee / contractor in their bank account. By law it is Midnight local time, but timing varies bank by bank. We are now considering this specific incident of delayed payments resolved. We will follow up in the coming days with updates and further guidance as needed. If you have any questions, please reach out via our Support Portal. Thank you again for your patience during this challenging time. We know it has been a hard day for you and your customers. Finally, as mentioned earlier, our CEO, Andrew, will be sending a message recapping today's events and outlining the steps we'll be taking to ensure reliable payments going forward.
Great news. All remaining payments that were not sent earlier have now been fully released to the Federal Reserve through Blue Ridge Bank via Increase. These payments will go out in the 4:45pm ET Same Day ACH window. As a reminder, funds will settle with the receiving bank by 6pm local time; note that while funds are often made available immediately, it is up to the discretion of the bank itself on when they make those funds available to the employee / contractor in their bank account. By law it is Midnight local time, but timing varies bank by bank. In the meantime, we will continue to actively monitor the status of these payments with Increase to ensure smooth delivery. We plan to follow up with confirmation as we begin to receive reports of employees and contractors receiving payment later this evening. Thank you for your patience and support. We know it has been an unexpected and trying day for you and your customers. We feel thankful and privileged to be operating from a position of strength, with a full, liquid balance outside SVB. This evening, our CEO, Andrew, will be sending a message recapping today's events and outlining the steps we'll be taking to ensure reliable payments going forward.
We are continuing to work closely with Increase to ensure payments go out today. We have identified all employees & contractors who have yet to be paid. We have now provided all required information to their team to create and submit the ACH file required to send payments, and are actively working with them to ensure that file is submitted on time. We jointly expect to meet the 4:45pm ET Same Day ACH window. Funds will settle with the receiving bank by 6pm local time; note that while funds are often made available immediately, it is up to the discretion of the bank itself on when they make those funds available to the employee / contractor in their bank account. By law it is Midnight local time, but timing varies bank by bank. We appreciate your continued understanding and patience. We will follow up with more details no later than 4:45pm ET.
We have continued to receive some positive reports that employees & contractors have received their funds. We have determined that these payments were made out of a payment attempt that we submitted to Blue Ridge Bank via Increase earlier this morning. We are now working with Increase to jointly verify whether all or only some employee & contractor payments have been sent, with the goal of understanding who, if anyone, still needs to be paid today. We remain committed to paying all employees & contractors today and will send out any additional payments, if needed, by latest the final Same Day ACH submission window at 4:45pm ET through Blue Ridge Bank via Increase. We appreciate your continued understanding and patience. We will follow up between 2:45-3:00pm ET with an update.
** Update re: today’s payments - Action Recommended ** Silicon Valley Bank has now officially been closed by the FDIC. We have included a link to more information below. There are three things we want to communicate: First, Check’s corporate funds are not held at SVB; they are now at JP Morgan Chase. Check remains stable and fully liquid. Second, we remain committed to ensuring that all employees and contractors are paid before the end of the day. Currently, the status of those payments remains unclear. Within the last few minutes we have received multiple reports from employees of payments landing in their bank accounts, generally shown as pending. We do not yet know if all payments have been made. We are investigating the status of those payments as quickly as possible. If we are not able to 100% confirm that all payments have been made through SVB ahead of the final Same Day ACH submission window at 4:45pm ET, we will make employee payments through Blue Ridge Bank via Increase. We will follow up by 1:30pm ET with an update. https://www.fdic.gov/news/press-releases/2023/pr23016.html
Payments were unfortunately not submitted by the 10:30am ET cut-off for Same Day ACH settlement by 1pm. There are multiple additional Same Day ACH submission windows this afternoon, in practice all of which settle around End of Day local time. We are committed to ensuring that all employees and contractors are paid before End of Day. Here are more details on why payments were not submitted in the 10:30am ET window, and what will happen from here: Check is coordinating with both our primary bank, SVB, and our fallback bank, Blue Ridge via Increase. Our priority is to ensure that everyone is paid today, while minimizing the risk that everyone is paid twice. Check had coordinated with Blue Ridge via Increase and was fully prepared to submit payments ahead of the 10:30am cutoff. However shortly after 10am we received word from SVB that they had constructed a single master ACH file for the entire bank that they would be submitting, and that they could not pull Check’s payments from. Unfortunately SVB missed the 10:30am cutoff for submitting that file, but they are still endeavoring to do so today. Check is coordinating with Blue Ridge via Increase to ensure we are prepared to submit Same Day ACH payments for this afternoon’s submission windows. Our hope is to gain some confidence from SVB on whether or not they will be able to successfully submit their master ACH file. If we are not able to get certainty from them, in either direction, we will proceed with Same Day payments through Blue Ridge via Increase. We are aiming to get employee & contractor payments out for the next Same Day ACH window at 2:45pm ET, but are guaranteeing we will get payments out by the last Same Day ACH window at 4:45pm ET. Link to more information about ACH windows below. We will make the decision re: which window we send payments to as we learn more from our two banking providers. We appreciate your patience and will continue to keep you updated through the day as the situation evolves. We will share a status update at 1pm ET. https://www.nacha.org/system/files/2021-03/SDA_Schedules_and_Funds_Availability.pdf
** Update re: today’s payments - Action Recommended ** After monitoring our scheduled payments this morning, we are not aware of any payments successfully settling at receiving banks of employees / contractors. We have decided to shift our employee & contractor payments for today’s payrolls to process via a different banking provider. Recap from yesterday: Silicon Valley Bank (SVB) issued an earnings report yesterday morning that raised concerns about the bank’s liquidity. Check relies on SVB as our primary ACH processor, so we have been monitoring the situation closely and actively preparing for contingency plans to ensure employee & contractor payments arrive today. When will funds be received? Our current expectation is that funds will be transferred to employee / contractor banks by 1pm local time for all Direct Deposits dated for today. We will confirm that timing by 10:30am ET. Note that while funds are often made available immediately, it is up to the discretion of the bank itself on when they make those funds available to the employee / contractor in their bank account. By law it is Midnight local time, but timing varies bank by bank. What actions are required? While no action is required of you, however, due to funds landing later than usual today, we would suggest you notify your clients of this expectation to reduce your support volumes. Please see below for a template for those communications. The actions we are taking should resolve the issue and remit payments to employees & contractors today, and no additional action is required of you. We will continue to monitor to confirm that the payments successfully land in accounts. We apologize for any confusion this issue may have caused. Nothing is more important to Check than timely and accurate payments. We remain focused on reinforcing our payments infrastructure to ensure the reliability of our payroll service to you and your customers. We will follow up with more specific details about our go-forward plan later today. Template for customers [Hi customer], Due to an issue with our banking provider, employee & contractor funds for payrolls that were meant to be paid today are delayed. We are confident that payments for your employees & contractors will land today. We will follow up with more details around specific payment timing as we receive confirmation. We apologize for this inconvenience. Nothing is more important to our service than processing timely and accurate payments for you. We will work with our banking provider to identify the issue and resolve it on a go forward basis.
Report: "Delayed Payments"
Last updateAfter confirming the successful landing of funds within several more of the impacted accounts, we are now marking this issue as Resolved. If you have any questions related to this incident, please contact our Support team via the Support Portal in Console. Again, we sincerely apologize for this payments delay. We know how important timely and accurate payments are to you and your customers. We will be conducting Root Cause Analysis to gather a more thorough explanation of this incident and future remediation.
Several employee and contractor accounts that were impacted by today’s payments delay have now received funds. This confirms that the ACH file has been processed and Direct Deposits will be posted by 5pm local time. We will continue to monitor this issue but at this time, we expect it to be resolved. We will follow up again at 5pm ET.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We have confirmed our ACH file has been received. All Direct Deposits dated for today will be posted by the bank and funds will be posted at 5pm local time. Please notify your clients of this expectation. While this should resolve the issue and remit payments to affected employees, we will continue to monitor to confirm that the payments successfully land in accounts. We sincerely apologize for this delay. Nothing is more important to Check than timely and accurate payments. We will be conducting a Root Cause Analysis over the coming few days that will contain a more thorough explanation of this incident and future remediation.
We have isolated the issue to one of our banking providers and are actively investigating root cause and resolution paths with that provider. We are simultaneously exploring other means to send employee payments today. We remain confident that employee payments will land today. We will send an updated status within the next hour or if we have meaningful updates to share in the interim.
Brief update: You may notice that the status for these payments is currently Paid. Please disregard this status. These payments have still not landed and we are investigating both the issue as well as methods to get employees paid today.
We are aware that employee payments have not landed this morning. We are investigating this with the highest degree of urgency. It is our goal to ensure that all impacted will receive payment today. We will send an updated status within the next hour.
Report: "Check API Degraded Performance"
Last updateThis issue has been resolved as of 2:36pm ET. Check API experienced intermittent degraded performance today at 11:36amET (6 min), 12:19pmET (3 min), and 2:24pmET (12 min). We have taken immediate action to prevent similar disruptions going forward.
The issue has been identified and mitigated as of 2:36pmET. We will continue to monitor the Check API for any degraded performance. All impacted partners will now see immediate resolution.
Check API is experiencing increased error rates and webhook delivery latency. Issue is under investigation.
Report: "Applied For companies unable to approve payrolls"
Last updateWe have continued seeing error rates return to normal and are making the incident as resolved. Check will conduct a postmortem and a root cause analysis will be provided on request. If you continue to experience issues, please reach out to support@checkhq.com.
A fix has been deployed and we are monitoring the results.
We have identified an issue where companies that have marked some tax params as Applied For are unable to approve payrolls. Companies submitted for a start date of today will experience delays until the issue is resolved. We have identified the issue and are actively deploying a fix. We will provide another update within the next in 30 minutes.
Report: "Elevated 2023 Arizona Payroll failures"
Last updateThis incident has been resolved.
We have confirmed an issue with previewing and approving payrolls in 2023 with employees paid in Arizona. We are currently working on a fix.
Report: "Inconsistent Document Request Statuses"
Last updateDocument inconsistencies have been resolved.
The immediate problem is resolved. We have moved all affected document requests into the correct state and are keeping our eyes open for any further inconsistencies.
We have identified and are in the process of fixing an issue where some document request statuses may be incorrect in Console (i.e., showing pending when in fact the request is open, or vice versa). We will update once the data has been corrected.
Report: "Document Uploads Not Updating Statuses In Some Cases"
Last updateDocument uploads and requirements appear to be working properly, so we've resolved this incident.
We have deployed a fix for this issue and are monitoring to ensure no further uploads end up in this state. We'll fully resolve if we see no more affected docs for the next 60 minutes (until 1:18PM Pacific time).
We've updated the status of documents already uploaded to reflect the correct status. The root issue is not yet resolved, but we'll be updating statuses every 15 - 20 minutes until it's resolved.
We're aware of an issue impacting our document upload feature where documents are being uploaded but are not changing the status of the task. We'll provide an update shortly.
Report: "Console Downtime"
Last updateWe have continued seeing system access return to normal and are declaring this incident resolved. The total downtime for our systems was from 5:27 - 5:45 PM ET. If you continue to experience issues please reach out to support@checkhq.com.
We have determined the root cause and implemented a fix. Please attempt to log out and log back in if you're still experiencing issues within Console. If you experience issues with Onboard, please attempt to reopen the onboarding link.
We are investigating elevated error rates across Check API, Console, and Onboard.
We are continuing to investigate this issue.
We are aware that users are experiencing errors while using Check Console, and are investigating.
Report: "Historical Payroll Data Upload Failures"
Last updateThis incident has been resolved.
We are aware of an issue that is preventing uploading historical payroll data in Console. The issue has been identified and we are working on a fix.
Report: "Issue accessing sandbox tab within Console"
Last updateThe fix has been deployed. If you are still experiencing issues, a refresh should fix the issue.
Our Engineers have identified the issue, and are deploying a fix.
We are investigating an issue where users are experiencing issues accessing the sandbox tab within Console.
Report: "Increased Payroll Preview 500s"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring for any issues.
We are currently investigating the issue with increased 500s in payroll preview.
Report: "Issues creating employees/contractors"
Last updateWe have not observed any instability after the initial window. We will continue to monitor for any instability and work with our vendor to increase the stability of our system
"From 16:25 EST to 16:31 EST we observed issues with creating employees and contractors in sandbox and production. This is due to an issue with a geolocation vendor, we are escalating this to our vendor and will continue to monitor the situation."
Report: "Webhooks Delayed"
Last updateThis incident has been resolved.
We have deployed the fix and are observing successful processing of webhooks. We will continue monitoring until all pending webhooks are sent.
We have identified an issue with processing webhooks which has caused webhooks to be delayed. The issue has been identified and the delayed webhooks will be sent soon.
Report: "MO state tax not calculating"
Last updateWe've identified the impacted payrolls and will be reaching out today with next steps to get these resolved. If you do not receive any communication from Check today, you were not impacted by this incident. Please reach out to us if you have any questions or need additional information on this incident.
As we've continued our investigation, we have identified the root cause of the issue and have additional details to share. <b>To clarify the issue, any payroll previews and/or approvals generated after 11:26 AM EDT on September 9th did not include Missouri State Tax withholding for employees earning wages in a Missouri workplace</b>. We are also in the process of rolling out a fix for this issue and will provide an update when that has deployed.
Any payroll with an employee working in Missouri has not been calculating Missouri state tax correctly since Sept 8th 2022 7PM EST.
Report: "Delays in sending webhooks"
Last updateWebhook delivery times have returned to normal.
We have seen improvements in webhook delivery times and are continuing to monitor.
We are currently investigating an issue causing delays in sending webhooks.
Report: "Customers unable to create employees via POST employees"
Last updateThe fix has been deployed. Employees and contractor creation has been verified in production.
A fix is currently being deployed and we are monitoring the situation. We expect the next update within 10 minutes.
The root cause has been identified and we are working on a fix.
We are investigating an issue affecting some customers to create employees when the employee country field is not properly populated.
Report: "Some company tax documents unavailable through API and Console"
Last updateAccess to all company tax documents has been restored.
Some company tax documents for Q2 2022 are temporarily unavailable through the Check API and Console. Our engineers are currently working on restoring access to these documents.
Report: "Plaid link token degredation"
Last updatePlaid has recovered and we have confirmed link sessions are working as usual.
Plaid is experiencing an incident with creating link tokens. We are awaiting a status update from the vendor.
Report: "Payments and Bank Account endpoints experiencing degradation due to high vendor latency"
Last updateThis incident has been resolved.
High latencies have subsided. We will continue to monitor the performance.
The issue has been identified and a fix is being implemented.
Our vendor is currently investigating this issue. At this time, Payments APIs and Bank Account endpoints may experience poor or unstable performance.
Report: "Bank Account creation and updates inacessible due to Vendor service degradation"
Last updateOur vendor rolled back a faulty system upgrade and all systems are operational now.
Our vendor implemented a fix and is now monitoring.
We are waiting on guidance from our vendor to inform timelines. At this time bank account creation and updates are degraded / inaccessible.
Report: "/bank_accounts endpoints unavailable"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Faulty metrics reporting code is causing our /bank_accounts endpoints to be unavailable. We have identified the issue and are imminently deploying a solution.
Report: "Vendor performance degradation impacting /bank_accounts endpoints"
Last updateThis incident has been resolved.
A vendor performance degradation is resulting in inconsistent or inaccessible bank account creations or updates at this time.
Report: "Companies Unable to Approve Payroll"
Last updateWe have released a fix and confirmed that we've restored the ability for companies to approve payroll.
We're aware of an issue impacting the ability to approve payrolls and are actively looking into solutions.
Report: "Check API System Outage"
Last updateCheck-api was unavailable in production for ~3 minutes between 10:10 AM - 10:13 AM while resolving a database migration issue.
Report: "Vendor performance degradation impacting /bank_accounts endpoints"
Last updateThe vendor implemented a fix and are monitoring for additional issues.
A vendor performance degradation is resulting in inconsistent or inaccessible bank account creations or updates at this time.
Report: "Delays on sending webhooks"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are seeing delays in webhook delivery in both live and sandbox environments. We are working to resolve this immediately.
Report: "Employee and contractor credit payments delayed"
Last updateThis incident has been resolved.
All affected employee and contractor payments have been sent and will be received this afternoon.
We are investigating a payment processor vendor outage that affected employee and contractor credits scheduled to settle this morning. We are working with urgency to get these payments sent out as soon as possible.
Report: "Degraded availability for Bank Account endpoints"
Last updateThis incident has been resolved.
A vendor outage is resulting in degraded availability of our /bank_account endpoints. The problem has been identified and a fix is on the way. We are actively monitoring the situation.
Report: "IRS Downtime Impacting Company Verifications"
Last updateThis incident has been resolved.
We're seeing expected responses from the IRS and are working through the verification queue. We will continue to monitor.
The IRS is recovering and is able to service some of our business verification requests. We are still working through the queue of companies which we were unable to immediately verify.
The IRS is currently experiencing an outage, impacting our ability to verify any new business that sign up with Check. Further details forthcoming.
Report: "Intermittent employee/contractor creation issues"
Last updateThis incident has been resolved.
Last issue was observed at 11:19am EST, but we will continue to monitor.
We are investigating intermittent issues with 500s in our employee and contractor API
Report: "Payroll list endpoint temporarily unavailable"
Last updateThis incident has been resolved.
A fix has been applied to the database connection pooler, we are monitoring that there are no errors.
We have continued to notice database issues as the situation has progressed. We are attempting another fix to our database connections.
A fix has been applied and the payroll list endpoint should load as normal. Sandbox is still recovering.
The payroll list endpoint was temporarily returning a 500. We have since reverted the change and this should be resolved. We will continue to monitor.
Report: "Check API system outage"
Last updateCheck-api was unavailable in production for ~10 minutes, and sandbox was down for about 20 minutes while resolving an AWS networking issue.
Report: "Payroll status reporting delay"
Last updateThis incident has been resolved and all affected payrolls are showing their correct statuses. Note: the original post about this incident contained a typo on the date: the payrolls that were affected had a payday if 2022-01-21. The root cause of this issue was a degradation in the webhook delivery subsystem from our payments processor.
We are currently investigating an issue where some payrolls with a payday of 2021-01-21 remain in a `processing` status when they should have transitioned to a `paid` status. We have no reason to believe that any employees have not actually received their net pay - we believe this issue is isolated to the status of the payroll and is caused by a reporting delay with our banking partner. We are actively investigating this matter.
Report: "Degraded Payroll Calculations and Geocoding"
Last updateWe are no longer seeing errors for payroll preview, approval, and creation of API resources with location information. Our system is fully operational. This incident has been resolved.
The root cause has been identified as an AWS network outage, which has been resolved. Error rates have decreased on payroll preview, approval, and the creation of API resources with location information. We are monitoring for resolution.
We are investigating a partial degradation that impacts payroll preview and approval, and the creation of API resources with location information. We will provide updates shortly.
Report: "Partial System Outage"
Last updateThis incident has been resolved. All services are functioning normally at this time.
We have resolved the issue. All systems are operational and we are currently monitoring.
Check API is down. Console, Onboard and Components are impacted. We understand the root cause of this issue and are working on resolving the issue.
Refer to next update.
We have migrated Console to a backup environment. At this point, all Check services are hosted on environments that are not impacted by the AWS outage. We will continue to monitor the health of our systems.
We have migrated Onboard to a backup environment, bypassing the AWS outage.
We have migrated Check API to a backup environment, bypassing the AWS outage. Console, Onboard, and Components users may continue to see elevated errors.
We are continuing to investigate the issue.
We are continuing to investigate the issue.
We have identified a partial outage to our system due to an AWS outage. Check API, Onboard, Console, Check Express, and webhook processing are impacted. We are continuing to investigate.
Report: "Temporary Sandbox Data Loss"
Last updateThis incident has been resolved.
From August 23 at 10:38:35 AM UTC to August 24 at 1:10:51 AM UTC, we experienced an outage in our sandbox environment. This has resulted in a temporary loss of data for any sandbox activity between 4:38 PM UTC on August 23 and 1:10 AM UTC on August 24th. We are actively working on restoring the missing data from this time period.