Historical record of incidents for Procurify
Report: "Major Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Major Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Slow Load Times and Application Unavailability"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Report: "Bill sync unavailable for bills without invoice numbers"
Last updateThis incident has been resolved.
The issue is resolved. No action is required on your side
Bill syncs to accounting software will show an error if you have not provided an invoice number. The issue has been identified and a fix is being deployed.
Report: "Netsuite Master Data Sync Vendors missing"
Last updateThis incident has been resolved.
A fix has been implemented. We will continue monitoring before resolving this incident.
A fix has been implemented. Any vendors that were erroneously deleted will be re-added as they were. It will take some time for the data to update fully. The next update will be once this is complete.
We are continuing to investigate the issue. The next update will be once we have a timeline for the fix, or in 2 hours, whichever comes first.
We are continuing to identify and implement a fix. The fix is expected to restore the deleted Vendors. out of an abundance of caution we have disabled the ongoing sync of Account Codes temporarily.
We are continuing to investigate this issue.
We are currently investigating an issue where if you have Master Data Sync enabled for Netsuite (Pulling vendors from Netsuite). All vendors are showing as Deleted. We are working with our Integration Partner for a fix and expect vendors will be re-added soon.
Report: "Missing AP Navigation Links"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring the results. AP navigation is now showing
The issue has been identified and a fix is being implemented.
We are continuing to investigate the issue. To navigate to Create Bill go to /bills/create route of your domain. To navigate to Manage Bills go to /bills/requests/draft route of your domain.
We are continuing to investigate the issue. To navigate to Create Bill go to /bills/create route of your domain. To navigate to Manage Bills go to /bills/requests/draft route of your domain.
We are continuing to investigate the issue. To navigate to Create Bill go to /bills/create route of your domain. To navigate to Manage Bills go to /bills/requests/draft route of your domain.
We are investigating an issue where Create Bill and Manage Bill navigation links are not displaying. Our team is currently working to fix the issue. To navigate to Create Bill go to /bills/create route of your domain. To navigate to Manage Bills go to /bills/requests/draft route of your domain. We apologize for any inconvenience caused. We will provide another updated in 15 minutes. Thank you for your understanding.
Report: "Degraded Performance: API"
Last updateAt 11:08 am PT a configuration change to protect our application against automated bot attacks was released. Unfortunately, the ruleset was too board and it blocked a small subset of legitimate traffic. The change was reverted at 11:45am and the application is now running as expected. We apologize for the inconvenience.
We have resolved an issue that intermittently degraded the performance of our API endpoints between 11:08 a.m. and 11:45 a.m. PT. This issue affected a subset of requests to our application endpoints. We apologize for any inconvenience and thank you for your understanding.
Report: "Microsoft Business Central Accounting Integration - Bill sync failures."
Last updateThis incident has been resolved. Please reach out to support if you have any further issues.
A fix has been deployed. Customers may re-sync and bills stuck in "in-progress" to retry. We will continue to monitor to confirm there are no further or related issues.
Syncing Bills to Microsoft Business Central dynamics 365 is currently failing. It will result in a stuck "in progress" state. Our integration partner is working on a fix. Once resolved, we will move this stuck bills to "Failed" and you will be able to re-sync successfully. The next update will be once we hear back from our partner and we have verified the issue is resolved.
Report: "[Partial Outage]: Web application"
Last updateThank you for your patience, this incident is now resolved
We have fixed the infrastructure issue with unhealthy pods that caused the incident and we are continuing to investigate internally for the root cause. We will continue to monitor this situation and update with any new information.
We have fixed underlying issue and have restored service. We are continuing to investigate the root cause of the issue with our cloud provider
We have restarted some infrastructure components which has increased the performance of the system. We continue to investigate the root cause. We will provide an update again in 15 minutes or when we have new information
We are continuing to investigate the ongoing incident internally and with our cloud provider. We will provide another update in 15 minutes or when we have new information
We are continuing to investigate this issue.
We are investigating a partial outage for our web application service. Our team is currently working to restore the service.
Report: "Service Outage"
Last updateThis incident has been resolved.
This incident has been resolved. We will continue to monitor the affected services.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are investigating a full outage for our web application service. Our team is currently working to restore the service. We apologize for any inconvenience caused. We will provide another updated in 15 minutes. Thank you for your understanding.
Report: "Major Outage: Web Application"
Last updateProcurify was down between 11:45AM-11:49AM PST. We have resolved the issue and will continue to monitor the affected services. Thank you for your understanding.
Report: "Major Outage: Web Application"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified a major outage of our web application services. Our team has identified the issue and is working to resolve the issue. We apologize for any inconvenience caused. We will provide another update in 15 minutes. Thank you for your understanding.
Report: "Intermittent API service unavailability"
Last updateWe have deployed a fix to improve performance of our services. We will continue to monitor the situation. We apologize for any inconvenience caused. Thank you for your understanding.
Report: "Login Issue"
Last updateThis incident has been resolved.
We have deployed a fix and login should be working again for all users. We will continue to monitor the situation
We are currently investigating an issue where users cannot log into Procurify. Users who are currently logged in are not affected.
Report: "Accounting Integration Bill Syncs"
Last updateThis incident has been resolved.
This incident has been resolved with an update to our integration with your Accounting Platform. If any bills were synced during this incident, They would have successfully synced with your accounting software after being "In progress" for some time, however, the attachment may not have. You may need to manually upload the attachment in your accounting software for the affected bills. Please reach out to support if you have any questions
An issue has been identified and a fix is in progress. Syncing bills will report as "In progress" for an extended period. There is no need to retry a sync, It would have completed.
Report: "Accounting Integration Bill sync errors"
Last updateThis incident has been resolved.
This has been resolved, New syncs should no longer fail. For any syncs stuck in "in progress" - They will be fixed soon, Do not attempt to re-sync these.
We are currently investigating an issue resulting in Bill Syncs being stuck in in-progress. It would be best not to retry stuck syncs while we look into this.
Report: "Partial Outage: AP Automation Inbox Email"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are investigating a partial outage of our AP Automation Inbox Email service. Our team is currently working to restore the service. This outage affects customers with AP Automation enabled. • Inbox emails are not displaying in the "Create Bill" page We apologize for any inconvenience caused. We will provide another updated in 15 minutes. Thank you for your understanding.
Report: "Major Outage: Web Application"
Last updateBetween 2023-12-20 10:07 PST and 2023-12-20 10:16 PST, our Web Application services were unavailable. We have identified and fixed the issue. This incident has been resolved. All systems are operational. Thank you for your understanding.
Between 2023-12-20 10:07 PST and 2023-12-20 10:16 PST, our Web Application services were unavailable. We have identified and fixed the issue. We will continue to monitor the affected services. Thank you for your understanding.
We are investigating a major service interruption for our Web Application service. Our team is currently working to restore the service. This outage affects: • Web Application We apologize for any inconvenience caused. We will provide another update in 15 minutes. Thank you for your understanding.
Report: "Partial Outage: Authentication Services"
Last updateThis incident has been resolved.
Between 2023-12-18 1:35 PST and 2023-12-18 5:55 PST, our Web Authentication services were intermittently unavailable. We have identified and fixed the issue. We will continue to monitor the affected services. Thank you for your understanding.
We are continuing to work on a fix for the partial outage to our Authentication service. Our team is currently working to restore the service. In the meantime, clearing cookies in the browser may fix the issue. We apologize for any inconvenience caused. We will provide another update in 30 minutes. Thank you for your understanding.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are investigating a partial outage for our Authentication service. Our team is currently working to restore the service. This outage affects: • Authentication services We apologize for any inconvenience caused. We will provide another updated in 15 minutes. Thank you for your understanding.
Report: "Partial Outage: Login"
Last updateThis incident has been resolved. All systems are operational. Thank you for your understanding.
Between 2023-12-13 12:00 PM PST and 2023-12-13 11:40 PM PST, our login services were unavailable to a subset of users. We have identified and fixed the issue. We will continue to monitor the affected services. Thank you for your understanding.
Our team is working on deploying a fix for this issue. We will provide another update in one hour.
Our team is working on deploying a fix for this issue. We will provide another update in one hour.
Our team is working on deploying a fix for this issue. We will provide another update in one hour.
Our team is working on deploying a fix for this issue. We will provide another update in one hour.
Our team is working on deploying a fix for this issue. We will provide another update in one hour.
Our team is working on the fix for this issue. We will provide another update in one hour.
We are continuing to work on a fix for this issue. We will provide another update in one hour.
We are continuing to work on a fix for this issue. We will provide another update in one hour.
We have identified the issue. Our team is currently working to restore the service for affected users. We will provide another update in one hour.
We are investigating a partial outage for our Authentication service. Our team is currently working to restore the service. This outage affects: • Some users are unable to log in We apologize for any inconvenience caused. We will provide another update in one hour. Thank you for your understanding.
Report: "Accounting integrations sync unavailable"
Last updateThis incident has been resolved.
This is an intermittent issue. We will contnue to monitor over the next few hours. If you have bills stuck in "Syncing" It would be best to leave them, and not re-attempt.
Some bill syncs may be stuck in "Syncing". We are looking into the issue.
Report: "Login/SSO"
Last updateWe have identified that the login page issue was due to an outage in our third-party authentication provider. The issue has been confirmed to be resolved on their end.
The incident related to the login page has been resolved. We will continue to monitor the issue.
We are currently investigating an issue where customers are unable to log into the application. If you are logged in then there is no impact to you. Please remain logged in until this incident is resolved
Report: "Major Outage - Accounting Integrations"
Last updateBetween 2023-10-04 14:34 PDT and 2023-10-04 14:56 PDT, accounting integrations were unavailable. We have identified and fixed the issue. This incident has been resolved. All systems are operational. Thank you for your understanding.
Report: "Major Outage"
Last updateBetween 2023-10-02 12:52 PDT and 2023-10-02 12:57 PDT, our web application services were unavailable. We have identified and fixed the issue.
Report: "Major Outage"
Last updateThis incident has been resolved. All systems are operational. Thank you for your understanding.
All services restored and we will continue to monitor the affected services. Thank you for your understanding.
It was noted that the Spending Card service had yet to reboot. We are investigating this issue.
Most services and applications were restored. A few applications that were still rebooting were being monitored such as Spend Card and Pay Service. We are continuing to monitor for any further issues.
An investigation began at 00:43 and a root cause was identified. By 01:02, the root cause was fixed and monitoring began.
Multiple applications and services began to report failures
Report: "Major Outage"
Last updateBetween 2023-06-09 07:36 PDT and 2023-06-09 07:54 PDT, our services experienced instability resulting in periods of unavailability. This incident has been resolved and all systems are now operational. Thank you for your understanding.
We are investigating a full outage of our services. Our services are now restored and our team continues to monitor to ensure continued stability. We apologize for any inconvenience caused. Thank you for your understanding.
Report: "Service Outage"
Last updateThis incident has been resolved. All systems are operational. Thank you for your understanding.
We are continuing to monitor for any further issues.
Between 2023-05-24 11:01 AM PDT and 2023-05-24 11:03 AM PDT , all Procurify services were unavailable. We have identified and fixed the issue. We will continue to monitor the affected services. Thank you for your understanding.
Report: "Issue with PunchOuts for Amazon Business"
Last updateIntermittent outages have been resolved for Amazon Business PunchOuts
We've identified that Amazon Business is having intermittent Service outages for PunchOuts and customers may experience errors when attempting to enter their portal. We'll monitor the situation and hope they can resolve the issue soon.
Report: "Issues with login / logout"
Last updateOur third party authentication provider has resolved the incident related to login failures.
Updating from Partial Outage to Operational.
Login service is restored. Our third party authentication provider confirmed they've implemented a fix and are monitoring it. We will provide an update in 1 hour as we continue to monitor the health of the third party authentication provider.
We have confirmed that this incident is caused by an outage in our third party authentication provider Some customers are able to log in and log out at this time. If you are logged in, please remain logged in until this incident is fully resolved We will provide an update in 15 minutes or when we have more information from our authentication provider
We are continuing to investigate an outage with our third party authentication provider Some customers are able to log in and log out at this time. If you are logged in, please remain logged in until this incident is fully resolved
We are investigating whether this incident is caused by an outage in our third party authentication provider Reminder to stay logged in until this incident is resolved to continue to use the application
We are currently investigating an issue where customers are unable to log into or log out of the application. If you are logged in then there is no impact to you. Please remain logged in until this incident is resolved
Report: "Bill Creation Partial Outage"
Last updateThe bill creation page in the accounts payable module was experiencing errors upon bill submission. The incident is now resolved and the team is monitoring the system.
Report: "Degraded Performance"
Last updateAn upgrade to our database caused an unexpected performance degradation and as a result we initiated a database failover
Report: "Application Degraded Performance"
Last updateOur application experienced degraded performance for about 5 minutes at this time when a configuration change internally caused a database failover
Report: "Degraded Performance"
Last updateOur application experienced degraded performance during an unplanned failover of our database
Report: "Single Sign On Unavailable"
Last updateThis incident has been resolved.
We've confirmed that the issue with SSO should be resolved. We're continuing to monitor.
We have applied a fix and are validating it to confirm that the issue has been resolved.
We are investigating an issue where customers cannot sign into Procurify using the Single Sign On option
Report: "Partial Service Outage"
Last updateBetween 2022-11-29 11:38 PDT and 2022-11-29 13:41 PDT, all Procurify services were partially unavailable due to an outage experienced within our API infrastructure. We are working to mitigate the impact of such an outage should it reoccur in future. All systems are operational. This incident has been resolved. Thank you for your understanding.
Report: "AP Automation Partial Outage"
Last updateNo new issue since last observed.
The issue has been resolved and the last error was 1:04 PM PDT. We are monitoring the service in case any new issues arise.
The issues are still being worked on by our cloud provider and there are reports of error rates being improved but they do not have full recovery yet. Will provide further updates as recovery progresses.
We are still seeing some issues with the feature and monitoring the situation with the cloud provider.
We are observing a partial outage on the invoice inbox feature where the inbox emails may appear blank and some emails may be rejected. Currently, we are seeing issues with our cloud provider.
Report: "Login Errors and Degraded Performance"
Last updateWe have monitored our fix for the issue and are confident that the issue is resolved.
A fix has been implemented and functionality has been restored, we are monitoring the incident
We have identified the issue that is causing login failure and degraded performance and we are working to resolve the issue
We are continuing to investigate this issue.
We are currently investigating an issue where users are unable to login to the Procurify. Users who are currently logged in are experiencing degraded performance.
Report: "Service Outage"
Last updateBetween 2022-08-08 06:13 PDT and 2022-08-08 06:26 PDT, all Procurify services were unavailable due to an outage experienced within our storage infrastructure. We are working to mitigate the impact of such an outage should it reoccur in future. All systems are operational. This incident has been resolved. Thank you for your understanding.
Report: "Spending Card Outage"
Last updateThis incident has been resolved. All systems are operational. Thank you for your understanding.
Between 15:29 and 15:50 PST on Thursday, July 28 2022, Spending Card services were unavailable. We have identified and fixed the issue. We will continue to monitor our Spending Card services. Thank you for your understanding.
Report: "Service Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We've identified the issue and the fix has been implemented. We are currently monitoring.
We are currently investigating an outage with Procurify.
Report: "Degraded Performance"
Last updateOur fix has been deployed and performance has returned to normal, thank you for your patience
We are continuing to monitor for any further issues.
We have deployed a fix for our degraded performance and are monitoring the solution
The cause of our degraded performance has been identified and a fix is being implemented
We are currently investigating degraded performance across our Product where customers are seeing slow loading times
Report: "Spending Card issues"
Last updateOur spending card infrastructure continues to be stable. Thank you for your understanding.
Our spending card infrastructure is stable. We will continue to monitor.
We are investigating an intermittent issue with our spending card infrastructure which is affecting managing, loading funds, and transaction updates
Report: "Exports Delayed"
Last updateBetween 15:55 and 17:10 PST on Thursday, May 26 2022, our export task system experienced an issue where export processing was delayed. Exports prepared during this time have now completed. Thank you for your understanding.
Report: "Notification Email and Export Outage"
Last updateBetween 09:40 and 10:20 PST on Tuesday, May 3 2022, our notification email and export task systems experienced an outage. Notification emails were not sent during this time. Purchase order emails were unaffected. Exports prepared during this time did not complete. For exports which failed during this time, which currently show as "Preparing", we suggest creating new exports to download data. Thank you for your understanding.
Report: "Customers may not be able to log in"
Last updateThe incident has been resolved. Thank you all for your patience!
Our authentication provider has implemented a fix and situation has changed in a positive way - customers now should be able to log in. But we are continuing to monitor for any further issues. Thank you for your patience!
We have identified the issue with our authentication provider that is experiencing an outage at the moment.
We are currently investigating issues with logging in into the app.
Report: "Emails/Exports service outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Emails/exports were restored but we are continue to monitor for any further issues.
Emails/exports were restored but we are continue to monitor for any further issues.
A fix to resolve issue with emails not being sent has been implemented and now all customers should be able to send/receive emails. We are continue to monitoring the results.
An interim fix has been deployed to unblock the majority of email processing tasks and we are monitoring the results. Some customers may still experience emails not being sent. We are working on fixing it for all.
The issue has been identified and we are working on implementing the fix
We are currently investigating an issue with emails/export tasks are not processing
Report: "Service Outage"
Last updateThis incident has been resolved.
The issue with our provider AWS is resolved on their end and we are continuing to monitoring the application performance. Thank you all for your patience!
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating an outage across the Procurify application
Report: "Attachment Errors!"
Last updateFile server restored.
We are experiencing issues with our file server. Users may experience errors when uploading/downloading attachments, or if they are expecting attachments via emails. Thank you for your understanding.
Report: "Login fixed! Monitoring status!"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing problems with our login. If you are already logged in, please stay on, and avoid logging out. We are looking into this issue as we speak.
Report: "Operational (closely monitoring)"
Last update**Customer Impact** We aim for 99.9% platform uptime. From 15 to 29 September, uptime was reduced to 99.7% with some customers experiencing slow response times and/or limited access to the platform. **Status** Platform Operational. **Resolution** The root cause of the outages was related to Procurify’s database configuration and tuning, which links back to recent changes made by Amazon Web Services \(AWS\). We have worked closely with AWS and have identified ways to retain the necessary control to tune and maintain our databases independent of changes made by AWS. The database instances Procurify operates failover to standby replicas in case of availability issues. This process of failing over ensures high availability and happens occasionally. Our database provider, AWS, manages this failover process. During such a failover, the performance configurations of a production database did not transfer fully. This configuration change caused the database to experience performance bottlenecks that resulted in an outage. We were able to identify the root cause and fix the configuration issue to resume normal operations. Furthermore, we have found a solution to control and apply necessary configuration changes across database failovers without depending on AWS. This change helps with site reliability despite the infrastructure changes in the background required to ensure high availability. **Going Forward** We are investing in broader auditing and enhancements to our platform in order to be more resilient to the dynamic workloads from our customers. We have a strong team today, and we are adding senior talent specializing in database management to deepen our team's strengths. **Contact Information** If you have any questions or concerns, please reach out to our friendly and knowledgeable team of Procurify experts at [support@procurify.com](mailto:support@procurify.com).
This incident has been resolved.
Scheduled maintenance complete, team will continue to monitor site stability.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating degraded site performance.
Report: "Login errors or timeouts"
Last updateThis incident has been resolved.
Our authentication provider has reported that the login system performance has returned to normal. We will continue to monitor and provide updates.
Our authentication provider is working towards a resolution of the issue. We will continue to monitor and provide updates.
We have identified an intermittent issue where users may experience login error messages or login timeouts. Our authentication provider has identified the issue and is working on a resolution.