Historical record of incidents for Canix
Report: "Metrc API Having Intermittent Issues"
Last updateThis incident has been resolved.
Metrc's API is having intermittent issues today with occasional server errors across several states. Metrc is investigating urgently, and some syncs from Metrc in Canix may take longer than normal until they resolve this issue.
Report: "Some GET calls to Canix's API failing with 422 error"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring for any further disruption.
The issue has been identified and we're actively working on a fix.
We are seeing elevated error rates with 422 errors returned on some GET calls to our Canix API and are actively investigating.
Report: "Delayed Metrc Syncing - Worldwide Tech Incident"
Last updateMonitoring of the situation is complete. This issue is now resolved.
Metrc has confirmed their issues have been resolved, and syncs in Canix have completed. We will continue to monitor. Thank you for your patience.
We're experiencing delayed Metrc syncs as a result of Microsoft's worldwide tech outage today, July 19. We are reprocessing delayed syncs and expect normal syncing to return shortly. Any activity done in Metrc directly since the last synced time on the Refresh Metrc Data button will not be reflected until that time. Canix activity and submissions will operate as normal during this time. We will send additional updates as soon as they become available.
Report: "Active package view not loading"
Last updateActive package view temporarily would not load. Issue is now resolved.
Report: "Menu links not showing properly for some users"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring for any further issues.
The issue has been identified and a fix is being implemented.
We are seeing issues with menu links not loading properly for some users and are actively investigating.
Report: "Slow loading/timeouts"
Last updateCertain plants and packages table views/actions are loading slowly or timing out for large volumes of data.
Report: "Elevated Errors for Package and Transfer Transactions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
IMPORTANT: Canix customers may experience elevated errors for package and transfer actions at this time. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible.
Report: "Metrc Data Refresh & API Issues"
Last updateThis issue is resolved and our systems are now operating normally.
A fix has been implemented to resolve the issue and we are continuing to monitor the situation.
We have deployed a fix and are now monitoring to ensure there are no residual effects.
Canix customers may experience trouble with completing a Metrc refresh sync. Our Engineering team is aware of the issue and is working to investigate communication with the Metrc API.
Report: "Metrc Data Refresh Incident"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Canix customers may experience trouble with completing a Metrc refresh sync. Our Engineering team is aware of the issue and is working to investigate communication with the Metrc API.
Report: "Auth0 Intermittent Issues"
Last updateAuth0 has confirmed the outage has been resolved. Their full report on the now resolved issue is detailed here: https://status.auth0.com/incidents/3fnx9qxpmp0f Canix customers should no longer experience issues logging into the platform.
A fix has been implemented by our login service provider.
The situation has been identified as an issue with our service provider for logging into Canix.
Our service provider for logging into Canix is having intermittent issues. You may experience an inability to log into Canix at this time. We are working to resolve this ASAP, you can follow their status page directly here: https://status.auth0.com. Canix will update our status page as well, please check here for any additional updates.
Report: "Intermittent Downtime"
Last updateCanix is functioning as expected and issue is resolved.
Canix customers may experience intermittent timeouts or errors on actions on the Web Application. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "Issues communicating with the Metrc API"
Last updateMetrc API is functioning properly with no known errors or slow response time.
Update: We are continuing to receive errors and slow responses from the Metrc API. We are working with Metrc support to reach a resolution, and will continue to update here and our in-app banner.
We're seeing an increased level of "METRC servers are temporarily unavailable" errors from Metrc's API. We've contacted Metrc's support and escalated this issue with them. We will continue to update this page once we hear and have more details from Metrc.
Report: "Issues communicating with the Metrc API"
Last updateMetrc's API has stabilized enough that our functionality has normalized. We will continue to monitor in case the error rate from Metrc's API increases again, but Canix is now currently operating normally.
Update: Metrc's API appears to be slowly stabilizing, and we'll continue to monitor for any further issues.
We're seeing an increased level of "METRC servers are temporarily unavailable" errors from Metrc's API. We've contacted Metrc's support and escalated this issue with them. We will continue to update this page once we hear and have more details from Metrc.
We are seeing increased errors returned from Metrc's API this morning. Some Metrc submissions and syncs may not complete successfully until Metrc resolves these errors. Please continue to monitor status.canix.com for additional updates.
Report: "Issues communicating with the Metrc API"
Last updateMetrc's API has stabilized enough that our functionality has normalized. We will continue to monitor in case the error rate from Metrc's API increases again, but Canix is now currently operating normally.
Update: Metrc's API appears to be slowly stabilizing, and we'll continue to monitor for any further issues.
Identified that the error is due to a current outage from Metrc. Error from METRC (8445033): METRC servers are temporarily unavailable. Please try again in a moment. We will continue to monitor the issue and update here.
We are continuing to investigate this issue.
Canix customers may experience trouble with intermittent timeouts on actions on the Web Application. Our Engineering team is aware of the issue and is working to investigate issues communicating with the Metrc API. Please continue to monitor status.canix.com for additional updates.
Report: "Heroku Outage Impacting Canix Users"
Last updateCanix is functioning as expected. Issue with Heroku appears resolved as there have been no lingering issues with Canix functionality.
Heroku's status page has not been updated (https://status.heroku.com/) but Canix is functioning as expected. Updating status to Monitoring and will update to Resolved once confirmed on Heroku's status page.
Heroku Outage: Canix customers may experience trouble logging in and intermittent timeouts on actions on the Web Application. This is due to an outage with Heroku, we will continue to monitor their status updates here: https://downdetector.com/status/heroku/ and on their status page here (when it is operational again): https://status.heroku.com/ Canix users will be updated on our status page with outage updates.
Report: "Heroku Outage Impacting Canix Users"
Last updateHeroku status is operational and Canix users should not have further interruption with the web application.
We are continuing to work on a fix for this issue.
The issue has been identified as an outage with Heroku and we will continue to monitor the situation and update Canix customers.
Heroku Outage: Canix customers may experience trouble logging in and intermittent timeouts on actions on the Web Application. This is due to an outage with Heroku, we will continue to monitor their status updates here: https://downdetector.com/status/heroku/ and on their status page here (when it is operational again): https://status.heroku.com/ Canix users will be updated on our status page and within the in-product banner with outage updates.
Report: "Cloudflare Services Outage Impacting Canix Users"
Last updateCloudflare services are operational.
The issue has been identified as a partial outage based on region with Cloudflare per their status page: https://www.cloudflarestatus.com/
Cloudflare Services Outage: Canix customers in certain regions may experience trouble logging in and intermittent timeouts on actions on the Web Application. This is due to an outage with Cloudflare, we will continue to monitor their status updates here: https://www.cloudflarestatus.com and update Canix users on our status page and within the in-product banner.
Report: "Cloudflare Services Outage Impacting Canix Users"
Last updateCloudflare status page (https://www.cloudflarestatus.com) has been updated to all systems operational and internal engineering team verified Canix users should no longer experience downtime due to this issue.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Cloudflare Services Outage: Canix customers in certain regions may experience trouble logging in and intermittent timeouts on actions on the Web Application. This is due to an outage with Cloudflare, we will continue to monitor their status updates here: https://www.cloudflarestatus.com and update Canix users on our status page and within the in-product banner.
Report: "Trouble Logging In and Intermittent Downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Canix customers may experience trouble logging in and intermittent timeouts on actions on the Web Application. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "AWS Outage - Canix Update"
Last updateAs you may have heard, Amazon Web Services \(AWS\) had an outage this morning that impacted multiple websites, one of which was Canix. The issue was unfortunately out of our control, but thankfully AWS was able to restore services and we were operational within a few hours. We apologize for any issues this may have caused, and thank you for your patience while Amazon resolved the matter.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Canix customers may experience trouble logging in and intermittent timeouts on actions on the Web Application. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "Trouble Logging In and Intermittent Downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Canix customers may experience trouble logging in and intermittent timeouts on actions on the Web Application. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "Trouble Logging In and Intermittent Downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Canix customers may experience trouble logging in and intermittent timeouts on actions on the Web Application. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "Partial outage"
Last updateSome users may have recently experienced degraded performance issues with logging in and various pages having display issues. Our team has currently restored normal performance levels. We apologize for any inconvenience.
Report: "Intermittent Downtime"
Last updateOur team has resolved this issue and all pages are loading as expected.
We are continuing to work on a fix for this issue.
Canix customers may experience intermittent timeouts on actions on both the Web Application and Mobile Application. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "Mobile Package Functionality Inaccessible"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
The packages function within the Canix mobile application is currently inaccessible. You can navigate to the menu however, the buttons are nonresponsive. We're investigating the cause and will provide an update as soon as possible.
Report: "Unable to Transmitting Weight from Scale to Canix Mobile Application"
Last updateThis incident has been resolved. Thank you for your continued patience.
Canix has identified an issue that is preventing harvesting data from a scale to our platform. You may need to manually input the weight until this issue is resolved. Our Engineering team is aware and we are working to get this issue resolved as quickly as possible.
Report: "Canix Loading Blank Screen"
Last updateOur Engineering identified an issue related to a recent product release that prevented users from being able to log into Canix. Customers logging in were presented with a white screen. This incident affected our web application between the hours of 1:02 pm - 1:39 pm PST.
Report: "Unable to Change Prices on Sales Orders"
Last updateUsers can now edit Sales Orders as expected. This issue is resolved.
We've identified an issue that is preventing users from being able to update the line item subtotals on Sales Orders. Our Engineering Team is hard at work resolving this issue. We will send additional updates as soon as they become available.
Report: "Manufacturing pages may be slow loading"
Last updateWe identified an issue that was causing our Manufacturing modules to load slowly and cause timeouts. This issue has since been resolved. If you are still experiencing slow load times we suggest clearing your browser's history, cookies, and cache before reloading the page.
Report: "Trouble logging in"
Last updateThis issue is mitigated. From our understanding, it was a result of an error on our backend host. We have reached out to this service to assist us in doing a full investigation to ensure this does not happen again. You now have access to all Canix services. You may need to clear your browser history, cache, and cookies to ensure the pages load successfully.
Some users may be experiencing trouble when logging in to Canix. Our Engineering team is currently investigating. We will send additional updates as soon as they become available.
Report: "Intermittent Downtime "Error 525""
Last updateThank you for your continued patience. Our team was able to resolve the issue and all pages are loading as expected.
Canix is experiencing intermittent downtime. You may see the error "Error (525) SSL handshake failed" when loading certain pages. Our Engineering team is aware of the issue and is working to resolve it as quickly as possible. Please continue to monitor status.canix.com for additional updates.
Report: "Elevated Metrc API Errors"
Last updateWe have been notified that this issue is now mitigated. All Metrc submissions are processing as normal.
Metrc is experiencing significant delays. If you received this message while making submissions in Canix please do not attempt to resubmit. Our platform is built to manage these types of situations. Canix will re-try any submissions made during the Metrc downtime until they go through successfully. You may continue working in Canix as normal.
Report: "Pages may be slow loading"
Last updateOur Engineering team identified the source of the slowness, it was a result of performance and traffic issues. This incident has been resolved.
Our service provider is currently experiencing a slowdown and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible.
Report: "Degraded Performance"
Last updateUsers reported the inability to switch between licenses or select item types due to a change made to dropdown menus. This change was rolled back and the issue was resolved.
Report: "Sales Order Loading Error"
Last updateWe identified an issue that was resulting in a "Loading error" on the Sales Order page. It was a result of the backend not deploying correctly due to an unexpected blocker after we added a test coverage check. This issue was identified at 10:38 am PT and mitigated at 10:43 am PT.
Report: "Canix Web Application intermittently latency"
Last updateInventory Lookback was affected by over-fetching, which resulted in intermittent latency (or slowness). This issue was resolved as of 8:00 am PST. If you were experiencing any slowness this morning, this was likely the cause. Again, it has been resolved.
Report: "This is an example incident"
Last updateWhen your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.
Empathize with those affected and let them know everything is operating as normal.
As you continue to work through the incident, update your customers frequently.
Let your users know once a fix is in place, and keep communication clear and precise.