Historical record of incidents for Domotz
Report: "Login Failures in EU/APAC/US Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Login Failures in EU/APAC/US Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Login Failures in EU/APAC Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Login Failures in EU/APAC Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Backend infrastructure maintenance - European Data Center"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be undergoing scheduled maintenance during this time.
Report: "Login Failures in EU/APAC Region"
Last updateFollowing a comprehensive investigation, we have fully resolved the incident. The root cause was traced to a lag in our RDBMS, which in turn caused unpredictable performance issues across several system components. Our engineering team has rectified the replication delay, restored normal functionality, and conducted thorough validation checks. We will continue to monitor system performance closely to ensure sustained stability and prevent any recurrence.
We have deployed a corrective update and are closely monitoring its performance to ensure that the issue is fully resolved. Thank you for your patience as we validate the results and maintain system stability.
We have successfully identified the root cause of the issue and our development team is currently in the process of deploying a comprehensive solution. We anticipate that the fix will be implemented shortly and appreciate your continued patience as we work to fully resolve the matter.
We’re currently experiencing login issues in our US data center, which may prevent users from accessing the application or signing into their accounts. Users in the EU and APAC regions might also see delays or be unable to log in. Our team is actively investigating the root cause and will share updates as soon as we have more information. We apologize for the inconvenience and appreciate your patience. Thank you for your understanding.
Report: "Event Processing Delays"
Last updateThis incident has been resolved.
The issue has been identified and a fix is in place. We are now monitoring the situation.
We are currently experiencing delays in processing events from Collectors and Devices. Our team is actively working on a fix to resolve the issue as quickly as possible. We appreciate your patience and will provide further updates as soon as they are available.
Report: "Performance Degradation on the Domotz Portal – North America"
Last updateThis incident has been resolved. The reported slowness has been identified and eliminated, and Custom Scripts have been re-enabled on the affected devices. We thank you for your patience during this time.
A remediation has been applied, leading to a notable improvement in both App and WebApp performance. Our engineering team is actively monitoring system behavior and continues to address residual issues related to metrics retrieval—specifically historical data access via Custom Scripts and SNMP polling. We appreciate your patience as we work toward full resolution and will provide further technical updates as developments occur.
We're currently experiencing performance issues on the platform. Users may encounter slow loading times when accessing the Collector device list in the UI, as well as delays in processing historical data related to device status and metrics collection. Some integration scripts may also appear as "disabled." Our team has identified the root cause and is actively working on a fix. We're closely monitoring the situation and will share updates as progress is made.
Report: "Slowness on the Domotz Portal - North America"
Last updateThe issue has been fully resolved. All previously reported slowness has been eliminated, and functionality related to Custom Scripts and SNMP activities has been fully restored. We thank you for your patience during this time.
A fix has been implemented, and the App/WebApp performance has significantly improved. Our team continues to monitor the platform closely, as we are still addressing potential issues affecting the retrieval of metrics (and historical values) through Custom Scripts and SNMP. We appreciate your patience and will provide additional updates as the situation evolves.
We are currently experiencing performance issues on the platform. Users may notice slowness when loading the Collector device list in the UI, along with delays in processing historical data for device status and metrics collection. Our team has identified the cause and is actively implementing a fix. We are monitoring the situation closely and will provide further updates as the fix progresses.
Report: "Outage of remote connection feature in the US Cell"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Slowness in Portal/Webapp Loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some users may experience increased loading times or intermittent delays when accessing the Portal or Webapp.
Report: "Outage of remote connection feature in the US Cell"
Last updateThis incident has been resolved.
A fix has been implemented, and we are actively monitoring the results. Remote connections for all US EAST collectors should now be functioning as expected.
The issue persists and it is still affecting the EAST servers specifically. As a temporary workaround, moving the collector to a different region — for example, CALIFORNIA — allows the remote connection to work as expected (WEST servers are operational). We're continuing to investigate the root cause on the EAST side. Change Location: Through the Mobile App/WebApp/Desktop App select the Collector --> go to Collector Settings (Gear Icon) --> Edit Location: --> Move it to California - USA.
The issue persisted, and we are currently investigating it further.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Collectors Showing an Orange Bar in the US Cell"
Last updateI am pleased to inform you that the issue regarding the orange banner has been successfully resolved. Rest assured, the incident was isolated to the incorrect message alert displayed, and it did not impact the normal operations of Domotz in any way.
We are currently investigating this issue.
Report: "Login failures in the US region"
Last updateWe have successfully implemented a fix for the issue. After thorough monitoring, we can confirm that the incident has been fully resolved. All systems are now operating normally.
The problem has been recognized and is affecting a minority of our user base. A fix have been implemented and we are monitoring the results.
We are currently experiencing issues with logins in our US datacenter. Users may encounter difficulties logging into the application or accessing their accounts. Our team is actively investigating the root cause of the issue. We will provide an update as soon as more information becomes available. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue. Thank you for your understanding.
Report: "Login failures in the US region"
Last updateThe issue affecting the US region has been successfully resolved. Users can now log in without any problems, and all Domotz functionalities are operating as expected. A Root Cause Analysis (RCA) will follow after a full review. We will provide a comprehensive report detailing the incident, its resolution, and steps taken to prevent future occurrences.
We have successfully restored all system functionality, and now users can log in. We are currently conducting a resync to ensure that all data is up-to-date and accurate.
Efforts are underway to restore data and ensure the system is functioning as expected. We are prioritizing data integrity and system stability as part of this process. Further updates will be provided as we make progress.
We have identified the issue as related to AWS OpenSearch Managed Services. Our team, in collaboration with AWS, is currently applying a resolution. Further updates will be provided as the situation progresses. Thank you for your patience.
**Description:** We are currently experiencing issues with logins in our US datacenter. Users may encounter difficulties logging into the application or accessing their accounts. **Impact:** - Users in the US region may be unable to log in or may experience delays in accessing the application. **Next Update:** Our team is actively investigating the root cause of the issue. We will provide an update as soon as more information becomes available. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue. Thank you for your understanding.
Report: "Webapp Performance Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "New Collectors Activation in the EU Datacenter"
Last updateWe are pleased to confirm that the incident has been effectively resolved. Our system is currently functioning at maximum capacity, with all services operating smoothly.
We are pleased to announce that all system functionality has been successfully restored. Users can now activate new collectors without any issues. Our team is actively monitoring the situation to detect any side effects. If all remains stable, we plan to close the incident soon. We appreciate your patience and understanding during this time. Thank you for your continued support.
We are currently experiencing issues with the activation of new collectors in our EU datacenter.
Report: "Webapp Data availability in US region"
Last updateThis incident has been resolved.
A fix has been implemented, and we are currently monitoring the results. The issue was limited to the topology map rendering and, in some cases, the Interfaces tab of devices.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Webapp Data availability in EU region"
Last updateThe incident in our European datacenter has been successfully resolved. The network topology visualization and device status check functionalities are now operating as expected. A Root Cause Analysis (RCA) will follow after a full review. We will provide a comprehensive report detailing the incident, its resolution, and steps taken to prevent future occurrences.
We are continuing to monitor for any further issues. We will keep you posted on a resolution.
We have performed and emergency restart for the backend components responsible for the affected functionalities and we recovered the data availability for the EU data center. Services are recovering.
Our team has identified the root cause of the issue and is actively working towards a resolution. We understand the impact this may have on your workflow and are prioritizing the restoration of full service functionality.
We are currently experiencing an issue within our European datacenter that is affecting specific functionalities of our application, notably the network topology visualization and device status check. This disruption may be causing these parts of the app to not work as expected.
We are currently investigating this issue.
Report: "Backend Emergency infrastructure maintenance - US Region"
Last updateThe maintenance activity has been successfully completed, and the service is now fully operational.
Emergency Maintenance Activity Notification Affected Services: User Logins in the US region. Description: We are currently conducting emergency maintenance activities. This maintenance is essential to ensure the stability and performance of our services. Impact: During this maintenance window, users may not be able to login to the Domotz Portal. We apologize for any inconvenience this may cause and appreciate your understanding as we work to resolve the issue promptly. Estimated Duration: 30 minutes We will provide updates on the progress and inform you once the maintenance is complete. Thank you for your patience and cooperation.
Report: "Login failures - Webapp"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Some collectors have been reported offline"
Last updateThis incident has been resolved.
Everything is up and Running. An additional recovery procedure is in progress to improve data consistency
A fix has been implemented and we are monitoring the results.
We are still observing some collectors fluctuating between being online and offline. Our team is conducting a deeper investigation to determine the root cause.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Login failures"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Login failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Users experiencing issue in logging into the Domotz service"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Users experiencing issue in logging into the Domotz service"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Performance Issues in US Data Center"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Customers may experience missing Device Up/Down events
Report: "Some Windows Collectors/Agent have been reported offline"
Last updateThis incident has been resolved.
Most of the agents have successfully reconnected and are now operational. Our team continues to closely monitor the ongoing situation.
The most recent release of the Domotz Windows Collector\Agent has resulted in a subset of agents to stop running. This issue specifically impacts the Domotz Collector running on the MS Windows operating system. To restore the Domotz Collector's functionality, please follow these steps: - Uninstall the Domotz Agent - Visit https://portal.domotz.com/portal/domotz_software to obtain the latest installer. - Run the installer on the affected Agent host. - resume the current agent at http://ipaddressofthedomotzhost:3000 (remember to use http and not https)
Report: "Login falure in EU region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Users experiencing issue in logging into the Domotz service (North American)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Users experiencing issue in logging into the Domotz service (North American / Europe Data Center)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Performance Issues in North America"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Operational issue due to infrastructure partial failure - EU Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Users are not able to access to the Domotz Agents"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "General connectivity issue with our Domotz Cloud infrastructure"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented. There are still some delays and general slowness of the platform. We are monitoring the situation.
The issue has been identified and a fix is being implemented.
Issue affecting login and activating of new Domotz Agents. Our Engineering team is investigating in this issue.
Report: "Login falure in EU region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Users are not able to access to the Domotz Agents"
Last updateThis incident has been resolved.
The access for all the type of users has been restored. We are monitoring the situation to avoid any further issue.
The issue has been identified. Admin users as well as all the Team Member users should now have access to all their own Agents. Field Operators and Residential Customer Facing App users might still have issue accessing their own Agents (as well as Agents shared with different teams)
We are currently investigating this issue.
Report: "Users experiencing issue in logging into the Domotz service (North American Data Center)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate in this issue. Apparently there is a higher, and sustained incoming traffic and requests to our backend.
We are currently investigating this issue which has also an impact on the performance of users that are currently logged in.
Report: "Users experiencing issue in logging into the Domotz service (North American Data Center)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are restarting some backend components to resolve the issue we are experiencing.
The issue has been identified and a fix is being implemented.
We are experiencing an overload in the number of requests to our North America data-center. Users might experience failures in login into the service. Our team is investigating in this issue.
Report: "Performance Issues in Europe Data Center"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Remote Connections in North America - East coast"
Last updateWe have been experiencing an issue with the server hosting the Remote Connection capability for the North America - East based Agents and network. The Engineering team have identified the issue and resolved with the deployment of a new server for that functionality.
Report: "Remote Connection issues"
Last updateThis incident has been resolved.
We are experiencing an issue with the servers hosting the Remote Connection capabilities (VPN on Demand, TCP Connection and direct http/https Remote Connections). The Engineering team is investigating in this issue which is causing some of the remote connection sessions to fail.
Report: "Performance Issues in North America - delays expected"
Last updateThis incident has been resolved.
We have completed the increase in the size of the cloud services. Response time for loading the Domotz App/WebApp and data has decreased significantly. We will continue monitoring the situation.
The Engineering team is looking into this issue, and increasing the pool of services available to cope with the increased load. During this phase, users might experience some errors while loading the App/WebApp.
We are experiencing some delays on our North America datacenter. Users retrieving information through the Domotz App or WebApp are experiencing some significant delays.
Report: "Overload on the Cloud infrastructure - Service unavailable"
Last updateThis incident has been resolved. However, during the outage, some users might have received "false positive" notifications for Agents disconnections. As a matter of fact, due to the outage, the cloud service responsible for monitoring the Agent status (by receiving heartbeats from each Agent) was not able to cope with the incoming requests, and triggered false Alerts to the users. Domotz users are invited to discard those notifications occurred during the outage (between ~11:00 and ~11:30 UTC). Moreover, the status of the Agents falsely triggered offline during the outage might still be reported as offline. The team is re-synchronizing the status of all the Agents. We invite the users to contact support (support@domotz.com) in case of any further issue.
A fix has been implemented, so that the Cloud services are able to cope with the increased load. The Engineering team will keep monitoring the situation in order to make sure that a further increase in the load won't cause any further disruption.
The Engineering team has identified the issue. The scheduled maintenance activity for this morning, in conjunction with an increased number of connected users, caused an increased load of the cloud infrastructure. The team is increasing the capacity of the cloud services to cope with the increased load.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Users experiencing issue in logging into the Domotz service (North American Data Center)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Performance Issues in North America"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The Engineering team has identified the root cause of the issue. It is fixing the issue.
We are currently experiencing an issue on the Cloud infrastructure for our North America data-center. Users might experience issues in login to the platform.
Report: "Users experiencing issue in logging into the Domotz service (North American Data Center)"
Last updateThis incident has been resolved.
The service has been restored. The infrastructure team is monitoring the status of the service.
The issue has been identified and a fix is being implemented.
Due to periodic maintenance of the Domotz Cloud infrastructure, an unexpected issue in accessing the Domotz Agents has been recorded in our North America Data Center. The service will be restored as soon as possible.
Report: "Users experiencing issue in logging into the Domotz service (European Cell)"
Last updateDue to periodic maintenance of the Domotz Cloud infrastructure, login services were temporary affected (between 10:02 AM and 10:09 UTC). The infrastructure team is making sure that such issues do not occur in the future when a similar operation is required.
Report: "Users are experiencing issue in logging into the Domotz service (European Cell)"
Last updateThere has been an issue in the login process for the European datacenter between 10:55 AM and 11:05 AM. The issue, due to a temporary overload of the service, has been resolved.
Report: "Login issue due to overload of the system"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the root cause of the issue. The Engineering team is looking on a fast way to resolve it.
We are continuing investigating the issue
We are currently investigating this issue.
Report: "Login issue due to infrastructure partial failure - US Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "EU Data Center - Login issue due to overload of the system"
Last updateThis incident has been resolved. Our IT Infrastructure will keep an eye, in order to ensure the issue does not occur again.
The Engineering team has identified and resolved the root cause of the issue. We will continue to monitor the status of the system, due to some of the services still experiencing a sustained high load.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Login issue due to overload of the system"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A fix has been implemented. The Engineering Team will continue to monitor the platform in order to spot any further issue.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.