Historical record of incidents for UXCam
Report: "Degraded UXCam dashboard performance"
Last updateAt 08:45 UTC, we observed that certain components of the dashboard were not loading as expected. Upon investigation, we identified that one of the database servers was operating at peak capacity, leading to performance degradation. To mitigate the issue, we immediately took the necessary actions to manage the load and restore normal operations. As a result, users experienced degraded performance for approximately 30 minutes. By implementing appropriate measures, we successfully stabilized the database and fully restored the dashboard functionality. We will conduct a post-incident review to improve resilience.
Report: "Delay in session processing"
Last updateThis incident has been resolved.
The fix has been successfully deployed, and sessions are now processing at the normal rate. However, it will take approximately 20 to 25 minutes for the backlog of sessions to be fully processed. We are closely monitoring the situation and will provide updates as needed.
We are continuing to work on a fix for this issue.
We are currently experiencing delays in session processing. However, no sessions have been lost, and all data remains intact.
Report: "Outage in session replay"
Last updateThis incident has been resolved.
We are facing issues with the session replay due to a problem with our Cloud service provider. No sessions will be lost due to this issue.
Report: "Delay in session-processing"
Last updateThis incident has been resolved.
We are facing some delays with the session processing however no sessions are lost.
Report: "Outage in UXCam Dashboard"
Last updateThe incident has been successfully addressed. To summarize, while carrying out routine server optimization, unexpected networking issues arose, leading to the problem. We have pinpointed the root cause and are implementing measures to prevent similar occurrences in the future. We appreciate your patience and understanding throughout this process.
We have applied the fix and the dashboard has been restored to normal working condition. We are actively monitoring.
The issue has been identified and are in the process of applying necessary fixes. We will keep you posted.
We are currently facing issue with the UXCam Dashboard (https://app.uxcam.com). Our team is currently investigating.
Report: "Dashboards are not working"
Last updateThis issue has been resolved. All the dashboards are functioning as expected.
We are currently experiencing issues with the customizable dashboard and funnels. The team is working to identify and resolve the issue.
Report: "Outage in UXCam Dashboard and Verification service"
Last updateOn December 20, 2023, our routine upgrade unexpectedly caused an outage from 5:35 pm to 5:40 pm (UTC), rendering the UXCam Dashboard and Verification service temporarily unavailable. Our team promptly identified the issue and successfully restored the system. Sessions that occurred during the outage are being uploaded subsequently. We apologize for any inconvenience caused by this disruption.
Report: "We are facing session processing delay at the moment"
Last updateAll the delayed sessions have been processed. Everything is now in a normal operational state.
The delayed sessions have started to get processed. We are monitoring the situation as it gradually returns to its usual operational state.
A partial AWS outage caused this. We don't expect any loss of sessions due to this incident.
We are facing a session processing delay and we are currently investigating the issue.
Report: "Partial outage in UXCam dashboard"
Last updateThis incident has been resolved.
The issue has been addressed, and a fix has been successfully implemented. As a result, the dashboard has been restored to its normal state. We are currently monitoring the situation to ensure everything remains stable.
We are actively working on the solution, and progress has been made in restoring the dashboard to a partial state. However, some issues still remain. Our team is dedicated to resolving these problems and returning the dashboard to its normal functioning state.
We have identified the issue with one of our data infrastructures. Our team is already working on the fix.
We are currently investing an issue with the UXCam dashboard (app.uxcam.com) not serving requests.
Report: "Outage/Degraded performance issue in UXCam Dashboard"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating a degraded performance issue with the dashboard frontend (app.uxcam.com).
Report: "Outage in UXCam Dashboard"
Last updateOn Jun 30, 2023, during a routine deployment, we faced an unexpected outage that led to the UXCam Dashboard being unavailable from 12:10 pm to 12:21 pm (UTC). We identified the cause and implemented the fix, and were able to recover the system. We apologize for any inconvenience that this has caused.
Report: "Partial outage in session processing"
Last updateOur system has recovered and is operating normally.
The number of sessions in the queue waiting to get processed has started to decrease. We expected around 30 mins for the system to get back to normal. We will keep you posted.
Our system is still catching up to the high traffic and running at maximum capacity to process the incoming sessions. We will update you with the latest status in about 15 mins.
During an infrastructure migration process, we faced an unexpected downtime of around 15 mins. This impacted the upload of sessions from the devices. The change has been reverted and the system is operating in a normal state. The sessions that failed to upload during the time of the issue have started to upload, and due to the burst of traffic, we are facing delays in sessions getting processed. Our system is catching up. We will keep you all posted.
Report: "Delay in sessions to appear in the UXCam Dashboard"
Last updateThe issue with partial delays for data to appear in the dashboard has been resolved. All systems are back to normal.
We are facing partial delays for recorded user data to appear on the dashboard. No data has been lost, and our system is catching up in processing the latest incoming sessions.
The issue has been identified and a fix is being implemented. Delay has already been resolved for parts of the dashboard.
We are observing delays in sessions appearing on the dashboard. The team is already on it. From initial investigation, we don't think there has been any loss in the data.
Report: "Outage in UXCam Dashboard"
Last updateWe faced issues with Query Engine, which impacted our application dashboard. This issue was caused by the AWS cloud service not being able to provide the compute instances to lunch and scale-up our query engine.
Report: "Outage in UXCam Dashboard"
Last updateOn July 28, 2022, during a routine deployment, we faced an unexpected outage that led to the UXCam Dashboard being unavailable from 7:31 am to 8:20 am (UTC). We identified the cause and implemented the fix, and were able to recover the system. We apologize for any inconvenience that this has caused.
Report: "Delay in session processing"
Last updateThis incident is now resolved
We experienced high load in our system causing delay in session processing. We upgraded our system to handle the load, and the system is recovering. We are monitoring closely.
Report: "Downtime and degraded performance because of AWS issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Our system is now recovering and we are monitoring closely.
We are observing downtime and degraded performance in verify-service, session-processing, and dashboard because our cloud-provider AWS is having issues.
Report: "Outage in UXCam Dashboard"
Last updateOn May 17, 2022, during a routine deployment, we faced an unexpected outage that led to the UXCam Dashboard being unavailable from 8:17 am to 9:04 am (UTC). We identified the cause and implemented the fix, and were able to recover the system. We apologize for any inconvenience that this has caused.
Report: "Session Verification Outage"
Last updateOn March 31, 2022, while upgrading our infrastructure, some breaking changes were applied accidentally which lead to the failure in APIs for one of our services. The breaking change was quickly detected and reverted within a minute. Between 10:02 UTC to 10:04 UTC all the sessions sent were lost. We apologize for any inconvenience that this has caused.
Report: "Partial session recording malfunction"
Last updateOn April 13, 2022, we saw an elevated number of error responses in our APIs after the deployment of new changes which led to partial session loss from 09:24 UTC to 09:28 UTC. The change was immediately rolled back and the APIs were restored to the normal working state. The issue affected 10% percent of the total active sessions during the time window. We identified the bug that led to the issue and the fix was implemented. We apologize for any inconvenience that this has caused.
Report: "Session recording malfunction: Android devices"
Last updateSession recording on Android devices was malfunctioning between 2022/02/16 11:20 UTC and 2022/02/17 05:00 UTC. The exact scope of lost sessions is under investigation.
Report: "Delay in session processing"
Last updateThis incident has been resolved.
We continue to experience delays with processing sessions, however, no data has been lost. The system is recovering and we will continue with the updates.
We are facing delays with session processing due to the outage in the cloud infrastructure provider (AWS).
Report: "Session Processing and Dashboard are not working"
Last updateThis incident has been resolved.
The issue has been identified and fixed. We are monitoring on it.
We are currently investigating this issue.
Report: "Delay in Processing Sessions"
Last updateStarted time: 2021/09/11 20:30 Resolved time: 2021/09/12 12:25 We experienced an issue with a database which resulted in a delay with session processing services. This led to sessions appearing after some delay in the dashboard. The maximum delay was 7.39 hours.
Report: "Delay in processing Sessions"
Last updateThis incident has been resolved. Now all the sessions will be processed normally.
All the queued session has been processed and we are monitoring the server.
The issue has been identified and we are working on it.
We are currently investigating the issue.
Report: "All dashboard are not working"
Last updateThis incident has been resolved.
Everything is now working as intended. We are closely monitoring this issue.
Fix has been implemented and we are still working on the issue to make it work completely.
The issue has been identified and currently working for a fix.
Report: "Session Verification is not working"
Last updateThis incident has been resolved.
A fix has been implemented and running smoothly. We are closely monitoring.
A fix has been implemented and we are monitoring the issue.
The issue has been identified and a fix has been implemented.
We are currently investigating into this issue
Report: "This is an example incident"
Last updateWhen your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.
Empathize with those affected and let them know everything is operating as normal.
As you continue to work through the incident, update your customers frequently.
Let your users know once a fix is in place, and keep communication clear and precise.