Is YOOBIC Down Right Now? Check if there is a current outage ongoing.

The incident has been resolved. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team.

The service is back to normal and we are actively monitoring the results.

We are currently experiencing degraded performance with media & files upload, which may result in slower response times. Our engineering team is actively investigating the issue and working to improve the performance as quickly as possible. While the degraded performance may impact your experience, rest assured that we are prioritizing the resolution to provide a smoother and more reliable service. Our team is committed to restoring normal performance and ensuring a seamless user experience. We will provide regular updates on this page to keep you informed about the progress of the incident. Please bear with us as we work diligently to address the performance issues and deliver the level of service you expect.

The incident has been resolved. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team.

The service is back to normal and we are actively monitoring the results.

We are currently experiencing degraded performance with media & files upload, which may result in slower response times. Our engineering team is actively investigating the issue and working to improve the performance as quickly as possible.While the degraded performance may impact your experience, rest assured that we are prioritizing the resolution to provide a smoother and more reliable service. Our team is committed to restoring normal performance and ensuring a seamless user experience.We will provide regular updates on this page to keep you informed about the progress of the incident. Please bear with us as we work diligently to address the performance issues and deliver the level of service you expect.

The incident has been fully resolved and webhooks performance is now back to normal. We understand the impact this incident may have had on your operations, and we apologize for any inconvenience caused. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team. We are here to assist you and ensure a smooth experience moving forward.

We are currently experiencing degraded performance with our webhook services, which may result in slower response times. Our engineering team is actively investigating the issue and working to improve the performance as quickly as possible. While the degraded performance may impact some of your integrations or automations that rely on webhooks, rest assured that we are prioritizing the resolution to provide a smoother and more reliable service. Our team is committed to restoring normal performance and ensuring a seamless user experience. Please note that the application and the general end-user experience is not impacted by this incident. We will provide regular updates on this page to keep you informed about the progress of the incident. Please bear with us as we work diligently to address the performance issues and deliver the level of service you expect. Thank you for your understanding and cooperation. We appreciate your continued support as we work towards resolving the degraded performance and enhancing your overall experience.

We experienced a partial outage that affected the recurring campaigns feature. As a result, some campaigns were not published as expected at 8:00 AM UTC. Our engineering team identified the issue and deployed a fix to resolve it. Impacted campaigns were successfully published at 4:30 PM UTC. We understand the inconvenience this may have caused and sincerely apologize for the disruption. If you require further assistance or have any questions, please do not hesitate to reach out to our support team. Thank you for your understanding.

The incident has been fully resolved and you should be able to use our services. We understand the impact this incident may have had on your operations, and we apologize for any inconvenience caused. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team. We are here to assist you and ensure a smooth experience moving forward.

We are currently experiencing a partial outage that is affecting the platform. As a result, users may experience limited or intermittent access to the YOOBIC application. Our engineering team is actively investigating the issue and working to restore full-service availability. We understand the impact this may have on your workflow and assure you that we are making every effort to restore full functionality as quickly as possible. We will continue to provide regular updates on this page to keep you informed about the progress of the incident. Our team is committed to resolving this issue and minimizing any impact on your operations. Thank you for your understanding and cooperation during this time. We appreciate your continued support as we work towards a resolution.

The incident has been fully resolved and you should be able to use our services. We understand the impact this incident may have had on your operations, and we apologize for any inconvenience caused. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team. We are here to assist you and ensure a smooth experience moving forward.

We are actively investigating the issue you may be experiencing and will provide updates as soon as possible. We appreciate your patience and understanding during this process.

We are currently experiencing a partial outage that is affecting email notifications. As a result, users may experience limited or intermittent access to these features. Our engineering team is actively investigating the issue and working to restore full-service availability. We understand the impact this may have on your workflow and assure you that we are making every effort to restore full functionality as quickly as possible. We will continue to provide regular updates on this page to keep you informed about the progress of the incident. Our team is committed to resolving this issue and minimizing any impact on your operations. Thank you for your understanding and cooperation during this time. We appreciate your continued support as we work towards a resolution.

This incident has been resolved.

We are continuing to investigate this issue.

We are currently experiencing degraded performance with our services, which may result in slower response times. Our engineering team is actively investigating the issue and working to improve the performance as quickly as possible. While the degraded performance may impact your experience, rest assured that we are prioritizing the resolution to provide a smoother and more reliable service. Our team is committed to restoring normal performance and ensuring a seamless user experience. We will provide regular updates on this page to keep you informed about the progress of the incident. Please bear with us as we work diligently to address the performance issues and deliver the level of service you expect. Thank you for your understanding and cooperation. We appreciate your continued support as we work towards resolving the degraded performance and enhancing your overall experience.

The incident has been fully resolved and you should be able to use our services. We understand the impact this incident may have had on your operations, and we apologize for any inconvenience caused. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team. We are here to assist you and ensure a smooth experience moving forward.

Our engineering team has deployed a fix to resolve the issue and is actively monitoring the results. The services are now back to fully operational status. Further updates will be provided as our team works towards restoring normal service.

Our engineering team is still actively looking into the ongoing incident which is impacting our services. The following services are affected: - Mobile application - Web application - Public API - Scheduler We understand the impact this has on your experience and want to assure you that resolving the issue is our top priority. We appreciate your patience and support during this time. Further updates will be provided as our team works towards restoring normal service.

We are currently experiencing degraded performance with our services, which may result in slower response times. Our engineering team is actively investigating the issue and working to improve the performance as quickly as possible. While the degraded performance may impact your experience, rest assured that we are prioritizing the resolution to provide a smoother and more reliable service. Our team is committed to restoring normal performance and ensuring a seamless user experience. We will provide regular updates on this page to keep you informed about the progress of the incident. Please bear with us as we work diligently to address the performance issues and deliver the level of service you expect. Thank you for your understanding and cooperation. We appreciate your continued support as we work towards resolving the degraded performance and enhancing your overall experience.

The incident has been fully resolved and you should be able to use our services. We understand the impact this incident may have had on your operations, and we apologize for any inconvenience caused. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team. We are here to assist you and ensure a smooth experience moving forward.

Our engineering team has deployed a fix to resolve the issue and is actively monitoring the results. The services are now back to fully operational status.

Our engineering team has deployed a fix to resolve the issue and is actively monitoring the results. The services are now back to fully operational status. Further updates will be provided as our team works towards restoring normal service.

Our file display service is currently experiencing an outage due to an issue with our provider's API access. We are actively working with them to resolve this issue and anticipate full resolution within the next hour. We apologize for any inconvenience this may cause.

Our engineering team has identified the cause of the ongoing issue and is currently implementing a fix. We appreciate your patience as we work towards resolving the situation. Further updates will be provided as our team works towards restoring normal service.

We are currently experiencing a partial outage that is affecting files viewer and files upload. As a result, users may experience limited or intermittent access to these features. Our engineering team is actively investigating the issue and working to restore full-service availability. We understand the impact this may have on your workflow and assure you that we are making every effort to restore full functionality as quickly as possible. We will continue to provide regular updates on this page to keep you informed about the progress of the incident. Our team is committed to resolving this issue and minimizing any impact on your operations. Thank you for your understanding and cooperation during this time. We appreciate your continued support as we work towards a resolution.

## Summary On 24 July 2023, our application experienced degraded performance with slower response times for our customers. The incident was caused by a higher server CPU usage, due to a scheduled task calculating aggregation metrics. This incident report provides a detailed account of the timeline, resolution, root cause analysis, impact, corrective actions, and conclusion related to the outage. ## Timeline Events that happened on the 24th of July, 2023: * 07:40am \(UTC\): Our server monitoring system detected higher CPU usage and increased response times. * 07:50am: The issue was escalated to the CTO, who initiated a crisis conference call with the Head of Backend and other Backend engineers for investigation. * 08:00am: The initial investigation focused on determining if the issue was related to a scheduled task, impacting the load on the database, utilizing analytics tools. * 08:05am: The team decided to kill the scheduled task. * 08:10am: The load stopped and the performance progressively returned to normal * 08:15am: The performance was back to normal, and the outage was resolved. ## Root cause analysis The root cause of the issue was traced back to a scheduled task impacting MongoDB database load, and causing the outage. ## Impact During the outage, our application experienced higher response times for some customers, including invalidation of their token forcing a logout, because the token could not be refreshed. It is probable that the same scheduled task played a role in the outage of the 17th July and we are actively monitoring that task in the coming weeks. ## Corrective actions To prevent similar incidents in the future, we have implemented the following corrective actions: * Dispatch the load of the scheduled task on the secondary nodes of MongoDB. * Adding new indexes to ease the workload of the scheduled task. * Evaluating the benefit of adding dedicated secondary nodes to MongoDB for this type of calculation unloading the workload from the primary nodes. ## Conclusion We apologize for the inconvenience caused by the outage. Our team worked diligently to identify and resolve the issue promptly. We are taking the necessary steps to prevent similar incidents in the future and improve the overall stability and reliability of our application. We understand the critical nature of our services for frontline workers, and we are committed to continuously improving our infrastructure and processes to ensure the best possible experience for our valued customers. If you have any further questions or concerns, please don't hesitate to contact our support team. Thank you for your understanding.

This incident has been resolved.

Our engineering team has identified the cause of the ongoing issue and is currently implementing a fix. We appreciate your patience as we work towards resolving the situation. Further updates will be provided as our team works towards restoring normal service.

Our engineering team is still actively looking into the ongoing incident which is impacting our services. The following services are affected: - Mobile application - Web application - Public API - Scheduler - Integrations centre We understand the impact this has on your experience and want to assure you that resolving the issue is our top priority. We appreciate your patience and support during this time. Further updates will be provided as our team works towards restoring normal service.

We are currently experiencing degraded performance with our services, which may result in slower response times. Our engineering team is actively investigating the issue and working to improve the performance as quickly as possible. While the degraded performance may impact your experience, rest assured that we are prioritizing the resolution to provide a smoother and more reliable service. Our team is committed to restoring normal performance and ensuring a seamless user experience. We will provide regular updates on this page to keep you informed about the progress of the incident. Please bear with us as we work diligently to address the performance issues and deliver the level of service you expect. Thank you for your understanding and cooperation. We appreciate your continued support as we work towards resolving the degraded performance and enhancing your overall experience.

# Incident report 17 July 2023 ## Summary On 17 July 2023, our SaaS application experienced an outage resulting in high response times for our customers. The incident was caused by a query plan rebuilt, triggered by a new index created on the mission collection in the previous days. This incident report provides a detailed account of the timeline, resolution, root cause analysis, impact, corrective actions, and conclusion related to the outage. ## Timeline Events that happened on the 17th of July, 2023: * 08:05am \(UTC\): Our server monitoring system detected higher CPU usage and increased response times. * 08:15am: The issue was escalated to the CTO, who initiated a crisis conference call with the Head of Backend and other Backend engineers for investigation. * 08:20am: Initial investigation focused on determining if the issue was related to MongoDB or Heroku, utilizing analytics tools. * 08:25am: It was confirmed that the problem was on the MongoDB side, and further analysis revealed the new index on the mission collection as the root cause. * 08:28am: The team began rebuilding the query plan on the server side. * 08:30am: The performance progressively returned to normal, and the outage was resolved. ## Root cause analysis The outage was caused by the unexpected rebuilt of the query plan, triggered by the implementation of a new index on the mission collection designed to improve the performance of specific queries on the mission collection. This rebuilt was specific to the production environment context and was not triggered during previous tests in staging environments. The outage was resolved by forcing ramp up of queries using the index and rebuilding the query plan on the server side. These actions were taken promptly to restore normal performance. ## Impact During the outage, our SaaS application experienced high response times, with the 95th percentile exceeding 20 seconds. ## Corrective actions To prevent similar incidents in the future, the following corrective actions have been taken: * A thorough review process will be implemented for any index creation or modification, ensuring ramp up query plan building right after deployment. ## Conclusion We apologize for the inconvenience caused by the outage and resulting in higher response times. The incident was promptly addressed by our engineering team, who successfully identified the root cause and implemented corrective actions to restore normal performance. We remain committed to delivering a reliable and efficient SaaS application for our frontline worker customers and will continue to take proactive measures to prevent future occurrences.

The incident has been fully resolved and you should be able to use our services. We understand the impact this incident may have had on your operations, and we apologize for any inconvenience caused. If you have any lingering concerns or encounter any unexpected issues, please do not hesitate to contact our support team. We are here to assist you and ensure a smooth experience moving forward.

Our engineering team is still actively looking into the ongoing incident which is impacting our services. The following services are affected: - Mobile application - Web application - Public API - Integrations centre We understand the impact this has on your experience and want to assure you that resolving the issue is our top priority. We appreciate your patience and support during this time. Further updates will be provided as our team works towards restoring normal service.

We are actively investigating the issue you may be experiencing and will provide updates as soon as possible. We appreciate your patience and understanding during this process.

We are currently experiencing a major outage that has resulted in the unavailability of our services. During this outage, users may be unable to access our application/service. Our engineering team is actively investigating the issue and working to restore full functionality as quickly as possible. We understand the impact this may have on your workflow and assure you that we are making every effort to restore full functionality as quickly as possible. We are committed to keeping you informed about the progress of the incident. Regular updates will be provided on this page as our team works towards a resolution. We appreciate your patience and understanding during this challenging time. Thank you for your support as we work diligently to restore normal service and minimize any disruptions to your workflow.