Is Merinio Down Right Now? Check if there is a current outage ongoing.

During our recent efforts to enhance the performance of the application, a piece of profiling code intended to diagnose issues was unintentionally deployed to production. This resulted in users being unable to edit shifts or punch in using the punch terminal overnight from 10:15PM to 7:45AM. The issue has been identified and resolved, and full functionality has been restored. We apologize for the inconvenience this caused and are taking steps to strengthen our deployment process to prevent similar issues in the future. Thank you for your patience and understanding.

**Background:** On the afternoon of February 19th, 2021, our main API servers began exhibiting significantly elevated error rates coupled with extreme latency. This issue caused widespread unresponsiveness across our infrastructure for about half an hour, severely impacting user experience by preventing app loading and creating substantial delays in page transitions. **Root Cause Analysis:** The initial remedy involved expanding the capacity of our production cluster and rebooting several key services. A thorough investigation of the logs revealed that the request load on our systems had surged by approximately 3000%. This dramatic increase led to escalating latency, culminating in widespread request timeouts. The surge was traced back to a recent modification in the bulk edit tool within Merinio 2.0. This change inadvertently caused the entire user list to refresh on all logged-in devices for each user edit, as opposed to the previous setup where the list refreshed only upon page changes post-resource modification. **Immediate Response:** To mitigate the immediate impact, we have temporarily disabled real-time updates for changes executed through the bulk edit tool. Our team is actively developing a more optimized solution to handle such scenarios efficiently. **Impact:** This incident significantly disrupted operations for our users, leading to a degraded experience. We deeply regret the inconvenience and interruption caused to our users during this period.

This incident has been resolved, we will continue to monitor the situation closely.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue where Merinio responds intermittently and sometimes does not load.

**Root Cause Analysis:** The core issue was traced back to an overlooked renewal of the SSL certificate. Essential renewal notifications were inadvertently directed to our spam folder, resulting in a missed update of the certificate. This oversight led to the service interruption experienced by our users. **Impact:** During the disruption, users encountered difficulties in accessing our services. We acknowledge the inconvenience caused by this incident and deeply regret the impact on our users' experience. **Resolution and Prevention:** Upon identification of the root cause, we have implemented a shift to DNS validation for future SSL certificate renewals. This change introduces a fully automated process, significantly reducing the likelihood of similar incidents in the future.

At approximately 7:00 PM today, we experienced an unexpected disruption in our services due to an expired SSL certificate. Our monitoring systems alerted us to the issue, and our team immediately began investigating. The issue was identified as a missed renewal of the SSL certificate. The renewal notification was mistakenly routed to our spam folder, leading to an oversight in the certificate's renewal. During this period, users may have experienced difficulty accessing our services. We understand the inconvenience this caused and are taking steps to prevent a recurrence.

The incident has been resolved. We have temporarily increased capacity to ensure smooth operation for the remainder of the weekend until we can get a more permanent fix in place.

A fix has been implemented and we are monitoring the results.

The issue has been identified and we are working on a temporary mitigation.

We are observing issues with the web application responsiveness and availability.

This incident has been resolved.

We isolated the issue to time off and attendance reports taking longer than expected, we have implemented a temporary date limit to these exports while we continue to work on a long term solution.

The issue has been identified and a fix is being implemented.

We have noticed that a subset of requests have started failing. We are currently investigating.

This incident has been resolved.

An issue causing the application to hang when changing filters has been discovered. If the application still hangs, please close the tab and try again.

This incident has been resolved.

We are continuing to work on a fix for this issue.

The issue has been identified, a bulk creation tool was causing a much higher than normal load on the servers. We are temporarily disabling the feature as we work on a more robust solution.

There currently seems to be an issue with increased latency on all connected systems. We are investigating the issue.

This incident has been resolved.

A bad deploy caused the web application to load infinitely when refreshing the app. A fixed version of the application has been deployed and we are monitoring to make sure that the issue has been resolved.

We are currently investigating this issue.

This incident has been resolved.

A rogue machine was causing our servers to fail their health checks, which was causing sporadic 502 and 404 errors. The machine has been terminated and the servers now appear to be passing their health checks. We will continue to monitor this issue.

We are currently investigating this issue.

This incident has been resolved and was due to a corrupted Google Play listing.

A update for the android app that was released on August 19th causes the app to hang during initial load. This issue is currently being resolved and version 2.25.2 is currently in review with a fix.

The incident has been resolved, we will continue to monitor the situation and expect one or two more hiccups over the next few hours.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

A spike in email related events getting sent to the API servers caused our database to temporarily exceed its limit of operations per second, causing a systemwide slow down of all requests that required database access. We have temporarily fixed the issue by disabling email delivery and read receipts as we scale our database's peak performance to better handle spikes of this nature. Email delivery and read receipts will be re-enabled once the new database is up and running.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue, the general area of the problem has been located and a fix is being worked on.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

Following this Monday's release with a change in internal authentication protocols, users were reporting that they were being logged out every hour. We investigated the issue and eventually found the culprit. A fix has been pushed as of 12:17 PM on November 26th.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some users report that they are unable to login.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Due to a recent release, page styling does not appear and application is unusable, we have isolated the source of the issue and are working on a fix.

This incident has been resolved.

Incoming calls for most phone numbers is back to normal operation, we will continue to monitor the situation.

Our provider has updated their status page and we are awaiting further news. https://status.twilio.com/incidents/6gd7kdy789l3

Incoming calls do not currently connect on some networks. We have identified the source upstream of our voice provider and will communicate news as we receive it.

We've found the source of this incident and are working on a fix, but it should no longer occur for now.

We are currently investigating an issue where the API takes several times longer to respond to requests than usual.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved, we believe this may have been caused by a particularly active integration saturating all available server bandwith.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue, a current workaround would be to create attendance requests using the phone system or by right clicking on a shift to create an absence.

This incident has been resolved, sorry for any inconvenience this may have caused.

There is an issue following a release where the login page will infinitely reload, caused by missing permissions on an API route. We have identified the source and are implementing a fix.

This incident has been resolved.

We have identified the source of the issue and are currently working on a fix.

We are currently investigating an issue where the application client loads indefinitely for some users.