Historical record of incidents for Mews
Report: "Premium Chat Support - Service Disruption"
Last updateWe are currently experiencing an issue with our Premium Chat support functionality available to our Premium Support clients. This means that Premium clients may be unable to reach our support team via chat at this time. We are investigating the issue right now, In the meantime, we encourage you to reach out via chatbot, email or phone for urgent matters. We apologize for the inconvenience and appreciate your patience while this is being resolved. We will provide further updates as soon as more information becomes available. Thank you for your understanding. — Mews Support Team
Report: "Broad Internet issues (Mews PMS unaffected)"
Last updateMultiple cloud providers and services are reporting disruptions and outages at the moment. Mews PMS remains unaffected, but our teams continue to monitor.
Report: "Elevated errors on Space Report"
Last updateWe are observing increased errors on Space Report page for some properties. We are investigating the root cause and working on the fix.
Report: "Investigating issues with Mews Booking Engine"
Last updateWe are currently investigating reports of an issue affecting the Mews Booking Engine. The Mews team is actively working to identify the cause, and we will provide updates as soon as possible.
Report: "Issue with Mews Guest Portal"
Last updateWe discovered an issue with the communication between two of our Guest Experience APIs in our demo and production environments. We were able to switch the new features off temporarily that are responsible for the Guest Experience APIs communicating whilst we investigated. This allowed the Guest Portal to be accessed fully once again. We have updated the library we use for API communication and have made observability and alerting a key focus over the next few weeks of development.
Production and demo environments are confirmed to be working again.
Some customers were unable to access the guest portal due to an internal connection issue. This has been resolved and we have started to determine the cause. Production and demo environments have been confirmed as working fully again.
The Mews team identified the issue with the Guest Portal and is working on a resolution. Further updates will follow as we make progress.
Report: "Service Disruption Notice"
Last updateThe issue has been fully resolved by our service provider, we've verified the recovery on our end and will continue to monitor for stability, but no further impact is expected.
Our service provider has implemented a fix for the issue, and we are currently seeing continued signs of recovery. We are closely monitoring the situation to ensure full stability is restored across all services and we'll provide further updates as the resolution is confirmed by our service provider.For more details, you can follow updates from our provider directly here: https://status.launchdarkly.com/incidents/v1np1l7lpdrd
We’ve confirmed the root cause of the issue with our service provider, who has been notified and is actively working on a fix. As of now, they’ve made partial progress in resolving the outage, and we’re seeing signs of improvement in our services.
We’re currently experiencing a service disruption due to an issue with one of our technology partners. As a result, some features may appear offline or unavailable at the moment. Our team is actively investigating and working to restore full functionality as quickly as possible. We’ll continue to share updates as we have them. Thank you for your patience and understanding.
Report: "Service Disruption Notice"
Last updateThe issue has been fully resolved by our service provider, we've verified the recovery on our end and will continue to monitor for stability, but no further impact is expected.
Our service provider has implemented a fix for the issue, and we are currently seeing continued signs of recovery. We are closely monitoring the situation to ensure full stability is restored across all services and we'll provide further updates as the resolution is confirmed by our service provider. For more details, you can follow updates from our provider directly here: https://status.launchdarkly.com/incidents/v1np1l7lpdrd
We’ve confirmed the root cause of the issue with our service provider, who has been notified and is actively working on a fix. As of now, they’ve made partial progress in resolving the outage, and we’re seeing signs of improvement in our services.
We’re currently experiencing a service disruption due to an issue with one of our technology partners. As a result, some features may appear offline or unavailable at the moment. Our team is actively investigating and working to restore full functionality as quickly as possible. We’ll continue to share updates as we have them. Thank you for your patience and understanding.
Report: "Reservations coming from channel managers are missing."
Last update### **Problem** On 17th May 2025, we experienced an issue in our production system that caused the unintentional removal of key services used for the asynchronous processing of reservations and payments. As a result, around 12,000 reservation messages—particularly from channel partners—were lost or delayed. Some terminal payment events were also affected during this time. ### **Action** Once the issue was identified, we: * Reversed the change that caused the problem. * Rebuilt the services responsible for handling reservations and payments. * Switched to a backup method for processing reservations to minimize further disruption. * Notified affected customers and partners, advising them to reprocess any missing data from the affected period. ### **Causes** This issue was caused by a configuration error in a system update, which mistakenly removed important data services. The problem was not caught during the review, partly because of how the system presented the changes. It also deployed automatically to all environments, including production, making the impact immediate. ### **Solutions** To prevent this from happening again, we are: * Adding stronger checks and alerts in the asynchronous processing system before any large-scale system changes are made. * Improving how system changes in asynchronous processing are previewed to ensure potential issues are clearly flagged during our review process. * Adjusting how we deploy updates across environments to enable safer testing of changes. * Strengthening our system’s ability to fall back to other processing methods if something goes wrong. * Enhancing monitoring and alerting to catch similar problems faster.
Channel manager reservation processing is fully operational. Investigation of missing reservations will follow.
New reservations are now being created in MEWS. We are investigating if there any lost reservations and if so how many.
Reservations created in OTA and channel managers are not created in Mews. We are investigating the root cause.
Report: "Degradation of performance"
Last updateThe issue is now fully resolved. Mews is now operating as expected.
The team has identified short period of elevated CPU utilization on primary database. After 15 min all metrics are back to normal. The team is investigating root cause. We are closely monitoring the system to ensure stability.
Report: "Degradation of performance"
Last updateThe issue is now fully resolved. Mews is now operating as expected.
The team has identified short period of elevated CPU utilization on primary database. After 15 min all metrics are back to normal. The team is investigating root cause. We are closely monitoring the system to ensure stability.
Report: "Missing reservations from Sunday's incident (18th of May)."
Last updateAll channel manager partners were informed about this outage to perform a retry operation. However, if you’re still missing any reservations from this timeframe, please manually push them from your intranet to ensure they are properly delivered to Mews. We will keep monitoring the situation and we apologize for the inconvenience caused.
During the period on Sunday between 03:38 PM and 04:23 PM UTC, an error occurred while processing reservations to Mews. We are currently reaching out to affected partners to perform a retry operation if it hadn’t already been done. However, if you’re still missing any reservations from this timeframe, please manually push them from your intranet to ensure they are properly delivered to Mews. We apologize for the inconvenience caused.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
We are continuing to investigate the issue and assess any potential impact. In the meantime, we kindly ask you to reach out directly to your channel manager to re-push the affected reservations. Please note: Manual reservation creation is recommended only for same-day check-ins. Manually created reservations cannot be modified via the channel manager. We appreciate your patience and understanding as we work to resolve this.
We are currently investigating how many reservations are missing for which connection so we can contact partners for data recovery.
Report: "Unplanned Outage - we are working hard on a solution"
Last updateWe have identified and successfully mitigated the source of the outage.
We experienced an unplanned outage after a deployment, this has been reverted and we continue to monitor the situation
Report: "Delay in April Subscription and Commission Invoices"
Last updateWe’re pleased to inform you that all transactions have been successfully processed and confirmed. Invoice emails have now been dispatched to customers. We understand how important these documents are for your financial operations and appreciate your patience throughout this process.
Our team is continuing to work diligently to resolve the issue. We understand how important these documents are for your financial operations and sincerely apologize for the continued inconvenience. Resolving this matter remains a top priority, and we will notify you immediately once the invoices are available.
We are currently experiencing delays in the delivery of Mews Subscription and Payment Commission invoices for April. Our team is actively working to resolve the issue and ensure all invoices are sent out as soon as possible. We understand the importance of timely invoicing and apologise for any disruption this may cause to your operations. Thank you for your patience and understanding.
Report: "Some terminal payments are failing to process"
Last updateThis incident has been resolved.
Terminal payments are now processed correctly and are completed as expected.
Some terminal payments are failing to process and are stuck in a pending state.
Report: "System operational with potential power outage impact in some countries"
Last updatePower has been restored in most areas affected by the outage in Spain, Portugal, France, and Italy. Hence, we are resolving this incident. Our systems are operating smoothly.
Our systems are currently functioning normally. However, please be aware that due to a major power outage in Spain, Portugal, France and Italy there may be intermittent disruptions in those countries beyond our control.
Report: "Degraded perfomance"
Last updateThe issue is now fully resolved. Mews is now operating as expected.
Platform have experienced degraded performance from our backend instance. Our auto healing procedure restarted the instance. Performance of all instances is back to normal. We are closely monitoring the situation.
Report: "Downgraded Performance"
Last updateThe issue is now fully resolved. Mews is now operating as expected.
The team has resolved the issue, and we are closely monitoring the system to ensure stability.
We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.
Report: "Slowness on search bar"
Last updateWe think the impact from the issue is over. The search bar is now working without delays.
We are monitoring the performance of the search bar. It seems to be back to normal now.
We've identified the issue and mitigated the impact while we work on resolving the problem. Users should be able to use the search bar again without any issues.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Downgraded Performance"
Last updateOne of our instances encountered an issue, which has now been fully resolved. We are no longer observing any errors, and Mews is now operating as expected.
The Mews team identified the issue and is working on a resolution. Further updates will follow as we make progress.
We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.
Report: "Delayed delivery of emails due to 3rd party outage"
Last updateThe issue is now fully resolved. Mews is now operating as expected.
Delivery of emails is restored with minimum delay. We are closely monitoring the system to ensure stability.
We have identified issues with all emails being sent from Mews, this appears to be caused by our 3rd party provider - SendGrid. You can follow their status page for more updates https://status.sendgrid.com/ We are monitoring the situation closely.
The Mews team identified the issue with 3rd party product outage. We are closely monitoring the situation.
Report: "Downgraded Performance"
Last updateOne backend instance was misbehaving. Platform team restarted it. We are no longer observing errors. The issue is now fully resolved. Mews is now operating as expected.
The Mews team identified the issue and is working on a resolution. Further updates will follow as we make progress.
We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.
Report: "Investigating currency issue with Mews Booking Engine"
Last updateThe Mews Booking Engine was incorrectly using the users local currency rather than the configured currency. This has now been resolved.
We are currently investigating reports of an issue affecting Mews. The Mews team is actively working to identify the cause, and we will provide updates as soon as possible.
Report: "Severely degraded POS service"
Last update### Problem On March 26, 2025, our Point of Sale \(POS\) system experienced a major incident due to heavy load, which rendered the service unavailable to customers. The root cause was identified as insufficient Input/Output Operations Per Second \(IOPS\) allocated to the storage used by the POS's database. ### Action Upon detection of the incident, the team promptly initiated recovery actions executing specific operations in AWS and on the database. The service was restored, and monitoring was put in place to ensure stability. ### Causes The incident was caused by a combination of factors: * Insufficient IOPS allocated to the database storage. * A gradual increase in the number of merchants and users, leading to a higher database load. * Execution of various heavy queries in the hours leading up to the incident. ### Solutions To prevent similar incidents in the future, the following actions will be taken: * Review and optimize database configurations, including checkpoint frequency and memory settings. * Implement better monitoring and alerting mechanisms to detect potential issues before they escalate. * New indexes were added to the database to improve query performance. * Increased the storage IOPS from 12k to 24k and to a larger instance size.
Severely degraded POS service
Report: "POS server outage"
Last updateWe identified a degradation in the POS database and rebooted it to release resources. After the reboot, the environment returned to standard performance.
Report: "Login issue on Mews POS app"
Last update### Problem On 17 February 2025, increased levels of TLS handshake errors were detected during invoice submissions to the Croatian tax agency. A hotfix was released to address the TLS issue, but it inadvertently changed a secret key during a configuration clean-up. This change rendered all logged-in user sessions invalid, causing users to receive "Token invalid" errors and be logged out of the application. The incident affected all currently logged-in users on both Android and web admin platforms, necessitating re-authentication. ### Action As it was a configuration change with an immediate domino effect on all active sessions, all actions were retroactive. A hotfix was deployed that added fallback logic to attempt access token validation with the old key in case of an error. This is a preparatory step for supporting robust key rotation in the future. ### Causes The incident was influenced by both direct and indirect causes. The failure to properly configure the secret key led to immediate user session invalidation. Detection of the incident was delayed due to suppressed validation errors and the absence of alarms for surges in these errors. The resolution was hindered by the timing of the incident and the decision to avoid further user frustration by not invalidating sessions once again during peak hours. ### Solutions Dealing with sensitive credentials poses significant challenges. Any operation involving keys now must be confirmed and signed off by at least two individuals, aligning with industry’s best practices. Additional tooling to validate the presence of all secrets will be implemented to further safeguard from this incident re-ocurring in the future.
Due to an unforeseen issue that has been detected and fixed, all login sessions need to be restarted. All users who experience the "Invalid token" error message are advised to clear the app data from the Android device in order to log in again. Help center article: https://help.mews.com/s/article/Troubleshooting-slow-systems-in-Mews-POS?language=en_US#h6 A hotfix was rolled out, enabling devices with no active sessions between 18:00 and 20:00 to log in safely.
Report: "Changing the product order in the Mews POS webshop temporarily disabled"
Last updateWe have been experiencing slow server response and degraded performance of our web dashboard lasting 9:45-10:15 CET The ability to change the order of products in our webshop has been disabled until the fix can be deployed. This will prevent the total degradation of back office as seen this morning. We will make a server side change to improve the performance of changes in the database, we expect that to be ready at the start of next week. The backoffice interface will change from a drag-and-drop function that changes immediately to an edit and save action so all changes are batched into a single transaction
Report: "Downgraded Performance"
Last update**Problem** On March 13 at 15:24 UTC, end-users experienced degraded performance across Mews products. The degraded performance status remained until 16:08 UTC. **Causes** The Mews Application received increased traffic, and we had issues auto-scaling our application to take the additional load. **Action** The problem was resolved by the Platform Team scaling out the infrastructure manually. **Solution** We have upgraded to more substantial hardware to enhance overall system performance. Additionally, the Mews platform team is currently identifying opportunities to improve our performance monitoring solution to reduce detection time.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
We can see the regression in performance has improved and continue to monitor the situation
We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.
Report: "Relative products cannot be created with decimal values in the new product screen"
Last updateThe fix has been deployed to production. The team has confirmed the issue has been mitigated with customers.
The Mews team identified the issue and is working on a resolution. Further updates will follow as we make progress.
Report: "Downgraded Performance"
Last update### Problem An automated background job consumed excessive database resources, causing degradation of Mews services and impacting overall application performance. ### Action The engineering team quickly identified the problematic processes and terminated them, while simultaneously updating our status page to keep customers informed of the situation. ### Causes A scheduled background processing job generated unexpectedly high resource utilization, leading to database performance constraints that affected application responsiveness. ### Solutions The team has implemented immediate measures to prevent recurrence and is conducting further analysis to optimize the underlying processes. We are also reviewing our job scheduling policies to ensure resource-intensive tasks are executed during periods of lower system load.
We have disabled automatic processes that caused the degraded performance.
We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.
Report: "Elevated privilege errors in rates change"
Last updateWe resolved the issue. All impacted customers have now correct privileges.
We're experiencing an elevated level of privilege related errors in rate change and are currently looking into the issue.
Report: "Issues with printing invoices in France"
Last updateAfter monitoring doing this all day, we are confident the issue is resolved and should no longer be reproduced.
The team has resolved the issue, and we are closely monitoring the system to ensure stability.
We got information that French properties can't print bill nor invoices. The Mews team is actively working to identify the cause, and we will provide updates as soon as possible.
Report: "Queue Processing Delays"
Last updateOur background processing infrastructure was running behind for about 45 minutes which caused delay in processing channel manager connections. Some reservations may have been delayed. Any missing reservations from the Booking.com direct integration must be manually downloaded by ID. For missing reservations from other channel managers, please report them to the Mews support team.
Report: "Investigating issues with French invoicing"
Last updateWe have monitored the issued and it seems working properly again. No customers should be affected right now.
The team has resolved the issue, and we are closely monitoring the system to ensure stability.
We are currently investigating an issue with invoicing that is affecting French customers and preventing them from printing, sharing or emailing invoices.
Report: "Failing reservation payments with the latest version of Mews Kiosk app"
Last updateThe issue is now resolved. The latest version of Mews Kiosk application is fully functional.
We have resolved the issue and we are closely monitoring the system to ensure stability.
The issue is affecting the latest version of the Mews Kiosk application. Customers are advised to not update the application until the issue is resolved. Further updates will follow as we make progress.
We are currently investigating reports of failed payments affecting Mews Kiosk. We are actively working to identify the cause, and we will provide updates as soon as possible.
Report: "Premium Chat Support - Service Disruption"
Last updateThe issue has been resolved, and Premium Chat is now fully available to all Premium clients again. Thank you for your patience and understanding - we look forward to assisting you! — Mews Support Team
We've identified the issue and are currently working on a fix. Thank you for your understanding.
We are currently experiencing an issue with our Premium Chat support functionality available to our Premium Support clients. This means that Premium clients may be unable to reach our support team via chat at this time. We are investigating the issue right now, In the meantime, we encourage you to reach out via chatbot, email or phone for urgent matters. We apologize for the inconvenience and appreciate your patience while this is being resolved. We will provide further updates as soon as more information becomes available. Thank you for your understanding. — Mews Support Team
Report: "Errors making reservations"
Last updateOn the 12th of February 2025, Mews users have experienced issues with various actions involving reservations like creating reservations, reservation check-in and check-out or displaying and exporting some reports. The issue was caused by a database schema and application version mismatch. ### Action We noticed the issue almost immediately and performed actions to correct the database schema and application version mismatch. We then proceeded to synchronize any channel manager reservations that have failed to synchronize during the outage. ### Causes A bug in our automatic deployment process has caused the database schema to be updated before the application, leading to the forementioned version mismatch. ### Solutions The automatic deployment process will be revised and processes will be put in place to prevent similiar issues in the future.
The team has resolved the issue and all reservation operations are back to normal. Any possible reservations failing to be synchronized with channel managers are now successfully synchronized.
The team has resolved the issue, and we are closely monitoring the system to ensure stability.
We're experiencing an elevated level of errors during reservations and are currently looking into the issue.
Report: "Issues with email delivery"
Last updateWe have confirmed the full recovery of email delivery functionality and consider this issue resolved.
Our third-party email delivery provider has confirmed that functionality has been fully restored. We are actively monitoring the situation to ensure full recovery. Thank you for your patience.
We are currently experiencing an issue with email delivery due to a problem with our third-party email delivery platform. Our team has contacted them to resolve the issue, and we are awaiting their response. We will provide updates as soon as more information becomes available. Thank you for your patience.
We are investigating reports of login issues with Mews. The Mews team is actively working to identify the cause, and we will provide updates as soon as possible.
Report: "Investigating issues with Mews Services"
Last updateWe have generated fresh new updates with the most recent data to prevent retrying outdated data. All new updates were successfully synchronized to booking.com
We have automatically re-enabled all disabled operations by incident for booking.com integrations. We will continue with retrying all failed updates.
We are waiting for booking.com to fix the issue on their side.
We are investigating an issue with Booking.com integrations outage.
We are currently investigating reports of an issue affecting Mews. The Mews team is actively working to identify the cause, and we will provide updates as soon as possible.
Report: "Booking.com integrations operations outage"
Last updateAll the operations are now restored for all affected Booking.com integrations
The error is no longer occurring. We are preparing to automatically restore all operations to their original state. This process may take 1-2 hours. If you prefer to expedite the process, you can manually enable operations to bring the system back online more quickly.
We can see the issue is now more stable. There are still some issues but we are monitoring the situation.
Booking.com is currently aware of the issue on their side and are currently fixing the issue. Please refer to their own status page: https://connect.booking.com/status/incident/378
Booking.com is currently aware of the issue on their side and are currently fixing the issue.
We are investigating an issue with Booking.com integrations outage.
Report: "Chat & Communications to Support"
Last updateOur provider, Salesforce, has confirmed that the issue affecting our chat functionality and ticket creation has been resolved. If you continue to experience any issues please reach out to us via phone. Thank you for your patience. You can find more information directly on Salesforce status page: https://status.salesforce.com/incidents/13729
We are currently experiencing intermittent issues with our Chat support and Help centre, due to an ongoing incident on our tooling provider's side. This may affect being unable to get in contact with our Support team. Our provider, Salesforce, has acknowledged the issue and is actively working on a resolution. In the meantime, we encourage you to reach out via chatbot, email or phone for urgent matters. We apologize for the inconvenience and appreciate your patience while this is being resolved. We will provide further updates as soon as more information becomes available. Thank you for your understanding. — Mews Support Team
Report: "Chat Functionality Unavailable for Premium Clients"
Last updateOur provider has confirmed that the issue affecting our Premium Support Chat functionality has been resolved. If you continue to experience any issues, please reach out to us via email or phone. Thank you for your patience.
We are currently experiencing an issue with our Chat support functionality available to our Premium Support clients, which is unavailable due to an ongoing incident on our tooling provider's side. This means that Premium clients may be unable to reach our support team via chat at this time. Our provider, Salesforce, has acknowledged the issue and is actively working on a resolution. You can follow their updates directly here: https://status.salesforce.com/incidents/13728 In the meantime, we encourage you to reach out via chatbot, email or phone for urgent matters. We apologize for the inconvenience and appreciate your patience while this is being resolved. We will provide further updates as soon as more information becomes available. Thank you for your understanding. — Mews Support Team
Report: "Downgraded Performance"
Last updateThe impact from the issue is over.
We have noticed a downgraded performance of the system. Changes were made already and performances should be back shortly. Thank you for your patience.
Report: "Booking.com integrations operations got automatically disabled."
Last updateAll the operations are now restored for all affected Booking.com integrations
The connection between Mews and Booking.com failed due to authorization issues. We have contacted Booking.com support for further information. The error is no longer occurring. We are preparing to automatically restore all operations to their original state. This process may take 1-2 hours. If you prefer to expedite the process, you can manually enable operations to bring the system back online more quickly.
We are investigating whether issues is on our side or booking.com side.
Report: "Downgraded performance"
Last updateThe impact from the issue is over.
The Mews application experienced about 10min window of slowness or timeouts. The performance is recovered now and were are monitoring the situation.
Report: "Issue with online payment processing and channel manager reservations"
Last updateThe incident has been resolved. Online payment processing and reservation delivery from distribution partners is back to normal. All scheduled payments that are in pending status or failed with a note "Communication with payment gateway failed." could be safely retried. However, the incident may have caused some reservations not to integrate successfully. We recommend checking with your distribution partner to see whether the system: - Failed to deliver reservations during the incident interval. - Could not automatically recover the reservations after the incident interval. We apologize for the inconvenience.
A fix have been deployed and services are being restored. We are monitoring the incident.
The fix did only temporarily restore the service. We are still experience problems with online payment processing.
A fix have been deployed and services are being restored. We are monitoring the incident.
The problem is identified as a network problem. PCI Proxy is working to fix the problem.
We have identified an issue with processing online payments and receiving reservations through the Mews application. The issue is related to PCI Proxy having an unplanned disturbance. We are working together with them to resolve the problem.
Report: "Payment and Reservation Issues"
Last update### Problem Customers, guests and channel managers are experiencing issues with processing online payments and sending reservations to Mews due to a communication failure with the payment gateway. The errors first occurred on on 2025-01-20 01:01 UTC and the service was restored on 2025-01-20 02:23 UTC. ### Action We informed about the disruption on our public status page and also monitored the situation after the third party service was restored to ensure Mews systems was recovering. ### Causes The outages was caused by a planned maintenance window from one of our third party providers for payment processing. Mews failed to notify customers and partners about a the planned maintenance. ### Solutions Mews will adjust our process for planned outages announcement from third parties to make sure we can take mitigating actions or communicate the service disruption.
All services have been fully restored and are operating normally. We apologize for any inconvenience caused during this incident. Thank you for your patience.
Service performance is gradually improving. Card processing operations are being restored. We continue to monitor the situation. We will provide a final update once all services are fully operational.
We are currently experiencing service degradation affecting card processing due to third-party provider issues (PCI Proxy provider). Our team is in contact with the provider and working on resolution. Besides payments, it's also affecting Reservations from Channel Managers. We will update this status page once service is fully restored.
We have noticed a downgraded performance of the system, affecting payments, and reservations. Our team is investigating the issue. Thank you for your patience as we work to resolve this.
Report: "Downgraded performance"
Last updateThe performance problem has been resolved.
We mitigated the performance issue and keep monitoring the situation. We still wait for Microsoft to fix the root cause, but user experience should no longer be impacted.
We have identified an issue on the side of our cloud provider Microsoft Azure and are working on a resolution.
We are investigating downgraded performance of Mews application. We will provide more details soon.
Report: "Properties not able to charge via Mews Payments"
Last updateDue to an incident on PCI proxy's side (https://status.pci-proxy.com/incidents/m75j2vp8mbxd) there was a degradation impacted Mews payments between 7.25 UTC to 8.20 UTC. This also affected reservation processing of channel manager reservations and potentially availability updates.
Report: "Unplanned Outage - Mews Analytics"
Last update**On the morning of Monday, 6th January,** users were unable to open the Mews Analytics reports. **Problem:** The embedded Power BI token expired, causing the Mews Analytics reports to fail to load. **Action:** We generated a new token and applied it to the configuration. After updating the token, the issue was resolved. **Preventative Measures:** We have updated our proactive management process for token expiration to prevent similar issues in the future.
This incident has been resolved.
We believe the impact of the issue has been resolved. All reports have been updated accordingly.
Mews Analytics is now accessible again. Please note that data refresh is currently in progress, so some reports may still display outdated information.
We have identified the root cause of the outage and are actively working on implementing a resolution.
We are currently experiencing an unplanned outage of Mews Analytics. We are working on solving this problem as quickly as possible. Please continue checking this page for updates on the status.
Report: "Delay in reservations coming from channel manager connections."
Last updateAll delayed reservations were created.
We have restarted the automated processes for creating the reservations from channel manager.. All reservations are being processed now. It will take few minutes to create all reservations.
The reservations coming from channel managers got delayed.
Report: "Mews down due to database replication issues"
Last update### Incident summary **On** **November 17th between 9:53 and 15:24 UTC,** a significant issue with Mews database replication rendered the Mews PMS application unresponsive. During this time, it was not possible to change any existing data or to store new data in the Mews PMS. ### Impact For 5 hours and 31 minutes, customers experienced significant service disruptions, including slower response times and unexpected errors when trying to access functions in the Mews PMS. Reservations coming from channel managers and internal processing were not successful. ### Actions taken Mews immediately took the following actions to resolve the issue: performing a failover to a secondary region; upsizing the database; and contacting Microsoft support. Shortly thereafter, we recreated a read-only replica and made our first attempts to disable Microsoft's Change Data Capture \(CDC\). At 10:55 we began to see the first signs of recovery for read-only transactions due to the above actions. Acting upon instructions from Microsoft support, we truncated the transaction log at 14:50, after which any lost data would only have affected features built on top of CDC and therefore would have been recoverable. After several attempts to disable CDC, which were stopped due to the full transaction log, we successfully disabled it at 15:06. This allowed us to recover the database and restore proper performance of the Mews PMS by 15:24. Following these steps, significant work was required to reprocess new reservations that may have been lost because they were not retried by channel managers when Mews first reported the error during their attempted creation. ### Causes The cause of this incident is rooted in Microsoft's Change Data Capture \(CDC\), a tool for maintaining data integrity and ensuring that systems are up-to-date with the latest information. Mews began utilizing CDC one month prior to this incident and it had so far run without issues. Due to an error with Microsoft CDC, free space in the transaction log was depleted. When the transaction log becomes full, the database blocks any further data from being entered. Microsoft performed a Root Cause Analysis \(RCA\) to understand precisely what happened, and identified that the cause was a race condition between an index rebuild and a row update. This error prevented CDC scans from writing to the transaction log. When CDC fails, it retries from the last successful point and encounters the same error repeatedly, eventually leading to transaction-log depletion. ### Preventative measures Mews fully intends to avoid system outages of this type in the future. To do so, we are improving our database monitoring and alerting so that we are made aware of any similar incidents immediately, and can take steps to remediate them before they result in system outages.
The impact of this issue is over. Since 15:25 UTC the performance is back to normal.
We have confirmed that background processing (including channel manager updates) should be up to date and believe the impact from this issue is over. We still keep actively monitoring the situation.
The performance is currently stable and Mews Platform team is closely monitoring the situation.
We are seeing significant improvement in the past 20mins, but there is still performance degradation present as data is reprocessing.
We have identified an issue with our database replication (which provides high resilience and availability for our customers), and are taking action to mitigate. We are seeing small improvements on our production availability, though we are not yet in a resolved state.
The Mews Platform team continues investigating the database issues.
We are investigating issues with database replication. The mews application is still very unresponsive.
We are still seeing performance issues on the side of our database and are investigating the root cause.
We have identified a technical issue on the side of our database. We have performed failover to secondary region and are seeing first signs of improvement.
We are seeing serious degradation across all services. We are investigating what is going on.
We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.
Report: "Database maintenance"
Last updatePlease be advised that we are performing database maintenance to improve long-term performance over the next 96 hours. The system will remain accessible and perform as usual. However, during this work, some operations may occasionally run slightly slower, and spikes in timeouts may occur on certain API endpoints. We are actively monitoring system performance and making adjustments to minimize disruptions. If you experience any issues, we sincerely apologize. Your patience is greatly appreciated as we complete this necessary work to enhance our system. The maintenance is expected to conclude by 18.12 at 23:59 UTC.
Report: "Payment Processing Issue with Stripe Terminals"
Last update### Problem On **2024-02-12 11:05:24 \(UTC\),** one of our internal tools unlinked PSP accounts with Mews customer accounts. This resulted in payment failures due to missing presence of PSP accounts. We successfully corrected the links and restored the system by **2024-02-12 14:12 \(UTC\)**. ### Action We restored the lost data and linked back the PSP accounts to Mews customer accounts using database backups. ### Causes The issue occurred due to the tool running unexpectedly through an automation. The tool has been removed from the code base preventing it from executing again. ### Solutions We have unblocked all payments, and we have restored the links to the PSP accounts as a result of our recovery efforts. We are sorry for the inconvenience and we thank you for your patience as we work to keep the system reliable.
The issue causing error messages when charging payments through Stripe Mews terminals has been fully resolved as of December 2, 16:12 CET. Thank you for your patience while we worked to address this.
We have had some properties advising of an error message when trying to charge payments through Stripe Mews terminals We are currently looking into this. Thank you for your patience while we investigate.
Report: "Issues with search bar"
Last updateThe issue affecting the search bar has been fully resolved. Monitoring confirms the stability of the system, and the search bar is working as expected.
The issue causing an error when clicking on the search bar has been resolved. The search bar should now function as expected. We are actively monitoring the situation to ensure stability.
We are currently investigating an issue where clicking on the search bar triggers an unexpected error. The team is actively working to diagnose and resolve the problem.
We're experiencing some errors on the search bar, showing an unexpected error when you click on it
Report: "Issues with terminal preauthorizations"
Last updateWe have resolved the issue with terminal preauthorizations.
There may be issues with charging and cancelling terminal preauthorizations. We are deploying a fix and will you keep updated.
Report: "Issue with reservation assignment"
Last updateWe have resolved all issues with the Booking.com reservations, and all data has been synchronized accordingly
We are investigating an issue with booking.com reservations. This is affecting only a small subset of hotels with a direct booking.com integration. We are working on a solution and will update affected customers shortly.