Is Mono Solutions Down Right Now? Check if there is a current outage ongoing.

The Ecwid control panel and shops are now loading correctly again. All the data (products, orders, your store settings, etc) are safe and intact.

Our 3rd party ecommerce provider Ecwid is having an incident and you might experience some issues with Ecommerce. We will keep you updated.

The issue with RAI login has been resolved and the response time is back to normal.

We are currently investigating an issue with RAI response time. This might affect login and general RAI navigation.

We are monitoring the traffic and we don't see anything concerning at the moment.

We are currently investigating an issue with 429 errors some users might experience when visiting a site.

At approximately 12:00 today, we experienced an issue with one of the servers in our European fleet, impacting our editor and sites for some of our European clients. The downtime lasted between 5 minutes. While we will continue closely monitoring, we consider the incident resolved.

We see editor and sites are coming back up and we are monitoring the status.

We are experiencing some issues with editor and sites not available. We are currently investigating.

The issue with editor login has been resolved.

The editor is now available and our team is monitoring for any possible issues.

We are currently investigating some issues with accessing to the editor.

The incident has been resolved

The issue has been identified and our developers are working on a solution.

We are currently investigating some issues with publishing sites and uploading files to the editor.

The issue has been resolved.

We are currently investigating some issues we may experience with .SE domains.

The incident has been resolved.

We are currently experiencing some issue with adding SSL and accessing to newly created sites. We are investigating.

The traffic is back to normal and everything is operational.

We are currently experiencing a DDoS attack and it might take longer time to access the editor.

The traffic is back to normal and everything is operational

We are experiencing a large DDoS on our nameservers.

The issue with the nameservers has been resolved.

We are currently experiencing some issues with our NS2, and are working to resolve the problem

The issue with RAI login has been resolved. It might take a few minutes before you are able to login

We are experiencing some issues with RAI and it will be resolved shortly

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing issues with sites after being published, we are looking in to it.

Everything is operational

We are experiencing lowered performance due to a large DDoS attack

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our 3rd party Jira ticketing system is expiriencing some issues. If you are not able to create a ticket at http://partner.monosolutions.com/ please reach out to Mono Support through your direct Slack channel or write directly to partnersupport@monosolutions.com

The incident has been resolved.

We have isolated the problem and implemented the changes needed and we are monitoring.

We are investigating an issue with our nameservers being unreachable.

the DNS attack has been successfully resolved.

We are experiencing a DNS attack and there might be degraded performance

The issue with editor access from Chrome has been resolved.

Since the release of Chrome 115, users may experience the inability to access the editor. Alternatively, we suggest you try using a different browser while we work to solve the problem.

The incident has been resolved. Control panels and storefronts should work as expected

Unresolved incident: Ecwid control panels load slowly or don't load at all.

The incident has been resolved and RAI and the editor are operational.

At the moment there seems to be some issues with RAI and the editor. We are looking into it.

We are resolving this incident. We have monitored issues related to the email provider migration and it seems that everything is operational.

Update from the email provider: Incident resolved The engineering team identified some of the VMs along with IMAPs were identified in the stopped state and started them back up. the affected users on prod B should be able to log in with no issues as of 19:06 UTC. We are closing the incident. Incident Start Time: 03-30-2023 17:45:00 UTC Incident End Time:03-30-2023 19:06:00 UTC Total Duration:1 hour, 21 minutes We will keep monitoring

Update from the email provider: We are investigating an issue preventing some users on Cluster B from logging into Webmail. Our engineering team has been engaged. We will provide an update once we have additional information.

The issues with emails should now be resolved. If the issues related to Outlook (or other email client) inbox still persists, the quickest way to solve it is to completely remove and re-add the email account on the mail client. We will keep monitoring with the email provider for any possible further issue.

The issue with the webmail login has been resolved. We are still monitoring the general email issue.

There are issues at the moment with the webmail login and we are waiting for an update from the email provider. We will keep monitoring the issue.

We are still monitoring for any possible issues with emails. This is an update from the email provider: "On Monday, 13/03/2023 we initiated a migration on our prodB environment. For 99% of our users the migration has been completed and you should expect a return to normal operating. We have received reports of users describing issues with duplicated email messages on their IMAP clients. This is related to the means in which IMAP clients determine whether a message has already been downloaded locally. The quickest way to address this issue is to completely remove and re-add the email account on the mail client. This ensures that any incorrect caching information kept by the mail client is discarded, and presents a consistent view of mailbox contents."

On 13.03.23 there has been an issue with our email provider. The issue has been solved but we are aware that some email accounts are full because of that incident and we are in active contact with the email provider to further investigate. It is our priority to solve the issue.

The incident has been resolved. RAI, editor, and websites are operational.

We have implemented some measures to contain the DNS attack that caused this issue. RAI and editor seem to start working again and we are monitoring the status.

We are still experiencing issues with RAI and websites and we are actively investigating.

We have blocked the DDos, websites and RAI is now working normally again

The issue has been identified as a DNS attack (DDos). Our network team is actively working on this

We are currently experiencing RAI login issues and sites that are not showing online. We are actively looking into this

The incident has been resolved. We had a spike in DNS traffic towards a few specific domain names, this blocked our DNS servers from responding to actual requests.

At the moment there seems to be some issue with RAI and editor. We are looking into it.

Our development team has resolved the issue and both RAI and the editor are again fully accessible.

We currently have an issue with RAI and editor login and we are investigating

This incident has been resolved. RAI and editor are operational.

We are currently investigating an issue with accessing RAI and the editor

A DDoS attack caused some of our services to go temporary down. While we stop the vast majority of DDoS attacks we do occasionally see a larger attack that makes it through. Our development team managed to stop the attack and our services are back to normal.

We are currently investigating issues with accessing the editor.

The incident has been resolved. Files can now be uploaded to the editor.

We are continuing to investigate this issue.

There is an issue with uploading files in the editor at the moment and we are investigating

This incident has been resolved.

The issue with RAI has been resolved and it is operational again.

We are currently experiencing issues with RAI login. We are investigating

The incident has been resolved. The team will continue monitoring the server load.

We were under DNS attack and we have implemented measures to block the attack. The Mono services should be already online or becoming online as we speak. The team is still monitoring the servers.

We are currently investigating the issue.

The server performance has been fixed and the sites are loading fast again.

We are currently investigating an issue with slower load of site resources thus creating a slow down on websites live rendering.

The issue with publishing has now been resolved. It was caused by one of our CDN's and we'll continue to monitor the issue.

We are still investigating the issue.

We are currently investigating an issue where users cannot see their published changes reflected on the live site.

Sites and editor loading times are starting to normalise.

We are currently investigating an issue causing the editor and live sites to load slow.

This incident has been resolved.

This is caused by operational glitch. We are monitoring the platform and we can see the different services coming back online.

We are currently investigating this issue.

We experienced an DDOS attack for serval hours on our system originating from Russia. This resulting in our services to become partial unavailable \(429 - Too Many Requests\).Compared to previous incidents, we are seeing the attacks are becoming bigger. Resources internally have been redistributed to focus on the ongoing investigation, and additional improvements to our services. Actions have already been take to make the CDN´s more resilient, to theses kind of attacks. Further actions will be taken, when the investigation is done, and will be communicated out.

We believe that this is now resolved but we will continue to monitor.

A fix has been implemented and we are seeing sites and editor starting to load fine.

We are still investigating this. We are being under heavy load from few russian IPs. The team is working on a solution.

We are currently investigating an issue that causes some users to experience difficulties accessing the editor and sites.

All editors and sites are operational again.

We are currently investigating an issue that causes some users to experience difficulties accessing the editor and sites.

Ecwid's developers have released a fix, and all Ecwid stores and control panels are up and running again. We are monitoring the results now.

Users may experience that the Ecwid storefronts and control panels are currently unavailable. Ecwid is investigating the issue and we'll update our status page according to Ecwid's findings.

We experienced temporary instability which caused the editor to momentarily become unavailable. Our servers quickly recovered and the editor is again available.

We are currently investigating an issue with the editor which may lead to some people not being able to access the Site Editor.

RAI and the editor are now up and working again. The incident was caused by an unusual load on our DNS.

The issue has been fixed and we are already seeing the editor coming back online. We will provide more information once this has been resolved.

We are continuing to investigate this issue.

We are currently investigating the issue.

The issue was caused by some of our European CDN nodes becoming unavailable due an internal anomaly that was causing some servers to crash. Affected nodes were properly fixed/updated and all issues addressed.

This has been resolved. We will be providing details later.

We are currently having an issue where some users may not be able to access the editor.

### **Summary** After a restart of the sattelite service in the Frankfurt AWS region, EU partners were unable to create new sites from RAI. Engineers tracked down the root cause to a configuration problem of PHP. ### **Impact** Not able to create new sites from RAI, European partners. HAL not able to successfully call the create site method from the backend. ### **Root Causes** Non persistent hot fixes where made to the server. Upon refreshing the fleet instances, the hotfix was lost. As a result the sattelite treated warnings as errors when HAL was calling the "createSite" function. ### **Resolution** The issue was resolved temporarily by disabling error reporting.

This is now resolved. Creation of sites through RAI is now working. We will udpate the incident page with Postmortem later.

We are experiencing issues with creating new sites through RAI. This includes copying/creating templates as well. If you provision sites through API you should be able to continue doing so. The team is and will continue investigating the issue.

The issue has been solved. We had a diskspace issue that was interfiering with some of the SSL orders.

We are still investigating this. The issue only affects small amount of users.

We are continuing to investigate this issue.

We are currently investigating an issue with installing new SSL certificates on domains.

Incident resolved The email provider engineering team has completed checks and confirmed all services are fully restored.

The email provider team are manually bringing services back online and will continue to post updates on this. Thank you for your patience.

We are experiencing an issue affecting Webmail, Pop and IMAP connections for users on Cluster B. Our email provider engineering team is currently investigating this, and we will post updates as they are available.

We had a short case of 1600 events in a few minutes from a russian IP, but we have mitigated the issue. Editor is now accessible again

We are currently investigating why users may experience issues accessing the editor.

This incident has been resolved.

The team has identified the issue and we have released a hotfix and we are about to release a permanent fix as well. We are monitoring the stability of the service.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We currently experiencing issues when you create a new Site in RAI. We are actively looking into this

### Summary Following a previous incident on the same day, a database instance at our back-ends in Frankfurt became unavailable. This rendered the back-ends to return 404s that would be subsequently cached at our CDNs. Once the issue was identified, the database was synced with the master and the back-end service was restored. There was a number of websites that required a cache clear in order for the 404s to become stale and get the correct content cached. ### Impact * Editor inaccessible in Frankfurt * Non cached sites responding with 404s in Frankfurt ### Root Causes Redis slave instance in our VPC In Frankfurt was unresponsive. ### Trigger Master Redis instance was unavailable earlier. ### Resolution  - Manually restart the service. ## Action Items * Alert when the Redis slave is not available or not in sync with master. ## Lessons learned * We need better alerting for Redis availability issues. ## Timeline GMT\+2 2021/06/28 12:28 CET First support reports 2021/06/28 12:28 CET Engineering engaged on the incident 2021/06/28 12:34 CET Incident announced on status page 2021/06/28 13:31 CET Root cause identified 2021/06/28 13:39 CET Back-end service restored 2021/06/28 13:46 CET Status Page updated 2021/06/28 13:46 CET Affected sites still need manual cache clearing upon reporting 2021/06/28 16:32 CET Incident resolved

This incident has been resolved.

We have had networking issues that made our Redis master slave replication fail. The issues is now identified and fixed, we are in the process of updating missing data entries

A fix has been implemented and we are monitoring the service. Both sites and editor should be becoming operational again. We will provide postmortem once all has been resolved.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified and the team is working on fixing it.

We are currently experiencing slow response time when trying to use RAI. We are investigating the issue.

This incident has been resolved.

Issue has been fixed and we are operational again.

We have rolled out a misconfiguration to our Content Delivery Network, currently we are failing to serve images to sandbox sites. This is not affecting live sites. We will keep you updated on the progress.

Tool load speed has improved gradually after the team increased the server instances in the effected regions. Development team is still investigating the issue with the images. We will keep you updated on the progress.

We’re currently experiencing degraded performance issues with editor. Our team is currently working to restore normal performance levels. We apologize for any inconvenience. We will keep you updated on the progress.

The issue has been resolved and maps are loading again.

There is an availability incident for Mono maps and our development team is working on resolving the issue. Cached maps will still be served.

### Date 2021/05/122021/0 ### Authors Stefanos Agelastos ### Status Resolved ### Summary Our third party provider Equinix Metal experienced an BGP networking incident causing degraded performance at our CDN node in Frankfurt. ### Impact * Slow response time for sites and editor * Sites reported as not responding at all, from 6:00 CET till 09:00 CET ### Root Causes Frankfurt CDN node experienced an outage due to third party provider incident. Equinix Metal, our third provider, announced a networking availability incident affecting multiple BGP Anycast connections and routes on their network. ### Trigger Third party networking incident. ### Resolution  - Remove the affected CDN node from traffic. This stop the affected node from announcing our IPs and clients are directed to the rest of our CDN network. ### Detection On Call Alerts. ## Action Items * Stream line provisioning of CDN Nodes from our third providers ‌ ## Timeline GMT\+2 2021/05/12 06:22 CET First alerts 2021/05/12 06:35 CET Engineering engaged on the incident 2021/05/12 07:50 CET Third Party provider Equinix Metal [announces an incident](https://status.equinixmetal.com/incidents/995zzxff5bw0) 2021/05/12 08:23 CET Updated Mono Status Page 2021/05/12 09:30 CET Affected CDN node removed from live traffic 2021/05/12 11:57 CET Third Party provider Equinix Metal announced a fix is in place 2021/05/12 12:42 CET Frankfurt CDN node is put back in traffic 2021/05/12 12:46 CET Our system is operational and the incident is resolved

The incident is resolved and we will provide a postmortem shortly.

Our internet routing provider is experiencing issues with their services which have impacted our services. We have switched over to our second provider and as a result we are starting to see our services getting back to normal.

We are currently investigating why users experience issues with slowness and sites being down. We'll update you as soon as we know more.