Is Umbraco Down Right Now? Check if there is a current outage ongoing.

We have identified an upstream provider incident affecting the Umbraco Cloud platform. Internal hostnames in the format project.{region}.umbraco.io are currently unresponsive. We will continue to monitor this incident and provide an update within an hour.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

At this time we will perform scheduled maintenance on the primary Heartcore database.During the maintenance window the content delivery and GraphQL APIs will continue to be available and serving content. Additionally, the back-office will still be available for editing content.However, the flow of content from the backoffice into the delivery APIs will be temporarily paused. Any content publishes that happen during this window will be deferred until maintenence concludes.

This incident has been resolved.

We are continuing to monitor for any further issues.

Since the Heartcore maintenance window this morning (ended around 7 CET) the Heartcore Rest API and Preview API haven't been ingesting CRUD REST API operations correctly. Webhooks did also not fire during this time. A fix has been implemented and we are monitoring the results. All CRUD REST API operations done during the incident have been applied, no data was lost.

The incident has been resolved.

We are currently investigating this issue. Environment creation in the Europe West fails.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue where customers are unable to delete environments from their project.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The Heartcore GraphQL API underwent a degraded quality of service for a period of approximately two hours, beginning at 13:00 on March 21st (UTC). During this time, some queries returned a 429: Too Many Requests HTTP response. The issue was organically resolved around 15:00 UTC. We determined the cause to be increased traffic volume to the service. Additional capacity has been provisioned to prevent the issue happening again.

This incident has been resolved. Please reach out to our support team if you were affected by this issue.

A fix for this issue has been implemented and we are monitoring the results. Please reach out to our support team if you have been impacted by this issue.

We have identified an issue causing some Umbraco Cloud Development environments to become unavailable. Umbraco Heartcore customers may notice the incorrect environment alias in the API Browser. Our team is currently working on a fix for this issue. We will have an update on this within the next hour. If you have been affected by this issue, please reach out to our support team.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We have identified an issue where users currently can't add environments to existing projects in the Umbraco Cloud Portal. The first identified issue has been reported at 13:56 UTC.

This incident has been resolved.

A fix for this issue has been implemented and we are currently monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating a medium severity issue where customers cannot create new Heartcore projects on Umbraco Cloud. We will provide an update in one hours time.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We have identified the root cause and are rolling out a fix.

We are currently investigating a medium severity issue where customers cannot create new projects on Umbraco Cloud. We will provide an update in one hours time.

This incident has been resolved.

A fix has been successfully deployed, and we are now actively monitoring the system to ensure stability and performance.

We have identified the root cause of the issue and are actively working on releasing a fix. The resolution is in progress, and we will provide further updates as soon as the fix is deployed.

We are currently investigating reports of some customers being unable to save content in the backoffice.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Users of Umbraco products may be seeing reports of licenses not being validated when viewing status of licenses in the backoffice, and noting failed license checks in the logs. Product usage should not be disrupted as all have a grace period in case of issues with the licensing service. We have identified the problem and are rolling out a fix.

We experienced a brief outage of the Cloud Portal for around 10 minutes due to an erroneous deployment. Our Cloud Team identified the cause and rolled back to the previous version.

This incident has been resolved.

We have applied policies to mitigate the issue and fix the affected projects. We are currently monitoring the situation.

The issue is currently being mitigated and monitored.

A fix has been rolled out - we are monitoring the situation.

The issue has been identified to be on the Microsoft Azure side - we are in contact with the Microsoft team and are awaiting resolution.

When renaming or creating new Umbraco Cloud projects, the backoffice can't be accessed and a CloudFlare error page is reached. We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an operational issue with Heartcore webhook deliveries. A fix has been implemented and will be released momentarily. No webhook invocations have been lost and delayed webhooks will be processed once the fix is in place.

This incident has been resolved.

We are currently experiencing issues due to a subset of Azure services having performance issues. Umbraco Cloud Websites are unaffected. We are monitoring the situation: https://azure.status.microsoft/en-us/status

Umbraco Cloud hosting has not been affected by the CrowdStrike Falcon agent incident.

Our team continues to monitor the situation.

We are aware that some Azure Virtual Machines have been impacted by a BSOD issue, as reported by Azure status page (https://azure.status.microsoft/en-us/status). Umbraco Cloud hosting is not affected - our team will continue to monitor the situation.

The incident has been resolved/.

The issue has been identified as a problem with Azure Active Directory B2C. Azure reports this issue is currently being mitigated.

UmbracoID is currently not available - Umbraco Cloud portal is not reachable. We are currently investigating this issue.

This incident has been resolved.

Small subset of customers encountered 522 errors when visiting websites hosted on Umbraco Cloud.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We've received reports of users unable to login to the backoffice of newly created projects. We are currently investigating this issue.

This incident has been resolved.

The fix is having a positive effect on the GraphQL service. We are continuing to monitor the fix throughout the afternoon.

A fix has been implemented and we are monitoring the results.

The issue has been identified. A fix is being worked on.

We are continuing to investigate this issue.

We are currently investigating this issue.

We've identified an incident from 07:30 AM to 8:30 AM GMT+1 . Heartcore APIs were unavailable during the period. We have investigated and resolved the issue.

The Heartcore APIs are continuing to operate as normal. This incident is now resolved.

We have identified and resolved the cause of the issue. Heartcore APIs should be available again and we are continuing to monitor.

We have received preliminary reports of the Heartcore APIs being unresponsive. We are investigating the issue now.

Our supplier was able to identify the root cause of the issue and implement a fix. All Umbraco Cloud services should now function as normal. This issue has been resolved.

We are still seeing degraded performance with project/environment creation in the EU West region. Our supplier has identified a potential mitigation technique and is seeing improvements. We will continue to monitor and update the situation here as information becomes available.

We have identified an issue from one of our suppliers that could impact the ability to create new projects or environments in the West Europe region. We are currently in contact and will update this message as further information comes available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Umbraco Cloud operations intermittently fail. You might experience errors during operations such as: - Project creation - Environment addition - Change project plan - CI/CD API operations We have identified the issue and are working on a fix.

This incident has been resolved.

A fix has been implemented and the authentication emails are being delivered again; we are monitoring the affected components.

We have identified the root cause of the problem and we are in contact with the relevant service providers. It appears that the Outlook email provider in particular is affected by this problem. For urgent requests, please note you can reset the Multi-Factor Authentication by reaching out to Umbraco Support.

Two-factor authentication emails for UmbracoID may take more than usual to arrive. We are currently investigating this issue. If you are affected by the issue, please note you can reset the Multi-Factor Authentication by reaching out to Umbraco Support.

This incident has been resolved.

A fix has been applied and pending webhooks should have all been delivered. For the next 6-8 hours there may be brief intermittent delays as poisoned messages that caused the issue are periodically retried and handled by the patch.

We are aware of an issue that is currently causing delivery of webhooks from Umbraco Heartcore sites to be significantly delayed. We have created and deployed a fix for this issue, and are monitoring as the system works through and delivers the backlog of delayed webhooks.

The root cause has been identified and a fix has been deployed and confirmed working. The root cause was due to an expired certificate for internal services used in the Umbraco Cloud Portal.

We are aware of degraded performance for the Cloud Portal in the UK-region. We are investigating to find the root cause and will update you in an hour (12:00 CET)

This incident has been resolved.

A patch containing a fix has been rolled out for the affected environments. We will continue to monitor the issue until further notice.

We are pleased to announce that the auto-upgrade process with the new patches for the Umbraco Cloud issues has begun. This update is aimed at addressing the problems identified in CMS versions 8.18.10, 10.8.1, and 12.3.4. Impact: The affected users, particularly those logging into a backoffice running Umbraco versions 8.18.10, 10.8.1, or 12.3.4, may have experienced functionality disruptions or other anomalies in the following scenarios: -Environments with a multilingual setup. - Environments including a rich text editor and using custom style sheets (CSS) on one or more content nodes. Current Actions: Patch Creation: New patch versions for Umbraco 8, 10, and 12 have been developed. These include versions 8.18.11, 10.8.2, and 12.3.5, respectively. NuGet Accessibility: The patch versions are now accessible on NuGet. Auto-Upgrade in Progress: Cloud projects running the affected patch versions have begun the auto-upgrade process to these new patches on Tuesday evening. Monitoring and Next Steps: Our team is closely monitoring the auto-upgrade process to ensure smooth and effective deployment. We anticipate that upgrading to these new patch versions will resolve the known issues on Umbraco Cloud. A further update will be provided once all cloud projects with the affected versions have been upgraded with the new patch versions.

Our ongoing investigation into the issues following the recent security patch on Umbraco Cloud has led us to the conclusion that a patch is required for all versions that were affected by the security update. This includes CMS versions 8.18.10, 10.8.1, and 12.3.4. Impact: Users logging into a backoffice running either 8.18.10, 10.8.1, or 12.3.4 and with either of these setups: - a multilingual setup - including a rich text editor and using custom style sheets (CSS) on one or more content nodes are likely to experience functionality disruptions or other anomalies. These issues have been linked to the recent security patch deployment. Current Actions: We are currently creating a new patch version for Umbraco 8, Umbraco 10, and Umbraco 12. The patch versions, 8.18.11, 10.8.2, or 12.3.5 will be accessible in NuGet. Cloud projects running these patch versions will be auto-upgraded with these patches later this Tuesday evening. We expect an upgrade to the new patch versions will resolve any known issue on Umbraco Cloud.

Status CET 15:40: Our ongoing investigation into the issues following the recent security patch on Umbraco Cloud has led us to the conclusion that a patch is required for all versions that were affected by the security update. This includes CMS versions 8.18.10, 10.8.1, and 12.3.4. Current Impact: Users may experience functionality disruptions or other anomalies when logging into the backoffices of mentioned CMS versions. These issues have been linked to the recent security patch deployment. Current Actions: We are currently creating a new patch version for Umbraco 8, Umbraco 10, and Umbraco 12. When these are created, they will be deployed to the affected cloud environments, which we expect to resolve the issue

We are currently investigating this issue.

The incident has been resolved. Cloudflare's incident has been marked as resolved and we have not seen any further issues in the Cloud Portal.

Cloudflare has now marked the issue as resolved. https://www.cloudflarestatus.com/incidents/hm7491k53ppg We are now seeing restored service to the Umbraco Cloud Portal. We will continue to monitor stability.

We are continuing to monitor the Coudflare incident, and remain in contact with our suppliers. The current status of Umbraco Cloud Portal Services: -Project Creation: Available (Degraded Performance) -Environment Creation: Available (Degraded Performance) -Change Plan: Available (Degraded Performance) -Custom Hostnames Addition: (Degraded Performance)

We are continuing to monitor the Coudflare incident, and remain in contact with our suppliers. The addition of custom hostnames is continuing to be affected. The current status of Umbraco Cloud Portal Services: -Project Creation: Available (Degraded Performance) -Environment Creation: Available (Degraded Performance) -Change Plan: Available (Degraded Performance) -Custom Hostnames Addition: Currently Unavailable

We are continuing to monitor the Cloudflare incident for any updates to the situation. Currently some services have been restored, however you may still see degraded performance in these areas. Umbraco Cloud Portal Services: -Project Creation: Available (Degraded Performance) -Environment Creation: Available (Degraded Performance) -Change Plan: Available (Degraded Performance) -Custom Hostnames Addition: Currently Unavailable We will continue to update this incident as soon as possible.

We are continuing to monitor the ongoing Cloudflare issue, and are in touch with our suppliers about this issue. We will continue to update this incident as soon as possible.

We are continuing to work on a fix for this issue.

We have identified an issue negatively impacting major Umbraco Cloud Portal services: - Project Creation: Currently unavailable - Creating environment: Currently unavailable - Change plan: Currently unavailable - Custom Hostnames addition: Currently unavailable Please refrain from initiating new projects or environments, change plans or custom hostname additions during this period. This is due to the Cloudflare API currently experiencing issues - https://www.cloudflarestatus.com/incidents/hm7491k53ppg We will update this incident as soon as possible.

On October 30, 2023, Umbraco Cloud experienced a service disruption due to an incident at our network service provider, Cloudflare. The incident began at 20:03 UTC, impacting the availability and functionality of various services. The Umbraco Cloud Portal was down during this period, and also a limited number of cloud projects. Timeline: - 20:03 UTC: Cloudflare reports an investigation into availability issues impacting several services. - 20:05 UTC: First update from Cloudflare, the investigation continues. - 20:23 UTC: Second update from Cloudflare, the investigation continues. - 20:33 UTC: The issue is identified, and a fix is being implemented by Cloudflare. - 20:34 UTC: A fix has been implemented, and monitoring begins to ensure stability. - 20:35 UTC: Cloudflare reports the incident as resolved. Impact: The incident affected the availability of the Umbraco Cloud Portal, hindering access to certain cloud projects. Authentication processes, as well as loading functionalities, were particularly impacted, causing inconvenience to our users. Resolution and Mitigation: Upon identifying the root cause, Cloudflare implemented a fix at 20:34 UTC, which restored the services to their normal functionality. Our engineering team remained in close contact with Cloudflare throughout the incident, ensuring that necessary steps were taken to mitigate the impact on Umbraco Cloud services. Alternative routing solutions were explored to minimize service disruption.

This incident has been resolved.

We have seen a reduction in these issues since 02:03 CEST, and continue to monitor the situation.

We are continuing to monitor the issue and discuss potential resolutions with our 3rd party suppliers.

We have identified the root cause of the issue. We continue being in contact with our suppliers to fix the issue.

We are currently investigating the issue. We've contacted our 3rd party provider Cloudflare and we're working towards resolution.

# Summary On Wednesday 26th of July, we had an incident impacting operations for adding new environments, changing plans, and creating projects, including trial flow, resulting in instances where customers were unable to do necessary operations on their projects. The affected component was a third-party vendor responsible for delivering a package feed to source additional applications that are used by Umbraco Cloud to run and manage environment websites. The incident was registered between 8:23 CET and resolved at 22:47 on the 26th of July # Root Cause Analysis Umbraco Cloud is using Myget to host a type of application called site extensions, which is a form of symbiotic component to an Umbraco Application running Umbraco Cloud. Pending on tier and product we utilize different site extensions to function as entry points for managing, and working with Umbraco under the hood, and is a required component on Umbraco Cloud.  Late on the 25th of July throughout the 26 and hours into the 27th Myget seized to respond to pulling Nuget Packages on Request \([https://status.myget.org/787400873](https://status.myget.org/787400873)\) and did so for around 40 hours.  MyGet was a known Single Point of Failure \(SPoF\), on Umbraco Cloud, and we have previously relied on stable uptime for that third-party provider. Internally the process for handling such an issue was in place, and we followed the playbooks for incident management. With the very limited knowledge regarding MyGet recovery, we initiated working on a fallback solution at 13:37CET and began testing it on our internal Development environments thereafter. At 16:09 CET we deployed our fallback solution to Production, where we saw an instant recovery of services.  At 17.35 We discovered that while our US and UK region was recovering from the incident gracefully, customers in the European region were still experiencing issues. At 20:28 The European region was recovering as expected and we changed from Actively fixing the issues to monitoring.  At 22:47 The Issue was resolved. **What happened to MyGet?** Unfortunately, we have yet to hear from Myget regarding the issue, and at this point in time, no efforts have been made on their behalf to ensure that this will be resolved moving forward. **Actions based on root cause analysis** First, we’ve already identified the provider as a SPoF. We have updated our risk assessment from negligible \(based on historical high uptimes\) to _critical_. We are actively working on solidifying the applied fallback solution, with the end goal of replacing Myget as a Nuget Package Provider in favor of multiple alternative fallback sources and ensuring a redundant Nuget Feed for Umbraco Cloud.  If you have any questions related to the above please feel free to contact your partner manager, reach out through our support channels or the Umbraco Cloud issue tracker on GitHub: [https://github.com/umbraco/Umbraco.Cloud.Issues/issues](https://github.com/umbraco/Umbraco.Cloud.Issues/issues)

This incident has been resolved.

Hotfix has now been deployed in the European region and project creation and environment addition is working again. Creation of trials is now possible as well. The incident is now in monitoring state and we will close the incident as resolved within the next hour. Reach out to support@umbraco.com in case of any issues.

A fix has been rolled out to useast01 and uksouth01 regions.

We are continuing to work on a fix for the issue. We have evaluated our fallback options and have started work on implementing a fallback provider for Umbraco Cloud Site Extension nuget packages.

Our upstream provider, myget.org is experiencing an issue which is affecting our ability to create new projects or add new environments to existing projects. See https://status.myget.org/ for more details. We continue monitoring for updates as well as evaluating fallback options. Website front-ends continue working.

Our upstream provider, myget.org is currently experiencing an issue which is affecting our ability to create new projects or perform deployments. See https://status.myget.org/ for more details. Website front-ends continue working. We will resolve this incident once we have received confirmation the issue has been resolved from our upstream provider.

The incident is now resolved. From 01:34 to 01:52 and 07:16 to 08:52 UTC websites on Umbraco Cloud responded with intermittent 500 HTTP errors from Cloudflare.

A subset of websites on Umbraco Cloud respond with intermittent 500 HTTP Errors. This is due to an ongoing incident in Cloudflare our DNS provider. We are currently investigating and will follow up with an update within the hour.

This incident has been resolved.

We are continuing to monitor for any further issues.

The Third Party Provider, MyGet, has recovered and the affected services on Umbraco Cloud have recovered as well. We will continue to monitor the situation.

We see that MyGet is slowly recovering the Feed Protocols and Package Downloads services. https://status.myget.org/ Once they are up again, Umbraco Cloud infrastructure issuance should recover. We will continue to monitor the status for the coming hours. Next status update will be no later than 17:00 CEST

We are continuing to monitor for any further issues.

We see that MyGet is reporting the Feed Protocols and Package Downloads services as "down": https://status.myget.org/ Once they are up again, Umbraco Cloud infrastructure issuance should recover. We will continue to monitor the status for the coming hours.

We are continuing to investigate this issue.

We have identified an issue with a third party provider which affects the issuance of new infrastructure. This affects, change plan, add environment, and potentially create new Projects. A new update will be provided no later than 16:00 CEST.

The incident has been resolved on Azure's end. Our investigation and testing have confirmed that all systems and services are functioning as expected on our end.

Azure Status History link - https://azure.status.microsoft/en-us/status/history/

We've experienced several customer sites being down between 5:03 and 6:18 CEST today. Underlying hosting provider issues have stopped and we are now in the monitoring phase.

This incident has been resolved.

Underlying hosting provider issues have stopped and we are now able to provision new projects and new environments in the West Europe region. Running websites in the West Europe region remain unaffected, while we continue to monitor the situation.

We found an issue with provisioning of new Umbraco Cloud sites and new environments in Europe. We are working together with our hosting provider to find and implement a solution. Running websites in the West Europe region are unaffected.

This incident has been resolved, and we are seeing a positive ingestion in telemetry

The status page from Azure has been updated, and it seems like Azure is recovering, as well as Umbraco Cloud. We'll keep monitoring our internal performance indicators and provide an update latest at 1030 CET

We have identified the source to be a Microsoft Azure Api that Umbraco Cloud is depending on to provision and change infrastructure. As such you'll see a degraded performance when creating and changing plans etc. Website hosting is not affected. Our partners at Microsoft have provided us with the following status page https://azure.status.microsoft/en-gb/status

We found an issue with provisioning of new Umbraco Cloud sites and new environments in Europe. We are working together with our hosting provider to find and implement a solution. Website hosting is not affected.

This incident has been resolved.

The hot-fix rollout of the autoupgrade to all Heartcore projects is now complete. The content API as well as content updates in Umbraco backoffice are now working as expected. We continue monitoring and will close the incident within the next hour.

We have identified the underlying issue preventing content updates. We have sucesfully tested a hotfix on a set of internal projects as well as select customer projects. We will now begin rolling out an emergency hotfix to all the affected heartcore projects as an automatic upgrade. Heartcore API as well as backoffice content updates will start working after the emergency update is rolled out. We will post the next update once the roll out is complete.

We are currently investigating an issue related to Umbraco Heartcore content updates not working.

We've just concluded the last rounds of checks and reporting, and we are back to normal. Again we are sorry for the inconvenience that you might have had, and if you are experiencing any Heartcore issues, please reach out to support.

All of our telemetries are reporting that we are back to normal. We will keep monitoring the sites for now, to ensure that everything is as it seems. If you are having issues, please reach out to support. We are sorry for the inconvenience that you might have had.

We have started rolling out a fix for all affected Heartcore Projects, and we start to see a positive trend in our monitoring.

We have identified a fix, and we'll start rolling out new releases to Heartcore shortly. Next update no later than 1500 DK Time

We have identified the issue with one of the dependencies Issue: - Heartcore has an assembly redirect for System.Diagnostics.DiagnosticSource to version 4.0.3.1 - The latest CMS patch upgrades System.Diagnostics.DiagnosticSource to version 4.6.25519.3 - The assembly redirect in web.config still points to 4.0.3.1 We are still deciding on a course of action.

While deploying the latest upgrades for CMS 8.18.6 and Deploy 4.7.4, we discovered a significant regression of uptimes for Heartcore projects. We are currently investigating the issue and will start restoring websites as soon as we find the issue. We will provide an update no later than 12.00 DK time.

# Summary On Monday, February 13, 2023, we experienced a high load of connections from our GraphQL server, which caused connections to the central database server to fail. For the duration of the incident it was not possible to get a stable connection to the database and as a result of this content and schema ingestion were paused. The REST API, Preview API, and GraphQL API were mostly unavailable for the 4 hours the incident lasted. We apologize for the inconvenience that this has caused our customers, and assure you that we are working on initiatives to further improve both the GraphQL server and the central database used as part of our Content Delivery Platform. Throughout the incident we have involved Microsoft to debug the issue from their side and help pinpoint the connection issues. The following sections detail the timeline, issue, and actions that relate to the February 13th issue caused by a high number of connections made from our GraphQL server to our central database server. # Incident Timeline On February 13th at 15:24 CET we received an alert about failed connections to our central database because too many clients \(API Server instances\) were already connected. Upon receiving this alert we looked at the health of the database, which showed high and consistent CPU usage. We proceeded to look at active and failed connections, as well as connecting the database to gather details about the issue, which caused the alert to fire. While investigating the issue we quickly discovered that the database was not accepting connections while showing very few active connections. We then escalated the issue through support channels with our vendor. We proceeded to restart the API servers to ensure that no existing database connections were in use. This initially worked and the APIs started serving requests again, which was posted on status page at 16:42 CET. We continued to monitor the database and API servers for the next hour and at 17:47 CET the connection issue started again. At around 18:00 CET we jumped on a call with a support engineer from our vendor to further debug the issues we were experiencing. We isolated the GraphQL server as being the culprit of the issue, as it was continuously making an excessive amount of connections causing congestion in the database server. With both API servers offline we were able to access the database and increase the number of connections available to the GraphQL server. We then continued to gradually turn on the API servers starting with the REST API and Preview API. After monitoring health for about 15 minutes we continued to bring the GraphQL API back online while monitoring the number of active and failed connections, as well as incoming requests. At 19:14 CET we updated status page as both APIs had been serving requests for a longer period of time without any failed connections. We continued to monitor the APIs for another hour before starting the ingestion of queued schema and content updates from all Umbraco Heartcore projects. We finished ingestion after about an hour and continued to monitor the health of all three APIs for another hour until reporting the issue as resolved at 21:00 CET on our status page. Since February 13th we have been investigating the issue centered around the GraphQL server and have been actively monitoring the health of the database server. # Root Cause Analysis The incident centered around our GraphQL server and the central database in our Content Delivery Platform. The REST and Preview APIs were only affected because of the congestion in the database server, which is unfortunate as we have been following recommendations to avoid this type of “noisy neighbor” situation. After a conversation with Microsoft it’s now clear that there are certain limitations with the database tier we are currently using, which indicates that a shared gateway can handle fewer connections per second than what we send under high load. The recommendation is to move to a different database tier, which doesn’t have a shared gateway. Not having a shared gateway means that all connections go to the database and the user limitations will have the desired effect - and establishing new connections will be faster. Last year we made various performance improvements to the GraphQL server to serve complex queries faster. One of these improvements introduced parallelization as the results for a query is retrieved. The implication of this is that a very complex query could spawn many connections \(n\+1\). This is not feasible as we need to be able to handle many complex queries at the same time without jeopardizing the health of the database server. Additionally, for each query sent to the GraphQL server, the schema needs to be resolved so the query can be validated and transformed into database queries. The way this is currently done is by generating a text-based schema by retrieving the document types from the database. The end result can be fairly big \(megabytes\) and because of this a lot of data goes back and forth between the GraphQL server and the database. As this currently happens on each \(uncached\) request it imposes a lot of work in the database, which we believe can be avoided. We are already in the process of preparing a newer and faster version of PostgreSQL Server for our Content Delivery Platform, which will have a positive impact on many fronts - both performance, connections, and availability. This update will happen within the next few months \(current target is end of March\) and will be posted to our status page as scheduled maintenance. **Actions based on the root cause analysis** First and foremost we will continue with the database upgrade, as this is already in progress, and as mentioned above it will have a positive impact. Specifically for the GraphQL Server we have planned several improvements of the utilization of database connections - basically limiting the number of connections that we need to make to serve a GraphQL query request: * We will look into reducing the amount of parallelization that is currently done within the GraphQL server, so we can reuse the same connection for a single request. * We will rework how the schema is handled per request by loading document types \(used for generating schema objects\) into memory instead of generating a text-based schema. * We will cache document types in memory in the GraphQL server to avoid continuously looking up the same document types over a period of time to serve requests for the same environment. The implementation of rate limiting is still in our backlog, but we anticipate starting a first implementation at the instance level. This will be advertised some time in advance of it being rolled out. We believe all of these actions will have a significant impact on the stability and performance of the GraphQL server. If you have any questions related to the above please feel free to contact your partner manager, reach out through our support channels or the Heartcore issue tracker on GitHub: [https://github.com/umbraco/Umbraco.Heartcore.Issues/issues](https://github.com/umbraco/Umbraco.Heartcore.Issues/issues)

The issue has been resolved and ingestion of schema and content updates runs as normal with operations back to normal. We expect to release a post mortem by the end of next week with more details about the issue and what has been done to solve it. We apologize for the inconvenience that this incident has caused!

After a period of monitoring we have restarted ingestion of schema and content updates. Everything has now been processed. We will continue to monitor all APIs for a while longer to ensure the health of all services before we close the incident.

We are continuing to monitor for any further issues.

After a longer debugging session with our database vendor, the APIs are back online and we are actively monitoring the connectivity and working on getting the queue of schema and content updates processed.

There is currently no connectivity to the database and we are working with our database vendor to resolve the problem.

We are continuing to monitor for any further issues.

The database is back online serving requests for all 3 APIs. We'll keep monitoring to ensure its fully recovered. Currently, ingestion of schema and content updates are paused. We expect to start the ingestion again within 30-60 minutes.

The central database used for the Umbraco Heartcore APIs is currently not accessible and we are working on getting it back online.

We are currently investigating an issue with connectivity to the Umbraco Heartcore APIs - REST API, Preview API and GraphQL API.

This incident has been resolved.

Microsoft Azure has been reported to be recovering and with all status indicated from Azure now in the green, it seems like we are recovering on Umbraco Cloud as well. We'll keep monitoring the situation and keep you informed of any updates.

We've got the latest statement from Microsoft. "We have identified a recent change to WAN as the underlying cause, and have taken steps to roll back this change. Our telemetry shows consistent signs of recovery from 09:00 UTC onwards across multiple regions and services, and we are continuing to actively monitor the situation. With WAN networking now seeing recovery, we are working to ensure full recovery for impacted services." Azure Status https://status.azure.com/en-gb/status

The latest information from Microsoft "Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365, PowerBI. We've determined the network connectivity issue is occurring with devices across the Microsoft Wide Area Network (WAN). This impacts connectivity between clients on the internet to Azure, as well as connectivity between services in datacenters, as well as ExpressRoute connections. The issue is causing impact in waves, peaking approximately every 30 minutes. We are actively investigating and will share updates as soon as more is known. This message was last updated at 09:24 UTC on 25 January 2023"

We've got the latest statement from Microsoft. "Azure Networking - Multiple regions - Investigating Starting at 07:30 UTC, we're aware of a networking issue impacting connectivity to Azure for a subset of users. We are actively investigating and will share updates as soon as more is known." We'll keep you updated when we've more information.

We are currently experiencing degraded performance on Umbraco Cloud. We have identified challenges with provisioning new Cloud Websites, Logging into Umbraco Id, and slow load times.

This incident has been resolved.

A fix have been implemented we are monitoring

We found an issue with provisioning new Cloud Sites and environments in Europe. This is due to an unexpected capacity limit with our hosting provider. We have identified that fix, and we have started implementation. Website hosting is not affected. Expect an update at around 10:30 DK Time

Moving this incident to resolved as the rolled out fix had the desired effect. Same fix has been applied to other regional configurations to ensure we don't run into the same issue again.

A fix has been rolled out and we are currently monitoring to verify everything is working as expected. Please note that the first occurance of the issue was observed at 15:06 CET.

Regional services for deployments, project and environment creation are currently inaccessible due to an infrastructure connectivity issue. We are currently debugging the situation trying to find a solution to bring all regional services online. Only the US East region is affected. Trying to access projects in US East through the Umbraco Cloud Portal currently doesn't work as it relies on these regional services for environment details. All websites remain only and available, as only internal services are affected.

With our Microsoft partners, we have found a workaround to solve deployment issues. While this is a temporary workaround, we'll schedule a maintenance window to address the issue for Umbraco versions 9 and 10. The workaround is to update the global.json inside the cloud repository, using exactly dotnet version 6.0.401 and disabling roll forward, requiring an exact match on the runtime You can read more about global.json here: https://learn.microsoft.com/en-us/dotnet/core/tools/global-json Today (Tuesday 13 December) we will start rolling out a configuration component to all websites on Cloud. this will enable us to update the configuration without any impact on running websites. Thursday, we'll be updating the global.json with the aforementioned details to all Umbraco 9 and 10 Cloud Websites. The roll-out of the configuration component as well as the update to global.json will be non-intrusive. For local development ensure that you have the correct version of .NET SDK installed on your machine. You can find the appropriate at https://dotnet.microsoft.com/en-us/download/dotnet/6.0 For Umbraco Documentation regarding the issue, please visit https://docs.umbraco.com/v/10.x-lts/umbraco-cloud/set-up/working-locally#tools

We are continuing to investigate the issue along with Microsoft support and have escalated the severity level of the issue. Sorry for the inconvenience caused. As a temporary solution some of our affected customers have managed to temporarily resolve the issue by removing the global.json file or by enforcing the version of the dotnet SDK to 6.0.401 It is important to note that the above workaround is only temporary and might have to be reverted after we find a proper fix.

We are seeing a subset of users having difficulty deploying to Umbraco Cloud. We are currently investigating the cause, and have reached out to our partners at Microsoft. Projects affected are running versions 9 and 10 of Umbraco. More specifically the issue is revolving around adding and removing environments, upgrades, and deploying schemas between environments. The website's runtime is not affected.

This incident has been resolved.

We are investigating an issue where our baseline services are not working as expected. Sorry for the inconvenience.

# .Net 7 Rollout on Microsoft Azure On Thursday, October 20 2022 Our partners at Microsoft started rolling out a pre-release version of the upcoming .NET version 7. This unfortunately resulted in a subset of customers not being able to deploy changes to Umbraco Cloud or deploy changes between environments. The way that Umbraco Cloud works is tightly bound to the current SDK and runtime of .NET, and is susceptible to changes in the underlying framework. Over the last couple of weeks, the roll-out has been scaled across Azure Data Centers, and more customers have gradually become affected. We apologize for the inconvenience that this has caused our customers, and assure you that we have taken steps to address the specific issue and are working on initiatives to ensure that this type of issue will not happen again. ## Root Cause Analysis In preparation for the upcoming release of .NET 7 multiple preview versions are getting deployed to Microsoft Azure \(our hosting provider\). We did not anticipate that this would cause a change in the default behavior on how we build and deploy changes internally on our Umbraco Cloud Projects. Additionally, the preview versions that were being pushed to our infrastructure, contained a bug that made the Umbraco Website unable to build on the current hardware. The issue started occurring on Thursday, October 20, 2022, at that point, it was only a small subset of customers were affected by the rollout. Nevertheless, we did reach out to our partners and Microsoft to gain an understanding of the issue at hand. On Wednesday 2nd of November, we saw a sharp uptick in the number of tickets coming in, and we took immediate action. When provisioning a new Website, we explicitly set the current .NET runtime, however, we discovered that if a version of the .NET runtime was installed automatically via Microsoft Azure roll-out then it would not enforce our settings. This caused the underlying infrastructure to utilize the latest available version of .NET - in this case, the pre-release of .NET 7. The default behavior for our infrastructure is that it is running on the latest version of .NET installed. To hotfix the issue we communicated that applying a global.json to the repository, containing the targeted SDK would get the build process up and running again. The default .NET behavior is documented here \([https://learn.microsoft.com/en-us/dotnet/core/tools/global-json](https://learn.microsoft.com/en-us/dotnet/core/tools/global-json)\) ‌ On Friday the 5th we rolled out a hotfix to a core component in Umbraco Cloud that would allow us to patch all [ASP.NET](http://ASP.NET) Core-based projects \(Umbraco 9 and 10\). The component is also installed on non- [ASP.NET](http://ASP.NET) Core projects \(Umbraco 7 \+ 8\), and though the component was tested for Umbraco 9 and 10, the release introduced a regression for Umbraco 8 sites. Our support department was quickly discovering the malfunctioning component and reached out to the Platform Team, which halted the rolling out of component upgrades, then proceeded to patch the component, and re-released the hotfix for the component. Monday the 7th, we created and tested an internal application, to target and migrate the affected projects. Tuesday the 8th, we patched all affected websites on Umbraco Cloud. ## Actions based on the root cause analysis When creating a new Umbraco Cloud Website, we are creating a new bundle that contains the default files for a new Umbraco Cloud Website. This includes file structures, project files, readme, and configuration files. We’ve amended the bundle to include the global.json file that includes the targeted .NET runtime, and all new Umbraco Cloud projects are created with the new global.json Additionally, we are creating internal procedures and guardrails for testing the component that was released onto Cloud On Friday the 5th. This includes a series of automated tests that will cover the scenario where the component is running on different project types.

We have rolled out and updated adding global.json to all .NET core-based projects (Umbraco 9 + 10), this ensures that the correct SDK is selected when deploying to Umbraco Cloud Websites. An RCA will be available by the end of this week.

On Friday, November 5 component was being hotfix on all Umbraco Cloud Websites. The component allows us to upgrade all .net core based projects (Umbraco 9 + 10) to include a global.json in the root of the repository to control the correct runtime of .net. All affected websites will be patched over the next few days. Sorry for the inconvenience this has caused! If you are still experiencing issues, please follow the guide below. https://our.umbraco.com/documentation/Umbraco-Cloud/Troubleshooting/Deployments/#issues-with-net-7-deployments-between-local-and-envs

We have rolled out a fix for creation of v10 Umbraco projects on cloud, we continue to work on fixing affected projects. As mentioned before if your project is affected, either reach out to support or follow this guide: https://our.umbraco.com/documentation/Umbraco-Cloud/Troubleshooting/Deployments/#issues-with-net-7-deployments-between-local-and-envs

We have identified the root cause to be correlated with the rollout of .net 7 on to Azure. We have identified steps needed to mitigate the issue and we are working on a rollout strategy on cloud. If your project is affected, either reach out to support or follow this guide: https://our.umbraco.com/documentation/Umbraco-Cloud/Troubleshooting/Deployments/#issues-with-net-7-deployments-between-local-and-envs.

We are still investigating the issue with creation of new projects/environments and when changing plans. If you encounter this issue, please reach out to our support. We are still working with our partner Microsoft to resolve the issue.

We are currently investigating an issue that causes customers not to be able to create new v9 and v10 Umbraco projects on Umbraco Cloud. We are currently working with our partner Microsoft to resolve the issue.

All issues related to the preemptive hotfix have been rolled out to all Cloud Customers. Sorry for the inconvenience this might have caused.

We have identified the issue as a hotfix to an internal component being rolled out Umbraco Cloud. We have amended the hotfix and are actively updating all affected websites.

In relation to the ongoing incident https://status.umbraco.io/incidents/fq4cz99rl6gy a fix was rolled out, which has caused an issue creating projects on Cloud for Umbraco v7, v8 and Heartcore. The team is rolling out a fix to get this resolved in the next couple of hours. We apologize for any inconvenience caused.