Historical record of incidents for Liferay Cloud Platform
Report: "Infrastructure Services Instability"
Last updateWe are experiencing issues with our infrastructure provider. We are monitoring to mitigate any potential impact. Symptoms on our platform are unavailability of the Cloud Console, Deployment failures and backup failures.
Report: "Kubernetes-upgrades"
Last updateA fix has been implemented and we are monitoring the results.
Our infrastructure provider is performing kubernetes upgrades from 8:00 AM Pacific time to 8:00 PM Pacific time and customers on standard availability might suffer a brief downtime
Report: "api.liferay.com down because of expired SSL certifitace"
Last updateThe cert used on our haproxy-vm had expired and has been replaced
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "files.liferay.com and releases.liferay.com were temporarily down"
Last updateSome systems were unable to verify the validity of the new certificate for the liferay.com domain, so we had to update the CA certificate chain on the web server and this required restarting the web server. The systems are now accessible and operational.
Report: "We are experiencing an outage on our main firewalls that are handling the incoming traffic of the LAX DataCenter."
Last updateAll inbound traffic to the Diamond Bar DataCenter was being blocked due to an error in our inbound firewalls. This affected access to services hosted there such as releases.liferay.com and files.liferay.com. The firewalls have been fixed. The DataCenter is operational again.
Report: "Brief infrastructure downtime in South America"
Last updateA service interruption on March 29, 2025, impacted the southamerica-east1-p3-c1 cluster, causing temporary downtime for projects hosted within it. This disruption was caused by increased latency on the File System during a routine file system maintenance of the service provided by our cloud infrastructure partner. All services are fully restored.
Report: "Certificate on releases.liferay.com will expire 4/8/25"
Last updateThe certificate for releases.liferay.com has been renewed and is being applied to the server. Downtime expected to be about 10 minutes
Report: "Brief infrastructure downtime in Europe West"
Last updateA service interruption on March 28, 2025, impacted the europe-west3-p6 cluster, causing temporary downtime for projects hosted within it. This disruption was caused by increased latency on the File System during a routine file system maintenance of the service provided by our cloud infrastructure partner. All services are fully restored.
Report: "Certificate on customer.liferay.com has expired"
Last updateDate and time it occurred: March 17th 17:05 UTC Description: The certificate that was renewed was not reloaded on the webserver of the LFU server, which is responsible for storing the large customer.liferay.com files. Resolution: IT restarted the webserver to reload the new certificate and the certificate reload automation was configured. Date of resolution: March 17th 19:26 UTC
Report: "The releases.liferay.com unreachable"
Last updateDate and time it occurred: March 24th 13:34 UTC Description: The webserver hosting releases.liferay.com crashed due to a locked process on the host. Resolution: IT restarted the web server. Date of resolution: March 24th 16:22 UTC
Report: "Brief infrastructure downtime in many regions"
Last updateWe identified that a brief period of downtime affected some customers using older DXP versions. Liferay's self healing capabilities restored all environments promptly. This disruption was caused by increased latency on the File System during a routine file system maintenance of the service provided by our cloud infrastructure partner. The maintenance happened on Feb 25th. To mitigate the risk of recurrence, we strongly recommend upgrading to the 2025.Q1 release. This release enables a transition to improved file systems, designed to enhance performance and stability.
Report: "Builds page on Cloud Console not showing content"
Last updateFix applied. This incident has been resolved.
The issue is has been mitigated and no customer should have the problem anymore. The permanent fix will be deployed on Tuesday Feb 18th.
The issue was identified, and we applied the workaround. We continue working on the final fix.
Some customers are not able to see the list of Builds on the Cloud Console. The issue is intermittent. We are currently investigating.
Report: "Instability in the US-Central cluster impacting customer projects"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Customers are experiencing deployment issues in many regions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Issues logging into cloud console also affecting Jenkins Builds"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue was resolved. We continue monitoring to grant platform availability.
The team is still monitoring and working on the issue as the issue might happen on rare occasions.
A fix has been implemented and we are monitoring the results.
Some users experience issues with logging into the cloud console when not using SSO. The team is currently investigating.
Report: "Unable to restore backups for services hosted in Dammam (Saudi Arabia) Region"
Last updateWe have successfully resolved the issue impacting backup restoration activities in the Dammam, Saudi Arabia region.
Our infrastructure provider has informed us that they are actively working to fix the issue. They estimate a resolution by the second week of November.
A mitigation was identified. If you need to perform backup restorations in this region, please reach our Help Center.
Our infrastructure provider has an ongoing issue with resource allocation that prevents backup restoration activities for services hosted in Dammam, Saudi Arabia Region. We're closely monitoring the situation and will provide updates as soon as possible.
Report: "Potential Liferay’s webserver instability"
Last updateFix applied. This incident has been resolved.
Occasionally, Liferay’s webserver service (nginx) deployed on customer environments may undergo brief periods of unavailability as a result of infrastructure instabilities. Liferay's inherent self-healing mechanism is designed to automatically identify and rectify these instabilities and no action is needed from customers. If problems persist after restart, contact Liferay support. We apologize for any inconvenience and are working on a long-term solution as soon as possible.
Report: "False positive file removals in US SaaS clusters"
Last updateThe issue has been resolved.
All the missing files were restored.
Issue has been identified and we are working on the files restoration.
We are currently investigating an incident where our antivirus software erroneously removed certain files from our SaaS clusters in the United States. A limited number of files were identified as infected, which have now been confirmed as false positives. This issue may cause browser page loading disruptions and result in a small number of documents (33 in total including non-production systems) being missing. Please be assured that our backups remain unaffected, and we are actively working on restoring the affected files. We apologize for any inconvenience this may have caused and appreciate your understanding as we resolve this issue. Thank you for your cooperation. No components were marked as affected by this incident.
Report: "Intermittent issue while restoring a backup"
Last updateIssue has been identified and fixed. For optimal performance on backup and restore processes we recommend to always use the latest images available. More information on how to optimize and configure the services can be found on https://help.liferay.com/hc/en-us/articles/26902463990029-Backup-restoration-failing-due-to-document-library-download-extraction-issues
We have identified an intermittent issue that may hinder the ability to restore a backup. If there is urgent need to execute a restore and it fails, we encourage customers to create a critical ticket and we will do our best to help mitigate the problem. There might be data loss during the workaround. Due to this issue we are limiting changes to SaaS projects.
Report: "Some Scheduled Backups Are Not Being Created"
Last updateThe regression found was fixed and there is no more problems with the scheduled backups. There is no action needed from customers.
We are continuing to work on a fix for this issue.
We have identified an issue with some scheduled backups not being created as expected. As a workaround customers can trigger manual backups
Report: "Some Scheduled Backups Are Not Being Created"
Last updateWe have identified the cause and we have mitigated all occurrences.
We have identified an issue with some scheduled backups not being created as expected. Database backups are still being successfully created on our infrastructure. We recommend to take a manual backup in case the scheduled one failed.
Report: "Scheduled Backups not being created"
Last updateWe have identified the cause and we have mitigated all occurrences.
We are currently experiencing issues with scheduled backups as they are not being created as expected. Our team is actively working to resolve this issue. We apologize for any inconvenience this may cause and appreciate your patience as we work to fix it.
Report: "Dynatrace license expired"
Last updateDynatrace adjusted the License expiration. It's working, now.
Dynatrace is unavailable at this time due to license expired issue. We are working with them to restore the service. There is no impact for customer projects.
Report: "Console monitoring data display issue"
Last updateThe issue has been resolved.
The fix for the issue has been created and it is scheduled to be applied in today's maintenance window.
The issue has been identified and the team is working to resolve it.
We are currently investigating the issue.
Report: "UI problem showing service status"
Last updateIssue was fixed and UI is operational again.
Services status on Console are not updated as ready after a deployment even though they are successful. This is only a UI issue and services are working normally.
Report: "Console is very slow or unreacheable"
Last updateThe issue has been resolved.
The issue is identified and the team is working to resolve it.
We are currently investigating this issue.
Report: "Problems to log into the console"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Console is unstable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and the team is working on a fix.
Console is unstable at this moment. We are investigating this issue.
Report: "Important information about Liferay License"
Last updateThis incident has been resolved.
We have previously applied a Liferay license fix to all customer's environments. However if a `license.xml` is being deployed from your workspace it can cause a regression. Please remove any `license.xml` file from the Liferay folder (`/liferay/configs/*`) to prevent issues when deploying. DXP Cloud manages licenses and any `license.xml` file committed to your workspace is now expired and a deprecated workspace artefact.
Report: "DXP license expiration warning"
Last updateThe problem has been fixed and customers with running Liferay Service image equal to or newer than liferaycloud/liferay-dxp:*-4.2.0 should not see the Expiration Warning anymore. For older Liferay images, a restart is required.
We have applied a fix and are currently checking projects.
We are reopening this incident as some customers are still receiving License warnings. Engineering team is working to identify the problem. There is no loss in functionality and only admin users receive the warning.
Report: "DXP license expiration warning"
Last updateLicenses have been applied. Liferay service has to be restarted at the earliest convenience to complete the process.
New licenses will be available after the next planned maintenance window (Nov 17th). After that a restart of the Liferay service will be required to apply the changes to environments. We will update the page once the new licenses are deployed.
Plans are in place to update the keys. There is no issue with the usability of the platform and we recommend to use the "Remind me later" option that will disable the popup temporarily. Additional information: The warning is only visible to users with Administrator Role The warning delay is user specific, each admin can choose to hide it, it does not affect the other users’ view By default the warning is postponed by 7 days
We are investigating this warning. There is no impact for customers at this point.
Report: "Deployment Errors"
Last updateThe issue has been resolved.
We discovered an increase in deployment errors. Though live applications should continue to be healthy, this is causing unsuccessful deployments for multiple customers.
Report: "Planned maintenance on Console"
Last updateThis incident has been resolved.
Maintenance is complete. We will monitor the performance now.
We are currently undergoing planned maintenance on the Console to improve the response time and some customers may experience a degraded performance.
Report: "Slow performance on Liferay Cloud Console"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
Problem has been identified, and actions to restore are in place. Affected pages on console are Overview, Activities, Alerts, Builds and Deployments.
We are currently investigating the issue.
Report: "Elevated error rates on europe-west2"
Last updateThis incident has been resolved.
Mitigation work is currently underway. We do not have an ETA for mitigation at this point.
We are working with the cloud provider to resolve the issue
Report: "We are experiencing issues with data storage in some regions"
Last updateThis issue has been resolved and all systems are operational. We appreciate your patience and understanding.
Nearly all customer systems are back online as well as the API and Console applications. We will continue to monitor the situation.
Issue has been identified and we will monitor as all systems are back on line.
We are still actively investigating this issue. Our API and Console applications along with multiple customer applications are severely impacted.
We are investigating the issue with Cloud Provider
Report: "Intermittent issues on logging capabilities"
Last updateLogging functionality has been restored.
The Cloud provider reports elevated ingestion time for logging capabilities. We are currently monitoring.
Report: "Deploy or restart functions are hanging"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Some customers may be experiencing deployment or restart issues. We are currently investigation this.
Report: "Login Errors on console.liferay.cloud"
Last updateLogging into console.liferay.cloud is now fully operational.
An incident has been identified with our identity provider due to a widespread cloud provider outage that is preventing users from logging into console.liferay.cloud.
Report: "Platform wide downtime"
Last updateThis incident has been resolved.
All services are up. Our Cloud provider reported that there might be issues with applying changes to load balancer configuration. We recommend to delay changes, deployments on DXP Cloud until further notice.
The network problem has been identified by the cloud provider and corrective measures have been applied. Currently all Liferay DXP Cloud have been restored. We will continue to monitor the return of the services.
Our cloud provider is experiencing some issues, causing downtimes to the following regions. Investigation work is still underway
Report: "Incident affecting Compute Engines"
Last updateThis incident has been resolved.
Issue has ben resolved and we are monitoring.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
Our cloud provider is experiencing some issues, causing downtimes to the following regions. Mitigation work is still underway
Report: "Login Timeouts on Console"
Last updateLogin error rates have completely subsided and the DXP Cloud Console should be fully accessible now.
Our authentication provider is experiencing some issues, causing timeout errors when users try logging into the DXP Cloud Console. We are actively engaged on the issue and will provide further status updates as soon as we have more information.
Report: "Deployments are Failing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue that leads to Deployment Failures.
Report: "Missing Backup page in DXP Console"
Last updateThe issue has been solved.
A hot fix has been deployed and the Backup page in DXP Cloud console is now functional. We will keep monitoring this issue to address any further incidents.
We are currently working on a problem on the Backup page in DXP Cloud console for customers that have a non Liferay Cloud image (custom image). The fix should be released soon and no additional effort will be needed from customers. We are sorry for this inconvenience and we are always working to avoid this kind of incident.
Report: "Authentication Error"
Last updateAccess to console and API has been restored. No extra action is needed.
We have located the issue and are currently working on a fix.
We are currently investigating an issue with login in to DXP Console.
Report: "OpenVPN Timeout Error"
Last updateIn our recent release, the OpenVPN timeout error has been resolved. If you have any continuing issues, please disconnect and reconnect your VPN.
We have discovered a known issue where VPN's using the OpenVPN protocol will timeout on all connections if the VPN is restarted. This can be caused by updating values and configurations on the VPN settings. A fix for this issue should be released in the following days. In the meantime, we suggest refraining from making updates the your VPN service. If you need to make an update to your VPN or find your VPN timing out on all connections, please open a support ticket and we will work to restore the connection of your VPN.
Report: "Pod Sandbox Changed Error"
Last updateNew nodes were launched with upgraded software that should eliminate the issue. To avoid further disruption, these new nodes were created in parallel to the old ones, and the migration to new nodes will be complete once you perform a redeploy of your services. If this issue is occurring in your environment, please perform a redeploy.
We’ve observed a subtle increase in these rare events of pod sandbox changed errors. Many have been confirmed through customer reports as well so we are currently investigating to see if there’s any correlation between them and to fix the root cause. Impact: This error ultimately causes a self-healing restart of the service. With High Availability (HA) environments, this should cause very minimal downtime as requests will be routed to the healthy instance. With Standard (STD) environments, this restart will cause a 5-10 minute downtime to occur as the single instance will not be available as it is being restarted. Details: A Docker process can become unstable or crash. Those events can happen due to low-level activities on the underlying infrastructure like network errors, system cleanup, unanswered requests, or other operating-system issues. It is the role of the self-healing capabilities of the Kubernetes system to spot crashed containers and automatically restart them. In this case, the Docker process crashed, Kubernetes noticed the problem, killed the broken process, and restarted a new container. In setups without High Availability, during the restart process, the application will be unavailable. In HA setups, you may notice minimal downtime. Workaround: Currently, there is no workaround for this issue. We are working diligently to resolve this issue and prevent any similar issues from appearing in the future.
Report: "Certificate Expiration"
Last updateThis incident has been resolved.
The fix has been applied and is now propagating. All users should now be able to log in. If any issues regarding log-in persist, please create a support ticket.
A certificate related to our platform login operation has expired. This is blocking users from logging in with a warning from their browser. We are actively working to resolve the issue and will provide an update soon.
Report: "MySQL Read-Only Error"
Last updateThe read-only error experienced with your DB is related to an issue with the legacy HA mode for Cloud SQL. Cloud SQL checks the availability of the master instance using different checkpoints to know if the instance is healthy and in particular, one of them is the writability of the disk; in case the master is not ok because of the writability, the traffic will be routed to the failover replica causing the state of the DB to be "read-only. The long-term solution is to set up an HA mode that does not use an explicit failover replica. The new HA replication mode has some added benefits such as increased replication performance and will have no cost implications for our customers The Liferay team will coordinate with customers to provide a detailed path to perform this migration. Meanwhile, should the behavior happen the recommended workaround is to restart the Liferay service.
Some customers are experiencing an error that limits their database service to read-only operations. The root cause of this error is related to a change in the underlying HA configuration by our infrastructure partner. This error can be fixed by restarting the Liferay Service. We are working to prevent this from happening again and new information will be posted soon. If you have questions or need further assistance, please open a Support ticket.
Report: "Errors Creating External HTTP(S) Load Balancers"
Last updateThis incident has been resolved.
One of our cloud infrastructure providers has reported an issue creating external HTTP(S) load balancers. This could cause errors or delays when deploying to an environment for the first time. Below is the related incident: https://status.cloud.google.com//incident/cloud-networking/21003
Report: "Search Cluster Errors During Initialization"
Last updateThis incident has been resolved. For information regarding deploying/restarting the Search service in your production environment, please read the previous updates to this incident. We appreciate your patience and understand throughout this incident. If you have any remaining questions or issues, please open a Support ticket.
This issue should be completely resolved for all environments, except for production environments where the Search service was deployed between Feb 11, 2021, 1:31:12 PM PST and Feb 18, 2021, 1:50:53 PM PST. If you did not deploy the Search service to your production environment within this time frame, your environment should be completely fixed of this issue and you can continue normal operations. If you deployed within this time period, we have released a fix that will take effect on your next deployment or when you restart the search service (no version upgrade is required), which you can do at your earliest convenience. DO NOT delete the search service or rename the search data volume BEFORE deploying/restarting. If you choose not to deploy/restart for some time, your Liferay and search service will continue operating smoothly, so you can deploy/restart when you are ready. To summarize, if you performed a deployment of your Search service to Production within this time period, simply perform a regular deployment/restart at your earliest convenience. If you have any concerns, please open a Support ticket.
The initial fix which should resolve the issue from occurring for the majority of customers has been successfully completed. For customers that have clustered Search services and that renamed their volume, we still suggest that you continue to temporarily avoid deployments while we conduct further tests. For all other customers, you are safe to proceed with your deployment operations. Further updates will be provided as new information is available. If you have questions or issues, please open a Support ticket.
We have identified the issue and are beginning to roll out a fix for all customers. For customers whose Search service was restored by following the support *instructions, we will create replica shards for primary shards that do not have replicas in their search service. This is necessary to ensure that Elasticsearch indices are successfully replicated across nodes during only the next rolling update. We will be conducting a rolling update of the search service in order to realize this solution. No customer action is required. We suggest that all customers temporarily avoid deployments to affected environments until the rollout of the solution is complete. We will post another update here when this process has concluded (approx. 4 hours). *https://help.liferay.com/hc/en-us/articles/360056704432
We have noticed an increase in errors when initializing search services, including an unhealthy search cluster status and unsuccessful deployments. We believe this is due to a Volume mounting regression in a recent release. Our engineers are actively working on a non-disruptive resolution. We have prepared a workaround for this issue so you can remove the cluster errors from your environment. Please see this documentation for details regarding the workaround: https://help.liferay.com/hc/en-us/articles/360056704432. In case of any issues or questions, please contact our support team by opening a ticket.
Report: "Activation Key Expiration Warning"
Last updateDXP Cloud has implemented a new licensing mechanism within the platform that will resolve the licensing message behavior. This new release will keep licenses updated with a valid DXP Cloud subscription. A mandatory maintenance window to implement this fix will be scheduled to start on December 29th, 2020, with all projects updated by January 8th. Some non-production environments may see the fix implemented prior to the scheduled maintenance. To implement this fix within your project prior to the maintenance, please either restart or re-deploy the Liferay service. This action is required in order to update to the new licensing model and avoid this licensing message.
We are continuing to work on a fix for this issue.
Some DXP Cloud customers have noticed a license key expiration warning in their Liferay instances today. A fix is being implemented. More information can be found in the article: https://help.liferay.com/hc/en-us/articles/360054550091-DXP-Cloud-Activation-Key-Expiration-Warning