Historical record of incidents for Agility CMS
Report: "Agility Instance Access Outage"
Last updateThis incident has been resolved
A fix has been implemented and we're monitoring the results
We're currently investigating this issue and it's scope
Report: "Brief Delay to API Content Updates"
Last updateWe experienced a brief delay with updates to API content globally. We identified the issue and released a fix at 4:32PM ET.
Report: "[Resolved] Scripts added to Rich Text fields may not be coming through when published"
Last updateThe issue has been resolved and a fix has been released
We are currently investigating an issue where scripts that have been added to content via a Rich Text Field may not be coming through when published.
Report: "[Resolved] API Content Not Updating for Some Customers"
Last updateWe had an update to optimize our dynamic pages with large lists that introduced an issue with some customers. We have implemented a fix and are continuing to monitor.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "[Investigating] Degraded Performance with Internal Cloud Storage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented to mitigate this issue and we are continuing to monitor.
We believe we've identified the issue and are working on a fix
We are currently investigating this issue, some customers may receive a 503 or 401 error
Report: "Currently investigating an issue with our FetchAPI"
Last updateAll services have been restored and should be functioning normally
We are currently investigating an issue with our FetchAPI, please stay tuned for updates. We appreciate your patience as we investigate.
Report: "Fetch API - issue with nested dynamic pages not appearing in API"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the issue.
We have identified the source of the issue and a fix is being implemented.
We are investigating an issue where nested dynamic pages are not appearing in the API. We will have an update soon on a fix for this issue.
Report: "An Azure issue may be causing dropped connections or slow response times to Agility services in the USA"
Last updateThis incident has been resolved.
An Azure issue may be causing dropped connections or slow response times to Agility services. We are monitoring this and will continue to provide updates.
Report: "An Azure issue may be causing dropped connections or slow response times to Agility services."
Last updateThis incident has been resolved.
An Azure issue may be causing dropped connections or slow response times to Agility services. We are monitoring this and will continue to provide updates.
Report: "Login Page Not Loading"
Last updateThis incident has been resolved.
Out login provider has addressed the issue and the login page is working again. We will continue to monitor the issue.
We are currently investigating an incident with our login provider where the login page is not loading. Users who are already logged in are not affected.
Report: "Intermittent Connectivity Errors - Content Manager - All Regions"
Last updateOur authentication provider is reporting operational status and we are considering this issue to be resolved.
Our authentication provider has engineers investigating the issue on their side. We are continuing to monitor.
We have isolated the issue to be with our authentication provider, who is not reporting an outtage. We are following up with them and will continue to monitor.
We are getting intermittent connectivity reports from our monitoring tools regarding the Content Manager App. While the apps are currently responsive in our own tests, and the reports are coming back negative now, we are continuing to monitor.
Report: "Fetch API - Changes not coming through (USA)"
Last updateThe fix for this issue has been deployed and the issue has been resolved.
We have identified the source of this issue and engineers are working on a fix in case this scenario happens again. In the meantime, changes are going through again as normal.
We are continuing to investigate this issue.
We are investigating an issue where updates to the Fetch API are not coming through in the US region for some customers.
Report: "Intermittent Errors accessing CDN assets and API requests in Europe"
Last updateDue to an incident with Azure in the West Europe data region, some Agility customers in the region experienced increased latency and/or errors accessing assets or API requests from a period starting at 3:30 AM EST to 5:40 AM EST. This issue has been resolved at the data region level by Azure.
Report: "Content Manager Issues - Classic and New UI"
Last updateThis incident has been resolved.
The fix has been rolled out and we are actively monitoring.
We have identified the issue and a fix is being rolled out.
Some customers are experiencing intermittent connectivity issues. We are actively investigating this.
Report: "Azure CDN Cert Errors (cdn.agilitycms.com)"
Last updateThis incident has been resolved and validated globally.
A fix has been implemented and we are monitoring the results.
The issue has been identified a fix is being implemented by Microsoft on our behalf.
We have had confirmation of this issue with Microsoft. Expect an update within the hour on this incident.
We are continuing to investigate this issue with Microsoft.
We are currently investigating an issue with our CDN Service over Azure affecting the cdn.agilitycms.com domain. We have engaged Microsoft to help us solve this.
Report: "Content Manager intermittent connection issues"
Last updateWe have observed that all components are functioning properly. The services that were reporting errors have completed diagnostics and all services are working as normal.
The connection issues have stabilized. We are actively monitoring all layers of the platform as we seek more answers on the connection issues.
We are getting reports of intermittent connection issues to the content manager in multiple regions. Our team is actively investigating.
Report: "CDN Origin Errors - Europe"
Last updateThis incident has been resolved.
The issue has been identified and a fix has been implemented.
We are investigating an issue with our CDN service in Europe.
Report: "Content Manager - Canada: Experiencing login delays and timeouts"
Last updateThis issue has been resolved.
We are continuing to monitor this issue.
Our engineers are working on this issue and have observed that the system is now performing optimally again. We are continuing to monitor.
We are investigating an issue regarding login delays and timeouts on the Content Manager in Canada.
Report: "Investigating reports of connection timeouts in the Content Manager, USA"
Last updateThis incident has been resolved.
We have identified the issue with a serverless resource and have re-allocated the resource.
We are current investigating reports of connection timeouts in the Content Manager, USA.
Report: "Issue with Azure network connectivity in Europe affecting some customers."
Last updateThe Azure team has confirmed that they have mitigated this issue.
We have observed no issues with our European customers in more than 60 minutes, however, as Azure support is continuing to report this as a potential issue, we are continuing to monitor.
Azure support has confirmed that their team has identified the issue and is working on a fix. Our team has confirmed that currently all Europe instances are online, but we are continuing to monitor.
We are currently working with Azure to resolve an issue with connectivity to back-end data resources.
Report: "Log In Issues"
Last updateThis incident has been resolved.
We have confirmed that this is an issue with our login provider. We are currently working with them to resolve it.
Some users are experiencing issue when logging in. We are investigating this with our login provider.
Report: "Performance Degradation on US Content Manager"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue causing performance degradation on our US Content Manager.
Report: "Issues with Updating Content"
Last update## Incident On October 4th, early in the morning, our systems reported that content was taking longer than normal to update. The outage prevented the latest content from being available through the Agility Content Fetch API. This resulted in content updates not being reflected on customer websites and apps that use this API. ## Cause The cause of the content processing delay was identified as a software defect in the logic around processing specific content events. ## The Fix A fix was implemented within hours to resolve the issue and successfully past our quality assurance. It was rolled out to all of our customers automatically. ## Outcome We were able to resolve the software defect within hours of the outage being detected and have implemented a fix to ensure that case cannot happen again.
This incident has been resolved.
We are currently investigating some issues regarding updating content on our platform.
Report: "Elevated Error Rates from our Authentication Provider"
Last updateThis incident has been resolved.
We are currently monitoring the issue and awaiting more information from our authentication provider
Some customers are experiencing elevated error rates from our authentication provider. We are actively monitoring the situation and will provide updates immediately as they are available.
Report: "Content Fetch (REST) API calls are not returning a contentLinkDepth greater than 5"
Last updateThe API update to protect against over-fetching has been rolled-back and will be re-scheduled at a later time. We will notify all affected customers before releasing this.
We have identified that this affects a very small subset of customers and identified all API calls that are affected. Since this could be a breaking change we have decided to rollback this update until our customers have confirmed they have updated how they call the API.
We have released a patch to improve the stability of our API to address some data over-fetching scenarios. If you are calling the Fetch API with a contentLinkDepth of 5 or greater, you may notice the response of the API has changed to limit the amount of data returned by the API.
Report: "Subset of Canadian customers cannot Sync classic .NET sites"
Last update## Incident After a security update was applied on June 17th, we received some reports from a subset of our Canadian customers that they could not synchronize content to their websites via our classic .NET integration. ## Cause A networking security update was applied to our environment which enforced rules that applications in our US datacenter could no longer communicate with databases in our Canadian datacenter. ## The Fix If you are experiencing any sync failures since June 17th and you are using a classic .NET integration, check your `web.config` or `appSettings.json` and ensure that the `contentServerUrl` is set to `https://contentserver-ca.agilitycms.com`. ## Outcome We were able to identify all customers who needed to update their settings in their website and have reached out to them and confirmed then have switched over. In the future, we will schedule maintenance windows \(through this Status Page\) and ask customers to subscribe to updates so they are aware ahead of time \(at least 1 week\) of any major changes in our platform that may require them to take action.
A recent security update has brought to light some misconfigured, Canadian classic .NET websites which were incorrectly connecting to the USA region content server. A configuration change is required to the web.config/appSettings.json to use contentserver-ca.agilitycms.com rather than contentserver1503.agilitycms.com
Report: "Increased Errors in connectivity to the CMS"
Last update## Incident On June 17th, between 9-10pm ET some customers in the Canadian region were experiencing elevated errors while performing typical operations within the CMS. ## Cause A networking security update was applied to our environment and had some repercussions to the connectivity between our Canadian data center and our global services. ## The Fix The update was rolled-back immediately, restoring service. ## Outcome Updates have been made to our test environment to better reflect our production environment so that we can ensure future, similar updates can be made with less risk.
Customers in the Canadian Region were experiencing intermittent errors while accessing the manager-ca.agilitycms.com