Is FMX System Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved. Again. We sincerely apologize for this interruption and any inconvenience this may have caused.

A fix has been implemented and we are now monitoring.

The main application should be available now and we are still working through the other services

We deployed a critical hotfix to address an issue in our system. While this deployment was designed to occur without downtime, an internal misconfiguration inadvertently triggered maintenance mode. We have identified the issue and are actively working on remediation steps. We sincerely apologize for this interruption and any inconvenience it may have caused.

Earlier today, we deployed a hotfix to address a previously identified calendar-related issue. Inadvertently, additional changes were included in the deployment, which resulted in the calendar returning errors and being temporarily inaccessible for approximately 3-4 minutes. The issue was quickly identified and resolved, and full functionality has been restored. We apologize for any inconvenience this may have caused and are reviewing our deployment processes to prevent similar occurrences in the future.

Earlier today, we deployed a critical hotfix to address an issue in our system. While this deployment was designed to occur without downtime, an internal misconfiguration inadvertently triggered maintenance mode, resulting in approximately 5-10 minutes of unexpected downtime. We sincerely apologize for this interruption and any inconvenience it may have caused. We plan to implement measures to prevent this from happening in future deployments and are reviewing our processes to ensure greater reliability moving forward.

This incident has been resolved.

We have implemented a fix and are continuing to monitor.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We're currently investigating reports of customers that are receiving a 502 error while logging into Mapping and Mobile.

Through our investigation we have identified a subset of customers that were potentially affected by this issue. If you are one of the customers that we have identified, our Customer Success team will be reaching out to you via phone or email. What we know at this time is that we experienced multiple hardware failures that required us to roll back a subset of our databases to an earlier restore point. Any data that was added, edited, or deleted between 10:00pm EDT, and 11:00pm EDT on 10/22/2024 for these identified customers has been corrupted and our Customer Success team will be reaching out to you to answer any further questions. We are still in the process of investigating alternative remediation steps and will be conducting an internal postmortem to better understand and prioritize future infrastructure changes and processes to remove the possibility of this ever happening again. We know that any amount of data corruption can be a problem, and we apologize for any inconvenience this issue has caused.

On 10/22/2024 at approximately 10:30 PM EDT FMX experienced a server failure that required manual intervention to restore it. On 10/23/2024 at 4:10 AM EDT we had successfully restored the affected databases and are continuing to investigate the extent of data that was not restored between 10 and 11 pm EDT

The incident has been marked as resolved by Microsoft.

We have received reports that some users can't access their site. Microsoft has reported that there is an issue between their cloud and AT&T customers. They're currently looking into the issue. You can follow their status page https://azure.status.microsoft/en-us/status for more details. If available, you can try switching over to a mobile hot spot.

We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **The root cause of the outage** On the evening of July 18th, we discovered that the FMX platform was unavailable. After investigating we determined that the cause of this was due to an entire region outage at the Microsoft Azure datacenters across the entire US-Central region. You can read more about it [here](https://azure.status.microsoft/en-us/status/history/) from the Azure team. Tracking ID: `1K80-N_8` **Solution** To solve the problem, we simply monitored the situation and waited for the Azure remediations to be implemented. **Future mitigations** On July 30th, we conducted our own internal postmortem discussing future mitigation tactics that we can employ to reduce our reliance on an individual region or cloud provider, improve communication, and improve our monitoring capabilities. This discussion aligned very closely with the [Azure Well-Architected Framework](https://learn.microsoft.com/en-us/azure/well-architected/). ‌ Regards, FMX Team

This incident has been resolved by our cloud service provider. We will continue to keep an eye out for any further issues. Again, our sincere apologies for any service disruption.

Our cloud provider has deployed a fix that appears to have resolved the current incident. We'll continue to monitor the situation and provide updates as they happen.

We are seeing sites come back online. We're continuing to monitor the incident to ensure our cloud service provider's mitigation is successful.

According to our cloud provider they've `determined the underlying cause and are currently applying mitigation through multiple workstreams. ` We're continuing to monitor the situation and will provide updates as they become available.

We are continuing to monitor and follow our cloud providers status pages and will provide more updates as they become available.

Our Cloud Service Provider is experiencing an outage. We will update this message as we have more details. We apologize for the inconvenience.

This incident has been resolved.

A fix has been implemented to resolve PDF generation errors. We are continuing to monitor site performance.

We are currently investigating degraded site performance and errors when generating PDFs. We will update our status page with additional details as they become available. We apologize for any inconvenience.

This incident has been resolved.

A fix has been implemented, and we're monitoring the results.

We have identified a fix for this issue. We are working on testing the fix now. We'll update this message once our fix passes testing and is out for release. We still anticipate that reporting dashboards should be up in a couple of hours.

We have identified an issue with our reporting dashboards. We are investigating solutions for a fix. We anticipate that reporting dashboards should be up in a few hours. We'll update this message as we progress on this issue. We apologize for the inconvenience.

This incident has been resolved. Application performance should be back to normal.

The application is back online. However, we may see degraded performance until our system can recover from the issue. We will report back when our application performance is back to normal. We apologize for the inconvenience.

We are continuing to investigate an issue related to our servers. We will report back when we have more information. We apologize for the inconvenience.

We are currently investigating an issue related to our servers. We will report back when we have more information. We apologize for the inconvenience.

This issue is now resolved. We apologize for any inconvenience this issue has caused.

We are rolling out a fix now. We expect full service to be returned in 20 minutes or less. We apologize for any inconvenience this issue has caused.

We have identified an issue that is impacting users ability to resolve work requests. We have identified a fix and are working to implement it. We will respond back on our status page with a timeline shortly. We apologize for any inconvenience this issue has caused.

This issue has been resolved. Users that experienced the issue before may need to "hard refresh" the login page by holding the shift key and clicking the browser's refresh button if they're still experiencing the issue. We apologize for any inconvenience this issue has caused. We recommend bookmarking a direct link to your FMX site and using it as your primary login method in the future, as it is the most direct path to your site. Your direct link is <your hostname>.gofmx.com (example.gofmx.com). If you don't know your hostname, you can find a direct link by clicking any link back to FMX through any email notification. From there, you will be prompted to log in. You can also contact the FMX Support Team at support@gofmx.com for assistance.

We are aware of an issue impacting the universal sign-on function on our marketing website https://www.gofmx.com Users who attempt to sign on through our marketing website will receive a page not found error. Please note that you can still sign on to your FMX site via a direct link, <your hostname>.gofmx.com (example.gofmx.com). If you don't know your direct link, you can find one by clicking any link back to FMX through any email notification. From there, you will be prompted to log in. You can also contact the FMX Support Team at support@gofmx.com for assistance. Once you've identified your direct link to your FMX site, we recommend bookmarking it and using it as your primary login method in the future, as it is the most direct path to your site. We apologize for any inconvenience this issue has caused and will update this message when the universal sign-on service is restored.

This incident has been resolved. This outage was the result of a bug that our Cloud Service Provider, Microsoft Azure, has in the service offering that FMX consumes. Microsoft Azure has assured us that they are currently testing a fix for this issue. We apologize for the inconvenience!

We're currently investigating a site outage that began at 2:40 pm EST. Our typical responses to this known issue with our Cloud Service Provider do not seem to have had an impact. We'll report back as we have more information. We apologize for the inconvenience!

This incident has been resolved. This outage was the result of a bug that our Cloud Service Provider, Microsoft Azure, has in the service offering that FMX consumes. Microsoft Azure has assured us that they are currently testing a fix for this issue. We apologize for the inconvenience!

We're currently investigating a site outage that began at 1:12 pm EST. Our typical responses to this known issue with our Cloud Service Provider do not seem to have had an impact. We'll report back as we have more information. We apologize for the inconvenience!

The FMX application experienced a 40-minute outage starting at 7:55 pm and ending at 8:33 pm EST. This outage was the result of a bug that our Cloud Service Provider, Microsoft Azure, has in the service offering that FMX consumes. Microsoft Azure has assured us that they are currently testing a fix for this issue. We apologize for the inconvenience!

This incident has been resolved. We'll continue to monitor the application for any issues. We apologize for the inconvenience!

We're currently investigating a site outage that began at 5:44 pm EST. Our typical responses to this known issue with our Cloud Service Provider do not seem to have had an impact. We'll report back as we have more information. We apologize for the inconvenience!

Our Cloud Service Provider, Microsoft Azure, has a bug in their service that caused an outage from 11:30 PM EST to 3:30 AM EST. We were delayed in resolving the issue, due to the lateness of the hour and the manual nature of the fix. We are now working on automating the recovery process that Azure has advised us on so that any future incidents of this problem will recover automatically. Azure is in the process of preparing their permanent fix for this bug for deployment. We are continuing to work with them on any possible escalations to their deployment process. We will update our status page once we have confirmed that Azure's fix was deployed. We deeply apologize for the inconvenience.

This incident has been resolved.

Our Cloud Service Provider, Microsoft Azure, has a bug in their service that is causing this, the other brief outages we've had, and the 9/12 outage. We detected the issue at 9:28 AM EST, and are implementing the temporary fix that Azure has advised us on, and we expect that service should return shortly. Azure is in the process of preparing their permanent fix for this bug for deployment. We are continuing to work with them on any possible escalations to their deployment process. We will update our status page once service returns and again when we have confirmed that Azure's fix was deployed. We apologize for the inconvenience.

Our Cloud Service Provider, Microsoft Azure, has a bug in their service that caused both this and 9/12 outage. We detected the issue at 2:09, implemented a temporary fix that Azure has advised us on, and the service returned at 2:18. Azure is in the process of preparing their permanent fix for this bug for deployment. We will update our status page once we have confirmed that the fix was deployed. We apologize for the inconvenience.

Our Cloud Service Provider, Microsoft Azure, has a bug in their service that caused this, the brief outage yesterday, and the 9/12 outage. We detected the issue at 3:00, implemented a temporary fix that Azure has advised us on, and the service returned at 3:10. Azure is in the process of preparing their permanent fix for this bug for deployment. We are continuing to work with them on any possible escalations to their deployment process. We will update our status page once we have confirmed that the fix was deployed. We apologize for the inconvenience.

This incident has been resolved.

FMX is back online. We are still investigating this issue and are in contact with our Cloud Service Provider. We've updated our status to degraded performance as we continue to monitor the situation. We will update this page with more information as we have it. We apologize for the inconvenience!

We are still investigating this issue and are in contact with our Cloud Service Provider. We will update this page with more information as we have it. We apologize for the inconvenience!

We are still investigating this issue but are upgrading its status to a major outage. We will update this page with more information as we have it. We apologize for the inconvenience!

We are currently investigating this issue. We will update this page with more information as we have it. We apologize for the inconvenience!

We've determined that the fixes that we introduced last week have solved the acute performance issue impacting our application. We are prioritizing additional application performance improvements to help alleviate any ongoing general performance problems. We apologize for the inconvenience!

We are continuing to investigate this issue. We are still not currently seeing any timeouts related to the issue, but we are continuing to monitor whether our fix is fully effective and are continuing our investigation into any other possible causes. Please continue to reach out to support@gofmx.com with any issues you encounter. We'll continue to update this post as we have more information.

We are continuing to investigate this issue. We are not currently seeing any timeouts related to the issue, but we are continuing to monitor whether our fix is fully effective and are continuing our investigation into any other possible causes. Please continue to reach out to support@gofmx.com with any issues you encounter. We'll continue to update this post as we have more information.

We are continuing to investigate this issue. We have released an adjustment to our fix to account for additional scenarios. We are monitoring whether our fix is effective and are continuing our investigation for any other possible causes. Please continue to reach out to support@gofmx.com with any issues you encounter. We'll continue to update this post as we have more information. We apologize for the inconvenience!

We are continuing to investigate this issue. We are seeing a reduced number of timeouts since our last hotfix, but we need to adjust our fix to account for additional scenarios, and we're working on that now. Please continue to reach out to support@gofmx.com with any issues you encounter. We'll continue to update this post as we have more information. We apologize for the inconvenience!

We are continuing to investigate this elusive issue and look for solutions. We implemented a fix for one possible cause around 4 pm EST today. We are monitoring whether that fix was effective and are continuing our investigation for any other possible causes. We'll update the post as we have more information. We apologize for the inconvenience!

We are investigating an issue that is causing intermittent timeouts for our application. We are in the process of adding additional logging to determine the cause. We'll update the post as we have more information. We apologize for the inconvenience!

This incident has been resolved.

FMX is completing an investigation to discover if any of the services we provide or third-party services we use are impacted by the Apache log4j vulnerability, log4shell. We have verified that the data you’ve entered into the FMX application (<hostname>.gofmx.com) is not impacted by log4shell. We do use a third-party service to provide our reporting dashboards. Like many services, they are impacted by log4shell. However, the part of this service impacted by the vulnerability is well guarded in the FMX infrastructure and is only accessible by FMX employees. Thus, we consider it highly unlikely for this vulnerability to ever have been exploited. Some of the internal services we use to provide customer support, sales, and marketing services do contain organizational-level customer data and are impacted by the vulnerability. Many of our vendors have since mitigated the vulnerability and we expect the rest to do so very soon. We are continuing to monitor our vendors’ responses and if we become aware of any unauthorized access to this organizational-level data, we will notify any impacted customers without delay. Organizational-level data can include organization name and contact information, contract value, product-use data, meeting notes, customer support conversations, etc. Updates will be posted to our status page as any additional information becomes available. Please reach out to support@gofmx.com with any questions. For more information about log4shell please review CVE-2021-44228 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228) and the Apache Log4j2 (https://logging.apache.org/log4j/2.x/security.html) post.

A fix has been implemented and we are currently monitoring results. Please reach out to support@gofmx.com if you are still unable to access your reporting dashboards.

Reporting dashboards have been partially restored at this time. You may still be experiencing issues loading dashboards on your site. We will update periodically as more information is available.

The issue has been identified and a fix has been implemented. Reporting dashboards may be offline for a few hours. We apologize for the inconvenience. We will update the status of this incident periodically and as we have more information.

We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **The root cause of the outage** On the morning of February 9th, we discovered that the page response time for the FMX application was higher than normal. For reference, “page response time” refers to the length of time it takes to load an individual page in an application. After investigating we determined that the cause of this high page response time was due to a lack of available hardware in the Microsoft Azure datacenter where our application is hosted. At any given time, FMX can tap into additional server capacity available in our datacenter to handle the load of increased traffic. On Tuesday morning as our traffic began to increase, we were unable to autoscale to a higher number of servers because there was no additional hardware available for us to scale up to.  **Solution** To solve the problem, we manually removed all of our servers and redeployed our application in a new cluster of servers. This allowed us to access additional hardware elsewhere in Azure. During this period, users were unable to access FMX at all. Once the application was fully re-deployed, users were once more able to access the application, and the application’s page response time was once more back in its normal range. **Future mitigations** After we confirmed the root cause of the outage, we alerted Microsoft to the problem and asked for additional guidance. They alerted us that although this problem is quite rare, it is possible that we may run into it again. For context, we have used Azure as our hosting provider for 8 years and have only encountered this issue once.  * Microsoft provided us with some advice on how to mitigate the problem without taking the application offline should we encounter it again.  * Additionally, we have added alerts for when an autoscale failure occurs so that we can respond more quickly to the problem should it ever recur.  * Lastly, we’re currently exploring options to improve our page response time during the time it would take for us to mitigate this problem if it reoccurs. Once more, we deeply apologize for this outage and we will be taking steps to ensure that in the unlikely event we have this problem in the future, that we will limit the amount of disruption as much as possible. ‌ Regards, FMX Team

This issue has now been resolved. Normal site performance should be restored. We will continue to monitor the situation closely.

A fix has been implemented and we're currently monitoring the results. We will post a post-mortem when we have more details about the cause of the issue. We apologize for any inconvenience.

We are continuing to investigate this issue. We apologize for the inconvenience.

We are currently investigating an issue that is causing degraded performance. We apologize for any inconvenience.

The universal FMX log in page (https://www.gofmx.com/login/) was not allowing users to log in. Organization-level log in pages were unaffected.

We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **Root cause of outage:** On 8-26 we discovered high priority bug in our application and determined that it would be disruptive enough to our users that we should deploy mid-week fix. On the evening of 8-26, we attempted to deploy this fix twice and were unsuccessful both times. We later determined that there was a problematic change included in this deployment. During the second deployment attempt, a rare set of circumstances followed: Windows Update restarted the build server mid-deployment, resulting in Azure, our hosting service, continuing the deployment in the background without our knowledge. As designed, Azure refused to deploy the broken app to live servers due to health probe failures \(a protective measure to prevent deploying bad code\). After experiencing an hour of health probe failures and having received no additional cancellation attempts from us, Azure, according to its design, deployed the broken application. **Contributing factors:** We rely on alerts to inform us of application and feature outages outside of our core hours from 8 am to 6 pm EDT. While we received alerts, they were in email and instant messaging format. This, coupled with the overnight timing of the outage, delayed the time to respond. **Future mitigations:** As a result of this issue are taking the following actions: * We are setting up a tiered, automated, calling service when alerts are issued so that we ensure team members are aware of them. * When we cancel a deployment we will now verify that the corresponding cloud service deployment is actually cancelled, to protect against this rare set of circumstances in the future. Once more, we deeply apologize for this outage and we will be taking steps to ensure it does not happen again. Regards, FMX Team

This incident has been resolved. We greatly apologize for the inconvenience!

A fix has been implemented and we're monitoring the results. Service should now be returning.

The issue has been identified and a fix is being implemented. We expect a return of service in 1 hour. We greatly apologize for the inconvenience.

We are currently investigating this issue. We greatly apologize for the inconvenience!

Reporting dashboard service has been restored. We apologize for the inconvenience!

We are again extending our expectation for return of server to 4:00 PM EDT due to performance bottlenecks encountered during the recovery process. We apologize for the inconvenience and thank you for your continued patience.

We're extending our expectation for return of service to 3:00 PM EDT due to performance bottlenecks encountered during the recovery process. We apologize for the inconvenience and thank you for your continued patience.

A fix has been implemented and we are monitoring the results.

We're currently experiencing a reporting outage that began early this morning. We're working diligently to resolve the issue and expect a full return of service by 12:00 PM EDT. We apologize for the inconvenience.

We were impacted by a outage of our Cloud Service Provider, Microsoft Azure this morning at 12:30 AM. Microsoft Azure is investigating the issue and will be deploying a mitigation. We apologize for the inconvenience. Please view Microsoft Azure's status page for more information: https://status.azure.com/en-us/status/history/

We encountered a brief outage. We were able to resolve the issue within moments but we know that we likely caused some inconvenience and for that we apologize.

This issue has been now been resolved and full functionality should now be restored. Once more, we apologize for the inconvenience.

We have identified the issue and are deploying a fix. We expect full service to be restored in approximately 90 minutes if not sooner. Again, we apologize for the inconvenience.

We are currently investigating this issue. We apologize for the inconvenience. We will provide updates on this page as we work towards a resolution.

This incident has been resolved. Thank you for your patience.

We are currently investigating an issue related to our reporting dashboards. We will update this page when we have more information. We apologize for the inconvenience.

This incident has been resolved.

We are currently investigating this issue. We will update this page as we gather more information. We apologize for the inconvenience.

We experienced a brief outage of our reporting dashboard. This incident should now be resolved

This incident has been resolved.

It appears as if application performance is returning to normal. We're continuing to monitor the situation to ensure that performance remains level.

We identified an issue related to our database this morning that is causing slow response times across our application. We've implemented a fix but performance will require some time to fully recover. We apologize for the inconvenience and will update this page when the issue is fully resolved.

This incident has been resolved.

We are experiencing a outage of our reporting dashboards. We are in the process of implementing a fix that will return service. We expect full service to return in a few hours. We greatly apologize for the inconvenience.

This incident has been resolved. We greatly apologize for the inconvenience.

We are experiencing a outage of our reporting dashboards. We are in the process of implementing a fix that will return service. We expect full service to return in a few hours. We greatly apologize for the inconvenience.

We are now seeing full service return for reporting dashboards. If you experience any further difficulties please contact support@gofmx.com. Once again, we greatly apologize for the inconvenience.

Identified - We are experiencing a outage of our reporting dashboards. We have discovered that the source of the problem is likely with the third-party service we use to deliver reporting dashboards. We are in the process of implementing a fix that will mitigate the issue and are following up on an open support request to the service. We expect full service to return in a few hours. We greatly apologize for the inconvenience. We will be searching for another third-party service to deliver dashboards in the future.

We are now seeing full service return for reporting dashboards. If you experience any further difficulties please contact support@gofmx.com. Once again, we greatly apologize for the inconvenience. We are taking measures to prevent this issue from happening again.

We are experiencing a recurrence of the same reporting issue as yesterday. We have discovered that the source of the problem is likely with the third-party service we use to deliver reporting dashboards. We are in the process of implementing a fix that will mitigate the issue and are submitting a support request to the service. We expect full service to return in a couple hours. We greatly apologize for the inconvenience.

This issue appears to be resolved. We will continue to monitor the situation closely to make sure that there are no recurrences. We sincerely apologize for any inconveniences.

A fix has been implemented and dashboards appear to be back online for all customers, but we will continue to monitor the issue to make sure that the service remains up.

We are continuing to work on a fix for this issue.

We believe we have identified the issue and are working to implement a fix. We should know in roughly 2 hours if that fix was successful. Once again, we apologize for any inconvenience to all those affected by this outage.

We are currently investigating an issue with our Reporting dashboards. This issue is causing some dashboards to be offline. We greatly apologize for any inconvenience this issue may be causing and will release updates to this page as we have more information.

This issue appears to be resolved. We are working to implement ways to prevent an issue like this in the future and also to implement some monitoring options to notify us should this situation ever arise again. Once again, we appreciate your patience and apologize for the inconvenience.

We've implemented a fix and are monitoring results. All notifications that were generated during the outage should now be received via email. If any notifications are missing please contact support@gofmx.com. Again, we sincerely apologize for the inconvenience.

We are currently experiencing an outage that is exclusively affecting our automatic notifications. All notifications are still being recorded and will be sent upon the resolution of this outage. We sincerely apologize for any inconvenience this issue may be causing your team. We are working to resolve this issue as swiftly as possible and will continue to update this page as we have more information.

Microsoft Azure Engineers have mitigated the underlying root cause of the issue. We have seen a complete a return of performance to FMX sites. However, we are continuing to monitor the issue and will update our status once we are confident that all sites will maintain normal performance. Please refer to Microsoft Azure's Status page for more updates on this issue: https://azure.microsoft.com/en-us/status/ We apologize for the inconvenience this issue may have caused.

Microsoft Azure Engineers have identified the underlying root cause of the issue. They are applying mitigation steps and are starting to see signs of recovery. We are continuing to seeing a return of performance on many FMX sites. We are continuing to monitor the issue and will update our status once all sites have returned to normal performance. Please refer to Microsoft Azure's Status page for more updates on this issue: https://azure.microsoft.com/en-us/status/ We apologize for the inconvenience this issue may have caused and we will share new updates as soon as they are available.

We are continuing to monitor for any further issues.

We are seeing a return of performance on many FMX sites. We are continuing to monitor the issue and will update our status once all sites have returned to normal performance. Please refer to Microsoft Azure's Status page for more updates on this issue: https://azure.microsoft.com/en-us/status/ We apologize for the inconvenience this issue may have caused and we will share new updates as soon as they are available.

We are continuing to work on a fix for this issue.

This issue is more extensive than we originally thought. We are closely monitoring communications from the third-party platform, Microsoft Azure and are looking for ways to mitigate the issue on our side. We will continue to update our status page with more information as we have additional updates. We greatly apologize for the inconvenience.

We have identified an issue with a third-party platform that we use to deploy FMX. We are currently working to implement a fix that should resolve the issue. We estimate performance to be fully restored in one hour. We will continue to update our status page with more information as we have it.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the application to ensure that the fix was effective.

We've identified a system stability issue related to a bug fix that was introuduced 2-17-2019. The team is currently working to resolve the issue. We expect to have this issue resolved in less than 3 hours and will continue to update this message as progress is made. We apologize for any inconvenience.

This incident has been resolved.

The performance issues have been corrected. We sincerely apologize for the inconvenience this has caused and are exploring options to mitigate this type of risk in the future.

Our hosting provider, Microsoft Azure, has fully restored power to the impacted datacenter. They are currently working on restoring services to full capacity. We are implementing an alternative solution to the issue to provide relief in the event that Azure services are not restored in the near future.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We're experiencing a system stability issue that's causing poor performance and slow response times. The issue is related to an outage that our hosting provider, Microsoft Azure, is currently experiencing. We're actively working with Microsoft to resolve the issue and will post updates as soon as possible.

An interaction between our automated deployment process and a misconfigured customer resulted in a system outage. The issue was remedied quickly when noticed early the next morning. Steps were taken to increase transparency around system issues and this status page was created as a result.