Historical record of incidents for Amperity
Report: "Amperity is experiencing performance issues in AWS with the query service"
Last updateThe query service of the Amperity platform in AWS are experiencing performance issues. We're working on the problem and expect it to be resolved shortly.
Report: "Amperity is experiencing an outage"
Last updateMany parts of the Amperity platform are currently experiencing an outage. We’re working to fix the problem as quickly as we can. We’ll share another update shortly.
Report: "Amperity AI Assistant Outage - Resolved"
Last updateBetween 4:30 PM PT on June 5th and 10:30 AM PT on June 6th, the Amperity AI assistant was unavailable. This issue has now been resolved. We’re taking steps to improve our monitoring and recovery processes to prevent future disruptions. Thank you for your patience, and please don’t hesitate to contact our Support team if you have any questions.
Report: "Amperity is tracking an issue with tik-tok campaign executions in AWS"
Last updateAll failed workflows have been retried and the incident is now closed.
We are continuing to monitor for any further issues.
Amperity has identified an issue with tik-tok campaign executions in AWS. The issue was identified as having started 20 hours ago, and the cause has been fixed. We are monitoring and restarting any failed workflows. No action is required from our customers at this time.
Report: "Amperity is experiencing performance issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Some parts of the Amperity platform are experiencing performance issues. We're working on the problem and expect it to be resolved shortly.
Report: "Amperity is experiencing an outage"
Last updateThis incident has been resolved. If you have any questions, please submit a ticket to Support and we will follow up. Thank you for your patience.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Most parts of the Amperity platform are experiencing an outage. We’re working to fix the problem as quickly as we can. We’ll share another update shortly.
Report: "Amperity is experiencing a partial outage"
Last updateWe’re back! Amperity should be up and running normally. Thanks for bearing with us.
A fix has been implemented to recover orchestration scheduling and we are monitoring the results.
We're experiencing an issue with scheduling Orchestration jobs which may impact our customers on Azure. We're working on the issue and expect to resolve it shortly.
Report: "Amperity is experiencing a partial outage"
Last updateAmperity services are functioning as expected and this incident is resolved. Azure support continues to work in mitigation of the issue.
Amperity has restored our services and all workflows are working as expected. Azure support is continuing to address their outage and we are in close communication with their team.
Azure support has identified the issue and is working on the mitigation.
We are continuing to investigate this issue.
Azure is having an outage that is impacting Amperity. We're working with Azure support on the issue and expect to resolve it shortly.
Report: "Amperity is experiencing a partial outage"
Last updateWe have implemented a fix and restarted the failed queries and workflows, and the system is operational.
We're experiencing an outage that may impact some customers. The impact is to Queries, Segments and Campaigns. We're working on the issue and expect to resolve it shortly.
Report: "Amperity is experiencing a partial outage"
Last updateAzure Support has resolved the issue and Amperity is functioning normally.
A workaround has been implemented to restore functionality, while Azure Support continues to work on the permanent resolution.
Azure has acknowledged this as a widespread issue impacting multiple customers and is working on resolving the issue.
We're experiencing an incident caused by an Azure storage outage that may impact some customers in the US East 2 region. We're working with Azure Support on the issue and expect to resolve it shortly. The following are impacted: Queries Customer360 Streaming Ingest Campaigns
Report: "Amperity is experiencing a partial outage in Workflow processing"
Last updateOur monitoring shows that all the Workflows that were stuck have completed successfully, Workflows started after the fix was deployed are also running to completion and the system is healthy.
We have deployed a hotfix and are monitoring.
We're experiencing an outage that may impact Workflow processing for some customers. We have identified the potential cause and are working on resolving the issue.
Report: "Amperity is experiencing performance issues"
Last updateWe have confirmed that the issue has been resolved. Thank you for your patience and if you have any questions, please contact Support
Some parts of the Amperity platform experienced performance issues due to a backlog of query executions. We have identified the issue and has been resolved. We will continue to monitor throughout the evening.
Report: "Amperity is experiencing performance issues related to the Azure platform"
Last updateThis incident has been resolved. We apologize for the inconvenience. If there are any issues or questions please reach out to Amperity Support and we will follow up.
Some parts of the Amperity platform are experiencing performance issues due to an Azure incident. Amperity is working with Azure to resolve the incident and will provide updates shortly.
Report: "Amperity is experiencing degraded performance in Azure"
Last updateThe application functionality has been fully restored and all failed workflows have been restarted. We apologize for the inconvenience and if you encounter any issues, please submit a Support ticket.
The affected services have been recovered and the application functionality has been restored. We are monitoring for further interruption. If your workflows were impacted, those will be restarted.
The performance of some internal Amperity services degraded, and as a result some application features are unavailable on our Azure platform, including workflow management, source configuration, in-app notifications, and queries. Our Platform Engineering teams have identified the services impacted, and are working to restore those services along with any impacted workflows. We will update this incident as we work to restore all features of the Amperity application.
Report: "Amperity is experiencing an outage of AWS"
Last updateAWS has resolved the incident and all Amperity services are fully functional. We apologize for the inconvenience and if you encounter any issues, please submit a Support ticket.
AWS has identified the issue and has applied a resolution and we will continue to monitor.
Most parts of the Amperity platform are experiencing an outage. We’re working with AWS to fix the problem as quickly as we can. We’ll share another update shortly.
Report: "Ongoing Azure incident impacting Amperity performance"
Last updateWe’re back! Amperity should be up and running normally. Thanks for bearing with us.
A fix has been implemented and we are monitoring the results.
We are currently experiencing a degraded state of operations due to a global networking outage at Azure. In response to this outage, we have transitioned to our failover data center and are observing favorable progress. Our team is actively monitoring the actions Azure is taking to resolve this issue. We are also monitoring our customers' workflows and taking appropriate actions to resolve any failures. We understand the impact this may have on your business, and we sincerely apologize for any inconvenience caused. We appreciate your patience and understanding during this time. If you have any further questions or concerns, please don't hesitate to reach out to our support team, who will be happy to assist you.
Report: "Amperity not affected by Microsoft outage"
Last updateWe are aware of a Microsoft outage affecting Windows PC. Please be assured that Amperity is not affected, but we are monitoring the situation and will update as necessary.
Report: "Amperity is experiencing performance issues"
Last updateThe issue has been resolved and if you were affected, you may receive backlogged email notifications from the last 24 hours. We apologize for the inconvenience and appreciate your understanding.
We are experiencing an issue with our mail provider which is affecting our ability to send email notifications for customer managed workflows. We are working on the resolution and will provide an update shortly
Report: "Ongoing Azure incident impacting Amperity performance"
Last updateAzure has resolved the incident but Amperity will continue to monitor the situation. Thank you for your patience and for any questions please reach out to our Support team.
Our team is actively monitoring the actions Azure is taking to resolve this issue. We are also monitoring our customer workflows and taking appropriate actions to resolve failures. We understand the impact this may have on your business and we sincerely apologize for any inconvenience caused.
We are currently experiencing a degraded state of operations due to an outage at Azure. As a result, services that rely on heavy compute will be running slower than their usual performance levels during this period. Our team is actively monitoring the actions Azure is taking to resolve this issue. We are also monitoring our customer workflows and taking appropriate actions to resolve failures. We understand the impact this may have on your business and we sincerely apologize for any inconvenience caused. We appreciate your patience and understanding during this time. If you have any further questions or concerns, please don't hesitate to reach out to our support team, who will be happy to assist you.
Report: "Azure is currently experiencing a incident which is affecting Amperity"
Last updateThe incident has been resolved and Amperity will continue to monitor.
Amperity has mitigated the issue and will continue to monitor.
The incident is continuing and Amperity is working with Microsoft to resolve the incident.
Amperity has failed over from the impacted Azure region and monitoring the resolution.
Some parts of the Amperity platform are experiencing performance issues. . Amperity is working to mitigate this issue while Azure resolves their issue.
Report: "Workflows delayed across all platforms"
Last updateThis incident has been resolved.
A fix has been deployed and all workflows have been restarted.
All workflows are currently delayed across all Amperity platforms. The issue has been identified and our Platform Engineers are in the process of deploying a fix. All impacted workflows will be restarted.
Report: "Elevated error levels in Amperity on the Azure platform"
Last updateThis incident has been resolved.
Our Platform Engineering team has restarted the impacted services. We will continue to monitor our systems until we are confident we are fully recovered.
An internal service issue on our Azure platform is causing elevated error rates in Amperity. In addition, new workflows may fail to start. Our Platform Engineering team is investigating the issue and we will update this incident as the root cause is identified.
Report: "Incident impacting workflows on AWS"
Last updateNo further impacts have been identified in the course of our monitoring, and we will now be marking this incident resolved.
We are continuing to monitor for any further issues.
All impacted workflows have been unstuck and workflows are proceeding nominally. Our Platform Engineering team will continue to monitor the system and we'll resolve this incident once we've identified the root cause.
Our Platform Engineering team is restarting the impacted services and some of the affected domain transforms are beginning to succeed. We will continue to monitor the situation and update this incident as we work to resolve the issue.
On our AWS platform, a number of workflows have become stuck waiting for custom domain transforms to be saved, just prior to the start of Stitch. Our Platform Engineering team is investigating the issue and working to determine a mitigation strategy to restart those workflows. We will continue to update this incident as we work to resolve the issue.
Report: "Partial outage impacting some in-app features on AWS"
Last updateOur Platform Engineering team has identified the root cause of this incident, and a hotfix has been deployed to prevent it from reoccurring in the future.
All impacted services have been restarted, and all impacted workflows have been restarted. The partial outage lasted from approximately 1:37PM to 1:53PM PST. Our Platform Engineering team will be closely monitoring the system while they investigate further and identify a fix for the root cause.
Some internal Amperity services were unavailable, and as a result some application features are unavailable on our AWS platform, including workflow management, source configuration, in-app notifications, and orchestrations. Our Platform Engineering has identified the services impacted, and is working to restart those services along with any impacted workflows. We will update this incident as we work to restore all features of the Amperity application.
Report: "Internal configuration service outage on AWS platform caused partial outage"
Last updateFrom 12:37PM PST to 13:05PM PST, an outage of the internal configuration management service for the Amperity application was unavailable on our AWS platform. As a result, many features on the Amperity application were temporarily unavailable. Our Platform Engineering team were alerted to the issue and service was promptly restored. All impacted workflows have been restarted.
Report: "Partial outage impacting some in-app features on AWS"
Last updateBetween 9:50 AM PST and 10:40 AM PST, some internal Amperity services were unavailable, and as a result some application features were unavailable on our AWS platform. The outage impacted workflow management, data source configuration, in-app notifications, and orchestrations, as well as the ability to save some kinds of related configuration. Our Platform Engineering team responded promptly to the incident, and service has been restored. They are currently evaluating root cause, and will take further actions to ensure that the issue doesn't reoccur. All impacted workflows have been restarted successfully.
Report: "Planned Azure Platform Maintenance May Impact Amperity"
Last updateAzure's maintenance window closed at 0500 PST this morning with no further impacts to Amperity.
No further impact has been noted to Amperity as a result of Azure's planned maintenance. We continue to closely monitor our systems. Azure's maintenance window closes at 05:00 PST on October 31st, at which point we will close out this incident if they do not further extend the timeline.
At about 3:22 PM PST, our systems alerted that Azure had rebooted one of our database instances. Our systems recovered quickly, and all systems are now functioning normally. As there may be additional reboots for other instances, we will keep this notification open until Microsoft notifies us that maintenance is complete.
Sometime between 21:00 UTC on October 30th and 12:00 UTC on October 31st Microsoft may be taking down one of Amperity's core database instances for brief maintenance. During that brief time, some workflows may fail and some parts of the Amperity application may be unavailable, but workflows will be automatically restarted when service is restored. Once our systems have been restarted and/or when Microsoft announces that their maintenance is complete, we will close out this incident.
Report: "Errors orchestrating to Facebook destinations"
Last updateAfter a fix executed by Meta, impact on Amperity has been mitigated. We are declaring this resolved.
After prompts from Amperity's Platform Engineering Team, Meta has identified a root cause for the observed orchestration errors. We will continue to monitor the situation and provide updates accordingly.
The Amperity Platform Engineering team continues to monitor the situation and has sought additional guidance from Meta. We will post updates as we learn more.
The team continues to monitor the ongoing situation. We will post additional updates as we learn more.
The Amperity Platform Engineering team continues to wait on guidance from Meta. We will post additional updates as we learn more.
The Amperity Platform Engineering team continues to monitor the situation as we wait on guidance from Meta. We will post additional updates as we learn more.
The Amperity Platform Engineering team continues to wait on guidance from Meta. We will post additional updates as we learn more.
The Amperity Platform Engineering team is continuing to observe additional failures; we are waiting on guidance from Meta. We will post additional updates as we learn more.
Our Platform Engineering team has identified an increase in errors while orchestrating to Facebook for both destinations and campaigns. Failed orchestrations may be retried but may still fail. Amperity has reached out to Meta Developer Support, and we are awaiting additional guidance or root cause.
Report: "Ongoing AWS Incident Impacting Ad Hoc Queries"
Last updateWhile the AWS incident is still not marked closed, the impacts on Amperity seem to have been mitigated and we are declaring this incident resolved.
Amazon Web Services continues to work towards resolving the issues that are impacting queries on the Amperity platform. Positive improvements have been noted but their issues are not yet fully resolved, so we will keep this incident open until we are confident that things have returned to normal. Our Platform Engineering team continues to monitor their status.
Amazon Web Services is continuing to take steps to address resource contention within the AWS subsystems that are impacting queries in Amperity. Our Platform Engineering team is continuing to monitor the situation.
Amazon Web Services has taken steps to address resource contention within the AWS subsystems that are impacting queries in Amperity. Our Platform Engineering team is monitoring the situation and is beginning to see some signs of improvement.
An ongoing incident within Amazon Web Services is impacting ad-hoc queries running within the Amperity platform for our AWS customers. Queries that are part of scheduled workflows will be automatically retried, but users issuing ad-hoc queries may encounter failures. Those failures are transient and frequently succeed upon retry. The Amperity Platform Engineering team is monitoring the AWS incident and we will continue to provide updates here as we learn more.
Report: "Delayed Data to Google Ads"
Last updateAll impacted delays have completed successfully, and this incident has been resolved.
Data transmission to your Google Ads systems is currently experiencing a delay. We are actively working to address this issue and will provide updates as soon as we have resolved the delay.
Report: "Failing Orchestrations and Campaign Sends to Facebook"
Last updateAll impacted workflows have completed successfully, and this incident has been resolved.
Facebook connections appear to recovering and we're successfully updating audiences. Our Platform Engineering team is monitoring all recovered workflows to ensure they complete successfully.
At approximately 8:02AM PDT, our platform detected a failure in orchestrations and campaign sends using our Facebook connector. Couriers and other connections to Facebook are working normally. Our Platform Engineering team is investigating the issue and has requested assistance from Meta. This incident affects tenants on both AWS and Azure. We will provide updates as we gather more information and will retry all failed orchestrations once connectivity is restored.
Report: "Emergency Maintenance: Internal cluster upgrades"
Last updateMaintenance was completed with no customer impact.
Our Platform Engineering team will be performing some emergency maintenance to scale up our Aurora cluster in our AWS environment. We do not expect there to be any notable impacts and we will close this incident once the work has been completed.
Report: "Storage issue impacting a limited number of Stitch workflows in Azure"
Last updateThis morning, beginning at about 7:30AM PDT, a table storage issue was discovered that caused a limited number of Stitch workflows to fail on the Azure platform. The impacted tables were restored by our Platform Engineering team and those workflows were restarted successfully.
Report: "AWS connectivity issue caused some ingest workflows to fail"
Last updateAll workflows have been restarted and we are closing out this incident.
At approximately 7:50AM PDT, our Platform Engineering team identified that some ingest workflows were failing due to an internal connectivity issue in Amazon Web Services. Our team has restarted the impacted services and we are restarting and monitoring failed workflows at this time.
Report: "Amperity is experiencing an incident which is affecting some workflows"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Amperity Azure SFTP server down for emergency maintenance"
Last updateEmergency maintenance has concluded and clients may now connect as usual.
In order to perform emergency maintenance, the Amperity Platform Engineering team has shut down the SFTP server used by our customers on Azure. We expect this action to take approximately 30 minutes, and we will post additional updates once the work has concluded. Customers on AWS will be unaffected.
Report: "Ongoing issue identified with Microsoft Azure Storage Accounts"
Last updateThe fix has been verified as working, and all affected workflows have been restarted.
A fix has been identified and we are in the process of restarting affected workflows.
We are currently experiencing high read latencies impacting our customer tenants that make use of Microsoft Azure storage accounts. Our team is investigating and we have engaged with our partners in Azure support to identify the root cause of the issue. This incident impacts active managed workflows only for customers on the Azure platform.
Report: "Our third party partner Snowflake is experiencing a partial outage of their service"
Last updateThis incident has been resolved.
Snowflake has implemented a fix and Amperity will continue to monitor.
Snowflake continues to work on the fix to resolve this incident.
Snowflake has identified the issue and is implementing a fix.
Report: "Amperity is experiencing a partial outage"
Last updateThis incident has been resolved.
All impacts have been mitigated and our Azure platform is fully operational
We have identified and resolved the issue that was impacting our Azure customers, and we're addressing any impacts to workflows that were running at the time. We will post additional updates until the issue is fully resolved.
We're experiencing an outage that may impact the ability of customers in Azure to make changes within the application. We're working on the issue and will continue to post updates as we work on a resolution.
Report: "Amperity is experiencing performance issues"
Last updateAmperity has confirmed that the mitigation put in place by Microsoft has resolved the issue.
MSFT has confirmed that they have mitigated the outage and Amperity is seeing improvements. Amperity will continue to monitor until we have high confidence of resolution
Microsoft has identified the issue as an outage on one of their regions and is working on resolving the issue. While Microsoft works to resolve their outage, Amperity will move the affected clusters to a productive region. Workflows may still be slow to start or may run slower than normal.
Amperity is actively working with Azure engineering to determine the source of the issue. Workflows may still be slow to start or may run slower than normal.
Some parts of the Amperity platform are experiencing performance issues. We're working on the problem and expect it to be resolved shortly.
Report: "Partial Outage in Azure: Workflows Impacted"
Last updateThis incident has been resolved. Our engineering team will be performing an After Action Review to determine root cause and any necessary follow-up.
A fix has been implemented and workflows should now be running. We are monitoring to ensure that everything has restarted properly.
We are continuing to investigate this issue.
Our engineers are investigating an incident impacting workflows within our Azure platform. We will provide a status update within the next hour.
Report: "Amperity is experiencing performance issues"
Last updateThis incident has been resolved.
Some parts of the Amperity platform are experiencing performance issues. We're working on the problem and expect it to be resolved shortly.
Report: "Queries and Segments unresponsive"
Last updateThis incident has been resolved.
Queries and Segments are currently unresponsive. A fix is currently being deployed.
Report: "(Azure only) Amperity is experiencing a partial outage"
Last updateThe mitigation has proved successful and the incident has been resolved by Amperity and our upstream provider. Thank you for you patience, understanding and partnership.
We continue to monitor the mitigation that was implemented and thus far, the mitigation is showing positive results. We will provide an update by 9:30PM PST.
We have identified the cause of the issue and have implemented a mitigation. We will monitor the effectiveness of the mitigation and provide an update by 8PM PST or earlier, depending on the outcome of the mitigation. Thank you for your patience and understanding.
Amperity is currently experiencing a partial outage due to an outage in our upstream cloud provider (Azure). We are attempting to implement a mitigation.
Amperity is currently experiencing a partial outage due to an outage in our upstream cloud provider (Azure). We are attempting to implement a mitigation.
Amperity is currently experiencing a partial outage due to an outage in our upstream cloud provider (Azure). We are attempting to implement a mitigation.
Amperity is currently experiencing a partial outage due to an outage in our upstream cloud provider (Azure). We are attempting to implement a mitigation.
Amperity is currently experiencing a partial outage due to an outage in our upstream cloud provider (Azure). We are currently investigating the issue.
Report: "Select queries may fail to run in Azure."
Last updateFrom 1:23pm through 2:18pm today a small number of queries may have failed to run in our Azure environment. The underlying issue has been addressed and systems are again fully functional.
Report: "Workflows and Notifications are currently degraded for AWS customers."
Last updateThis incident has been resolved.
The issue has been resolved and impacted workflows are being restarted and monitored. Some notifications may not reflect the correct state of a task - please see the workflow page if you have questions about the status of a particular workflow.
While many workflows are progressing, the team has identified an issue which is causing some workflows and all notifications to fail in our AWS environment. The team has identified the root cause of this issue and is currently working on a fix.
Report: "Facebook Integrations are currently disabled"
Last updateThis incident has been resolved.
We are are currently experiencing integration issues with our Facebook connector for all customers and we are working with Meta on possible resolutions.
Report: "Platform Provider Network Outage"
Last updateThe issue has been resolved at our platform provider and all systems are fully functional once again.
We are currently investigating an issue with a platform provider that may be causing intermittent application issues.
Report: "Amperity is undergoing emergency maintenance in Azure"
Last updateThe maintenance in Azure has been completed and the platform is fully operational.
Amperity's Azure environment is undergoing emergency maintenance in response to repeated network instability. During this time you may experience degraded performance or a brief interruption in service.
At 3:00 PT Amperity services will be undergoing emergency maintenance in response to repeated network instability within our Azure environment. During this time, you may experience degraded performance. The estimated time of completion is 4:00 PT.
At 3:00 PT Amperity services will be undergoing emergency maintenance in response to repeated network instability within our Azure environment. During this time, you may experience degraded performance. The estimated time of completion is 4:00 PT.
Report: "Login partial outage"
Last updateThis incident has been resolved.
This issue is now resolved.
We are currently experiencing an issue with our login provider - users may experience some issues logging into the Amperity application.
Report: "Amperity app in Azure is currently experiencing issues"
Last updateThe underlying issue in the infrastructure has been resolved.
We are continuing to monitor the results from the fixes we've applied.
We're continuing to see normal functionality in the application and are working towards addressing the root cause in partnership with Azure.
We are continuing to see normal functionality in the application and workflows. We are still working to address the underlying root cause.
We are continuing to work on a fix for the underlying root cause of instability. The application and workflows are continuing to function normally.
We're continuing to investigate the root cause of the issue in our Azure infrastructure; the Amperity application and workflows are continuing as normal.
We've mitigate the immediate issue and the Amperity platform is working, including any running workflows. We're continuing to try to identify the root cause.
The issue has been identified and a fix is being implemented.
We are investigating an issue affecting the Amperity application in Azure.
Report: "Amperity app in Azure is currently experiencing issues"
Last updateThis issue has been resolved.
A fix has been implemented and we're monitoring for any additional issues.
We are investigating an issue affecting the Amperity application in Azure.
Report: "We are investigating an issue with workflows in our Azure application."
Last updateThis issue has been resolved.
A fix has been implemented and we're monitoring for any additional issues.
We are currently investigating this issue.