Is Piano Software Down Right Now? Check if there is a current outage ongoing.

**Incident explanation:** * Following the deletion of a site requested by a customer, one dataflow export was automatically deactivated. Only one customer was affected by this issue. ‌ **Actions done:** * The site has been removed from the export, and the export has been reactivated. Missing data were resent. ‌ **Actions to do:** * A redesign of the export planning system, currently underway, will prevent this from happening again in the future.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available.We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products.Please excuse us for any inconvenience caused.

The issue has been identified and fixed.

Currently, previewing templates is not working on sandbox.piano.io.

The issue has been identified and fixed.

Currently, previewing templates is not working on sandbox.piano.io.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

This incident has been resolved.

An update has been released for the VX and ID double opt-in deactivation flow in the EU environment. This update addresses an issue where some users were not removed after failing to confirm their accounts. As a result, there may be a larger than usual drop in user counts while the backlog is processed. This change does not affect applications that are not using the double opt-in functionality. It will also not affect users who have already successfully opted in or are within the window to opt in. Please reach out to support@piano.io if you have any questions.

This incident has been resolved.

An update has been released for the VX and ID double opt-in deactivation flow in the EU environment. This update addresses an issue where some users were not removed after failing to confirm their accounts. As a result, there may be a larger than usual drop in user counts while the backlog is processed.This change does not affect applications that are not using the double opt-in functionality. It will also not affect users who have already successfully opted in or are within the window to opt in.Please reach out to support@piano.io if you have any questions.

**Incident explanation:** <ul><li>Some imports caused a processing lag.</li></ul> **Actions performed:** <ul><li>The issue was fixed.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Incident explanation:** <ul><li>A data warehouse sizing issue with our provider caused delays in processing D-1 data.</li></ul> **Actions performed:** <ul><li>The processing has been relaunched.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and all scheduled exports have been sent.

We are currently experiencing a delay in data generation for yesterday on Piano Analytics. Scheduled exports may be still pending to be sent. Our technical teams are actively working to resolve the issue. Please excuse us for any inconvenience caused.<br><br><b>Affected customers </b>- Less than 1%

**Incident explanation:** <ul><li>A service went under maintenance for a security update which caused a processing lag.</li></ul> **Actions performed:** <ul><li>No actions performed.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a delay in real-time processing on Piano Analytics products. Please excuse us for any inconvenience caused.

**Incident explanation:** <ul><li>A property that changed type in the data model blocked the processing.</li></ul> **Actions performed:** <ul><li>The property issue was fixed.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

Piano Insight is experience a production incident. Our engineers are investigating the issue.

RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2025/05/2025-04-23-RCA-Activation-cross-services-issue.pdf).

This incident has been resolved.

A fix has been applied and the team are now monitoring the results.

We are continuing to investigate this issue.

Piano's US environment is currently experiencing an incident where clients may experience errors in the Composer dashboard and VX access check is failing. Outside of access checking, Composer experiences are executing normally.

Today between 01:47 and 08:41 UTC Piano Amplifier (Socialflow) experienced an incident where approximately 50% of publishing media, like images and videos, to X (Twitter) were not working. The issue was identified and resolved. Root cause: Twitter replaced their upload.twitter.com certificate on May 1st at 21:22 UTC. Our workers didn't reload the certificate and some of the forks were still using the old one. To fix the issue, Piano restarted the Twitter workqueues to reload the new certificate.

This incident has been resolved.

Our traffic is back up and the team are monitoring the situation. The services were affected between 07:45 - 07:56 UTC and 08:20 - 08:33 UTC. The cause is related to a 3rd party incident which is described here: https://health.aws.amazon.com/health/status#ec2-ap-northeast-1_1744705274.

Piano VX is currently experiencing issues in the AP region. Clients may notice API errors, delayed responses and inability to access https://dashboard-ap.piano.io. We are investigating the issue.

Piano experienced dashboard login issues between 07:54 and 10:16 UTC. The problem was isolated to SSO users with dedicated links, whereas the Piano login page was functioning normally. The issue was identified and resolved.

Our developers have found the solution and are running the batches for the articles that missed processing. The backlog batch should take 2-3 days to complete. The new articles shouldn't have any issues. If you continue to face any issues, please reach out to support@piano.io

The issue has been identified and a fix has been implemented. The search index is in the process of recovery, which may take up to several hours to fully complete.

Previously the issue was limited to the EU region, but the other regions are affected, too. Our engineers are investigating the issue.

The issue has been identified and a fix has been implemented. The search index is already in the process of recovery, which may take up to a couple of hours to fully complete.

We are currently experiencing an issue where the search index is not being updated for some clients. This may result in Piano Content returning older articles as backfill. Our engineers are investigating the issue.

Between 6:50:31 - 6:52:12 UTC VX /publisher endpoints experienced an elevated level of errors. The issue was identified and resolved, all endpoints are now functioning correctly.

There was an incident on Insight/DMP data collection between 2025-03-04 12:40 and 12:48 (UTC). Traffic data was partially lost during that time.

The issue has been resolved. After monitoring, we have identified no negative after effects.

Our devs have implemented the fix, and we are monitoring the results.

Piano Insight is facing a lag in the real-time traffic display. Our engineers are currently investigating the issue

This incident has been resolved.

We are continuing to monitor for any further issues.

A mitigation has been put in place by the email service provider. The email processing in the EU region is back to full working behavior.

Piano ESP email delivery may be affected in the EU region due to an incident on the email service provider side. The incident can be monitored here: https://status.mailgun.com/incidents/b60dthm3t8gk

**Incident explanation:** * We had an issue with a data visualization component. The last release of this component integration was unstable. **Actions performed:** * The last release has been rolled back and the pending exports have been unlocked and sent. **Actions to come:** * We will investigate on the origin of the problem.

The issue has been resolved and all exports have been sent.

We are currently experiencing delays in the generation of some Dashboard exports. Our technical teams are working to resolve this issue so that these exports may be sent as soon as possible. We apologise for any inconvenience caused.

**Incident explanation:** <ul><li>One service was encountering some errors.</li></ul> **Actions performed:** <ul><li>The service was rebooted.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and all exports have been sent.

We have fixed the issue. We are now working on sending exports.

We are currently experiencing delays in the generation of some Dashboard exports. Our technical teams are working to resolve this issue so that these exports may be sent as soon as possible. We apologise for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

**Incident explanation:** <ul><li>One service was encountering some errors.</li></ul> **Actions performed:** <ul><li>The service was rebooted.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and all exports have been sent.

We are currently experiencing delays in the generation of some Dashboard and Explorer exports. Our technical teams are working to resolve this issue so that these exports may be sent as soon as possible. We apologise for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

Between the hours of 08:01 UTC - 10:39 UTC, there were major network connectivity issues observed between our regional data centre in Japan and our main data centre in Europe. As a result, some of the following services were affected for customers in the APAC region: - Some API requests failed completely - UI unavailable

A fix has been implemented and we are monitoring the results.

/traffic APIs are failing with 500, caused by network issues between our data centers.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

The issue was identified and resolved by 09:15 UTC.

Publisher dashboard login https://dashboard.piano.io/ is unavailable since 09:02 UTC, we are investigating the issue.

**Incident explanation:** <ul><li>On Saturday, January 25 there was a network outage in our Tokyo Data Center. As a result we lost most of ARI events for the Japanese customers between 4 PM JST and 11 PM JST.</li><li>The outage was somewhere on our service provider's network. We were unable to control it.</li></ul> **Actions performed:** <ul><li>The network outage has been resolved.</li></ul> **Actions to come:** <ul><li>No more action needed</li></ul>

The issue has been resolved and the delay in real-time has now been caught up.

We have experienced an issue in data collection for some Japanese customers on Ad Revenue Insights. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- Less than 1%

ESP experienced issues within email preparation logic between 05:00 UTC and 12:45 UTC in the EU region. This caused significant delays of up to 6 hours in email distribution. The issue has been identified and resolved, all emails were sent successfully.

**Incident explanation:** <ul><li>An issue prevented users from reaching our interface.</li></ul> **Actions performed:** <ul><li>The last release has been rollback</li></ul> **Actions to come:** <ul><li>No more action to do.</li></ul>

The issue has been resolved and all our tools and interfaces are fully available.

We are currently experiencing availability issues with our tools and interfaces. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Incident explanation:** <ul><li>Slowdown on one of our processing services.</li></ul> **Actions performed:** <ul><li>Restarting the service</li></ul> **Actions to come:** <ul><li>No more action</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Incident explanation:** <ul><li>A component update took longer than expected and caused a delay in data collection for some customers.</li></ul> **Actions performed:** <ul><li>Service restart after update.</li></ul> **Actions to come:** <ul><li>No more action needed</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. No data was lost. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data collection on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Incident explanation:** <ul><li>Authentication issues on our interfaces following a network configuration change.</li></ul> **Actions performed:** <ul><li>Rollback on the network configuration change.</li></ul> **Actions to come:** <ul><li>No more action to do.</li></ul>

The issue has been resolved and all our tools and interfaces are fully available.

We are currently experiencing availability issues with our tools and interfaces. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Incident explanation:** <ul><li>Some properties were sent empty in the VX events.</li></ul> **Actions performed:** <ul><li>The issue was fixed.</li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved. The properties are alimented again in the events.

The root cause of the issue has been identified, and our technical team is actively working on its resolution. This issue affected the properties of the following events: vx.conversion, vx.conversion.offsite, transaction.refund.offsite, subscription.action, and subscription.action.offsite. We apologize for any inconvenience caused.

Since December 23rd, experience properties (ID, name) have been missing from vx.conversion events, causing the "Experience Turnover" metric and all metrics derived from it to show 0. This impacts most Out-of-the-Box Boards, which rely on these metrics. Our technical team is actively working on a fix. We apologize for any inconvenience.<br><br><b>Affected customers </b>- Less than 5%

Some applications may be experiencing issues with VX Flow report where the data is either incorrect or missing for the period between 16-26 December. Our team have identified and fixed the cause. No data was lost and will be reinstated. Missing data will be fixed by December 30th, incorrect data will be fixed in the second week of January.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2024/12/2024-12-12-Piano-VX-checkout-issues.pdf).

The incident has been resolved.

Rollback for both regions has been completed. The team will continue to investigate the root cause.

We are continuing to investigate this issue.

Piano VX checkout is experiencing an incident where some users are not able to complete payments. The issue seems to be related to a recent release, which is being rolled back.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

**Incident explanation:** <ul><li>There was a cache problem on a geolocation instance.</li></ul> **Actions performed:** <ul><li>A fix has been released.</li></ul> **Actions to come:** <ul><li>No more action needed.</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. <br><br><b>Affected customers </b>- All customers

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified that the /access/check endpoint returns empty results even for the users who have entitlement to the resource. As a result, users cannot access the content that is entitled to them.

**Incident explanation:** <ul><li>An access right issue prevented users from reaching our interface.</li></ul> **Actions performed:** <ul><li>The access issue was fixed. </li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and all our tools and interfaces are fully available.

We are currently experiencing availability issues with our tools and interfaces. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Incident explanation:** <ul><li>Real time delay on data processing (data collection part).</li></ul> **Actions performed:** <ul><li>Increase the number of servers and restart the service.</li></ul> **Actions to come:** <ul><li>No more action needed.</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

C1x segmentation and propensity model scoring experienced a significant disruption between 13:05 and 13:11 UTC. The issue was identified and resolved. The team is currently investigating the root cause.

This incident has been resolved.

We experienced a brief delay of approximately 15 minutes in displaying traffic data in DMP Insight. A fix has been implemented, and we are currently monitoring the results. Based on our current information, there was no loss of traffic data. After the planned maintenance was completed, our systems came back online and had a backlog of paused, queued up events to resume processing. Some parts of our system did not recover 100% automatically, and this resulted in a delay in ingesting the old, as well as the new, real time events. Once our engineers manually recovered all of those affected system components, the backlog of events were quickly processed and the traffic reports in Insight looked correct.

We are continuing to investigate this issue.

The team are currently investigating the issue.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2024/12/2024-12-02-RCA-CX-InsightAPI-down.pdf).

This incident has been resolved.

Starting at around 08:50 UTC this morning, we experienced an issue with our main database. This caused new incoming events to be unprocessed and queries to fail. Our engineers identified the problem and deployed a fix. After the fix was implemented, the queued events began processing and feeding into the database, and the systems returned to normal operations at around 11:30 UTC. Based on our current knowledge, there was no data loss during this period.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented. Impact: - No traffic in Insight Dashboard - Cxense API endpoints response with 500 error - Low availability for a customer cube partition across all data centers

We are currently investigating this issue.

**Incident explanation:** <ul><li>Delay in real time data processing due to a value problem in a virtual site mapping table.</li></ul> **Actions performed:** <ul><li>Fix deployed to unblock the situation</li></ul> **Actions to come:** <ul><li>Another fix will be deployed to resolve the problem once and for all</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused.<br><br><b>Affected customers </b>- All customers

**Explanation:** * Real time delay on data processing due to a certificate expiration of one of our cloud providers **Actions:** * Manual server installation without using the package manager **Actions to come:** * Investigate the origin of the incident with the concerned provider

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

Between 07.13 and 07.20 UTC Piano VX experienced an elevated level of errors on the access endpoints. The following endpoints were affected: /api/v3/access/list /api/v3/access/check /api/v3/publisher/user/access/list /api/v3/publisher/user/access/check /api/v3/access/token/list The issue has been identified and resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

Piano Cxense experienced an outage between 08:05 and 08:23 UTC. A number of Cxense subsystems, including access to the dashboard, were down. Composer experience executions in AP, AU and EU regions were also affected. The issue was identified and resolved. No data was lost and all operations have now resumed to normal.

On November 27, 2024 Piano ESP experienced an incident during which email sending was delayed in the AP region. The incident lasted between 02:00 and 23:30 UTC, the issue was identified and resolved. No emails were lost during the incident, email sending was resumed and the functionality returned to normal operations.

**Incident explanation:** <ul><li>We had a node issue on one of our servers.</li></ul> **Actions performed:** <ul><li>A replacement was made and fixed the issue. </li></ul> **Actions to come:** <ul><li>No more actions to come.</li></ul>

The issue has been resolved and the delay in real-time has now been caught up. All scheduled exports have been sent.

We are currently experiencing a delay in real-time data processing on Piano Analytics. Scheduled exports will be sent once the data is ready. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused. <br><br><b>Affected customers </b>- All customers

The issue has been fixed and all new reports are processing normally. Unfortunately, all reports that were scheduled prior to the fix will not process and should be re-triggered manually.

Piano VX is currently experiencing issues with generating Subscription log report. The issue has been identified and is currently being fixed.