Historical record of incidents for Panoply
Report: "Trials fail to be created due to failure to create a group"
Last updateEncountered sporadic failure for new trial signups.
Report: "White screen on login and switch between accounts"
Last updateUsers are encountering a whitescreen (no error or anything) when logging in to the platform and/or when switching between accounts.
Report: "Instagram connector down"
Last updateMeta has depreciated some resources and update is needed.
Report: "Partial Data Source Failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We identified that some of our data sources are failing and are not visible in the platform. The root cause was identified and we are working on a fix.
Report: "Inability to access to Redshift accounts through external tools"
Last updateWe’ve resolved the Redshift connectivity issues by updating our SSL certificates for our Redshift data warehouses.
We are currently investigating the inability to access Redshift accounts through external tools. The issue is currently affecting only external connections and the platform and data sources are working are expected.
Report: "Data sources in Redshift account do not ingest data"
Last updateThe incident has been resolved
A fix was deployed and all data sources are expected to properly ingest data. We are monitoring the system to verify that everything is back to normal
We've identified the reason for the inability to ingest data into Redshift account and working on a fix
We are currently investigating the reason for inability to ingest data using data sources on Panoply's Redshift accounts
Report: "Data sources fail to run"
Last updateThe issue was resolved and the data sources collection processes are back to normal
The fix was deployed and we are monitoring the system to verify its stability
We've identified the issue and are deploying a fix to this issue.
We are currently investigating an increase in errors in multiple different data sources
Report: "Data sources failures"
Last updateThe issue was resolved and every failing data source should complete successfully the next time it will run.
We've deployed a fix to the issue and monitoring the status of all the different data sources
The root reason for this issue can be found here: https://github.com/paramiko/paramiko/issues/2038 The team is still working on a solution for that on our end.
Update: This issue is currently affecting the following data sources: Postgres Mongo MsSQL ElasticSearch Jira BigQuery Hubspot MySQL SFTP WooCommerce Netsuite Google Sheets BigCommerce Shopify Google Analytics Google Cloud Storage
We've identified an issue that is affecting multiple different data sources. Our team is currently working on solving this issue and getting all the failed data sources to work.
Report: "Issue with starting jobs"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are encountering issues with our Job System. We are currently investigating this issue.
Report: "Meta (Facebook) data sources are not available"
Last updateAfter reviewing the incedent, unfortunately, we cannot find something to improve in Panoply work. In retrospective there was no issue or problem with the platform and the reason the app was disabled was because the tester did not have valid credentials to the platform. It is very unsettling that FB decided to disable the app and just then inform us of the actual caveat. After the credentials were provided the app was approved within a few hours. People involved - alon oded Mykyta
All four Meta (Facebook & Instagram) data sources are available for creation and collection.
The following data sources are currently not available (users cannot log in to new data sources and existing data sources will fail to collect data): * Facebook Ads * Facebook Pages * Facebook Posts * Instagram Business We are currently working with the Meta (Facebook) team to resolve this issue.
Report: "Data sources cannot be configured"
Last updateWe've fixed the data sources configurations and users can now resume configuring their data sources and adding new ones
We've identified an issue when updating data sources configurations. We are implementing a fix for this issue.
Report: "Google Analytics authentication process"
Last updateWe have fixed the issue and Google Analytics data source would be online shortly
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have disabled the ability to create a new Google Analytics data source. All existing sources are working properly and data is being collected on schedule. We are testing a solution
Report: "Inability to connect Google Data Studio to Panoply"
Last updateDue to a code change implemented by Google in Google Data Studio, Panoply can no longer connect to their BI platform. We are in contact with Google to assess a fix. Please contact the Director of Customer Success (chris@panoply.io) for guidance
We've identified an inability to connect Google Data Studio to Panoply and investigating the issue
Report: "External connections to Panoply"
Last updateThe external connections issues to the Panoply accounts were resolved
We are currently investigating an outage in connections between external tools to multiple different Panoply accounts
Report: "Issues with starting jobs"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Jobs are returning to their normal scheduling behavior. We are continuing to monitor the situation to verify that everything works smooth.
We are encountering issues due to a major outage in AWS itself. We are in contact with AWS representatives and are working to resolve the issue.
Report: "Job system scheduling issue"
Last updateThis incident has been resolved.
The issue was resolved and we are now monitoring our data sources job queue.
We have identified an issue with the job scheduler affecting customer jobs. Because of this we have temporarily disabled the scheduling of new jobs until the issue is resolved. We are working hard to resolve the matter as quickly as possible. Your data is safe and no action is required by the users. Job queue will be ingested once the incident is resolved.
Report: "Data sources page not loading"
Last updateThe issue was resolved and the data sources page is fully functional
A fix was deployed. We're monitoring the data sources page to verify that the issue will not happen again
We've identified the cause of the issue and working on a fix
We are currently investigating the data sources page failure to load
Report: "Issues Triggering Jobs"
Last updateThe issue is resolved and jobs are now scheduled as usual.
We found that our rate of triggering jobs is lower than usual. We are currently investigating this issue.
Report: "Text ingestion issue"
Last updateThe issue is resolved. A fix was deployed and verified.
Cases identified where after ingestion textual fields are saved as a date value instead of the text. Our investigations show this only affects texts like "XXXX,YYYY" where X can be any letter and Y can be a letter or a number. There can be any number of characters before and after the comma. Our team identified the cause for the issue and are working on a fix.
Report: "Delays in ingestion for Panoply SDK Data sources"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're seeing delays in ingestion for data sources of type Panoply SDK. Other data source types are unaffected.
Report: "Warehouse Connectivity Issues"
Last updateThe issue was resolved, and all systems are operational.
The issue was found and resolved. We are now monitoring the system to make sure everything operates as expected.
We are experiencing some connectivity issues when trying to connect to a warehouse. We are currently investigating the issue.
Report: "Panoply website is down"
Last updatePanoply website is available again
The Panoply website (www.panoply.io) is currently down and we are investigating the reason. You can still login to the Panoply platform by going directly to https://platform.panoply.io/
Report: "Data warehouses connectivity issues"
Last updateWhile monitoring we did not see any unexpected connections issues to the data warehouses and all data sources were working as expected
We've identified and fixed the issue. All data warehouses and sources should work as expected now. We monitoring their accessibility to verify that it stays that way
We are investigating connectivity issues to our data warehouses and potential delays in data sources' runs
Report: "Job system scheduling issue"
Last updateThis incident has been resolved.
The issue was resolved and we are now monitoring our data sources job queue.
We have identified an issue with the job scheduler affecting customer jobs. Because of this we have temporarily disabled the scheduling of new jobs until the issue is resolved. We are working hard to resolve the matter as quickly as possible. Your data is safe and no action is required by the users. Job queue will be ingested once the incident is resolved.
Report: "Facebook Ads data source not connecting"
Last updateFacebook Ads data source is working as expected since 7:30pm GMT
The fix was deployed and we're currently monitoring. Any previous data source run should be retried automatically or can be collected manually through the data sources screen.
We've identified the issue and releasing a fix shortly
We are currently investigating the Facebook Ads connectivity issue
Report: "DB connectivity issues"
Last updateThis incident has been resolved.
A fix has been deployed and we're monitoring
We believe we have identified the issue which was causing connectivity problems. We've applied a temporary workaround to make service more available while we prepare a more permanent fix to prevent the issue from occurring again.
We are investigating a db connectivity issues which is affecting some customers. We are sorry for any inconvenience and will update when we have more information
Report: "Scheduled Maintenance"
Last updateThis incident has been resolved.
We are performing some scheduled maintenance on our system today.
Report: "Platform UI is not available"
Last updateUI and jobs work as expected. If a job was effected by this issue it will retry and should complete successfully
The platform's UI was offline and some jobs were failing for about 40 minutes We've implemented a fix to the issue and now monitoring the processes. The UI and jobs work as expected.
Report: "Platform UI degraded performance"
Last updateMonitoring as completed with no outstanding issues. The issue is resolved. The system is fully operational
We've fixed the issues and the data sources and UI components are now fully operational. The jobs that failed will be retried and all the pending jobs will start working shortly as we will start to ingest the jobs backlog. We are currently monitoring the system to make sure it is stable.
The current issue is affecting the following: * sources page * jobs page * running jobs We’re working on resolving the issues and all jobs will be rescheduled
We have identified the issue and are currently working on restoring the platform to a fully operational status.
Some customers are experiencing degraded performance while using the platform UI. We are currently investigating these issues.
Report: "Delay in scheduled data sources"
Last updateAll scheduled data source issues have been resolved.
We have identified and corrected the issues that caused delays in scheduled data sources. Currently monitoring.
We're seeing delays in scheduled data sources runs and investigating the cause for this issue
Report: "Data sources default destination issue"
Last updateAll data sources work as usual now. Failed sources will complete on retry or next scheduled run
The fix was deployed and all data sources should work as expected We're monitoring all the sources to confirm the fix
We've identified the cause and releasing a fix in the next few minutes
We're seeing a rise in failures in several different data sources that use a default destination setting Data sources that their destination was changed are not effected
Report: "Postgres connector failing"
Last updateWe've confirmed Postgres jobs now complete successfully. Jobs that previously failed will either be handled by the retry mechanism (if retries have not been exhausted) or upon the next scheduled time.
We've applied a fix and are monitoring relevant jobs
We've identified the source of the problem and are working on a fix
We're investigating elevated errors and failures in the Postgres connector.
Report: "Scheduled data sources delays"
Last updateAll scheduled data sources will now trigger on time. Due to the backlog some delays in ingestion are possible
Scheduled data sources are now pending for execution
We've identified the source of the issue and we're fixing it
We're seeing delays in triggering scheduled data sources. We're currently investigating the issue.