Historical record of incidents for Taplytics
Report: "Elevated Response Times"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue causing elevated response times across all SDKs and APIs and for push notifications
Report: "Elevated Response Times"
Last updateThis incident has been resolved
Services appear to be back to normal, we will continue to monitor
A fix has been implemented and we are monitoring as response times and error rates normalize
We are continuing to work on a fix for this issue.
The issue has been identified as a database latency issue and we are working on a resolution
A fix has been implemented and the response times are starting to normalize.
The issue has been identified as a database latency issue and we are working on a resolution
We are currently investigating an issue causing elevated response times across all SDK and API endpoints.
Report: "Minor Push Notification Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue related to push notifications. Users may experience delayed push notifications.
Report: "Elevated error rates and response times on Taplytics APIs"
Last updateThe incident has been resolved and services are back to normal
We are currently investigating elevated error rates on our APIs and increased latency on the management dashboard.
Report: "Minor Push Notification Delays"
Last updateAn incident occurred from around 9:00 - 9:25 am EST that may have caused minor delays in receiving push notifications.
Report: "Degraded User Insights"
Last updateThis incident has been resolved.
The issue may cause slowness when loading and updating User Insights data
Report: "Degraded User Insights and Push Performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Users may experience delays in receiving push notifications
Report: "Elevated SDK Response Times"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're observing higher than normal response times on our SDKs. A fix is being worked on.
Report: "Degraded Performance on Taplytics webpage"
Last updateThis incident has been resolved.
The Taplytics.com website is experiencing degraded performance and may take longer than usual to load. The Taplytics dashboard (logged-in) is operating as expected.
Report: "Degraded User Insights and Push Delivery Performance"
Last updateThis incident has been resolved.
We've identified an issue that is causing slowness in loading new data on the User Insights page as well as experiencing degraded performance for Push Notifications
Report: "Delays in Processing Event Data and Updating Dashboard Results"
Last updateThis incident has been resolved.
We’ve identified an issue with the saving of event data at the time resulting in delays in updating Events on the dashboard
Report: "Degraded Push Performance"
Last updateThis incident has been resolved.
An outage with a third-party provider is causing degraded performance for Push Notifications.
Report: "Delays in Processing Event Data and Updating Dashboard Results"
Last updateThis incident has been resolved.
We've identified an issue with the saving of event data at the time resulting in delays in updating Events on the dashboard. A fix has been implemented and dashboard results data should update over time. We'll continue to monitor the results.
Report: "Delays in Processing Event Data and Updating Dashboard Results"
Last updateThis incident has been resolved.
A fix has been implemented and dashboard results data should update over time. We'll continue to monitor the results.
We've identified an issue with the saving of event data at the time resulting in delays in updating Events on the dashboard
Report: "Degraded User Insights and Push Delivery Performance"
Last updateThis incident has been resolved - The User Insights page and Push notification delivery are working as expected
We've identified an issue that is causing slowness in loading new data on the User Insights page as well as experiencing degraded performance for Push Notifications
Report: "Users initializing their SDKs via CDN may experience delays in receiving config file updates."
Last updateWe have identified the issue to be related to an outage caused by an upstream service provider, which has since been resolved. All services are operating as expected.
We are currently investigating this issue.
Report: "Delays in processing and display of experiment results and user insight data"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Delays in processing and display of experiment results and user insight data"
Last updateThis incident has been resolved.
The root cause of the investigation into an increase in the number of MSK Connect applications that are not processing data in the US-EAST-1 Region with a 3rd party upstream service provider has been identified. A resolution is currently being worked on, and our systems are recovering. A full recovery is expected to be completed by 3:55 PM EST.
We are currently investigating an increase in the number of MSK Connect applications that are not processing data in the US-EAST-1 Region with a 3rd party upstream service provider.
Some users may observe a delay in the processing and display of experiment results and user insight data. Please forward any questions or concerns to support@taplytics.com.
Report: "Degraded User Insights and Push Delivery Performance"
Last updateThis incident has been resolved - The User Insights page and Push notification delivery are working as expected
We've identified an issue that is causing slowness in loading new data on the User Insights page as well as experiencing degraded performance for Push Notifications
Report: "Degraded Push Delivery Performance"
Last updateThe above incident has now been fully resolved. Pushes deliveries are no longer experiencing a delay.
We are currently investigating degraded performance, pushes are delivering with a slight delay.
We are currently investigating an issue with an upstream service provider that's causing delays in user receiving push notifications.
Report: "Events Pipeline Outage"
Last updateThis incident has been resolved.
We are continuing to monitor our systems while we recover. Minor delays in saving events are expected at this time.
A fix has been implemented and we're currently monitoring the results. Events may be delayed while we recover. Response times are back to expected values.
The issue has been identified and a fix is being implemented.
We are continuing to investigate the issue at this time. Users may additionally notice increased response times on Mobile and Web SDKs.
We are currently investigating issues with our Events pipeline causing events not to be saved
Report: "Delay in Loading Results Data"
Last updateThe incident has been resolved, there are no longer delays on Results pages
The issue has been identified and a fix is being implemented.
Report: "Taplytics Dashboard not rendering CSS"
Last updateThis incident has been resolved.
A fix has been implemented, and we are monitoring the results.
We are currently investigating an issue with the rendering of the Taplytics Dashboard's User Interface.
Report: "API Degraded Performance"
Last updateWe have observed a temporary increase in our API error rate, which has since recovered.
Report: "Taplytics Dashboard Degraded Performance"
Last updateThis incident has been resolved.
We are currently investigating an issue with the loading time of the Taplytics Dashboard.
Report: "Degraded Events Pipeline"
Last updateEvents have been backfilled, the incident has been resolved.
A fix has been implemented and we're in the process of backfilling the events on the dashboard. We are monitoring its progress at this time.
We've identified an issue with the saving of event data at the time resulting in delays in updating Events on the dashboard
Report: "Cloudflare Downtime"
Last updateThis incident has been resolved.
We are continuing to monitor as Cloudflare rolls out their fix.
Cloudflare has implemented a fix, and we are monitoring services returning to normal.
We are currently monitoring a outage from Cloudflare, view their status page for updates: https://www.cloudflarestatus.com/
Report: "Degraded Push Performance"
Last updateThe incident has been resolved
The issue has been mitigated and we'll continue to monitor our platform
The issue has been identified, we have added a fix and our platform is recovering at this time
Taplytics is currently experiencing degraded performance on our push platform and are investigating at this time
Report: "CSV Upload Service Disabled"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We've identified an issue relating to CSV imports and have disabled the functionality while we investigate.
Report: "Push Notification and Elastic Search Degraded Performance"
Last updateThe incident has been resolved.
We are seeing a recovery in push notification performances and will continue to monitor for any further changes.
We’ve identified an issue with our upstream provider and you may see degraded push notification delivery performance as a result
Report: "Elastic Search Degraded Performance"
Last updateThis incident has been resolved.
We are seeing a recovery in push notification performances and will continue to monitor for any further changes.
We are continuing to monitor for any further issues.
We’ve identified an issue with our upstream provider and you may see degraded push notification delivery performance as a result
Report: "Degraded Push Notification Service"
Last updateThe incident has been resolved and performance has returned within full standard operating parameters.
We’ve identified an issue with our upstream provider and you may see degraded push notification delivery performance as a result
Report: "Degraded User Insights and Push Notification Performance"
Last updateThis incident has been resolved. Preventative measures have been implemented to eliminate future occurrences.
We are observing degraded performance on the Taplytics User Insights page and with push notifications.
Report: "Degraded User Insights and Push Notification Performance"
Last updateThis incident has been resolved.
The incident has been mitigated. We are monitoring the data's recovery at this time.
We have implemented a fix and all impacted services are currently recovering.
We are observing degraded performance on the Taplytics User Insights page and with push notifications.
Report: "Taplytics documentation currently unavailable"
Last updateThis incident has been resolved.
Taplytic's documentation page (https://docs.taplytics.com/) is currently down due to connectivity issues to Common Runtime apps in the US and EU regions with Heroku.
Report: "Degraded AWS Push Performance"
Last updateThis incident has been resolved.
Report: "AWS Outage"
Last updateAWS has now restored power to all instances and network devices within the affected data center and systems are recovering. We will continue to monitor our systems throughout the day.
We are continuing to monitor the situation. We don't detect any outages in our systems and degraded performance should be within standard operating parameters.
We're aware of a current AWS outage and are currently investigating the issue. We will provide a status update once we have more information.
Report: "Cloudflare Outage"
Last updateThe Cloudflare incident has been resolved.
We are continuing to monitor for any further issues.
Cloudflare has implemented a fix for the CDN and latency issues and Taplytics is seeing improvements in our services. We will continue monitoring the situation.
Cloudflare is experiencing CDN and latency issues which are impacting Taplytics services at this time. Cloudflare statuspage: https://www.cloudflarestatus.com/
Report: "AWS Degraded Performance"
Last updateAWS network device issues resolved.
AWS is continuing to report degraded performance. We are monitoring and do not currently see any impact on Taplytic's systems.
Report: "AWS Degraded Service"
Last updateAWS services are operating normally.
We are continuing to investigate this issue.
We are observing degraded performance with AWS' us-east-1 region. We are monitoring and currently observing no impact on any Taplytics systems and will update within 15 minutes.
Report: "Degraded API Service"
Last updateAn Upstream Outage from Google Cloud affected our API Services. We have done an investigation and have resolved the issue. The API Service is operating as normal.
Report: "Push Notifications Outage"
Last updateThe issue has been resolved and all push notifications have already been sent. We are back to normal.
Our Engineers have identified and fixed the issue. Currently, the Push notifications are queued and slowly processing the backlog. We are monitoring it and will update as soon as we are back to normal.
We are currently encountering an issue with our Push Notification service and we are already in the progress of investigating it.
Report: "Degraded Events pipeline"
Last updateTaplytics experienced a degradation to our Events service that occurred between Aug 21 and Aug 23, 2021. This caused event data to back up and resulted in a loss of some events on Aug 21 and Aug 22. The majority of events for Aug 23 were successfully recovered. There was no impact to the delivery and execution of all core functionality including: Experiments, Feature Flags and Push Notifications during this incident. Taplytics users may have experienced a decrease of project wide dashboard metrics during this period. As a result of this incident, alerting and monitoring services have been improved and additional system measures are in place to prevent this issue from recurring.
This incident has been resolved.
The event's data delay is still recovering and we will continue to monitor it.
Issue: We have encountered a degraded service with our events pipeline. Impact: Dashboards will show delayed events data. Action Item: Currently, the service is already recovering and being monitored.
Report: "Degraded Cloudflare service"
Last updateThis incident has been resolved.
Base on the Cloudflare status page, A fix has been implemented, and they are monitoring the results.
There is a Cloudflare outage that affected our services. Currently, our services are back and we are monitoring closely
Report: "Intermittent slowness for Dashboard events"
Last updateThe issue has been fixed and will be monitored.
The Issue has been Identified and we are monitoring the services
We are seeing intermittent slowness in our data events, this will only impact results temporarily. The investigation is in progress
Report: "Intermittent Push Notification Instability"
Last updateThis incident has been resolved.
Performance is back to operational.
We are continuing to investigate this issue.
We are observing some intermittent instability with our push notification service. Some of our customers may be affected. We are currently investigating.
Report: "Degraded Service - DNS Instability"
Last updateWhile migrating DNS records for the [api.taplytics.com](http://api.taplytics.com) endpoint, we observed increased DNS propagation times for individual ISPs. Subsequently, we observed that the invariable side effects of the above resulted in degraded performance, for some of our clients. We have restored services to operational levels. Thank you for your understanding,
The incident is now resolved. We will continue to monitor closely.
Recovering. Currently awaiting changes of the DNS to propagate.
We are currently investigating an issue with our DNS Servers
Report: "AWS Partial Outage Affecting Taplytics Production Environment"
Last updateLatest Update from AWS: We have restored all traffic to Kinesis Data Streams via all endpoints, and have resolved the error rates invoking CloudWatch APIs. We are continuing to closely monitor Kinesis and work toward full recovery of all services. We have identified the root cause of the Kinesis Data Streams event, and have completed immediate actions to prevent recurrence. Kinesis and CloudWatch are operating normally. https://status.aws.amazon.com/ Taplytics Status Update: All affected components have been restored; Push Notification functionality is now fully operational for some of our clients that may have been been impacted.
AWS Outage Update - Increased Error Rates 12:15 PM PST: We continue to work towards recovery of the issue affecting the Kinesis Data Streams API in the US-EAST-1 Region. We also continue to see an improvement in error rates for Kinesis and several affected services, but expect full recovery to still take up to a few hours. The issue continues to also affect other services, or parts of these services, that utilize Kinesis Data Streams within their workflows. While features of multiple services are impacted, some services have seen broader impact and service-specific impact details are below. We continue to work towards full recovery. 10:47 AM PST: We continue to work towards recovery of the issue affecting the Kinesis Data Streams API in the US-EAST-1 Region. For Kinesis Data Streams, the issue is affecting the subsystem that is responsible for handling incoming requests. The team has identified the root cause and is working on resolving the issue affecting this subsystem. The issue also affects other services, or parts of these services, that utilize Kinesis Data Streams within their workflows. While features of multiple services are impacted, some services have seen broader impact and service-specific impact details are below. For real-time updates directly from our hosting provider - AWS - please head over here: status.aws.amazon.com Taplytics Status Update: - Intermittent outage affecting Push Notification feature functionality. - No additional issues have been detected/reported for the Taplytics platform.
We are continuing to monitor for any further issues.
AWS Status Update 9:12 AM PST: - The Kinesis Data Streams API is currently impaired in the US-EAST-1 Region. - This issue is also impacting other services, including ACM, Amplify Console, API Gateway, AppMesh, AppStream2, AppSync, Athena, AutoScaling, Batch, CloudFormation, CloudTrail, CloudWatch, Cognito, Connect, DynamoDB, EventBridge, IoT Services, Lambda, LEX, Managed Blockchain, Marketplace, Personalize, Resource Groups, SageMaker, Support Console, Well Architected, and Workspaces. -This issue has also affected our ability to post updates to the Service Health Dashboard. -We are continuing to work towards resolution. Taplytics Status Update - Taplytics Status Update: - Intermittent outages affecting Push Notification feature functionality. - No additional issues have been detected/reported for the Taplytics platform. We are continuing to monitor and are remaining on stand-by until AWS deploys a solution to the problem they're experiencing. For more real-time updates directly from AWS, please see here: https://status.aws.amazon.com/
AWS Status Update: - Increased Error Rates - 7:30 AM PST: Kinesis is experiencing increased API errors in the US-EAST-1 Region. This is also causing issues with Cognito, IoT Core, EventBridge and CloudWatch. This issue has also affected our ability to post updates to the Service Health Dashboard. Taplytics Status Update: - Intermittent outage affecting Push Notification feature functionality. - No additional issues have been detected/reported for the Taplytics platform. We will continue to monitor this event in real-time.
We are continuing to monitor for any further issues.
Affected: Amazon Kinesis Data Streams (N. Virginia) Location: N. Virgina [North America] Effect: Increased API error rates Taplytics Downstream Effects: Instability, intermittent outage for Push Notification platform feature functionality For more information regarding this AWS partial outage, please refer to: https://status.aws.amazon.com/
We are actively monitoring the AWS status page in parallel to our internal monitors for any Taplytics platform performance impact. We will post AWS updates piping in from their end accordingly, as well as any additional synchronous effects on our systems from this partial outage caused by our third party hosting vendor.
We are currently on stand-by and are awaiting AWS to resolve the problem. We will update consistently with relevant status updates. Taplytics Affected Components: Push Notifications
Report: "Experiments and Feature Flags Delivery"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring performance to ensure that we have achieved resolution of the issue.
Root cause identified. We are in the process of rolling out a hotfix shortly. Thank you for your continued patience.
We are currently investigating reports of Experiment delivery issues on iOS. Engineers have been dispatched to investigate; we will continue to post updates as more information becomes available.
Report: "Dashboard: Launch Control tab is missing"
Last updateThe issue with the missing Launch Control tab has been resolved. The main result has been that users wouldn't have been able to access the Launch Control page and individual Feature Flags. We are conducting a further internal investigation of this issue and will be making appropriate improvements to our systems to help prevent or minimize future recurrence.
We are investigating reports of the Launch Control tab missing from the main dashboard. Our team is currently actively looking into the situation. There are no known issues related to launch control delivery at this time.