Historical record of incidents for Keen
Report: "Incident detected with the S3 stream uploader"
Last updateThis incident has been resolved.
The issue is resolved.
The issue has been identified and a fix is being implemented.
Report: "Degraded performance of Compute API"
Last updateThis incident has been resolved.
We are observing degraded performance of Compute API due to increased volume of queries. We are investigating the issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Writing events might be delayed and some queries might fail"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Writing events might be delayed and some queries might fail"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Some events streamed to S3 may be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Cached Datasets updates might be delayed"
Last updateThis incident has been resolved.
Big backlog of data is being processed and some indexes updates might be delayed.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Some actions may return errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Some actions, especially ones that use access keys, may return errors. We're investigating the issue.
Report: "Query durations are higher than usual and some queries might fail"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Kafka Inbound - messages are delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating the issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Events streaming to S3 buckets may be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Kafka Inbound - messages are delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Potential data inconsistencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Most of the inconsistencies have been fixed. We are still scanning the remaining outliers.
We are still progressing with the fixing discovered inconsistencies.
We have identified three Clients affected by the data inconsistencies. The required steps have been taken to address discovered issues.
The minor data inconsistencies impact a handful of customers. The inconsistencies are being fixed using data from backups.
We are currently investigating this issue.
Report: "Keen Website and Query API partial outage"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Events streaming to S3 buckets may be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Our monitoring shows that streaming events to Keen is slower than usual. We are currently investigating the issue."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are currently investigating this issue.
Report: "Stream and Query API degraded performance"
Last updateYou may observe longer response time and increased error rates
Report: "S3 streaming is temporarily disabled"
Last updateThis incident has been resolved.
Due to an error in S3 streaming we had to completely disable this feature.
Report: "Problems with access to Dashboards via Keen Website"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
You might observe errors when calling API using k-n.io domain. Also problems with Custom SSL Domain might occur.
We are currently investigating this issue.
Report: "Stream API degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our monitoring shows that streaming events to Keen is slower than usual. We are currently investigating the issue.
Report: "Degraded performance"
Last updateThis incident has been resolved.
Performance degradation was caused by Internet connectivity issue on our cloud provider side: https://status.aws.amazon.com/ The issue has been resolved and we are monitoring our services performance.
We are experiencing network performance degradation due to a vendor. We are further investigating to understand the issue impact on our services.
Report: "Apache Log4j Vulnerability - No impact to the Keen platform"
Last updateKeen is aware of a new, critical vulnerability in the Apache Log4j library, CVE-2021-44228. Our Engineering team investigated the impact of the Log4j remote code execution vulnerability and have determined that no part of our platform is vulnerable. We use Logback as a primary logging framework. On the other hand our Apache Kafka, Apache Zookeeper, Apache Storm and Apache Cassandra currently use Log4j 1.2.17, which is not affected by this issue. We are not at risk of breach via the above vulnerability.
Report: "Streaming events to Keen works very slowly"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Due to the scheduled maintenance of one of our databases, we have faced the issue with longer response times of the Stream API.
Report: "Events streaming to S3 buckets may be delayed."
Last updateThis incident has been resolved.
A long queue of events is being processed and will take up to 3 hours to clear up
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Delayed processing of events streamed to Keen"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Everything should be back to normal within around 1 hour (once all the events are processed).
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Delayed processing of events streamed to Keen"
Last updateWe are experiencing network performance degradation due to a vendor.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
The long queue of events is being processed and will take up to 4 hours to clear up
The issue has been identified and a fix is being implemented.
Report: "Refreshing of Datasets might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Refreshing of Datasets might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Datasets might contain stale data due to delayed refreshes. We are investigating this issue.
Report: "S3 streaming might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "S3 streaming might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Some events streamed to Keen are delayed to be accessible by queries"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
1. Usually an event streamed to Keen is available for querying within 10s from being submitted to Keen. 2. Rough estimate is that the incident only applies to about 5% of events streamed to Keen. 3. Some events might be delayed by more than 3.5 hours. We are working to resolve the issue.
Report: "Events write response time is higher than usual"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Events might be delayed"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Writing events might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Events write response time is higher than usual"
Last updateWe experienced a long response time from the downstream AWS MSK cluster and a root cause is still being investigated. The incident lasted from 1/29/2021 6:00 PM PST till 1/29/2021 7:50 PM PST. Really apologize for the inconvenience.
Report: "S3 streaming might be delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Delayed processing of events streamed to Keen"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating the issue. Events sent to Keen in the last 45 minutes may not be available for querying right now.
Report: "Cached Datasets are not refreshed since 9PM UTC"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Keen Website Partial Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Some queries (extractions) may be failing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.