Is Kisi Down Right Now? Check if there is a current outage ongoing.

Google Cloud Platform is experiencing an outage. Unlocks are not affected, but there may be delays for event related processing (webhooks, event history, ...).

The Event Service—Kisi’s core system for capturing and forwarding events—experienced a degradation after a recent feature deployment. An unclosed-connection bug gradually exhausted CPU and memory resources, throttling database writes. Our monitoring detected elevated error rates. A redeployment addressed the bug, but due to a large backlog some delayed event webhooks were unfortunately dropped. However, no events were lost. **Service Impact:** - Event visibility in the Kisi Admin Dashboard and via API was degraded during the degradation window (new events were not yet persisted). - Event Webhook deliveries and report generation were paused until full recovery. **Next Steps:** - We will adapt our system to process all event webhooks and event related jobs regardless of backlog size - We will isolate our event pipeline to avoid spillover effects from new functionality

The most recent Kisi API release caused elevated error rates. The release has been rolled back and API error rates are back to normal. Unlocks were affected during this time.

Fix implemented by our email service provider seems to have fixed the issue. All our outgoing emails have been delivered since 08:30 UTC.

A fix has been implemented by our email delivery service provider and we are monitoring results. As of 08:30 UTC a majority of emails are being delivered.

An issue is ongoing causing email delays for all customers. We have identified our email delivery service as the cause of this incident. See https://status.mailgun.com/incidents/30w5l1zqcw5r Currently, a majority of our outgoing emails are affected.

Our email delivery provider has implemented a fix and is monitoring the results. Emails are being delivered successfully since 13:52 UTC.

A majority of emails are being delivered as of 13:52 UTC. We will continue to monitor the issue.

We have identified our email delivery service as the cause of this incident. See https://status.mailgun.com/incidents/8bn2px3nyfy4 Currently, a majority of our outgoing emails are affected.

We are investigating an issue causing email delays for all customers. The incident started 13:10 UTC, and is ongoing.

The release has been rolled back. Unlock failure rates and API error rates are back to normal.

The most recent Kisi API release caused elevated error rates. We are rolling back and expect the issue to be resolved in a minute or two. Unlocks are affected.

Cisco Meraki camera snapshots are working normally. The elevated error rates could be attributed to a single Kisi organization.

The rate of failed Cisco Meraki camera snapshots is higher than normal, we are investigating the issue. We will provide an update by Monday, 2023-04-25 09:00 UTC with current details.

We experienced a short (less than 10 minutes) API outage. As a result, unlocks via cloud were failing and web dashboard was unavailable. Offline cache unlocks (https://docs.kisi.io/concepts/offline_support) were working as usual. The issue has now been resolved and we are doing everything to make sure it will not happen again.

All incorrect unlock events have been fixed.

We are currently correcting the affected unlock events.

We discovered an issue that caused in-app unlocks to fail for a small group of users that belong to groups with restrictions between 2024-01-24 11:12 UTC and 2024-01-25 14:19 UTC. Although we have resolved the underlying problem causing unlock failures, there was a misreporting of these failures as successes in the unlock events. We are actively addressing and correcting the affected events.

Our systems are processing jobs at their normal rate again.

We are working on a fix to mitigate the impact of the service outage. The fix will be deployed shortly.

We are continuing to work on a fix for this issue.

We have identified an issue with background jobs leading to severely degraded throughput. Outgoing emails, alerts, and webhooks are among the impacted features. The root cause seems to be an outage at one of our service providers (https://status.elastic.co/incidents/07bw653d2677). Our team is working on mitigating the issue.

Our DNS provider has deployed a fix and this incident should be resolved. We are taking actions to avoid similar incidents in the future.

Our DNS provider is reporting an outage that affects mainly Latin America, but we are also getting indications that other regions are affected. For more information, please see: https://rackspace.service-now.com/system_status?id=child_service_status&service=2309ceb0db6cf200e93ff2e9af961913

We are investigating an issue where users in some regions could be experiencing problems resolving Kisi domains. The incident is mainly affecting users in South America that are using Google's DNS servers.

This incident has been resolved.

We are continuing to monitor for any further issues.

Our infrastructure provider had intermittent connectivity issues. The error rates are back to normal and our engineers will continue monitoring the issue.

We are currently seeing an elevated error rate. Our engineers are investigating the underlying issue.

Our email provider is reporting degraded performance, but we are not seeing any unusual email delivery delays at this point.

Our email provider has identified the problem and is implementing a fix.

Our email provider is reporting email delivery latency issues. Up to 5% emails may be delayed by more than one minute.

This incident has been resolved.

We are continuing to monitor for any further issues.

We have rolled out a fix to all affected controllers and unlocks should work as usual.

The issue is affecting customers with Controller Pro 1.0 and Controller Pro 1.1 with wireless locks attached. We will start rolling out a fix in a few minutes.

We have identified a configuration issue affecting a small number of Kisi customers. We are working on fixing the problem.

The Azure networking issue has been resolved. Please visit Azure status history for more information: https://azure.status.microsoft/en-us/status/history/ (Tracking ID: VSG1-B90)

We are monitoring the impact of an Azure outage. Some Active Directory user imports are currently failing. SCIM and SSO for Kisi customers that are using Azure services might also be affected. Kisi customers that are not using Azure services are not affected. For more information and updates, please see the Azure status page https://status.azure.com/en-us/status

The issue has been resolved as of 3:46 PM, Dec 29, 2022 (UTC).

We are investigating an issue regarding increased API response times. The impact right now is higher than normal response times for about 5% of our requests, including unlocks. We have identified the component responsible for the delays, and we are working with our service providers to find a solution. The issue started just after 10 AM, Dec 29, 2022 (UTC).

An API issue was affecting the availability of the following features between 11:12 and 16:04 UTC: - Creating new users - Setting or updating user passwords - Confirming new users

This incident has been resolved.

Delivery times are back to normal, we are investigating how to avoid these issues in the future.

We have allocated additional resources in order to improve email processing times.

We have identified an issue with our email delivery system. Some emails will be delivered late.

Between 12:16 UTC and 12:50 UTC we experienced problems updating controller and reader configurations. This impacted some scheduled unlocks during this period. The incident has been fully resolved.

This incident has been resolved.

Email delivery times are back to normal. We have identified the root cause and we are working on a fix.

We are investigating an issue causing email delays for all customers.

This incident has been resolved.

The platform is back, but background jobs are still processing slower than normal. The apparent cause seems to be a problem with one of our underlying platform providers.

A fix has been implemented and we're monitoring the results.

We are investigating a major service outage. Updates will be posted shortly.

The Kisi API was unavailable for just under three minutes, from 08:22:45 to 08:25:33 UTC. Our investigation shows that a bug in our platform provider caused requests to be routed incorrectly for a few minutes after our last deploy. We will adjust the way we deploy future versions of the API to avoid triggering this bug.

The Kisi API is currently unavailable. All requests are returning a 503 status.

The incident has been resolved and it is now possible to authenticate Google Calendar and Google Directory integrations.

We are investigating authentication problems with our Google Calendar and Google Directory integrations.

Email delivery is working again.

We are working with our email provider to resolve the issue.

We have an ongoing issues delivering emails. All outgoing emails are affected. We expect it to be resolved shortly.

A fix has been deployed and we are monitoring the results.

A fix has been deployed and we are monitoring the results.

We will deploy a code change that addresses this issue within an hour.

Most API calls are unaffected. The unlock failure rate is currently normal. We have identified the issue and are working on a fix.

The issue seems to be related to some scheduled database maintenance. We have taken some temporary measures to ensure API availability and we are working on identifying the root cause.

We are continuing to investigate this issue.

We have noticed an increased number of errors in both background jobs and endpoints. We are currently investigating.

A fix has been deployed and we will continue to monitor performance.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Azure user provisioning is temporarily disabled due to performance issues. This means no users are currently synced from Azure. We are working on enabling it again.

Azure user provisioning is working again.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

Azure user provisioning is temporarily disabled due to performance issues. This means no users are currently synced from Azure. We are working on enabling it again.

Our underlying platform provider is back up and service has been restored. We continue to monitor the situation closely in light of the current situation in Europe, especially with regards to DDoS threats.

We are continuing to investigate this issue.

The issue lies with our underlying platform provider. We are currently working on mitigation.

We are currently investigating an issue with the Kisi API.

The system is now back up. We are constantly working on various resilience improvements to the API. As part of one of our resilience improvement releases, a configuration error inadvertently denied some of our API traffic. We have taken steps to ensure this does not happen again.

Our engineers has found the issue and are working on bringing the system back up.

Our engineers are continuing to investigate the issue.

Our engineers are continuing to investigate the issue.

Our engineers are continuing to investigate the issue.

We are currently investigating an issue with the Kisi API.

An issue with a release caused failing API calls and background jobs for less than 10 minutes. The issue has been resolved.

Our webhook integration handling was under heavy load which resulted in webhooks being processed slower than normal.

Error rates and throughput are back to normal.

The API is back up and we are processing a backlog of events. Services are up but response times are still above normal.

We have identified an issue with our platform provider and are investigating unusual traffic patterns.

We are continuing to investigate this issue.

We are investigating a major service outage. Updates will be posted shortly.

Emails are no longer being resent.

A service integrating with Kisi is experiencing synchronization issues. This may lead to a high amount of access emails being resent. The partner is notified and working on a solution.

Heroku is performing a rollback, Kisi not affected.

An API patch was applied and NFC unlocks should be functional again.

NFC Unlocks are currently disabled due to a change in the Kisi system. Functionality is expected to be included in the next Android app release.

A new version of the Android app is available through the Play Store with fixes to the BlinkUp process.

The Blinkup Process on the Android Mobile App currently does not work. This will be fixed with the next release of our Android App.

Our update to 3.3.0 should have solved the issue. Again, we are very sorry for any inconvenience caused by the previous update.

iOS App crashes after update. Can be resolved by deleting the Kisi App and re-installing it. We are re-submitting a modified version to the App store and should be released mid of next week. For the ones who want to dig in deeper: The cause of the issue is updating via Apple's App store: http://www.gottabemobile.com/2016/01/19/ios-9-problems-fixes/