Is Gini Down Right Now? Check if there is a current outage ongoing.

This had been a false positive flagged by our routine automated checks.We are investigating and correcting the monitoring component to adjust for that.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This had been a false positive flagged by our routine automated checks. We are investigating and correcting the monitoring component to adjust for that.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This was a false positive incident. The API is and was operational.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

A configuration issue in our network caused partial outages of the APIs. The issue is resolved and the system is back in a healthy state.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

Short authorization endpoint outage due to an issue with the database. We applied fixes and monitor them to ensure the problem is solved

Major outage of all APIs due to a defective configuration in the underlying network infrastructure.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved

We are currently facing an outage in parts of our system. We are currently investigating this issue.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

This incident has been resolved.

We are currently facing an outage in parts of our system. We are currently investigating this issue.

The incident has been resolved successfully.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

The incident has been resolved - all systems are operational again

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

Dear customers, a few minutes ago we received the message via e-mail that the network infrastructure of our data center operator myLOC/Mivitec has been compromised. For security reasons, the entire data center in Munich will be taken offline shortly so that no further damage can occur. According to the first analyses of the data center operator your customer data are NOT affected. However, our service will be unavailable shortly (approx. at 6:00 pm). We are in close contact with myLOC/Mivitec and will keep you informed as soon as we learn more.

For around 90 seconds Pay API got scheduled on a server undergoing maintenance. Issue has been resolved

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently investigating this issue.

Configuration in storage cluster was overwritten 2 days prior during the incident fix

Incorrect configuration of the routing led to a negative impact on essential components

isolated, short-term problems in the storage cluster after the migration in December prevented the processing of requests.

The network outage has been solved. We keep monitoring the system. Full functionality has been restored.

We are experiencing networking issues that causes our infrastructure not to communicate with external world and doesn't let internal components talk to each other due to failed DNS resolution

Storage servers became full and unable to store more documents. Degradation caused the outage and processing problems. We managed to restore the service and preserve all the data. We are investigating why alert wasn't created when server storage capacity was reaching its limit

We had an issue while upgrading a network component in the cluster. Unfortunately this broke the cluster-networking for all services.

This incident has been resolved - a fix has been implemented.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

We had to deal with a major storage degradation which caused the outage and processing problems. We managed to restore the service and preserve all the data. The implemented fix will need to be revisited in an upcoming maintenance window.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Towards the end of our planned maintenance tasks, we encountered a problem that caused a temporal storage cluster outage. The degradation of the storage cluster affected the document processing and resulted in an outage of our external services. We were able to overcome the failed upgrade and thus restore full functionality. We are deeply sorry for the unplanned outage and will test these kind of procedures even more thorough in the future.

We worked through the backlog and the processing times are back to normal. We will continue the investigation into the issues that lead to this outage to improve our resiliency even further.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Documents processing is not operational at the moment and we are investigating further.

We are currently investigating this issue.

a hardware problem was discovered with the storage-cluster and a solution has been implemented to resolve the issue

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

the datacenter hosting our infrastructure was having routing issues. these issues have been resolved and connectivity has been restored. we are monitoring the situation.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently recovering from an outage in our storage cluster. This means that PDFs and images cannot be downloaded from the API. Document processing and getting extractions work as expected.

This incident has been resolved.

The network is back and we are monitoring the situation closely.

We received notification from our datacenter provider that they need to perform a emergency maintenance on our internet infrastructure and this will lead to a few minutes of downtime for our external APIs. We are sorry for the short notice and will update the incident accordingly.

Starting Friday around 15:00 we received alerts about reduced API availability and around 15:30 our entire external network was down. Our investigations quickly revealed that this outage is out of our immediate control and we contacted our datacenter provider for additional assistance. Looking from the outside it became clear that our network was no longer announced via BGP or that the routes kept changing. As we learned during the outage this was done by our provider as part of a DDoS mitigation which we fell victim to. Neither Gini nor our provider were the actual target but we were still affected and had to take countermeasures. After it became apparent, that redistributing the traffic alone will not stop the attack, our provider decided to use a number of scrubbing centres to remove the unwanted traffic and gain back control. The attack continued until Sunday morning but our services regained full stability from Saturday 03:30 onwards. We discussed additional options to reduce the time it took to fend off the attack and we and our datacenter provider will continue to optimise the procedures and tools.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

API availability is impacted. We are in contact with our datacenter provider to understand what is happening. The problem seems to be BGP related.

We experienced some connectivity issues between 00:53 and 02:39 which resulted in a total downtime of 19 minutes. During that timeframe our APIs were not reachable and no documents could be accepted and processed. We are investigating the events that led to this outage and will provide updates accordingly.

Our production system was affected by an outage that lasted, with varying degrees of impact for our partners, from 25.11.2019 23:00 until 26.11.2019 03:15. **Document processing and API availability was seriously impacted until 01:15**. The initial outage was caused by a failure in one of our Kubernetes clusters. We operate a redundant control plane but this specific problem still resulted in a downtime of the Kubernetes API server. The API server is vital to the functionality of the cluster and without it the problem spread into wide parts of our infrastructure. After the API functionality was restored we needed longer than anticipated to identify, understand and fix a networking issue that was caused by the API outage in the first place. After the networking problems were fixed we finished the reactivation of affected data stores and databases to restore full service functionality. **No data was lost and all the documents that we accepted have been processed**. We strive to provide the best service and availability to our partners and customers and we didn’t live up to this goal. We already identified ways to circumvent similar issues in the future and we are already working on the implementation. Feel free to contact us at [technical-support@gini.net](mailto:technical-support@gini.net) if you have any questions about Gini or this outage.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

The APIs are back and documents are being processed. We will continue to monitor the system closely for the rest of the night.

The issue has been identified and a fix is being implemented.

Our external API endpoints are currently offline. We are investigating together with our infrastructure provider. IPv6 connections work as expected as of now.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified a source of the problem and deployed a fix. Documents are being processed again. We continue to monitor the situation

We're investigating an elevated number of document processing errors. We are investigating the situation

This incident has been resolved.

The incident is confirmed to be resolved.

We identified an issue in the processing of documents that drastically limited the throughput of document processing, ultimately leading to queued documents not being processed. We restored the processing throughput to its expected level and are monitoring the situation closely. We will take further steps to mitigate the issue in the future.

We identified a problem with an internal service that impacted the availability of api.gini.net while performing internal maintenance operations. The API was brought back up immediately and a fix has been implemented in order to mitigate future occurrences of the problem.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue is still under investigation. The limitation in the access only applies to IPv6 connections; IPv4 connections remain unaffected.

We are currently investigating the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We identified a problem with an internal service that impacted the availability of api.gini.net. We brought the api back online and are working on a sustainable fix.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Datacenter is reachable again. We're working closely with our datacenter provider to identify the root cause

We're investigating a complete loss of network connectivity to and from our datacenter in Munich.

This incident has been resolved.

We deployed a fix and extractions can be accessed as expected. We're monitoring the situation closely

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

The storage tier is back up and operates as expected and we did not lose any data. We're monitoring the situation closely

The problem has been identified and we're working on a fix.

We're currently having problems with our storage tier and documents and rendered images can't be stored or retrieved. Extractions are not affected.