Is Fluxx Inc. Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved. We experienced a service degradation affecting some clients, resulting in slow responses and temporary downtime. The root cause was traced to a suboptimal database query using an incorrect index, which increased load on the RDS instance. We resolved the issue by upgrading the affected RDS instance to a larger size, which restored performance. Additionally, a fix addressing the inefficient query has been implemented and will be included in an upcoming release

We are currently investigating this issue.

This incident has been resolved.

Maintenance will occur on 4/8/2025 Start of maintenance window: 11:00 AM PST Estimated end: 12:00 PM PST Estimated duration: 1 hour

This incident has been resolved.

Maintenance will occur on 4/10/2025 Start of maintenance window: 11:00 AM PST Estimated end: 12:00 PM PST Estimated duration: 1 Hour

This incident has been resolved.

Maintenance will occur on 4/08/2025 Start of maintenance window: 5:00 PM PST Estimated end: 6:00 PM PST Estimated duration: 1 Hour

This incident has been resolved.

Maintenance will occur on 4/10/2025 Start of maintenance window: 5:00 PM PST Estimated end: 6:00 PM PST Estimated duration: 1 Hour

This incident has been resolved.

Maintenance will occur on Start of maintenance window: 5:00 PM PST 4/01/2025 Estimated end: 6:00 PM PST 4/01/2025 Estimated duration: 1 hour

This incident has been resolved.

Maintenance will occur on January 28th 2025 Start of maintenance window: 5:00 PM PST Estimated end: 6:00 PM PST Estimated duration: 1 hour

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved. All impacted documents on affected servers have been cleared. Document uploading and validation services are stable.

Document uploading and validation have been restored. We are addressing impacted documents on affected servers and actively monitoring traffic.

We have identified the cause of the issue and are taking steps to address this issue on affected servers.

We are currently experiencing an issue where document validation is causing uploads to hang, impacting some of our servers. Documents are uploaded, but cannot be downloaded after uploading. Our team is actively investigating and working to resolve the problem as quickly as possible.

This incident has been resolved

Our team is currently monitoring the system to ensure all services are fully restored and stable.

We have identified the root cause of the issue and implemented a fix.

We are currently investigating this issue. Behavior presents as intermittently slow loading cards.

We have confirmed all Grantmaker services are once again operational.

The dashboard loading cause has been identified and a fix has been implemented.

We are currently investigating this issue.

A spike of database load caused the system degradation. This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

AWS notified us of a health issue with some of our clusters, may affect some Production sites.

The performance issue in United States has been resolved. All systems are currently operating within normal parameters.

We are currently investigating a performance issue in the United States Region

During the incident, users were presented with errors when loading detailed views of records. This was also coupled with degraded performance while loading dashboards. A processing node became unstable and unresponsive. This then prevented traffic from being passed to functioning nodes. The affected node was removed from the infrastructure and traffic was rerouted to functioning nodes. The incident was resolved within 3 hours and 30 minutes.

This incident has been resolved.

All services have been restored and we are monitoring the situation

We are continuing to investigate this issue.

We are investigating a partial outage in the US Region

This incident has been resolved.

We have identified the issue and are responding

Multiple customer sites in the US Region were unresponsive due to database contention. The situation has been resolved and engineering is working to reduce the possibility that this can occur in the future

At 1:30 PST today we deployed a patch meant to correct an issue with uploaded documents a previous version to be current. Unfortunately the patch fix created an issue where all documents disappeared from the documents component. The patch was reverted an systems were back to normal by 3:30 PST. Investigation into the original issue is ongoing.

Incident resolved

All clusters are performing as expected at this time.

We are continuing to see some degradation of performance and are working to correct it.

We are continuing to monitor the situation and will advise as futher information becomes available

The previous incident has left some infrastructure in an affected state. Engineers are working to clear this up and correctly route traffic to the responsive infrastructure.

The incident has been resolved and all systems are now fully functional.

We are continuing to monitor the situation.

The solution has been implemented and we are monitoring results.

Engineering has identified the issue and is working on putting the fix into production

We’re experiencing an unexpected outage on numerous us-east-1 prod sites. Engineering is already investigating and we will update as this progresses

Sites should all be operational now.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

It was announced recently that Apache Log4j2 has a serious vulnerability that can result in an attacker performing Remote Code Execution on compromised servers. While Fluxx does have some java services, none of these use Apache Log4j2. Ancillary services that are used by Fluxx do use log4j2 and as soon as the vulnerability was announced, mitigating efforts were taken across the entire infrastructure to ensure that our infrastructure was secure. None of the services that are used by Fluxx that use log4j2 are internet facing, and further, we have put in place measures to eliminate the exploitation of the stated vulnerability. We are also actively working with our vendors to ensure that they are not compromised, and if/where they are using log4j2 to confirm that they also have mitigated the vulnerability or upgraded to fixed versions of log4j2

You may have heard of a recently announced vulnerability in log4j, a popular logging package for Java programs. You can find technical details on this vulnerability at this page: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228. While Fluxx uses some Java services, there is no impact to our clients identified at this point. Our team is actively monitoring our system for any suspicious activity and taking actions to mitigate potential risks. For updates please check back on this page. Thank you, Fluxx Product Team

All Services are now operational.

AWS is seeing services return to stability. Fluxx relies on some of the services that are still experiencing impact, but we are seeing service approaching normal. We will continue to monitor the situation and advise when we are fully returned to operation.

The ongoing AWS outage on the US East Coast is affecting Fluxx back-end systems. At this time we are continuing to monitor the AWS situation and will take steps to improve things as soon as possible. Thank you for your understanding.

From AWS: We continue to experience increased API error rates for multiple AWS Services in the US-EAST-1 Region. The root cause of this issue is an impairment of several network devices. We continue to work toward mitigation, and are actively working on a number of different mitigation and resolution actions. While we have observed some early signs of recovery, we do not have an ETA for full recovery. For customers experiencing issues signing-in to the AWS Management Console in US-EAST-1, we recommend retrying using a separate Management Console endpoint (such as https://us-west-2.console.aws.amazon.com/). Additionally, if you are attempting to login using root login credentials you may be unable to do so, even via console endpoints not in US-EAST-1. If you are impacted by this, we recommend using IAM Users or Roles for authentication. We will continue to provide updates here as we have more information to share.

We are continuing to monitor for any further issues.

from AWS [11:26 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time. Root logins for consoles in all AWS regions are affected by this issue, however customers can login to consoles other than US-EAST-1 by using an IAM role for authentication.

From From https://status.aws.amazon.com/ We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified root cause of the issue causing service API and console issues in the US-EAST-1 Region, and are starting to see some signs of recovery. We do not have an ETA for full recovery at this time.

From https://status.aws.amazon.com/ We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified the root cause and are actively working towards recovery.

Current AWS Services affected Amazon Connect (N. Virginia) Amazon DynamoDB (N. Virginia) Amazon Elastic Compute Cloud (N. Virginia) AWS Management Console AWS Support Center

AWS is now also reporting errors in the APIs used in our primary region From https://status.aws.amazon.com/ We are experiencing elevated error rates for EC2 APIs in the US-EAST-1 region. We have identified root cause and we are actively working towards recovery.

AWS Status: (https://status.aws.amazon.com/) 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery.

Fluxx in the US is suffering from ongoing widespread AWS performance and connectivity issues. We are working to understand how to provide better access and increase performance of the system.