Is Beefree SDK Down Right Now? Check if there is a current outage ongoing.

Between June 13th at 5:40 AM UTC and 7:20 AM UTC, some users experienced issues with the authorization services used by the Beefree SDK. Our team worked to resolve the issue during this time, including performing a service reboot and retrieving the necessary logs and diagnostics to restore full functionality. All services are now fully operational, and we’re continuing to investigate the root cause to help prevent similar issues in the future. Thank you for your patience and understanding.

### Duration: 1 hour, 37 minutes ### Impact: Beefree SDK customers experienced slowdowns and errors when loading the editors, particularly on the Auth and Utils endpoints. API SDK calls were impacted by slowness and errors. ### Summary: On Saturday, March 8, 2025, we updated our infrastructure and services to support a multi-region deployment, enabling our SDK to operate across two AWS regions—Dublin, Ireland, and Frankfurt, Germany. The scheduled maintenance was successfully completed at 11:36 AM EST, with additional adjustments made on March 9 and the morning of March 10 \(CET\). However, as traffic increased on **Monday, March 10, 2025**, the **Beefree SDK and Beefree SDK API** began experiencing **intermittent issues**, which later escalated into a **stable outage**. During the escalation, we investigated the root cause and explored multiple possible fixes. Given the time constraints and uncertainty of an immediate resolution, **we prioritized stability and swiftly redirected traffic to a single region, which restored normal service**. ## Current status: Services are **normal** with no further detected issues. We are actively monitoring for any residual effects. ## Root cause: The issue was highly traffic-dependent, as no slowdowns or failures were observed in the days prior or during load testing. Autoscaling worked correctly, ruling it out as a contributing factor. The problem originated from the use of shared resources within the EKS cluster, specifically DNS resolution. A preliminary analysis indicates that the primary cause was a slowdown in CoreDNS within our AWS infrastructure. As traffic surged, CoreDNS pods became overwhelmed with DNS queries, leading to delayed responses. However, CPU and memory usage remained stable, showing no signs of resource exhaustion. We are conducting further investigations to validate this finding and ensure there are no additional contributing factors. ## Issue timeline **10:38 AM EDT** Our monitoring systems triggered phone alerts, notifying developers of the issue. Our automated monitoring service also posted an alert in our internal Slack channel. Internal escalation was officially initiated. **10:41 AM EDT** A cross-functional incident response meeting was held with developers and support team members. It quickly became evident that the issue was causing slowdowns across the entire infrastructure. The team began troubleshooting to identify the root cause. **11:45 AM EDT** The issue was identified as an anomaly in the internal network management of the newly deployed multi-region AWS setup. To mitigate the impact, we redirected all traffic for essential services to a single region \(AWS Dublin, Ireland\). **12:05 PM EDT** Traffic redirection was fully completed, and infrastructure slowdowns resolved almost immediately. We began monitoring the effectiveness of the solution. **12.35 PM EDT** After 30 minutes of stable monitoring, we officially closed the escalation ## Next steps: Our team is currently conducting a full Root Cause Analysis \(RCA\) to completely understand the underlying cause. We will post updates here as we continue to receive them, including when the RCA is available upon request from our team. ‌ We will evaluate additional monitoring, testing, and safeguards to prevent recurrence. #### If you are still experiencing issues, please contact our [support](mailto:pluginsupport@beefree.io) team. No further action is needed by you at this time. ‌ **We sincerely apologize for the disruption and appreciate your patience.** We will provide an update once we complete our analysis.

The issue has been resolved, and all services are now operating normally. We will continue to monitor the situation closely over the next few days to ensure stability. Thank you for your patience.

We have identified the root cause of the issue and implemented a solution to address the intermittent issues affecting key services. Our team is closely monitoring the situation to ensure system stability.

Update: We are currently identifying the root cause of the issue and are actively working on a solution. Our team is dedicated to resolving the problem as quickly as possible, and we will continue to provide updates on our progress.

We are currently experiencing a partial outage that is affecting our login system. As a result, users may encounter 403 (Forbidden) and 504 (Gateway Timeout) errors when attempting to access our services via our login endpoint. Users may experience login failures, including 403 and 504 errors, and access to certain endpoints may be delayed or unavailable. Our team is actively investigating the issue and working towards a resolution.

The issue has been mitigated by rolling back the impacted component, and all services are now stable. While the immediate incident is resolved, we will be conducting further work to ensure a long-term fix and prevent recurrence.

The co-editing functionality is now working again. The service issue has been resolved. We are monitoring the service closely to ensure it continues functioning as expected.

We have identified the cause of the issue with the service powering the Co-editing functionality. Our team is actively working on a solution and will provide further updates shortly.

We are currently investigating an issue with our service that powers our Co-editing functionality. As a result, customers using Co-editing are currently unable to launch the builder. Our team is already actively working to resolve the issue, and we will provide updates as soon as we have more information.

The image editor experienced a brief outage and was unavailable for approximately 1 minute. The issue has been resolved, and full functionality has been restored.

We no longer see our CDN URL listed in popular ad blockers, and we confirmed that most customers no longer experience issues with images. There’s a small chance that some corporate filters or niche ad blockers may still temporarily block images until the fix propagates. In the unlikely case that you’re still experiencing issues, please reach out to our support team and we’ll investigate.

We are continuing to monitor for any further issues.

The issues affecting content visibility are steadily improving. The feeds that impact various blocklists are updating, and we anticipate a return to normal by tomorrow morning UTC. However, some residual effects, such as browser ad blockers, are still flagging content, and may persist temporarily. While significant progress has been made, we recommend checking this status page to stay updated on ongoing monitoring. Thank you for your patience as we continue to address this issue.

We are investigating an issue where some images hosted on our CDN are not displaying for users due to a temporary listing of our CDN URLs on a spam blacklist. This affects users with specific ad blockers or antivirus settings, causing images to be invisible in libraries, designs, and landing pages. The delisting request has been approved, but it may take time for the update to propagate. Our team is actively monitoring the situation, reviewing systems to address vulnerabilities, and exploring solutions to prevent recurrence. For now, affected users can try disabling ad blockers or whitelisting our CDN.

Our systems experienced degraded performance due to an unexpected surge in oversized JSON files. This caused temporary delays and occasional failures for a small percentage of users during specific timeframes: 8:12-8:24 AM PST 9:23-10:16 AM PST 11:21-11:47 AM PST To address the issue, we scaled our infrastructure, implemented safeguards to handle oversized requests, and deployed a fix on Monday, November 18, to prevent future parser hangs.

The co-edit worker services were redeployed, effectively recycling old connections and allowing new services to establish connections. The number of co-edit workers was also reduced to slow the rate of connection accumulation. An action plan and mitigation strategy were implemented to prevent reoccurrence.

A fix has been implemented at 2024-09-25 15:27:05.790 UTC and we are monitoring the results

An error occurred with our co-editing worker Healthcheck starting 2024-09-25 15:09:35.656UTC

From approximately 23:57 -> 00:09, an issue with an expired token in server to server calls. We have remedied this issue and more information will be available as we complete our RCA.

Outage in component 'Bee Plugin Utils Template'. The team is investigating.

From approximately 23:57 -> 00:09, an issue with an expired token in server to server calls. We have remedied this issue and more information will be available as we complete our RCA.

Outage in component 'Bee Plugin Utils Template'. The team is investigating.

On July 9th, 2023 \(Sunday\), starting from 18:30 \(CEST\), the BEE HTML generation engine experienced a gradual degradation of performance that went on until 21:47 \(CEST\), when the server was restarted. The engine experienced a second period of degraded performance on July 10th, 2023 \(Monday\), from 16:30 \(CEST\) to 17:30. During that first episode on July 9th, an increasing number of requests received a timeout after 30 seconds with a 504 error response. During the 1-hour interval on July 10, approx 10% of the calls returned a 504 error response. While we were working to resolve this, there were brief times the generation may have been unavailable in the following days. However, we are confident this issue is now resolved following an emergency release. Our monitoring tools did not detect the first degraded performance issue immediately because our health checks reported all systems were OK, even on the unresponsive instances of the HTML Generation Engine. We updated our monitoring tools after the two episodes of degraded performance so that we were able to prevent the issue and act accordingly. In the following days, the service had no more experienced degraded performance. Root Cause The issue was caused by a change in the way we handle the environment configuration settings of the HTML Generation Engine. This change was published on July 5th and did not raise any issue until five days later. After proper troubleshooting, we identified that the problem was happening only when specific types of incorrect JSON files were sent to the engine. Action Items \*We improved our monitoring tools, adding additional parameters to keep the HTML Generation Engine under surveillance. \*We added new specific performance tests to our HTML Generation Engine deployment pipeline. \*We included a sample of the invalid JSON files causing the issue to the collection of JSON files that we test the HTML Generation Engine with before every public deployment. \*We deployed a specific fix for the problem, and a new version of the HTML Generation Engine was deployed to production on July 18th, 2023.

You could be experiencing intermitting errors while generating pages and emails.

This incident has been resolved.

We are continuing to investigate this issue.

There was no interruption of service. Our monitoring system detected an issue with a deprecated API no longer used in production.

Outage in component 'Bee Parser Healthcheck'. The team is investigating.

The issue affecting the render component has been resolved. Customers may have experienced a slowdown in thumbnail creation.

Outage in component 'BEE renderer Healthcheck'. The team is investigating.

For 4 minutes, starting at 8:24 AM PST, an instance of the Image Editor's proxy service exited unexpectedly, and traffic was routed to healthy instances. End users should not have experienced disruption as the Image Editor plugin is designed to function normally if the proxy is unavailable.

BEE image editor experienced an outage for approximately 4 minutes. An investigation is underway.

Outage in component 'BEE image editor Healthcheck'. The team is investigating.

This incident has been resolved.

Outage in component 'BEE developers portal Healthcheck'. The team is investigating.

The incident started at 4:27 am PT and was resolved at 4:30 am PT. This issue only affected customers trying to change the Plugin settings via the configuration portal during the 3-minute outage. There was no interruption of service affecting the BEE Plugin Builder.

Outage in component 'BEE developers portal Healthcheck'. The team is investigating.

From approximately 1:46 PST to to 2:04 PST, some customers may have experienced 500 errors due to some backend latency issues. The issues should now be resolved.

We are seeing full recovery of services and related components. Please reach out to Support if you are still seeing issues following the clearing of any browser cache.

We're beginning to see recovery in most endpoints. We've identified the issue and are working on full resolution ASAP. Some features may be limited at this time.

We are continuing to investigate this issue.

Currently, all services/endpoints are down. We are investigating and working to restore as soon as possible.

We've received confirmation that the widespread issues have been resolved. If you're still seeing issues, please clear your cache or contact support.

We’ve received some reports of slower than normal loading times and loading failures. Thus far, we’ve identified the issue as part of a larger internet issue occurring across the United States. Our services are operating as normal otherwise. We recommend, if possible, trying a different network connection. We will continue to monitor this situation and provide updates as we can.

We experienced a limited scope issue with authorization following a deployment early this morning. In rare cases, when the uid was passed as an integer--not a string--authorization would fail to validate the uid. While we do recommend always passing the uid as a string--and state as such in our documentation--edge cases where the uid was an integer weren't covered in tests. We've since pushed a fix, will include these edge cases in the future, and no changes are required on your side at this time.

We're currently experiencing some issues with authentication. This seems to be low impact, and most customers will not be affected. We've identified the issue, however, and are currently working on deploying a fix.

From approximately 8:52 to 9:11 PSDT, authentication was down for BEE Plugin. This resulted in the failure to load for BEE Free, BEE Pro, and BEE Plugin. Upon further inspection, there was an issue with the database that hosts information for authentication as well as our CDN, which was fixed when the size was increased and changed to be elastic.

From 07:05 PSDT to 07:20 PSDT, we experienced an outage. During this time, requests to load the editor were either delayed or would fail and give 403 errors. We do apologize for the inconvenience this may have caused. Upon further inspection, this was a result of an issue with a code deployment and the latest version of the code not being available for the editor to load. We do take these matters very seriously and have already added further safeguards to prevent this from happening in the future. We're also hard at work developing closer to real time alerts for any down time in the future as a result of customer feedback.

We're seeing latency go down and authorization should now be operational.

We are continuing to investigate this issue.

We are currently investigating an issue related to the authentication portion of BEE Plugin. At this time, the editor may have slower load times or may fail to load at all. We will provide an update once the issue has been identified.

We experienced a brief down time this morning that should be resolved as of 9:15 AM PST. More details will be provided as available. Please try reloading and contact us if you're still seeing issues.

We realized an issue with the trust relationship between our back end services and AWS. We have updated this relationship to be more secure and should see no further issues.

You may experience some loading issues with BEE Plugin and/or your page may look strange (content blocks out of place, calls to our service or content returning errors). We are investigating this issue and suspect something on the AWS side.

We have received confirmed reports that this issue has been resolved on the part of the ISP.

Please be advised that we are currently investigating latency issues related to Comcast. We do apologize for the inconvenience and hope everything will be remedied soon. If you are experiencing this issue, please reach out to Comcast and let them know.

On March 25th, 2019, the BEE Plugin system experienced a rare downtime event. System up-time is of paramount importance to us, and we are very sorry for what happened. As some of you know, many applications within MailUp Group \(BEE's parent company\) rely on BEE Plugin - just like you do. When there is any degradation of performance or outage of any kind, our own company is negatively affected, just like you are. So we owe it to you - and to us - to aim for the highest level of availability and performance. And we are very much committed to that. The March 25th issue occurred within - and was isolated to - legacy infrastructure, which had been scheduled for an upgrade on March 9th. As previously communicated, the March 9th upgrade was successful, but had to be rolled back to assist customers using BEE Plugin authorization tokens in undocumented ways. We decided to roll back because we are committed to supporting each and every BEE Plugin customer with a smooth migration when the system is upgraded. As Murphy's Law dictates, one of the very portions of the system that was scheduled to be upgraded on March 9 - which relies extensively on [Redis](https://redis.io) for in-memory data storage - failed on March 25. Despite very extensive research, we have so far been unable to exactly pinpoint why Redis became unresponsive within our AWS environment. This portion of the BEE Plugin system will be replaced with a new-generation infrastructure on [April 6](https://support.beefree.io/hc/en-us/articles/360022828272-April-6-2019-Scheduled-Maintenance). Extra resources have been deployed in the meantime to ensure that no further downtime events occur between now and April 6. Once again, we sincerely apologize for the downtime. If you have any questions, certainly reach out. All the best, The BEE Team

The incident has been resolved and all our systems are fully operational.

All our systems are operational. We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Our team is still investigating the origin of the authentication issue.

We're experiencing an elevated level of authentication errors that causes the editor to stop loading.

This incident has been resolved.

Our team identified an issue that impacted one of our DB instances and fixed it. The Plugin is operational and our team is monitoring the services.

We are currently investigating an issue that is causing loading errors

Amazon Web Services resolved the connectivity problems that caused the FSP API service degradation. The service is running without errors or performance issues.

This service degradation impacts the usage of our default FSP API, causing slow responses and occasional loading errors.

We found that one of the load-balanced machines in our AWS environment had become corrupted, which led to the intermittent issues. The AWS environment has been updated and the issue appears to have been resolved. In the short term, we implemented a few changes to minimize any chances of recurrence of this problem: - We increased the authorization token life to 120 seconds. - We modified the UID validator endpoint so that it will trigger a retry request if authorization fails. In the medium term, we plan to review the entire authorization system to make it even more robust and scalable.

The issue was identified as a problem with AWS instances that had been replaced. Sice that, the loading issue stopped in our tests.

We received reports of intermittent authorization issues related to the UID validation. The editor loads normally when the page is refreshed.