Historical record of incidents for Apigee
Report: "Apigee Edge for Public Cloud - Management Infrastructure maintenance is scheduled for Friday, May 16, 2025, from 8:00 PM PT to 10:00 PM PT"
Last updateApigee Edge for Public Cloud - Management Infrastructure maintenance is scheduled for Friday, May 16, 2025, from 8:00 PM PT to 10:00 PM PT.During this scheduled maintenance period, the listed services may experience a single unavailability event for up to 10 minutes. Invocations of these services may return a 4xx status code:* Management Services UI* Management Services API(s)* Monetization Management APIs* New Proxy Deployments* DevPortal Service* Microgateway Services* Edge Query (Analytics Stats APIs)All inquiries should be directed to Apigee Support at https://cloud.google.com/support-hub/FAQs* Will this change affect runtime traffic?No* Will this change have any effect on the monetization product?Please be advised that while the monetization runtime will remain unaffected, the monetization management APIs will experience a period of unavailability during the scheduled maintenance window.* Will this change impact new proxy deployments or updating existing proxy revisions?During this scheduled maintenance period, customers may not be able to deploy or update their proxies.* Will this change affect Analytics?While Analytics statistics APIs will be unavailable during the maintenance period, no analytics data will be lost. The entirety of analytics data will be accessible following the completion of the maintenance.* Will this change impact EMG?During the scheduled maintenance window, the deployment and scaling of new Edge MicroGateway (EMG) instances will be suspended. Existing EMG instances will not be affected. However, the restart of the Edge MicroGateway service is not possible during this period.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Apigee released OPDK 4.53.00.04 , a patch version for Apigee Edge for Private Cloud"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
Apigee released OPDK 4.53.00.04, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:OPDK 4.53.00.04: https://docs.apigee.com/release/notes/4530004-edge-private-cloud-release-notes
Report: "Apigee released OPDK 4.52.02.03 , a patch version for Apigee Edge for Private Cloud"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
Apigee released OPDK 4.52.02.03, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:4.52.02.03: https://docs.apigee.com/release/notes/4520203-edge-private-cloud-release-notes
Report: "Apigee Edge for Public Cloud Monetization infrastructure maintenance schedule"
Last updateApigee Monetization Infrastructure maintenance for Apigee Edge for Public Cloud is scheduled as follows, per region.During the maintenance period, Apigee Monetization billing, reports, and Monetization Management APIs will be unavailable for customers with Monetization enabled. Apigee Edge runtime traffic will remain unaffected by this maintenance activity. Rating of monetized runtime calls will recommence upon system restoration. Following the completion of rating, data for the affected period will be incorporated into the reports for that day.Apigee Edge customers without the Monetization feature enabled will experience no impact.Note: This maintenance update does not introduce new features. The system and its existing functionalities will continue to operate as previously configured.Should you require assistance or have any inquiries, kindly reach out to Apigee Support via the following link: https://cloud.google.com/apigee/support/.Components affected for Customers with Monetization enabled: * Apigee Monetization billing and reports. * Monetization related APIs # Management API calls on developer/apps/product # Mint API calls on Rate plans / Product bundles / NotificationsRegion Maintenance WindowAsia Pacific Apr 22, 2025 11:30 AM - Apr 22, 2025 5:30 PM UTCAsia Pacific Apr 23, 2025 11:30 AM - Apr 23, 2025 5:30 PM UTCEMEA Apr 23, 2025 3:30 AM - Apr 23, 2025 9:30 AM UTCEMEA Apr 23, 2025 3:30 AM - Apr 23, 2025 9:30 AM UTCAmericas Apr 28, 2025 8:30 AM - Apr 28, 2025 2:30 PM UTCAmericas Apr 29, 2025 8:30 AM - Apr 29, 2025 2:30 PM UTCFAQs:What is changing?A maintenance activity will be performed on the backend systems supporting Monetization. Is any customer action necessary?No.Will customers be able to access Monetization reports during the maintenance period ?Reporting functionality will be unavailable during this scheduled maintenance period for customers with monetization enabled. Customers without monetization enabled will experience no impact.Monetized runtime call rating will recommence upon system restoration. Following the rating process, the corresponding data for the affected period will be incorporated into the daily reports.Will customers be able to use Management Functionality/APIs during this period ?For customers with monetization enabled, the following Management APIs will be unavailable.1. API products2. App3. Developer4. Company5. Rate plan6. Reports7. Notifications8. Transactions SearchThis is an indicative list of APIs concerned with Monetization-specific entities or those utilized by the Monetization system. Management APIs addressing entities such as proxies and deployments, which do not directly affect Monetization, may also be used. For Customers who have Monetization disabled, Management APIs remain accessible.How can customers verify their Monetization enablement status?To verify Monetization enablement status, customers can utilize this API: https://apidocs.apigee.com/docs/organizations/1/routes/organizations/%7Borg_name%7D/get. A response containing the property as depicted below indicates that Monetization is enabled. Conversely, the absence of the property or a false value signifies that it is disabled.{ "name": "features.isMonetizationEnabled", "value": "true"}
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Apigee released OPDK 4.53.00.03, a patch version for Apigee Edge for Private Cloud"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
Apigee released OPDK 4.53.00.03, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:4.53.00.03: https://docs.apigee.com/release/notes/4530003-edge-private-cloud-release-notes
Report: "Apigee released OPDK 4.52.02.02, a patch version for Apigee Edge for Private Cloud"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
Apigee released OPDK 4.52.02.02, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:4.52.02.02: https://docs.apigee.com/release/notes/4520202-edge-private-cloud-release-notes
Report: "Apigee Edge for Public Cloud Single Sign-On (SSO) patch release is scheduled for Sunday, March 16, 2025, at 9:30 PM Pacific Time."
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
An Apigee Edge for Public Cloud Single Sign-On (SSO) patch release is scheduled to commence at 9:30 PM Pacific Time (PT) on Sunday, March 16, 2025. The deployment will occur in a phased manner by region. It is anticipated that this release will not disrupt Apigee Edge services.Release Notes: https://docs.apigee.com/release/notes/250316-apigee-edge-public-cloud-release-notes-sso
Report: "Apigee Edge increased error rates for some customers in europe-west2"
Last updateThis incident has been resolved.
Issue has been mitigated
Some of Apigee Edge Public Cloud customers in europe-west2 experiencing elevated number of HTTP 500 responses and increased latencies due to infrastructure issues in Apigee Datastore cluster.
Report: "Traffic disruption in eu-west-1, elevated number of HTTP 500 errors"
Last updateThis incident has been resolved.
Issue has been mitigated, normal traffic flow resumed. Team suspects partial network outage in single region, we continue investigating.
We are currently investigating issues affecting Apigee Edge customers in eu-west-1. Some customers may observe elevated number of HTTP 500 responses due to issues with Apigee Datastore in the region.
Report: "Security Advisory: [CVE-2024-6387]: [OpenSSH Vulnerability]"
Last updateA remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Please refer to Google Cloud Security bulletin published: GCP-2024-040 : https://cloud.google.com/apigee/docs/security-bulletins/security-bulletins#gcp-2024-040
Report: "Security Advisory: [CVE-2024-6387]: [OpenSSH Vulnerability]"
Last updateA remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Please refer to Google Cloud Security bulletin published: GCP-2024-040 : https://cloud.google.com/apigee/docs/security-bulletins/security-bulletins#gcp-2024-040
Report: "NAT Incident affecting some Customers in us-central1"
Last updateThis incident has been resolved.
Mitigation was successful, we're now monitoring the incident and confirming impact.
Team has narrowed down potential cause and performing a mitigation attempt
Team suspects incorrect configuration leading to some percentage of customer traffic affected. We are actively investigating this issue.
Report: "Networking Disruption for GCP Customers"
Last updateThis issue has been resolved for all affected projects as of Thursday, 2024-05-16 18:10 US/Pacific. We thank you for your patience while we worked on resolving the issue.
Most regions' network functionality has recovered. We are still actively monitoring for any remaining issues.
Some customers hosted on GCP may experience latency and/or 5XX errors in their runtimes due to a Networking Issue on GCP. Some Edge Cloud orgs may experience errors during API proxy deployment as well. The impact to most organizations were resolved automatically and the Apigee team is investigating the issue further and taking appropriate steps to mitigate it further. Apigee service will recover as soon as this Networking issue is resolved. The underlying issue status can be tracked at https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#2c2sBHWU84yPDJ8y1ar4
Report: "Service Incident affecting the Developer Portal service"
Last updateThe incident was restored at 1:18 PM PDT and services continue to be stable. We apologize for any inconvenience this caused you. A full RCA is in flight to identify and track actions to prevent a future occurrence.
The issue was identified and mitigating actions taken to restore services as of ~1:15pm PT. Our technical teams are continuing to monitor the service to ensure no further service disruptions while working to understand the root cause.
We are currently investigation an issue causing Developer Portals to not load. We'll provide updates as they become available. Thank you for your patience!
Report: "Apigee Edge increased error rates for some customers in eu-west-1"
Last updateThis incident has been resolved.
Customer traffic is no longer impacted. We continue to monitor the issue.
Unhealthy router nodes has been replaced, we continue to investigate this incident.
Some of the customers whose Edge Cloud organizations in eu-west-1 region might be experiencing API traffic failures due to unknown issue with router nodes. The incident is under investigation now.
Report: "Apigee Datastore issues (Cassandra) affecting API runtime traffic of Edge Cloud organizations in europe-west1 region"
Last updateThe incident was mitigated several hours after this initial posting but left in monitoring status for an extended time period given similar issues in this region. No further outages were recorded post mitigation. A full RCA is in flight to track actions to prevent a recurrence of this issue. We apologize for the inconvenience this caused.
We are continuing to investigate this issue.
Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.
Report: "Apigee Datastore issues (Cassandra) affecting API runtime traffic of Edge Cloud organizations in europe-west1 region."
Last updateThe emergency maintenance to perform infrastructure maintenance was completed successfully. The technical teams will continue to monitor this cluster closely to ensure the service continues to remain stable including investigating the root cause further for opportunities to prevent recurrence. We apologize for any inconvenience this has caused for your business, but we appreciate your partnership and patience through this.
The incident has been mitigated, and has been stable for the last 6 hours. While we investigate the root cause, we'll continue to monitor this issue closely to ensure service stability. Thank you for your patience and understanding.
We are continuing to investigate this issue.
Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.
Report: "Apigee Datastore issues (Cassandra) affecting API runtime traffic of Edge Cloud organizations in europe-west1 region."
Last updateThe issue is mitigated and it is being monitored.
We are continuing to investigate this issue.
Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.
Report: "Issues affecting customers accessing Apigee Edge UI from EMEA region"
Last updateThis issue has been fully mitigated and continues to remain stable. We will continue to monitor this issue but do not expect any further service impact. Thanks you for your patience and partnership through this.
The issue has been identified and a mitigation is being implemented. Next update will be provided post that implementation.
Edge UI is experiencing intermittent issues for users accessing it from Europe. Proxy develop tab is not loading for some users. We are actively investigating this issue.
Report: "Apigee Integrated Developer Portal - Intermittent Access Issues"
Last updateThe issue with Apigee Integrated Developer Portal has been resolved for all affected users as of Monday, 2023-09-11 12:25 US/Pacific. We thank you for your patience while we worked on resolving the issue.
We are currently investigating issues accessing the Apigee Integrated Developer Portal from Apigee Edge and Apigee X. Users may experience errors like "Failed to load users", "Failed to load zone" or "Service Unavailable when interacting with the developer portal. We will post updates are they become available.
Report: "EdgeCloud: Intermittent runtime impact for some customers in US-West"
Last updateThe issue was identified to a single node issue in one of our Cassandra pods in the US-West region which was mitigated. We're not seeing any further errors post mitigation but will continue to monitor. We'll do a review of this incident and take the necessary actions to improve our systems to prevent a recurrence. Thanks again for your patience and partnership!
We're currently investigating an issue for some customers encountering intermittent 5xx errors for some of their API runtime calls in the US-West region. All of the right resources are engaged in this investigation. No other services are impacted by this issue. We'll provide an update shortly. Thank you for your patience!
Report: "Stale records causing deployment issue in Apigee Edge Cloud"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing an issue with Apigee beginning on Thursday, 2023-06-22 03:18 US/Pacific. Our engineering team continues to investigate the issue. We apologize to all who are affected by the disruption. Customers are not able to perform any deployment activities while using Apigee Edge Cloud
We are currently investigating this issue.
Report: "The Apigee hosted targets calling Azure endpoints may be experiencing timeout starting June 14th 10 AM PDT"
Last updateThis incident has been resolved.
The issue has been mitigated as of June 16th 9:30 AM PT
Starting June 14th 10 AM PDT, the Apigee customers using Azure endpoints in their Apigee Hosted Targets may be experiencing timeouts. The issue is currently investigated by GCP teams. We will update as we progress.
Report: "Issue with Cassandra"
Last updateNo further issues encountered since the last status page update. Marking this incident as resolved. Post incident analysis is in flight to determine preventative actions and other improvement opportunities to reduce the risk of recurrence. Thank you for your patience and partnership through this.
Apigee SRE team has mitigated the cassandra datastore errors and affected customers should be seeing their API runtime traffic error rates return to normal
We have identified an issue with Cassandra for a subset of organizations in eu-west-1 causing datastore errors, and may impact runtime traffic.
Report: "Error in fetching API monitoring via UI or API in Europe"
Last updateThis incident has been resolved.
A fix has been implemented. We are monitoring the results now.
We are experiencing issues with fetching data for API monitoring in Europe region. The users of Edge Cloud will experience intermittent errors when loading api monitoring via the UI or APIs. There is no runtime traffic impact.
Report: "Runtime errors in us-west"
Last updateThis issue has been resolved following the mitigation rollout. All DNS resolutions are stable in the us-west region, and runtime issues are confirmed to be resolved.
The team has rolled out a mitigation, and we are monitoring recovery in us-west. Once we have confidence the region is stable and the mitigation is completed (expected within the next 2 hours), we will begin re-enabling us-west for customers who have been failed out of that region.
We are continuing our work to identify the root cause of the issue implement a mitigation. In the meantime, we have failed over customer who have multi-regional setups that we have identified as impacted to their alternate regions.
We have identified an issue with DNS failures on Apigee Edge in US-West that may result in failed runtime calls. Teams are actively working on remediation.
Report: "EdgeCloud: Datastore incident in us-central1"
Last updateSome EdgeCloud customers would have encountered "datastore" related error responses yesterday for API runtime related requests in the US-Central1 region. All other products and services were unaffected. We were able to identify and mitigate the impact, and have monitored service stability since the incident. If you are experiencing continued impact related to this issue, please file a case with Apigee Support for further investigation. We apologize for the impact this may have had on your business, and will take the necessary actions to prevent similar issues in the future. Impact start time: 2023-04-12 17:03 PDT Impact end time: 2023-04-12 17:48 PDT
Report: "Apigee Edge Scheduled Release - Mon March 13, 2023"
Last updateThe incident has been resolved and the release notes can be viewed at http://apigee.com/docs/release-notes/content/apigee-edge-release-notes
We are continuing to investigate this issue & will post updates here.
No interruptions due to this issue. Release task is complete and the release notes can be viewed at http://apigee.com/docs/release-notes/content/apigee-edge-release-notes
We have an upcoming release window scheduled for Mon March 13, 2023. This release is not expected to cause any interruption to Apigee Edge UI. Once the release is complete, release notes will be published at: http://apigee.com/docs/release-notes/content/apigee-edge-release-notes
Report: "Apigee Log4J CVE-2021-44228 Response Update"
Last updateThis incident has been resolved.
Log4j v1.x: We are aware of the recent upgrade to CVE-2021-4104 which affects versions of Log4j 1 which are used on Apigee Edge and OPDK. This CVE requires a specific configuration of Log4j 1 which Apigee Edge does not use and which Apigee OPDK does not ship as a default configuration. User Supplied Log4j 1 Java Callouts Apigee SaaS (X and Edge): Customers can upload vulnerable configurations of Log4j 1 in their custom resources, but CVE-2021-4104 is mitigated due to Java Security Manager restrictions. Apigee OPDK and Hybrid Users should upgrade all of their custom resources of Log4j to the latest version. Users can mitigate uploaded vulnerable versions of Log4j 1 in Hybrid and OPDK by enabling a custom java callout security policy: Hybrid: https://cloud.google.com/apigee/docs/api-platform/develop/adding-custom-java-callout-security-policy OPDK: https://docs.apigee.com/api-platform/reference/java-permission-reference
Updated December 15, 2021 15:22 PST Google Apigee is actively following the security vulnerability in the open-source Apache “Log4j 2" utility (CVE-2021-44228 and CVE-2021-45046). We encourage you to update to the latest version of Log4j 2. We are currently assessing the potential impact of the vulnerability for Apigee products and services. This is an ongoing event and we will continue to provide updates through this page and our customer communications channels. Background: The Apache Log4j 2 utility is a commonly used component for logging requests. On December 9, 2021, a vulnerability was reported that could allow a system running Apache Log4j 2 version 2.14.1 or below to be compromised and allow an attacker to execute arbitrary code. On December 10, 2021, NIST published a critical Common Vulnerabilities and Exposure alert, CVE-2021-44228. More specifically, Java Naming Directory Interface (JNDI) features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from remote servers when message lookup substitution is enabled. ==== Platform ==== Apigee X: Apigee X does not use Log4j 2. Apigee Hybrid: Current supported versions of Hybrid do not use Log4j 2. Unsupported versions of Hybrid used Log4j 1.x, but it is not included in any of the currently supported versions. Customers on Hybrid 1.4 or lower are unaffected by this CVE but should still upgrade to a supported version. Apigee Edge: Apigee Edge’s default configuration contained Log4j 2 but was not vulnerable to Log4j 2 (CVE-2021-44228). Apigee OPDK: Apigee OPDK’s default configuration contained Log4j 2 but was not vulnerable to Log4j 2 (CVE-2021-44228). The service “apigee-machinekey” includes Log4j 2 and does not process any user-provided input. It is not susceptible to the vulnerability CVE-2021-44228. ==== User Supplied Log4j 2 Instances ==== Apigee SaaS (X and Edge): Customers can upload vulnerable versions of Log4j 2 in their custom resources, but CVE-2021-4228 is mitigated due to Java Security Manager restrictions. Apigee Hybrid and OPDK: Users can mitigate uploaded vulnerable versions of Log4j 2 in Hybrid and OPDK by enabling a custom java callout security policy: Hybrid: https://cloud.google.com/apigee/docs/api-platform/develop/adding-custom-java-callout-security-policy OPDK: https://docs.apigee.com/api-platform/reference/java-permission-reference ==== Versions Updates ==== Log4j v1.x: Apigee Edge and Apigee OPDK contain Log4j 1.x and Log4j 2.x libraries. Log4j 1 is not part of this particular assessment. All instances of Log4j 1 will be upgraded across all Apigee products in upcoming releases to the latest version of Log4j 2. Upgrades to SaaS services and releases for OPDK are expected in January. ==== More Information ==== Information on this page is based on findings in our ongoing investigations. Please see these helpful articles published to the Apigee Community: How to Detect Network Probes on Traversing Your Apigee Proxies- https://www.googlecloudcommunity.com/gc/Apigee/Detecting-Attempts-to-Exploit-Log4j-CVE-2021-44228-on-Apigee/td-p/178123 How To Mitigate Log4Jv2 Attacks CVE-2021-44228 Traversing Apigee Proxies - https://www.googlecloudcommunity.com/gc/Apigee/How-To-Block-requests-that-exploit-Log4j-CVE-2021-44228-on/td-p/178138
Report: "Errors while login with SAML in Apigee UI"
Last updateThe fix was completely rolled out in production and this incident is now fully resolved.
We have identified the problem with the unavailable SAML login button which some customers may be experiencing. The fix is currently being tested. Once the fix is confirmed, we'll provide an ETA for the delivery of this solution. Thank you for your patience as we work through this.
The 'Validation of protocol message signature failed' errors observed on SAML login to Edge should be mitigated and we're still investigating an issue where the SAML login button is not available.
A fix has been implemented and we are monitoring the results.
We are investigating failures while logging in to Apigee UI. Customers using saml / sso to login to Edge Cloud UI are experiencing errors. This incident does not cause any traffic impact.
Report: "Apigee Live Portals problem"
Last updateThis incident has been resolved as of 10:44 AM PT.
Apigee Live Portals management operations are currently experiencing intermittent issues with our backend services. We are aware of the problem and actively working to mitigate its effects. Existing Live portals themselves should not be affected.
Report: "Apigee Support Portal Problems"
Last updateWe have resolved the problem. The portal can be used to create and update cases.
We are continuing to investigate this issue.
We have identified a problem with Apigee Support portal and currently investigating the problem. Impact : Creating new cases or updating cases using the portal is returning errors. For submitting issues, please use this form: https://docs.google.com/forms/d/19BPxvgMXNE6_0khhDA23wr1f1qY_QNlGkhtC91Lk_JA OR if you have access to the GCP support portal, please file cases there in the interim. Thanks for your patience and understanding.
Report: "Users are unable to access the Apigee Edge UI. The timeout errors and issues with the permissions are reported when users are trying to access the Apigee Edge UI."
Last updateThis incident has been resolved.
We are currently investigating the issue. Further updates will follow.
We are continuing to monitor for any further issues.
The cause of the issue has been identified at the Apigee load balancers. We have mitigated the incident and current being monitored by the Apigee team.
We are currently investigating the issue. We will update as we progress the investigation. Runtime traffic is not affected.
Report: "Networking issue in US-East"
Last updateAll mitigations have now been implemented and completed.
An initial mitigation has been deployed, and a further mitigation step will be done tomorrow at ~8am PDT, lasting for ~ 2 hours.
The issue was limited to the time period of 9:45-10:40AM PDT. We have found the causal factor and are implementing a remediation to ensure it does not recur. We will update this page when the remediation is completed, within the next 4 hours.
We are currently investigating an issue affecting some customers in US-East1 who are seeing connection timeouts while connecting to target servers.
Report: "Intermittent issues affecting customers accessing apigee Edge UI from EMEA region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Intermittent issues affecting customers accessing Apigee Edge UI from EMEA region loading userroles and proxy trace pages. There is no impact to runtime traffic.
Report: "Errors in Apigee Analytics report jobs"
Last updateThis incident has been resolved.
Services continue to look good post restoration. Resolving this incident.
A fix has been implemented and we are monitoring the results.
We are investigating errors with Apigee Analytics report jobs.
Report: "Intermittent Datastore errors in us-central1 region"
Last updateThis incident has been resolved.
Services continue to look good post restoration. Resolving this incident.
A fix has been implemented and we are monitoring the results.
We are currently investigating intermittent datastore errors affecting subset of customers in us-central1.
Report: "Bad Gateway Error from Management API"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently aware of an issue with our Management APIs resulting in "Bad Gateway Error", and are working internally to identify the cause and fix the issue.
Report: "Intermittent issues affecting customers accessing Apigee Edge UI"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently actively investigating Intermittent issues affecting customers accessing Apigee Edge UI loading pages. More updates will follow.
Report: "Apigee API Services are experiencing elevated error rates, latencies or service unavailability"
Last updateThis incident has been resolved.
The issue was mitigated by 2022-07-19 17:08:00 UTC. The latency and "datastore errors" seen during runtime are no longer present. Customers performing proxy deployments to europe-west2 might have experienced deployment errors before the mitigation. More updates will follow.
Apigee services are experiencing elevated error rates, latencies or service unavailability in europe-west2. Google teams are actively working to restore the services, see details at https://status.cloud.google.com/incidents/fmEL9i2fArADKawkZAa2.
Report: "Apigee Integrated Portals and Specs intermittent loading issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
The issue is also impacting Analytics Services. Customers may experience intermittent failures when trying to access those pages via the UI or APIs.
We are continuing to investigate this issue.
we are currently actively investigating an issue that is causing intermittent page loads on Apigee Integrated Portals, Apigee Integrated Portals Admin, and Specs on apigee.com.
Report: "Errors in Apigee UI Login via SAML"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are investigating reports of Apigee Edge UI login issues via SAML. Users may not be able to access the Apigee Edge UI if SAML is enabled on the Apigee Edge UI. This incident does not affect customers without SAML enabled. This incident does not affect runtime traffic. We will post updates here as we identify and resolve this issue.
Report: "Edge UI and Management API Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix has been implemented.
After further investigation, this issue is not limited to SAML enabled orgs. All of the right resources are engaged to identify and solve this issue. We'll provide an update as soon as one becomes available. Thanks again for your patience and understanding.
The Apigee team continues to investigate this issue which seems to impact SAML enabled orgs. The troubleshooting is progressing but unfortunately we do not have an ETA as this time. We do apologize for the delay in solving this issue.
We've done a partial rollback of a change in US East as a mitigating action but hasn't solved the problem. The Apigee team continues to work this critical incident, and will provide an update shortly. This issue is impacting administrative access to the Edge UI for some users, and should not be impacting runtime traffic. We apologize for this inconvenience, and appreciate your patience!
Users of the Apigee Edge UI and Apigee Management APIs may receive error messages accessing certain product areas. Apigee teams are actively working to correct these errors.
Report: "Apigee SSO DR Testing"
Last updateThe testing has ended.
Apigee SSO will be conducting disaster recovery testing starting at 3:00 PM PDT. This is expected to last one hour. This affects only Apigee Edge customers using Apigee SSO for user logins into the Management UI. There is no expected impact to runtime traffic. User logins normally served by US East may see increased request latency for the duration of the test. No other impact is expected.
Report: "Login errors observed on SAML-enabled Apigee Integrated Developer portals"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Starting at approximately 6 AM PDT, customers logging in to SAML-enabled Apigee Integrated Developer portals started encountering an elevated rate of login errors. We are actively investigating the issue and will update this page periodically.
Report: "Retroactive Incident: Datastore issue in US-East causing intermittent errors"
Last updateThis is a retroactive post on an incident that occurred this morning between 05:39-06:18 PST. We detected elevated errors with our Cassandra service in the US-East region related to a single node event. During this time, customer API calls in this region would have seen intermittent API 5XX errors for approximately 5-10% of their calls for approx. 40 mins. Services in all other regions were not impacted by this incident. The node in question was removed from service to mitigate impact pending further analysis. We're investigating the root cause of this incident and will make the necessary changes to prevent a recurrence of this issue. We apologize for the inconvenience this may have caused you and your business -- and we don't take that lightly. If you have any further questions or may be seeing impacts related to this, please file a case with Support.
Report: "5xx Error Rates due to AWS"
Last updateThis incident has been resolved.
AWS has recovered from the recent unavailability of zone USE1-AZ4.
Due to an ongoing incident in AWS, Apigee Edge Cloud customers hosted in US-EAST-1 may be experiencing 5xx errors. https://status.aws.amazon.com/
Report: "Log4j Vulnerability - CVE-2021-44228"
Last updateApigee has conducted a detailed investigation and we believe that the Edge and OPDK products are not vulnerable to this exploit.
Apigee acknowledges the reports received regarding the potential vulnerability of log4j - https://nvd.nist.gov/vuln/detail/CVE-2021-44228 . We are actively investigating the impact, and at this stage we have yet to determine the scope of impact within Apigee’s core infrastructure. This space will be updated with findings.
Report: "Increased latency and timeout in europe-west1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Analytics Data Delays in europe-west1"
Last updateAs at 2021-09-01 21:40 US/Pacific, all analytics data backlogs have been processed. The Analytics data delay has been resolved and Analytics services are fully recovered.
We have deployed a new pipeline in europe-west2 region to process the analytics data backlogs. It is expected to take a couple of hours from 2021-09-01 21:40 US/Pacific to catchup.
Apigee has identified delays in data analytics that may affect customer organizations located in the europe-west1 region. Symptoms: Data analytics dashboards may not be up-to-date. You may not be able to view realtime analytics records starting from Wednesday, 2021-09-01 17:00 US/Pacific. Analytics APIs continue to work. Runtime traffic is not affected. Apigee is actively working on addressing this issue. We will post updates here as we identify and resolve this issue.
Report: "Apigee UI 50x Errors"
Last updateThis incident has been resolved.
We have taken corrective action to resolve this. Apigee teams are actively monitoring to ensure continued stability.
We are continuing to investigate this issue.
We are investigating reports of Apigee Edge UI errors. Users may see "Error: 50x" displayed in the UI, or be unable to access various portions of the UI. We will post updates here as we identify and resolve this issue.
Report: "API Monitoring Down"
Last updateThis incident has been resolved.
The API Monitoring issue has been mitigated and we are currently monitoring. We expect that it will take up to two hours for all monitoring data to backfill.
Apigee Sense has recovered, but the API Monitoring issue is still being mitigated. We expect a resolution within the hour. Monitoring Data will be backfilled when service is restored.
We have identified the root cause and are working on a mitigation.
API Monitoring is currently unavailable. We are investigating the cause and will update when we have more information, or within 30 minutes. Runtime traffic is unaffected, but you will not be able to access any Monitoring data at this time.
Report: "Syslog logging failures"
Last updateThis incident has been resolved.
The Message Processor fix has been rolled out, and we are monitoring the results for any errors.
Apigee is aware of a bug introduced in a recent version of our Message Processors which occasionally affects Message Logging policies which use Syslog. Symptoms: Syslog servers stop receiving logs from Apigee Customer Action and Mitigation: Please file a P1 Support Case with Apigee Support. The mitigation will be a restart of your Message Processors and a rollback to a previous stable version. Fix: We have a patch which will be deployed to all pods over the coming week.
Report: "Apigee Integrated Portals loading issue"
Last updateA fix was identified and rolled out. This incident is now resolved.
We are aware of an issue that some Apigee Integrated Portals customers may experience issues with their portals loading. Apigee teams are actively working to mitigate this. We will provide updates periodically.
Report: "Apigee X instance creation fails due to GKE release"
Last updateThis incident has been resolved.
we are currently investigating an issue affecting only Apigee X Paid and Eval customers onboarding is broken due to this issue. Apigee Edge cloud remains unaffected by this issue.
Report: "Datastore Errors in AWS EU-Central-1"
Last updateAWS Availability Zone euc1-az1 experienced an outage today at roughly 1:25PM Pacific. In-flight requests to our database servers in that zone were rejected as the machines were unavailable, but only in-flight requests at the instance of the outage were affected. There is no ongoing impact, as our High Availability immediately routed all incoming requests to available nodes. We will continue to monitor the situation but expect no other issues to arise.
We are continuing to work on a fix for this issue.
Customers may see an increase in Datastore errors for runtime calls in EU-Central-1, including access to KVM, Cache, Quota, and KMS policies. We are working to restore services.
Report: "Apigee UI Issues in US-EAST"
Last updateThis incident has been resolved.
Apigee teams have released a fix, and are continuing to monitor for any other impact.
Apigee teams have released a fix, and are continuing to monitor for any other impact.
We are actively investigating an issue with the Apigee UI in the US-EAST region. The proxy editor, trace, and offline trace features may be impacted. This is not affecting runtime traffic.
Report: "Apigee Support Portal Issues"
Last updateThis incident has been resolved.
The issue appears to be resolved, but we are continuing to monitor the situation.
Apigee's Support Portal is experiencing intermittent issues due to a service provider outage. If you are unable to submit a case, please submit via this form, and we will contact you immediately.: https://docs.google.com/forms/d/19BPxvgMXNE6_0khhDA23wr1f1qY_QNlGkhtC91Lk_JA