Apigee

Is Apigee Down Right Now? Check if there is a current outage ongoing.

Apigee is currently Operational

Last checked from Apigee's official status page

Historical record of incidents for Apigee

Report: "Apigee Edge for Public Cloud - Management Infrastructure maintenance is scheduled for Friday, May 16, 2025, from 8:00 PM PT to 10:00 PM PT"

Last update
Scheduled

Apigee Edge for Public Cloud - Management Infrastructure maintenance is scheduled for Friday, May 16, 2025, from 8:00 PM PT to 10:00 PM PT.During this scheduled maintenance period, the listed services may experience a single unavailability event for up to 10 minutes. Invocations of these services may return a 4xx status code:* Management Services UI* Management Services API(s)* Monetization Management APIs* New Proxy Deployments* DevPortal Service* Microgateway Services* Edge Query (Analytics Stats APIs)All inquiries should be directed to Apigee Support at https://cloud.google.com/support-hub/FAQs* Will this change affect runtime traffic?No* Will this change have any effect on the monetization product?Please be advised that while the monetization runtime will remain unaffected, the monetization management APIs will experience a period of unavailability during the scheduled maintenance window.* Will this change impact new proxy deployments or updating existing proxy revisions?During this scheduled maintenance period, customers may not be able to deploy or update their proxies.* Will this change affect Analytics?While Analytics statistics APIs will be unavailable during the maintenance period, no analytics data will be lost. The entirety of analytics data will be accessible following the completion of the maintenance.* Will this change impact EMG?During the scheduled maintenance window, the deployment and scaling of new Edge MicroGateway (EMG) instances will be suspended. Existing EMG instances will not be affected. However, the restart of the Edge MicroGateway service is not possible during this period.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Report: "Apigee released OPDK 4.53.00.04 , a patch version for Apigee Edge for Private Cloud"

Last update
In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Apigee released OPDK 4.53.00.04, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:OPDK 4.53.00.04: https://docs.apigee.com/release/notes/4530004-edge-private-cloud-release-notes

Report: "Apigee released OPDK 4.52.02.03 , a patch version for Apigee Edge for Private Cloud"

Last update
In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Apigee released OPDK 4.52.02.03, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:4.52.02.03: https://docs.apigee.com/release/notes/4520203-edge-private-cloud-release-notes

Report: "Apigee Edge for Public Cloud Monetization infrastructure maintenance schedule"

Last update
Scheduled

Apigee Monetization Infrastructure maintenance for Apigee Edge for Public Cloud is scheduled as follows, per region.During the maintenance period, Apigee Monetization billing, reports, and Monetization Management APIs will be unavailable for customers with Monetization enabled. Apigee Edge runtime traffic will remain unaffected by this maintenance activity. Rating of monetized runtime calls will recommence upon system restoration. Following the completion of rating, data for the affected period will be incorporated into the reports for that day.Apigee Edge customers without the Monetization feature enabled will experience no impact.Note: This maintenance update does not introduce new features. The system and its existing functionalities will continue to operate as previously configured.Should you require assistance or have any inquiries, kindly reach out to Apigee Support via the following link: https://cloud.google.com/apigee/support/.Components affected for Customers with Monetization enabled: * Apigee Monetization billing and reports. * Monetization related APIs # Management API calls on developer/apps/product # Mint API calls on Rate plans / Product bundles / NotificationsRegion Maintenance WindowAsia Pacific Apr 22, 2025 11:30 AM - Apr 22, 2025 5:30 PM UTCAsia Pacific Apr 23, 2025 11:30 AM - Apr 23, 2025 5:30 PM UTCEMEA Apr 23, 2025 3:30 AM - Apr 23, 2025 9:30 AM UTCEMEA Apr 23, 2025 3:30 AM - Apr 23, 2025 9:30 AM UTCAmericas Apr 28, 2025 8:30 AM - Apr 28, 2025 2:30 PM UTCAmericas Apr 29, 2025 8:30 AM - Apr 29, 2025 2:30 PM UTCFAQs:What is changing?A maintenance activity will be performed on the backend systems supporting Monetization. Is any customer action necessary?No.Will customers be able to access Monetization reports during the maintenance period ?Reporting functionality will be unavailable during this scheduled maintenance period for customers with monetization enabled. Customers without monetization enabled will experience no impact.Monetized runtime call rating will recommence upon system restoration. Following the rating process, the corresponding data for the affected period will be incorporated into the daily reports.Will customers be able to use Management Functionality/APIs during this period ?For customers with monetization enabled, the following Management APIs will be unavailable.1. API products2. App3. Developer4. Company5. Rate plan6. Reports7. Notifications8. Transactions SearchThis is an indicative list of APIs concerned with Monetization-specific entities or those utilized by the Monetization system. Management APIs addressing entities such as proxies and deployments, which do not directly affect Monetization, may also be used. For Customers who have Monetization disabled, Management APIs remain accessible.How can customers verify their Monetization enablement status?To verify Monetization enablement status, customers can utilize this API: https://apidocs.apigee.com/docs/organizations/1/routes/organizations/%7Borg_name%7D/get. A response containing the property as depicted below indicates that Monetization is enabled. Conversely, the absence of the property or a false value signifies that it is disabled.{ "name": "features.isMonetizationEnabled", "value": "true"}

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Report: "Apigee released OPDK 4.53.00.03, a patch version for Apigee Edge for Private Cloud"

Last update
In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Apigee released OPDK 4.53.00.03, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:4.53.00.03: https://docs.apigee.com/release/notes/4530003-edge-private-cloud-release-notes

Report: "Apigee released OPDK 4.52.02.02, a patch version for Apigee Edge for Private Cloud"

Last update
In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Apigee released OPDK 4.52.02.02, a patch version for Apigee Edge for Private Cloud.Please refer to the release notes for details:4.52.02.02: https://docs.apigee.com/release/notes/4520202-edge-private-cloud-release-notes

Report: "Apigee Edge for Public Cloud Single Sign-On (SSO) patch release is scheduled for Sunday, March 16, 2025, at 9:30 PM Pacific Time."

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

An Apigee Edge for Public Cloud Single Sign-On (SSO) patch release is scheduled to commence at 9:30 PM Pacific Time (PT) on Sunday, March 16, 2025. The deployment will occur in a phased manner by region. It is anticipated that this release will not disrupt Apigee Edge services.Release Notes: https://docs.apigee.com/release/notes/250316-apigee-edge-public-cloud-release-notes-sso

Report: "Apigee Edge increased error rates for some customers in europe-west2"

Last update
resolved

This incident has been resolved.

monitoring

Issue has been mitigated

investigating

Some of Apigee Edge Public Cloud customers in europe-west2 experiencing elevated number of HTTP 500 responses and increased latencies due to infrastructure issues in Apigee Datastore cluster.

Report: "Traffic disruption in eu-west-1, elevated number of HTTP 500 errors"

Last update
resolved

This incident has been resolved.

monitoring

Issue has been mitigated, normal traffic flow resumed. Team suspects partial network outage in single region, we continue investigating.

investigating

We are currently investigating issues affecting Apigee Edge customers in eu-west-1. Some customers may observe elevated number of HTTP 500 responses due to issues with Apigee Datastore in the region.

Report: "Security Advisory: [CVE-2024-6387]: [OpenSSH Vulnerability]"

Last update
identified

A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Please refer to Google Cloud Security bulletin published: GCP-2024-040 : https://cloud.google.com/apigee/docs/security-bulletins/security-bulletins#gcp-2024-040

Report: "Security Advisory: [CVE-2024-6387]: [OpenSSH Vulnerability]"

Last update
Identified

A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Please refer to Google Cloud Security bulletin published: GCP-2024-040 : https://cloud.google.com/apigee/docs/security-bulletins/security-bulletins#gcp-2024-040

Report: "NAT Incident affecting some Customers in us-central1"

Last update
resolved

This incident has been resolved.

monitoring

Mitigation was successful, we're now monitoring the incident and confirming impact.

investigating

Team has narrowed down potential cause and performing a mitigation attempt

investigating

Team suspects incorrect configuration leading to some percentage of customer traffic affected. We are actively investigating this issue.

Report: "Networking Disruption for GCP Customers"

Last update
resolved

This issue has been resolved for all affected projects as of Thursday, 2024-05-16 18:10 US/Pacific. We thank you for your patience while we worked on resolving the issue.

monitoring

Most regions' network functionality has recovered. We are still actively monitoring for any remaining issues.

investigating

Some customers hosted on GCP may experience latency and/or 5XX errors in their runtimes due to a Networking Issue on GCP. Some Edge Cloud orgs may experience errors during API proxy deployment as well. The impact to most organizations were resolved automatically and the Apigee team is investigating the issue further and taking appropriate steps to mitigate it further. Apigee service will recover as soon as this Networking issue is resolved. The underlying issue status can be tracked at https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#2c2sBHWU84yPDJ8y1ar4

Report: "Service Incident affecting the Developer Portal service"

Last update
resolved

The incident was restored at 1:18 PM PDT and services continue to be stable. We apologize for any inconvenience this caused you. A full RCA is in flight to identify and track actions to prevent a future occurrence.

monitoring

The issue was identified and mitigating actions taken to restore services as of ~1:15pm PT. Our technical teams are continuing to monitor the service to ensure no further service disruptions while working to understand the root cause.

investigating

We are currently investigation an issue causing Developer Portals to not load. We'll provide updates as they become available. Thank you for your patience!

Report: "Apigee Edge increased error rates for some customers in eu-west-1"

Last update
resolved

This incident has been resolved.

monitoring

Customer traffic is no longer impacted. We continue to monitor the issue.

identified

Unhealthy router nodes has been replaced, we continue to investigate this incident.

investigating

Some of the customers whose Edge Cloud organizations in eu-west-1 region might be experiencing API traffic failures due to unknown issue with router nodes. The incident is under investigation now.

Report: "Apigee Datastore issues (Cassandra) affecting API runtime traffic of Edge Cloud organizations in europe-west1 region"

Last update
resolved

The incident was mitigated several hours after this initial posting but left in monitoring status for an extended time period given similar issues in this region. No further outages were recorded post mitigation. A full RCA is in flight to track actions to prevent a recurrence of this issue. We apologize for the inconvenience this caused.

investigating

We are continuing to investigate this issue.

investigating

Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.

Report: "Apigee Datastore issues (Cassandra) affecting API runtime traffic of Edge Cloud organizations in europe-west1 region."

Last update
resolved

The emergency maintenance to perform infrastructure maintenance was completed successfully. The technical teams will continue to monitor this cluster closely to ensure the service continues to remain stable including investigating the root cause further for opportunities to prevent recurrence. We apologize for any inconvenience this has caused for your business, but we appreciate your partnership and patience through this.

monitoring

The incident has been mitigated, and has been stable for the last 6 hours. While we investigate the root cause, we'll continue to monitor this issue closely to ensure service stability. Thank you for your patience and understanding.

investigating

We are continuing to investigate this issue.

investigating

Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.

Report: "Apigee Datastore issues (Cassandra) affecting API runtime traffic of Edge Cloud organizations in europe-west1 region."

Last update
resolved

The issue is mitigated and it is being monitored.

investigating

We are continuing to investigate this issue.

investigating

Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.

Report: "Issues affecting customers accessing Apigee Edge UI from EMEA region"

Last update
resolved

This issue has been fully mitigated and continues to remain stable. We will continue to monitor this issue but do not expect any further service impact. Thanks you for your patience and partnership through this.

identified

The issue has been identified and a mitigation is being implemented. Next update will be provided post that implementation.

investigating

Edge UI is experiencing intermittent issues for users accessing it from Europe. Proxy develop tab is not loading for some users. We are actively investigating this issue.

Report: "Apigee Integrated Developer Portal - Intermittent Access Issues"

Last update
resolved

The issue with Apigee Integrated Developer Portal has been resolved for all affected users as of Monday, 2023-09-11 12:25 US/Pacific. We thank you for your patience while we worked on resolving the issue.

investigating

We are currently investigating issues accessing the Apigee Integrated Developer Portal from Apigee Edge and Apigee X. Users may experience errors like "Failed to load users", "Failed to load zone" or "Service Unavailable when interacting with the developer portal. We will post updates are they become available.

Report: "EdgeCloud: Intermittent runtime impact for some customers in US-West"

Last update
resolved

The issue was identified to a single node issue in one of our Cassandra pods in the US-West region which was mitigated. We're not seeing any further errors post mitigation but will continue to monitor. We'll do a review of this incident and take the necessary actions to improve our systems to prevent a recurrence. Thanks again for your patience and partnership!

investigating

We're currently investigating an issue for some customers encountering intermittent 5xx errors for some of their API runtime calls in the US-West region. All of the right resources are engaged in this investigation. No other services are impacted by this issue. We'll provide an update shortly. Thank you for your patience!

Report: "Stale records causing deployment issue in Apigee Edge Cloud"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are experiencing an issue with Apigee beginning on Thursday, 2023-06-22 03:18 US/Pacific. Our engineering team continues to investigate the issue. We apologize to all who are affected by the disruption. Customers are not able to perform any deployment activities while using Apigee Edge Cloud

investigating

We are currently investigating this issue.

Report: "The Apigee hosted targets calling Azure endpoints may be experiencing timeout starting June 14th 10 AM PDT"

Last update
resolved

This incident has been resolved.

investigating

The issue has been mitigated as of June 16th 9:30 AM PT

investigating

Starting June 14th 10 AM PDT, the Apigee customers using Azure endpoints in their Apigee Hosted Targets may be experiencing timeouts. The issue is currently investigated by GCP teams. We will update as we progress.

Report: "Issue with Cassandra"

Last update
resolved

No further issues encountered since the last status page update. Marking this incident as resolved. Post incident analysis is in flight to determine preventative actions and other improvement opportunities to reduce the risk of recurrence. Thank you for your patience and partnership through this.

monitoring

Apigee SRE team has mitigated the cassandra datastore errors and affected customers should be seeing their API runtime traffic error rates return to normal

investigating

We have identified an issue with Cassandra for a subset of organizations in eu-west-1 causing datastore errors, and may impact runtime traffic.

Report: "Error in fetching API monitoring via UI or API in Europe"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented. We are monitoring the results now.

investigating

We are experiencing issues with fetching data for API monitoring in Europe region. The users of Edge Cloud will experience intermittent errors when loading api monitoring via the UI or APIs. There is no runtime traffic impact.

Report: "Runtime errors in us-west"

Last update
resolved

This issue has been resolved following the mitigation rollout. All DNS resolutions are stable in the us-west region, and runtime issues are confirmed to be resolved.

monitoring

The team has rolled out a mitigation, and we are monitoring recovery in us-west. Once we have confidence the region is stable and the mitigation is completed (expected within the next 2 hours), we will begin re-enabling us-west for customers who have been failed out of that region.

identified

We are continuing our work to identify the root cause of the issue implement a mitigation. In the meantime, we have failed over customer who have multi-regional setups that we have identified as impacted to their alternate regions.

identified

We have identified an issue with DNS failures on Apigee Edge in US-West that may result in failed runtime calls. Teams are actively working on remediation.

Report: "EdgeCloud: Datastore incident in us-central1"

Last update
resolved

Some EdgeCloud customers would have encountered "datastore" related error responses yesterday for API runtime related requests in the US-Central1 region. All other products and services were unaffected. We were able to identify and mitigate the impact, and have monitored service stability since the incident. If you are experiencing continued impact related to this issue, please file a case with Apigee Support for further investigation. We apologize for the impact this may have had on your business, and will take the necessary actions to prevent similar issues in the future. Impact start time: 2023-04-12 17:03 PDT Impact end time: 2023-04-12 17:48 PDT

Report: "Apigee Edge Scheduled Release - Mon March 13, 2023"

Last update
resolved

The incident has been resolved and the release notes can be viewed at http://apigee.com/docs/release-notes/content/apigee-edge-release-notes

investigating

We are continuing to investigate this issue & will post updates here.

investigating

No interruptions due to this issue. Release task is complete and the release notes can be viewed at http://apigee.com/docs/release-notes/content/apigee-edge-release-notes

investigating

We have an upcoming release window scheduled for Mon March 13, 2023. This release is not expected to cause any interruption to Apigee Edge UI. Once the release is complete, release notes will be published at: http://apigee.com/docs/release-notes/content/apigee-edge-release-notes

Report: "Apigee Log4J CVE-2021-44228 Response Update"

Last update
resolved

This incident has been resolved.

monitoring

Log4j v1.x: We are aware of the recent upgrade to CVE-2021-4104 which affects versions of Log4j 1 which are used on Apigee Edge and OPDK. This CVE requires a specific configuration of Log4j 1 which Apigee Edge does not use and which Apigee OPDK does not ship as a default configuration. User Supplied Log4j 1 Java Callouts Apigee SaaS (X and Edge): Customers can upload vulnerable configurations of Log4j 1 in their custom resources, but CVE-2021-4104 is mitigated due to Java Security Manager restrictions. Apigee OPDK and Hybrid Users should upgrade all of their custom resources of Log4j to the latest version. Users can mitigate uploaded vulnerable versions of Log4j 1 in Hybrid and OPDK by enabling a custom java callout security policy: Hybrid: https://cloud.google.com/apigee/docs/api-platform/develop/adding-custom-java-callout-security-policy OPDK: https://docs.apigee.com/api-platform/reference/java-permission-reference

monitoring

Updated December 15, 2021 15:22 PST Google Apigee is actively following the security vulnerability in the open-source Apache “Log4j 2" utility (CVE-2021-44228 and CVE-2021-45046). We encourage you to update to the latest version of Log4j 2. We are currently assessing the potential impact of the vulnerability for Apigee products and services. This is an ongoing event and we will continue to provide updates through this page and our customer communications channels. Background: The Apache Log4j 2 utility is a commonly used component for logging requests. On December 9, 2021, a vulnerability was reported that could allow a system running Apache Log4j 2 version 2.14.1 or below to be compromised and allow an attacker to execute arbitrary code. On December 10, 2021, NIST published a critical Common Vulnerabilities and Exposure alert, CVE-2021-44228. More specifically, Java Naming Directory Interface (JNDI) features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from remote servers when message lookup substitution is enabled. ==== Platform ==== Apigee X: Apigee X does not use Log4j 2. Apigee Hybrid: Current supported versions of Hybrid do not use Log4j 2. Unsupported versions of Hybrid used Log4j 1.x, but it is not included in any of the currently supported versions. Customers on Hybrid 1.4 or lower are unaffected by this CVE but should still upgrade to a supported version. Apigee Edge: Apigee Edge’s default configuration contained Log4j 2 but was not vulnerable to Log4j 2 (CVE-2021-44228). Apigee OPDK: Apigee OPDK’s default configuration contained Log4j 2 but was not vulnerable to Log4j 2 (CVE-2021-44228). The service “apigee-machinekey” includes Log4j 2 and does not process any user-provided input. It is not susceptible to the vulnerability CVE-2021-44228. ==== User Supplied Log4j 2 Instances ==== Apigee SaaS (X and Edge): Customers can upload vulnerable versions of Log4j 2 in their custom resources, but CVE-2021-4228 is mitigated due to Java Security Manager restrictions. Apigee Hybrid and OPDK: Users can mitigate uploaded vulnerable versions of Log4j 2 in Hybrid and OPDK by enabling a custom java callout security policy: Hybrid: https://cloud.google.com/apigee/docs/api-platform/develop/adding-custom-java-callout-security-policy OPDK: https://docs.apigee.com/api-platform/reference/java-permission-reference ==== Versions Updates ==== Log4j v1.x: Apigee Edge and Apigee OPDK contain Log4j 1.x and Log4j 2.x libraries. Log4j 1 is not part of this particular assessment. All instances of Log4j 1 will be upgraded across all Apigee products in upcoming releases to the latest version of Log4j 2. Upgrades to SaaS services and releases for OPDK are expected in January. ==== More Information ==== Information on this page is based on findings in our ongoing investigations. Please see these helpful articles published to the Apigee Community: How to Detect Network Probes on Traversing Your Apigee Proxies- https://www.googlecloudcommunity.com/gc/Apigee/Detecting-Attempts-to-Exploit-Log4j-CVE-2021-44228-on-Apigee/td-p/178123 How To Mitigate Log4Jv2 Attacks CVE-2021-44228 Traversing Apigee Proxies - https://www.googlecloudcommunity.com/gc/Apigee/How-To-Block-requests-that-exploit-Log4j-CVE-2021-44228-on/td-p/178138

Report: "Errors while login with SAML in Apigee UI"

Last update
resolved

The fix was completely rolled out in production and this incident is now fully resolved.

monitoring

We have identified the problem with the unavailable SAML login button which some customers may be experiencing. The fix is currently being tested. Once the fix is confirmed, we'll provide an ETA for the delivery of this solution. Thank you for your patience as we work through this.

monitoring

The 'Validation of protocol message signature failed' errors observed on SAML login to Edge should be mitigated and we're still investigating an issue where the SAML login button is not available.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are investigating failures while logging in to Apigee UI. Customers using saml / sso to login to Edge Cloud UI are experiencing errors. This incident does not cause any traffic impact.

Report: "Apigee Live Portals problem"

Last update
resolved

This incident has been resolved as of 10:44 AM PT.

investigating

Apigee Live Portals management operations are currently experiencing intermittent issues with our backend services. We are aware of the problem and actively working to mitigate its effects. Existing Live portals themselves should not be affected.

Report: "Apigee Support Portal Problems"

Last update
resolved

We have resolved the problem. The portal can be used to create and update cases.

investigating

We are continuing to investigate this issue.

investigating

We have identified a problem with Apigee Support portal and currently investigating the problem. Impact : Creating new cases or updating cases using the portal is returning errors. For submitting issues, please use this form: https://docs.google.com/forms/d/19BPxvgMXNE6_0khhDA23wr1f1qY_QNlGkhtC91Lk_JA OR if you have access to the GCP support portal, please file cases there in the interim. Thanks for your patience and understanding.

Report: "Users are unable to access the Apigee Edge UI. The timeout errors and issues with the permissions are reported when users are trying to access the Apigee Edge UI."

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating the issue. Further updates will follow.

monitoring

We are continuing to monitor for any further issues.

monitoring

The cause of the issue has been identified at the Apigee load balancers. We have mitigated the incident and current being monitored by the Apigee team.

investigating

We are currently investigating the issue. We will update as we progress the investigation. Runtime traffic is not affected.

Report: "Networking issue in US-East"

Last update
resolved

All mitigations have now been implemented and completed.

monitoring

An initial mitigation has been deployed, and a further mitigation step will be done tomorrow at ~8am PDT, lasting for ~ 2 hours.

identified

The issue was limited to the time period of 9:45-10:40AM PDT. We have found the causal factor and are implementing a remediation to ensure it does not recur. We will update this page when the remediation is completed, within the next 4 hours.

investigating

We are currently investigating an issue affecting some customers in US-East1 who are seeing connection timeouts while connecting to target servers.

Report: "Intermittent issues affecting customers accessing apigee Edge UI from EMEA region"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Intermittent issues affecting customers accessing Apigee Edge UI from EMEA region loading userroles and proxy trace pages. There is no impact to runtime traffic.

Report: "Errors in Apigee Analytics report jobs"

Last update
resolved

This incident has been resolved.

monitoring

Services continue to look good post restoration. Resolving this incident.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are investigating errors with Apigee Analytics report jobs.

Report: "Intermittent Datastore errors in us-central1 region"

Last update
resolved

This incident has been resolved.

monitoring

Services continue to look good post restoration. Resolving this incident.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating intermittent datastore errors affecting subset of customers in us-central1.

Report: "Bad Gateway Error from Management API"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently aware of an issue with our Management APIs resulting in "Bad Gateway Error", and are working internally to identify the cause and fix the issue.

Report: "Intermittent issues affecting customers accessing Apigee Edge UI"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

We are currently actively investigating Intermittent issues affecting customers accessing Apigee Edge UI loading pages. More updates will follow.

Report: "Apigee API Services are experiencing elevated error rates, latencies or service unavailability"

Last update
resolved

This incident has been resolved.

identified

The issue was mitigated by 2022-07-19 17:08:00 UTC. The latency and "datastore errors" seen during runtime are no longer present. Customers performing proxy deployments to europe-west2 might have experienced deployment errors before the mitigation. More updates will follow.

investigating

Apigee services are experiencing elevated error rates, latencies or service unavailability in europe-west2. Google teams are actively working to restore the services, see details at https://status.cloud.google.com/incidents/fmEL9i2fArADKawkZAa2.

Report: "Apigee Integrated Portals and Specs intermittent loading issue"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

The issue is also impacting Analytics Services. Customers may experience intermittent failures when trying to access those pages via the UI or APIs.

investigating

We are continuing to investigate this issue.

investigating

we are currently actively investigating an issue that is causing intermittent page loads on Apigee Integrated Portals, Apigee Integrated Portals Admin, and Specs on apigee.com.

Report: "Errors in Apigee UI Login via SAML"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are investigating reports of Apigee Edge UI login issues via SAML. Users may not be able to access the Apigee Edge UI if SAML is enabled on the Apigee Edge UI. This incident does not affect customers without SAML enabled. This incident does not affect runtime traffic. We will post updates here as we identify and resolve this issue.

Report: "Edge UI and Management API Errors"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix has been implemented.

investigating

After further investigation, this issue is not limited to SAML enabled orgs. All of the right resources are engaged to identify and solve this issue. We'll provide an update as soon as one becomes available. Thanks again for your patience and understanding.

investigating

The Apigee team continues to investigate this issue which seems to impact SAML enabled orgs. The troubleshooting is progressing but unfortunately we do not have an ETA as this time. We do apologize for the delay in solving this issue.

investigating

We've done a partial rollback of a change in US East as a mitigating action but hasn't solved the problem. The Apigee team continues to work this critical incident, and will provide an update shortly. This issue is impacting administrative access to the Edge UI for some users, and should not be impacting runtime traffic. We apologize for this inconvenience, and appreciate your patience!

investigating

Users of the Apigee Edge UI and Apigee Management APIs may receive error messages accessing certain product areas. Apigee teams are actively working to correct these errors.

Report: "Apigee SSO DR Testing"

Last update
resolved

The testing has ended.

identified

Apigee SSO will be conducting disaster recovery testing starting at 3:00 PM PDT. This is expected to last one hour. This affects only Apigee Edge customers using Apigee SSO for user logins into the Management UI. There is no expected impact to runtime traffic. User logins normally served by US East may see increased request latency for the duration of the test. No other impact is expected.

Report: "Login errors observed on SAML-enabled Apigee Integrated Developer portals"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

Starting at approximately 6 AM PDT, customers logging in to SAML-enabled Apigee Integrated Developer portals started encountering an elevated rate of login errors. We are actively investigating the issue and will update this page periodically.

Report: "Retroactive Incident: Datastore issue in US-East causing intermittent errors"

Last update
resolved

This is a retroactive post on an incident that occurred this morning between 05:39-06:18 PST. We detected elevated errors with our Cassandra service in the US-East region related to a single node event. During this time, customer API calls in this region would have seen intermittent API 5XX errors for approximately 5-10% of their calls for approx. 40 mins. Services in all other regions were not impacted by this incident. The node in question was removed from service to mitigate impact pending further analysis. We're investigating the root cause of this incident and will make the necessary changes to prevent a recurrence of this issue. We apologize for the inconvenience this may have caused you and your business -- and we don't take that lightly. If you have any further questions or may be seeing impacts related to this, please file a case with Support.

Report: "5xx Error Rates due to AWS"

Last update
resolved

This incident has been resolved.

monitoring

AWS has recovered from the recent unavailability of zone USE1-AZ4.

investigating

Due to an ongoing incident in AWS, Apigee Edge Cloud customers hosted in US-EAST-1 may be experiencing 5xx errors. https://status.aws.amazon.com/

Report: "Log4j Vulnerability - CVE-2021-44228"

Last update
resolved

Apigee has conducted a detailed investigation and we believe that the Edge and OPDK products are not vulnerable to this exploit.

investigating

Apigee acknowledges the reports received regarding the potential vulnerability of log4j - https://nvd.nist.gov/vuln/detail/CVE-2021-44228 . We are actively investigating the impact, and at this stage we have yet to determine the scope of impact within Apigee’s core infrastructure. This space will be updated with findings.

Report: "Increased latency and timeout in europe-west1"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Analytics Data Delays in europe-west1"

Last update
resolved

As at 2021-09-01 21:40 US/Pacific, all analytics data backlogs have been processed. The Analytics data delay has been resolved and Analytics services are fully recovered.

identified

We have deployed a new pipeline in europe-west2 region to process the analytics data backlogs. It is expected to take a couple of hours from 2021-09-01 21:40 US/Pacific to catchup.

investigating

Apigee has identified delays in data analytics that may affect customer organizations located in the europe-west1 region. Symptoms: Data analytics dashboards may not be up-to-date. You may not be able to view realtime analytics records starting from Wednesday, 2021-09-01 17:00 US/Pacific. Analytics APIs continue to work. Runtime traffic is not affected. Apigee is actively working on addressing this issue. We will post updates here as we identify and resolve this issue.

Report: "Apigee UI 50x Errors"

Last update
resolved

This incident has been resolved.

monitoring

We have taken corrective action to resolve this. Apigee teams are actively monitoring to ensure continued stability.

investigating

We are continuing to investigate this issue.

investigating

We are investigating reports of Apigee Edge UI errors. Users may see "Error: 50x" displayed in the UI, or be unable to access various portions of the UI. We will post updates here as we identify and resolve this issue.

Report: "API Monitoring Down"

Last update
resolved

This incident has been resolved.

monitoring

The API Monitoring issue has been mitigated and we are currently monitoring. We expect that it will take up to two hours for all monitoring data to backfill.

identified

Apigee Sense has recovered, but the API Monitoring issue is still being mitigated. We expect a resolution within the hour. Monitoring Data will be backfilled when service is restored.

identified

We have identified the root cause and are working on a mitigation.

investigating

API Monitoring is currently unavailable. We are investigating the cause and will update when we have more information, or within 30 minutes. Runtime traffic is unaffected, but you will not be able to access any Monitoring data at this time.

Report: "Syslog logging failures"

Last update
resolved

This incident has been resolved.

monitoring

The Message Processor fix has been rolled out, and we are monitoring the results for any errors.

identified

Apigee is aware of a bug introduced in a recent version of our Message Processors which occasionally affects Message Logging policies which use Syslog. Symptoms: Syslog servers stop receiving logs from Apigee Customer Action and Mitigation: Please file a P1 Support Case with Apigee Support. The mitigation will be a restart of your Message Processors and a rollback to a previous stable version. Fix: We have a patch which will be deployed to all pods over the coming week.

Report: "Apigee Integrated Portals loading issue"

Last update
resolved

A fix was identified and rolled out. This incident is now resolved.

investigating

We are aware of an issue that some Apigee Integrated Portals customers may experience issues with their portals loading. Apigee teams are actively working to mitigate this. We will provide updates periodically.

Report: "Apigee X instance creation fails due to GKE release"

Last update
resolved

This incident has been resolved.

investigating

we are currently investigating an issue affecting only Apigee X Paid and Eval customers onboarding is broken due to this issue. Apigee Edge cloud remains unaffected by this issue.

Report: "Datastore Errors in AWS EU-Central-1"

Last update
resolved

AWS Availability Zone euc1-az1 experienced an outage today at roughly 1:25PM Pacific. In-flight requests to our database servers in that zone were rejected as the machines were unavailable, but only in-flight requests at the instance of the outage were affected. There is no ongoing impact, as our High Availability immediately routed all incoming requests to available nodes. We will continue to monitor the situation but expect no other issues to arise.

identified

We are continuing to work on a fix for this issue.

identified

Customers may see an increase in Datastore errors for runtime calls in EU-Central-1, including access to KVM, Cache, Quota, and KMS policies. We are working to restore services.

Report: "Apigee UI Issues in US-EAST"

Last update
resolved

This incident has been resolved.

monitoring

Apigee teams have released a fix, and are continuing to monitor for any other impact.

investigating

Apigee teams have released a fix, and are continuing to monitor for any other impact.

investigating

We are actively investigating an issue with the Apigee UI in the US-EAST region. The proxy editor, trace, and offline trace features may be impacted. This is not affecting runtime traffic.

Report: "Apigee Support Portal Issues"

Last update
resolved

This incident has been resolved.

monitoring

The issue appears to be resolved, but we are continuing to monitor the situation.

identified

Apigee's Support Portal is experiencing intermittent issues due to a service provider outage. If you are unable to submit a case, please submit via this form, and we will contact you immediately.: https://docs.google.com/forms/d/19BPxvgMXNE6_0khhDA23wr1f1qY_QNlGkhtC91Lk_JA