Historical record of incidents for Apollo Graph, Inc.
Report: "Elevated Error Rates"
Last updateWe are currently investigating this issue.
Report: "Elevated Error Rates"
Last updateWe are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Elevated Error Rates"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are seeing services recover.
We've identified the source of the issue and are working to implement a mitigation.
We are continuing to investigate this issue.
We're experiencing an elevated level of API errors and are currently looking into the issue.
We are investigating elevated error rates on our APIs
Report: "Elevated Error Rates"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are seeing services recover.
We've identified the source of the issue and are working to implement a mitigation.
We are continuing to investigate this issue.
We're experiencing an elevated level of API errors and are currently looking into the issue.
We are investigating elevated error rates on our APIs
Report: "Notice: GraphOS Scheduled Maintenance, 2025/05/19"
Last updateDuring part of this period, Routers and Gateways which use IP-based allow lists will be unable to connect to Uplink. Contact Apollo Technical Support to help mitigate this.
As part of scheduled maintenance on May 19, 2025, between 17:00 UTC and 21:00 UTC to update IP addresses behind our APIs, we will be temporarily changing the IP address associated with Uplink: uplink.api.apollographql.com. The current Egress IP Address is 34.117.186.194Customers who use IP Allow Lists to connect to Apollo, and do not update their lists to include the new IP, will be temporarily unable to access Uplink while Apollo conducts this maintenance. If you are a customer using IP Allow Lists, and have the IP Allow list updated with the new IP, there will be no downtime to any Apollo services.Required Actions if using IP Allow ListsThe temporary IP address may be obtained by contacting Apollo Technical Support via our Help Center, Studio, or by emailing support@apollographql.com.Add the temporary IP to your allow list for uplink.api.apollographql.com.Keep the current IP (34.117.186.194) in your allow list, as service will return to this IP after the maintenance period.We appreciate your understanding and are committed to ensuring a smooth transition with minimal disruption. If you have any questions or concerns, please reach out to Apollo Technical Support.
During part of this period, Routers and Gateways which use IP-based allow lists will be unable to connect to Uplink. Contact Apollo Technical Support to help mitigate this.
We apologize for the previous notification sent out in the morning of April 15, 2025, which had the incorrect dates related to this scheduled maintenance.As part of scheduled maintenance on May 19, 2025, between 17:00 UTC and 21:00 UTC to update IP addresses behind our APIs, we will be temporarily changing the IP address associated with Uplink: uplink.api.apollographql.com. The current Egress IP Address is 34.117.186.194Customers who use IP Allow Lists to connect to Apollo, and do not update their lists to include the new IP, will be temporarily unable to access Uplink while Apollo conducts this maintenance. If you are a customer using IP Allow Lists, and have the IP Allow list updated with the new IP, there will be no downtime to any Apollo services.Required Actions if using IP Allow ListsThe temporary IP address may be obtained by contacting Apollo Technical Support via our Help Center, Studio, or by emailing support@apollographql.com.Add the temporary IP to your allow list for uplink.api.apollographql.com.Keep the current IP (34.117.186.194) in your allow list, as service will return to this IP after the maintenance period.We appreciate your understanding and are committed to ensuring a smooth transition with minimal disruption. If you have any questions or concerns, please reach out to Apollo Technical Support.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Traces and signatures missing from Insights Studio"
Last updateWe will follow up with affected customers.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified a fix.
We are still investigating this issue.
We are investigating the issue.
Report: "Traces and signatures missing from Insights Studio"
Last updateWe are investigating the issue.
Report: "Problem loading Proposals"
Last updateIncident has been resolved.
Fix rolled out, now monitoring.
We have identified a potential fix, and will be rolling it out shortly.
We have implemented a partial fix, Proposals is now partially degraded
April 5 20:35 UTC - we have identified a problem with the Proposals and any Schema Checks that have Proposal Checks enabled.
Report: "Problem loading Proposals"
Last updateApril 5 20:35 UTC - we have identified a problem with the Proposals and any Schema Checks that have Proposal Checks enabled.
Report: "High Studio UI latency"
Last updateThe incident has been resolved
We implemented a fix and are seeing improved load times for proposals and the rest of the studio UI.
We have identified the issue in schema proposals. We are working on a fix to improve page load times.
Page load times are high for studio UI.
Report: "High Studio UI latency"
Last updateThe incident has been resolved
We implemented a fix and are seeing improved load times for proposals and the rest of the studio UI.
We have identified the issue in schema proposals. We are working on a fix to improve page load times.
Page load times are high for studio UI.
Report: "Increased error rates across Apollo services"
Last updateThe incident has been resolved. We are seeing error rates and latency for our GraphQL API have recovered back to normal.
We have rolled out a fix and the error rate and latency are returning to normal. We are continuing to monitor the outcome.
We have identified the issue and are working on fixing it.
We are continuing to investigate the issue. Uplink and metric ingestion are also currently unaffected.
We are seeing increased error rate and latency when talking to Apollo's GraphQL API. Uplink and metric ingestion are currently unaffected.
Report: "Increased error rates across Apollo services"
Last updateThe incident has been resolved. We are seeing error rates and latency for our GraphQL API have recovered back to normal.
We have rolled out a fix and the error rate and latency are returning to normal. We are continuing to monitor the outcome.
We have identified the issue and are working on fixing it.
We are continuing to investigate the issue. Uplink and metric ingestion are also currently unaffected.
We are seeing increased error rate and latency when talking to Apollo's GraphQL API. Uplink and metric ingestion are currently unaffected.
Report: "Embedded Sandbox loading issue"
Last updateBetween 8:46am ET 3/5/2025 and 8:16am ET 3/7/2025 the Embedded Sandbox was experiencing issues loading. This has now been fixed.
Report: "Issues with schema publishing"
Last updateBetween 23:10 UTC and 23:44 UTC, all schema publishes for monographs failed. Subgraph publishes for graphs using Apollo Federation were unaffected.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are investigating an issue impacting schema publishing.
Report: "Studio not loading for graph pages"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Report: "GraphOS studio page load failures"
Last updateThe following pages failed to load for 1 hour and 15 minutes - Check details for proposals, linter and custom checks - Proposal Changes and Editor linter drawer
Report: "Schema Check and Publish Degradation"
Last updateLaunches and checks are back to normal and operating as expected
We have implemented a fix and are monitoring. The situation is resolving and we are monitoring it.
We are experiencing high latency running a build. We have identified the issue and are working on a fix. This will manifest as slow or failing publishes and checks.
Report: "Schema Check and Publish Degredation"
Last updateLaunches and checks are back to normal and operating as expected
We have implemented a fix and are monitoring. The situation is resolving and we are monitoring it.
We are experiencing high latency running a build. We have identified the issue and are working on a fix. This will manifest as slow or failing publishes and checks.
Report: "Launches, Uplink, Checks, Degraded"
Last updateThis incident has been resolved.
Upstream service has recovered, monitoring the results.
Outage lasted about 12 minutes and the upstream service is recovering, monitoring.
Report: "Schema publishing and checks degradation"
Last updateLaunches and checks are back to normal and operating as expected
The issue has been identified and a solution has been deployed. The situation is resolving and we are monitoring it
We are continuing to investigate this issue.
We are noticing some degradations related to schema publish and checks. We are investigating the issue.
Report: "Increased error rates across Apollo services"
Last updateThis incident has been resolved. Please note that reading data from Uplink was operational throughout this entire incident. We had noted Uplink as affected as we were seeing other non user-facing issues with the service.
The fix has been deployed and we are continuing to monitor the resolution.
We are starting to see some recovery but are still working to fully deploy the fix.
We are applying a fix for this issue now. Uplink has remained operational throughout this incident.
We are continuing to work on a fix for this issue.
The issue has been identified and we are working on a fix.
We are continuing to investigate this issue.
We are investigating elevated error rates across our services. We will provide more information soon.
Report: "Apollo GraphQL website outage in some locations"
Last updateOur hosting provider has resolved the issue.
Our hosting platform is continuing to investigate.
Our hosting platform is continuing to investigate.
The fault has been identified by our hosting platform and we are waiting for a resolution from them.
We have identified that the issue is with our website hosting platform and we have created a support request to them.
We are continuing to investigate this issue.
We are continuing to investigate the issue. It appears that South America and Oceania are affected, but no other regions.
The apollographql.com website is not accessible in some locations including Australia and New Zealand. Others such as United States and United Kingdom are not affected. Studio is not affected in any location. We are investigating but it seems to be an issue with our downstream provider.
Report: "Increased error rate in schema checks"
Last updateThe error rate in Schema Checks has recovered.
We are seeing a slightly increased error rate in Schema Checks. We are investigating the issue.
Report: "Increased check latency"
Last updateWe have identified the issue and resolved it.
We are continuing to work on a fix for this issue.
We have identified the source of the scaling issues and are working on a fix. We are also seeing increased latency for any pages and operations related to schema proposals.
We are seeing increased latency for schema checks in studio.
Report: "High Latency in Contracts and Downstream Checks"
Last updateThis incident has been resolved.
We have located the source of the issue and are seeing latency begin to improve. Will continue to monitor and resolve the status once latency returns to expected performance.
We're continuing to investigate. Checks and publishes are continuing to complete successfully, but with high latency. Thanks for your patience!
We are continuing to investigate this issue.
We are currently investigating high latency and downgraded performance in checks and schema publishes. This is not currently expected to prevent check completion, but does impact the time to completion.
Report: "Schema composition degraded"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating some issues with the composition of Supergraph schema
Report: "Traces delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are actively investigating an issue that involves a delay in availability of traces from live traffic.
Report: "Checks and Publishes broken for federation versions < 2.4"
Last updateWe're confident this issue has been resolved and it has also been resolved on GCP's side. Thank you again for your patience. For more information, please check the link below: https://status.cloud.google.com/incidents/1yphfNLPHEnwJcWqwxbu
Our upstream provider has applied a mitigation and it appears to have resolved the degraded performance of the features listed below. We are monitoring to ensure there are no regressions.
An incident has been created and you can follow the updates here: https://status.cloud.google.com/incidents/1yphfNLPHEnwJcWqwxbu We will also continue to monitor and update accordingly.
Our upstream provider is mitigating the issue, though they do not have an estimate on resolution time at this moment. We will continue to monitor the situation and update accordingly. Thank you again for your patience.
We are also noticing elevated error rates for GCP Uplink which we believe is also a cause of the ongoing issue with our upstream provider. We are continuing to watch for updates. Users should still be falling back to AWS Uplink if configured to do so.
We have identified the issue to be with one of our upstream providers. They are working to identify and fix the issue, so we will continue to update this page as we get more information.
We are continuing to look into this incident and are working to resolve it. Thank you for your ongoing patience.
We have identified an issue and are working to resolve it. Users may also be seeing delays in schema notifications. Thank you for your patience.
We are currently investigating issues running builds for federation versions less than 2.4.
Report: "Elevated error rates across multiple services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Router and Rover installation affected by GitHub outage"
Last updateThe upstream incident has been resolved.
The upstream incident appears to be at least partially resolved; Apollo Router and Apollo Rover installation scripts are now functional again.
Installation of Apollo Router and Apollo Rover via `router.apollo.dev` and `rover.apollo.dev` currently fails due to an incident with an upstream provider. We expect this will function again when the upstream incident is resolved (https://www.githubstatus.com/incidents/kz4khcgdsfdv).
Report: "Increased error rates"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an increase error rate affecting a number of our components. We will post another update shortly.
Report: "Studio UI not loading some pages"
Last updateThis incident has been resolved
We have rolled out a fix and are seeing all pages load again. Will continue monitoring before resolving the incident.
We have identified an issue impacting the studio.apollographql.com UI and are working on a fix. Some pages, such as Explorer are not loading.
Report: "Unable to create new Cloud Routers (Dedicated / Serverless)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Currently, we are unable to create new cloud routers for our dedicated and serverless platforms. We are investigating the root cause.
Report: "Metrics ingestion issues"
Last updateThis incident has been resolved
We have implemented a fix and are seeing success rates return to normal. We will continue to monitor the service while we confirm recovery.
We continue to investigate this issue
We are investigating an issue with metrics ingestion and you may see elevated failure rates when submitting usage reports. We are actively working on resolving the issue.
Report: "Datadog Forwarding: Delayed metrics"
Last updateThis incident has been resolved.
We have seen consistent success here over the last 15+ minutes. Given the duration of this incident, we will be observing till the top of the hour to close this out. We appreciate your patience in this incident.
We are seeing customer data flowing normally. There will be missing data for our subset of customers who were previously impacted, but no data loss for any other customer. We will be observing this for another 15 minutes and if things remain stable we will move to monitoring.
We are continuing to investigate this issue. We are still seeing partial degradation for a subset of our customers. We will report back as new information arises.
We are continuing to investigate this issue. We are still seeing partial degradation for a subset of our customers. We will report back as new information arises.
We are seeing signs that we are still having issues with DataDog Forwarding for a subset of our users. We are continuing to investigate.
We are seeing signs that the issue is resolved but will continue to monitor.
We are continuing to investigate this issue.
We are continuing to investigate this issue. Note that some customers will be seeing a delay in metrics forwarding to DataDog, but most customers will be unaffected.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are experiencing issues <a href="https://www.apollographql.com/docs/graphos/metrics/datadog-integration/">Forwarding metrics to Datadog</a>. Enterprise customers may see delayed Datadog metrics at this time. Our team is actively investigating the issue and will provide updates frequently until resolution. Thank you for your patience.
Report: "studio.apollographql.com inaccessible"
Last updatestudio.apollographql.com was inaccessible for returning browser users for 20 minutes
Report: "Partial outage of GraphQL API"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
There is a partial outage of the GraphQL API (graphql.api.apollographql.com) responsible for Studio UI and custom legacy integrations. This outage does not affect the Platform API used by Rover, the Uplink API used by Router, or the Usage Reporting API. We have identified the root cause and are working on remediation.
Report: "Partial outage of Schema Reporting"
Last updateSchema Reporting had a partial outage from 2024-05-31 04:04 to 18:06 UTC. This only affected the "Schema Reporting" feature of Apollo Server (https://www.apollographql.com/docs/apollo-server/api/plugin/schema-reporting/), which can only be used by graphs that do not use Apollo Federation. Other methods of schema publishing (including all methods of schema publishing for federated graphs and all graphs that use Apollo Router or Apollo Gateway) were unaffected. During this period, a subset of graphs did not have their active schemas updated in GraphOS. A secondary issue (a misconfiguration of our monitoring system specific to Schema Reporting) caused us to believe that this issue resolved immediately after it started to occur, leading to the 14-hour length of this incident. This misconfiguration has also been resolved.
Report: "Email notifications degraded"
Last updateThis incident has now been resolved.
The upstream provider has implemented a fix, and we are monitoring the results.
We are currently experiencing issues with email notifications due to an upstream provider. We are monitoring the situation closely, and will provide updates as soon as the upstream provider is stable.
Report: "Email notifications degraded"
Last updateThe upstream provider has implemented a fix, and this incident has been resolved.
We are continuing to work on a fix for this issue.
The upstream provider has identified the problem and is working on a fix for the issue.
We are currently experiencing issues with email notifications due to an upstream provider. We are monitoring the situation closely, and will provide updates as soon as the upstream provider is stable.
Report: "Missing field usage insights data"
Last updateBetween 2024-05-04 18:00 UTC to 2024-05-04 22:00 UTC, one of our insights data sources containing field usage stats was not ingesting data. This affects all graphs for that time window, and a small number of graphs for up to 1 day before the start time. Within these times, you may notice missing field usage data on the Insights page. We are working on a way to limit the impact to just affecting client attribution of field usage and we will update this incident when we have updates.
Report: "Failures in GraphQL API"
Last updateThis incident has been resolved. We appreciate your patience and we apologize for the inconvenience this may have caused.
We are beginning to see recovery in our services. We will continue to monitor the situation and update the incident shortly.
We have identified the issue and have applied a fix.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating failures in GraphQL API. You may see some errors throughout the platform.
Report: "Metrics ingestion delayed"
Last updateThis incident has been resolved.
Metric ingestion is recovering and we are monitoring the status.
The issue has been identified and a fix is being implemented.
We have identified the issue and are working on a fix.
We are investigating an issue with our metrics database and metrics ingestion is delayed.
Report: "Some builds are failing to complete"
Last updateThis incident has been resolved.
We have deployed a fix and are monitoring the results.
We have identified the issue and are working on a resolution.
We are seeing some occasional build failures but most builds are progressing as normal.
We are investigating some issues related to builds not progressing or failing to complete.
Report: "Metrics Degraded"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are actively investigating issues with our metrics ingestion pipeline. This issue will affect loading metrics in Studio.
Report: "Elevated error rates for schema publishing"
Last updateWe have seen an increase in failure rate for schema publishes / launches. We believe we have diagnosed the cause and a fix has been rolled out. We are continuing to monitor the situation.
Report: "Metrics degraded"
Last updateThis incident has been resolved.
Loading times for insights and metrics are still elevated. We are going to take additional actions to improve performance over the next few hours.
Our systems have returned to a normal state. We'll continue to monitor and share updates within the next hour.
We are continuing to investigate this issue.
We are currently investigating an issue with loading metrics in Studio.
Report: "Metrics degraded"
Last updateThis incident has been resolved.
We are making progress on resolving the latency in field metrics and all other issues have been resolved.
We are continuing to work through the latency in field metrics and we will keep monitoring the recovery.
We are still experiencing latency in field metrics but other metric latencies have recovered. We expect complete recovery to take several more hours.
We have identified the issue and are beginning to recover. No data has been lost but we expect complete recovery to take several more hours.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating an issue with loading metrics in Studio.
Report: "Publish and Checks"
Last updateWe have completed running through the backlog of outstanding check and publishes requests. All systems have returned to operational.
We have begun seeing recovery to most systems. We are beginning to make progress against our backlog of publish and check requests. Will provide an ETA for full recovery when available.
Report: "Studio API Availability Impaired"
Last updateThis incident has been resolved
We have completed running through the backlog of outstanding check requests and all systems have returned to operational. We will continue to monitor for 30 minutes and resolve the incident.
We are continuing to work through our checks backlog. Presently, based on our burn rate through our queue, we are estimating less than 15 minutes till we are entirely caught back up. We will update here again in 15 minutes, or once our queue is through.
Publishes have recovered, still working through the backlog of check requests
We're making progress on our backlog of builds. ETA 15-30 minutes
Fix has been shipped. We are beginning to make progress against our backlog of publish requests. Will provide an ETA for full recovery when available.
We are still in the process of deploying the code fixes. Publishes and Checks continue to be degraded
We have identified another scaling issue in our build systems, and are deploying a fix now. Publishes and Checks continue to be degraded
Our systems are working through a backlog of build requests. Publishing and Checks continue to be degraded
We continue to monitor recovery to most systems. Publishing and checks are still degraded.
We have begun seeing recovery to most systems. Publishing and checks are still degraded
We believe to have root caused our availability issues and are working on a fix
We are currently investigating this issue.
Report: "Cloud Router Provisioning Interruption"
Last updateThis incident has been resolved by our upstream provider. Thank you for your patience.
The upstream routing provider continues to work on a fix for this issue.
The upstream routing provider continues to work on a fix for this issue.
The issue has been identified and a fix is being implemented by our upstream routing provider.
We are experiencing issues with the creation of GraphOS Cloud Routers due to an upstream routing provider interruption. Provisioned Cloud Routers are currently stable and serving traffic as usual. Endpoint availability is also not affected by this issue. Our team is closely monitoring the situation, and we will provide updates as soon as the service is stable. Thank you for your patience.
Report: "Our API is returning intermittent errors"
Last updateThis incident has been resolved.
Our monitors indicate that our fix has allowed the API to recover. We believe this mitigates the issue and we will follow-up internally. Thank you for your patience.
We have committed and reviewed a fix and are monitoring the rollout.
We believe we understand the problem and are applying a fix.
We are currently investigating an issue which is resulting in intermittent errors in both Studio and for those consuming from our GraphQL API.
Report: "Cloud Router Provisioning Interruption"
Last updateThis incident was resolved 3 hours ago at 14:42 UTC, as described in the previous update message. At that time, the "Cloud Routing" component on this status page was marked as "Operational" (rather than "Degraded performance"), however the overall "Incident Status" was inadvertently left as "Investigating" rather than "Resolved". This corrects that. Thank you for your understanding!
This incident has been resolved and operations have returned to normal. Thank you for your patience.
We are experiencing issues with the creation and updates of Cloud Routers due to an upstream hosting provider interruption. Provisioned Cloud Routers are currently stable and serving traffic as usual. Endpoint availability is also not affected by this issue. Our team is closely monitoring the situation, and we will provide updates as soon as the service is stable. Thank you for your patience.
Report: "Lint Checks failing to report Composition Hints"
Last updateRollback has successfully fixed the issue. We will be conducting a postmortem this week. We appreciate your patience.
We deployed code that caused Composition Hints to not be reported in Lint Check tasks. Composition hints and errors were accurately reported for publishes and Build tasks in checks. Users with checks configured to fail on Lint Composition Hints will have seen checks pass incorrectly.
We deployed code that caused Composition Hints to not be reported in Lint Check tasks. Composition hints and errors were accurately reported for publishes and Build tasks in checks. Users with checks configured to fail on Lint Composition Hints will have seen checks pass incorrectly.
Report: "Degraded Schema Checks"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Schema checks can successfully be triggered, but they are very slow to resolve. We are currently investigating this issue.
Report: "Cloud Router Provisioning Interruption"
Last updateThis incident has been resolved and operations have returned to normal.
Our upstream provider has implemented a fix. Creating and updating Cloud Routers should be functioning normally, but we are actively monitoring the situation to ensure safe and reliable use of our services.
We are experiencing issues with the creation of Cloud Routers due to an upstream hosting provider interruption. Provisioned Cloud Routers are currently stable and serving traffic as usual. Endpoint availability is also not affected by this issue. Our team is closely monitoring the situation, and we will provide updates as soon as the service is stable. Thank you for your patience.
Report: "Cloud Router Provisioning Interruption"
Last updateThis incident has been resolved and operations have returned to normal.
We are experiencing issues with the creation and updates of Cloud Routers due to an upstream hosting provider interruption. Provisioned Cloud Routers are currently stable and serving traffic as usual. Endpoint availability is also not affected by this issue. Our team is closely monitoring the situation, and we will provide updates as soon as the service is stable. Thank you for your patience.