Historical record of incidents for Sourcegraph
Report: "Sourcegraph hosted services outage"
Last updateOur upstream service provider is experiencing an outage and services are affected.
Report: "Sourcegraph Cloud infrastructure outage"
Last updateWe've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Our upstream service provider is experiencing an outage and services are affected.
Report: "Infrastructure Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A workaround has been applied and the services are now online again. We will continue to monitor service health closely
The issue has been identified as a GCP outage. We are applying a workaround to restore our services
We are continuing to investigate this issue: - Cody Gateway access is affected, but remains available for workspaces - Individual workspaces should be accessible but there may be partial outages.
We are continuing to investigate this issue: - Cody Gateway access is affected - Individual workspaces should be accessible but there may be partial outages.
We are currently investigating an incident with our cloud infrastructure which appears to have failed. Attempts are being made to re-provision the affected services.
Report: "Cody Gateway is offline"
Last updateThe issue has been resolved.
Cody Gateway is operational again, but we are continuing to monitor and investigate.
Cody Gateway is experiencing an outage, affecting all Cody usage
Report: "Sourcegraph.com sign-in outage"
Last updateSourcegraph.com sign-ins are back, and the issue has been resolved.
Sourcegraph.com sign-ins are currently experiencing an outage, affecting Sourcegraph.com public code search, Cody Free, and Cody Pro. Workspaces (Enterprise Starter) and Enterprise customers are unaffected.
Report: "Errors on Sourcegraph Cloud while using Cody"
Last updateAll Cloud instances using Cody are back to fully operational.
After pushing a change to our services, we are seeing all Cody services on Cloud instances work again. We are monitoring.
We have received some reports from Cloud customers running into issues using Cody with error lookup cody-gateway-msp.sourcegraph.com no such host We are taking remedial action
Report: "Sourcegraph.com Maintenance"
Last updateScheduled maintenance has been performed, services should be operating normally.
We will be undergoing scheduled maintenance on our infrastructure during this time, and services may be temporarily unavailable.
Report: "Sourcegraph.com data migration"
Last updateThis incident has been resolved.
The migration is progressing well, we're still rebuilding indexes and restoring access to all repositories, but most functionality is restored.
We are migrating storage backends for Sourcegraph.coms OSS code search, while this is ongoing expect incomplete search results and some slowness. We will update along the way.
Report: "Cody Free / Cody Pro issues"
Last updateIssue resolved.
We've identified an issue affecting Cody Free / Cody Pro users, and are working to resolve it.
Report: "Cody Authentication Issues"
Last updateThis incident has been resolved.
A fix has been applied and we are monitoring the situation.
We're investigating an authentication issue affecting Cody services. Pro and paid users are currently being incorrectly identified as free users, resulting in access limitations and error messages (HTTP 429/403). This issue is impacting both the web interface and IDE integrations. Our engineering team is actively working to resolve this problem and restore proper account recognition. We apologize for the inconvenience and will provide updates as we have more information.
Report: "Sourcegraph.com service degradation"
Last updateAfter taking corrective action we are no longer seeing issues signing into sourcegraph.com nor any slowness when using Cody. All services fully operational.
We have received reports of slowness while navigating sourcegraph.com and Cody Free/Pro users. We are investigating
Report: "Sourcegraph.com service degradation"
Last updateA Sourcegraph.com outage was affecting our public code search at sourcegraph.com/search, Cody Free, and Cody Pro services for about 45 minutes. Workspaces (Enterprise Starter) and Enterprise customers are not affected.
Report: "Sourcegraph.com service degradation"
Last updateThis incident has been resolved.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Sourcegraph.com service degradation"
Last updateThis incident has been resolved.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Sourcegraph.com service degradation"
Last updateThis incident has been resolved.
Sourcegraph.com issues is affecting our public code search at sourcegraph.com/search, Cody Free, and Cody Pro services.
Report: "Sourcegraph.com service degradation"
Last updateA Sourcegraph.com outage was affecting our public code search at sourcegraph.com/search, Cody Free, and Cody Pro services for about 35 minutes. Workspaces (Enterprise Starter) and Enterprise customers are not affected.
Report: "Sourcegraph.com service degradation"
Last updateA Sourcegraph.com outage was affecting our public code search at sourcegraph.com/search, Cody Free, and Cody Pro services for about 43 minutes. Workspaces (Enterprise Starter) and Enterprise customers are not affected.
Report: "Sourcegraph.com service degradation"
Last updateA Sourcegraph.com outage was affecting our public code search at sourcegraph.com/search, Cody Free, and Cody Pro services for about 23 minutes. Workspaces (Enterprise Starter) and Enterprise customers are not affected.
Report: "Sourcegraph.com service degradation"
Last updateAll services are back online.
All affected services should be back online, but we are still investigating what happened.
Sourcegraph.com issues is affecting our public code search at sourcegraph.com/search, Cody Free, and Cody Pro services. Workspaces (Enterprise Starter) and Enterprise customers are not affected.
Report: "Sourcegraph.com Performance Degradation Affecting Cody Services"
Last updateWe think the impact from the sourcegraph.com issue is over but we are still monitoring.
While the immediate service disruption has been resolved, we are actively monitoring the situation. All Cody services and sourcegraph.com functionality have returned to normal operation. Our engineering team continues to investigate the root cause.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Sourcegraph.com is experiencing slow page loads"
Last updateThis incident has been resolved, Sourcegraph.com is fully operational again.
A fix has been implemented and we are monitoring the results.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Sourcegraph Cloud outage"
Last updateCustomer-facing impact has been addressed.
We've deployed the fix, and we will continue to monitor.
Our fix has been applied to all Sourcegraph Cloud instances, and all Cloud instances should be healthy again. We are now performing some final verification.
We have a confirmed resolution and have already fixed some Sourcegraph Cloud instances, and are in the process of applying the fix across all Cloud instances.
Some Sourcegraph Cloud instances are experiencing an outage. We're currently trying to resolve the issue, and will provide an update soon.
Report: "Sourcegraph services scheduled maintainence"
Last updateThe scheduled maintenance is complete, and all systems should be operational again.
We are performing a scheduled upgrade for online Sourcegraph services - during this time, users may encounter issues with Cody Pro, signing in to Sourcegraph.com, and Cody Analytics access.
Report: "Cody Gateway recovered"
Last updateThis incident has been resolved.
The issue has mostly recovered now but we will wait a while for confirmation before closing this
Cody Gateway is down currently and dotcom user's can use Cody
Report: "Sourcegraph.com inaccessible"
Last updateSourcegraph.com public code search and Cody Pro were affected by downtime that lasted about 1.5 hours.
Report: "Sourcegraph Cloud Executors Not Allowing Batch Changes"
Last updateCloud customers have been upgraded to the latest version which resolves this issue. Customers with persisting issues creating batch changes using executors should reach out to the Sourcegraph support team.
Cloud customers that have reported the issue have been patched and the patch has been applied to other cloud customers as well. If you are reading this and still experiencing the issue please reach out to support@sourcegraph.com immediately.
If you are reading this message currently and have already reached out to Sourcegraph Support about this issue please be advised that a fix has been deployed to your instance and Batch Changes should now be working as intended. If you are reading this and have not reached out to Sourcegraph Support about this issue a fix will be deployed to your instance by Tuesday EOD.
A fix has been created and will be rolled out over the weekend and should be rolled out to affected customers by Monday EOD. Please follow closely for updates.
We've found the root cause of the issue and are currently creating a fix for it. Thank you for your patience and please stay tuned.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Sourcegraph.com maintenance event"
Last updateWe have completed our maintenance event - Cody Pro and Sourcegraph.com public code search should now be functioning as usual now.
We are performing a schedule upgrade for Sourcegraph.com - during this time, users may encounter issues with Cody Pro and public code search on Sourcegraph.com.
Report: "Sourcegraph.com and Cody Free/Pro experiencing high latency"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Cody Gateway unexpected status code 500 error"
Last updateAfter the taking remedial action Cody Gateway is responding as expected. All services are fully operational.
We have identified and fixed an issue that affects Cody Gateway with error "unexpected status code 500". We are monitoring.
Report: "GPT4o and Mistral 8x7b not working for chat"
Last updateWe think the impact from the issue is over.
GPT-4o and Mixtral 8x7B are working again. We will be monitoring before we mark this resolved.
Problem: Asking Cody a question using GPT4o or Mistral 8x7b models fails due to a regression caused by a recent update. Impact: This failure results in errors within VS Code and Cody Web platforms. Fix: An update is currently being rolled out.
When Asking Cody a question using GPT4o or Mistral 8x7b models fails with the following errors: VS Code: "SourcegraphNodeCompletionsClient: isError(parseEvents(bufferText)) Error: invalid completion event" Cody Web: "Request Failed: Cannot read properties of undefined (reading 'length')" Please use another model for the meantime.
Report: "Cody Gateway access is not reachable"
Last updateWe've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Autocomplete is now working. Still closely monitoring.
VS Code: chat works, but not autocomplete. Still working on the issue.
We have discovered ❌ VS Code doesn't work ✅ Cody Web works ✅ JetBrains works
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Request Failed: Sourcegraph Cody Gateway: unexpected status code 4xx"
Last updateThis is a duplicate issue, closing.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Some open source repos from GitHub are missing in Code Search results"
Last updateThis incident has been resolved. Happy searching.
731k repositories have been indexed for search.
Current status: all repositories (989,820) have been synced/cloned. 677k have been indexed and should be finished sometime tomorrow.
Current status: 990k repositories synced/cloned from GitHub. 644k have been indexed. Getting closer to finished.
Current status: 870k repositories synced/cloned from GitHub. 611k have been indexed.
Current status: 700k repositories synced/cloned from GitHub. 557k have been indexed. Updating impact from Major to Minor.
Current status: 557k repositories synced/cloned from GitHub. 528k have been indexed.
Current status: still at 527k repositories synced/cloned from GitHub. 489k have been indexed (+126k).
Current status: 527k repositories synced/cloned from GitHub. 363k have been indexed.
Current status: 422k repositories synced from GitHub. 297k have been indexed.
We are currently resynchronizing the GitHub repositories. This process will take some time to complete, based on the current rate of syncing, it’ll take ~48 hours to finish syncing all the deleted repos. Time to finish indexing is unknown at this time.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
We are currently resynchronizing the GitHub repositories. This process will take some time to complete, based on the current rate of syncing, it’ll take ~48 hours to finish syncing all the deleted repos. Time to finish indexing is unknown at this time.
We are currently resynchronizing the GitHub repositories. This process will take some time to complete, and we will provide updates on our progress.
We're currently investigating the issue, and will provide an update soon.
Report: "sourcegraph.com is degraded"
Last updateWe think the impact from the issue is over.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Email delivery failures"
Last updateThis incident has been resolved.
Our email provider has implemented a fix and we're seeing error rates go down.
We've spotted that something has gone wrong with email deliveries from Sourcegraph.com. This affects emails notifications (e.g. for access token creation) and password resets. We're currently investigating the issue, and will provide an update soon.
Report: "Users getting logged out and sessions timing out due to issues with redis"
Last updateWe think the impact from the issue is over.
All services are up and healthy
All services are up and healthy
Users are able to log in again and not be logged out randomly
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Sign in doesn't work."
Last updateWe've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
The signin page throws an error when used. A previous change was identified as the culprit and a fix is being deployed.
Report: "Searches are taking a long time to complete"
Last updateThe issue has been and services are operational.
We've applied some improvements and will continue to monitor performance
Searches are taking a long time. In an effort to improve performance, we’re applying some mitigations. While these changes are being rolled there might be a brief period where search is unavailable.
Report: "Syntax highlighting not working on Sourcegraph.com"
Last updateWe are not longer seeing elevated syntax highlighting errors.
We've noticed that syntax highlighting is timing out for various languages on Sourcegraph.com. We're investigating the issue.
Report: "Cloud instances may intermittently be down due to upstream issue"
Last updateAfter taking remedial action and monitoring we have not seen any instance in Sourcegraph Cloud have the errors mentioned. All services are fully operational.
Some instances may see 404 errors on their Sourcegraph Cloud instance due to a GCP outage. We are monitoring and taking remedial action as needed. See here for more: https://status.cloud.google.com/incidents/skbvoobU4btcxyAthEmu#2c2sBHWU84yPDJ8y1ar4
Report: "docs.sourcegraph.com is unavailable"
Last updateThis incident has been resolved.
docs.sourcegraph.com is currently unavailable. We're currently investigating the issue, and will provide an update soon.
Report: "Cody Gateway Embeddings generation degraded performance"
Last updateError rates have gone back to normal over the weekend.
We're seeing increased usage of embeddings generation causing some requests to be rejected with high load errors. We're working on increasing capacity.
Report: "sourcegraph.com is taking long to respond"
Last updateNo issues reported in a while. Issue is therefore resolved.
We've mitigated the problem for now. Monitoring site performance
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "sourcegraph.com is responding"
Last updateSite is responding and incident mitigated
We've spotted that something has gone wrong. We've mitigated the issue and monitoring performance of the site
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "Cody Gateway Embeddings generation degraded"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are seeing some requests for embeddings fail or take a long time.
Report: "Scheduled Maintenance (complete)"
Last updateSourcegraph.com has successfully completed maintenance. Search and repository operations are restored.
Sourcegraph.com is undergoing scheduled maintainence - availability of repositories for search may be affected during this time.
Report: "Sourcegraph.com downtime"
Last updateAt 4:45PM UTC, Sourcegraph.com went offline due to a configuration issue. The issue was identified and fixed around 5:10PM UTC, with most Sourcegraph.com functionality restored by 5:25PM UTC. This may have caused brief issues with extensions on on-premises Sourcegraph instances.
Report: "All Python packages unavailable on Sourcegraph.com"
Last updateAt 2am PT all Python packages became unavailable on Sourcegraph.com. This issue was resolved at 7:35am PT and all Python packages are now available again.
Report: "Networking partner outage"
Last updateThis is to confirm that this outage has been resolved and all impacted Sourcegraph services are now up and running for all regions. We appreciate your patience and understanding.
It has come to our attention that some of our Sourcegraph.com and managed instance customers may be experiencing connectivity issues due an ongoing outage with our networking partner. We are beginning to see signs of of recovery however and will share further updates on the situation as they emerge. We apologize for any inconveniences experienced.
Report: "Issue with GitLab Permissions"
Last updateThis issue has now been resolved. A patch has been released which should address this issue for on-prem customers and fixes have also been deployed to sourcegraph.com.
Our Engineering team is currently working on a fix to address this issue. We will communicate further updates as soon it is applied. Thanks for your patience.
We have identified an issue that can cause you not to get search results that you would expect or see repositories that have been added to Sourcegraph. You are impacted if you are a Sourcegraph.com user using GitLab.com or an on-prem customer using GitLab version 15.0 or later, or older Gitlab versions that have OAuth access token expiration enabled. Otherwise you are not impacted. If you are impacted, you can remediate this by removing and re-adding GitLab personal code host connection every two hours (if you are a Sourcegraph.com user) or signing out and in every two hours with GitLab OAuth (if you are an on-prem user). We are currently working to resolve this issue and regret the inconvenience. We will provide an update when functionality is restored.
Report: "Unable to add repositories to Cloud"
Last updateThis is now fully resolved! Again, we are sorry for the inconvenience and thank you for your patience. If you have any questions, email support@sourcegraph.com.
We have fixed the issue and the situation is stabilizing. Repositories will sync, but may feel a little delayed for a little longer. We are monitoring and will update again when the issue is fully resolved. Thank you for your patience!
We are aware that you cannot add repositories on Cloud and we are on it! We are sorry for the inconvenience and will update when functionality is restored. If you have any questions, please email support@sourcegraph.com.
Report: "Search is slow on Sourcegraph.com"
Last updateThis issue is now resolved. Again, we are sorry for any inconvenience this caused. If you have any questions, please email support@sourcegraph.com.
We have identified the issue and it looks to be self-correcting -- if you noticed search improving over the last ~45 minutes, that is correct. We are monitoring to make sure everything is fully restored and will update again when we consider this resolved. Again, if you have any questions in the meantime, please email support@sourcegraph.com.
We are aware that search is intermittently slow on Sourcegraph.com right now. We are on it! We will let you know when this is resolved or if we have any updates to share in the meantime. We are sorry for the inconvenience. If you have any questions, please email support@sourcegraph.com.