Historical record of incidents for Semgrep
Report: "Occasional latency issues in Semgrep AppSec Platform"
Last updateWe're investigating an issue with occasional latency increases in Semgrep AppSec Platform. Platform responsiveness is generally normal but there may be brief periods where the platform loads slowly or some requests time out.
Report: "High latency causing API timeouts and login delays"
Last updateWe're currently investigating increased latency on the Semgrep platform.
Report: "Some scheduled managed scans are not starting"
Last updateWe're currently investigating an issue that has been preventing some managed scans from starting on schedule.
Report: "Project syncs delayed in Semgrep AppSec Platform"
Last updateWe have implemented a fix and the system is in a healthy state.
Since approximately 18:00 UTC, project sync in the Semgrep AppSec Platform has been delayed or not completing. We have identified the cause of the issue and are working to resolve it.
Report: "Managed Scans delayed for users with network access controls"
Last updateManaged Scans are now healthy and running normally.
The delayed scans have run successfully and Managed Scans are running normally. We are continuing to monitor to ensure scan health.
For users with network access controls to their SCM, Managed Scans were delayed starting at 18:20 UTC. We've identified the issue and scans are resuming.
Report: "Managed scans failing to start"
Last updateManaged scans were not successfully starting for about fifteen minutes. The problem has been identified and rolled back.
Report: "Delays in Managed Scans, PR/MR comments"
Last updateThis issue has been resolved, and Managed Scans are being triggered normally. For any scheduled scans that did not run during this period, use the "Run a scan" option in the Semgrep AppSec Platform to initiate a new scan as needed.
We've identified that the primary impact of the issue was scheduled Managed Scans for all tenants not initiating during this period. Other services, including Managed Scans on PRs and MRs, may only have experienced a slight delay. We have applied a fix for the issue and are now monitoring to ensure the platform is healthy.
Since approximately 20:15 UTC, Managed Scans and PR/MR comments are delayed and some may have failed. We have identified the issue and are working to restore full functionality.
Report: "Managed scans failing"
Last updateManaged scan jobs were failing to run for about twenty minutes. The faulty deploy was identified and rolled back immediately.
Report: "UI performance degraded for some organizations"
Last updateWe've confirmed that UI and login issues are resolved.
The fix is rolled out and we're monitoring to ensure functionality is restored.
As part of the remediation for the underlying issue, we made changes that temporarily impacted login for some users. This was expected and is self-resolving as the fix continues to roll out.
We've identified the cause of the latency and are implementing a fix.
We're investigating reports of degraded performances across pages in Semgrep AppSec Platform for some organizations.
Report: "Intermittent errors and slowness in Semgrep AppSec Platform"
Last updateWe've continued monitoring the fixes implemented earlier, and the platform has now stabilized.
We have applied a fix and are monitoring the results. We are still seeing some intermittent issues and are investigating further.
Report: "Semgrep AppSec platform intermittently slow or not loading"
Last updateThis issue is resolved, and the platform is now stable.
We have applied a fix and are monitoring the results. The platform appears to be stable.
We've received reports that Semgrep AppSec Platform is encountering problems loading or is running slowly. We're currently investigating the cause.
Report: "Delays in starting Managed Scans"
Last updateThis incident has been resolved.
We've applied a fix and scans and PR comment behavior should be returning to normal. We are monitoring the situation.
We're continuing to investigate the issue. We are also seeing that posting PR comments may be delayed.
We're currently investigating an issue with delays in starting Managed Scans and will update as soon as we have more information.
Report: "Degraded scan and app service"
Last updateScans and app service have been restored.
The fix is deployed and service is returning to normal. We'll continue to monitor.
Scans and app service are degraded. We have identified the cause and are working on rolling out a fix.
Report: "Project settings are unavailable in Semgrep AppSec Platform"
Last updateThis issue has been resolved, and project settings are available as normal.
We have released a fix for the issue and are confirming the result.
We've identified an issue with access to project settings in the platform and are working on a fix.
Report: "Supply chain dependency and license processing degraded"
Last updateBetween approximately 16:00 UTC and 22:00 UTC, Semgrep's dependency and license processing experienced failures due to an issue with a release. Supply chain findings were not affected during this time. New dependencies added to scanned projects during this time would not have appeared on the Dependencies page, and license policy violations were not processed and did not appear in pull requests. As of 22:15 UTC, scans are functioning normally. For affected pull request scans during this window, please close and re-open the pull request or push a new commit to re-trigger a new scan.
Report: "SMS scanning is degraded"
Last updateThis issue has been resolved and Managed Scans are running normally.
This issue has been resolved and Managed Scans are running normally.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "GitHub Outage Impacting Semgrep Managed Scanning"
Last updateGitHub has marked the issue as resolved. Semgrep Managed Scanning has returned to being fully operational.
We are currently experiencing a partial outage in Semgrep Managed Scanning services due to an ongoing issue with GitHub's services, which are currently non-operational. This only affects customers that host their repositories on GitHub.com. We are actively monitoring the situation and will provide updates as soon as more information is available or the issue is resolved. For live updates on GitHub's status, please visit their official status page at https://www.githubstatus.com/. Thank you for your understanding and patience.
Report: "Semgrep AppSec platform failing to load"
Last updateThis issue has been resolved and the platform is operating normally. If you saw any scans for pull or merge requests hanging during this time, you can close and re-open the request, or add a commit, to start a new scan.
We've applied fixes and are now monitoring results. The platform is able to load and appears healthy.
The AppSec platform is running slowly or failing to load, and we're investigating the issue. Some scans may be having trouble reporting back to the platform.
Report: "Semgrep AppSec platform running slowly or failing to load"
Last updateThis incident has been resolved.
We have identified the issue and the change has been reverted. We're now monitoring to ensure stability.
We have received reports that the AppSec platform is running slowly or failing to load, and we're investigating the issue. Some scans may be having trouble reporting back to the platform.
Report: "Problems starting scans from 2024-11-27 00:05-00:20 UTC"
Last updateThis issue has been resolved and scans are kicking off successfully.
We observed an elevated rate of errors when starting scans during this period. The issue was identified and the change has been reverted. We're now monitoring the health of the system.
Report: "AppSec Platform: Sign in with GitLab intermittently failing"
Last updateThis incident has been resolved.
GitLab has deployed a fix, and we will continue to monitor our metrics.
We've identified that attempting to sign in to Semgrep AppSec Platform with GitLab is not always successful. GitLab is currently undergoing extended planned maintenance which may be affecting availability.
Report: "Managed scans are failing since 21:30 UTC"
Last updateThis issue has been resolved and Managed Scans are running normally.
We've applied a fix and scans are running normally again. We're monitoring to ensure stability.
We've identified the issue and are applying a fix.
Report: "Managed scans failing since 17:50 UTC"
Last updateThis issue has been resolved and scan behavior is normal.
We've applied a fix for the issue and scans appear to be recovering.
We've identified that managed scans are failing since 17:50 UTC and are working to resolve the issue.
Report: "Intermittent Failures Onboarding Github Repos"
Last updateThis incident has been resolved
A fix has been deployed and we will continue to monitor the results.
Github app creation onboarding is occasionally returning errors
Report: "Inconsistent behavior of Supply Chain rule updates, some advisories since August 1st may be missing"
Last updateMonitoring indicates this issue is resolved and Supply Chain Advisories are now populating consistently.
A fix was implemented and rule sync has resumed. We're now monitoring the results to ensure that the fix is stable.
We've identified the issue with the advisory update and are working to resolve the problem.
We've observed that Supply Chain Advisories created since August 1st 2024 are not consistently populating in Semgrep. We are investigating this issue and will provide an update once we've identified the problem.
Report: "GitHub checks not completing for Managed Scans"
Last updateFrom 16:10 UTC to 20:45 UTC, Managed Scans running as GitHub checks were not being correctly marked as completed. The Semgrep scans completed and returned correct results but the check status was not updated. To clear out of date checks for any unmerged PRs, you can close and re-open the affected PR. This should trigger a new check that will complete successfully.
Report: "Unable to connect additional GitHub Cloud organizations"
Last updateThis incident has been resolved.
The fix is in production and we're monitoring to ensure the problem is resolved.
We've found the source of the issue and we're preparing to deploy a fix.
It's not currently possible to connect Semgrep deployments to additional GitHub Cloud organizations for source code management in the AppSec platform. Existing connections are not affected. We're investigating the cause.
Report: "Finding details page on semgrep.dev not loading for some customers using broker"
Last updateThis incident has been resolved.
Monitoring showing that system components related to broker are operational again.
A fix has been implemented and we are monitoring the results.
We are currently investigating the scope and cause of the issue
Report: "SMS Full Scans Not Running"
Last updateThis incident has been resolved.
Pull requests are still being scanned, but since 5 pm PST on Thursday, Sep 12, scheduled full scans have failed to run. We have identified the source of the problem and will have a fix up shortly.
Report: "Jira tickets not being associated with findings"
Last updateWe've backfilled associations between findings and Jira tickets, and have gathered a list of duplicate Jira tickets created during the outage for affected deployments. We will reach out to affected customers for next steps.
We've been monitoring and are seeing Jira tickets being successfully linked to associated findings. We've determined that Jira tickets created within the last five days were not associated with findings, and have prepared a backfill job to reassociate these tickets with the appropriate findings. We intend on running this backfill job tomorrow.
We've deployed a fix and are monitoring results.
We're seeing Jira tickets being successfully created, but not being linked to associated findings. We've identified the root cause and are deploying a fix. We're looking into how to reassociate Jira tickets with findings for associations that were not created. We will reach out to affected customers for next steps, and we will continue to update the status page as we release the fix and monitor.
Report: "Semgrep prod is down"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Elevated rate of errors when creating scans"
Last updateWe have been monitoring and are no longer seeing an elevated error rate. Scan creation is now healthy.
We have rolled back the code we identified as causing the issue and are monitoring the results.
We are seeing an elevated rate of 500 errors when creating scans. We believe we have identified the issue and are working to resolve it.
Report: "Semgrep scans failing due to connectivity issues with AWS STS (Security Token Service)"
Last updateThe AWS STS outage issue has been mitigated, and semgrep scans should be healthy and operational again.
Earlier today, Semgrep scans were failing due to an AWS STS outage, which resulted in SSL validation errors and connectivity issues. AWS has been actively investigating the issue, and the situation is showing signs of improvement. With the ongoing recovery, we expect services to stabilize. We will continue to monitor the issue.
Report: "Managed scans failing"
Last updateSemgrep Managed Scans were failing between approximately 16:00 UTC and 18:30 UTC due to an issue with a release. Scans are healthy again as of 18:30 UTC. For failed scans of pull requests, please close and re-open the pull request to re-trigger the check. You can also choose to push a new commit.
Report: "Existing findings being marked as re-opened"
Last updateThis incident has been resolved.
A fix has been released and we're monitoring the situation.
We observed that there was an increase in the existing findings being re-opened. We are currently investigating it.
Report: "Some SMS projects with diff scanning configured are not scanning"
Last updateNo further errors have been observed. Please reach out to support if you're seeing unexpected behavior around diff scans in PRs with Managed Scanning
A fix has been implemented and we are monitoring the results.
We've identified the source of the issue and are working on a fix.
It appears this started on July 30 and is only impacting some customers.
Report: "Many scans experiencing error submitting results"
Last updateThis issue has been resolved and scans are fully functional.
Scans are fully operational as of 21:00 UTC. A small number of scans failed to report results during the affected window. If you are seeing errors related to reporting scan results, try re-running the scan, or reach out to Semgrep Support with any questions or concerns.
From approximately 20:40 to 20:50 UTC, many Semgrep scans experienced an error submitting results to the Semgrep AppSec Platform. We have implemented a fix and are monitoring the results.
Report: "Some users unable to log in to semgrep.dev"
Last updateThis issue has been resolved. All users should be able to log in to the platform successfully.
A fix has been implemented and login via SSO should now be allowing access to the platform.
This issue is only affecting users logging into the Semgrep AppSec Platform via SSO. We are continuing to investigate.
Some users are currently unable to log in to the Semgrep AppSec Platform at semgrep.dev. We are investigating the issue.
Report: "Managed Scanning is degraded"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The service is up and running. We are monitoring the scans
Managed Scanning is degraded. Team is actively investigating and remediating, focusing on restoring diff scans first. Updates to follow.
Report: "Users unable to login"
Last updateThis issue has been resolved and the platform is fully operational.
We are continuing to monitor for any further issues.
We've deployed a fix for the issue and are now monitoring to ensure everything is stable.
Users are unable to log in. We are actively investigating this issue.
Report: "Login not working for all Semgrep tenants"
Last updateThis issue has been resolved. Users can now log in and use the platform normally.
We've deployed a fix for the issue and are now monitoring to ensure everything is stable.
Users on all Semgrep tenants are unable to log in. We are actively investigating this issue. Logged in users can continue to use the platform.
Report: "Semgrep App is experiencing an outage"
Last updateOn Monday, March 11th Semgrep Cloud Platform experienced a large, unexpected increase in traffic that uncovered a misconfiguration in one of our web services. This misconfiguration meant that we were not caching static data, resulting in our web service handling 10-15x its normal traffic. As traffic was spiking, the web service hosts were unable to handle the dramatically increased load, and began to crash loop. After configuring caching on our static data, our services quickly exited their crash loops and began to operate as normal.The Semgrep Engineering Team has performed a postmortem since the incident and have determined a number of action items that will help prevent and mitigate issues like this in the future. 1. Configure all static data to be cached through our CDN 2. Implement improved metrics and alerting at multiple levels of our infrastructure 3. Revaluate resource allocation for our critical web services
All traffic appears to be operating normally now that the fix has been applied
A fix as been applied and the team is monitoring and starting to conduct a post-mortem
We are continuing to investigate this issue.
Hello all, We've seen a spike in errors on the Semgrep API backend. The team is investigating and working on a fix. We apologize for the inconvenience.
Report: "Semgrep scans failing and parts of website down"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Supply Chain issues not correctly visible"
Last updateThe issue has been resolved. We are individually reaching out to affected parties who ran scheduled scans between 12:19am UTC and 1:45am UTC. If you ran a scan during this period, your Supply Chain page may be missing results. We will contact affected customers within 24 hours for resolution. Results will appear at the latest on your next full scan.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We have identified an issue where Supply Chain results are incorrectly showing results. The team has identified the root cause and is working on resolving the issue. We apologize for the inconvenience.
Report: "Semgrep Docker release removed the bash shell, causing issues in CI"
Last updateThe rollback is complete and we've confirmed that the latest Docker release has the bash shell available.
The release has been rolled back, and we are monitoring to ensure the rollback is working as expected.
We've identified an issue with the most recent release of the Semgrep Docker container which removed the bash shell, causing issues in CI for some users. We are working on a mitigation for the issue.
Report: "Semgrep App is experiencing an outage"
Last updateThis incident has been resolved
This incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been applied and the team is continuing to monitor the performance.
Hello all, We've seen a spike in errors on the Semgrep API backend. The team is investigating and working on a fix. We apologize for the inconvenience.
Report: "Occasional Secrets scan failures and mis-attributed findings"
Last updateThis incident has been resolved.
We determined that since Friday, Feb 23 22:30 UTC, a bug related to Semgrep Secrets has caused a small number of scans to fail, and some Semgrep Secrets findings to appear as Semgrep Code findings. Only a small percentage of scans using Semgrep Secrets were affected. The issue has been addressed and future scans will not be affected. We are continuing to work to clean up the erroneous findings.
Report: "Semgrep App Backend Down 18:24-18:29 UTC"
Last updateThe Semgrep App Backend was down for 5 minutes during a migration that locked the database. We are monitoring the situation and believe it has resolved.
Report: "Semgrep Cloud Platform and Docs not loading consistently"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are seeing timeouts and server errors on https://semgrep.dev, including Semgrep Cloud Platform and Semgrep Docs. Engineers are currently investigating.
Report: "Semgrep Supply Chain Rules blocking pipelines even when not configured."
Last updateThis incident has been resolved.
We have deployed a fix for this issue and engineering is seeing improved performance. We are continuing to monitor the fix for stability.
We are currently investigating an issue regarding those who are having scans blocked due to some Semgrep Supply Chain findings even if not configured. We are looking into this and will continue to update the status page.
Report: "Findings from semgrep-legacy rules"
Last updateThis incident has been resolved.
On Jan 20-21, 2024 a handful of semgrep-legacy rules were added to actively running semgrep rulesets, creating new and erroneous findings for users who had configured those rulesets. We have patched the problem, so these findings will no longer appear in scans. We are also actively working on a patch to remove these findings without users needing to take action.
Report: "Semgrep Secrets False Positives"
Last update2024-01-09 21:00 UTC Engineering has identified and is currently investigating an issue where Semgrep Secrets surfaced false positive findings. The cause was determined to be that rules from an internal training were accidentally applied to customer scans. The only impact to customers were the false positives shown by the Secrets product, no sensitive data was displayed to customers. 2024-01-09 22:00 UTC We disabled the ability to add new Semgrep Secrets rules to prevent additional rules from being mistakenly applied to customer scans while we continued to remediate the underlying issue. 2024-01-10 01:00 UTC We removed the previously mentioned rules. Scans completed after this time would remove any false positives findings previously identified. Customers that still see false positives should re-run a scan against the impacted projects. Please reach out to Semgrep support if this does not resolve the issue.
Report: "Semgrep Supply Chain Vulnerabilities page not Loading"
Last updateEngineering has deployed a fix and has deemed this incident to be resolved.
Engineering has identified the root cause of the issue and is now working on deploying an immediate fix.
Engineering is currently investigating further on the root cause of the issue.
Engineering has identified the root cause of the issue and is now working on deploying an immediate fix.
Engineering has identified and is currently investigating an issue with the Semgrep Supply Chain Vulnerabilities not loading for customers. We will respond back when relevant updates are present.