Historical record of incidents for env0
Report: "Issue with creating deployments in env0 SaaS"
Last updateThis incident has been resolved.
We are currently investigating a sporadic issue with creating deployments in the env0 SaaS. env0 agents are currently unaffected.
Report: "Issue with creating deployments in env0 SaaS"
Last updateThis incident has been resolved.
We've have resolved the issue and are currently monitoring the fix. The env0 SaaS should be fully operational
We are currently investigating a sporadic issue with creating deployments in the env0 SaaS. env0 agents are currently unaffected.
Report: "Issue with creating deployments in env0 SaaS"
Last updateThis incident has been resolved.
We are currently investigating a sporadic issue with creating deployments in the env0 SaaS.env0 agents are currently unaffected.
Report: "Deployments fail in SaaS runners"
Last updateThis incident has been resolved.
A recent docker image upgrade caused a dependency issue, our teams are working on a fix.
Report: "Errors while entering the app"
Last updateOutage resolved.
Revert is done. Monitoring...
Issue identified in one of our services. Revert in progress (~20m)
We are continuing to investigate this issue. Auto revert is upcoming.
We are seeing several errors when customers access the app. The issue is in investigation. Auto revert is upcoming.
Report: "Deployments are not started"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We noticed that deployments are not getting started, our team is working on a fix for the issue.
Report: "We're seeing some deployments either failing to start or report completion"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We're seeing some deployments either failing to start or report completion"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Request Timeout Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We noticed some timeout errors in our system. Our team is on it.
Report: "Timeout Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We identified some timeout errors in our system, our team is working on it.
Report: "Issues with GitLab integration"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "An issue handling workflow file from GitLab Enterprise Edition"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified and are working to resolve an issue when trying to deploy custom flows fetched from GitLab Enterprise Edition. We will roll out a fix shortly.
Report: "Deployment fails when using commit hash"
Last updateThis incident has been resolved.
We currently having some issues completing deployment when using git refs (commit hashes) in place of branch name. The issue has been identified and a fix is being issued.
Report: "Errors in accessing env0 UI"
Last updateThis incident has been resolved.
We've identified an issue with accessing the UI of env0 - our team is working on rolling out a fix
Report: "PR/CD Status checks not reported on GitHub"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue with how we report git providers status checks. We are looking into rolling out a fix within the next 30 minutes
Report: "Issues performing clone actions with GitHub"
Last updateThe source of the issue was found and the incident has been resolved. All operations with GitHub.com are now working as usual
There are sporadic issues when cloning from GitHub in env0, either when git cloning a GitHub repository, or when downloading a module with a GitHub source. A small number of such git clones fail due to network issues with GitHub.com This seems to be an issue in GitHub, though non has been reported as of yet. We are looking further into the issue
Report: "PR Plans and CD fail to report status check"
Last updateThis incident has been resolved.
We have identified an issue with how we report git providers status checks. We are looking into rolling out a fix within the next 30 minutes
Report: "Some deployments fail to start"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Deployments fail to start"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Errors reported on the env0 application UI"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "env0 UI issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Users are experiencing issues in env0 UI, our RnD members are investigating it.
Report: "Environment service degradation"
Last updateenv0 is back to operational state
Environment service degradation - querying environments and deploying are not fully available
Report: "Deployment service degradation"
Last updateThe issue is resolved and the application is operational
We face issues with the deployment service that cause steps not to log and the frontend to not load deployment relevant data.
Report: "UI app auto updates degradation"
Last updateThis incident has been resolved.
We rolled out a fix, currently monitoring.
We identified an issue with our AWS AppSync integration. This means the env0 UI auto updates are degraded. Some customers experience stuck loading stats, loading deployment logs, or stating deployment. As a temporary workaround, you can try to click on our “refresh” button (2 blue circular arrows). That should load your data right away. We are deeply sorry. Our team is monitoring and working on a fix.
Report: "Degraded performance starting new deployments"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Users may experience failures and timeouts when triggering new deployment.
Report: "Gitlab Service Disruption"
Last updateThis incident has been resolved.
Gitlab webhooks and GitOps flows are interrupted because of ongoing incident in gitlab. For more information you can go to: https://status.gitlab.com/
Report: "Delays in Deployments"
Last updateThis incident has been resolved.
A fix was implemented and we are trying to monitor if the issue is resolved.
The issue has been identified and we are working on a fix.
We are continuing to investigate the issue together with AWS support. It looks like this is an issue with our EKS cluster. Customers who are using the Self-hosted agent should not be affected.
We are seeing a lot of deployments delays. At first look, it seems to be coming from AWS Step function API where we are getting a lot of network errors and timeouts. We are currently contacting AWS to find out what is the root cause of this issue.
Report: "Increased Error Rates and Delays in Deployments due to AWS"
Last updateThis incident has been resolved.
We're experiencing an issue with deployments, and overall elevated error and delays in the application. The source of the problem is an AWS outage that affects our application, mostly around API Gateway, EKS, Lambda services on "us-east-1" region. For more information about the AWS outage please refer to AWS status page - https://health.aws.amazon.com/health/status We are currently monitoring the outage, and trying to find workarounds for it.
Report: "Incident with starting new deployments"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some of the deployments fail to start, we are taking a look.
Report: "AWS issues"
Last updateOur login service is now fully restored following the recent AWS outage. You can now access our services without any issues.
Our service is now fully restored following the recent AWS outage. We want to assure you that we are actively monitoring the situation to ensure continued stability. If you encounter any further concerns, please reach out to our support team.
We are currently experiencing a service disruption with our service. This is due to a significant outage at AWS: https://health.aws.amazon.com/health/status We will keep monitoring and updating
We are continuing to investigate this issue.
We are continuing to investigate this issue.
The system is unstable due to regional issues in us-east1
We are continuing to investigate this issue.
Our system is currently unstable due to issues with auth0
Report: "Deployment Logs may be unaccessible"
Last updateThe incident has been resolved
We've uploaded a fix, and are monitoring the issue
We're experiencing an issue with deployment logs not being accessible via the env0 UI. We've identified the source of the issue and working on resolving the issue
Report: "Sporadic timeouts when starting new deployments"
Last updateDeployments on env0 SaaS should now start as usual. Deployment images on env0 SaaS platform will be pulled from mirror till GitHub resolve their outage of ghcr.io.
We're experiencing issues starting new deployments due to an outage of ghcr.io. Our team's working rerouting image pull to a mirror - deployments may take longer to start and can timeout
Report: "Deployments may experience degraded performance due to GHCR issues"
Last updateGithub has reported the issue as resolved
Implemented a short-term mitigation for SaaS users, deployments will be slower but should work
We are continuing to monitor for any updates.
Github Packages is experiencing degraded availability. This can cause a significant delay in all deployments - regardless of customers' VCS Providers.
Report: "Elevated errors in Authentication and Management APIs. Issues Logging In"
Last updateAuth0 authentication services had downtime, which affected env0 as well env0 users had issues logging in
Report: "Deployments may experience degraded performance due to GHCR issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
Deployments may experience degraded performance due to GHCR issues
Report: "Issue with loading the App"
Last updateThis incident has been resolved.
Due to an issue with a third party provider (MyFonts), The front-end application might not load for certain customers.
Report: "Incident with starting new deployments"
Last updateThe root cause (GHCR) has been resolved and deployments on env0 are starting as usual.
Due to an issue with Github and GHCR, new deployments will not start or take a very long time to start.
Report: "GitHub Enterprise Pull Request Integration Problem,"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with GitHub Enterprise Pull Request Integration.
Report: "Auth0 flakiness"
Last updateThis incident has been resolved.
auth0 seems to be having increased flakiness, the source seems to be cloudflare which has increased error rates
Report: "Issue with Private Module Registry"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're seeing failures when trying to deploy Templates using our Private Module Registry. We are currently investigating this issue
Report: "Drift Detection runs fail"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We've identified an issue with Drift Detection runs and are working to resolve it.
Report: "Downtime due to third party provider issues"
Last updateThis incident has been resolved.
Cloudflare (and consequentially Auth0) have reportedly implemented a fix and monitoring the results. And so are we
The issue has been identified and a fix is being implemented.
Report: "Elevated API Errors"
Last updateThis incident has been resolved.
A fix was rolled out successfully and we're monitoring the results
We've identified a possible cause and are looking to roll out a fix
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "we're experiencing issues with Gitlab integrated templates - this might affect your deployments or gitflow triggers"
Last updateWe've rolled out the fix that will allow GitLab tokens to be refreshed. You will need to re-authenticate your using GitLab user/bot with env0 by editing the Template in env0 that integrates with that GitLab user, click the "authorize another GitLab user" and authorize with your user/bot again. Once again we're terribly sorry for the inconvenience - we're in touch with GitLab trying to understand how it came to pass that we were not alerted about this change in their API to avoid such future incidents.
GItLab.com (the Saas) updated their version to 15.x. which deprecates our authorization approach. We are working on a solution, in the meantime, if you must use GitLab you can re-authorize by going into the template, selecting "authorize another Gitlab user"
we're experiencing issues with Gitlab integrated templates - this might affect your deployments or gitflow triggers
Report: "Deployments are taking longer to start"
Last updateenv0 deployments are at peak performance again
Deployments are now operational but may take longer to start than usual. We're still working on getting env0 back to peak performance
Deployments are now operational but may take longer to start than usual. We're still working on getting env0 back to peak performance
env0 deployments will take a long time to start or will time out. The root cause seems to be a global issue with ghcr.io, which is hosting our deployment image. We are looking into it.
Report: "Slowness in env0 deployments"
Last updateThis incident has been resolved. If you have an environment that appears "In Progress" when it has already finish, then clicking on the "abort" button will resolve the issue.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue. Effects of this issue include slowness in deployments, inability to approve or cancel deployments
Some customers are currently experiencing slowness in env0, for deployments and drift detections. We are currently investigating the issue
Report: "Slowness in env0 SaaS deployments"
Last updateThis incident has been resolved.
The issue has been resolved. All new deployments should now start as usual
Some env0 deployments are currently slower than usual. We are investigating the source of this issue
Report: "Partial System Outage"
Last updateThe incident has been resolved. All env0 components are now fully operational
We are continuing to investigate this issue.
Some aspects of env0 are currently not working properly, including secrets, cost, and AWS assume role We are looking
Report: "Partial System Outage"
Last updateThis incident has been resolved.
A fix has been deployed ad we're monitoring the results
We've identified a cause for a partial outage in env0 - ETA on a being back online is 15-20 minutes