Historical record of incidents for AppVeyor
Report: "GCP cloud is experiencing issues"
Last updateGCP cloud is experiencing issues which affects AppVeyor builds running on its infrastructure.
Report: "AppVeyor CI web portal is not responding"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "macOS builds degraded performance"
Last updateThis incident has been resolved.
The network requests in macOS workers are either slow or failing. We are investigating the issue.
Report: "AppVeyor CI is not responding and builds are staying in Queued mode"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "NuGet restore slow speed"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Build cache is intermittently failing to upload or download cache items"
Last updateThis incident has been resolved.
AppVeyor's object storage provider has identified the issue and is working on the fix.
Report: "Intermittent networking issues while accessing NPM registry"
Last updateThis incident has been resolved.
A temporary workaround has been implemented in the form of NPM HTTP proxy. We are working on a permanent solution.
Report: "Performance degradation"
Last updateThis incident has been resolved.
We are noticing performance degradation of some AppVeyor services. Builds maybe affected because of that. We are investigating.
Report: "Build cache is not available"
Last updateThis incident has been resolved.
We are working with our hosting provider to resolve the issue. To temporary disable cache restore in your projects you can define `APPVEYOR_CACHE_SKIP_RESTORE=true` environment variable on either project settings or in global account config.
Report: "GitHub is experiencing degradation of services"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
GitHub incident: https://www.githubstatus.com/incidents/dsf2qtzh4jpz
Report: "Builds are failing to start on GCP cloud"
Last updateThis incident has been resolved.
There is an ongoing known issue with Google Compute Engine right now and Google team are working on it.
We are currently investigating this issue.
Report: "GitHub is current experiencing issues"
Last updateThis incident has been resolved.
GitHub team has rolled out a fix and monitoring.
GitHub incident: https://www.githubstatus.com/incidents/80d0cs6kpsps
Report: "Builds are failing to start in selected environments"
Last updateThere was an intermittent connectivity issue between AppVeyor and primary build environment's data center.
We are currently investigating this issue.
Report: "AppVeyor is not responding"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
There were issues migrating database schema during AppVeyor update. We are working on the fix.
Report: "Builds are failing to start on Azure cloud"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating the issue with failing/delayed builds on Azure cloud. The builds using `Windows Server 2019` image are affected. Customers can switch to `Visual Studio 2019` image that offers the same software/features to get the builds going.
Report: "Builds are failing to start on GCE cloud"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating the issue with provisioning new build VMs in GCE cloud.
Report: "GitHub is experiencing queue backlogs affecting some webhooks deliveries"
Last updateThis incident has been resolved.
GitHub is currently experiencing queue backlogs affecting some webhooks deliveries: https://www.githubstatus.com/
Report: "Degraded GitHub webhook notifications"
Last updateThis incident has been resolved.
There is an ongoing incident on GitHub side: https://www.githubstatus.com/
Report: "Unable to run builds using 'Windows Server 2019' image"
Last updateFixed Windows Server 2019' image has been deployed.
We understand the root cause and working on redeployment of the image. This problem does not affect builds using any other image.
Report: "Builds on GCE environment are failing"
Last updateThis incident has been resolved.
There is an ongoing issue with Google Compute Engine: https://status.cloud.google.com/incident/compute/19003
Report: "GitHub webhooks delayed delivery"
Last updateThis incident has been resolved.
GitHub incident: https://www.githubstatus.com/incidents/nf4gcjtcsqdb
Report: "Intermittent nuget packages restore issues"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Problem is related to the network infrastructure problems ion Azure West US (https://azure.microsoft.com/en-us/status/).
Report: "GitHub webhooks rejection"
Last updateThis incident has been resolved.
The allowed webhook IP addresses range has been updated and GitHub webhooks should be working again. We apologize for the inconveniences caused by this incident!
We are currently investigating this issue.
Report: "Slow response and builds are queued"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
There was a huge number of ongoing malicious requests to AppVeyor exploiting certain functionality and degrading AppVeyor database performance.
We are currently investigating this issue.
Report: "Builds are not starting"
Last updateThis incident has been resolved.
AppVeyor has been affected by network infrastructure issues on Azure in West US region: https://azure.microsoft.com/en-us/status/ It looks like the service is getting back up and we are monitoring it.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "AppVeyor is not available"
Last updateIt's been resolved. The problem was a database migration during AppVeyor update preventing roles from a normal start. Sorry for the inconveniences caused by this issue.
We are investigating.
Report: "Errors provisioning build VMs on GCE cloud"
Last updatePrimary build environment has been recovered.
Customers see "Cannot assign job to clouds/groups 'pro, pro-backup'. All clouds within the group are either failing, or offline, or busy." error on failed build details. There was a power issue in our primary data center. We are currently performing recovery procedure.
Report: "NuGet restore issues"
Last updateThe issues has been fixed.
We are observing the errors in all build environments while restoring NuGet packages during the builds. NuGet team has confirmed the issue on their status page: https://status.nuget.org/ which is related to CDN provider service degradation in North America: https://status.verizondigitalmedia.com/#network-status
Report: "Issues with switching node version"
Last updateWe have switched node version manager to download a feed from a healthy blob.
'install-Product node <version>' fails with 'Unable to connect to the remote server' error. Root cause of the problem is unresponsive Azure blob container. We are working with Azure on restoring the functionality.
Report: "Sporadic errors accessing build cache"
Last updateThis incident has been resolved.
The issue has re-occurred. We are still working with our hosting provider to resolve it.
Report: "AppVeyor cannot list repositories for GitHub App authorization"
Last updateThis incident has been resolved.
It should be working now.
GitHub is experiencing issues: https://status.github.com/messages
Report: "Sporadic errors accessing build cache"
Last updateThis incident has been resolved.
We are observing sporadic 403 (Forbidden) errors in builds working with build cache. We notified our hosting provider about the issue and are awaiting for their response.
Report: "GitHub is having major issue"
Last updateThis incident has been resolved.
https://status.github.com/messages
Report: "AppVeyor builds are queued"
Last updateThis incident has been resolved.
We are still seeing the huge queue. It will take more time to drain it to the normal state. Sorry about inconveniences.
We've resolved the issue with queued builds.
AppVeyor builds are delayed because of increased webhook calls from GitHub. We are working on clearing up the queue.
Report: "NuGet restores return 503 error"
Last updateNuGet.org is no longer degraded.
We are receiving reports from customers that nuget restores from v2 feed sporadically return 503 Service Unavailable errors. An issue filed on NuGet Gallery project: https://github.com/NuGet/Home/issues/7333
Report: "Slow and intermittent chocolatey downloads"
Last updateThis incident has been resolved.
Customers can experiences unstable chocolatey downloads. Please watch http://status.chocolatey.org/. As a workaround add 'appveyor-retry' before 'choco install'
Report: "Intermittent errors while uploading artifacts to Google storage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
There is an ongoing incident on GCP: https://status.cloud.google.com/incident/storage/1800
We are currently investigating this issue.
Report: "Intermittent issues with Chocolatey packages installation"
Last updateChocolatey maintenance completed.
Chocolatey is performing maintenance (https://twitter.com/chocolateynuget/status/1031915993646424067). To workaround or at least decrease error rate, add 'appveyor-retry' before 'choco install'
Report: "AppVeyor nuget feed timeouts"
Last updateUsers experienced intermittent timeouts why restoring nuget packages hosted on AppVeyor project and account feeds.
Report: "Slow cloning from GitHub"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Intermittent networking issues"
Last updateThis incident has been resolved.
The fix has been applied.
Some customers are reporting about intermittent connectivity issues in their builds.
Report: "Connectivity issues to nuget.org"
Last updateVerizon Digital Media Services (CDN service used by Azure) has routed the network path to a different ISP which fixed the issue. The reason for the packet loss on that specific link is still being investigated by Verizon.
We've deployed a workaround (along with <a href="https://www.appveyor.com/updates/2017/07/19/">other changes</a>) to all three images: <code>Visual Studio 2013</code>, <code>Visual Studio 2015</code> and <code>Visual Studio 2017</code>. Let us know if you are still experiencing the issues restoring nuget packages. In the mean time we continue working with Azure, Verizon and Rackspace on a permanent solution.
While working with NuGet, Azure and Rackspace teams on resolving this issue we are deploying a temporary workaround to build worker images.
The issue is a "broken" route between Rackspace data center and US-based CDN edge server for api.nuget.org. Unfortunately, there is no ETA yet. We continue working with nuget team to resolve the issue as soon as possible.
Temporary workaround is to redirect nuget restore to European CDN edge server by adding this to <code>appveyor.yml</code>: <pre> hosts:<br> api.nuget.org: 93.184.221.200 </pre> If you are configuring your projects via UI you can add host mappings on "Environment" tab of project settings. The fix above will work only if <code>nuget restore</code> command is run after <code>install</code> phase. To update hosts file <b>before</b> <code>install</code> phase please use the following solution: <pre> init:<br>- ps: '[System.IO.File]::AppendAllText("C:\Windows\System32\drivers\etc\hosts", "`n93.184.221.200`tapi.nuget.org")' </pre> Please note this is a temporary workaround that should be removed once the issue is resolved. Sorry for the inconveniences.
We are investigating intermittent connectivity issues to api.nuget.org.
Report: "Intermittent connectivity issues to some resources hosted on CloudFlare CDN"
Last updateThere was congestion/high utilization issue last days at Equinix IX Chicago to the CloudFlare hosting datacenter. This was mitigated by CloudFlare networking team and the problem should no longer be noted.
Build worker VMs running in LiquidWeb data center are experiencing intermittent connectivity issues to selected resources hosted on CloudFlare CDN. This affects download/installation of certain Chocolatey packages and node modules (such as node-gyp). We are actively working with both LiquidWeb and CloudFlare to resolve the issue.
Report: "GitHub commits delay"
Last updateThis incident has been resolved.
GitHub is currently experiencing delays with commits and webhooks deliveries: https://status.github.com/messages
Report: "Sporadic 500 (Internal Server Error) while restoring some packages"
Last updateThis incident has been resolved.
We are no longer receiving reports with this issue.
Customers may experience sporadic 500 errors while restoring NuGet packages in their builds. Related NuGet issue: https://github.com/NuGet/NuGetGallery/issues/4933
Report: "Timeout restoring Nuget packages"
Last updateNuget.org is back to normal
Nuget.org is degraded: https://status.nuget.org/
Report: "Some builds may start with delay"
Last updateAzure Service Bus team helped us to identify and fix the issue.
Customers may notice that some builds may start with significant delays somewhere between 5 and 10 minutes. We believe the cause of that is issues with Azure Service Bus queue used by AppVeyor backend workers. We are still investigating and working with Azure support. Thank you for your patience.
Report: "Build VMs are not getting provisioned in GCE cloud"
Last updateDisks provisioning time is back to normal.
This issue might be related: https://status.cloud.google.com/incident/storage/18001
Persistent SSD disks are failing to provision in GCE cloud. We are investigating.
Report: "https://ci.appveyor.com returns 'service is unavailable' error"
Last updateUnhealthy web front-end server identified and fixed.
Error is intermittent. We believe we found unhealthy web front-end server.
We are investigating "The service is unavailable" error being returned from https://ci.appveyor.com. Will update shortly.
Report: "Nuget package restore failures"
Last updatenuget.org up and running.
Nuget.org experiencing an outage. Check https://status.nuget.org for details.
Report: "GitLab projects cannot be added and built"
Last updateGitLab API V4 support implemented and deployed
GitLab discontinued API V3 support which AppVeyor uses. We are working on fix.