Historical record of incidents for Anchor Hosting
Report: "Domain configuration pending error message"
Last updateWe're aware of an issues affecting customer websites with Kinsta loading. See here for details on the outage: https://status.kinsta.com/incidents/xxy7fv9qk77f.
Report: "Kinsta outage"
Last updateCloudflare is reporting everything is restored on their end: https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc. All customers continue to remain healthy. Most customers were restored at 4:20 EST. In total around 23 minutes of outage.
All sites currently back online. Will keep monitoring the situation until everything is stable.
This outage is due to an issue with Cloudflare. Kinsta is aware of the problem and working on a solution: https://status.kinsta.com/incidents/9bqvq6tbnc8l.
Within the last 10 minutes site monitors are showing many customers sites with Kinsta are offline. We are aware and working on a resolution. Will post details here.
Report: "Site outages and critical errors"
Last updateAt this time all sites have been restored.
A fix is being rolled out to all affected customers. Some websites have already been restored. Will continue to monitor until everyone affected has been restored.
Kinsta sysadmin are working on the issue. It appears that a batch of websites are incorrectly routing traffic from production environments to staging environments. The production sites appear healthy however are unavailable until proper domain routing is restored.
We are seeing a batch of customers receiving errors ranging from critical errors, staging urls and PHPmyadmin pages. Working with Kinsta to find out what's happening and restore affected customer websites.
Report: "PHP rollbacks"
Last updateAll customers identified with PHP 8 compatibility issues have been rolled back to PHP 7.4. If you notice any issues with your website, be sure to reach out to support: https://anchor.host/account.
Some customers websites are currently running into issues with PHP update applied late last night. Currently rolling back any customer affected to PHP 7.4 to stablize. I will be applying any compatibly fixes as needed before upgrading PHP. Sorry for the disruption.
Report: "Site outages at Google Cloud datacenters"
Last updateSite monitor is showing all sites are back online as of 10:30am. Kinsta is continue to monitor the situation.
Outages are now affecting many different Google Cloud datacenter locations. Google engineers are actively working to restore affect locations.
At 9am this morning, a number of customer websites at one of Google Cloud's datacenter are experiencing an outage. Kinsta is aware of the issue and is working to restore the affected websites. See here for more details: https://status.kinsta.com/incidents/zsb97sw6y69w.
Report: "Service Disruption"
Last updateCloudflare reported the issue is now resolved.
Getting reports that various customer sites are loading slow or experiencing a partial outage relating to disruption with Cloudflare. Kinsta and Cloudflare is aware of the issue and are working on it: https://status.kinsta.com/incidents/mlq5nl3v9cvd.
Report: "Cloudflare Errors"
Last updateCloudflare has mitigated the issue and is monitoring the situation.
The Cloudflare outage appears isolated to individuals located in Canada: https://www.cloudflarestatus.com/incidents/1yz6wmypbmb9.
Seeing reports of various customers seeing an error 1105 Temporarily unavailable from Cloudflare. Taking a look and will followup when more information is available.
Report: "Cloudflare outage affecting some customers on east coast"
Last updateCloudflare has resolved the issue: https://www.cloudflarestatus.com/incidents/zbzjv8sm3g94.
There appears to be a localized outage with the Cloudflare network affecting visitors on the east coast from reaching websites. They are aware of the problem and are working to restore services as shown here: https://www.cloudflarestatus.com/incidents/zbzjv8sm3g94.
Report: "Kinsta development URLs retiring for SFTP and SSH"
Last updateThe Anchor Hosting portal has been updated: https://anchor.host/account. Moving forward please use the listed IP addresses for SFTP and SSH connections.
Kinsta is working on improving their infrastructure and is making breaking changes to how their built in development urls operate (ex: sitename.kinsta.cloud). Anyone connecting via SFTP or SSH will need to replace their kinsta.cloud address with the IP or domain name of the website. I recommend using the IP address. Will be rolling out updates to https://anchor.host/account with updated IP info over the next few days. In the meantime feel free to reach out to support@anchor.host if you need help getting connected via SFTP or SSH.
Report: "Kinsta Datacenter Outage"
Last updateThis incident has been resolved.
At this time all websites are back online however the datacenter is still being monitor for stability.
Kinsta is aware of an issue affecting the US East 4 datacenter that is causing site outages. They are working to restore the affected websites. https://status.kinsta.com/incidents/f3tyyl0vm8sr
Report: "Kinsta Outage"
Last updateAll Live and Staging sites have been restored. The services are all fully available now.
From Kinsta, At approximately 2:20 PM EST, 11 August 2020, one of our Google Cloud virtual machines went offline due to an apparent hardware failure. We worked on this issue with Google Cloud Support for multiple hours without success. Eventually, we determined that our best course of action was to restore the affected sites using our most recent backup. We create backups of all of our virtual machines every four hours. In this case, the most recent backup was approximately three hours old at the time that the incident began. As a result, the backup we have restored does not contain any changes made to affected sites after 11:07 AM EST, 11 August 2020. As of 11:22 PM EST 11 August 2020, all LIVE sites which were affected by this incident have been restored from that backup. Our Engineering team is continuing to work on restoring affected STAGING sites. At this time, we are focused on the recovery of website data between the 11:07 AM and 2:20 PM EST. We will keep you updated on our progress as it pertains to recovering this data. We are deeply disappointed by this extended outage. Thank you for your patience with us during this event. We will continue to keep you updated regarding this incident until we are confident it has been fully resolved. Thank you for being a valued Kinsta customer.
Update from Kinsta: The ongoing issue is due to an apparent hardware failure at Google Cloud. We've been working with Google Cloud Support seeking resolution but at this time an ETA on restoration of the affected virtual machine is unavailable from Google Cloud. Due to the ongoing uncertainty regarding when the hardware failure will be resolved, we've made the decision to restore the affected machine from our most recent backup. Backup restoration is in process now. We will continue working with Google to seek the retrieval of all data leading up to the point where the hardware failure occurred and we will provide additional information as soon as possible.
Still no ETA on when affected websites will be restored. Kinsta is working with Google Support which is taking longer then usual for this type of issue. This outage is isolated to a individual hardware failure within a Google datacenter.
This issue has been narrowed down to an individual LXD within the Northern Virginia (US East 4) datacenter. Kinsta is aware of the issue and is working to restore. https://status.kinsta.com/incidents/qc639q78n6fg
Some customers with websites located within the Northern Virginia (US East 4) datacenter are currently offline. This appears to be an issue with a particular Google datacenter. Will report back as new information is known.
Report: "Outage affecting customers at South Carolina (US East 1) data center"
Last updateAll customer sites have been restored.
Starting 30 minutes ago, customers hosted at the South Carolina (US East 1) data center are experiencing outages. Kinsta is aware and is working to resolve. See more details here: https://status.kinsta.com/incidents/8qzy95tr7knd.
Report: "Iowa (US Central) data center outage"
Last updateAll affected sites have been restored.
A few customer websites are currently offline due to an outage at Iowa data center. Kinsta is working to restore affected websites.
Report: "Kinsta outage"
Last updateKinsts is reporting the issue has been resolved.
A number of websites hosted at Kinsta's Iowa datacenter are currently offline. This is a known ongoing issue: https://status.kinsta.com/incidents/9phncbhccc6w. They are working to restore the affected websites.
Report: "Kinsta Outage"
Last updateAll websites have been restored.
Kinsta engineers are actively working to restore the affected websites: https://status.kinsta.com/incidents/7379hfj1mrk3. At this time I do not have an ETA for when it will be resolved.
A number of customers that are hosted with Kinsta at the Northern Virginia (US East 4) location have just gone offline. Kinsta is working to restore the affected websites.
Report: "Google outage causing site outages"
Last updateLast remaining websites have been fully restored.
At this time most websites have been restored. The remaining offline sites are expected to be restored shortly.
Google believes they have identified the root cause and is expecting services to return to normal here shortly: https://status.cloud.google.com/incident/compute/19003.
Beginning an hour ago a major outage affecting Google infrastructure is affecting most of customer websites. Both Kinsta and Google are working to restore the affected regions. See more details on the incident here: https://status.kinsta.com/incidents/0mc8tyyywlj6.
Report: "Kinsta datacenter outage"
Last updateGoogle Cloud reported a [hostError](https://cloud.google.com/compute/docs/faq#hosterror) for one of Kinsta’s compute instances. These sorts of errors are quite rare and occur due to hardware or software failures at a Google Cloud data center. In this case, this error caused one of Kinsta’s virtual machines to go offline. Google restarted the affected compute instance as quickly as possible but that took several minutes and during that time all site containers associated with that instance were unavailable.
This incident has been resolved.
The host machine has been restored and all sites are back up and running.
Most websites are already restored. The remaining should be back within the next few minutes.
A number of websites located on the Northern Virginia (US East 4) datacenter are currently offline resulting in 502/504 errors. Kinsta is investigating. Appears to be an outage with the datacenter.
Report: "502 error outages"
Last updateKinsta has restored all affected websites.
Kinsta is working on an issue affecting a limited number sites with Anchor Hosting. More details here: https://status.kinsta.com/incidents/k412kpqnqgyj.
Report: "Kinsta outages"
Last updateAll sites have been restored.
Kinsta engineers performed a restart on the underlying server. Sites are slowing being restored.
Some sites on Kinsta's eastern datacenter appears to be loading slowly or failing to load. The first outage happened an hour ago. Kinsta is aware of the issue and is working to stabilize the affected sites.
Report: "Kinsta outage"
Last updateAll sites have been restored.
Kinsta sysadmins are working to restore the affected sites. https://status.kinsta.com/incidents/1zj4qg44xtv4
Just received notice that a number of sites at Kinsta just went offline. Working to resolve.
Report: "Kinsta Outage"
Last updateAll sites are back online.
A few sites hosted with Kinsta have just gone offline. This appears to be affecting all sites at the Iowa datacenter location. Will post more details as I have them.
Report: "WP Engine server outage"
Last updateAfter some investigation, it appears that the downtime was due to a host-level issue with WP Engine's infrastructure provider. WP Engine hasn't found any indication of excessive resource usage, and this appears to be an isolated event.
There appears to have be an issue with an automated mysql restart process with required some manual intervention in order to complete. With that now completed the server is back to normal operations. WP Engine will be doing a deeper look into why this server needed the mysql restart. At this time everything looks healthy.
Websites have been restored. This appears to be an isolated issue with one of WP Engine's server. Engineers are working on the server right now to pinpoint and resolve the issue.
A few minutes ago I received notice that a number of sites with WP Engine are offline. Working with WP Engine for more info and to resolve.
Report: "Partial Outage"
Last updateAt 5:10pm an outage affecting one of my WP Engine services experienced an outage affecting some customers. Everything appears to have stabilized within 20 minutes. Will post more details as I have info.
Report: "Slow performance on 'anchorhost4'"
Last updateYesterday there were two large spikes in traffic which affected performance and reliability of 'anchorhost4' (104.130.140.77). After reviewing it would appear the server was over capacity due to a large number of uncacheable requests directed to one of the installs. To prevent this from happening again I have moved the affected install over to a new hosting provider which is better configured to handle uncached requests.
The load has been back at normal levels for the last 30 minutes. Will continuing to monitor over the next few days to see if further action is needed.
Appears to be a large amount of unwanted traffic. A block has been put into effect to help reduce load.
Starting 15 minutes ago I've been seeing some slow performance with some brief 504 errors affecting sites on 'anchorhost4' (104.130.140.77). Working with WP Engine to resolve. Will update as I have more information.
Report: "Kinsta Outage affecting certain sites"
Last updateThis incident has been resolved.
As of a few minutes ago all websites have been restored. Google engineers are still working. I'll keep monitoring and close this ticket out after they complete their work.
The outage is due to an issue with the Google Datacenter: https://status.cloud.google.com/incident/compute/18004. They are aware of the problem and are working to restore.
Beginning a few minutes ago, certain sites with Kinsta are experiencing an outage. Kinsta is aware and is working on the issue.
Report: "Kinsta Outage affecting certain sites"
Last updateService has been restored.
A few customers hosted with Kinsta are experiencing an outage starting a few minutes ago. Kinsta engineers are working to resolve an issue affecting one of their US Central Load Balancers.
Report: "Kinsta 501 errors"
Last updateAfter 30 minutes of instability all sites are now healthy.
Beginning 20 minutes ago an issue with one of Kinsta's host machines is causing some sites to timeout with 501 errors. They are working to resolve.
Report: "WP Engine Outage"
Last updateThis incident has been resolved.
After 5 minutes of outage, all WP Engine servers appear to be back online . WP Engine says the will be posting more details here shortly. Will continue to monitor.
Just received noticed that multiple WP Engine servers are experience an outage. Will post more details as available.
Report: "Instability on 'anchorhost6' server"
Last updateWP Engine performed a server reboot in order to stabilize. I've manually offloaded some high usage installations. Will continue to monitor and followup with WP Engine to make sure this server is in a healthy state.
Starting around an hour ago there have been a number of outage reports on server 'anchorhost6'. Working with WP Engine in order to stabilize.
Report: "Instability on 'anchorhost6' server"
Last updateWP Engine performed a server reboot in order to bring things back online. Everything appears healthy now. Total of 30 minutes of instability.
Starting 10 minutes ago there have been a number of outage reports on server 'anchorhost6'. WP Engine is reviewing the server. All other servers are healthy at this time. Appears to be an isolated issue with this particular server.
Report: "'anchorhost2' stability issues"
Last updateAt this time all sites are stable. The root issue caused by database size has significantly been reduced. Will continue to monitor the server over the next few days for stability.
The load issue has escalated. WP Engine engineers are still working on this and are rebooting the server now to regain stability and get everything back and operational.
Starting 45 minutes ago the 'anchorhost2' is again experiencing some instability. Been working with WP Engine's to get stabilized.
Report: "'anchorhost2' stability issues"
Last updateThe load on the server stabilized around 3hrs ago. Will continue to monitor and move installations to other servers until this server remains stable.
WP Engineers are continuing to work on this server. While some of the websites are loading there is still a number of websites loading slow and/or displaying load errors.
Starting 20 minutes ago the 'anchorhost2' WP Engine server has been unstable. This is related to outages happening last week. While optimization were made the root issue has only now been identified as excessive database size. I'm working with WP Engine to move affecting installs to over boxes in order to stabilize this server.
Report: "'anchorhost2' stability issues"
Last updateAfter some brief instability this morning, the 'anchorhost2' server has remained stable throughout the day. Still working to trace down the source of slow down. Until that has been found and corrected I've begun internal migrations to other servers to help spread the higher then typical load.
This morning the server 'anchorhost2' has become unstable. Working with WP Engine to identify the source of the disruption and stabilize.
Report: "Server anchorhost2 offline"
Last updateAfter 30 minutes of instability everything is back to healthy levels. The outages was caused due to high I/O wait. Will be doing some investigations as to why that might be.
Beginning 20 minutes ago one of my WP Engine servers went offline. WP Engine engineers are actively working to stabilize.
Report: "Server anchorhost4 unstable continued"
Last updateEverything has continued to remain stable today with exception of one large spike resulting in a few small outages around 10:50am. WP Engine believes have figured out what went wrong with the migration and are working to improve their automations to prevent this from happening on other servers. At this time they believe everything is healthy with anchorhost4. I'll continue to monitor for stability.
Beginning last night the server became unstable resulted in multiple outages each around 10 to 20 minutes long. WP Engine is aware and is working to resolve. If not resolved shortly I have directed them to begin another migration to a new server.
Report: "Server anchorhost4 unstable"
Last updateWP Engine engineers believe they have resolved the issues with the new server and the datacenter. Will continue to monitor for stability.
Post migration everything was stable up until late night when WP Engine noticed some performance issues resulting in outages. They are currently working to stabilize.
Report: "Server anchorhost5 unstable"
Last updateThis incident has been resolved.
Last night the server continued to remain unstable resulting in numerous small outages. WP Engine reached out to the datacenter and had them replace the physical hardware on the server. Since that replacement (2 hours ago) everything has been stable and the server is looking healthy. Will continue to monitoring throughout today.
There are no clear causes to the instability caused on the anchorhost5 server today. WP Engine is pulling more logs and attempting to trace down the source of the disruption. In the meantime high traffic installations are being moved to other servers until a cause can be determined.
The engineers have finished their revisions which seem to be stabilizing the server however continuing to work with WP Engine to make sure it's healthy.
WP Engine engineers are attempting to stabilize the server through a series of tweaks to Nginx. Expected 10 minutes of outage beginning now on anchorhost5. All other servers are stables.
In followup to last night maintenance, https://anchorhost.statuspage.io/incidents/hhq8jdkf7t4n, the server anchorhost5 has just started experiencing instability issues resulting in sporadic outages. Working with WP Engine to stabilize the server.
Report: "'anchorhost2' stability issues"
Last updateA server reboot became necessary which resulted in some additional downtime. At this time all websites are back up and running.
Starting 10 minutes ago the server anchorhost2 (104.198.197.214) is experiencing some stability issues. Working to stabilize now. I expect it to be fully resolved within the hour.
Report: "Server Reboots"
Last updateOther then the unscheduled restarts earlier today, both servers have been stable.
The reboots were done by WP Engine engineers. They have identify a known issue affecting some of their servers running PHP7 and are working on a fix.
Beginning 10 minutes two of my WP Engine servers were rebooted, anchorhost1 and anchorhost2. Website on these servers were offline briefly however appear to be coming back online within 3 to 5 minutes. Checking with WP Engine for more info.
Report: "Slow performance on 'anchorhost2'"
Last updateHardware upgrade completed. Everything appears to be running smooth now.
On further investigation it appears the root problem appears to be related to the recent migration. To resolve as quick as possible a hardware upgraded will be installed on anchorhost2. During the update a brief outages of 10 to 15 minutes will occur.
There is a large amount of traffic incoming from a malicious source. WP Engine is putting in a few manual blocks to mitigate.
Starting 30 minutes anchorhost2 is running slower then it should be resulting in sporadic 504 errors. WP Engine is investigating that particular server. All other servers are healthy at this time.
Report: "Slow performance on 'anchorhost1'"
Last updateThis incident has been resolved.
Server upgrades has been completed. Websites should be coming back online now. Will continue to monitor for performance and stability with the additional resources.
Server upgrades are currently being processed by WP Engine. Will update once they have been completed.
WP Engine is working to add hardware resources to 'anchorhost1' in order to stabilize. While hardware is being added there will be a 10-15 minutes outage. They will be starting that here shortly.
I'm seeing slower performance on 'anchorhost1' starting within the hour. Working with WP Engine to identify and to stabilize. The server was stable after migration was completed late last night. The other servers are unaffected.
Report: "Instability issues with 'anchorhost1'"
Last updateThe instability is better however the server is continuing to run slower then normal affecting various backend pages. Will continue to work on this server until the backend performance improves.
One of my dedicated WP Engine servers is experiencing instability starting around 30 minutes ago. WP Engine is taking at look and attempting to stabilize. Will update once I have more information. All other servers are healthy.
Report: "Server Hardware Failure"
Last updateThe server has been restored. The affected websites total downtime was 45 minutes. Everything is now up and running.
Due to a hareware failure on anchorhost4 (104.155.154.175) a number of website are offline. The affected server is currently being repaired and is expected to be restored within the hour.
Report: "Slow load and sporadic outages on anchorhost1"
Last updateHigh load appears is due to high traffic. Examining traffic logs to identify which sites need to move to server with more capacity and will plan accordingly.
Starting 1 hour ago anchorhost1, one of my dedicated WP Engine server, begun giving sporadic outages due to high load. Working with WP Engine right now to stabilize this server. All other servers are healthy.
Report: "Amazon Outages"
Last updateAt this time everything appears to be running smooth.
Due to Amazon outages affecting a large portion of the web right now: https://status.aws.amazon.com/, some websites are experiencing issues intermittent issues.
Report: "Individual server timeouts"
Last updateThis incident has been resolved.
WP Engine engineers believe they have stabilized the server. Everything has been running smooth since 9:40pm. Will continue to monitor.
Server anchorhost4 (104.130.140.77) is currently experiencing random 502/504 timeout errors beginning around 8:30pm. Will update as more info is available.
Report: "Individual server outage"
Last updateAfter 8 minutes of outage, connection has been restored. Will post more info as it becomes available.
Server anchorhost5 (130.211.199.78) is currently experiencing an outages beginning 5 minutes ago which is affect all websites on that server.
Report: "Partial Outages"
Last updateAll sites are now loading successfully.
A few minutes ago I got notification that websites with Kinsta are experience an outage. All sites with WP Engine are unaffected. Will update as more details are available.
Report: "Slow loads and brief 504 timeouts"
Last updateSome of the intended installs have been moved. Will continue to monitor over the next 24hr to evaluate stability.
Beginning this afternoon the server 'anchorhost4' (104.130.140.77) started going slower then normal resulting in a few outages lasting only a few minutes. While very sporadic, these brief outages have increase as the evening progressed. I'm working to stabilize the server by offloading a few of the highest usage installs over to a new server. I expect those to be done within the next 2hrs and for performance to be restored there after.
Report: "Brief outages"
Last updateMigration completed. All affected sites have been updated with new IP or those responsible have been notified.
A batch migration has just been started. I estimate it to be completed within the next 3hrs.
The WP Engine onboarding team is planning to move a batch of installs from 'anchorhost2' to 'anchorhost5'. As soon as these are completed I expect load on 'anchorhost2' to return to normal levels. I don't currently have an ETA on that. Will update once that has started.
Internal migrations put extra load on 'anchorhost4' (104.130.140.77) which caused a brief outage at 10:07 AM. Everything came back online a few minute thereafter. In order prevent future internal migration issues, I'll be directing all internal migrations to a new server.
In continuation to yesterday brief outages, 'anchorhost2' (192.237.244.230) is experiencing some instability relating to overcapacity. Working with WP Engine to stabilize this server while continuing manual migrations to reduce load.
Report: "Brief outages"
Last updateThings are continuing to run a little slower then it should on the 'anchorhost2' server however have been running stable for the last 3 hours. Will continue with some internal migrations until the speed is fully restored.
I received a few down notices over night and also again within the last hour. It appears that 'anchorhost2' (192.237.244.230) is experiencing some instability relating to overcapacity. I'm working with WP Engine to alleviate the extra load. Also I plan to move a few installs off of this box to allow for some planned margin.