Sites Down

[anthony]

There was a major hardware failure that caused the servers to be down.

I wanted to give you a quick follow up. Power has been restored and we are working to get everything back online as fast as possible. Our datacenter space at CoreSite experienced what the engineers are explaining as a ground fault at the main power distribution panel. We also have additional WiredTree staff within the datacenter ready to ensure that each dedicated server, VPS and Hybrid are up and operational. We will be powering on servers soon to get everyone back up. Currently this looks to have affected about 20% of our datacenter space. Our datacenter space ([DLMURL]http://www.coresite.com/chicago-data-center.php[/DLMURL]) is protected by UPS and power generators in case of a power failure. Our understanding was that this failure at the main panel prevented those systems from doing their job. We expect to be hearing information from CoreSite and will be reporting back to you on this issue as we hear it from the engineers. We will also report an ETA as soon as we have more information.

Again, we are deeply sorry for the inconvenience and your patience and understanding while we sort out this issue is greatly appreciated.

 

[anthony]

No idea how long this silly game will occur... though obviously the host has a rare hardware issue that should hopefully be fixed ASAP.

 

[Death]

Hahaha..... I work in mysterious ways ...... Enjoying my Christmas spirit of messing with your heads! :p

 

[PerfectEmpire]

Man! I was in the middle of writing a 1,300 word post for my diary, and when I clicked "post reply" I nearly lost everything (thank goodness I wasn't using Internet Explorer)! I'm happy to see that the site is back up. Now I can paste my entry in and try again.

 

[Nicolette]

Wow, really lucky PerfectEmpire.

 

[AdamAnt]

I gotta get used to using Chrome or something else....

 

[anthony]

It is extremely rare that the sites would ever be down outside something I did myself... and this was a hardware failure.

They have posted an update to the issue:

This is an emergency notice to inform you that the power that feeds the rack that your server is in will be undergoing emergency maintenance starting at 9:00PM CDT tonight (Saturday, 12/11/2010). The issue is due to the ground fault power problem that CoreSite (WiredTree's datacenter provider) suffered this afternoon. CoreSite needs to shut down the circuits that power your servers and switch them over to new ones. This is a proactive replacement to ensure that your servers are being fed by stable power. They are pre-wriring the replacement circuits at this time and we will be issuing power down commands to your server starting at 9:00PM CDT and we expect downtime to minimal.

Switching the circuits takes about ~15min to do by CoreSite and as soon as we have a stable power environment, we will be powering on your server. The expected outage window will be .5 to 1hrs in length. While we do not expect your server to be down for that long, we are giving our staff and the CoreSite engineering team an ample downtime window to get this issue taken care of and get your server back online. All best practice measures will be taken to ensure downtime is minimal. Your patience and understanding while we work on this proactive emergency replacement is greatly appreciated.