Post by Spartan198 on Feb 11, 2011 0:35:52 GMT -5
From Jcink
It is with great regret today that I inform everyone that the system had a hardware failure.
During our routine maintenance today, something went horribly wrong, and it turned out to be a faulty drive. It was not expected. We quickly backed up all of the content as it was, and then shut down the server to begin the process of swapping out the bad drives.
Many attempts to use yesterday's backup failed. Every time we tried as well it took another few hours of time to see if it would work - moving around and recopying all of the boards takes a lot of time.
As a result we had to dig out the latest backup that worked from December. I did not want to leave the service offline any further.
I must stress that all hope is not lost with getting back the recent content. In fact it is far from it.
The backup we have from yesterday, from what I can tell, is mostly good. We had it up for a short time... but had to take it down after everything went "read-only" because of errors that remained from the old disk. This makes it bad for use in production but it can be picked-off of without a problem.
In the next few hours I'm going to try to have a tool out that can recover boards. Admins will be able to run this tool to recover their board back to the way it was, and I'm also going to try to work out rolling restorations. I know there's a lot of boards that currently don't exist which should be easy to fix. Unfortunately this is the best that can be done until I find a better way.
I know that this situation is bad and many are going to be angry, but I ask that you give us some faith; we are still comitted to restoring service the way it was. I can only apologize a thousand times for what has happened and hope you'll stick with us.
During our routine maintenance today, something went horribly wrong, and it turned out to be a faulty drive. It was not expected. We quickly backed up all of the content as it was, and then shut down the server to begin the process of swapping out the bad drives.
Many attempts to use yesterday's backup failed. Every time we tried as well it took another few hours of time to see if it would work - moving around and recopying all of the boards takes a lot of time.
As a result we had to dig out the latest backup that worked from December. I did not want to leave the service offline any further.
I must stress that all hope is not lost with getting back the recent content. In fact it is far from it.
The backup we have from yesterday, from what I can tell, is mostly good. We had it up for a short time... but had to take it down after everything went "read-only" because of errors that remained from the old disk. This makes it bad for use in production but it can be picked-off of without a problem.
In the next few hours I'm going to try to have a tool out that can recover boards. Admins will be able to run this tool to recover their board back to the way it was, and I'm also going to try to work out rolling restorations. I know there's a lot of boards that currently don't exist which should be easy to fix. Unfortunately this is the best that can be done until I find a better way.
I know that this situation is bad and many are going to be angry, but I ask that you give us some faith; we are still comitted to restoring service the way it was. I can only apologize a thousand times for what has happened and hope you'll stick with us.