Back to the Future
There was a catastrophic hard drive failure on the central database server very early this morning, Donncha was on the ball and switched all the traffic to a recent backup so most things would work while we investigated the hardware failure. This means that an old version of your site was shown for a few hours.
A few minutes ago we restored the up-to-date database and we’re currently syncing it to the backup to get back any posts you might have made during the semi-downtime. Even though we were able to recover everything, we’re looking at ways to make things even more redundant, so if this ever happens again the problems will be measure in seconds or minutes. Like I said in the last post, all of the new hardware hasn’t been brought fully online yet.
Technical details: The ext3 filesystem was badly corrupted but thanks to journalling there was no lost data.
Update: All posts have now been restored.
Thanks for all of your hard work this morning Matt and Donncha. Several hours worth of work over the weekend lost induced a mild state of panic this a.m. but all appears to be back to normal. Thanks again.
*snicker*
October 31, 2005 at 9:22 am
Mr. Matt, will the referrer button be restored as well?
Flickrdoodle
October 31, 2005 at 9:22 am
with the exception of the referer button…
*snicker*
October 31, 2005 at 9:23 am
Thanks for being quick and not losing data.
mike-burns
October 31, 2005 at 9:25 am
thank you. i got my lost posts back!!
Ric
October 31, 2005 at 9:38 am
[...] Asi es como anuncian en su blog, la repación del pequeño problema que han tenido los servidores de wordpress.com que durante todo el finde semana, al menos yo, he ido acarreando al servicio del blog. [...]
/home/darkomen » Regreso al Futuro
October 31, 2005 at 9:42 am
Matt, a big thanks to you for restoring back the posts…..and a big hug from me…..one post were written with a lot of time(though may seem average to some)
Rahul M
October 31, 2005 at 9:47 am
really really great job. thanx
lukasz
October 31, 2005 at 9:47 am
gr8 job
lukasz
October 31, 2005 at 9:48 am
[...] There was a major hardware failure this morning, however now everything is back to normal. See this post for more information. [...]
alicekey’s Blog » 囧…WordPress.com
October 31, 2005 at 9:54 am
You guys did a great job, and even took the time out to answer a few of my emails with some details. I appreciate that level of customer service, it came as a complete surprise on a free service like this.
You have my loyalty.
The Scribe
October 31, 2005 at 9:54 am
Yeah, well looks like something is still wrong elsewhere. I had registered for the account ryanl earlier, written a bit and now it redirects to ben_b. After this happened I was able to register the same user account again but it still takes me to the alternate control panel and doesn’t allow me to have any options whatsoever other than the dashboard.
ryanl
October 31, 2005 at 10:00 am
My latest post is still missing. Hope you can fix it.
http://gaby.wordpress.com/2005/10/31/freelance-your-blog-skills/
It was a really good post and a lot of work. In stead of making a backup I’ve posted links to it. ahhhhhhh How unprofessional of me.
Please make it come back.
gaby
October 31, 2005 at 10:05 am
[...] On my WordPress.com dashboard enclosed in thin red border I saw this announcement in bold: “There was a major hardware failure this morning, however now everything is back to normal. See this post for more information.” [...]
devnullnikita {memento mori et vita} » Blog Archive » WordPress.com temporarily down
October 31, 2005 at 10:13 am
Back to the Future
Suddenly posting didn’t work anymore, and the newest post was lost. Now I know why: “There was a catastrophic hard drive failure on the central database server very early this morning, Donncha was on the ball and switched all the traffic t…
roam,maol
October 31, 2005 at 10:19 am
[...] Suddenly posting didn’t work anymore, and the newest post was lost. Now I know why: There was a catastrophic hard drive failure on the central database server very early this morning, Donncha was on the ball and switched all the traffic to a recent backup so most things would work while we investigated the hardware failure. This means that an old version of your site was shown for a few hours. [...]
roam,maol » Back to the Future
October 31, 2005 at 10:20 am
[...] [edit]: oddio non che qui si stia meglio ora come ora… [...]
Flannel Blog » E avevo ragione…
October 31, 2005 at 10:25 am
[...] If you’re using an RSS aggregator to view this blog (like Bloglines, FeedLounge, NewsGator or any one of the hundreds of other options), then I apologize for the re-publishing frenzy of the past couple days. I tagged some of the more recent posts, and then WordPress hiccuped. Won’t happen again. [...]
In The Crowd » Too much of a good thing?
October 31, 2005 at 10:26 am
Wow! I am liking WordPress already. Great service. Thank you!
bestoftrendy
October 31, 2005 at 10:32 am
Thanks for restoring service. I really appreciate your service.
Pal
October 31, 2005 at 10:44 am
gotta love ext3 journaling.
Aaron Brazell
October 31, 2005 at 10:47 am
Well done over there; no lost data and only a few hours!! Good work…
mikep
October 31, 2005 at 10:48 am
Wonderful on everything being brought back up but I can’t get the “Referrer” link off the “Manage” section….
whatnottocrochet
October 31, 2005 at 10:55 am
[...] [...]
Flickrdoodle » Wordpress Referrer Link Missing
October 31, 2005 at 11:19 am
Thanks for sorting the problems out, good communication via the dashboard.
mspoke
October 31, 2005 at 11:43 am
Your work is appreciated.
On the other hand, I was already thinking “major re-writes” and stuff… and then it all came back… arrrfff.
Your work, as I said, *is* appreciated. Thanks all.
elpincha
October 31, 2005 at 11:47 am
Gaby, your post was made onto the backup hardware, and we just restored all of those. Your link works now and your post is back safe and sound.
Matt
October 31, 2005 at 12:03 pm
yay, my post is magically recovered.
gaby
October 31, 2005 at 12:03 pm
no, mine is combined with concious.wordpress.com
I’m forestglen.wordpress.com
my dashboard is frozen solid
I can see two of my posts.
my old password works
and computers are wonderful
and computers are not as nice as you guys.
no hurry, I’ll wait a while.
but since I had to recreate forestglen.wordpress.com
I want to keep it, thank you. I used 2 invites so far
to get this blog… worth it, too!
thank you.
forestglen
October 31, 2005 at 12:34 pm
[...] To those who left messages about my ‘now you see some, now you don’t’ posts WordPress.com migrated to a new datacenter over the weekend. As someone who has had the misfortune of doing so some settling in issues are always to be expected. So no it was not some Halloween stunt, nor a censorship thing (there goes that conspiracy theory), or me deleting posts. Rest assured I do not delete posts unless they are duplicates which I just did. Given I write each post directly into this weblog (without prep or backup) each one is written in realtime. Thanks to Donncha for sorting everything out and Matt for keeping me in the loop. [...]
Deaglan on WordPress » Appearing, disappearing and reappearing posts
October 31, 2005 at 12:47 pm
Thanks for the speedy work to restore service!!
samneeds
October 31, 2005 at 12:50 pm
Define “normal”
jonathanwsmith
October 31, 2005 at 1:06 pm
You’re really grat. Thx
annabialkowska
October 31, 2005 at 1:18 pm
Wow you guys really deserve a medal for this whole wordpress thing. If there’s anyway I can help out, just let me know.
helpdesk
October 31, 2005 at 1:54 pm
I’m seeing a post which now has somebody else’s name as the author even though the post was made by me, I can’t seem to get to the Feedback button as it disappears (using Firefox on Mac OS X). Any ideas why this has happened?
mspoke
October 31, 2005 at 1:57 pm
Great job on fixing this issue as quickly as you guys did. And thank you for everything you do.
Mark
October 31, 2005 at 2:02 pm
[...] UPDATE: They’re just back! All posts have been restored, now. Thanks to WordPress team! [...]
budigitalized » Blog Archive » Recovering The Entries: Check your Feed in the RSS Reader!
October 31, 2005 at 4:14 pm
aah ya me habia asustad. Que el Amo del Merol los bendiga
chavis
October 31, 2005 at 4:21 pm
great job.. last night, about 9 hours ago… some blogs at whatevertheuser.wordpress.com works but mine ain’t working? why is that so? is it because i used flock? are there any differences with blogs made from invites and the ones made with flock?
aissa
October 31, 2005 at 4:53 pm
[...] Back to the future by Matt [...]
CJ@Wordpress.com » Thanks god, it’s back
October 31, 2005 at 5:25 pm
If anyone is having the problem where a post has the wrong user please use the contact form and we’ll fix that up.
Matt
October 31, 2005 at 5:55 pm
[...] 早上以?看到WordPress的Dashboard中有一?红红的notice，早上?务器?到了一个严?的硬件错误，还?他们把所有的数?都备份了。 [...]
We Paranoid Love Life » Wordpress crashed this morning
October 31, 2005 at 6:12 pm
[...] From “Matt on WordPress” blog: [...]
Global Noise Online » ADMIN: posts restored
October 31, 2005 at 6:23 pm
[...] buat informasi lebih jelasnya lo bisa baca di : http://matt.wordpress.com/2005/10/31/back-to-the-future/ [...]
[ I.Do.Blog ] » Blog Archive » WP ngeCrash !
October 31, 2005 at 9:18 pm
[...] UPDATE: MacManX alerted me, in the comments of this post, to the fact that Matt has put up a post about this issue. In the post, Matt explains what happened, how the WordPress.com team responded and that fact that no data was lost: Donncha was on the ball and switched all the traffic to a recent backup so most things would work while we investigated the hardware failure. This means that an old version of your site was shown for a few hours. [...]
Wordpress.com “major drive failure” « Tom Raftery’s I.T. views
November 1, 2005 at 12:56 am
[...] WordPress.com is too busy with it’s hardware stuff to take care of the task of adding more useful features. Have not gotten around to shifting to the new place yet. Let’s see. [...]
Miles to go… » Technology News…
November 7, 2005 at 2:15 am