ADMIN: ilXor.com database troubles

Message Bookmarked

Bookmark Removed

The server went down at 10:30 GMT last night. This is what happened, and what's being done about it:

What happened?
Due to ilXor's popularity, an access log grew large enough to fill the system hard disk. The server's owner has been traveling for the last few weeks and so was unable to spot this. When the disk was nearly full, the server attempted to insert a message into the database, suddenly ran out of space, and the database became corrupted.

Has anything been lost forever?
No, although most recent messages are no longer online, they will all be retrieved when the message database has been repaired.

What's online now?
Over the last few days an optimised copy of the database was being created to accelerate certain functions, but due to the size of the message database, the server had only reached 9th September this year. This copy of the database is now in service.

Why weren't the thread titles lost?
The thread titles were stored in a separate database which (along with the Users and other databases) were undamaged.

When will the original messages be restored?
As soon as the server finishes repairing them, which could be today, tomorrow, or maybe not until some time next week. It's impossible to estimate.

Why do many threads now start with "The body of this question..."?
The server requires these placeholders in order to function normally. As it says, the original question and replies will be restored as soon as possible.

Any other questions?

― Graham (graham), Friday, 27 December 2002 04:23 (twenty-three years ago)

:`(

thanks for fixing it graham

― ron (ron), Friday, 27 December 2002 04:43 (twenty-three years ago)

If anyone has cached copies of threads on their computer, post them in the appropriate place (I'll hide the email addresses later). Get on it.

― Graham (graham), Friday, 27 December 2002 04:48 (twenty-three years ago)

nearly had a heart attack earlier...hats of for getting on the case fella!

oh, and happy festivities everyone.

― Charlie (Charlie), Friday, 27 December 2002 05:16 (twenty-three years ago)

alas i have nothing helpful here

― ron (ron), Friday, 27 December 2002 05:45 (twenty-three years ago)

Graham, where would we be without you?

― smee (smee), Friday, 27 December 2002 08:41 (twenty-three years ago)

outside in the fresh air probably

― duane, Friday, 27 December 2002 08:48 (twenty-three years ago)

Fresh air? Never heard of it...

― smee (smee), Friday, 27 December 2002 08:51 (twenty-three years ago)

this is all the hobbits fault

― Queen G (Queeng), Friday, 27 December 2002 08:56 (twenty-three years ago)

Which Hobbit? And why haven't you mentioned ass mucus yet?

― smee (smee), Friday, 27 December 2002 08:59 (twenty-three years ago)

queen g, can you aim me, ive forgotten yr name.

― anthony easton (anthony), Friday, 27 December 2002 09:10 (twenty-three years ago)

Three cheers for Graham! Hip hip hooray, hip hip . . . well you get the idea. Three always seemed like overkill in my book.

― That Girl (thatgirl), Friday, 27 December 2002 09:14 (twenty-three years ago)

(I did kind of fuck up actually that I hadn't mentioned, so thankyou, but anyway please don't cheer)

The unread database seems to be acting funny.

― Graham (graham), Friday, 27 December 2002 09:50 (twenty-three years ago)

Heh cheers anyway Graham. Oh those tantalising thread titles!

― Tom (Groke), Friday, 27 December 2002 10:35 (twenty-three years ago)

i'm glad everything's getting back to normal.
props to graham again!

― joan vich (joan vich), Friday, 27 December 2002 11:04 (twenty-three years ago)

Thanks for taking care of this stuff, Graham. (I hope the technical side of ILX doesn't cost you an excessive amount of time, but that's probably wishful thinking.)

― Rockist Scientist, Friday, 27 December 2002 14:01 (twenty-three years ago)

wow that was fixed quickly. i was afraid i'd be ilxor-less for a week.

― Maria (Maria), Friday, 27 December 2002 17:30 (twenty-three years ago)

Thst's cool .... BTW who runs ILXor now, on what? I know it used to be an ACS at Phil Greenspun's site. Now you seem to have moved it to something PHP based? What is it? How the hell does it cope with this much traffic?

― phil jones (interstar), Friday, 27 December 2002 18:45 (twenty-three years ago)

Andrew runs it from a rucksack containing a server and Graham or something.

― N. (nickdastoor), Friday, 27 December 2002 18:52 (twenty-three years ago)

I foudna 4GB access log on one of my servers the other day. Not that huge but big enough to cause problems on a smallish drive. 4GB of data noone would ever want/need to look at = delete!

― g (graysonlane), Friday, 27 December 2002 20:49 (twenty-three years ago)

somebody deposited 74 bicycles in my front garden? can you help?

― vincent tuquedenne, Friday, 27 December 2002 21:36 (twenty-three years ago)

Yes Graham, sort that out!

(Seriously, we are very lucky to have someone who can sort out unpredictable problems for us OVER CHRISTMAS! Thanks, Graham.)

― Martin Skidmore (Martin Skidmore), Friday, 27 December 2002 21:51 (twenty-three years ago)

No you're not, where's Andrew when you need him?

― Graham (graham), Friday, 27 December 2002 23:25 (twenty-three years ago)

Cheers, Graham. I doff my hat to you.

― weasel diesel (K1l14n), Saturday, 28 December 2002 00:21 (twenty-three years ago)

In other admin news:
Just before the server problems, I completely rewrote the New Answers page generator. It now caches the list of threads, but the counts for new answers, total and unread are completely up to date and recalculated every time (doing this takes very little effort by the server). This means new/updated threads don't appear/move to the top immediately, but the counts next to them will change. The list is updated fully every 2 minutes.

I've just made it near impossible for (identical) double posts. The general slowness of the server while it repairs the database has led to the first instance not yet being in the database when it checks for duplicates. The system can now tell when the first message is queued to be inserted, and won't check for duplicates until after it is.

― Graham (graham), Saturday, 28 December 2002 02:59 (twenty-three years ago)

Hey Graham, just tried posting a new message and got an error message that read "Problem inserting message: GET LOCK FAILED". Is this maybe related to the new double-posting checker?

― Sean Carruthers (SeanC), Saturday, 28 December 2002 14:05 (twenty-three years ago)

Yes, yes it is. Thanks for mentioning it. I've fixed it now.

― Graham (graham), Saturday, 28 December 2002 14:19 (twenty-three years ago)

hey, I was just wondering if all this server stuff was responsible for ILX working so so slowly, it's painful. Is there anything I can do in my settings to get it to work faster? Or do I just need more paintence? (sorry to be such a moaner)

― jel -- (jel), Saturday, 28 December 2002 17:12 (twenty-three years ago)

The server is v.busy rebuilding the old database. I don't think there's much you can do to speed it up.

― Graham (graham), Saturday, 28 December 2002 17:45 (twenty-three years ago)

A little jig?

― N. (nickdastoor), Saturday, 28 December 2002 17:53 (twenty-three years ago)

my browser has been showing visited links on ilxor recently, which it has never done before (in opera, that is)

graham, have you changed something that has made this start to happen? if so, a thousand blessings on you and yours

― ron (ron), Sunday, 29 December 2002 03:40 (twenty-three years ago)

Thanks, Graham.

― sundar subramanian (sundar), Sunday, 29 December 2002 05:04 (twenty-three years ago)

Hooray! Thank you. And thank you again because a problem I had since 12 hours ago has gone again before I even got round to doing a proper bugreport or moan or whatever it would've turned out as, heh.

― Rebecca (reb), Sunday, 29 December 2002 06:18 (twenty-three years ago)

That was me pissing you off Rebecca.

Everything's back. Apart from whatever people tried to post after the boards went down on Friday night, I don't think a single message has been lost.

― Graham (graham), Sunday, 29 December 2002 11:25 (twenty-three years ago)

You're a star, Graham. I shall buy you a drink if you'reon the crawl with us today.

― Martin Skidmore (Martin Skidmore), Sunday, 29 December 2002 13:09 (twenty-three years ago)

thanks for fixing it all graham, you rule.

― di smith (lucylurex), Monday, 30 December 2002 05:20 (twenty-three years ago)

Graham, good to hear that New Answers is now behaving acceptably. :)

The two stand-in sysadmins are doing a good job of making sure everything runs smoothly (they caught the log problem at about the same time as Graham, and acted decisively in fixing it), so you have them - in addition to Graham - to thank for the server's wellbeing over the next 7 months and during the past month.

― Andrew (enneff), Monday, 30 December 2002 11:31 (twenty-three years ago)

thanks to everyone who is giving their time for this!!

― maura (maura), Monday, 30 December 2002 15:41 (twenty-three years ago)

thank you stand in sysadmins

― ron (ron), Monday, 30 December 2002 15:46 (twenty-three years ago)

hurrah for Graham.

― DV (dirtyvicar), Monday, 30 December 2002 16:10 (twenty-three years ago)

I'm really sorry about the couple of long-ish outages and slowness since this, caused mainly by me own stupidity. I want to make the one tonight the last ever - I've solved a lot of the things that were unstable before.

The good news is it's now finished the work it was doing that made it so easy to knock over, and now a few things will be much much faster, such as searching by username, date, email address etc. I'm still working on speeding up keyword search, but it looks like there's nothing inherently slow about the way it works, there's something else making it go so slowly, so it should be fixable.

― Graham (graham), Friday, 3 January 2003 00:39 (twenty-three years ago)

Graham, you are an officer AND a gentleman. Keep up the exemplary work. (These kudos go out to anyone who's had a hand in getting these bitches up & running, of course. Feel free to switch the nouns to suit your own gender, if necessary.)

― David R. (popshots75`), Friday, 3 January 2003 01:16 (twenty-three years ago)

Yes, Graham's grrreat! I am sending him a present for all that he does here -- I was going to mail it today if not for the COCKFARMING BLIZZARD that hit the area.

― Nicole (Nicole), Friday, 3 January 2003 01:25 (twenty-three years ago)

Rah for Graham! He's a good soul he is. :-)

― Ned Raggett (Ned), Friday, 3 January 2003 15:55 (twenty-three years ago)

I haf just unleashed hell (or unending thread revivals. One of the two).

― Graham (graham), Sunday, 5 January 2003 04:10 (twenty-three years ago)

Why do I get the feeling a lot of folks will be bookmarking the bra and shaved pubes threads?

― Nicole (Nicole), Sunday, 5 January 2003 04:13 (twenty-three years ago)

haha that's cool

― ron (ron), Sunday, 5 January 2003 04:14 (twenty-three years ago)

(nicole may have missed the point)

― Graham (graham), Sunday, 5 January 2003 04:15 (twenty-three years ago)

I see now...I didn't even notice the refresh bit the first time I clicked on the link. D'oh and d'oh.

― Nicole (Nicole), Sunday, 5 January 2003 04:20 (twenty-three years ago)

they still will bookmark them when they fly by...

― ron (ron), Sunday, 5 January 2003 04:30 (twenty-three years ago)

You must be logged in to post. Please either login here, or if you are not registered, you may register here.