Sorry about that

[number8]

Yesterday the threads weren't loading for some unknown reason so I decided to just upgrade to the latest version of VB to see if that'll fix it. The upgrade went haywire and threw up all those errors instead, so I closed the forum while I fixed it. I think everything's fine now...?

[D_Davis]

Thanks for your work. Seems to be working.

[Raiders]

Indeed. All looks fine. vB updates are wonderful, eh? That's why we ran 3.6 for five years.

[Mara]

Thanks! Glad to have it all back.

[Spun Lepton]

Thanks, #8. Appreciate all your work.

[megladon8]

Thank you 8!

Is it sad that I basically sat with my finger on the "refresh" key for 2 days?

[Kurosawa Fan]

Thanks 8!

[dreamdead]

Thanks for the speedy fix, n8! Glad to have the site back!

[Dukefrukem]

Thanks 8.

If Raiders was still running the site he would claim it still has 99% up time or some shit.

I'm 150% bustin your balls :D

[Dukefrukem]

I'd also like to point out that the upgrade fixed all 8 problems documented here:
http://matchcut.artboiled.com/showthread.php?22-Bugs-Suggestions-Issues&p=469405&viewfull=1#post469405

[number8]

Really? Sweet.

I mean, I totally knew it would and thought it was time to address those bugs so I did it.

[MadMan]

I wondered why the TapeTalk ap on my smart phone kept sending me error messages every time I tried to check match-cut. Well I'm glad the site is back because I would then have to spend more time on RT and the Corrie. Yikes.

[number8]

Got exploited. Someone put malicious code into the forum. All fixed now. Sorry it took so long. It was 20 minutes to fix, but took all day to find out what was wrong.

In the future, though, I beg all of you to please refrain from rudely telling me outside the forum to go and fix it when things go wrong. Trust me, I'm on MC all the time, day and night. I'm usually the first to know when it's down, and I don't go, "Meh, I'll deal with it later" when it does.

Getting messages telling me to hurry during my birthday dinner last night sucked, even if it was said jokingly. Because it did make me briefly think that I should wrap up my night and go home to deal with it. Just having that cross my mind felt like shit, because it made the rest of the night seem like I was rebelling against the requests and indignantly refusing to take care of my responsibilities.

[D_Davis]

Fuck that shit.

I would have left it broken for a few days out of spite.

Might actually be nice to shut down the site for the holidays so no one has to worry about it.

[Dead & Messed Up]

Got exploited. Someone put malicious code into the forum. All fixed now. Sorry it took so long. It was 20 minutes to fix, but took all day to find out what was wrong.

In the future, though, I beg all of you to please refrain from rudely telling me outside the forum to go and fix it when things go wrong. Trust me, I'm on MC all the time, day and night. I'm usually the first to know when it's down, and I don't go, "Meh, I'll deal with it later" when it does.

Getting messages telling me to hurry during my birthday dinner last night sucked, even if it was said jokingly. Because it did make me briefly think that I should wrap up my night and go home to deal with it. Just having that cross my mind felt like shit, because it made the rest of the night seem like I was rebelling against the requests and indignantly refusing to take care of my responsibilities.

That's pretty bullshit. Thanks for fixing it, and sorry to hear that it got in the way of festivities.

[Dukefrukem]

Fuck that shit.

I would have left it broken for a few days out of spite.

Might actually be nice to shut down the site for the holidays so no one has to worry about it.

You would do that.

I'll openly admit I sent the tweet: Hey @artboiled Happy Birthday.. now fucking fix @MatchCut1 <3 Duke

Sorry it bummed you out.

[Raiders]

I'll openly admit I sent the email:

"You better not be enjoying a fucking birthday celebration while I sit here in the dark, cold and alone and staring at a blank space where the forums should be while everyone else I know is out doing things like work, eating, shopping, hanging out, living life, etc. and I am forced to be relegated to this pathetic existence. FIX IT NOW OR I WILL BASH YOU ON MYSPACE."

Yeah, sorry about that. It does seem a little callous in retrospect.

[Dukefrukem]

MYSPACE still exists?

[Irish]

First, Happy Birthday.

Second, I'll speak only for myself here:

The down times are annoying. Especially as there have been more & more of them in the last year. People come here every day & they get used to the site being up & expect that. Hence, I assume, those messages you got.

Everybody has day jobs & lives & those take priority over this site. I don't think anyone has a problem with that.

But given the age of smartphones & twitter apps, why did it take 10 hours for any admin or mod to even acknowledge there was a problem? (For my money, everyone one the mod team should have access to the @matchcut1 twitter account.)

This site seems so fragile now that every time it goes down I tend to wonder if this is "it," that the forums are done & won't be coming back.

It takes 10 seconds to post "Shit, site is down. At my job, will fix when I get home" to twitter. It might take a few minutes to set Apache to redirect all requests to a page with a short "working on it" message.

That kind of acknowledgement goes a long way to allaying user anxiety, and staving off the kind of messages you got at your dinner. If you don't do that, it looks to the outside like you're either ignoring the problem or that you don't care.

Does most of this suck? Yeah. But that's the admin job. Admin jobs always suck.

2 cents.

[Skitch]

I'm more irritated by the lack of pepper talk on twitter than the outage. Thanks for the work 8!

[Spinal]

Seriously? A free site run by volunteers goes down for a day and we've got to give someone crap for not notifying us immediately that they know about the problem? On their birthday? I do not get it.

[eternity]

We appreciate the heck out of what you do for us, 8. Compared to the response Rotten Tomatoes would have to its forum's technical hiccups, MC could be down for weeks and you'd still be doing a hell of a job. A day or two every once in a blue moon is what, 99.9% uptime? Can't complain.

[Gizmo]

I always assume the issue is with the host server and not the site itself, it'll be back eventually. *shrug*

[Dead & Messed Up]

This site is a privilege, not an entitlement. The people who run it owe me nothing.

[Irish]

Seriously? A free site run by volunteers goes down for a day and we've got to give someone crap for not notifying us immediately that they know about the problem? On their birthday? I do not get it.

You can kill the messenger if you like.

If number8 complains that the user messages upset him, I only suggest a simple solution: Notify users early and often. If one does this, more often than not, negative response will disappear.

Is this fair or rational? Dunno. But it's the nature of the beast. He might not have known what he was getting into when he signed up to be admin, but it is what it is.

Also, I'd argue this site stop being "free" when that Facebook button went up & multiple ads appeared. *shrug*

[Spinal]

You can kill the messenger if you like.

If number8 complains that the user messages upset him, I only suggest a simple solution: Notify users early and often. If one does this, more often than not, negative response will disappear.

Is this fair or rational? Dunno. But it's the nature of the beast. He might not have known what he was getting into when he signed up to be admin, but it is what it is.

Also, I'd argue this site stop being "free" when that Facebook button went up & multiple ads appeared. *shrug*

Nothing about the way number8 has handled the task indicates to me that he is anything less than capable and responsive. Indeed, I'm thankful that he does it at all. Because, as this minor incident shows, it's a pretty thankless job.

Until you said something, I had forgotten that there were even ads on this site. It would be silly to make the argument that the site stopped being free because someone has clearly taken the least obtrusive action possible to help keep us from having to pay money.

Again, I don't understand the tone. Talk about looking a gift horse in the mouth.

[Irish]

Nothing about the way number8 has handled the task indicates to me that he is anything less than capable and responsive. Indeed, I'm thankful that he does it at all. Because, as this minor incident shows, it's a pretty thankless job.

Until you said something, I had forgotten that there were even ads on this site. It would be silly to make the argument that the site stopped being free because someone has clearly taken the least obtrusive action possible to help keep us from having to pay money.

Again, I don't understand the tone. Talk about looking a gift horse in the mouth.

What tone? All I am suggesting is better communication. Why is that idea so offensive to you?

It doesn't have to be 8. This site has an over abundance of admins and mods. Any one of whom could toss out a heads up.

Like it not, when one takes on the responsibility of a site like this, or any service, there's a certain implicit social contract. It might be perceived as shrill and awkward for a guy like me to point that out. But ignoring the fact doesn't make it go away.

Also: You might have forgotten this site has ads, but Google, Facebook, and their associated networks haven't and never do. That kinda goes to the point.

[Spinal]

What tone?


He might not have known what he was getting into when he signed up to be admin, but it is what it is.



Also, I'd argue this site stop being "free" when that Facebook button went up & multiple ads appeared. *shrug*

This tone.

[Irish]

This tone.

Say again? I'm a dummy, Spinal, but I'm not seeing a "tone." I'm only trying to acknowledge, repeatedly, that, yes, admin jobs suck and always will. They are thankless in multiple ways. Sometimes, people don't know that when they sign up for them & later find themselves stuck. And then burnt out.

But if you want people to respect that job, for whatever it is, you've got to acknowledge it's a two way street.

The ad thing is a pet peeve. When I suggested a whole bunch of changes to the site way back when, people misread what I wrote & a few people became hypercritical and nasty because they thought I was advocating advertising.

Then, curiously, ads went up on the site. Those same people demurred or said nothing. Now here's you, defending those ads. Frankly, that reversal leaves me a bit mystified.

But you can't claim that the site is "free," and beyond that I'd argue that "free" doesn't imply some kind of blanket virtue.

[Ezee E]

I think there's been a total of four or five outages in Match Cut since number8 and I have taken over. None of which lasted more then 72 hours, and all have been addressed within a day on twitter. Some quicker then others. I will not put Match Cut over my job, even if it means taking ten seconds out of it. I addressed it the first time this week, the second time, I completely spaced it out when I received an e-mail from vB.

I'm not sure of this "social contract" that you discussed Irish, but it seems that only you and Duke are the upset members here about this, and even Duke isn't that upset, he just wants to be able to post. While the site has ads and facebook connection, they both require $0 from you, and are also optional to have on.

So really, I don't feel any compassion here Irish.

[Dukefrukem]

Well beyond the ad argument, we all donated to the site because we love it and we love coming here. So yeh, it's not that free.

But to Irish's point, I think 8 now understands why Raiders didn't want to run the site anymore.

[Dead & Messed Up]

You can kill the messenger if you like.

This only works as a phrase if you're delivering a message from someone else, which you've said you aren't ("I'll only speak for myself here").

[Irish]

This only works as a phrase if you're delivering a message from someone else, which you've said you aren't ("I'll only speak for myself here").

Well, think about that for a second. Do you really want to parse semantics in English euphemisms with me while I'm in a peevish, argumentative mood?

;)

[EyesWideOpen]

But to Irish's point, I think 8 now understands why Raiders didn't want to run the site anymore.

Because of people like Irish.

[Raiders]

Because of people like Irish.

To be fair, he's only playing devil's advocate here, which as I am sure he will admit is something he enjoys quite a bit. And that's not why. The site would have been down for days with the way I felt about doing it a year ago. I didn't want to have to think about it anymore. It was a "me thing."

But seriously guys, it is a free site and you are entitled to nothing. "Social contract" is a nice way of saying you decided you were owed something.

[Spinal]

But you can't claim that the site is "free," and beyond that I'd argue that "free" doesn't imply some kind of blanket virtue.

Do you notice that when you use the word, you put quotation marks around it? I am not. I am not paying to use the site. This is a fact. It is not a claim.

[Irish]

Do you notice that when you use the word, you put quotation marks around it? I am not. I am not paying to use the site. This is a fact. It is not a claim.

The quotes are there because "free" doesn't mean the same as "no cost." The fact that you, and others, don't understand this is troubling but not surprising. (Because outside tech & advertising, very few people do).

To Duke's point, people donated ~$400 to keep this site running. So: Not "free" in any context. And not a gift horse. More of a paid ride.

Disclaimer: I did not donate. This might make what I'm saying shrill but it doesn't make me wrong.

[Raiders]

Wikipedia is a free service. I have also donated to Wikipedia. This is not a contradiction. There is no compulsory payment to enjoy this site.

[Skitch]

My biggest complaint when the site goes down is my mild panic that it won't come back or I've been banned...or that everyone moved the site and didn't tell me so they wouldn't have to ban my annoying ass. :D

Seriously don't go anywhere, I love you guys.

[Irish]

There is no compulsory payment to enjoy this site.

Wikipedia also doesn't have ads or Facebook Connect. That goes to what I was saying how "free" doesn't mean "no cost." (As an aside, I also don't understand why "free" is some kind of virtue that inures anyone from criticism or suggestion).

Also: What's the collective excuse to the people that donated hard cash?

[Lazlo]

Wikipedia also doesn't have ads or Facebook Connect. That goes to what I was saying how "free" doesn't mean "no cost." (As an aside, I also don't understand why "free" is some kind of virtue that inures anyone from criticism or suggestion).

Also: What's the collective excuse to the people that donated hard cash?

It's not a business where service is guaranteed. We paid because we saw it as a way to help keep a site we enjoy going, not because we were purchasing a product or a service. It's more like a social club where there are optional dues. No one freaks out if a meeting gets unexpectedly cancelled once in a while. 8 and EeezE aren't employees of the site or its users. They're nice enough to help keep the thing running. It's a hobby, not a burdensome responsibility. It's pretty shitty to give them shit.

[Dukefrukem]

My biggest complaint when the site goes down is my mild panic that it won't come back

This, which is exactly why Irish is saying it would be nice if there was some kind of announcement during the downtime. A tweet is not difficult. I see Eric and Arya do it all day.

[Raiders]

Wikipedia also doesn't have ads or Facebook Connect. That goes to what I was saying how "free" doesn't mean "no cost." (As an aside, I also don't understand why "free" is some kind of virtue that inures anyone from criticism or suggestion).

You can turn both off. Not to mention you are seriously twisting the meaning of "cost."


Also: What's the collective excuse to the people that donated hard cash?

No excuse. This site is volunteerism. If you can't deal with the down-time reasonably, find a different site. People donated to help the site carry-on. They were not given any guarantee of up-time or have any reason to expect people to drop what they are doing on their birthday to fix this. 98% of the people here get this.

[Lazlo]

This, which is exactly why Irish is saying it would be nice if there was some kind of announcement during the downtime. A tweet is not difficult. I see Eric and Arya do it all day.

Have a little faith that they're good dudes and will sort it out. Life goes on.

[Irish]

It's not a business where service is guaranteed. We paid because we saw it as a way to help keep a site we enjoy going, not because we were purchasing a product or a service. It's more like a social club where there are optional dues. No one freaks out if a meeting gets unexpectedly cancelled once in a while. 8 and EeezE aren't employees of the site or its users. They're nice enough to help keep the thing running. It's a hobby, not a burdensome responsibility. It's pretty shitty to give them shit.

Pretty shitty, eh? Curious reaction.

Let's pretend you, me, and Skitch are friends. Every week the three of us go to Sunday brunch, and each week a different guy pays the entire tab. This week, you and I had to cancel, but nobody told Skitch (which is a shame, because this week he was getting a "free" lunch). Skitch ends up sitting alone in a restaurant wondering what the hell. Sorry, Skitch!

Now, if he called you later and asked where you were & why we neglected to tell him lunch was off, would your response be anywhere near the realm of "Don't give me shit"?

Why do you think what I'm proposing -- more transparency, better communication -- is bad for anyone here?

[Irish]

You can turn both off. Not to mention you are seriously twisting the meaning of "cost."

The bulk of the Internet economy is built right on top of that twist. Google, Facebook, and Twitter make all their revenue from it (curiously, all "free" services).

More importantly: You can't shut it off easily on the desktop & you can't shut it off at all on mobile devices.


They were not given any guarantee of up-time or have any reason to expect people to drop what they are doing on their birthday to fix this. 98% of the people here get this.

No where in any of my posts about this did I express the idea that 8 needed to drop what he was doing and offer an immediate fix, or did I expect any guarantee of uptime.

All I asked was, "Why did it take so long for anyone to acknowledge the problem?" And suggest that if you want to curb negative responses, then inform your users early and often.

That's it. That was the whole message.

[Lazlo]

Pretty shitty, eh? Curious reaction.

Let's pretend you, me, and Skitch are friends. Every week the three of us go to Sunday brunch, and each week a different guy pays the entire tab. This week, you and I had to cancel, but nobody told Skitch (which is a shame, because this week he was getting a "free" lunch). Skitch ends up sitting alone in a restaurant wondering what the hell. Sorry, Skitch!

Now, if he called you later and asked where you were & why we neglected to tell him lunch was off, would your response be anywhere near the realm of "Don't give me shit"?

Why do you think what I'm proposing -- more transparency, better communication -- is bad for anyone here?

Well, seeing as no one had to go to any physical place and friends in that situation would be able to text or call each other, the analogy doesn't really line up.

It wouldn't be a bad thing for the Twitter account to be updated, but expecting these guys to drop everything and address it before all the other real responsibilities in their lives is a bit much. The site was down. That was unfortunate. But it came back. No one died or lost money, or got fired. Again, we're not running a business here. Of course I'd prefer the site to be up, which it is most of the time. But if it's down, I figure they're working on it and it will come back in due time. I'm thankful they want to spend their free time dealing with this stuff at all. Sorry they aren't meeting your performance standards, but in the end they don't owe you anything and they're doing an overall great job.

[Raiders]

The bulk of the Internet economy is built right on top of that twist. Google, Facebook, and Twitter make all their revenue from it (curiously, all "free" services).

They are for-profit corporations with shareholders. Not really comparable.

[Skitch]

With all due respect to Duke and Irish, Skitch would like it known that at no point did he imply or state any complaint about the outage or those who repair said outages. - Skitch's lawyers

[Irish]

Well, seeing as no one had to go to any physical place and friends in that situation would be able to text or call each other, the analogy doesn't really line up.

It's similar in that in both situations, there's an implied social contract.


expecting these guys to drop everything and address it before all the other real responsibilities in their lives is a bit much.

Seriously, where do you get this stuff from? I never even came close to implying anything like this.


They are for-profit corporations with shareholders. Not really comparable.

They all started out as freely offered side projects, and nobody's day job[1].

Ignoring your user base is never a good idea. Skitch's post is a case in point. At the very least, doing so tends to undermine confidence over time.

[1] To the literal minded: No, I'm not actually suggesting that Match Cut is in that league, wants to be, or ever could be.

[Dead & Messed Up]

Saying something like...


"Admins, thanks for getting everything up and running again - next time this happens, could you guys post a Tweet or something? :)"

might've been more effective than saying things like...


"The down times are annoying. Especially as there have been more & more of them in the last year. People come here every day & they get used to the site being up & expect that.

...why did it take 10 hours for any admin or mod to even acknowledge there was a problem?

This site seems so fragile now...

...it looks to the outside like you're either ignoring the problem or that you don't care.

Does most of this suck? Yeah. But that's the admin job."

[Spinal]

The quotes are there because "free" doesn't mean the same as "no cost."

Except it does. That is exactly what free means.

[Ezee E]

And I think we've come to a point where we all will agree to disagree. So unless there's anything productive left to be said....

http://images.sodahead.com/polls/003413605/102199617_merry_christmas_you_ filthy_animal_xlarge.jpeg

:lol:

[Spinal]

And I think we've come to a point where we all will agree to disagree. So unless there's anything productive left to be said....


I've got about 5 more posts to make on this topic. How much does that cost?

[MadMan]

I um feel bad number8 for not wishing you a happy birthday. If I did good, but I don't recall doing so. So yeah happy late birthday dude. And I'm glad the site came back. I wasn't worried that it would never come back, but if it did I would be sad and move on. Such is life.

[Thirdmango]

I thought 8 might still be on vacation so I just hoped the whole system wasn't down and waited.

[Winston*]

Pretty shitty, eh? Curious reaction.

Let's pretend you, me, and Skitch are friends. Every week the three of us go to Sunday brunch, and each week a different guy pays the entire tab. This week, you and I had to cancel, but nobody told Skitch (which is a shame, because this week he was getting a "free" lunch). Skitch ends up sitting alone in a restaurant wondering what the hell. Sorry, Skitch!

Now, if he called you later and asked where you were & why we neglected to tell him lunch was off, would your response be anywhere near the realm of "Don't give me shit"?

Why do you think what I'm proposing -- more transparency, better communication -- is bad for anyone here?

Wow, this is a shitty analogy.

[number8]

Well that happened again.

[Raiders]

I assumed you did it on purpose and found it hilarious.

[number8]

I don't even know what Facebook Connect is.

[D_Davis]

I assumed you did it on purpose and found it hilarious.

So did I. :)

[Mara]

Well that happened again.

Merry Christmas, buddy!

[number8]

Seriously, though, the forum keeps getting attacked and I'm still trying to figure out where the security breach is. The holes I'd been plugging were apparently not enough. If it goes down again, I preemptively apologize. I've never run vBulletin before this. It's a bulky system.

[Dukefrukem]

Did you check all your passwords on your server and make sure none of them are: "admin" or "password" :lol:

[Dead & Messed Up]

I assumed you did it on purpose and found it hilarious.

Same. "This must be just for Irish."

[number8]

In hindsight, "abc123" was a bad choice.

[Mara]

From now on, every time the forum goes down you may as well pretend that it was deliberate. Every time it goes up again, just post "You know what you did."

[Dukefrukem]

In hindsight, "abc123" was a bad choice.

Heh you being funny or is this the RC?

[D_Davis]

Seriously, though, the forum keeps getting attacked and I'm still trying to figure out where the security breach is. The holes I'd been plugging were apparently not enough. If it goes down again, I preemptively apologize. I've never run vBulletin before this. It's a bulky system.

You said holes you've been plugging....heh heheh heh heheh.

[megladon8]

Could the security issues be coming from the same person and/or place that was messing with me earlier in the year? Remember when my posts kept getting deleted?

[Dead & Messed Up]

I'm thinking this is all a false flag operation by Irish so he can fan the flames of user discord, usurp the admins, and become Match Cut Chancellor.

[Skitch]

I called Angie Jolie and Johnny Miller to help, but they just sent me a bunch of screensavers of floating numbers and a square monocle. Not helpful.

[Raiders]

The last time the forum was down, I tried to go in through a Google search to see what happened, and sure enough I was attacked by a trojan virus and redirected to some sex scenes countdown Youtube video. No issues as I have pretty intense security/firewalls.

[number8]

It is getting really fucking old.

[Skitch]

It is getting really fucking old.
:sad:

Sorry dude. Wish I could help.

[Irish]

Just out of curiousity, why are there a bunch of trash accounts with admin privileges?

- qaz001
- rizky
- Th3H4ck

^ It's this.

See:


Imperva said the compromised sites appear to have been hacked by one of two sets of exploit tools that have been released publicly online. The first was apparently used in a mass Website defacement campaign. A Google search for forums with the the rather conspicuously-named administrator account added in that attack (“Th3H4ck”) shows that many of the hack sites also are hosting malware. Among the sites apparently compromised is a support forum for the National Runaway Safeline and a site selling vBulletin add-ons.

http://krebsonsecurity.com/2013/10/thousands-of-sites-hacked-via-vbulletin-hole/

You got caught in a zero day exploit. Most likely because you left /install or /core/install scripts running on the live web server. Because this is a default install & completely unmodified, this site easily shows up in Google searches as a target.

The only thing you can do now is assume your entire system is compromised (this includes the blog at artboiled.com). Check your log files. See if any new files were added in public facing directories. Check to see if any of vBullentin's .php files were modified in the last month; it's likely that code was appended to a public facing file that is part of the system.

Roll back the database to a point before November 13th [scratch that -- make it October 13th, before this exploit became known]. Wipe everything. Reinstall the server from the ground up. Notify your webhost. Make sure you're only using SSH to access the server, and that account does not have root access. Move SSH so it runs off a different port than 22. Turn on FTP only when you need it. Shut off everything else.

Once vBulletin is back up and running, modify all the templates & remove any mention in the HTML output of JelSoft, vBulletin, and softwarw version numbers.

If you skip any of these steps, you will be dealing with this problem for a long time to come. It won't go away on its own. (You will also eventually get flagged by Google as a malware infected site; this includes your blog, Ary).

Users should assume bad people have all their private data. They should change their passwords and email addresses, especially if they use the same address, or password/address combo, anywhere else.

Edit: The shorter solution is to say fuck it, grab a copy of the database from a local store you trust & move this entire site (including artboiled.com) to a different webhost & server entirely, update vBulletin & change the URL in the process.

[number8]

I didn't. I deleted the install folder right after installing vb, and I checked the admin accounts when you posted that first post.

There was exploit code written into the php files. That's why I've been rewriting every vb file with a fresh copy from the installer, and deleted files that weren't supposed to be there.

[Irish]

Those accounts existed for at least a week before I drew attention to them, and when I did that was a full month after this exploit became known. That's plenty of time for the bad guys to do what they need to do.

You need to assume your entire system is compromised, because usually these guys will try and get shell access as soon as they can. If what Raiders reporterd is at all accurate, your problem extends way beyond vBulletin.

Edit: Do you still have a copy of that injected code? Any ideas what they were trying to do?

Edit2: From what I've read, this exploit goes back as far as late August, early September. There's a chance they modified a table in the database. I'd poke around & see if there's anything funny (like a lone table with a single stored procedure in it) & check your mySQL logs.

[number8]

Yea, I'm doing that now.

Aha, found something I didn't before: they installed some startup plugins that inserts the code that gives backdoor access to the root folder. Yikes.

[number8]

Never run your own website, kids.

[Skitch]

Never run your own website, kids.
And murder any person you meet who claims to be a hacker.