ClubSNAP's unexpected week-long downtime

[Edmund]

Hi Everyone,

As most of you know, ClubSNAP has been down for the past week due to a series of catastrophic hardware and software failures. To be honest, this is probably the worst incident we have encountered since the time we first started.

Initial attempts to fix hardware were met with failures after failures resulting in backup servers going down as well. Then more problems were encountered with portals/forums software AND database restoration meeting obstacles after obstacles. It feels like a mega marathon of hardware and software fixes for the past week and I am anticipating more to come. :sweat::sweat::sweat:

Anyway, we now have ClubSNAP back up & running on backup machines and we expect to have another round of downtimes when the main server is back from the "hospital". Please report all problems in FAQs and Feedback.

Kindly also note that as there's no way for us to retrieve the data as at initial downtime, we have restored the database to the last backup on Sunday, 28th March 2010, 3am (i.e. threads/posts created from that time to initial downtime at 4:30pm Sunday are no longer available).

During this period, we may also be down occasionally as we are no longer running on optimum hardware so do bear with us if that happens - the ClubSNAP Admin and Mod Teams will be monitoring the site closely so that issues can be rectified asap.

Thanks for the patience and welcome back everyone!!

 

[Wai]

Kindly also note that as there's no way for us to retrieve the data as at initial downtime, we have restored the database to the last backup on Sunday, 28th March 2010, 3am (i.e. threads/posts created from that time to initial downtime at 4:30pm Sunday are no longer available).

Glad to hear that the server and database are back online.

I received an important PM on last Saturday, before the last backup, but seems that it is missing in my PM box now

 

[neech7]

Glad to see CS back. Got tired of going to dpreview.

 

[Dream Merchant]

You guys did a phenomenal job Edmund!

I can't even begin to imagine the horrors you and your team have been through this past wek.

THANK YOU!

 

[NovJoe]

Great job in rendering the problems! Really glad to be back in ClubSnap !

 

[matthew1381]

WELL DONE !!

and thanks again !

 

[Darren]

Thank you to all members for their patience - it's been long sleepless nights over the last week or so, especially for Edmund.

Suffice to say that we are back online, albeit with some limitations, and as soon as the main server is fixed or when the new server is delivered and installed (whichever is earlier) we will restore full service (and more!)

In the meantime, please accept our apologies for the downtime (especially for our Advertisers - we will be extending your subscription by two weeks to make up) and please bear with us if there are occasional interruptions with the backup server.

 

[vaxvms]

well done...thks again

 

[sgciti55]

Thanks guys for the hard work.

 

[eugenep]

Thanks for the update. Hope this down time won't happen again * cross finger*

 

[Keiichi]

Glad that clubsnap is back... thx for all the hard work in restorting the system....

 

[Peng Kang Hill]

Thanks, guys. Round of applause to our heroes who got it back to life. Clap clap clap clap!!!

Peng Kang Hill

 

[Akatsuki]

thank you for bring CS back, hope your next round of integrating the new/repaired server will be trouble-free!

 

[raptor84]

KUdos for the hard work

 

[Edmund]

Just a fyi based on queries from other threads :
We are still running on backup hardware - now split between 2 servers to try and handle the load but still sorely lacking the horsepower needed to run things at full gamut. That's the reason why we've turned off some features (like search) to ease the loads and also why at times you may see a server too busy message (we've put a limit on the number of concurrent connections so just retry). It will be a couple more days before the new server arrives so until then, nothing much we can do other than monitor this setup. We are also hopeful that the old server returns from sickbay soon so that we can return things back to normality earlier. Suffice to say, both the new and older returning server will work to complement each other to further reduce downtimes caused by hardware failures.

In any case, we'll update everyone as things go along. Thanks for everyone's patience with us!! :sweat:

 

[Phocus]

Thanks Edmund and team, much appreciated

 

[zonal77]

Kudos to all the hardwork put in by the relevant in charge to get CS up and running again.

Thanks!

:thumbsup:

 

[yc2005]

That's the reason why we've turned off some features (like search) to ease the loads and also why at times you may see a server too busy message (we've put a limit on the number of concurrent connections so just retry). It will be a couple more days before the new server arrives so until then, nothing much we can do other than monitor this setup. We are also hopeful that the old server returns from sickbay soon so that we can return things back to normality earlier.

I'm sure most of us are more than happy to put up with slight inconveniences and have CS back up again.

 

[Calahan]

Thank you to CS admin team and those who helped for their hardwork. Cheers!

 

[Hexlord]

Thanks for the hard work in bringing things up to speed.