ClubSNAP's unexpected week-long downtime


Edmund

ClubSNAP Admin
Staff member
Jan 17, 2002
2,346
0
36
44
Singapore
eoslightbox.com
#1
Hi Everyone,

As most of you know, ClubSNAP has been down for the past week due to a series of catastrophic hardware and software failures. To be honest, this is probably the worst incident we have encountered since the time we first started.

Initial attempts to fix hardware were met with failures after failures resulting in backup servers going down as well. Then more problems were encountered with portals/forums software AND database restoration meeting obstacles after obstacles. It feels like a mega marathon of hardware and software fixes for the past week and I am anticipating more to come. :sweat::sweat::sweat:

Anyway, we now have ClubSNAP back up & running on backup machines and we expect to have another round of downtimes when the main server is back from the "hospital". Please report all problems in FAQs and Feedback.

Kindly also note that as there's no way for us to retrieve the data as at initial downtime, we have restored the database to the last backup on Sunday, 28th March 2010, 3am (i.e. threads/posts created from that time to initial downtime at 4:30pm Sunday are no longer available).

During this period, we may also be down occasionally as we are no longer running on optimum hardware so do bear with us if that happens - the ClubSNAP Admin and Mod Teams will be monitoring the site closely so that issues can be rectified asap.

Thanks for the patience and welcome back everyone!! :)
 

Wai

Senior Member
Jan 17, 2002
5,270
0
36
39
South Pole with Penguin
singastro.org
#2
Kindly also note that as there's no way for us to retrieve the data as at initial downtime, we have restored the database to the last backup on Sunday, 28th March 2010, 3am (i.e. threads/posts created from that time to initial downtime at 4:30pm Sunday are no longer available).
Glad to hear that the server and database are back online.

I received an important PM on last Saturday, before the last backup, but seems that it is missing in my PM box now
 

neech7

New Member
Dec 17, 2009
1,083
0
0
US
#3
Glad to see CS back. Got tired of going to dpreview.
 

Dream Merchant

Moderator
Staff member
Jan 11, 2007
9,659
6
38
#4
You guys did a phenomenal job Edmund!

I can't even begin to imagine the horrors you and your team have been through this past wek.

THANK YOU!
 

Darren

ClubSNAP Admin
Staff member
Jan 16, 2002
8,617
27
48
Melbourne
#7
Thank you to all members for their patience - it's been long sleepless nights over the last week or so, especially for Edmund.

Suffice to say that we are back online, albeit with some limitations, and as soon as the main server is fixed or when the new server is delivered and installed (whichever is earlier) we will restore full service (and more!)

In the meantime, please accept our apologies for the downtime (especially for our Advertisers - we will be extending your subscription by two weeks to make up) and please bear with us if there are occasional interruptions with the backup server.
 

sgciti55

New Member
Feb 1, 2008
69
1
0
#9
Thanks guys for the hard work.
 

eugenep

Senior Member
Nov 20, 2004
2,242
0
0
Changi
#10
Thanks for the update. Hope this down time won't happen again * cross finger*
 

Keiichi

Senior Member
Aug 8, 2005
523
0
16
#11
Glad that clubsnap is back... thx for all the hard work in restorting the system....
 

Jul 15, 2009
156
0
0
Toa Payoh
#12
Thanks, guys. Round of applause to our heroes who got it back to life. Clap clap clap clap!!!

Peng Kang Hill
 

Edmund

ClubSNAP Admin
Staff member
Jan 17, 2002
2,346
0
36
44
Singapore
eoslightbox.com
#15
Just a fyi based on queries from other threads :
We are still running on backup hardware - now split between 2 servers to try and handle the load but still sorely lacking the horsepower needed to run things at full gamut. That's the reason why we've turned off some features (like search) to ease the loads and also why at times you may see a server too busy message (we've put a limit on the number of concurrent connections so just retry). It will be a couple more days before the new server arrives so until then, nothing much we can do other than monitor this setup. We are also hopeful that the old server returns from sickbay soon so that we can return things back to normality earlier. Suffice to say, both the new and older returning server will work to complement each other to further reduce downtimes caused by hardware failures.

In any case, we'll update everyone as things go along. Thanks for everyone's patience with us!! :sweat:
 

Phocus

Senior Member
Aug 21, 2009
4,087
0
0
9
#16
Thanks Edmund and team, much appreciated :)
 

zonal77

New Member
Jan 12, 2010
168
0
0
#17
Kudos to all the hardwork put in by the relevant in charge to get CS up and running again.

Thanks!

:thumbsup:
 

yc2005

New Member
May 14, 2009
698
0
0
#18
That's the reason why we've turned off some features (like search) to ease the loads and also why at times you may see a server too busy message (we've put a limit on the number of concurrent connections so just retry). It will be a couple more days before the new server arrives so until then, nothing much we can do other than monitor this setup. We are also hopeful that the old server returns from sickbay soon so that we can return things back to normality earlier.
I'm sure most of us are more than happy to put up with slight inconveniences and have CS back up again. ;)
 

Jun 26, 2009
126
0
16
Toa Payoh
#19
Thank you to CS admin team and those who helped for their hardwork. Cheers!
 

Top Bottom