Page 1 of 4 123 ... LastLast
Results 1 to 20 of 78

Thread: ClubSNAP's unexpected week-long downtime

  1. #1
    ClubSNAP Admin Edmund's Avatar
    Join Date
    Jan 2002
    Location
    Singapore
    Posts
    2,336

    Default ClubSNAP's unexpected week-long downtime

    Hi Everyone,

    As most of you know, ClubSNAP has been down for the past week due to a series of catastrophic hardware and software failures. To be honest, this is probably the worst incident we have encountered since the time we first started.

    Initial attempts to fix hardware were met with failures after failures resulting in backup servers going down as well. Then more problems were encountered with portals/forums software AND database restoration meeting obstacles after obstacles. It feels like a mega marathon of hardware and software fixes for the past week and I am anticipating more to come.

    Anyway, we now have ClubSNAP back up & running on backup machines and we expect to have another round of downtimes when the main server is back from the "hospital". Please report all problems in FAQs and Feedback.

    Kindly also note that as there's no way for us to retrieve the data as at initial downtime, we have restored the database to the last backup on Sunday, 28th March 2010, 3am (i.e. threads/posts created from that time to initial downtime at 4:30pm Sunday are no longer available).

    During this period, we may also be down occasionally as we are no longer running on optimum hardware so do bear with us if that happens - the ClubSNAP Admin and Mod Teams will be monitoring the site closely so that issues can be rectified asap.

    Thanks for the patience and welcome back everyone!!

  2. #2
    Senior Member
    Join Date
    Jan 2002
    Location
    South Pole with Penguin
    Posts
    5,291

    Default Re: ClubSNAP's unexpected week-long downtime

    Quote Originally Posted by Edmund View Post
    Kindly also note that as there's no way for us to retrieve the data as at initial downtime, we have restored the database to the last backup on Sunday, 28th March 2010, 3am (i.e. threads/posts created from that time to initial downtime at 4:30pm Sunday are no longer available).
    Glad to hear that the server and database are back online.

    I received an important PM on last Saturday, before the last backup, but seems that it is missing in my PM box now
    We are HDD of PC & FT are MB add to storage;
    so PC never hangs with enormous storage capacity - LKY

  3. #3

    Default Re: ClubSNAP's unexpected week-long downtime

    Glad to see CS back. Got tired of going to dpreview.

  4. #4

    Default Re: ClubSNAP's unexpected week-long downtime

    You guys did a phenomenal job Edmund!

    I can't even begin to imagine the horrors you and your team have been through this past wek.

    THANK YOU!

  5. #5
    Senior Member NovJoe's Avatar
    Join Date
    Feb 2009
    Location
    ~Sunny Little Island~
    Posts
    5,655

    Default Re: ClubSNAP's unexpected week-long downtime

    Great job in rendering the problems! Really glad to be back in ClubSnap !

  6. #6

    Default Re: ClubSNAP's unexpected week-long downtime

    WELL DONE !!

    and thanks again !

  7. #7
    ClubSNAP Admin Darren's Avatar
    Join Date
    Jan 2002
    Location
    Melbourne
    Posts
    8,496
    Blog Entries
    2

    Default Re: ClubSNAP's unexpected week-long downtime

    Thank you to all members for their patience - it's been long sleepless nights over the last week or so, especially for Edmund.

    Suffice to say that we are back online, albeit with some limitations, and as soon as the main server is fixed or when the new server is delivered and installed (whichever is earlier) we will restore full service (and more!)

    In the meantime, please accept our apologies for the downtime (especially for our Advertisers - we will be extending your subscription by two weeks to make up) and please bear with us if there are occasional interruptions with the backup server.

  8. #8
    Member
    Join Date
    Dec 2005
    Location
    BEHIND YOU!!!!!!
    Posts
    675

    Default Re: ClubSNAP's unexpected week-long downtime

    well done...thks again

  9. #9

    Default Re: ClubSNAP's unexpected week-long downtime

    Thanks guys for the hard work.

  10. #10
    Senior Member
    Join Date
    Nov 2004
    Location
    Changi
    Posts
    2,242

    Default Re: ClubSNAP's unexpected week-long downtime

    Thanks for the update. Hope this down time won't happen again * cross finger*

  11. #11

    Default Re: ClubSNAP's unexpected week-long downtime

    Glad that clubsnap is back... thx for all the hard work in restorting the system....

  12. #12

    Default Re: ClubSNAP's unexpected week-long downtime

    Thanks, guys. Round of applause to our heroes who got it back to life. Clap clap clap clap!!!

    Peng Kang Hill

  13. #13

    Default Re: ClubSNAP's unexpected week-long downtime

    thank you for bring CS back, hope your next round of integrating the new/repaired server will be trouble-free!
    Playing with M43.

  14. #14

    Default Re: ClubSNAP's unexpected week-long downtime

    KUdos for the hard work
    Furry Photos - Photography for the Modern Pet

  15. #15
    ClubSNAP Admin Edmund's Avatar
    Join Date
    Jan 2002
    Location
    Singapore
    Posts
    2,336

    Default Re: ClubSNAP's unexpected week-long downtime

    Just a fyi based on queries from other threads :
    We are still running on backup hardware - now split between 2 servers to try and handle the load but still sorely lacking the horsepower needed to run things at full gamut. That's the reason why we've turned off some features (like search) to ease the loads and also why at times you may see a server too busy message (we've put a limit on the number of concurrent connections so just retry). It will be a couple more days before the new server arrives so until then, nothing much we can do other than monitor this setup. We are also hopeful that the old server returns from sickbay soon so that we can return things back to normality earlier. Suffice to say, both the new and older returning server will work to complement each other to further reduce downtimes caused by hardware failures.

    In any case, we'll update everyone as things go along. Thanks for everyone's patience with us!!

  16. #16
    Workshop/Shoot Organiser Phocus's Avatar
    Join Date
    Aug 2009
    Posts
    4,075

    Default Re: ClubSNAP's unexpected week-long downtime

    Thanks Edmund and team, much appreciated
    Phocus Academy School of Photography | www.phocus.com.sg | academy@phocus.com.sg| +65 9765 4224

  17. #17

    Default Re: ClubSNAP's unexpected week-long downtime

    Kudos to all the hardwork put in by the relevant in charge to get CS up and running again.

    Thanks!


  18. #18

    Default Re: ClubSNAP's unexpected week-long downtime

    Quote Originally Posted by Edmund View Post
    That's the reason why we've turned off some features (like search) to ease the loads and also why at times you may see a server too busy message (we've put a limit on the number of concurrent connections so just retry). It will be a couple more days before the new server arrives so until then, nothing much we can do other than monitor this setup. We are also hopeful that the old server returns from sickbay soon so that we can return things back to normality earlier.
    I'm sure most of us are more than happy to put up with slight inconveniences and have CS back up again.

  19. #19
    Member
    Join Date
    Jun 2009
    Location
    Toa Payoh
    Posts
    126

    Default Re: ClubSNAP's unexpected week-long downtime

    Thank you to CS admin team and those who helped for their hardwork. Cheers!

  20. #20

    Default Re: ClubSNAP's unexpected week-long downtime

    Thanks for the hard work in bringing things up to speed.
    Black Rabbit Photography
    DeviantArt | Multiply | Flickr

Page 1 of 4 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •