Music Banter - View Single Post

djchameleon · 08-27-2014, 11:33 AM

Quote:

Originally Posted by djchameleon

Today at noon they are going down for 30 min. Maintenance then when it comes up the new one should be in place.

well it was up for a tiny bit then came crashing down. Here is the update they posted last night.

Quote:

Update: We will be in Maintenance Mode through tomorrow morning. Developers have been working for 13 hours and need to go home to rest. They will get back to it first thing in the morning.

For details on what's happening, here is Steven's comment about some of the technicalities:

"We added extra servers this morning in anticipation of the rush of traffic. Once the servers were up and running, we put the old site into maintenance mode anticipating a quick change over. Unfortunately, there was a bug with the chat server that appeared out of nowhere once we added more servers to the cluster. As with many bugs, you don't know how long it will take to fix, you just dive right in.

By 3pm we decided to bring the site up with less chat servers, which would temporarily prevent the bug but we weren't sure how less servers would handle the load. Bottom line: they couldn't. People couldn't connect to them, which caused the site to stay black (we have addressed that while we were down today), which meant a ton of people were refreshing the page over and over again. This had a cascading effect of slamming the rest of the servers and it very quickly got worse.

Support tickets were flying in as people were reporting issues logging in, as well as connecting, etc. We decided that we had gathered enough information in the log to fix the biggest issues and so we put the site back into maintenance mode to allow us to tackle them, as well as give our support team time to respond to the people who emailed us.

We are going to bring the site back up again as soon as we can to see how well our fixes today worked, but you should expect that we might need to go back into maintenance mode again to continue fixing the issues. All of the developers are working late tonight and we are dedicated to getting the site back up and stable as soon as we can. Thank you again for your patience and understanding."