30 August, 2007

Scheduled Maintenance on Gold Server

Update 1 Sep 2007, 7AM: We have to arrange another time for the maintenance. Started accounts backup from 10:20PM last night till now, unfortunately it's still not completed yet. It would cause too long downtime (estimated 10 hours+) this way. We will find a better solution and arrange another time. Will keep you informed.

Please be informed that we are going to replace hard drives on Gold Server and implement hardware maintenance on this coming Friday night, 31st Aug 2007.

As you know, no hardware can last forever, so normally we replace HDD every 18-24 months on all servers as routine maintenance.

Now it's time for Gold Server.

Another reason that we decided to carry out hardware maintenance on Gold server now is that Gold server has been having hardware problems this month. Our investigation shows that possible causes include:

1)HDD bad sectors
2)Kernel bug
3)BIOS needs to be updated
4)Combination of above

To be honest, hardware failure is unavoidable as you can understand, but the worst thing this time is the bad timing.

Our original plan was to move as much accounts as possible to new server before we change HDD to avoid or reduce service disruption. However, the new server ordered has not been delivered till now and we don't expect it to arrive in next 1 or 2 days.

We simply cannot wait anymore considering current situation. So we decided to go ahead ASAP.

But, updating BIOS is a bit risky. If anything went wrong, the whole server would be spoiled and become useless. In it did happen, we must change the whole server. Therefore we bought a server from other hosting company, at rather high cost.

Please rest assured, we will try our best to complete the maintenance and keep servers in good shape.

The hardware maintenance for Gold Server is scheduled on 31st, Aug 2007, from around 11PM to 5AM next morning, 1st, Sep 2007 .

Expected downtime is 6 hours while we replace HDD, install OS, update BIOS/kernel. Hopefully it will be shorter if everything goes smoothly, but it might be longer.

You can check which server your account is on via IP address:

http://sghosting.blogspot.com/2005/05/how-do-i-know-which-server-my-site-is.html

Sorry for inconvenience caused and thank you for your kind understanding.

02 August, 2007

Problem with Gold Server

Update 12:20PM 29 Aug: It's up now. I know this is unacceptable to have same issue 3 times in one month. This is the worst case we have had in past few years. We're working on compensation plan for clients on Gold server. And most importantly, we will do hardware maintenance asap. More details will be posted when schedule is confirmed. Sorry for the downtime and trouble caused to users.

Update 11:50AM, 29 Aug: Gold server is having same issue again now. We know it's bad. We'll bring it up and update here.

Update 3pm, 21 Aug: Gold server is up now. We're investigating. Will update here if further action/maintenance is needed.

Update on 21 Aug: Gold server is having problem again today from around 11am. We're trying our best to bring it back asap. Sorry for the trouble. Will update here once we have further news.

Dear Customer,

There was problem on Gold server this morning and we managed to bring it up at around 1:55pm. Sadly while we were investigating further on the server, it went down again. Our server admin is on the way to datacenter to troubleshoot.

Very sorry for the inconvenience caused. We will try our best to bring it back as soon as possible.

Thank you for your kind understanding.

Update: It's up at around 5:50pm

There might be problem with hard disk or kernel, we will investigate to see if it's hard disk, if so, we will have to change HDD as soon as possible, most likely tonight or tomorrow night.

Therefore, unfortunately, there might be urgent HDD maintenance and downtime tonight or tomorrow night. We'll make announcement to all clients and post updates here.

Clients on Gold server, please do NOT do backup or upload anything until maintenance is complete.

Again, sorry for inconvenience and thank you for understanding.