Urgent Maintenance for Gold Server

Update 4:35AM: Done, and we are monitoring.

We noticed some disk error on gold server just now (2AM, 3 Nov, 2008) and have to carry out urgent maintenance on gold server starting from around 3:30AM to 6:30AM, hopefully it can be completed earlier.

Sorry for the inconvenience and thank you for kind understanding.

SWH

DDOS To Network

Update 9PM 15 Sep: Most sites should be up. If you still cannot access your domain, it's either because your ISP hasn't updated their DNS cache or your domain is not using our nameservers, you need to update A record for your domain yourself.

Thank you for your patience and understanding.


Update 7PM 15 Sep: The attack is still ongoing, and the worst thing is, we do not know when it will be over.

So we have managed to get new IP addresses for our server and changed our server IPs. If your domain is using our nameservers (you should), you do not need to do anything. We've made changes to our DNS IPs and sites should be up soon.

If your domain is NOT using our nameservers, please change server IPs yourself. Just change IP from 203.175.160.x to 203.175.163.x

For example, if you are currently using 203.175.160.74, change it to 203.175.163.74

If yours is reseller account, you need to change your nameserver IPs instead of account IP.

-------------End Update-----------

Bad News !

There's Heavy DDOS attack to our uplink provider's network, 1gbps attack, the worst one in past few years to whole Singtel network. All our servers (and many other hosting providers on same network) are not accessible from around 1:30pm 15 Sep.

The attack is against other server, not ours, but it's too heavy, all servers on the network are affected.

We are monitoring and trying our best to find a solution but cannot estimate when the attack will be over since it's out of our control, not even in Singtel's control.

Will post here when there's updates.

Sorry for inconvenience and thank you for kind understanding.

SWH

PS. backup server is not a solution for such DDOS attack. As the whole network is under attack, all servers are not accessible. It's not practical to maintain a backup server for all servers in another datacenter. Even if it's possible to maintain backup servers in another datacenter, moving all account data to the other datacenter, making changes to IP address, waiting for DNS update, it would be a few days later considering the amount of domains/accounts affected.

Scheduled Maintenance at Sintel EXPAN Datacenter

Please be informed that Singtel EXPAN datacenter will be conducting scheduled maintenance on 16 Aug AND 17 Aug, 2008.

All customers, especially VPS, dedicated server and server colocation customers, please do your own backup before implementation of the maintenance.

The following is given by SingTel.

************************************
Dear Customers,

SingTel EXPAN will be conducting* Cyclical UPS Parts (Capacitors and
Fans) Replacement* in accordance to guidelines set by UPS manufacturer;

_*Commencement Date/ Time*_**

*16-08-08/ 1700hrs (SGT) to 17-08-08/ 0730hrs (SGT)*


_*Purpose of Works*_**

Following the recommended cyclical parts replacement guidelines set by
the UPS manufacturer, the AC/DC capacitors and fans for the UPS 5-1 and
6-1 have to be replaced. This would ensure that the performance of UPS
5-1, 6-1 (1+1 setup - function as a pair) is maintained within
specifications.


_*Impact on Services*_* *

* During the implementation work, the customer load on the UPSes
will be transferred to external maintenance bypass.
* In this instance, Customer?s load will be supported by Raw power
from the PowerGrid which are back-up by the building generator
supply during the replacement works.
* In the event that the raw power supply from the grid is disrupted,
customer?s load on single power feed supply (A source/power strip)
will be affected.
* *Customers are advised to backup your critical data or divert your
critical services before the implementation period where neccessary.*

_*Remarks (See attached table)*_**

* For customers hosted in Com3 - level 4, above mentioned UPSes are
supplying power to PDU 9, 10A and 11A
* For customers hosted in Com3 - level 2, above mentioned UPSes are
supplying power to PDU 13A

Picture (Device Independent Bitmap)

Feel free to contact us should you require further clarification on this
matter.

Thank you.

Warmest Regards
EXPAN NOC
**********************************************

***********************************************
This is to inform you that SingTel will be carrying out a network maintenance activity for their Network Switch at SingTel premises by SingTel Network Engineer on 17 Aug 2008 from 12:01 AM till 04:00 AM Singapore Time (SGT). The circuit will be interrupted up to 5 minutes during this maintenance window and expect to have a network downtime when the interruption occurred.
***********************************************

Fire at ThePlanet Datacenter

Update at 5:30PM ２ June: It's up now. Hope it's finally fixed, instead of kind of testing.

There're only a few clients on our Windows server, but if you happen to be on Windows server and wondering why your web site and email is not accessible from this morning til this moment 5PM 1 June 2008, and counting, here's the news on fire at ThePlanet datacenter, where our Windows server is located:

From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.

Sadly it's not in our control and at this time we do not have an estimate...

Update at 1PM 2 June: It's still not up after more than 1 day. Here's update from the datacenter：

As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.

As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.

There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.

We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.

We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.

We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.

I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.

I plan to have an audio update tomorrow evening.

Until then,

Douglas J. Erwin
Chairman & Chief Executive Officer