Dear Customers,
Just received "Urgent Maintenance for UPS & Incident Report for COM3 Level2 UPS Interruption" from Singtel EXPAN. There was UPS fault yesterday afternoon and at this moment (1:00pm) there was problem again.
We are trying our best to get servers back online ASAP. Sorry for the inconvenience.
As indicated in their urgent notice, there will be urgent maintenance activities required for UPS 2-3 & 2-4 on the following date/ time:
Date/ Time: 17/06/07, SGT 1pm to SGT 3pm
Sorry for the late notice as we just received the notice from EXPAN.
Extract from PDF file from EXPAN:
---------------------------------------------------------
Interim IR - Com III UPS Interruption - 14Jun07.doc 1
Restricted only when filled completely
Unless indicated, document is “Uncontrolled” when printed.
Interim Incident Report
Items
Descriptions
Remarks
Reported by:
SINGTEL EXPAN Operations
Site:
COM III Data Center, Level 2
Date of incident:
14/06/2007
Time occurred:
1414hrs
Date/Time Reported:
14/06/2007 at 1414hrs
Date/Time Resolved:
14/06/2007 at 1416hrs
Review by:
Problem Descriptions:
14 June 2007
1414hrs – UPS fault alerts for UPS 2-3 and 2-4 were received by NOC. UPS 2-3 & 2-4 provide power to equipments hosted in COM III, Level 2 EXPAN Data Centre.
1415hrs – Onsite UPS engineer was activated immediately to check on the UPS.
1416hrs – UPS power was restored.
Findings:
Review of the UPS logs shows that UPS 2-3 and 2-4 inverters were off at 14:14:41hrs and 14:14:42hrs respectively and the load was not transferred to static bypass source. The PCB controller (Control and Communication board) is determined to be faulty.
Immediate Resolution:
Immediate resolution to prevent the UPS from going offline is to replace the PCB controller:
1. 2 hrs maintenance window is required
2. Customer loads will be transferred to External bypass source so that replacement of the faulty PCB controller (Control and Communication board) and a complete test of the UPS systems can be carried out.
3. Customer loads on External bypass source will be supported by Raw power source during the maintenance period.
4. Due to the faulty PCB controller, there will be a power disruption of up to 10mins when the transfer of load to external bypass is performed.
5. The proposed maintenance window will be scheduled as stated below:
Date/ Time: 17/06/07, SGT 1pm to SGT 3pm
Recommendations to customer:
1. Customer is required to shutdown all their equipments before commencement of the maintenance window before SGT 1pm. (For those whom have subscribed to manage system services, Singtel will assist in the shut down of equipments on 17/06/07, starting from SGT 11.30am)
2. Once the faulty UPS parts are replaced and complete testing of the UPS systems are carried out, SingTel will inform customer via email/ phone to start up their equipments.
Interim IR - Com III UPS Interruption - 14Jun07.doc 2
Restricted only when filled completely
Unless indicated, document is “Uncontrolled” when printed.
3. After the maintenance is completed, customer is required to start-up all their equipments. (For those whom have subscribed to manage system services, Singtel will assist in the start-up process.)
Remarks:
Feel free to contact us should you require further clarification.
We sincerely apologized for the inconvenience cause.
--------------------------------------------------------
Will keep you updated.
Update: all servers are up by 2:10pm
Again, pls note there will be maintenance on 17/06/07, SGT 1pm to SGT 3pm
