• BCOI outlines actions to enhance fire safety practices and awareness and improve network resiliency
• Singtel accepts findings and commits to implement recommendations

Singapore 16 December 2013 – The Singtel Board has accepted and endorsed the findings and recommendations of the independent Board Committee of Inquiry (BCOI) on the fire at the Bukit Panjang exchange on 9 October 2013. 

The BCOI’s findings and recommendations covered three main areas: fire prevention in exchanges, network reliability and resiliency, as well as public relations and communications.  The BCOI was assisted by independent technical consultant, Bell Labs, and  communications consultant, Bell Pottinger. 

Following a full investigation, the BCOI concluded that the most probable cause of the fire was hot works carried out during maintenance activities in the cable chamber.  An unauthorised blowtorch was used to heat and shrink a lead-based sealant to seal a cable duct.  The employee carrying out the hot works did not have the company issued blowtorch equipment with him and had borrowed the unauthorised blowtorch from a contractor.  The flame of the unauthorised blowtorch was almost twice as hot as that of a company issued Singtel blowtorch. The use of the unauthorised blowtorch most likely caused localised overheating, resulting in a slowburning fire in the cable chamber.   

“While the fire was due mainly to human error, the inquiry uncovered shortcomings in fire safety practices that require immediate rectification,” said Chairman of the BCOI, Mr Bobby Chin, “The technical expert has, however, affirmed that the network has a high degree of resiliency.   Singtel should continue to build on this.” 

Singtel is committed to strengthening its network to enhance Singapore’s status as a leading financial and business hub. “We recognise the impact of the incident on businesses and customers. The BCOI has given us a critical review of our operations, which will steer and motivate us to raise our standards. We will work with the industry to define a national engagement framework for greater collaboration to ensure that Singapore’s telecoms infrastructure remains resilient even during major network incidents,” said Mr Simon Israel, Chairman of Singtel.

Singtel management has accepted the BCOI’s findings and is committed to implementing the recommendations.  “We take full responsibility for the incident.  We recognise the inconvenience it caused our customers and the public, and we are sorry for this.  The BCOI review offers a valuable opportunity to take a step back and comprehensively assess our network design and operations. We want to assure customers that it is our top priority to prevent a recurrence and we have started work on the enhancements,” said Ms Chua Sock Koong, Group CEO of Singtel.

BCOI Key Findings and Recommendations

1. Fire Prevention in Exchanges

The BCOI found that the use of lead-based sealants, and the associated use of open flames, posed fire risks. The BCOI recommended replacing lead-based sealants with alternative materials that do not require heating (eg, Multi Cable Transit systems ⁽¹⁾), thereby removing the need for hot works in cable chambers. Singtel is already in the process of converting all duct seal installations to MCT Systems.  Completion of this exercise is estimated to be at the end of 2014.  Other recommendations included enhancing fire detection, fire watch practices and automatic fire suppression measures to augment existing smoke detectors and fire extinguishers.

The BCOI noted that in the interim, Singtel has switched to an alternative method for hot works, which uses a hot air blower without open flame, and enforced a longer fire watch. The BCOI found that Singtel had an existing set of safety protocols in relation to hot works and recommended a review of training for employees to ensure compliance with such protocols.

Other recommendations include:

Fire Prevention Recommendations

• Centrally monitor critical building alarms, including smoke detectors (and other fire-related warning systems), even if other contractors or organisations already provide such monitoring services.
• Take measures to limit the amount of polythyelene (or PE) material (2) for indoor use.  For new fibre optic cables being placed in exchanges, splice PE sheath cables to fire-retardant PVC sheath cables as soon as practicable upon entry of the cables into the cable chambers / exchanges.  For existing fibre optic cables already placed in exchanges, provide fire-retardant capability and consider replacing existing PE sheath cables with PVC sheath cables.
• Strengthen fire-stop practices to prevent fire spreading between floors and rooms of the exchanges and consider the use of fire-retardant materials (including intumescent foam or putty), even though the fire did not spread to any other rooms or floors in this incident.
• Enforce a more effective process to control and record physical access to cable chambers.

Safety Practices Recommendations

• Ensure employees are aware of and comply with safety protocols. In particular, enforce onsite supervision of high-risk activities and clarify roles and responsibilities.  Ensure formal training is conducted regularly for persons responsible for hot works. 
• Enforce regular equipment checks and ensure that only Singtel standard-issue equipment is used for carrying out works.

• Enhance existing fire watch practices.  It is noted that Singtel has already instituted a 24-hour fire watch after completion of hot works and supplemented fire watches with the use of handheld thermal imaging cameras.
• Review and tighten protocols relating to disabling and re-activating of smoke detectors during and after hot works.

2. Network Reliability and Resiliency

Bell Labs has advised the BCOI that Singtel has a robust core network architecture which has a high degree of resilience against incidents such as the fire. As a result, the impact of the fire was localised to the area served by the Bukit Panjang exchange. Bell Labs also found that Singtel took fast and effective measures in its incident management and applied innovative processes, which allowed multiple splicing teams to simultaneously carry out repair work in a limited space. 

The BCOI identified a number of areas of improvement in network design and stakeholder engagement. It recommended a stronger engagement with industry players in the existing three-tier industry structure for fibre services, and to establish a national framework that will foster a coordinated response from all telecom stakeholders on an agile, proactive basis in the event of a major network incident.

The BCOI also recommended that Singtel initiate regular exchanges with other global operators to share key learning points and best practices in network resilience and reliability.

Other recommendations include:

Recovery Recommendations

• Develop a plan to minimise the impact to customers during the transition from temporary solutions to permanent restoration measures.

Network Design and Architecture Recommendations

Develop and adopt automated systems and procedures to verify and maintain physical path diversity.  This will improve infrastructure resiliency by avoiding any overlapping sections where the main path and back-up path(s) could be severed by one fibre cut (known as “folded rings”). For the same reason, the main and back-up path(s) for mission-critical traffic should use separate entranceways to cabinets.

• Set up dialogue or collaboration amongst Singtel, regulators and key customer groups to improve understanding of resiliency requirements and strengthen service resiliency by offering equipment, path and/or exchange diversity for key services.  Encourage government agencies and enterprise customers to take-up protected services available. 
• Further improve wireless resiliency by ensuring path diversity for mobile base stations. To consider exchange diversity coupled with path diversity to improve  indoor wireless coverage resiliency.

Stakeholder recommendations

• Where the relevant facility is not owned by Singtel, there should be detailed and documented procedures between Singtel and the facility manager to ensure that there is a clear demarcation of roles and responsibilities. 
• Enhance international collaboration by initiating formal and regular technical exchanges of knowledge with comparable operators in the region, or globally.  This would enable the sharing of key learning points, best practices and other information related to exchange outages, service restoration and network reliability best practices.  Alternatively, initiate a benchmarking review across a range of comparable operators.

3.  Communications and Public Relations

Bell Pottinger advised the BCOI that Singtel was prompt and transparent in its release of information about the incident and found no areas requiring immediate improvement.  Singtel used multiple channels, including social media, to communicate effectively with customers.

To further enhance communications crisis response, the BCOI’s key recommendations were for Singtel to formalise crisis communications protocols which are currently based on ad hoc prior experience, extending crisis simulation drills to include key spokespeople and corporate communications staff, as well as adopt a more structured process for media and internal updates.

Other recommendations include:

Customer and Media Engagement Recommendations

• Institute standard procedures to issue information updates on a regular basis, even if there is no change of status.
• Introduce a media spokesperson at an earlier stage. Separate the roles of crisis command and media spokesperson.
• Tighten the processes around message creation and approval to ensure speed and coordination of response across multiple channels.
• Enhance online visibility by creating an “updates” section on Singtel’s website.

Messaging

• Ensure that the full context of future incidents are well-explained, thereby improving clarity of information, reducing the risk of distractions and better managing customer expectation on restoration timelines.

Other Recommended Communications Initiatives

• Consider initiating a media familiarisation campaign to build on technical transparency and ensure a core understanding of the network.
• Involve fresh viewpoints in crisis preparation and response by inviting input from cross-functional teams within Singtel and external experts to update processes and procedures.

The full BCOI report is available on www.singtel.com

                                                                                                                  ###

About the BCOI

The BCOI was established to provide an objective and expert review of the Bukit Panjang fire, and to make recommendations to further enhance Singtel’s network resilience. The BCOI used leading global experts to draw on international best practices in developing its findings.

The Terms Of Reference of the BCOI were to:

i.        Investigate the facts and circumstances leading to the fire

ii.        Review the effectiveness of the company’s incident management and response, and recommend improvements

iii.        Benchmark current network design and contingency processes against international best practices and standards

iv.        Recommend appropriate improvements to prevent future occurrences and to strengthen network resilience

The BCOI is chaired by Mr Bobby Chin Yoke Choong, a non-executive and independent director of Singtel and Chairman of its Risk Committee. Mr Chin was previously the Managing Partner of KPMG Singapore. He is supported by two other non-executive and independent directors, Mrs Fang Ai Lian and Mr Low Check Kian. Mrs Fang is the Chairman of Singtel’s Audit Committee. Mrs Fang was previously Managing Partner and then Chairman of Ernst & Young. Mr Low is a Member of Singtel’s Corporate Governance and Nominations Committee. Mr Low was one of the founding partners of NewSmith Capital Partners LLP.

(1) Multi Cable Transit system : A modular cable sealing system that provides protection from fire, smoke, water and blast pressures at the point where cables enter an exchange through the cable chamber.

(2) Polythyelene material is used to protect fibre cables in ducts and cable tunnels due to its water/gas/oil resistant properties.