O2 and Softbank Network Outage: Why and How it Happened?

The LTE (4G) mobile phone services of “SoftBank,” “Y!mobile,” Fixed Line Services called “Ouchi-no-Denwa” and a part of “SoftBank Air” services were not available for its use from 1:39 pm to 6:04 pm on Thursday, December 6, 2018, due to a network outage caused by Ericsson-made software errors related to its packet switches, covering our customers nationwide.

In an apology to its customers - Softbank has said that it is deeply apologizing to its customers for all inconveniences it caused. "We will strive to take all measures to prevent the same network outage," the statement mentioned.

Period of Network Outage

1:39 pm - 6:04 pm (JST) on Thursday, December 6, 2018

Services Affected by the Outage

“SoftBank” and “Y!mobile” 4G LTE mobile phone service
Fixed-line telephone service “Ouchi-No-Denwa”
“SoftBank Air”

Details of Impact

4G (LTE) of mobile services under the “SoftBank” and “Y!mobile” brands were not available for use across Japan. Along with this, traffic data surge occurred in the 3G network that caused further difficulties for service use.

The same situation occurred for “Ouchi-No-Denwa” and “SoftBank Air” services.

Cause of the Network Outage

At 1:39 pm on Thursday, December 6, 2018, the SoftBank Network Center detected software's malfunction in all of the packet switching machines manufactured by Ericsson, which are installed at the Tokyo Center and the Osaka Center, covering our mobile customers nationwide.

After the incident, SoftBank received a report from Ericsson that the software has been in operation since nine months ago and the failure caused by the same software also occurred simultaneously in other telecom carriers across 11 countries, which installed the same Ericsson-made devices.

The network was recovered to the normal operation by adapting the older version of the software to all packet switching machines.

Recurrence Prevention Measure

SoftBank will work together with Ericsson and thoroughly investigate this issue to prevent further recurrence.

What Ericsson Says

During December 6, 2018, Ericsson has identified an issue in certain nodes in the core network resulting in network disturbances for a limited number of customers in multiple countries using two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity).

Börje Ekholm, President and CEO, Ericsson, explained, “The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers. We work hard to ensure that our customers can limit the impact and restore their services as soon as possible.”

An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers. A complete and comprehensive root cause analysis is still in progress. Our focus is now on solving the immediate issues.

During the course of December 6, most of the affected customers’ network services have been successfully restored. We are working closely with the remaining customers that are still experiencing issues.

Meanwhile, O2 network in the UK has been restored and the company is initiating a full review to understand why and how it happened.