Friday, November 11, 2011

Juniper fail seen as culprit in site outages

Several reports of core dumps by Juniper Networks routers have lead to suspicions that a BGP update bug is behind site outages across the world.

The problems started occurring at 2:15pm on Monday, by all indications. Juniper Networks routers across the world dumped their main memory — a 'core dump' — and have needed to be restarted, according to reports on Twitter.

The chain of events appears to be that a Border Gateway Protocol (BGP) update did not play well with Juniper kit on the Junos 10.2 and 10.3 operating system and this led to the affected kit carrying out a core dump, taking down dependent sites in the process such as Level3, according to multiple reports on Twitter and other sites.

"This morning, Juniper learned of a BGP edge router issue that affected a small percentage of customers," Mark Bauhaus, an executive vice president for Juniper Networks' services, support and operations, told ZDNet UK on Monday evening. "A software fix is available, and we’ve been working with our customers to immediately deploy the fix."

The problems "most likely" stem from a BGP update affecting Juniper Networks gear running Junos 10.2 and 10.3, dotwaffle (Matthew Walster), a network engineer at network services company IX Reach, posted on Twitter shortly after the problems began.

"We just had a core sump on all our Juniper running Junos 10.3R2.11," NeoTelecoms, a French telecommunications company, wrote in a tweet.

HostVirtual issued a note to customers on Monday saying major ISP Level3 had suffered a failure that seemed "to be related to a rogue route(s) injected into the BGP table or a Juniper bug." Many other ISPs had been reporting Juniper Networks issues as well, it said.

"We are aware of reports of problem with core Juniper routers that may affect ISPs and intermediate connections and may slow some sites etc," MerulaSupport, a small UK-based ISP, wrote.

Level3 gave more information on the outage just after 5:00pm, which said the problem was due to some routers but did not name a specific networking vendor.

"Shortly after 9 am ET today, Level 3's network experienced several outages across North America and Europe relating to some of the routers on our network," the company said. "Our technicians worked quickly to bring systems back online. At this time, all connection issues have been resolved, and we are working hard with our equipment vendors to determine the exact cause of the outage and ensure all systems are stable."

Juniper Networks had not responded to a request for information on whether the problems affected systems running Junos 10.2 and 10.3 at the time of writing.

No comments:

Post a Comment