This website collects cookies to deliver better user experience. Cookie Policy
Accept
Sign In
The Wall Street Publication
  • Home
  • Trending
  • U.S
  • World
  • Politics
  • Business
    • Business
    • Economy
    • Real Estate
    • Markets
    • Personal Finance
  • Tech
  • Lifestyle
    • Lifestyle
    • Style
    • Arts
  • Health
  • Sports
  • Entertainment
Reading: Facebook Blames Engineering Error of ‘Our Own Making’ for Outage
Share
The Wall Street PublicationThe Wall Street Publication
Font ResizerAa
Search
  • Home
  • Trending
  • U.S
  • World
  • Politics
  • Business
    • Business
    • Economy
    • Real Estate
    • Markets
    • Personal Finance
  • Tech
  • Lifestyle
    • Lifestyle
    • Style
    • Arts
  • Health
  • Sports
  • Entertainment
Have an existing account? Sign In
Follow US
© 2024 The Wall Street Publication. All Rights Reserved.
The Wall Street Publication > Blog > Tech > Facebook Blames Engineering Error of ‘Our Own Making’ for Outage
Tech

Facebook Blames Engineering Error of ‘Our Own Making’ for Outage

Editorial Board Published October 5, 2021
Share
Facebook Blames Engineering Error of ‘Our Own Making’ for Outage
SHARE

A simple technical mistake caused a global outage Monday that left more than 2.9 billion internet users unable to access Facebook, Instagram, WhatsApp and other tools.

Contents
Newsletter Sign-upTechnology AlertThe Facebook Files

The roughly six-hour disruption, which was the largest in the company’s history, based on the number of users affected, arose when Facebook Inc. FB 2.06% was trying to do routine maintenance related to how internet data routes back and forth through its network systems, according to a company blog post Tuesday.

Seeking to get a read on Facebook’s networking capacity, engineers issued a networking command that inadvertently pulled all of Facebook’s data centers off the company’s network. That led to a cascade of failures that pulled all of Facebook’s properties off the internet.

Ultimately, Facebook engineers—a team of people who built one of the world’s most sophisticated networks—had to use pre-internet technology to solve the problem. They had to drive to data centers and restart systems there, the company said.

The outage was “caused not by malicious activity, but an error of our own making,” Santosh Janardhan, Facebook’s vice president of infrastructure, wrote in the blog post.

The outage had widespread and global ripple effects. It cut off essential communication in some areas of the world, upended e-commerce in some countries, hampered some small businesses and led others to see a marketing opportunity. In some quarters, it was a cause for reflection about the extent to which Facebook and its platforms are woven into global connectivity.

Internet giants such as Facebook have poured billions of dollars into their sprawling global data centers over the past decades, designing their own networking gear and the software that powers them.

That has allowed these companies to operate with unmatched speed and efficiency, but it also creates vulnerability. The scale and complexity required to operate and maintain such a network, and the extent to which its infrastructure is managed and controlled by one company, can lead to circumstances in which small errors can have outsize impact, networking experts say.


Newsletter Sign-up

Technology Alert

Major news in the technology sector.


“This is an outfit with infinite resources and some of the most talented people,” said Doug Madory, director of internet analysis at the network monitoring firm Kentik. He said Facebook may not have applied enough scrutiny to its own solutions and backup processes.

One key question that Facebook has yet to answer is why the company’s backup network, called an out-of-band network, failed to work on Monday. This network is designed to be separate from the rest of Facebook and was supposed to provide engineers with a way to fix systems remotely within minutes when they go down.

In his blog post, Facebook’s Mr. Janardhan said that the out-of-band network didn’t work yesterday, but didn’t explain why.

Instead, with engineers unable to reset their misconfigured gear, a cascading set of failures ensued.

Once the data centers were offline, servers that used the Domain Name System, or DNS, to direct internet traffic pulled themselves off the internet. DNS is what browsers and mobile phones use to find Facebook’s services on the internet, and without that connection it was “impossible for the rest of the internet to find our servers,” Mr. Janardhan said.

The DNS changes also disabled internal tools that would have allowed Facebook’s engineers to restore service remotely, forcing engineering staff to drive to data centers and restart systems there.

That took more time. “They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them,” Mr. Janardhan said. “So it took extra time to activate the secure access protocols needed to get people on-site and able to work on the servers.”

The Facebook Files

Write to Robert McMillan at [email protected]

Copyright ©2021 Dow Jones & Company, Inc. All Rights Reserved. 87990cbe856818d5eddac44c7b1cdeb8

TAGGED:Tech NewsWall Street Publication
Share This Article
Twitter Email Copy Link Print
Previous Article Carl Icahn Holds Significant Stake in Southwest Gas, Opposes Planned Deal Carl Icahn Holds Significant Stake in Southwest Gas, Opposes Planned Deal
Next Article Kim Jong-un regime ‘disappointed’ with Joe Biden, former CIA North Korea chief says Kim Jong-un regime ‘disappointed’ with Joe Biden, former CIA North Korea chief says

Editor's Pick

Alyssa Farah Griffin: ‘The View’ Co-Host is Pregnant With Child #1!

Alyssa Farah Griffin: ‘The View’ Co-Host is Pregnant With Child #1!

Studying Time: 3 minutes The View co-host Alyssa Farah Griffin is pregnant! On ‘The View,’ Alyssa Farah Griffin breaks the…

By Editorial Board 3 Min Read
Mandy Moore ‘Unrecognizable’ to Followers After Debuting New Face
Mandy Moore ‘Unrecognizable’ to Followers After Debuting New Face

Studying Time: 4 minutes Mandy Moore has followers scratching their heads. This…

6 Min Read
Arturo Gatti Jr. Reason behind Dying: Son of Boxing Legend Passes Away at 17
Arturo Gatti Jr. Reason behind Dying: Son of Boxing Legend Passes Away at 17

Studying Time: 2 minutes Aruturo Gatti Jr. — an aspiring boxer and…

3 Min Read

Oponion

EU Official Warns Elon Musk After Twitter Deal: ‘The Bird Will Fly by Our Rules’

EU Official Warns Elon Musk After Twitter Deal: ‘The Bird Will Fly by Our Rules’

TechThe platform will still be subject to the EU's forthcoming…

October 28, 2022

Senate Republicans plan to outdo Home with much more Medicaid cuts

Senate Finance Committee Republicans plan to…

June 17, 2025

Megan Fox: Machine Gun Kelly Should PROVE He Can Win Me Again

Studying Time: 3 minutes Megan Fox…

April 9, 2025

Perfumes Confirmed to Make Males Lean In—Irresistible Picks | Fashion

We independently consider all really helpful…

July 10, 2025

Document-setter: St. Ignatius star scores six TDs to beat Sacred Coronary heart Cathedral

Jarious Hogan had an evening he’ll…

November 2, 2024

You Might Also Like

WIRED’S Favourite PC Monitor Is  Off
Tech

WIRED’S Favourite PC Monitor Is $75 Off

In case you're uninterested in staring a tiny laptop computer display whereas working from residence, think about scooping up our…

3 Min Read
Your Cat In all probability Is not Ingesting Sufficient Water. A Fountain Can Assist.
Tech

Your Cat In all probability Is not Ingesting Sufficient Water. A Fountain Can Assist.

Evaluate Our PicksOthers We ExaminedCourtesy of PetkikPetkit Eversweet Max for $90: This techy computerized fountain will be both cordless or…

17 Min Read
Wish to Begin a Web site? These Are the Finest Web site Builders
Tech

Wish to Begin a Web site? These Are the Finest Web site Builders

Prime Web site BuildersFinest for Most IndividualsSquarespace CoreLearn ExtraFinest Low cost Web site BuilderHostinger Web site BuilderLearn ExtraFinest for Small…

5 Min Read
Specialised’s New Electrical Mountain Bike Is So Enjoyable I Forgot to Go House
Tech

Specialised’s New Electrical Mountain Bike Is So Enjoyable I Forgot to Go House

The following experience was on singletrack from my home to Spirit Mountain, Duluth’s downhill lift-accessed park with 24 trails starting…

4 Min Read
The Wall Street Publication

About Us

The Wall Street Publication, a distinguished part of the Enspirers News Group, stands as a beacon of excellence in journalism. Committed to delivering unfiltered global news, we pride ourselves on our trusted coverage of Politics, Business, Technology, and more.

Company

  • About Us
  • Newsroom Policies & Standards
  • Diversity & Inclusion
  • Careers
  • Media & Community Relations
  • WP Creative Group
  • Accessibility Statement

Contact

  • Contact Us
  • Contact Customer Care
  • Advertise
  • Licensing & Syndication
  • Request a Correction
  • Contact the Newsroom
  • Send a News Tip
  • Report a Vulnerability

Term of Use

  • Digital Products Terms of Sale
  • Terms of Service
  • Privacy Policy
  • Cookie Settings
  • Submissions & Discussion Policy
  • RSS Terms of Service
  • Ad Choices

© 2024 The Wall Street Publication. All Rights Reserved.

Welcome Back!

Sign in to your account

Lost your password?