Inside the Facebook blackout: How the social media giant lost $7bn in a day
On Monday, around 10 PM BST, Facebook and its other services--including WhatsApp and Instagram--went completely down, in a massive outage that's thought to be the worst since 2019. The websites were restored nearly 12 hours later, after an unprecedented interruption in global communications and the internet economy.
As netizens start to check back in their regular habitats, new data and reports offer insights into what went down on Facebook during the complete social media blackout.
What really happened?
According to a corporate statement by Santosh Janardhan, VP of Infrastructure at Facebook, "configuration changes on the backbone routers that coordinate network traffic between our data centres" are to be blamed for the massive outage. "This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt," wrote Janardhan. "We want to make clear at this time we believe the root cause of this outage was a faulty configuration change."
While Facebook officially blamed the outage on its configurations changes to its routers, many experts suggested that a regular update to Facebook's Border Gateway Protocol (BGP)--a map that directs Internet service providers all over the world to a particular website-- had caused the temporary outage.
To put it simply, during one of its regular updates Facebook may have removed the map that directed the world's computers to its websites. As a result, typing Facebook.com into a web browser caused browsers to return an error page because they had no idea where to find Facebook.com.
Inside Facebook headquarters, the outage nearly broke off all internal communications between the employees. The Verge reported that because Facebook requires employees to log in with their work accounts in order to access tools like Google Docs and Zoom, those services were also unavailable.
According to reports, to restore the service, Facebook engineers were quickly dispatched to one of its main US data centres in California, which indicated that a remote fix wasn't possible.
With Facebook, Instagram and WhatsApp down, users turned to Twitter and other social media outlets for the time being. Particularly on Twitter, memes and trolls overflowed with many brands joining the bandwagon.
In Bangladesh, increased activity was seen in almost forgotten apps like Viber and Telegram as users resorted to alternative social media channels.
Impact on the economy
In the nearly 12 hours long outage, Facebook's stocks dropped by about 5.5 per cent, and Mark Zuckerberg, Facebook CEO, lost nearly $7 billion. The outage had also cost the global economy about $160 million per hour, according to NetBlocks, a website that tracks internet outages and their impact on the global economy.
Elsewhere, small businesses and advertisers that relied on Facebook and its services, took a massive punch in the gut. Around the world, more than 6 million advertisers use Facebook to advertise their products or services. Many of these advertising models, including those with influencer campaigns, use specific time frames to post about a certain product or service.
The 12-hours outage of Monday meant a 12-hours window of total blackout where those products and services were not displayed or advertised to audiences, causing many companies, particularly small businesses, to incur massive losses.
Users affected in Bangladesh
In Bangladesh, Facebook and WhatsApp are synonymous with the internet. Facebook, in particular, is the gateway to a vast world of 'F-commerce' in the absence of reliable e-commerce services in the country.
From tee-shirts to furniture, online stores, large and small, make up for at least 300,000 such businesses that rely solely on Facebook for conducting their business. From showcasing products to placing an order to delivery updates and payment terms, over 80% of internet commerce in Bangladesh takes place through Facebook.
In the outage, many small businesses felt the heat. Naim Hasan, who runs a Facebook-based leather products store called House of Leathers, mentioned that he generates around BDT 70,000 of revenue per month through his Facebook business.
"As a Facebook-based store, outages like these are major problems for us. We faced delays and setbacks in orders and customer queries. Luckily, the outage was late at night and we could quickly rebound the following day when the services were restored," he said.
But not all were as lucky as Naim. Nadia Jahan, who works as a marketing executive at a digital advertising agency that handles the social media accounts for multiple local brands, said that the outage caused huge setbacks in their usual operations.
"As a digital agency, we almost work round-the-clock. We usually have a fixed social media calendar and everything gets posted according to schedule. Due to the outage, our entire calendar had to be shifted and now we are running behind schedule." said she. "We were also running active Facebook ad campaigns which no one saw or interacted with during the outage period, obviously causing quite a monetary loss," she added.
Possibility of future outages
Facebook is the largest social network in the world, with 2.89 billion monthly active users as of the second quarter of 2021. And although this outage was a rare occurrence, it cannot be claimed with certainty that it will never happen again. However, there is currently no evidence to suggest that any user data was compromised, at least no more than when Facebook is up and running.
Following the initial explanation, Facebook has released another, more detailed update, on the issue. The following is an excerpt from the recent blog post by Facebook's VP of Infrastructure:
"This outage was triggered by the system that manages our global backbone network capacity. The backbone is the network Facebook has built to connect all our computing facilities together, which consists of tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers.
Those data centers come in different forms. Some are massive buildings that house millions of machines that store data and run the heavy computational loads that keep our platforms running, and others are smaller facilities that connect our backbone network to the broader internet and the people using our platforms.
When you open one of our apps and load up your feed or messages, the app's request for data travels from your device to the nearest facility, which then communicates directly over our backbone network to a larger data center. That's where the information needed by your app gets retrieved and processed, and sent back over the network to your phone.
The data traffic between all these computing facilities is managed by routers, which figure out where to send all the incoming and outgoing data. And in the extensive day-to-day work of maintaining this infrastructure, our engineers often need to take part of the backbone offline for maintenance — perhaps repairing a fiber line, adding more capacity, or updating the software on the router itself.
This was the source of yesterday's outage."