{"id":11219,"date":"2022-07-14T06:05:19","date_gmt":"2022-07-14T06:05:19","guid":{"rendered":"https:\/\/zeru.com\/blog\/?p=11219"},"modified":"2022-08-16T09:51:59","modified_gmt":"2022-08-16T09:51:59","slug":"facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool","status":"publish","type":"post","link":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool","title":{"rendered":"Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_43 counter-flat ez-toc-counter ez-toc-light-blue ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-69f39669d9773\" class=\"cssicon\"><span style=\"display: flex;align-items: center;width: 35px;height: 30px;justify-content: center;direction:ltr;\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/label><label for=\"ez-toc-cssicon-toggle-item-69f39669d9773\"  class=\"cssiconcheckbox\">1<\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-69f39669d9773\" ><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\/#Facebooks_Outage_May_Have_Been_Caused_by_a_Bug_in_an_Audit_Tool\" title=\"Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool\">Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\/#Bug_in_audit_tool\" title=\"Bug in audit tool\">Bug in audit tool<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\/#DDoS_attack\" title=\"DDoS attack\">DDoS attack<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\/#Maintenance_mistakes\" title=\"Maintenance mistakes\">Maintenance mistakes<\/a><\/li><li class='ez-toc-page-1'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\/#Lack_of_multi-location_networks\" title=\"Lack of multi-location networks\">Lack of multi-location networks<\/a><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"Facebooks_Outage_May_Have_Been_Caused_by_a_Bug_in_an_Audit_Tool\"><\/span>Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<p><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/foiuWLL.jpg\" alt=\"what caused facebook outage\" \/><\/p>\n<p>Facebook&#8217;s outage lasted globally for six hours on Monday and Tuesday. It&#8217;s now restoring services worldwide. The cause is a configuration change on backbone routers, which coordinate network traffic between the company&#8217;s data centres. This configuration change cascaded throughout the network, causing the entire website to become unavailable. While the outage may have been a single problem, it has implications for the overall health of the company&#8217;s entire network.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Bug_in_audit_tool\"><\/span>Bug in audit tool<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It appears that a bug in an audit tool may have caused the Facebook outage. Facebook has stated that a bug prevented it from stopping a command. Because the audit tool is used to track all of the user interactions on its network, it is important to identify the exact cause and the root cause should be fixed as soon as possible. But the problem is not as simple as a bug in the audit tool. Facebook&#8217;s systems are designed to catch mistakes and identify the source of the problem as quickly as possible.<\/p>\n<p>The outage began when Facebook engineers issued a command during a routine maintenance job. The command caused Facebook&#8217;s data centers around the world to go down. The outage was caused by a bug in a program audit tool, which failed to stop a command. The outage left employees without access to email, internal tools, and even work passes. Some reports also claim that the Facebook headquarters were in &#8220;meltdown mode.&#8221;<\/p>\n<h2><span class=\"ez-toc-section\" id=\"DDoS_attack\"><\/span>DDoS attack<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The outage affected millions of Facebook users, which accounts for a substantial chunk of the Internet&#8217;s traffic. If the DNS servers for Facebook had become inaccessible, the DNS providers and ISPs would have taken a beating. Even if the DNS servers were still in place, greater volumes of queries would have flooded DNS providers and ISPs. As DNS is resilient, the resolvers would have tried to resolve to another nameserver, which would have resulted in quadrupling the DNS query volume.<\/p>\n<p>Although the outage was unprecedented, the cause of it is still unknown. Facebook has admitted that the outage was a mistake, but it&#8217;s not clear what the root cause of the problem is. The company does not say how many users were affected by the outage or when the problem would be resolved. However, a DDoS attack can be traced to several sources. Several researchers have tracked the attack since it began in November 2016.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Maintenance_mistakes\"><\/span>Maintenance mistakes<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The outage caused problems for users as well as employees, and is still being investigated, but the root cause is a simple mix of DNS and BGP configuration mistakes. Facebook has no way of telling exactly how many people were affected, but it is likely that thousands of people were inaccessible for some time. Facebook&#8217;s outage also put employees on the front lines of the issue. The company tries to avoid such incidents as much as possible, but this time, it failed to take preventative action.<\/p>\n<p>The cause of the Facebook outage was a simple configuration change by an engineer who was fat-fingered and sent the wrong command, setting off a series of technical issues. The engineer failed to check the status of the change with an internal auditing tool, triggering a cascade of problems. The command that failed was subsequently executed across Facebook&#8217;s backbone routers, affecting the company&#8217;s entire network and rendering engineers&#8217; access cards useless.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Lack_of_multi-location_networks\"><\/span>Lack of multi-location networks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>An outage on Facebook was due to a BGP update gone wrong, and it prevented remote users from reverting the change. Because the BGP server cannot reach the internet, people with network, physical and logical access were unable to access Facebook. At the time, it looked like a DNS issue. But later reports stated that the cause was a network outage. In this article, we&#8217;ll take a closer look at the issues surrounding Facebook&#8217;s outage.<\/p>\n<p>The outage was due to problems with Facebook&#8217;s Workplace system, which three-quarters of its employees use. Without this system, Facebook&#8217;s technical team was cut off from help in its main data center. The team had trouble accessing secure rooms and reverting faulty configurations on key systems. The outage even brought down the messaging service used by remote-working staff, making it more difficult to reach them to find the problem.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool Facebook&#8217;s outage lasted globally for six hours on Monday and Tuesday. It&#8217;s now restoring services worldwide. The cause is a configuration change on backbone routers, which coordinate network traffic between the company&#8217;s data centres. This configuration change cascaded throughout the network, causing [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":52160,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v19.7 (Yoast SEO v21.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Facebook&#039;s Outage May Have Been Caused by a Bug in an Audit Tool - Zeru<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Facebook&#039;s Outage May Have Been Caused by a Bug in an Audit Tool\" \/>\n<meta property=\"og:description\" content=\"Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool Facebook&#8217;s outage lasted globally for six hours on Monday and Tuesday. It&#8217;s now restoring services worldwide. The cause is a configuration change on backbone routers, which coordinate network traffic between the company&#8217;s data centres. This configuration change cascaded throughout the network, causing [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\" \/>\n<meta property=\"og:site_name\" content=\"Zeru\" \/>\n<meta property=\"article:published_time\" content=\"2022-07-14T06:05:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-08-16T09:51:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/zeru.com\/blog\/wp-content\/uploads\/Facebooks-Outage-May-Have-Been-Caused-by-a-Bug-in-an-Audit-Tool_11219.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"801\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Lizzie Yates\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Lizzie Yates\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\",\"url\":\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\",\"name\":\"Facebook's Outage May Have Been Caused by a Bug in an Audit Tool - Zeru\",\"isPartOf\":{\"@id\":\"https:\/\/zeru.com\/blog\/#website\"},\"datePublished\":\"2022-07-14T06:05:19+00:00\",\"dateModified\":\"2022-08-16T09:51:59+00:00\",\"author\":{\"@id\":\"https:\/\/zeru.com\/blog\/#\/schema\/person\/61005d9ec00b94bc50fbaf11b78aa55e\"},\"breadcrumb\":{\"@id\":\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/zeru.com\/blog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/zeru.com\/blog\/#website\",\"url\":\"https:\/\/zeru.com\/blog\/\",\"name\":\"Zeru\",\"description\":\"Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/zeru.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/zeru.com\/blog\/#\/schema\/person\/61005d9ec00b94bc50fbaf11b78aa55e\",\"name\":\"Lizzie Yates\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/zeru.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/zeru.com\/blog\/wp-content\/uploads\/19-150x150.jpg\",\"contentUrl\":\"https:\/\/zeru.com\/blog\/wp-content\/uploads\/19-150x150.jpg\",\"caption\":\"Lizzie Yates\"},\"description\":\"A content marketing strategist with the Zeru team for a little over 5 years, Lizzie Yates specializes in everything digital media with a particular focus on social media and technology. Her passion? To follow how the social media sites like Instagram, YouTube, Facebook, Twitter, and TikTok are maturing over time, and what businesses can do to keep up. She shares her insights on our blog in a true outpouring of knowledge and expertise. Her knowledge about technology and social media is vast, and she is always willing to share her insights with businesses to help them stay up-to-date with the latest trends.\",\"url\":\"https:\/\/zeru.com\/blog\/author\/writer\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Facebook's Outage May Have Been Caused by a Bug in an Audit Tool - Zeru","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool","og_locale":"en_US","og_type":"article","og_title":"Facebook's Outage May Have Been Caused by a Bug in an Audit Tool","og_description":"Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool Facebook&#8217;s outage lasted globally for six hours on Monday and Tuesday. It&#8217;s now restoring services worldwide. The cause is a configuration change on backbone routers, which coordinate network traffic between the company&#8217;s data centres. This configuration change cascaded throughout the network, causing [&hellip;]","og_url":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool","og_site_name":"Zeru","article_published_time":"2022-07-14T06:05:19+00:00","article_modified_time":"2022-08-16T09:51:59+00:00","og_image":[{"width":1200,"height":801,"url":"https:\/\/zeru.com\/blog\/wp-content\/uploads\/Facebooks-Outage-May-Have-Been-Caused-by-a-Bug-in-an-Audit-Tool_11219.jpg","type":"image\/jpeg"}],"author":"Lizzie Yates","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Lizzie Yates","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool","url":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool","name":"Facebook's Outage May Have Been Caused by a Bug in an Audit Tool - Zeru","isPartOf":{"@id":"https:\/\/zeru.com\/blog\/#website"},"datePublished":"2022-07-14T06:05:19+00:00","dateModified":"2022-08-16T09:51:59+00:00","author":{"@id":"https:\/\/zeru.com\/blog\/#\/schema\/person\/61005d9ec00b94bc50fbaf11b78aa55e"},"breadcrumb":{"@id":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/zeru.com\/blog\/facebooks-outage-may-have-been-caused-by-a-bug-in-an-audit-tool#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/zeru.com\/blog"},{"@type":"ListItem","position":2,"name":"Facebook&#8217;s Outage May Have Been Caused by a Bug in an Audit Tool"}]},{"@type":"WebSite","@id":"https:\/\/zeru.com\/blog\/#website","url":"https:\/\/zeru.com\/blog\/","name":"Zeru","description":"Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/zeru.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/zeru.com\/blog\/#\/schema\/person\/61005d9ec00b94bc50fbaf11b78aa55e","name":"Lizzie Yates","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/zeru.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/zeru.com\/blog\/wp-content\/uploads\/19-150x150.jpg","contentUrl":"https:\/\/zeru.com\/blog\/wp-content\/uploads\/19-150x150.jpg","caption":"Lizzie Yates"},"description":"A content marketing strategist with the Zeru team for a little over 5 years, Lizzie Yates specializes in everything digital media with a particular focus on social media and technology. Her passion? To follow how the social media sites like Instagram, YouTube, Facebook, Twitter, and TikTok are maturing over time, and what businesses can do to keep up. She shares her insights on our blog in a true outpouring of knowledge and expertise. Her knowledge about technology and social media is vast, and she is always willing to share her insights with businesses to help them stay up-to-date with the latest trends.","url":"https:\/\/zeru.com\/blog\/author\/writer"}]}},"_links":{"self":[{"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/posts\/11219"}],"collection":[{"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/comments?post=11219"}],"version-history":[{"count":2,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/posts\/11219\/revisions"}],"predecessor-version":[{"id":52159,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/posts\/11219\/revisions\/52159"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/media\/52160"}],"wp:attachment":[{"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/media?parent=11219"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/categories?post=11219"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zeru.com\/blog\/wp-json\/wp\/v2\/tags?post=11219"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}