7-2-19

“Half the Internet Was Down”: Cloudflare Tanks Resulting in Millions of Websites Being Shut Down

If you are wondering if a kill switch exists, we may have just seen evidence that the same thing could be accomplished by whatever it was that happened to Cloudflare on Tuesday morning when literally millions of websites just would not come up.  Among those were SonsOfLibertyMedia.com.

For about 30 minutes sites throughout the US and around the world simple went down.

According to Cloudflare CEO Matthew Prince, “Aware of major Cloudflare issues impacting us network wide. Team is working on getting to the bottom of what’s going on. Will continue to update.”

Network services provider Cloudflare is down for the majority of its users, bringing services like Discord, Marketo, Down Detector and DigitalOcean offline.

The full cause of the outage is unknown, with Cloudflare’s support page simply noting: “Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare.

“We are working to mitigate impact to Internet users in this region.”

Update, 14:15 UTC: “Cloudflare has implemented a fix for this issue and is currently monitoring the results.

“We will update the status once the issue is resolved.”

Update, 14:45 UTC: Services appear to be resuming. Cloudflare notes, however, that “Cloudflare Analytics processing is running behind. This affects timely delivery of customer data.

“We are working to mitigate impact.”

Update, 14:50 UTC:

Cloudflare explains: “We saw a massive spike in CPU that caused primary and secondary systems to fall over. We shut down the process that was causing the CPU spike. Service restored to normal within ~30 minutes. We’re now investigating the root cause of what happened.”

Update, 14:57 UTC:

“Cloudflare has resolved the issue and services have resumed normal operation.”

According to DigitalAttackMap, there was a lot of activity on the 1st, just ahead of #Cloudflare tanking.

No, it’s not just you: Half of the internet is down, including #Google, #Amazon, and #Reddit

Haki Social Media

This is the second time in two weeks this has occurred.  Again, from Data Center Dynamics:

A similar problem happened just last week, when Cloudflare was unavailable for roughly 1hr 42m.

That time, Cloudflare noted that the downtime was not its fault: “Earlier today, a widespread BGP routing leak affected a number of Internet services and a portion of traffic to Cloudflare. All of Cloudflare’s systems continued to run normally, but traffic wasn’t getting to us for a portion of our domains. At this point, the network outage has been fixed and traffic levels are returning to normal.

“BGP acts as the backbone of the Internet, routing traffic through Internet transit providers and then to services like Cloudflare. There are more than 700k routes across the Internet. By nature, route leaks are localized and can be caused by error or through malicious intent. We’ve written extensively about BGP and how we’ve adopted RPKI to help further secure it.”

In a blog post for DCD, Alex Henthorne-Iwane, of network monitoring company ThousandEyes, explained: “Route leaks happen fairly regularly. Routing is complicated. Humans make configuration errors, as we saw with the MainOne Nigerian ISP route leak that caused the global rerouting of some of Google’s traffic. And sometimes automation systems can trigger service disruptions, as we saw in the recent Google outage.”

Both ThousandEyes and Cloudflare pointed the blame at Verizon. Henthorne-Iwane said: “The fact that Verizon accepted and propagated a huge set of leaked routes from a downstream enterprise eyeball network turned what could have been an isolated issue into a significant outage.

“There has been some heated commentary on this topic from Cloudflare and other Internet observers, who have commented that Verizon should have caught this issue with one of multiple possible filtering mechanisms. Those critiques are more than fair on their face.”

This time around, however, it is unclear what was the cause of the outage.

Fortunately, things are back to normal, for the time being.

Sons of Liberty

#hacking #emp #terror #war #internetfail