Insecure Internet Provider Paradigm

I was without internet access for about five hours two days ago. I had been remotely connected to work when my session disconnected. This event isn't as unusual as it should be, so I manually ended the broken VPN tunnel and tried to reestablish it. It was about that time that I noticed that my PC had lost internet access altogether. I checked a few other things, and I even noticed that my cell phone's internet (even though 4G was indicated) was unaccessible. My cell and my broadband are both AT&T (the map of the outage from and I was relegated to over-the-air TV, terrestrial radio, and my HAM radios.

I still had cell voice capability (not that I could get through to the now jammed AT&T support line) and SMS, and I validated with a coworker that our office was also having connection issues to some remote offices. This is particularly odd, as most large corporations purchase parallel internet feeds from competitors to ensure that there is a connection when one of them has a technical glitch. As the facts of the matter have filtered out, the few answers that we've been given raise more questions about the infrastructure that exists and what might happen in the event of a malicious attack on the US infrastructure. This outage of what AT&T claims is a single hardware failure really highlights the issues around the fake competition between the very, very few providers of broadband and what could result in relatively easy targeted attacks on US infrastructure.

For starters, where was the redundancy? Any network engineer will tell you that you want multiple paths to critical systems or inside complex networks so you don't wind up with a single point of failure. If a backhoe cuts a fiber line, you need to have other (possibly slower) lines that the data can go across, and the system needs to be able to sense the obstruction and react accordingly. Instead, some vague "hardware failure" happened, and people in at least four states were without internet access. We don't know if it was a tower that fell, a line that got cut, or even a power cord that got pulled out when someone tripped over it. What we DO know is that what we as customers expected to be a robust self-policing network turns out to be shut off at a single point. This is bad basic network design from the people who design networks for a living.

Second (and possibly worse), why were all of the other carriers also impacted? It's simple - while the internet providers balk at the idea of being referred to as "common carriers," they have effectively been acting like them the whole time. Why would Verizon bother to pull fiber where someone else already pulled some? Instead, they just contract with a competitor (referred to as a "vendor" or "partner") to provide service in the area. This means that those customers buying a connection from AT&T and then also buying a redundant connection from Verizon for fault tolerance essentially have twice the bill for half the service and a false sense of security about the whole thing.

So, what are the potential impacts of these kinds of issues? Well, for starters, the internet is being relied upon heavily for a lot of things (as you probably know) including things that could endanger human life. If you are planning an emergency surgery and need to have a doctor in another state watch over your doctor's shoulder or talk him or her through the procedure over video call, it would be wise to have more than one carrier providing internet. It would be unwise to think that was enough.

What if I was a malicious attacker? It is fairly well known that critical infrastructure systems are all over the place on the internet right now. What better way to disrupt emergency response after scripting an attack on all of the systems in an area than by bringing down the internet connections to all of them? If you can fire off a script to set all of the systems on a Stuxnet-like path of self-destruction and then cut those very systems off from remote access, think about the damage that could be done by all of the overloading boilers, chillers, generators, and every other piece of unsecured hardware sitting on the internet.

The fact that I was without internet for five hours is upsetting, but looking at the reasoning and the larger effects is even more upsetting. The fact that these kinds of peering agreements aren't widely disclosed and the impact of them is not even looked at for potential disaster shows a shortsightedness that we should all probably expect from companies that take our money and renege on promises. What we can do about it, however, appears to be either very little or absolutely nothing thanks to their close ties in congress (thank you Citizens United).


Popular posts from this blog

Elysian Dayglow IPA

The Purge (2013) Security System

I Spit on Your Grave 2 (2013)