Analysis: AboveNet attacks

[rob () ROBERTGRAHAM COM (Robert Graham)]

Analysis: AboveNet attacks
Original: http://www.robertgraham.com/op-ed/oped-abovenet.html

On April 25, '00, the web-hosting company AboveNet was taken down by a
"hacker" who disabled the company's network equipment. AboveNet has avoided
giving out details of this attack, citing an ongoing FBI investigation. I am
a customer of AboveNet; my systems have recorded extensive information
related to the attacks, and it is my profession to analyze such information.
This document contains my analysis of the event.

Public information can be read in the following articles:

http://dailynews.yahoo.com/h/cn/20000426/tc/abovenet_victimized_by_attack_4.
html
http://www.zdnet.com/zdnn/stories/news/0,4586,2555422,00.html
http://www.zdnet.com/zdnn/stories/news/0,4586,2556074,00.html
http://www.msnbc.com/news/399753.asp
http://www.computerworld.com/home/print.nsf/(frames)/000427D962?OpenDocument
&~f

What the outage was

According to AboveNet, a "hacker" took down backbone switches in the main
colos. A colo (collocation) is a building where customers put their
webservers. It is a better solution than locating the servers in their own
building because the colo company guarantees high-bandwidth and
high-availability. These "switches" are high-end Cisco boxes handling
gigabits/second of traffic. All the customers in a particular colo connect
through the same "aggregation" switch before going out to the Internet.

One of the articles above writes:

Vixie said the exploit had nothing to do with a defect in the switch. He
said the attacker exploited a flaw in the switch's configuration management
process that the company has since changed.

This rules out the most obvious form of attack: exploiting unpatched
vulnerabilities in vendor equipment. Such defects are regularly found in all
networking products. Customers don't apply the fixes as they should, leaving
the systems open for attack. There is a list of about 20 defects for the
Cisco equipment listed at http://www.SecurityFocus.com (among a thousand
more for other products, so you shouldn't interpret this as meaning Cisco is
more vulnerable than anybody else). Several of these defects are "topical",
so my normal inclination is to assume that the victim was late in applying
the recommended fixes. However, AboveNet claims that this was not the case
in the attack. They are instead saying that the intruder exploited a
security hole left open by their own people.

Case Study: SNMP

The most obvious way that the intruder could have penetrated the system was
through SNMP (Simple Network Management Protocol). AboveNet uses this
protocol extensively in order to manage their equipment. However, SNMP is
well known for its security deployment problems. Two common faults with
deploying SNMP are to leave the default passwords of "public"/"private"
enabled and accepting commands from the Internet (instead of just from local
machines).

These problems are extremely common. Elite hackers have used this knowledge
for years to play around with ISP equipment, such as the amusing feat of
telling the ISP to hang up on a dial-up user. Unlike the February DDoS
attacks, this type of SNMP hacking isn't widely done by script kiddies. This
technique was recently re-publicized, which makes it a prime candidate. The
main reason why this technique isn't used more often is the difficulty in
mapping out the ISP looking for equipment to play with.

In the case of AboveNet, they actually tell everyone the IP addresses of
their switches. They post to their website map the current status of all
their equipment and Internet connections. They essentially publicize where
to find the equipment and classify it in a well-known category of attacks
that might bring it down.

Here is a possible scenario. Somebody read the recent postings about the
SNMP vulnerability so it was fresh in their mind. While surfing AboveNet's
site, they stumbled across the SNMP traffic graphs with the IP addresses
labeled at the top. They opened their handy SNMP manager, typed in the IP
addresses and "poof" found they could log in. They then walked through the
configuration options until they found the item that would disable the
machine.

There are also more subtle ways of hacking SNMP. I run a website hosted by
AboveNet that runs an intrusion detection system (IDS). A month ago, the IDS
triggered an alert from a network management system that tried to log into
my machine using the password "AB-001". Even if AboveNet had correctly
changed their password from "private" to "AB-001", they carelessly told me
the new one.

According to AboveNet's comments, something similar may have happened with
Telnet, HTTP, or T/FTP. In all likelihood, the technique used probably
wasn't well known in the script kiddy world, but well known to anybody who
maintains such equipment.

Broadcast Domains

Like most colos, AboveNet puts their customers on shared broadcast domains.
While the switches isolate customers from seeing each other's primary
traffic, it forces customers to see each other's secondary broadcast
traffic. I host a site at AboveNet. I use the strong word "force" because
not only does AboveNet force this traffic onto my machine, it appears that
they are also billing me for it.

This is extraordinarily dangerous. If one customer gets hacked, the intruder
can then use that machine as a base of operations to hack other machines.
Network technologies are not protected against attacks across the local
broadcast domain. It affords the hacker new ways of attacking machines that
would not otherwise be possible remotely from the Internet.

By analyzing the traffic that AboveNet forces on my machine, I see the
following interesting possible items:

* ARP poisoning and ARP spoofing can be used to tell the upstream router to
redirect traffic. This means one customer can claim to be another customer's
website, or even sit between the user and website in order to steal
passwords and credit cards while relaying the traffic.

* ICMP redirects providing much the same access.

* Responding to the numerous DHCP and BOOTP requests in order to
misconfigure other customer's devices. Of course, I'm assuming the
DHCP/BOOTP broadcasts are actually from other customers and not from
AboveNet (in which case, the situation would be even more disturbing).

* Connecting to the numerous WinNT servers who are broadcasting the
availability of NetBIOS on private (192.168.x.x) addresses. Again, this is
an attack that only works locally.

* A major website (as listed in
http://www.mediametrix.com/Top500/Top500.html) uses WinNT clustering. I can
bring down the cluster with my own broadcasts or possibly insert myself into
the mix.

* Several nearby servers are announcing a likely vulnerability in their
"Insight" manager program.

* One customer is apparently announcing that it is the "default router",
which could confuse other people into relay their traffic through it.
Another customer is trying to use SNAP/LLC encoding rather than standard
DIX. I could probably exploit this and relay their traffic.

At minimum, I could bring down any of these machines by flooding them with
extremely high rates of traffic (i.e. 148,800 SYNs/second) . The February
attacks needed to be "distributed" because that was the only way to flood
traffic at those high rates. A LDoS (Local DoS) is much easier than a DDoS.

Op-ed

I spend a lot of time in colos. I can tell you from first hand experience
that colo network security is practically non-existent. I know of at least
one major competitor to AboveNet who I can switch off with a touch of a
button (not my own discovery, somebody else's). The situation in the
industry is truly frightening. What happened to AboveNet can happen to any
other colo at any time. My decision to host at AboveNet partially reflects
my belief that more secure than others, but you should interpret this as
meaning that I find them slight less lax/careless.

Here is a guide of some things that colos need to fix in their
infrastructures:

* Turn off Internet access to their network equipment. Right now, I can ping
the switch hosting my website from anywhere on the Internet. This is
unacceptable; the equipment should only be reachable from the company's NOC
(Network Operations Center).

* Restrict each customer to his/her own VLAN on their colo switches. This
feature is built into their equipment already. This will isolate customers
from each other so that they will have no more access to each other than
anybody else from the Internet.

* Obtain regular audits from security companies. This has been standard
practice in financial Accounting for eons, and should be used in security.
Publish the name of the security auditors. If you get hacked, rake the
auditor's name through the mud. Right now I'm finding gazillions of security
holes just by analyzing the traffic AboveNet shoves down my throat; imagine
what real auditors could find if they actively probed systems and reviewed
designs.

* Install intrusion detection systems (IDS) monitoring network traffic. Of
course, since I work for a company that builds what I believe to be the
highest performing such system, I'm highly biased on this issue.

An additional procedure needed by AboveNet:
Stop billing me for the broadcast traffic from other customers in the same
building. This is carelessness bordering on the fraudulent. [NB: According
to how I read the documents, AboveNet bills according to the greater of
inbound vs. outbound traffic, and the inbound traffic is about 10% higher
than our outbound, and consists mostly of local colo traffic. The machine in
question receives millions of hits per month, so the amount of traffic we
are talking about isn't small potatoes.]

Conclusion

AboveNet's SJ2 facility (where my equipment is located) is impressive. Their
bandwidth is extremely high with redundant links. Their physical security is
fairly tight with bank-like paranoia. They get their electricity from two
different grids, then cleanse it with their own generators. In the off
chance the Big One hits, they have commitments from diesel providers that
will keep their facility up and running for months.

In contrast, their network security is nearly non-existent. True, it is
slightly better than "industry norms", but industry norms suck right now. I
expect that other colos will start experiencing the same problem as AboveNet
until they start addressing the problems outlined above.