2006.06.07 NANOG-NOTES Lightning talk notes

["Matthew Petach" <mpetach () netflight com>]

(I think these were the toughest to take notes on, since they went
by so fast; took the most cleaning up afterwards.  But they were also
the best talks of the 3 days.  I wish we could have flipped, and taken
more time on Tuesday for them so we really could have dug in and
asked the questions we were itching to ask.  ^_^;  --Matt)


2006.06.07 Lightning talks

Marty Hannigan, Renesys:
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/1-hannigan.pdf

Critical infrastructure, root server location
analysis
Where to stick your servers.  :)
he took some public info out there on root-servers.org
talked to some people, extrapolated from larger set of
data.
operator demographics.
in  US:
3 corp a, c, j
2 edu b and d
1 mil g
2 research e/h
3 nonprofit f, i, l
autonomica is responsible for l, but hosts "some"
instances on a CDN; CDN is a US formed entity
in EU:
1 non profit k
asia/japan:
1 nonprofit m

92% of system operated in US, 8% non-us;
5% margin of error +-.
US entity type
non-us 8%
us corp 39%
us mil 23%
us edu 15%
us nonprofit 15%

where?
in 54 countries
all religions
all methods of governance

politically:
79% are democratic governments
21% in other forms of government

global diversification for security and performance
instances spread across continents
different networks
different proceedures
different software
different hardwware
different weaknesses
 weaknesses become strength, since they are diverse;
 no one weakness knocks out all servers.
 little less open to insider malfeasance

Global distribution
NA 38%
EU 35%
Asia 12%
AUs 8%
east EU 3%
LA 2%
Africa 2%
ANT 0%

getting reasonable coverage in the world

situating a root server
relationship 101
who you know
 ICANN, operator, IX, and RIR relationships
 regulators
how you spin it
 national pride
 performance and security
 betterment of user experience

Threats
no different from anyone else
 direct attacks
 proxy attacks
 botnets
  easy money
  miscreants masking other activities

Not sure what motivations to attack root servers;
can't extort money from nonprofits

let's attack a root server
target $-root
 location; eu hosting facility
 multi-post cabinet config with cabling and power
  under floor
 unlocked cabinet, single factor facility entry
physical attack
  open cabinet door
  access to power
hijack attempt
 advertise a route
 return bad answers
network attack
spoof source
random host queries
packet floods

summary:
root system is less likely to be subject to insider
 attack or weakness
but can be attacked by layer 3
there is likely good resarch data coming across those
 interfaces
trend towards a collapsed root system, where root and
TLD share same hardware or networks should be more
closely examined.
slides will be up soon, talk to him in the hallway


NEXT, Anton Kapela
Network RTTs
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/2-kapela.pdf

I'm pinging 10: high rate active probes
we're pinging stuff really quickly
adjusted host kern.hz to 1000 select() gets pretty
 accurate +-1ms emmission accuracy
stuff is responding
Interesting 0.001% of data relates to end-to-end queuing

what has been sampled?
some cisco 7513s
IOS 12.3 mainline
linux 2.4.20
freebsd 4.8
NT4 sp6
various end-to-end paths on u-wisc network

raw data isn't terrible interesting.
in adaptive link layer protocols, see rate shifting
manifested in RTT
wireless, HPNA/HCNA, powerline ethernet
10,30,60,90 second peaks

fourier transforms, wavelet transforms, frequency domain
1000 seconds at 10ms intervals
break into composite, aggregate graph at top,
0-50hz span on x axis, y axis is contribution
summary of entire graph.
bottom right graph is rough 200 samples of a
range from 0-5hz, 100pps, deduce delay at half
that sampling rate.

delay is not a simple boring thing; has
scheduler delays, path dynamics not visible
before to see queue depths.

shark fins showed up; congestion events do
occur, are quite measurable.
when links are hot, queues are obvious, esp. on
highly multiplexed links.

bottom left, cubic resonance, several tens of
thousands of multiplexed flows hitting odd
resonance.

pinging windows machine, composite spectral
fingerprint; 10,20,25,30 spikes
Linux fewer spikes
freebsd low and flat
IOS is 10, 20, 30 and grass of 1hz spacing
below 10hz.

win32 delay spectrum also has 1hz fuzz below
10hz.

Sampled RTT and performed signal analysis of it;
now what?
is network time continuous? is round trip time
discreet or continous?
no changes in revealed as you go down lower
is delay a "signal' anyway
what's with the 0 hz DC component in the FT output?

could this be used for fingerprinting?
yes, could be like next nmap.
packet-level fingerprinting is trivial to fake; but
IP stack scheduler behaviour doesn't change so
easily.


NEXT:
Mikael Abrahamsson
Affect on traffic from the TPB bust
with Kurtis Lindqvist
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/3-abrahamsson.pdf

Bittorrent background
p2p protocol for filesharing.
text string, upload to tracker, get IPs of other clients
that have done the same thing, clients connect to each
 other, develop a swarm.
clients communicate even when tracker vanishes.
 just can't get new clients joining

Thepiratebay.org
run by a handful of individuals aged 22 to 28
used ~100mb at peak
peaked at 2M concurrent users
stats code in tracker indicated that total p2p
 traffic was close to 100gig/sec
thus far, largest bittorrent site/tracker in world

photo slide showing the physical gear
10 high-end small servers in half a rack in stockholm,
sweden
web frontends, db servers, trackers

on the stats
not an exact science
at least a german ISP had an outage at the same time
bust was around 12.00 CET may 31st (euro time)
data collected from Euro-IX members

some saw no difference.
Netnod aggregated, biggest drop, about 10+Gb drop
very quickly
in Netnod stockholm *very* visible.
stats server was slashdotted, lost an hour of stats.
LINX London, saw about 5Gb drop out of 80Gb
AMSIX dropped about 5Gb out of 160Gb

DECIX frankfurt, germany, drop before noon,
FCIX, helsinki, Finland
drop fairly visibile
NIX, in norway, drop also visible.
doesn't show private exhanges/private peerings
Brussels (BNIX) also saw drop.

netflow export from big US ISP,
large chunk of bittorrent traffic packets faded off.

Thepiratesbay.org was back online 72 hours later in
Amsterdam, Netherlands
and traffic started coming back
June 6th is a holiday, watch the stats this coming
week.  :)

Aftermath
Police took ALL hosted equipment at the same site
by the same hosting company (small one, only a few
racks), caused quite a few community web sites to
go down plus commercial customers
Has spawned a lot of discussion in Sweden regarding
all issues involved.  Front page material every
day, even video surveillance of the raid from
surveillance cameras has been posted on youtube.com
Accusations of police/politicians being influenced
by White House and MPAA and others

Q: Bill Norton: what about other tracker sites, why
didn't traffic just shift to them?
A: some did, but torrent files have the tracker hard
coded in them, so they can't just flip over to other
tracker sites on their own.

Q: Roland Dobbins, back up in several countries now
including Russia, is traffic back?
A: Keep watching the graphs.
And if you want to see the bust, search for
"pirate bay" and "police", there's one link on youtube.


NEXT:
Alex Pilosov/Pilosoft
Adam Rothschild/Voxel
Nathan Patrick/Sonic.net
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/4-pilosov.pdf

Passive Metro WDM
how it works
single mode fiber: mutiple wavelengths
also called "colours" or "lambdas"
coexisting separately
pluggable optics as enabler
low cost for passive optical equipment, particularly
grey market

Dark fiber IRUs are very cheap.
low opex/capex

how does it work?
O band Original     1260 - 1360
E band Extended     1360 - 1460
S band Short        1460 - 1530
C band Conventional 1530 - 1565
L band Long         1565 - 1625

implementation options
active WDM cisco 15xxx, cienna, movaz, others
passive WDM using optical filters
 self-assembled patch panels
 complete systems (CUBO)

pictures of components

GWDM/WWDM
wideband multiplexing (1350/1550)
2GE fdx per pair, 1 GE fdx per strand
single strand networking the receiver is *always*
wideband
low cost for transcievers (LX/ZX, <$500)
10GE possible (ER/LR)

Active xWDM
beyond this scope
everyone knows how to do it, it just costs more.

Passive CWDM
wavelength, wide channels, 8 channels
1470-1610 conventional
1270-1470 low range
cost is cheap ~$1000 per strand per end for
CUBOs,
$300-$1000 per GBIC depending on quality
(CUBO, Taiwanese hw manufacturers)
no Xenpaks, GBICs only
20nm channel spacing
low availability on 'low range' GBICs/SFPs

Passive DWDM
each channel is narrow
0.8nm == 24 dense channel per single coarse channel
160 channels easily
25Ghz spacing
research at 12.5Ghz
Xenpaks available $9k+
few GBICs at $1500+

Filters:
build/add as you grow by mixing and matching
available in various ranges (center wavelength,
bandpass width)
Going from GWDM to GWDM/CWDM to GWDM/CWDM/DWDM

Testing and management
optical power meter
communication is key
OOB access: HOOTS, cell phone
you need to talk site-to-site to coordinate
 make sure cell phones don't depend on fiber
optical power monitoring/APD receivers in GBICs
(show interface blah trans)
spectrum analyzer

Caveats
few complete commercial systems available
systems require clue and duct tape to put together
need to tune with attenuators if signal is too
strong, attenuators differ with wavelength
flaky GBIC/SFP vendors
small-time passive optical vendors
expensive equipment for testing (spectrum analyzer,
light sources, etc)
lack of operational expertise (get hit by a bus)

Exotic options
Circulators (same wave both ways)
Interleavers (half the light, double the waves)
CWDM light into DWDM channel (similar to above)
10GE LX4/LR multiplexing

Simple Economics
2GE GWDM ~$1k
8GE CWDM ~$5k-10k
2*10GE ~$5k-10k
N*10GE DWDM ~N*$10k
prices include passive and active components,
per end, fdx over one pair
Prices an order of magnitude lower than commercial
systems from Cienna, Cisco.

List of vendors
Cloudy YAYA, Orient DONG,
[lots of names on slide, go read it yourself]

Questions?  mail them!
alex () pilosoft com
asr () voxel net
np () sonic net

Q: Martin, what do you about timing?
A: No need for timing, each channel is separate,
 no timing needed to run this.

Q: mike hughes, linx; one thing to look at if you're
looking at GWDM/WWDM, or going bidir on one strand,
watch out for back reflections--running several channels
bidir would see itself reflected back, would declare
linkup
A: don't run two waves bidir on it--just don't do it,
it's not worth it, it's too ghetto.


NEXT:
Mohit Lad
Alerting prefix owners of hijacks in near-realtime
UCLA, joint work with a bunch of other names
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/5-lad.pdf

PHAS project?
Three properties of a security solution
ability to see "bad" information
ability to distinguish between "good" and "bad" info
incentive to fix the problem

The PHAS (prefix hijack alert system) approach
use updates from existing BGP monitors (route views and
RIPE RIS)
if false origination, send notification.
push complexity of detection to user
look at email registration to decide who is allowed to
 announce prefixes.
don't filter out false vs real changes.

PHAS origin monitor
131.179.0.0/16, UCLA block

recommend multiple email addresses, including some that
are *not* on your blocks!

Message Delivery
apply local rules before generating alarms
you shouldn't recieve duplicates of notifications
due to topological mesh-ness, it's difficult for a
hijacker to get all notifications for a block.

Evaluation: messages per AS
Dec 2005
map prefixes to origin AS using routing table
most AS receive less than 100 messages per month
 most less than 10
local filters can limit legitimate origin changes.

readily deployable
 routeviews and RIPE RIS already collect data
alarm generation not dependent on
cooperation from other networks
monitoring or knowing correct origins
alarm authentication: single source
low overhead.

summary
comprehensive study using archived data
developing near-realtime system
interested in receiving notifications

send email to:
 mohit () cs ucla edu
 massey () cs colostate edu
ongoing efforts
covered prefix hijack
false last hop
reference:
PHAS: usenix security 2006.

Q: Danny McPherson--that's associated with origin AS,
and origin AS could be spoofed, does it look at
combination of prefix, origin, and next hop up?
A: they are doing it on origin AS and next hop,
they'll do some more thinking about that case.


NEXT:
Rick Wesson, Support Intelligence [hehe]
Understanding abuse, aggregate it, push it back to
operators, let them know what they're doing to other
people.
[no slides, he does a live presentation of his tool]

How do I believe you?
realtime data visualization, Feb 8th, 2006
visualization.
130 different data sources, 90% passive;
10,000 domain aggregated spam trap, very
evil SMTP that filters and bans IP for some time.
1.2million events per day aggregated, about 700,000
unique IPs for the global internet.
BGP peers, aggregate based on announcements made.
Put into tool so network operators can visualize
their prefixes, drill in, and see abuse each
prefix generates.
hover over point, it shows the operator, IP address,
and what the problem was (spam, insecure web server, etc)

This shows problem areas that need to be addressed!
disseminate this information, help ISPs clean up their
networks.

Can also pass along information of abuse that has
happened to you.
If you have an AS, he can tell you what your AS has
been used for, abused for, owned, etc.

email him for more info...except he didn't list
his email info. ^_^;

Break!  short!