RE: gigabit router (was Re: Getting a "portable" /19 or /20)

[Roeland Meyer <rmeyer () mhsc com>]

> From: Craig Partridge [mailto:craig () aland bbn com]
> Sent: Tuesday, April 10, 2001 4:36 PM
>
> In message 
> <Pine.BSF.4.21.0104101753540.98098-100000 () overlord e-gerbil net>, "R
> ichard A. Steenbergen" writes:
>
> > Don't be absurd, I can walk into fry's and pick up a motherboard with
> > 64bit/66mhz PCI, some Netgear GA620's, and all the other 
> > components for a 1GHz computer for under $1000.
>
> OK, so your bus has 4.2 Gb/s of bandwidth.  But, alas, you're in a PC
> so you have to copy each packet from the line card, into main memory,
> examine it, and push it back out to a line card.  So each 
> packet consumes
> twice its size in bus bandwidth.  So 2 1 Gb/s line cards will consume
> 4 Gb/s backplane.  Assuming you can run the PCI at full rate (which in
> my experience is a big big if), you can connect two Ethernets.

There are a few server-class motherboards that have dual PCI-66 buses. One
on each bus and there's no problem. Incidently, those same MBs usually have
dual or quad CPUs.

> Incidentally, this isn't the full story either.  You have to 
> do a route lookup on each of those packets.  That's typically 5 to 10
memory
> accesses... 5 memory access times 1 Mpps per gigabit times 2 gigabits
> is 10 million lookups per second or 100 ns per lookup.  Allowing for
> time spent getting through the chip to the pins, you probably 
> need 60 or 70ns DRAM, which is doable.  Except, oops!, that completely
consumes
> your memory bandwidth... where are you going to find the cycles to
> get the packets in and out?

Huh?! We haven't had the 60ns limitation for over three years. What
time-vault have you been living in? PC100 RAM is error correcting at 10ns
and PC133 runs at ~8ns. In fact, I challenge you to show me where you can
even buy 60ns RAM these days. That's right up there with 3 GB HDDs, not
available because they can't find customers for any thing that small/slow,
except propriatary applications (they make you pay ... large). I just paid
$104 for 256MB of ECC PC100 RAM, at the local computer store. The discount
wasn't even worth the gas/hr to drive down to San Jose for (1989 Chevy
Suburban, in San Jose rush hour).

> PS: Side note, this illustrates where router vendors earn their bucks.

Router vendors earn their bucks by building routing/switching appliances.
The fact that they manage to get corps to cough up $200K for $15K worth of
hardware (COGm, generous) is yet another issue. I expect to see some serious
price drops in the next 6 months. I also expect to see commodity hardware
start making some serious in-roads (cause of the price-drop).

COGr = 1/2 COGm, manufacturers cost of raw material (chips, labor, copper,
and such)
COGm = Cost of Goods manufactured (fully burdened)
2*COGm = The resellers cost (including mfg's margins)
4*COGm = The distributors price
8*COGm = Retail price (What you pay, full retail)

[yeah, that's right. This means that, said RAM, cost $13 to make. econ 101]

> Find a way to move data over each bus only once (double your 
> bandwidth!).

Dual PCI66 bus.

> Design your memory subsystems to keep packets and routing 
> data separate (increase your memory bandwidth!).  

PC133 ECC RAM (8ns).

> Find a processor that 
> doesn't waste cycles doing virtual memory (improve your memory access 
> times!).

Dual/Quad 1Ghz PIIIs, anyone?

> Oh yes, and then add hot board swap, 

How many routers actually do this? Even when they do, it is more prudent to
find the time to shut it down for 5 minutes. It's not like solid-state
devices die all the time...no moving parts. Keep them cool and they'll go
obsolete before they fail. Unless it's a combat situation (where you don't
have time to swap anyway) there isn't much that warrants this level of
maintenance. You're much better off with a redundant hot fail-over
configuration so that you can bring one down for PM and replacement/upgrade.

> a working BGP implementation (quick, where's Tony Li working these
days:-)), 

I actually like GateD. It ain't fancy, but it works.

> a CLI,

Linux is handy. RT Linux might be better (maybe).

> and a power subsystem for a CO,

If you must include that, then my choice is a Liebert GT3600.

> and you're in business.

With current commodity price drops, in less than 6 months, the cost of doing
this will fall well below $1000, retail. Considering potential market
volume. A 100% proprietary hardware solution may not be feasible, at that
price-point. They will *have* to use commodity hardware, just to keep up the
margins.