Re: Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

[Blake Hudson <blake () ispn net>]

>  I would
> > also suggest using stateless firewall rules and routing on your WAN
> > devices.
>
> That does seem to be the common wisdom. I'm actually not 100% sure 
> what we've got in line. It's OpenWRT based all around, so I'm sure 
> IPTABLES (and maybe even some ebtables).
iptables performs state tracking. So does pf in BSD. Sooner or later 
you'll run out of room in your state table. This is kernel tunable, and 
the OpenWRT guys have probably tuned for their needs, but their market 
is devices serving  a few users, not (several) thousands. Even a pfsense 
box with GB's of RAM caps at 500k simultaneous flows. I would plan on an 
average of 1000 flows per residential user. Most users will use less, 
some will use more, and some poor sob will get DOS'd and use 10's or 
100's of thousands. If I were to deploy CGN/stateful software I would 
keep it out of the core and either push it to the edge (user routers) or 
to a CGN appliance/cluster as a discrete entity in the network; I'd let 
the routers focus on routing and the switches focus on switching.

> I've got pretty much every Cisco router/switch in our lab, and an 
> EdgeRouter.
>
> What mikrotik should I evaluate?
>
> Our lab : https://commons.thefnf.org/index.php/FNF_Lab
>
>
> > If you've automatically discounted big name gear due to upfront costs,
> > you might consider buying from a used equipment reseller (I can
> > recommend a few, if needed).
>
> No. It's mostly for the customization/scripting etc. "SDN" and all 
> that jazz.  ;)

OK then. Just wanted to make sure you weren't excluding anything due to 
perceived budget issues. I'd think of a Cisco/Juniper/Brocade/whatever 
router as a special purpose server. You can use that Dell and OSS, but 
you've got a lot of extras in a Dell that can cause it to fail and you 
can't hot swap line cards, CPU's, etc in a Dell. I haven't used 
Mikrotik, but several of my clients, especially the ones involved with 
wireless, have been happy with the support and appliance options. They 
have the advantage of OSS without the disadvantages of a general purpose 
Dell/IBM/whatever server.

> > If you do need to use NAT, I feel like 500+ users sharing a single NAT
> > IP will result in poor quality of service and more admin overhead.
>
> Quite possibly. However if it's just for long tail v4 only sites, I 
> wonder how much it matters?
Probably depends on the amount of v4 traffic you have on your network. 
My guess is that v4 flows (not necessarily bits) will be the majority of 
your traffic for many years. Even services that primarily utilize v6 may 
still have v4 content. I believe v4 is and will continue to be of vital 
importance even after all of your users have working v6 connectivity and 
devices with good v6 support.

> > I didn't see it mentioned, where (and to whom) are you multihoming?
>
> Kansas City Kansas. Joesdatacenter.com is the current tower PoP. We 
> can get transit from him, of course peer with KCIX , and we'll 
> probably get transit from another local ISP in town (CTC). Of course 
> level3/att/vz et al are all in town/on net and just a very short fiber 
> hop away from Joes if we want to go that route.
>
>
>  Do
> > you have a good working relationship with these folks (cell phone,
> > email contacts that reach someone promptly)?
>
> Yes. Very much so.

Sounds like you have that covered.


> Will you be considered a
> > facilities based ISP (and subject to CALEA or other regulation)?
>
>
> I'm not sure. CALEA compliance is a very big deal for us. Especially 
> in regards to making an open doc about being compliant and any 
> necessary patches to the FLOSS supply chain for compliance.
Looks like something that  warrants more investigation.

> As far as documentation goes, we're working on a FLOSS book:
> https://commons.thefnf.org/index.php/Building_a_local_network_in_your_neighborhood 
>
>
> which will help folks build low cost community based access networks.
>
> We are all about building a (business/technical/operational) model 
> which can be readily and easily replicated by existing community based 
> organizations and not need to wait on muni networks (with all of the 
> complexity/risk/unknown unknowns etc that implies). The current bit 
> about cities having to ASK the federal govt (mother may I build an 
> ISP, even though the bullys have said I can't)? Are you kidding me? 
> What happened to techies banding together, getting some management 
> "bridge" types to organize the community and put up a network!

Let me know how it goes and if you need any help (I'm in Lenexa).

--Blake