Re: Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

[charles () thefnf org]

> I highly recommend pfsense for a firewall (been using pfsense and
> m0n0wall for years), but do have some concerns about using it at scale
> for (several) thousands of users.

So far it's gone fairly well for the existing subscriber base. The 
current service footprint is ~1k homes. I think it's running on a Dell 
Poweredge ~29xxish , don't know for sure.

 Most of this relates to NAT/State
> tracking, some of it hardware related, some of it software.

Right.

 If
> possible, I would suggest you obtain a routable IP address per user
> and avoid the pitfalls of NAT (I know at some point this may become
> expensive).

Exactly.

If you start with IPv6 from day 1 you are in a lot better
> place to encourage customers to upgrade to IPv6 capable gear.


Yes. We are doing v6 to every end user CPE. Absolutely. It will be 
there, be turned on and we hope to send all netflix/facebook/google etc 
traffic over v6. The v4 will be CGN. (We think we can only get a /24 
reasonably).

@Comcast v6 team (and really anyone who has a large dualstack network 
(*waves* at Owen),

So you guys have v6 turned up. You passed 1tb of traffic. Didn't comcast 
also write some floss code for CGN? So presumably you'll have to start 
doing CGN soon.

Thoughts on long tail v4 only internet being seriously degraded by large 
scale CGN? (Maybe that's a new thread?) If the major properties are v6, 
shouldn't that be enough to keep the support costs down? (My friends in 
the MMORPG "cloud gaming" space tell me that my approach could wreak 
havoc with many game engines).

Thoughts on what happens when you've got v6 at your door and v4 at your 
CO? Who is running a network like this today (I imagine most small ISPs 
will be in that boat soon)?

(And also, what's up with people complaining about ARIN fees?). The air 
fiber radios FNF is installing in KC cost 5k capex. So enough already 
about a ONE TIME 1k fee and get your v6 space! (I agree with the posters 
who said if you can't afford the arin fee, GET OUT OF BUSINESS).


 I would
> also suggest using stateless firewall rules and routing on your WAN
> devices.

That does seem to be the common wisdom. I'm actually not 100% sure what 
we've got in line. It's OpenWRT based all around, so I'm sure IPTABLES 
(and maybe even some ebtables).


 This should simplify the functions performed by these boxes
> to reduce the need to troubleshoot, apply updates, etc (resulting in
> better availability).

Yeah. Of course.



 I haven't used pfsense in an ISP WAN router
> capacity, and personally feel a router from Cisco, MikroTik, or
> Ubiquiti's EdgeOS devices, etc may be more appropriate in this role.


I've got pretty much every Cisco router/switch in our lab, and an 
EdgeRouter.

What mikrotik should I evaluate?

Our lab : https://commons.thefnf.org/index.php/FNF_Lab


> If you've automatically discounted big name gear due to upfront costs,
> you might consider buying from a used equipment reseller (I can
> recommend a few, if needed).

No. It's mostly for the customization/scripting etc. "SDN" and all that 
jazz.  ;)


> If you do need to use NAT, I feel like 500+ users sharing a single NAT
> IP will result in poor quality of service and more admin overhead.

Quite possibly. However if it's just for long tail v4 only sites, I 
wonder how much it matters?


 My
> gut feeling is that <50 may be more appropriate, depending on the
> quality of service you want to provide. This provides some headroom if
> one user makes many connections (p2p, virus infection, DoS attack) and
> also lessens the number of subs you need to look at in cases of abuse
> that are reported as an IP/port. Individual pfsense servers in a
> cluster may provide scalable CGN services. I'm not sure how you want
> to handle logging of all that data, but pfsense should allow you to
> define rules that allow stateless auditing (ip 1.2.3.4, ports
> 1000-2000 always NAT to sub A). The XML config file or possibly the
> shell is probably the easiest way to define such rulesets at scale.

Right right. I'm very familiar with the XML config and CLI. We've gotten 
to know pfSense well in our AutoTunnel (RADIUS) work. We patched (and 
released back to upstream) hostapd and other bits to actually correctly 
implement the RFC :D

So we've got a solution that is multi gateway. So based on the login 
creds you use, you get dropped into an appropriate vlan / BMX tunnel and 
get routed out the appropriate gateway.



> I didn't see it mentioned, where (and to whom) are you multihoming?

Kansas City Kansas. Joesdatacenter.com is the current tower PoP. We can 
get transit from him, of course peer with KCIX , and we'll probably get 
transit from another local ISP in town (CTC). Of course level3/att/vz et 
al are all in town/on net and just a very short fiber hop away from Joes 
if we want to go that route.


 Do
> you have a good working relationship with these folks (cell phone,
> email contacts that reach someone promptly)?

Yes. Very much so.

Will you be considered a
> facilities based ISP (and subject to CALEA or other regulation)?


I'm not sure. CALEA compliance is a very big deal for us. Especially in 
regards to making an open doc about being compliant and any necessary 
patches to the FLOSS supply chain for compliance.

As far as documentation goes, we're working on a FLOSS book:
https://commons.thefnf.org/index.php/Building_a_local_network_in_your_neighborhood

which will help folks build low cost community based access networks.

We are all about building a (business/technical/operational) model which 
can be readily and easily replicated by existing community based 
organizations and not need to wait on muni networks (with all of the 
complexity/risk/unknown unknowns etc that implies). The current bit 
about cities having to ASK the federal govt (mother may I build an ISP, 
even though the bullys have said I can't)? Are you kidding me? What 
happened to techies banding together, getting some management "bridge" 
types to organize the community and put up a network!