Re: Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

[Blake Hudson <blake () ispn net>]

charles () thefnf org wrote the following on 7/23/2014 11:58 AM:
> This is a greenfield network. We've got Ubiquiti gear for the 
> backbone. Running a mix of QMP routers with BMX6 as the IGP linked 
> over AirOS l2 bridge "pseudowires". We'll be homed to two AS 
> upstreams. Using pfSense as the WAN edge routers.
>
> From all my reading of the list, it seems like key things to do in 
> this scenario:
>
> 1) Have full flow telemetry at all points to help with (D)DOS mitigation.
> 2) Do CGN in pools (so perhaps ~500 to 1k users behind each IP)?
> 3) Provision a /56 of v6 space to each end user. I was thinking of 
> having the CPE with CeroWRT and be multi SSID with a /64 per. I'm 
> interested in folks thoughts on this?
> 4) Upsell a public v4 address if someone requires it
> 5) Of course implement bcp38
>
> I'm mostly interested in technical feedback. Business model etc type 
> feedback is welcome as well, but not the primary purpose of this 
> message. :)

Charles, it sounds like you've got a lot of the technical items on your 
radar.

I highly recommend pfsense for a firewall (been using pfsense and 
m0n0wall for years), but do have some concerns about using it at scale 
for (several) thousands of users. Most of this relates to NAT/State 
tracking, some of it hardware related, some of it software. If possible, 
I would suggest you obtain a routable IP address per user and avoid the 
pitfalls of NAT (I know at some point this may become expensive). If you 
start with IPv6 from day 1 you are in a lot better place to encourage 
customers to upgrade to IPv6 capable gear. I would also suggest using 
stateless firewall rules and routing on your WAN devices. This should 
simplify the functions performed by these boxes to reduce the need to 
troubleshoot, apply updates, etc (resulting in better availability). I 
haven't used pfsense in an ISP WAN router capacity, and personally feel 
a router from Cisco, MikroTik, or Ubiquiti's EdgeOS devices, etc may be 
more appropriate in this role. If you've automatically discounted big 
name gear due to upfront costs, you might consider buying from a used 
equipment reseller (I can recommend a few, if needed).

If you do need to use NAT, I feel like 500+ users sharing a single NAT 
IP will result in poor quality of service and more admin overhead. My 
gut feeling is that <50 may be more appropriate, depending on the 
quality of service you want to provide. This provides some headroom if 
one user makes many connections (p2p, virus infection, DoS attack) and 
also lessens the number of subs you need to look at in cases of abuse 
that are reported as an IP/port. Individual pfsense servers in a cluster 
may provide scalable CGN services. I'm not sure how you want to handle 
logging of all that data, but pfsense should allow you to define rules 
that allow stateless auditing (ip 1.2.3.4, ports 1000-2000 always NAT to 
sub A). The XML config file or possibly the shell is probably the 
easiest way to define such rulesets at scale.

I didn't see it mentioned, where (and to whom) are you multihoming? Do 
you have a good working relationship with these folks (cell phone, email 
contacts that reach someone promptly)? Will you be considered a 
facilities based ISP (and subject to CALEA or other regulation)?

--Blake