Re: PlayStationNetwork blocking of CGNAT public addresses

[Mike Hammett <nanog () ics-il net>]

People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big 
content can be just as arrogant and just as deserving of hatred. 




----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

----- Original Message -----

From: "Tom Beecher" <beecher () beecher cc> 
To: "Tom Smyth" <tom.smyth () wirelessconnect eu> 
Cc: "NANOG" <nanog () nanog org> 
Sent: Sunday, September 18, 2016 8:15:08 AM 
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses 

This is, as many things are, a huge problem in communication. 

Sony tells ISP 'Hey, you have customers abusing us. Fix it!'. 
ISP says 'Oh crap, sorry, what's going on? We'll run it down.' 
Sony says nothing. 

Let's just stop here for a second. This is fundamentally no different then 
the 'I have a problem, it's the network! complaints we've all dealt with 
forever. You spend days/weeks/months working on it. Maybe you ultimately 
find a goofy switchport, or maybe you discover that the server HDDs were 
crapping the bed and the problem server was chugging because of that. But 
you had to spend tons of time working on it because you couldn't get the 
info you need because the reporter was CONVINCED they KNEW what it was. 

Why should Simon have to spend hours of engineering time fishing through 
traffic captures and logs when he doesn't even know what he's LOOKING for? 
What does PSN consider 'abuse' here? 

Does Simon have customers infected with botnets that are targeting PSN at 
times? Or does PSN assume nobody will ever have more than a couple 
Playstations in a house, so if they see more than N connections to PSN from 
the same IP, it's malicious, since CGN is likely not something they 
considered? ( If anyone wants to place beer wagers, I'm picking the later. ) 

I spend about 8 weeks this year going back and forth with a Very Large 
Website Network who had blocked a /17 of IP space from accessing ANY of 
their sites because of 'malicious traffic' from a specific /23. 5 of those 
weeks, their responses consisted of 'it's malicious, you go find it, should 
be obvious', 'you clearly don't know what you're doing, we're wasting our 
time', etc. Week 5, I was able to extract that it was a specific web 
crawler that they said was knocking their databases over. After a 
conversation with their CIO the following week, they came back and admitted 
that a junior system admin made some PHP changes on a bunch of servers that 
he didn't think was in production,and when we crawled THOSE servers, Bad 
Things Happened for them. We were doing nothing wrong ; they just refused 
to look, and found it easier to blame us. 

Simon's getting screwed because he's not being given any information to try 
and solve the problem, and because his customers are likely blaming him 
because he's their ISP. 

Sony needs to stand up and work with him here. 

On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.smyth () wirelessconnect eu> 
wrote: 

> Hi Simon, 
>
> as other responders have said it is an inherent issue with NAT in general, 
> on workaround is to limit the ratio of actual users to an external IPv4 
> address, the other thing we have seen from our Abuse contact emails from 
> PSN, is that malicious activity towards the PSN is often accompanied by 
> other malicious activities such as SSH brute force outbound and spaming... 
>
> I would suggest that 
>
> 1) limit the ratio of users to an external ipv4 address as much as possible 
> (which would reduce the impact of one compromised customer bringing down 
> play time for other clients behind the same nat 
>
> 2)do some "canary in the mine" monitoring for obviously malicious traffic 
> (loads of SMTP traffic outbound) and lots of connection requests to SSH 
> servers ... if you see that traffic from behind your CGNAT device .. just 
> temporarily block the internal ip of the user until they clean up their 
> devices. 
>
> this is the pain with NAT you have to do extra work in order prevent 
> infected users interrupting internet connectivity for other innocent 
> users... 
> I think you can use simple firewall rules on your edge router to identify 
> multiple connections to SMTP and SSH in a short period of time.. 
>
> If you do the minimum to detect that abuse then you cant be accused of 
> invading peoples privacy... (bear in mind obvious false positives) 
> (Monitoring systems etc) ... 
>
> Hope this helps, 
>
> On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon () slimey org> wrote: 
>
> > All, 
> >
> > We operate an access network with several hundred thousand users. 
> > Increasingly 
> > we're putting the users behind CGNAT in order to continue to give them an 
> > IPv4 
> > service (we're all dual-stack, so they all get public IPv6 too). Due to 
> the 
> > demographic of our users, many of them are gamers. 
> >
> > We're hitting a problem with PlayStationNetwork 'randomly' blocking some 
> > of our 
> > CGNAT outside addresses, because they claim to have received anomalous, 
> or 
> > 'attack' traffic from that IP. This obviously causes problems for the 
> other 
> > legitimate users who end up behind the same public IPv4 address. 
> >
> > Despite numerous attempts to engage with PSN, they are unwilling to give 
> us 
> > any additional information which would allow us to identify the 'rogue' 
> > users 
> > on our network, or to identify the 'unwanted' traffic so that we could 
> > either 
> > block it, or use it to identify the rogue users ourselves. 
> >
> > Has anyone else come up against the problem, and/or have any suggestions 
> on 
> > how best to resolve it? 
> >
> > Many thanks in advance, 
> >
> > Simon 
>
>
> -- 
> Kindest regards, 
> Tom Smyth 
>
> Mobile: +353 87 6193172 
> --------------------------------- 
> PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL 
> This email contains information which may be confidential or privileged. 
> The information is intended solely for the use of the individual or entity 
> named above. If you are not the intended recipient, be aware that 
> any disclosure, copying, distribution or use of the contents of this 
> information is prohibited. If you have received this electronic 
> transmission in error, please notify me by telephone or by electronic mail 
> immediately. Any opinions expressed are those of the author, not the 
> company's .This email does not constitute either offer or acceptance of 
> any contractually binding agreement. Such offer or acceptance must be 
> communicated in 
> writing. You are requested to carry out your own virus check before opening 
> any attachment. Thomas Smyth accepts no liability for any loss or damage 
> which may be caused by malicious software or attachments.