Re: Amazon network engineering contact? re: DDoS traffic

[Tom Beecher <beecher () beecher cc>]

Nobody should ever be forced to peer to get someone to address abusive
traffic originating from networks under their control.



On Thu, Nov 8, 2018 at 4:29 PM John <jw () nuclearfallout net> wrote:

> Zach,
>
> As mentioned before, I am open to peering (where possible) but have not
> received a response.
>
> My goal is to connect with someone at Amazon and work with them on a
> technical solution, which is why I posted asking for that here. Various
> factors mean that we can't just upgrade our way out of this one, and
> manual whac-a-mole on the sources has shown to have limited use.
>
> These attacks also impact Amazon and the networks in between the sources
> and targets, and they take time to handle by the abuse teams, so there
> are good reasons to investigate them further and find better ways to
> mitigate or prevent them.
>
> -John
>
> On 11/8/2018 1:12 PM, Zach Puls wrote:
> > Makes sense, that's understandable. Do you peer with AWS? If not, maybe
> opening up a peering agreement will give you a better contact, and a bit
> more pull when attacks occur? I know someone with a peering agreement with
> AWS, and they have been able to get resolutions fairly quickly when issues
> arise.
> > Other than that, I'm not sure of a solution other than more IP transit.
> >
> > Thanks,
> >
> > Zach Puls
> > Network Engineer | MEF-CECP
> > KsFiberNet
> >
> > -----Original Message-----
> > From: John Weekes <jw () nuclearfallout net>
> > Sent: Thursday, November 08, 2018 15:03
> > To: Zach Puls <zpuls () ksfiber net>
> > Cc: nanog () nanog org
> > Subject: Re: Amazon network engineering contact? re: DDoS traffic
> >
> > Zach,
> >
> > Yes, RTBH is used to distribute the null-routes that I mentioned.
> >
> > Unfortunately, even brief saturation events lasting just 5-10 seconds (a
> typical amount of time to detect the loss, issue the null-route, and see
> the traffic start to fall off as it is distributed upstream) can cause real
> damage to those customers who are sensitive to latency and packet loss. So
> while null-routes limit the duration of the impact, they can't eliminate it
> entirely. And, of course, the actual target of the attack
> > -- the now-null-routed IP address -- becomes unreachable, which was
> presumably the goal of the attacker.
> > -John
> >
> > On 11/8/2018 12:54 PM, Zach Puls wrote:
> > > No idea about an Amazon abuse contact, but do you have RTBH communities
> enabled with your upstream provider(s)? As a general practice, when you
> detect a (D)DoS attack in progress, it would help to automatically
> advertise that prefix to your upstream(s) with the black-hole community.
> This would at least help mitigate the effects of the attacks when they do
> occur, even if they come from a different source than AWS.
> > > Thanks,
> > >
> > > Zach Puls
> > > Network Engineer | MEF-CECP
> > > KsFiberNet
> > >
> > > -----Original Message-----
> > > From: NANOG <nanog-bounces () nanog org> On Behalf Of John Weekes
> > > Sent: Thursday, November 08, 2018 14:44
> > > To: nanog () nanog org
> > > Subject: Amazon network engineering contact? re: DDoS traffic
> > >
> > > We've been seeing significant attack activity from Amazon over the last
> two months, involving apparently compromised instances that commonly send
> 1-10G of traffic per source and together generate Nx10G of total traffic.
> Even when our overall upstream capacity exceeds an attack's overall size,
> the nature of load-balancing over multiple 10G upstream links means that an
> individual link can be saturated by multiple large flows, forcing our
> systems to null-route the target to limit impact.
> > > We've sent an abuse notification about every traffic source to Amazon,
> and specific sources seem to stop their involvement over time (suggesting
> that abuse teams are following up on them), but there is an endless parade
> of new attackers, and each source participates in many damaging attacks
> before it is shut down.
> > > Is there anyone at Amazon who can help with an engineering solution in
> terms of programmatically detecting and rate-limiting attack traffic
> sources, to our networks or overall? Or applying the kludge of a rate-limit
> for all Amazon traffic to our networks? Or working with us on some other
> option?
> > > At least one other large cloud provider has an automatic rate-limiting
> system in place that is effective in reducing the damage from repeat
> high-volume attacks.
> > > Emails to the Amazon NOC, peering contacts (since that would be another
> possible solution), and abuse department have not connected me with anyone.
> > > Thanks,
> > > John