Re: Nmap and Cisco Dos, clarification --

[lnapier () CISCO COM (Lisa Napier)]

Hi Andrew,

The message you are quoting from me was my attempt to refute the possible
causes suggested by another poster, not as a suggestion for the reasoning
behind the nmap problem.   I was attempting to indicate that I did not
think that ARP was the problem, but was waiting for further testing and
details, which I did not have at the time.

Apologies, as I was not clear in my message.  I believe my later follow up
message of 9/15/99 is exactly in-line with what Kenny had tested in house.

Just further clarification.

Thank you,

Lisa Napier
Product Security Incident Response Team
Cisco Systems

At 01:44 PM 9/22/1999 -0700, Lancashire, Andrew wrote:
> This is to clarify what is being put out by Cisco and what we are being told
> by Cisco.
>
> Two e-mails below is what Cisco is telling us and makes allot more sense
> than what Cisco is telling Bugtraq. The last post to Bugtraq made mention
> that the arp cache was filling up and allocating memory for both reachable
> hosts and unreachable hosts (incompletes).  Although what Lisa describes is
> true regarding the arp cache, it would not be true for our or most other
> sane persons environment.  Since routers will only arp for what is local,
> that would mean that for the arp cache to fill up and us all the memory all
> networks in the 10.x.x.x range would need to be local.  So that's not gonna
> happen but if you read the e-mail below that from Kenny (also at Cisco ) his
> explanation makes allot more sense considering we have hundreds of routers.
>
> Thank You
>
> Andrew
>
> P.S. Congratulations on the re-opening of PacketStorm
> ___________________________________
>
>
> Subject:      Re: Cisco and Nmap Dos
> To: BUGTRAQ () SECURITYFOCUS COM
>
>
> Hi all,
>
>
> I wanted to address the items listed here.  We are still investigating this
> problem, and hope to have more information later on in the week.
>
>
> Item 1, OSPF is not an issue.  According to the configuration information
> provided to us by the customer, OSPF is not in use.
>
>
> Items 2, 3 & 4.   IOS actually handles ARP in the following manner:
>
>
> When we receive a packet destined for something not already in our ARP
> table, we  enter an "incomplete" entry in the ARP table.  Then we will rate
> limit ARPs to once every 2 seconds to that destination.  Any additional
> packets to that same destination will be dropped until the ARP entry is
> completed.  This is on a per destination basis.
>
>
> "Incompletes" (ARP requests that have not been responded to) get dropped
> after approximately 1 minute from the last time we sent an ARP request for
> that non existent address.
>
>
> Incomplete entries MAY stay around longer, as the process that is
> responsible for cleaning up the ARP table may not have enough time to
> complete before it is called again, if we have a lot to clean up, which may
> be relevant to this case.  The incomplete entries will eventually get
> cleaned up, but it may take two or three minutes, two or three cycles of the
> process that cleans up the table.
>
>
> Under a dedicated, intense nmap scan, a very large number of ARP requests
> may be generated, causing the ARP table to grow very large with "incomplete"
> entries.  These entries consume memory.  As the amount of free memory
> declines and demand on the processor to handle outstanding packets
> increases, ARP processing falls behind and throughput on the router may
> decline significantly.  Once the scan is stopped, processing catches up and
> things return to normal.
>
>
> So, to my knowledge IOS should be doing the right thing, we only queue one
> ARP request at a time, every 2 seconds, until the ARP entry is resolved, we
> rate limit requests, dropping all additional packets, until the ARP entry is
> resolved, and we clean up the outstanding incomplete requests within a few
> minutes.
>
>
> I hope that helps address some of the concerns put forth.  Hopefully we will
> have further updates shortly.
>
>
> Thank you,
>
>
> Lisa Napier
> Product Security Incident Response Team
> Cisco Systems
> ___________
>
> _______________
>
> -----Original Message-----
> From:   khollis [SMTP:khollis () cisco com]
> Sent:   Wednesday, September 15, 1999 11:31 AM
> To:     wescotd () sutterhealth org
> Subject:        Regarding Case Number V44290
>
> Hello Dave, I've done some testing here with Nmap. I was able to create a
> test bed that can cause problems & symptoms similar to those you reported.
> But in summary, the router is functioning normally & depending on the
> network topology the behavior you experienced would be expected.
>
> > From show processor memory, the ip input process is the process that
> maintains the ip fast switching cache. This fast switching cache is used
> when forwarding packets to avoid interrupting the cpu for repetitive route
> table lookups. The key issue is the behavior of the fast cache and the way
> it gets built.
>
> There are a number of scenarios that will cause the fast cache to install an
> entry that essentially looks like a host route. For instance, with only 1
> path to a destination, we will install an entry into the fast cache that
> covers the entire network. Example: 100.0.0.0/8. However, when multiple
> equal cost paths to a destination exist, we will install an entry into the
> fast cache for each destination. Example: 100.0.0.1/32, 100.1.1.1/32,
> 100.2.2.2/32...and so on. This helps ensure load balancing. Additionally,
> depending on whether routes are directly connected, and/or subnetted, or the
> next hop of a static route, the results can vary.
>
> When running Nmap & scanning every address in a class A ip network, if
> conditions warrant the installation of a /32 entry into the fast cache this
> would allow the fast cache to consume a tremendous amount of memory and in
> that scenario all available dram could be consumed. This creates additional
> problems because there isn't enough memory to support other features on the
> router.
>
> Since Nmap isn't a normal application ran on networks, this issue isn't a
> concern in most networking environments. However, if this is a major concern
> you could run CEF (Cisco Express Forwarding). The behavior I just explained
> does not occur when running CEF. But you will need to run 12.0 on the Cat5
> RSM. Other workarounds such as disabling fast switching (no ip route-cache)
> or using max-paths 1 aren't really feasible. CEF is the better solution.
>
> Thanks,
> KennyH.
>
> _________________