Re: [Zenmap] Slow "Open" for large files

[Henri Doreau <henri.doreau () gmail com>]

2014-06-08 14:58 GMT+02:00 Jay Bosamiya <jaybosamiya () gmail com>:
> Hi All!
>
> While I was testing things for my other patch (the MemoryError one), I
> came upon this rather strange thing. A scan that I had run took 30
> seconds to run, about a second to save, but hours to load. The "Open"
> option was too slow for large files!
>
> Some analysis showed that the Open option was using string concatenation
> and this made the algorithm quadratic in time complexity. (Thanks Dan
> for pointing out that string concatenation is slow in Python).
>
> Modifying this to use StringIO immediately improves the speed. The same
> file opened in under half a minute!!!
>
> Attached is the patch for the same. I had to refactor some other parts
> of the code for zenmap/zenmapCore/NmapParser.py too (since _nmap_output
> is used in some places where nmap_output should be used).
>
> I have tested this patch and there seems to be no problem.
>
> Feedback is welcome.
>
> Cheers,
> Jay
>
> PS: The command I used to generate the large file is "nmap
> --packet-trace -r --top-ports 1000 127.0.0.1/24". The generated xml file
> (using "Save" from Zenmap) is about 20 MB in size!
Nice! Thanks Jay.

Have you considered using string.join which is a very efficient way to
concatenate strings in python, instead of stringIO?

I don't have a very large file at hand right now but on a 5MB one I
get slightly better loading times with the patch attached than with
cStringIO. If confirmed, it would make the code a (little) bit faster
and a bit simpler.

I think there are still slow "+=" operations on strings in
ScriptMetadata.py. They might have an impact with the NSE collection
growing...

Regards

-- 
Henri

Attachment: zenmap_strjoin.diff
Description:

_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/