Re: Python and asynchronous sockets question

[Scott Kragen <skragen () gmail com>]

Thats where thread pooling comes into play.   It gives the best of both
worlds.  You set a maximum amount of threads you think I/O wise you can
handle, let it be 2 or 10.  Then each task gets added to the que.  Lets use
an example of 5 threads and 10 requests.  It would put the first 5 requests
out.  As a request free's up it assigns it to the open thread until the que
is empty.  This way you don't crush your machine.  I usually don't work
with more then 10 threads.

S

On Thu, Jan 10, 2013 at 10:57 AM, allison nixon <elsakoo () gmail com> wrote:

> Yeah that's why I said I accidentally wrote it in python.  I just want to
> know if I can salvage this since I already wrote it.
>
> I  also just realized something that I might have been doing wrong that is
> hogging most of the resources... if I can get 200 requests per second with
> python i'll let you guys know...
>
>
>
> On Thu, Jan 10, 2013 at 10:37 AM, Martín <ole () makensi es> wrote:
>
> > Hi!
> >
> > be also advised that because of the GlL [1] of the cpython interpreter,
> > you may need to switch to an alternative interpreter implementation when
> > performance of high concurrency applications is a must.
> >
> > In other words, Python may not be the best choice when trying to go to
> > such performance limits.
> >
> > [1] http://en.wikipedia.org/wiki/Global_Interpreter_Lock
> >
> >
> > On Thu, Jan 10, 2013 at 4:22 PM, Scott Kragen <skragen () gmail com> wrote:
> >
> > > Allison,
> > >
> > > Have you looked into threading for python?  I have used this library in
> > > several of projects because writing a thread pool from scratch started to
> > > give me a headache.
> > >
> > >
> > > http://code.activestate.com/recipes/577105-synchronization-decorator-for-class-methods/
> > >
> > > The advantage of a thread pool is it can que based on the max amount of
> > > threads you define.
> > >
> > > Scott
> > >
> > > On Thu, Jan 10, 2013 at 10:06 AM, allison nixon <elsakoo () gmail com>wrote:
> > >
> > > > Say I'm writing a broadscanner for a pet project and I accidentally
> > > > wrote the entire thing in Python.
> > > >
> > > > Using asyncore and sockets, the best I can get is 3 http requests per
> > > > second, maybe 10 per second at the very max.  For a scanner this is of
> > > > course, very lame.
> > > >
> > > > My goal is to of course make as many http request as a desktop computer
> > > > will handle, so perhaps 200 per second with whatever number of sockets
> > > > waiting in the background for a response(which I check periodically for a
> > > > response and parse when I get it)
> > > >
> > > > Is there any way I can acheive 200 requests per second without learning
> > > > another programming language?
> > > >
> > > > _______________________________________________
> > > > Pauldotcom mailing list
> > > > Pauldotcom () mail pauldotcom com
> > > > http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
> > > > Main Web Site: http://pauldotcom.com
> > >
> > >
> > >
> > > --
> > >
> > > ----------------------------------------------------------------------------------------------------------------------
> > > There is only one metric in security that can be truly measured and that
> > > is failure!
> > >
> > > --- Jack Daniels
> > > ----------------------------------------------------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Pauldotcom mailing list
> > > Pauldotcom () mail pauldotcom com
> > > http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
> > > Main Web Site: http://pauldotcom.com
> >
> >
> > _______________________________________________
> > Pauldotcom mailing list
> > Pauldotcom () mail pauldotcom com
> > http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
> > Main Web Site: http://pauldotcom.com
>
>
>
> --
> _________________________________
> Note to self: Pillage BEFORE burning.
>
> _______________________________________________
> Pauldotcom mailing list
> Pauldotcom () mail pauldotcom com
> http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
> Main Web Site: http://pauldotcom.com



-- 
----------------------------------------------------------------------------------------------------------------------
There is only one metric in security that can be truly measured and that is
failure!

--- Jack Daniels
----------------------------------------------------------------------------------------------------------------------

_______________________________________________
Pauldotcom mailing list
Pauldotcom () mail pauldotcom com
http://mail.pauldotcom.com/cgi-bin/mailman/listinfo/pauldotcom
Main Web Site: http://pauldotcom.com