gevent - a concurrency framework for python
I am sure that part of your code is filled with IO related tasks which majority of the time consume most of the execution duration... so scale it.
Before you say, "Easy, I just add more threads and everything is running on parallel and faster", threads has cost of spawn time, memory footprint --> bringing limit to your scale!
a better solution might be to make your application not block on your IO operations (e.g use that "dead time"), and for that you can use several framework like concurrent.future, eventlet, gevent and other
In one of my new utilities (API crawler) that is highly impacted by IO procedure (since all it does, is query and wait), I chose to use GEVENT to make my code run concurrent and better utilize my system resources - the outcome was amazing!!!
At the end of the day its all a matter of what are your requirements, e.g. you might still need to spawn tasks into a distributed queue that handles those in a cluster, but that doesn't mean that each worker cannot consume task and run the, concurrent --> use hybrid solution (Such as celery for python) which maximize your throughput and still take advantage of persistency of tasks, Error handling, etc
p.s. I found a very good site which shows great real life example of usage of gevent here
Check out the google trend:
Before you say, "Easy, I just add more threads and everything is running on parallel and faster", threads has cost of spawn time, memory footprint --> bringing limit to your scale!
a better solution might be to make your application not block on your IO operations (e.g use that "dead time"), and for that you can use several framework like concurrent.future, eventlet, gevent and other
In one of my new utilities (API crawler) that is highly impacted by IO procedure (since all it does, is query and wait), I chose to use GEVENT to make my code run concurrent and better utilize my system resources - the outcome was amazing!!!
At the end of the day its all a matter of what are your requirements, e.g. you might still need to spawn tasks into a distributed queue that handles those in a cluster, but that doesn't mean that each worker cannot consume task and run the, concurrent --> use hybrid solution (Such as celery for python) which maximize your throughput and still take advantage of persistency of tasks, Error handling, etc
p.s. I found a very good site which shows great real life example of usage of gevent here
Check out the google trend:
gevent - a concurrency framework for python
Reviewed by Ran Davidovitz
on
4:10 PM
Rating:
Reviewed by Ran Davidovitz
on
4:10 PM
Rating:
No comments:
Post a Comment