[Python-ideas] a multiProcess scheduler

[Thales filizola costa]

Hi Nick,

I have just checked all the links you posted, they are indeed very
interesting and very efficient. However, I think those are very complicate
in terms of installation and setup, and I still see a lot of usages for a
multi-process scheduler.


2016-08-28 20:32 GMT-07:00 Nick Coghlan <ncoghlan at gmail.com>:

> On 29 August 2016 at 11:50, Thales filizola costa <thalesfc at gmail.com>
> wrote:
> > What do you guys think? How to improve it? Is it relevant enough to be
> > incorporated to std python ?
>
> There are actually quite a few distributed schedulers out there (which
> can expand beyond a single machine), but "python multiprocess
> scheduler" isn't likely to bring them up in a web search (as when
> you're limited to a single machine, multiprocessing.Pool or
> concurrent.futures.ProcessPoolExecutor is generally already good
> enough).
>
> At a Python level, Celery is probably the most popular option for
> that: http://www.celeryproject.org/
>
> Another well-established option is Kamaelia: http://www.kamaelia.org/Home.
> html
>
> Dask is a more recent alternative specifically focused on
> computational tasks: http://dask.pydata.org/en/latest/
>
> Once you move outside Python specific tooling, there are even more
> language independent options to play with, including the likes of
> Mesos and Kubernetes.
>
> Cheers,
> Nick.
>
> P.S. It's a fairly sad indictment of our industry that people think
> this is a sensible question to ask in developer interviews - the
> correct answer from a business efficiency perspective is "I wouldn't,
> I would use an existing open source task scheduler rather than
> inventing my own", just as the correct answer to "How would you
> implement a sort algorithm?" from that perspective is "I wouldn't, as
> the Python standard library's sorting implementation is vastly
> superior to anything I could come up with in 5 minutes on a
> whiteboard, and the native sorting capabilities of databases are also
> excellent". Reimplementing existing software from first principles is
> a great learning exercise, but it's not particularly relevant to the
> task of day-to-day software construction in most organisations.
>
> (Alternatively, if the answer the interviewer is looking for is "I
> wouldn't, I would use...", then it may be an unfair "Gotcha!"
> question, and those aren't cool either, since they expect the
> interviewee to be able to read the interviewer's mind, rather than
> just answering the specific question they were asked)
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160828/f024988f/attachment.html>