[Python-ideas] a multiProcess scheduler

[Nick Coghlan]

On 29 August 2016 at 11:50, Thales filizola costa <thalesfc at gmail.com> wrote:
> What do you guys think? How to improve it? Is it relevant enough to be
> incorporated to std python ?

There are actually quite a few distributed schedulers out there (which
can expand beyond a single machine), but "python multiprocess
scheduler" isn't likely to bring them up in a web search (as when
you're limited to a single machine, multiprocessing.Pool or
concurrent.futures.ProcessPoolExecutor is generally already good
enough).

At a Python level, Celery is probably the most popular option for
that: http://www.celeryproject.org/

Another well-established option is Kamaelia: http://www.kamaelia.org/Home.html

Dask is a more recent alternative specifically focused on
computational tasks: http://dask.pydata.org/en/latest/

Once you move outside Python specific tooling, there are even more
language independent options to play with, including the likes of
Mesos and Kubernetes.

Cheers,
Nick.

P.S. It's a fairly sad indictment of our industry that people think
this is a sensible question to ask in developer interviews - the
correct answer from a business efficiency perspective is "I wouldn't,
I would use an existing open source task scheduler rather than
inventing my own", just as the correct answer to "How would you
implement a sort algorithm?" from that perspective is "I wouldn't, as
the Python standard library's sorting implementation is vastly
superior to anything I could come up with in 5 minutes on a
whiteboard, and the native sorting capabilities of databases are also
excellent". Reimplementing existing software from first principles is
a great learning exercise, but it's not particularly relevant to the
task of day-to-day software construction in most organisations.

(Alternatively, if the answer the interviewer is looking for is "I
wouldn't, I would use...", then it may be an unfair "Gotcha!"
question, and those aren't cool either, since they expect the
interviewee to be able to read the interviewer's mind, rather than
just answering the specific question they were asked)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia