Adding behaviour for managing "task" dependencies

[Paddy]

On Oct 2, 3:28 pm, m.pricejo... at gmail.com wrote:
> Hi,
>
> I'm currently writing an animation pipeline in Python which is a
> system for controlling the flow of work and assets for a team of
> people working on a computer animated film. The system will be fairly
> large with a database backend.
>
> One particular problem that I'd like to address is the need for
> managing dependent tasks. Often you need to process one particular set
> of information before starting on another so when you batch them all
> together and send them off to a set of computers to process you need
> some way of formulating the dependencies. This preferably ends up
> looking a lot like a Makefile in the end with syntax a bit like:
>
> task task_name (dependencies):
>     commands to complete the task
>     ...
>
> Where the dependencies are a list of other tasks that need to be
> completed first.
>
> However I'd really like to do it in Python, but I'm thinking I'd might
> need to extend Python a bit in order to achieve the new syntax. I've
> seen attempts to do this within the Python syntax (Scons and buildIt)
> and I'm not a big fan of the way it ends up looking. I've worked at a
> place that has written it's own language to handle that sort of thing
> but then you end up with a language that is good for that but rubbish
> at everything else. Python seems like a good basis if I could tweak it
> slightly.
>
> One particular point that interests me is the idea of maintaining
> compatibility with Python modules so you still have all the
> functionality. This makes me think of the "from __future__ import ..."
> statements which, if I understand them correctly, can introduce new
> syntax like the with_statement, whilst still maintaining compatibility
> with older modules?
>
> Is this correct? Can anyone write a syntax changing module or is
> __future__ a hard coded special case? I realise I'll have to get into
> the C side of things for this. Are there other approaches to this that
> I really should be considering instead?
>
> Any thoughts would be most appreciated, though I would like to stress
> that I don't think Python should support the syntax I'm proposing I'd
> just like to know if I can extend a copy of it to do that.
>
> Mike

You could use a professional Job scheduling system such as LSF or Suns
Grid Engine that both support job dependencies (and a whole lot more).

For example, search for 'Job Dependencies' on this page:
  http://docs.sun.com/app/docs/doc/820-0699/6nce0ht7s?a=view

- Paddy.