Adding behaviour for managing "task" dependencies

[Stargaming]

On Tue, 02 Oct 2007 14:28:35 +0000, m.pricejones wrote:

> Hi,
> 
> I'm currently writing an animation pipeline in Python which is a system
> for controlling the flow of work and assets for a team of people working
> on a computer animated film. The system will be fairly large with a
> database backend.
> 
> One particular problem that I'd like to address is the need for managing
> dependent tasks. Often you need to process one particular set of
> information before starting on another so when you batch them all
> together and send them off to a set of computers to process you need
> some way of formulating the dependencies. This preferably ends up
> looking a lot like a Makefile in the end with syntax a bit like:
> 
> task task_name (dependencies):
>     commands to complete the task
>     ...
> 
> Where the dependencies are a list of other tasks that need to be
> completed first.
> 
> However I'd really like to do it in Python, but I'm thinking I'd might
> need to extend Python a bit in order to achieve the new syntax. I've
> seen attempts to do this within the Python syntax (Scons and buildIt)
> and I'm not a big fan of the way it ends up looking. I've worked at a
> place that has written it's own language to handle that sort of thing
> but then you end up with a language that is good for that but rubbish at
> everything else. 

Doesn't seem too bad if you just want to write down dependencies. Better 
have one tool/language doing it's job perfectly than having a big monster 
tool doing lots of jobs pretty bad.

This is also sometimes called domain-specific languages or `language 
oriented programming <http://www.onboard.jetbrains.com/is1/articles/04/10/
lop/>`_.

> Python seems like a good basis if I could tweak it slightly.
> 
> One particular point that interests me is the idea of maintaining
> compatibility with Python modules so you still have all the
> functionality. 

If you disagreed with my statements above, that's a good point. Would 
make deserialization of those nearly-Makefiles easier, too.

> This makes me think of the "from __future__ import ..."
> statements which, if I understand them correctly, can introduce new
> syntax like the with_statement, whilst still maintaining compatibility
> with older modules?

I might be wrong here but AFAIK those `__future__` imports just set some 
'compiler' flags. The documentation isn't saying much about the treatment 
of those statements but I guess they're handled internally (since 
__future__.py doesn't seem to do anything at all). See the `module 
documentation <http://docs.python.org/lib/module-future.html>`_ for 
details.
 
> Is this correct? Can anyone write a syntax changing module or is
> __future__ a hard coded special case? I realise I'll have to get into
> the C side of things for this. Are there other approaches to this that I
> really should be considering instead?

Syntax (as in: grammar) is compiled into the python binary. You can 
change the python source (as it's free) and recompile but, FWIW, I would 
not suggest this. You're _forking_ CPython then and well, this would be 
an additional project to maintain (merging all updates, fixes, releases 
into your fork etc.).

`PyPy <http://codespeak.net/pypy/>`_ could be slightly easier to adjust 
but still: extra project.

Another approach could be `Logix <http://livelogix.net/logix/>`_. It's an 
"alternate front-end for Python" with the ability to change its syntax on-
the-fly. I don't know how actively maintained it is, though.

> Any thoughts would be most appreciated, though I would like to stress
> that I don't think Python should support the syntax I'm proposing I'd
> just like to know if I can extend a copy of it to do that.

As stated above, there are several ways to change Python to support the 
syntax you want but hey, why do you need this? As far as I understood 
you, you want to use *normal* python functions and add dependencies to 
them. First thing popping into my mind is a simple function eg. ``depends
(source, target)`` or ``depends(target)(source)``. The second example 
would have the advantage of adapting neatly into the decorator syntax::

    def early_task():
        foo()
    
    @depends(early_task)
    def late_task():
        bar()

The wrapping `depends` function could decide whether `early_task` is done 
already or not (and invoke it, if neccessary) and finally pass through 
the call to `late_task`.

This is just an example how you could use existing syntax to modelize 
dependencies, there are other ways I guess.

HTH,
Stargaming