automatically generating file dependency information from python tools

[Moosebumps]

> I'm not entirely clear on what the purpose of this is. I normally
> think of "makefile" type information as something needed to compile
> a program. This is something that isn't usually needed for Python
> unless you're dealing with C extensions. Then I'd suggest looking at
> SCons (www.scons.org).

Well sorry for being so abstract, let me be a little more concrete.  I am
working at a video game company, and I have had some success using Python
for tools.  I am just thinking about ways to convince other people to use
it.  One way would be to improve the build processes, and be able to do
incremental builds of art assets without any additional effort from
programmers.  Basically I'm trying to find a way to do some work for free
with python.

The idea is that there are many different types of assets, e.g. 3D models,
textures/other images, animations, audio, spreadsheet data, etc.  Each of
these generally has some tool that converts it from the source format to the
format that is stored in the game on disk / in memory.  Hence they are
usually simple command line data crunchers.  They take some files as input
and just produce other files as output.

Currently, we don't have time to generate the dependency information
necessary for incremental building, so we generally just build everything
over again from scratch, which takes 20 PCs the entire night.  The problem
is that the pipeline changes frequently, and nothing is really documented,
especially the dependencies.  It would be nice if there was a way to
automatically get these from the individual data crunchers, which may be
written by many different people.  It eliminates the redundancy of having
dependency information in the source code of the individual tools, and also
in a separate file that specifies dependency info (like a makefile).

So instead rebuilding the whole game, or having to know exactly which files
to rebuild (which some people know, but many others don't), the "make" tool
would be able to read the dependency information generated, and check dates
on the source files to see what changes, and build the minimum number of
things to get the game up to date.  Currently lots of unnecessary things are
rebuilt constantly.

> What I'm getting is that you want to tie the individual programs
> to the files that they're processing. In other words, build a catalog
> of "if you have this kind of file, these are the availible programs that
> will process it."

Well, that is not exactly the point, but hopefully that information would
fall out of the automatic processing of the individual command line tools.

> So the basic question is: are the files coming in from the command
> line or are they built in? If the latter, I'd probably start out by
pulling
> strings that have a "." or a "/" or a "\" in them, and examining the
> context. Or look at calls to modules from the os.path library.

They could be either "statically" specified in the source code, or only
known at runtime.

> More than likely you'll find a number of patterns that can be
> processed and that will deal with the majority of programs. The
> thing is, if you've got a bunch of programmers doing that kind
> of work, they've probably fallen into habitual ways of coding
> the repetitive stuff.

Yes, that is true, and everything works OK now, but there are thousands and
thousands of lines of redundant code, and the build process is very slow.
I'm just trying to separate out the common parts of every tool, rather than
having all that information duplicated in dozens of little command line
utilities.

MB