With & marcos via import hooking? (Was Re: Scanning a file)

[Bengt Richter]

On Tue, 01 Nov 2005 07:14:57 -0600, Paul Watson <pwatson at redlinepy.com> wrote:

>Paul Rubin wrote:
>> jjl at pobox.com (John J. Lee) writes:
>> 
>>>Closing off this particular one would make it harder to get benefit of
>>>non-C implementations of Python, so it has been judged "not worth it".
>>>I think I agree with that judgement.
>> 
>> 
>> The right fix is PEP 343.
>
>I am sure you are right.  However, PEP 343 will not change the existing 
>body of Python source code.  Nor will it, alone, change the existing 
>body of Python programmers who are writing code which does not close files.

It might be possible to recompile existing code (unchanged) to capture most
typical cpython use cases, I think...

E.g., I can imagine a family of command line options based on hooking import on
startup and passing option info to the selected and hooked import module,
which module would do extra things at the AST stage of compiling and executing modules
during import, to accomplish various things.

(I did a little proof of concept a while back, see

    http://mail.python.org/pipermail/python-list/2005-August/296594.html

that gives me the feeling I could do this kind of thing).

E.g., for the purposes of guaranteeing close() on files opened in typical cpython
one-liners or single-suiters) like e.g.

    for i, line in enumerate(open(fpath)):
        print '%04d: %s' %(i, line.rstrip())

I think a custom import could recognize the open call
in the AST and extract it and wrap it up in a try/finally AST structure implementing
something like the following in the place of the above;

    __f = open(fpath) # (suitable algorithm for non-colliding __f names is required)
    try:
        for i, line in enumerate(__f):
            print '%04d: %s' %(i, line.rstrip())
    finally:
        __f.close()

In this case, the command line info passed to the special import might look like
    python -with open script.py

meaning calls of open in a statement/suite should be recognized and extracted like
__f = open(fpath) above, and the try/finally be wrapped around the use of it.

I think this would capture a lot of typical usage, but of course I haven't bumped into
the gotchas yet, since I haven't implemented it ;-)

On a related note, I think one could implement macros of a sort in a similar way.
The command line parameter would pass the name of a class which is actually extracted
at AST-time, and whose methods and other class variables represent macro definitions
to be used in the processing of the rest of the module's AST, before compilation per se.

Thus you could implement e.g. in-lining, so that

----
#example.py
class inline:
    def mac(acc, x, y):
        acc += x*y

tot = 0
for i in xrange(10):
    mac(tot, i*i)
----

Could be run with

    python -macros inline example.py

and get the same identical .pyc as you would with the source

----
#example.py
tot = 0
for i in xrange(10):
    tot += i*i
----

IOW, a copy of the macro body AST is substituted for the macro call AST, with
parameter names translated to actual macro call arg names. (Another variant
would also permit putting the macros in a separate module, and recognize their
import into other modules, and "do the right thing" instead of just translating
the import. Maybe specify the module by python - macromodule inline example.py
and then recognize "import inline" in example.py's AST).
    
Again, I just have a hunch I could make this work (and a number of people
here could beat me to it if they were motivated, I'm sure). Also have a hunch
I might need some flame shielding. ;-)

OTOH, it could be an easy way to experiment with some kinds of language
tweaks. The only limitation really is the necessity for the source to
look legal enough that an AST is formed and preserves the requisite info.
After that, there's no limit to what an AST-munger could do, especially
if it is allowed to call arbitrary tools and create auxiliary files such
as e.g. .dlls for synthesized imports plugging stuff into the final translated context ;-)
(I imagine this is essentially what the various machine code generating optimizers do).

IMO the concept of modules and their (optionally specially controlled) translation
and use could evolve in may interesting directions. E.g., __import__ could grow
keyword parameters too ...  Good thing there is a BDFL with a veto, eh? ;-)

Should I bother trying to implement this import for with and macros from
the pieces I have (plus imp, to do it "right") ?

BTW, I haven't experimented with command line dependent site.py/sitecustomize.py stuff.
Would that be a place to do sessionwise import hooking and could one rewrite sys.argv
so the special import command line opts would not be visible to subsequent
processing (and the import hook would be in effect)? IWT so, but probably should read
site.py again and figure it out, but appreciate any hints on pitfalls ;-)

Regards,
Bengt Richter