[Import-SIG] Idea: concept of a builder or transformer to compliment loaders

[Brett Cannon]

I should start off by saying I don't plan to pursue this idea, but I wanted
to write it down for posterity and in case anyone else has thought about
this.

That being said, the idea of macros and other source-transforming things
done to Python code has come up a few times on python-ideas as of late. Now
experimenting with this sort of thing using a custom loader is not hard,
and thanks to importlib.abc.ResourceLoader.source_to_code()
<https://docs.python.org/3/library/importlib.html#importlib.abc.InspectLoader.source_to_code>
it's fairly easy to do (by design; I tried to initially structure
importlib's APIs to making alternative storage backends easy as well as
alternative syntax stuff like Quixote from back in the day).

But one thing I realized is that while finders and loaders are necessary
for alternative code storage mechanisms, they are not the right abstraction
for tweaking code semantics. Really all you need is a function that takes
in source code and spits out a code object to use with exec() (hence
ResourceLoader.source_to_code() even existing). It somewhat sucks that
people who just want to tweak code semantics have to define a loader
subclass and instantiate a new finder when all that is mostly stuff that
doesn't concern them. It also sucks that they would have to do that for
every storage type, e.g. local files and zip files.

Now I don't have a solid solution to propose for this niche use case. It
makes me want to have some kind of way to register compiler functions, but
that would be limiting if it went source -> code object. AST -> AST would
allow for chaining much like Victor has proposed in the past, but it also
means that people who want a transpiler to go source -> source are left
out. And then there is the whole thing of how to get the loaders to know of
these transpilers/transformers/compilers as adding more global state to sys
feels dirty (maybe an attribute on finders that they can draw from if they
so choose?), but maybe it isn't that big of a deal as long as they are just
callables and people realize they must be re-entrant.

As I said, I don't plan to work on this, but I wanted to get my ideas
written down in case someone else cared.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20150528/4b0161ff/attachment.html>