[Python-Dev] Peculiar import code in pickle.py

[Brett Cannon]

On Wed, Jul 14, 2010 at 05:15, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Wed, Jul 14, 2010 at 9:05 PM, Steve Holden <steve at holdenweb.com> wrote:
> >> I have stopped fixing bugs related to this in import.c because of the
> >> annoying issues it causes and I expect the correct approach to gain
> >> traction at some point (plus get importlib bootstrapped in so I don't
> >> have to care about import.c anymore).
> >>
> > It's only a matter of time until someone decides to provide a C
> > implementation of importlib ;-)
>
> A C accelerated version of importlib would probably be an awful lot
> cleaner than the current import implementation. While the import code
> isn't quite the mess that we sometimes make it out to be (it does
> basically work after all, and most of the "problems" lie in dim dark
> corners that 99% of developers will never get close to), but it has
> definitely suffered from an accumulation of features on top of a core
> approach that has been pushed far beyond what it was originally
> designed to support.
>

So my dream is to finally get full compatibility for importlib in 3.3
(probably won't hit 3.2 as it requires me changing marshal.loads to take a
file path argument) and then try to bootstrap it in. Now bootstrapping can
be done with actually a minimal amount of C code as I can simply make the
bytecode for importlib a literal in C, get that loaded, and then import
importlib as found on the file system to allow people to override things.
But obviously I could also identify the true bottlenecks through profiling
and provide acceleration where useful. Trick is being reasonable about this
so as to not put other VMs at a disadvantage by making the bootstrap
solution too difficult to implement.


>
> That said, I believe the limiting factor in import speed is likely to
> remain the number of stat calls and other filesystem operations, so it
> will be interesting to find out just how significant a slowdown there
> is between import.c and importlib. If I'm right about the real source
> of bottlenecks in import performance, the difference may be
> surprisingly small.


So I started writing benchmark code in anticipation of needing to prove a
minimal performance difference to justify bootstrapping importlib. Right now
it only compares importing from sys.modules and built-in modules. You can
run it with ``./python.exe -m importlib.test.benchmark``. If you add a `-b`
option that will use the built-in __import__ implementation. I still need to
benchmark loading source, bytecode, writing bytecode, and loading
extensions. Maybe I will finish writing the benchmark code as the thing I do
while at EuroPython (that and finally getting to reviewing
http://bugs.python.org/issue2919 so that cProfile and profile can merge,
unless someone beats me to it, in which case I would be grateful =).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100714/06c5ac58/attachment.html>