[Speed] Lazy-loading to decrease python_startup time

[Brett Cannon]

On Sat, 24 Jun 2017 at 02:00 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 23 Jun 2017 23:03:57 +0000
> Brett Cannon <brett at python.org> wrote:
> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
> wrote:
> >
> > > As suggested, I'd like to discuss if lazy-loading is an option for
> > > improving python-startup time.And if could be done inside the scope of
> a
> > > GSoc project.
> > >
> >
> > It's a possibility and it could be done in the scope of a GSoC project
> > easily. Basically what would be needed is an importlib.util.lazy_import()
> > function which does mostly what is outlined in
> >
> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
> > but
> > where the proper lazy loader is set on the spec object as an extra step.
> > Then every module that is used during startup would use
> > importlib.util.lazy_import() for importing instead of the normal import
> > statement.
>
> My experience is that:
> - you want lazy imports to be implicit, i.e. they should work using the
>   "import" statement and not any special syntax or function invocation
> - you want a blacklist and/or whitelist mechanism to restrict lazy
>   imports to a particular set of modules and packages, because some
>   modules may not play well with lazy importing (e.g. anything that
>   registers plugins, specializations -- think singledispatch --, etc.)
>
> For example, I may want to register the "tornado", "asyncio" and "numpy"
> namespaces / packages for lazy importing, but not the "numba" namespace
> as it uses registration mechanisms quite heavily.
>
> (and the stdlib could be part of the default lazy import whitelist)
>

That's all true for an end user's application, but for the stdlib where
there is no knock-on effects from dependencies not being loaded lazily I
don't think it's quite as critical. Plus lazy loading does make debugging
harder by making loads that trigger an exception happen at the line of
first use instead of at the import line, so I don't know if it's desirable
to automatically make the whole stdlib be lazily loaded from the outset
(which is what would be required since doing it in e.g. sitecustomize.py
wouldn't happen until after startup anyway).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170624/5730bd47/attachment.html>