From brett at python.org  Fri Jun 23 19:03:57 2017
From: brett at python.org (Brett Cannon)
Date: Fri, 23 Jun 2017 23:03:57 +0000
Subject: [Speed] Lazy-loading to decrease python_startup time
In-Reply-To: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>
References: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>
Message-ID: <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q@mail.gmail.com>

On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com> wrote:

> As suggested, I'd like to discuss if lazy-loading is an option for
> improving python-startup time.And if could be done inside the scope of a
> GSoc project.
>

It's a possibility and it could be done in the scope of a GSoC project
easily. Basically what would be needed is an importlib.util.lazy_import()
function which does mostly what is outlined in
https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
but
where the proper lazy loader is set on the spec object as an extra step.
Then every module that is used during startup would use
importlib.util.lazy_import() for importing instead of the normal import
statement. What this would do is help guarantee that all modules that are
identified as part of startup never import a module needlessly as the lazy
loader would simply postpone the load until necessary. This would also
allow for pulling out local imports that are currently done in modules that
are a part of startup and make them global so they are more visible.

But I have no idea if this will actually speed things up. :) At worst it
would slow things down ever so slightly due to the extra overhead of lazy
loading for things that are known to be needed already during startup. At
best, though, is we accidentally discover modules that are being imported
needlessly at startup as well as not having to hide imports in functions
for performance. This fact that it may not be worth it is why I haven't
bothered to try it out yet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170623/ed7416e2/attachment.html>

From solipsis at pitrou.net  Sat Jun 24 04:59:49 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 24 Jun 2017 10:59:49 +0200
Subject: [Speed] Lazy-loading to decrease python_startup time
References: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>
 <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q@mail.gmail.com>
Message-ID: <20170624105949.62590bbf@fsol>

On Fri, 23 Jun 2017 23:03:57 +0000
Brett Cannon <brett at python.org> wrote:
> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com> wrote:
> 
> > As suggested, I'd like to discuss if lazy-loading is an option for
> > improving python-startup time.And if could be done inside the scope of a
> > GSoc project.
> >  
> 
> It's a possibility and it could be done in the scope of a GSoC project
> easily. Basically what would be needed is an importlib.util.lazy_import()
> function which does mostly what is outlined in
> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
> but
> where the proper lazy loader is set on the spec object as an extra step.
> Then every module that is used during startup would use
> importlib.util.lazy_import() for importing instead of the normal import
> statement.

My experience is that:
- you want lazy imports to be implicit, i.e. they should work using the
  "import" statement and not any special syntax or function invocation
- you want a blacklist and/or whitelist mechanism to restrict lazy
  imports to a particular set of modules and packages, because some
  modules may not play well with lazy importing (e.g. anything that
  registers plugins, specializations -- think singledispatch --, etc.)

For example, I may want to register the "tornado", "asyncio" and "numpy"
namespaces / packages for lazy importing, but not the "numba" namespace
as it uses registration mechanisms quite heavily.

(and the stdlib could be part of the default lazy import whitelist)

Regards

Antoine.



From brett at python.org  Sat Jun 24 12:28:29 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 24 Jun 2017 16:28:29 +0000
Subject: [Speed] Lazy-loading to decrease python_startup time
In-Reply-To: <20170624105949.62590bbf@fsol>
References: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>
 <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q@mail.gmail.com>
 <20170624105949.62590bbf@fsol>
Message-ID: <CAP1=2W7P1fi+xyp3SQw72QhCMNfydhw0ivtKSiaxu98+GDLtMA@mail.gmail.com>

On Sat, 24 Jun 2017 at 02:00 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 23 Jun 2017 23:03:57 +0000
> Brett Cannon <brett at python.org> wrote:
> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
> wrote:
> >
> > > As suggested, I'd like to discuss if lazy-loading is an option for
> > > improving python-startup time.And if could be done inside the scope of
> a
> > > GSoc project.
> > >
> >
> > It's a possibility and it could be done in the scope of a GSoC project
> > easily. Basically what would be needed is an importlib.util.lazy_import()
> > function which does mostly what is outlined in
> >
> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
> > but
> > where the proper lazy loader is set on the spec object as an extra step.
> > Then every module that is used during startup would use
> > importlib.util.lazy_import() for importing instead of the normal import
> > statement.
>
> My experience is that:
> - you want lazy imports to be implicit, i.e. they should work using the
>   "import" statement and not any special syntax or function invocation
> - you want a blacklist and/or whitelist mechanism to restrict lazy
>   imports to a particular set of modules and packages, because some
>   modules may not play well with lazy importing (e.g. anything that
>   registers plugins, specializations -- think singledispatch --, etc.)
>
> For example, I may want to register the "tornado", "asyncio" and "numpy"
> namespaces / packages for lazy importing, but not the "numba" namespace
> as it uses registration mechanisms quite heavily.
>
> (and the stdlib could be part of the default lazy import whitelist)
>

That's all true for an end user's application, but for the stdlib where
there is no knock-on effects from dependencies not being loaded lazily I
don't think it's quite as critical. Plus lazy loading does make debugging
harder by making loads that trigger an exception happen at the line of
first use instead of at the import line, so I don't know if it's desirable
to automatically make the whole stdlib be lazily loaded from the outset
(which is what would be required since doing it in e.g. sitecustomize.py
wouldn't happen until after startup anyway).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170624/5730bd47/attachment.html>

From solipsis at pitrou.net  Sat Jun 24 12:49:56 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 24 Jun 2017 18:49:56 +0200
Subject: [Speed] Lazy-loading to decrease python_startup time
References: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>
 <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q@mail.gmail.com>
 <20170624105949.62590bbf@fsol>
 <CAP1=2W7P1fi+xyp3SQw72QhCMNfydhw0ivtKSiaxu98+GDLtMA@mail.gmail.com>
Message-ID: <20170624184956.16dd1656@fsol>

On Sat, 24 Jun 2017 16:28:29 +0000
Brett Cannon <brett at python.org> wrote:
> >
> > My experience is that:
> > - you want lazy imports to be implicit, i.e. they should work using the
> >   "import" statement and not any special syntax or function invocation
> > - you want a blacklist and/or whitelist mechanism to restrict lazy
> >   imports to a particular set of modules and packages, because some
> >   modules may not play well with lazy importing (e.g. anything that
> >   registers plugins, specializations -- think singledispatch --, etc.)
> >
> > For example, I may want to register the "tornado", "asyncio" and "numpy"
> > namespaces / packages for lazy importing, but not the "numba" namespace
> > as it uses registration mechanisms quite heavily.
> >
> > (and the stdlib could be part of the default lazy import whitelist)
> >  
> 
> That's all true for an end user's application, but for the stdlib where
> there is no knock-on effects from dependencies not being loaded lazily I
> don't think it's quite as critical. Plus lazy loading does make debugging
> harder by making loads that trigger an exception happen at the line of
> first use instead of at the import line, so I don't know if it's desirable
> to automatically make the whole stdlib be lazily loaded from the outset
> (which is what would be required since doing it in e.g. sitecustomize.py
> wouldn't happen until after startup anyway).

Yes, you are right.  I was assuming that if we take the time to include
a lazy import system, we'd make it available for third-party
applications, though ;-)

Regards

Antoine.



From brett at python.org  Sat Jun 24 13:04:10 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 24 Jun 2017 17:04:10 +0000
Subject: [Speed] Lazy-loading to decrease python_startup time
In-Reply-To: <20170624184956.16dd1656@fsol>
References: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>
 <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q@mail.gmail.com>
 <20170624105949.62590bbf@fsol>
 <CAP1=2W7P1fi+xyp3SQw72QhCMNfydhw0ivtKSiaxu98+GDLtMA@mail.gmail.com>
 <20170624184956.16dd1656@fsol>
Message-ID: <CAP1=2W5eVqXYEhZHRFT5r3hsvx-Au3XEavXKofv-V86=jmMDkg@mail.gmail.com>

On Sat, 24 Jun 2017 at 09:50 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 24 Jun 2017 16:28:29 +0000
> Brett Cannon <brett at python.org> wrote:
> > >
> > > My experience is that:
> > > - you want lazy imports to be implicit, i.e. they should work using the
> > >   "import" statement and not any special syntax or function invocation
> > > - you want a blacklist and/or whitelist mechanism to restrict lazy
> > >   imports to a particular set of modules and packages, because some
> > >   modules may not play well with lazy importing (e.g. anything that
> > >   registers plugins, specializations -- think singledispatch --, etc.)
> > >
> > > For example, I may want to register the "tornado", "asyncio" and
> "numpy"
> > > namespaces / packages for lazy importing, but not the "numba" namespace
> > > as it uses registration mechanisms quite heavily.
> > >
> > > (and the stdlib could be part of the default lazy import whitelist)
> > >
> >
> > That's all true for an end user's application, but for the stdlib where
> > there is no knock-on effects from dependencies not being loaded lazily I
> > don't think it's quite as critical. Plus lazy loading does make debugging
> > harder by making loads that trigger an exception happen at the line of
> > first use instead of at the import line, so I don't know if it's
> desirable
> > to automatically make the whole stdlib be lazily loaded from the outset
> > (which is what would be required since doing it in e.g. sitecustomize.py
> > wouldn't happen until after startup anyway).
>
> Yes, you are right.  I was assuming that if we take the time to include
> a lazy import system, we'd make it available for third-party
> applications, though ;-)
>

That's step 2. :) While EIBTI for the lazy_import(), "practicality beats
purity" in making it easy to just flip a switch to make all imports lazy.
Plus I have not thought through the design of the "switch" solution yet
while the function solution is already done thanks to
importlib.import_module() (which I honestly thought of giving a 'lazy'
keyword-only argument to, but since the algorithm would change I figured it
was probably best not to go down that route).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170624/42b9e577/attachment.html>

From bhavishyagopesh at gmail.com  Fri Jun 23 15:17:30 2017
From: bhavishyagopesh at gmail.com (Bhavishya)
Date: Sat, 24 Jun 2017 00:47:30 +0530
Subject: [Speed] Lazy-loading to decrease python_startup time
Message-ID: <CAFRL-_fG+7FUTN-8DexCHSvkenU2gakTjUk4EC1gqmm=R0kCyA@mail.gmail.com>

As suggested, I'd like to discuss if lazy-loading is an option for
improving python-startup time.And if could be done inside the scope of a
GSoc project.

Thank You

Regards,
Bhavishya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170624/d131ea2b/attachment-0001.html>

From bhavishyagopesh at gmail.com  Sun Jun 25 03:02:31 2017
From: bhavishyagopesh at gmail.com (Bhavishya)
Date: Sun, 25 Jun 2017 12:32:31 +0530
Subject: [Speed] Speed Digest, Vol 36, Issue 1
In-Reply-To: <mailman.1.1498320002.9306.speed@python.org>
References: <mailman.1.1498320002.9306.speed@python.org>
Message-ID: <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>

Hello,
I have added the "lazy_import"
<https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py>
function but still working on adding  it  implicitly(To ensure that at
startup it is actually  used.)

Thanks Antoine for the suggestion

Regards,
Bhavishya

On Sat, Jun 24, 2017 at 9:30 PM, <speed-request at python.org> wrote:

> Send Speed mailing list submissions to
>         speed at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.python.org/mailman/listinfo/speed
> or, via email, send a message with subject or body 'help' to
>         speed-request at python.org
>
> You can reach the person managing the list at
>         speed-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Speed digest..."
>
>
> Today's Topics:
>
>    1. Re: Lazy-loading to decrease python_startup time (Brett Cannon)
>    2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 23 Jun 2017 23:03:57 +0000
> From: Brett Cannon <brett at python.org>
> To: Bhavishya <bhavishyagopesh at gmail.com>, speed at python.org,  Ramya
>         Meruva <meruvaramya116 at gmail.com>, Victor Stinner
>         <victor.stinner at gmail.com>
> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
> Message-ID:
>         <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q at mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com> wrote:
>
> > As suggested, I'd like to discuss if lazy-loading is an option for
> > improving python-startup time.And if could be done inside the scope of a
> > GSoc project.
> >
>
> It's a possibility and it could be done in the scope of a GSoC project
> easily. Basically what would be needed is an importlib.util.lazy_import()
> function which does mostly what is outlined in
> https://docs.python.org/3/library/importlib.html#approximating-importlib-
> import-module
> but
> where the proper lazy loader is set on the spec object as an extra step.
> Then every module that is used during startup would use
> importlib.util.lazy_import() for importing instead of the normal import
> statement. What this would do is help guarantee that all modules that are
> identified as part of startup never import a module needlessly as the lazy
> loader would simply postpone the load until necessary. This would also
> allow for pulling out local imports that are currently done in modules that
> are a part of startup and make them global so they are more visible.
>
> But I have no idea if this will actually speed things up. :) At worst it
> would slow things down ever so slightly due to the extra overhead of lazy
> loading for things that are known to be needed already during startup. At
> best, though, is we accidentally discover modules that are being imported
> needlessly at startup as well as not having to hide imports in functions
> for performance. This fact that it may not be worth it is why I haven't
> bothered to try it out yet.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/speed/attachments/
> 20170623/ed7416e2/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Sat, 24 Jun 2017 10:59:49 +0200
> From: Antoine Pitrou <solipsis at pitrou.net>
> To: speed at python.org
> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
> Message-ID: <20170624105949.62590bbf at fsol>
> Content-Type: text/plain; charset=US-ASCII
>
> On Fri, 23 Jun 2017 23:03:57 +0000
> Brett Cannon <brett at python.org> wrote:
> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
> wrote:
> >
> > > As suggested, I'd like to discuss if lazy-loading is an option for
> > > improving python-startup time.And if could be done inside the scope of
> a
> > > GSoc project.
> > >
> >
> > It's a possibility and it could be done in the scope of a GSoC project
> > easily. Basically what would be needed is an importlib.util.lazy_import()
> > function which does mostly what is outlined in
> > https://docs.python.org/3/library/importlib.html#
> approximating-importlib-import-module
> > but
> > where the proper lazy loader is set on the spec object as an extra step.
> > Then every module that is used during startup would use
> > importlib.util.lazy_import() for importing instead of the normal import
> > statement.
>
> My experience is that:
> - you want lazy imports to be implicit, i.e. they should work using the
>   "import" statement and not any special syntax or function invocation
> - you want a blacklist and/or whitelist mechanism to restrict lazy
>   imports to a particular set of modules and packages, because some
>   modules may not play well with lazy importing (e.g. anything that
>   registers plugins, specializations -- think singledispatch --, etc.)
>
> For example, I may want to register the "tornado", "asyncio" and "numpy"
> namespaces / packages for lazy importing, but not the "numba" namespace
> as it uses registration mechanisms quite heavily.
>
> (and the stdlib could be part of the default lazy import whitelist)
>
> Regards
>
> Antoine.
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
>
> ------------------------------
>
> End of Speed Digest, Vol 36, Issue 1
> ************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170625/dd4ad18a/attachment-0001.html>

From bhavishyagopesh at gmail.com  Sun Jun 25 03:22:31 2017
From: bhavishyagopesh at gmail.com (Bhavishya)
Date: Sun, 25 Jun 2017 12:52:31 +0530
Subject: [Speed] Speed Digest, Vol 36, Issue 1
In-Reply-To: <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>
References: <mailman.1.1498320002.9306.speed@python.org>
 <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>
Message-ID: <CAFRL-_eNZZxZW8Ok1ABtJhM92x6WyOqPdRvfcy-fs=Oh0rv1FA@mail.gmail.com>

Also,  the "fatoptimizer" project's compile fails due to missing
"ma_version_tag" (and works on changing it to "ma_version").

Regards,
Bhavishya

On Sun, Jun 25, 2017 at 12:32 PM, Bhavishya <bhavishyagopesh at gmail.com>
wrote:

> Hello,
> I have added the "lazy_import"
> <https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py>
> function but still working on adding  it  implicitly(To ensure that at
> startup it is actually  used.)
>
> Thanks Antoine for the suggestion
>
> Regards,
> Bhavishya
>
> On Sat, Jun 24, 2017 at 9:30 PM, <speed-request at python.org> wrote:
>
>> Send Speed mailing list submissions to
>>         speed at python.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://mail.python.org/mailman/listinfo/speed
>> or, via email, send a message with subject or body 'help' to
>>         speed-request at python.org
>>
>> You can reach the person managing the list at
>>         speed-owner at python.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Speed digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: Lazy-loading to decrease python_startup time (Brett Cannon)
>>    2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 23 Jun 2017 23:03:57 +0000
>> From: Brett Cannon <brett at python.org>
>> To: Bhavishya <bhavishyagopesh at gmail.com>, speed at python.org,  Ramya
>>         Meruva <meruvaramya116 at gmail.com>, Victor Stinner
>>         <victor.stinner at gmail.com>
>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>> Message-ID:
>>         <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q at mail.gm
>> ail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com> wrote:
>>
>> > As suggested, I'd like to discuss if lazy-loading is an option for
>> > improving python-startup time.And if could be done inside the scope of a
>> > GSoc project.
>> >
>>
>> It's a possibility and it could be done in the scope of a GSoC project
>> easily. Basically what would be needed is an importlib.util.lazy_import()
>> function which does mostly what is outlined in
>> https://docs.python.org/3/library/importlib.html#approximati
>> ng-importlib-import-module
>> but
>> where the proper lazy loader is set on the spec object as an extra step.
>> Then every module that is used during startup would use
>> importlib.util.lazy_import() for importing instead of the normal import
>> statement. What this would do is help guarantee that all modules that are
>> identified as part of startup never import a module needlessly as the lazy
>> loader would simply postpone the load until necessary. This would also
>> allow for pulling out local imports that are currently done in modules
>> that
>> are a part of startup and make them global so they are more visible.
>>
>> But I have no idea if this will actually speed things up. :) At worst it
>> would slow things down ever so slightly due to the extra overhead of lazy
>> loading for things that are known to be needed already during startup. At
>> best, though, is we accidentally discover modules that are being imported
>> needlessly at startup as well as not having to hide imports in functions
>> for performance. This fact that it may not be worth it is why I haven't
>> bothered to try it out yet.
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://mail.python.org/pipermail/speed/attachments/20170623
>> /ed7416e2/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sat, 24 Jun 2017 10:59:49 +0200
>> From: Antoine Pitrou <solipsis at pitrou.net>
>> To: speed at python.org
>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>> Message-ID: <20170624105949.62590bbf at fsol>
>> Content-Type: text/plain; charset=US-ASCII
>>
>> On Fri, 23 Jun 2017 23:03:57 +0000
>> Brett Cannon <brett at python.org> wrote:
>> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
>> wrote:
>> >
>> > > As suggested, I'd like to discuss if lazy-loading is an option for
>> > > improving python-startup time.And if could be done inside the scope
>> of a
>> > > GSoc project.
>> > >
>> >
>> > It's a possibility and it could be done in the scope of a GSoC project
>> > easily. Basically what would be needed is an
>> importlib.util.lazy_import()
>> > function which does mostly what is outlined in
>> > https://docs.python.org/3/library/importlib.html#approximati
>> ng-importlib-import-module
>> > but
>> > where the proper lazy loader is set on the spec object as an extra step.
>> > Then every module that is used during startup would use
>> > importlib.util.lazy_import() for importing instead of the normal import
>> > statement.
>>
>> My experience is that:
>> - you want lazy imports to be implicit, i.e. they should work using the
>>   "import" statement and not any special syntax or function invocation
>> - you want a blacklist and/or whitelist mechanism to restrict lazy
>>   imports to a particular set of modules and packages, because some
>>   modules may not play well with lazy importing (e.g. anything that
>>   registers plugins, specializations -- think singledispatch --, etc.)
>>
>> For example, I may want to register the "tornado", "asyncio" and "numpy"
>> namespaces / packages for lazy importing, but not the "numba" namespace
>> as it uses registration mechanisms quite heavily.
>>
>> (and the stdlib could be part of the default lazy import whitelist)
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Speed mailing list
>> Speed at python.org
>> https://mail.python.org/mailman/listinfo/speed
>>
>>
>> ------------------------------
>>
>> End of Speed Digest, Vol 36, Issue 1
>> ************************************
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170625/fba35396/attachment-0001.html>

From brett at python.org  Sun Jun 25 12:11:59 2017
From: brett at python.org (Brett Cannon)
Date: Sun, 25 Jun 2017 16:11:59 +0000
Subject: [Speed] Speed Digest, Vol 36, Issue 1
In-Reply-To: <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>
References: <mailman.1.1498320002.9306.speed@python.org>
 <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>
Message-ID: <CAP1=2W7qqRPGjK+PhS=e8cCa6=KkdMpTawsffeZwv_BU=3hjuQ@mail.gmail.com>

An easier way to guarantee its use is to have the lazy import print
something under 'Python -v' and then make sure every logged import has a
corresponding lazy message to go with it.

On Sun, Jun 25, 2017, 08:33 Bhavishya, <bhavishyagopesh at gmail.com> wrote:

> Hello,
> I have added the "lazy_import"
> <https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py>
> function but still working on adding  it  implicitly(To ensure that at
> startup it is actually  used.)
>
> Thanks Antoine for the suggestion
>
> Regards,
> Bhavishya
>
> On Sat, Jun 24, 2017 at 9:30 PM, <speed-request at python.org> wrote:
>
>> Send Speed mailing list submissions to
>>         speed at python.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://mail.python.org/mailman/listinfo/speed
>> or, via email, send a message with subject or body 'help' to
>>         speed-request at python.org
>>
>> You can reach the person managing the list at
>>         speed-owner at python.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Speed digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: Lazy-loading to decrease python_startup time (Brett Cannon)
>>    2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 23 Jun 2017 23:03:57 +0000
>> From: Brett Cannon <brett at python.org>
>> To: Bhavishya <bhavishyagopesh at gmail.com>, speed at python.org,  Ramya
>>         Meruva <meruvaramya116 at gmail.com>, Victor Stinner
>>         <victor.stinner at gmail.com>
>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>> Message-ID:
>>         <CAP1=2W41_xTDt7gkuUWnmYA=+
>> Bz9Ox-XT-pAjtXkDGg4TDrY5Q at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com> wrote:
>>
>> > As suggested, I'd like to discuss if lazy-loading is an option for
>> > improving python-startup time.And if could be done inside the scope of a
>> > GSoc project.
>> >
>>
>> It's a possibility and it could be done in the scope of a GSoC project
>> easily. Basically what would be needed is an importlib.util.lazy_import()
>> function which does mostly what is outlined in
>>
>> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
>> but
>> where the proper lazy loader is set on the spec object as an extra step.
>> Then every module that is used during startup would use
>> importlib.util.lazy_import() for importing instead of the normal import
>> statement. What this would do is help guarantee that all modules that are
>> identified as part of startup never import a module needlessly as the lazy
>> loader would simply postpone the load until necessary. This would also
>> allow for pulling out local imports that are currently done in modules
>> that
>> are a part of startup and make them global so they are more visible.
>>
>> But I have no idea if this will actually speed things up. :) At worst it
>> would slow things down ever so slightly due to the extra overhead of lazy
>> loading for things that are known to be needed already during startup. At
>> best, though, is we accidentally discover modules that are being imported
>> needlessly at startup as well as not having to hide imports in functions
>> for performance. This fact that it may not be worth it is why I haven't
>> bothered to try it out yet.
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.python.org/pipermail/speed/attachments/20170623/ed7416e2/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sat, 24 Jun 2017 10:59:49 +0200
>> From: Antoine Pitrou <solipsis at pitrou.net>
>> To: speed at python.org
>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>> Message-ID: <20170624105949.62590bbf at fsol>
>> Content-Type: text/plain; charset=US-ASCII
>>
>> On Fri, 23 Jun 2017 23:03:57 +0000
>> Brett Cannon <brett at python.org> wrote:
>> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
>> wrote:
>> >
>> > > As suggested, I'd like to discuss if lazy-loading is an option for
>> > > improving python-startup time.And if could be done inside the scope
>> of a
>> > > GSoc project.
>> > >
>> >
>> > It's a possibility and it could be done in the scope of a GSoC project
>> > easily. Basically what would be needed is an
>> importlib.util.lazy_import()
>> > function which does mostly what is outlined in
>> >
>> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
>> > but
>> > where the proper lazy loader is set on the spec object as an extra step.
>> > Then every module that is used during startup would use
>> > importlib.util.lazy_import() for importing instead of the normal import
>> > statement.
>>
>> My experience is that:
>> - you want lazy imports to be implicit, i.e. they should work using the
>>   "import" statement and not any special syntax or function invocation
>> - you want a blacklist and/or whitelist mechanism to restrict lazy
>>   imports to a particular set of modules and packages, because some
>>   modules may not play well with lazy importing (e.g. anything that
>>   registers plugins, specializations -- think singledispatch --, etc.)
>>
>> For example, I may want to register the "tornado", "asyncio" and "numpy"
>> namespaces / packages for lazy importing, but not the "numba" namespace
>> as it uses registration mechanisms quite heavily.
>>
>> (and the stdlib could be part of the default lazy import whitelist)
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Speed mailing list
>> Speed at python.org
>> https://mail.python.org/mailman/listinfo/speed
>>
>>
>> ------------------------------
>>
>> End of Speed Digest, Vol 36, Issue 1
>> ************************************
>>
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170625/c751ee14/attachment.html>

From bhavishyagopesh at gmail.com  Mon Jun 26 03:02:47 2017
From: bhavishyagopesh at gmail.com (Bhavishya)
Date: Mon, 26 Jun 2017 12:32:47 +0530
Subject: [Speed] Speed Digest, Vol 36, Issue 1
In-Reply-To: <CAFRL-_eNZZxZW8Ok1ABtJhM92x6WyOqPdRvfcy-fs=Oh0rv1FA@mail.gmail.com>
References: <mailman.1.1498320002.9306.speed@python.org>
 <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>
 <CAFRL-_eNZZxZW8Ok1ABtJhM92x6WyOqPdRvfcy-fs=Oh0rv1FA@mail.gmail.com>
Message-ID: <CAFRL-_fBBOPHM6STRugVzUdYBsuX+oNAe+-Ai1+Kvb-eaVe8xQ@mail.gmail.com>

Hi,
I'm having some trouble making the lazy_import "implicit", I tried
following approaches:

1)overriding "__import__ " using 'builtins.__import__ =
lazy_import.lazy_import_module' but this can't take more than 2 args
whereas "__import__" takes upto 5 args, so now I have to make changes to
"_bootstrap.py"(which I shouldn't probably make) and __init__.py to
accomadate everything new.

2)Using "meta_path" and "path_hooks" to register custom "Loader" and
"finder" but somehow it's breaking during 'make' ...probably due to
'finder'(unable to import stuff like '.org*')...Working on this.

3)The Whitelist option(that Antoine suggested) ,somewhat hardcode, i.e. to
identify modules and in place of "import" add the lazy_import_module
function.

 Any suggestion?

Regards,
Bhavishya

On Sun, Jun 25, 2017 at 12:52 PM, Bhavishya <bhavishyagopesh at gmail.com>
wrote:

> Also,  the "fatoptimizer" project's compile fails due to missing
> "ma_version_tag" (and works on changing it to "ma_version").
>
> Regards,
> Bhavishya
>
> On Sun, Jun 25, 2017 at 12:32 PM, Bhavishya <bhavishyagopesh at gmail.com>
> wrote:
>
>> Hello,
>> I have added the "lazy_import"
>> <https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py>
>> function but still working on adding  it  implicitly(To ensure that at
>> startup it is actually  used.)
>>
>> Thanks Antoine for the suggestion
>>
>> Regards,
>> Bhavishya
>>
>> On Sat, Jun 24, 2017 at 9:30 PM, <speed-request at python.org> wrote:
>>
>>> Send Speed mailing list submissions to
>>>         speed at python.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         https://mail.python.org/mailman/listinfo/speed
>>> or, via email, send a message with subject or body 'help' to
>>>         speed-request at python.org
>>>
>>> You can reach the person managing the list at
>>>         speed-owner at python.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Speed digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Re: Lazy-loading to decrease python_startup time (Brett Cannon)
>>>    2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Fri, 23 Jun 2017 23:03:57 +0000
>>> From: Brett Cannon <brett at python.org>
>>> To: Bhavishya <bhavishyagopesh at gmail.com>, speed at python.org,  Ramya
>>>         Meruva <meruvaramya116 at gmail.com>, Victor Stinner
>>>         <victor.stinner at gmail.com>
>>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>>> Message-ID:
>>>         <CAP1=2W41_xTDt7gkuUWnmYA=+Bz9Ox-XT-pAjtXkDGg4TDrY5Q at mail.gm
>>> ail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
>>> wrote:
>>>
>>> > As suggested, I'd like to discuss if lazy-loading is an option for
>>> > improving python-startup time.And if could be done inside the scope of
>>> a
>>> > GSoc project.
>>> >
>>>
>>> It's a possibility and it could be done in the scope of a GSoC project
>>> easily. Basically what would be needed is an importlib.util.lazy_import()
>>> function which does mostly what is outlined in
>>> https://docs.python.org/3/library/importlib.html#approximati
>>> ng-importlib-import-module
>>> but
>>> where the proper lazy loader is set on the spec object as an extra step.
>>> Then every module that is used during startup would use
>>> importlib.util.lazy_import() for importing instead of the normal import
>>> statement. What this would do is help guarantee that all modules that are
>>> identified as part of startup never import a module needlessly as the
>>> lazy
>>> loader would simply postpone the load until necessary. This would also
>>> allow for pulling out local imports that are currently done in modules
>>> that
>>> are a part of startup and make them global so they are more visible.
>>>
>>> But I have no idea if this will actually speed things up. :) At worst it
>>> would slow things down ever so slightly due to the extra overhead of lazy
>>> loading for things that are known to be needed already during startup. At
>>> best, though, is we accidentally discover modules that are being imported
>>> needlessly at startup as well as not having to hide imports in functions
>>> for performance. This fact that it may not be worth it is why I haven't
>>> bothered to try it out yet.
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <http://mail.python.org/pipermail/speed/attachments/20170623
>>> /ed7416e2/attachment-0001.html>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Sat, 24 Jun 2017 10:59:49 +0200
>>> From: Antoine Pitrou <solipsis at pitrou.net>
>>> To: speed at python.org
>>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>>> Message-ID: <20170624105949.62590bbf at fsol>
>>> Content-Type: text/plain; charset=US-ASCII
>>>
>>> On Fri, 23 Jun 2017 23:03:57 +0000
>>> Brett Cannon <brett at python.org> wrote:
>>> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
>>> wrote:
>>> >
>>> > > As suggested, I'd like to discuss if lazy-loading is an option for
>>> > > improving python-startup time.And if could be done inside the scope
>>> of a
>>> > > GSoc project.
>>> > >
>>> >
>>> > It's a possibility and it could be done in the scope of a GSoC project
>>> > easily. Basically what would be needed is an
>>> importlib.util.lazy_import()
>>> > function which does mostly what is outlined in
>>> > https://docs.python.org/3/library/importlib.html#approximati
>>> ng-importlib-import-module
>>> > but
>>> > where the proper lazy loader is set on the spec object as an extra
>>> step.
>>> > Then every module that is used during startup would use
>>> > importlib.util.lazy_import() for importing instead of the normal import
>>> > statement.
>>>
>>> My experience is that:
>>> - you want lazy imports to be implicit, i.e. they should work using the
>>>   "import" statement and not any special syntax or function invocation
>>> - you want a blacklist and/or whitelist mechanism to restrict lazy
>>>   imports to a particular set of modules and packages, because some
>>>   modules may not play well with lazy importing (e.g. anything that
>>>   registers plugins, specializations -- think singledispatch --, etc.)
>>>
>>> For example, I may want to register the "tornado", "asyncio" and "numpy"
>>> namespaces / packages for lazy importing, but not the "numba" namespace
>>> as it uses registration mechanisms quite heavily.
>>>
>>> (and the stdlib could be part of the default lazy import whitelist)
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> Speed mailing list
>>> Speed at python.org
>>> https://mail.python.org/mailman/listinfo/speed
>>>
>>>
>>> ------------------------------
>>>
>>> End of Speed Digest, Vol 36, Issue 1
>>> ************************************
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170626/6ad4a4b7/attachment-0001.html>

From brett at python.org  Mon Jun 26 18:05:46 2017
From: brett at python.org (Brett Cannon)
Date: Mon, 26 Jun 2017 22:05:46 +0000
Subject: [Speed] Speed Digest, Vol 36, Issue 1
In-Reply-To: <CAFRL-_fBBOPHM6STRugVzUdYBsuX+oNAe+-Ai1+Kvb-eaVe8xQ@mail.gmail.com>
References: <mailman.1.1498320002.9306.speed@python.org>
 <CAFRL-_ddSyyKQC9gNEQ+S+wRVRo7Z18XMWrbwNNjRakcW-qerg@mail.gmail.com>
 <CAFRL-_eNZZxZW8Ok1ABtJhM92x6WyOqPdRvfcy-fs=Oh0rv1FA@mail.gmail.com>
 <CAFRL-_fBBOPHM6STRugVzUdYBsuX+oNAe+-Ai1+Kvb-eaVe8xQ@mail.gmail.com>
Message-ID: <CAP1=2W5-W11K7AmqQKDxrbptZb=dC8DM_hVowv+=HNRijftuEA@mail.gmail.com>

On Mon, 26 Jun 2017 at 01:07 Bhavishya <bhavishyagopesh at gmail.com> wrote:

> Hi,
> I'm having some trouble making the lazy_import "implicit", I tried
> following approaches:
>
> 1)overriding "__import__ " using 'builtins.__import__ =
> lazy_import.lazy_import_module' but this can't take more than 2 args
> whereas "__import__" takes upto 5 args, so now I have to make changes to
> "_bootstrap.py"(which I shouldn't probably make) and __init__.py to
> accomadate everything new.
>

Don't override __import__ basically ever.


>
> 2)Using "meta_path" and "path_hooks" to register custom "Loader" and
> "finder" but somehow it's breaking during 'make' ...probably due to
> 'finder'(unable to import stuff like '.org*')...Working on this.
>

I don't know what you mean by this, e.g. what is ".org*"?


>
> 3)The Whitelist option(that Antoine suggested) ,somewhat hardcode, i.e. to
> identify modules and in place of "import" add the lazy_import_module
> function.
>

I actually don't think that's what Antoine meant. I think he was suggesting
setting up a finder and loader as normal that would do lazy loading but
controlling it with a whitelist/blacklist to then potentially fail
accordingly so the normal finders/loaders would do the right thing.


>
>  Any suggestion?
>

As I said in my other email, just don't bother with this right now. To
quickly see if things will be faster you should just edit the modules that
are imported at startup directly to use a lazy_import() function and
measure the performance.

-Brett


>
> Regards,
> Bhavishya
>
> On Sun, Jun 25, 2017 at 12:52 PM, Bhavishya <bhavishyagopesh at gmail.com>
> wrote:
>
>> Also,  the "fatoptimizer" project's compile fails due to missing
>> "ma_version_tag" (and works on changing it to "ma_version").
>>
>> Regards,
>> Bhavishya
>>
>> On Sun, Jun 25, 2017 at 12:32 PM, Bhavishya <bhavishyagopesh at gmail.com>
>> wrote:
>>
>>> Hello,
>>> I have added the "lazy_import"
>>> <https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py>
>>> function but still working on adding  it  implicitly(To ensure that at
>>> startup it is actually  used.)
>>>
>>> Thanks Antoine for the suggestion
>>>
>>> Regards,
>>> Bhavishya
>>>
>>> On Sat, Jun 24, 2017 at 9:30 PM, <speed-request at python.org> wrote:
>>>
>>>> Send Speed mailing list submissions to
>>>>         speed at python.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>         https://mail.python.org/mailman/listinfo/speed
>>>> or, via email, send a message with subject or body 'help' to
>>>>         speed-request at python.org
>>>>
>>>> You can reach the person managing the list at
>>>>         speed-owner at python.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of Speed digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>    1. Re: Lazy-loading to decrease python_startup time (Brett Cannon)
>>>>    2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Fri, 23 Jun 2017 23:03:57 +0000
>>>> From: Brett Cannon <brett at python.org>
>>>> To: Bhavishya <bhavishyagopesh at gmail.com>, speed at python.org,  Ramya
>>>>         Meruva <meruvaramya116 at gmail.com>, Victor Stinner
>>>>         <victor.stinner at gmail.com>
>>>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>>>> Message-ID:
>>>>         <CAP1=2W41_xTDt7gkuUWnmYA=+
>>>> Bz9Ox-XT-pAjtXkDGg4TDrY5Q at mail.gmail.com>
>>>> Content-Type: text/plain; charset="utf-8"
>>>>
>>>> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
>>>> wrote:
>>>>
>>>> > As suggested, I'd like to discuss if lazy-loading is an option for
>>>> > improving python-startup time.And if could be done inside the scope
>>>> of a
>>>> > GSoc project.
>>>> >
>>>>
>>>> It's a possibility and it could be done in the scope of a GSoC project
>>>> easily. Basically what would be needed is an
>>>> importlib.util.lazy_import()
>>>> function which does mostly what is outlined in
>>>>
>>>> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
>>>> but
>>>> where the proper lazy loader is set on the spec object as an extra step.
>>>> Then every module that is used during startup would use
>>>> importlib.util.lazy_import() for importing instead of the normal import
>>>> statement. What this would do is help guarantee that all modules that
>>>> are
>>>> identified as part of startup never import a module needlessly as the
>>>> lazy
>>>> loader would simply postpone the load until necessary. This would also
>>>> allow for pulling out local imports that are currently done in modules
>>>> that
>>>> are a part of startup and make them global so they are more visible.
>>>>
>>>> But I have no idea if this will actually speed things up. :) At worst it
>>>> would slow things down ever so slightly due to the extra overhead of
>>>> lazy
>>>> loading for things that are known to be needed already during startup.
>>>> At
>>>> best, though, is we accidentally discover modules that are being
>>>> imported
>>>> needlessly at startup as well as not having to hide imports in functions
>>>> for performance. This fact that it may not be worth it is why I haven't
>>>> bothered to try it out yet.
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL: <
>>>> http://mail.python.org/pipermail/speed/attachments/20170623/ed7416e2/attachment-0001.html
>>>> >
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Sat, 24 Jun 2017 10:59:49 +0200
>>>> From: Antoine Pitrou <solipsis at pitrou.net>
>>>> To: speed at python.org
>>>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time
>>>> Message-ID: <20170624105949.62590bbf at fsol>
>>>> Content-Type: text/plain; charset=US-ASCII
>>>>
>>>> On Fri, 23 Jun 2017 23:03:57 +0000
>>>> Brett Cannon <brett at python.org> wrote:
>>>> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh at gmail.com>
>>>> wrote:
>>>> >
>>>> > > As suggested, I'd like to discuss if lazy-loading is an option for
>>>> > > improving python-startup time.And if could be done inside the scope
>>>> of a
>>>> > > GSoc project.
>>>> > >
>>>> >
>>>> > It's a possibility and it could be done in the scope of a GSoC project
>>>> > easily. Basically what would be needed is an
>>>> importlib.util.lazy_import()
>>>> > function which does mostly what is outlined in
>>>> >
>>>> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module
>>>> > but
>>>> > where the proper lazy loader is set on the spec object as an extra
>>>> step.
>>>> > Then every module that is used during startup would use
>>>> > importlib.util.lazy_import() for importing instead of the normal
>>>> import
>>>> > statement.
>>>>
>>>> My experience is that:
>>>> - you want lazy imports to be implicit, i.e. they should work using the
>>>>   "import" statement and not any special syntax or function invocation
>>>> - you want a blacklist and/or whitelist mechanism to restrict lazy
>>>>   imports to a particular set of modules and packages, because some
>>>>   modules may not play well with lazy importing (e.g. anything that
>>>>   registers plugins, specializations -- think singledispatch --, etc.)
>>>>
>>>> For example, I may want to register the "tornado", "asyncio" and "numpy"
>>>> namespaces / packages for lazy importing, but not the "numba" namespace
>>>> as it uses registration mechanisms quite heavily.
>>>>
>>>> (and the stdlib could be part of the default lazy import whitelist)
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Subject: Digest Footer
>>>>
>>>> _______________________________________________
>>>> Speed mailing list
>>>> Speed at python.org
>>>> https://mail.python.org/mailman/listinfo/speed
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> End of Speed Digest, Vol 36, Issue 1
>>>> ************************************
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170626/85125803/attachment.html>

From bhavishyagopesh at gmail.com  Wed Jun 28 01:07:11 2017
From: bhavishyagopesh at gmail.com (Bhavishya)
Date: Wed, 28 Jun 2017 10:37:11 +0530
Subject: [Speed] To start work on "fat-python".
Message-ID: <CAFRL-_fpdwg0yU87cyfPPoi-b1AFG1d-5+ZWzCcRY25sYF=z9Q@mail.gmail.com>

Hello,
As you might have seen(on ML) "Lazy-loading" didn't worked and increased
the startup time instead.

In the meantime I had started with  "fat-python"(adding pure methods etc)
so is that the right thing to do or you suggest something else...

Thank You
Regards,
Bhavishya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20170628/6c9e709e/attachment.html>

From victor.stinner at gmail.com  Wed Jun 28 08:29:35 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 28 Jun 2017 14:29:35 +0200
Subject: [Speed] To start work on "fat-python".
In-Reply-To: <CAFRL-_fpdwg0yU87cyfPPoi-b1AFG1d-5+ZWzCcRY25sYF=z9Q@mail.gmail.com>
References: <CAFRL-_fpdwg0yU87cyfPPoi-b1AFG1d-5+ZWzCcRY25sYF=z9Q@mail.gmail.com>
Message-ID: <CAMpsgwYZ342SfCcL3Ud6C3bxEzP+L6fzeCjNfdBrh=eEi4E-Zw@mail.gmail.com>

Hi,

FYI last year I proposed FAT Python as a GSoC project, but I failed to
pick a candidate. You may be interested by my project page:
http://fatoptimizer.readthedocs.io/en/latest/gsoc.html

And the TODO list:
http://fatoptimizer.readthedocs.io/en/latest/todo.html

The good news is that I rebased my FAT Python project recently on Git.
Previously, my fork was based on Mercurial, but CPython moved to Git
in the meanwhile.

To really test FAT Python, not only fatoptimizer, see instructions at:
https://faster-cpython.readthedocs.io/fat_python.html#getting-started

Sadly, it seems like there are bugs since I rebased my patches. Even
if I disabled function inlinling, I still get this strange error:
---
haypo at selma$ ./python -X fat -m test -v test_sys
...
test test_sys crashed -- Traceback (most recent call last):
  ...
  File "/home/haypo/prog/python/fatpython/Lib/test/libregrtest/save_env.py",
line 286, in get_sys_argv
AttributeError: 'list' object has no attribute 'description'
---

It's a bad in the optimizer, I don't get it when the FAT mode is disabled:
---
haypo at selma$ ./python -m test test_sys
...
Tests result: SUCCESS
---

Moreover, it seems like once .pyc files are created in FAT mode,
non-FAT mode (default mode) still uses the optimized .pyc files, which
is a bad :-/

haypo at selma$ ./python -X fat -m test -v test_sys
^C
haypo at selma$ find -name "*.pyc"
...
./Lib/test/__pycache__/__main__.cpython-37.pyc
...

The filename should contain "fat" somewhere. I made a mistake in my
PEP 511 backport:
http://bugs.python.org/issue26145

Good luck with all these things :-) FAT Python is a big project. The
problem is that I failed to prove that it really makes Python faster.
I stopped the project just after Dave Malcolm implemented function
inlining (which is still experimental = buggy), whereas this
optimization is likely to make CPython faster in a significant way.
It's just that I lost motivation, I put too much energy in this
project :-) So I switch to something else (which was FASTCALL).

Victor

2017-06-28 7:07 GMT+02:00 Bhavishya <bhavishyagopesh at gmail.com>:
> Hello,
> As you might have seen(on ML) "Lazy-loading" didn't worked and increased the
> startup time instead.
>
> In the meantime I had started with  "fat-python"(adding pure methods etc) so
> is that the right thing to do or you suggest something else...
>
> Thank You
> Regards,
> Bhavishya

From songofacandy at gmail.com  Fri Jun 30 08:31:05 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 30 Jun 2017 21:31:05 +0900
Subject: [Speed] About issue 30815
Message-ID: <CAEfz+TzhbMOr7VxX=8V46RzdH_LHg+cywkN4kM3sXnbuekZX9A@mail.gmail.com>

Hi, Bhavishya.

I filed new issue: https://bugs.python.org/issue30815

This issue may be relatively easy and small.
I think it's good for your first step of successful optimization.

If you want to solve it, please assign yourself at issue tracker.

Regards,

INADA Naoki  <songofacandy at gmail.com>

From victor.stinner at gmail.com  Fri Jun 30 08:54:10 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 30 Jun 2017 14:54:10 +0200
Subject: [Speed] About issue 30815
In-Reply-To: <CAEfz+TzhbMOr7VxX=8V46RzdH_LHg+cywkN4kM3sXnbuekZX9A@mail.gmail.com>
References: <CAEfz+TzhbMOr7VxX=8V46RzdH_LHg+cywkN4kM3sXnbuekZX9A@mail.gmail.com>
Message-ID: <CAMpsgwYM0kDw9RMYs1p31DqZvBLhf4YZuP9AVSMD4dUXB1bPBA@mail.gmail.com>

2017-06-30 14:31 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:
> This issue may be relatively easy and small.
> I think it's good for your first step of successful optimization.

I'm not sure about "easy and small", but since I wrote a similar patch
5 years ago (and that I wrote _PyUnicodeWriter), I would be interested
to see benchmark results in 2017 :-) As I wrote in the issue, two
different implementations for StringIO was be required according to
the author of the C implementation of StringIO, Antoine Pitrou.

Victor