The speed of glob()

[Chris Angelico]

On Wed, Jul 29, 2020 at 7:44 PM Gisle Vanem <gisle.vanem at gmail.com> wrote:
>
> Chris Angelico wrote:
>
> >> Has anybody noticed the speed of 'glob()' has
> >> decreased somewhere between v3.6 and v3.10.
> >>
> >> I got these results:
> >>     Python 3.6.5:
> >>       1st run: 0.14694
> >>       2nd run: 0.09506   <- *always* the fastest
> >>     Python 3.7.7:        <- from Nuget
> >>       1st run: 0.12440
> >>       2nd run: 0.09602
> >>     Python 3.10.0:       <- from Git repo
> >>       1st run: 0.15922
> >>       2nd run: 0.12424
> >>
> >> 'glob()' in Python 3.6.5 is consistently 30% faster on
> >> the 2nd run compared to 3.10.0.
> >>
> >
> > Unfortunately that's very hard to compare. If you're building Python
> > from source, the parameters may be VERY different from a prepackaged
> > binary.
>
> I guess so w/o knowing why. I just tried this from MS' AppStore:
> Python 3.8.5:
>    1st run: 0.12121
>    2nd run: 0.07674
>
> Fastest I've tried so far.

That suggests that there's a lot of irrelevant noise in those numbers. Ah well.

> > Are you able to redo this test with more consistent Python builds? For
> > instance, you can go as far as 3.8 using python.org binaries, or
> > alternatively, build all your tests from git (by checking out
> > different branches).
>
> I do not want to put to much stress on this. Maybe I'm just 'ass-u-me'ing
> too much?
>
> The important thing is that any Python3.x is much faster on both
> the 1st and 2nd run compared to my Python 2.7.
>

Ah, now THAT has a number of very good explanations, and most likely
IS meaningful. There've been several optimizations aimed at hammering
the file system less to get the same info, so I'm not at all surprised
that you're seeing an improvement.

BTW, I just noticed something. The path you're using for testing
purposes is "e:/net". Is that network-attached or local? If it's a
remote mount of some sort, then that will make a HUGE difference, and
it also means that the sync command you're using probably won't ensure
that you have a clean testing platform. But if that's local, then no
probs, don't mind me.

ChrisA