[issue22524] PEP 471 implementation: os.scandir() directory scanning function

[STINNER Victor]

STINNER Victor added the comment:

I also ran the benchmark without cache on disk. It looks like my hard drive is too slow to see a real speedup of scandir(). The maximum speedup is 2.7 seconds lesser when testing the "c" scandir (24.089 => 21.374 seconds): "1.1x as fast".

I modified the benchmark to flush all caches, I added the following line into do_scandir_walk():

   os.system("sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'")

I also commented the first call to do_scandir_walk() ("Priming the system's cache..."), just to make the benchmark faster.

(See attached clear_system_cache.patch for all changes.)

Result of the modified benchmark without cache:

haypo at smithers$ python3 setup.py build && for scandir in generic python c; do echo; echo "=== $scandir ==="; PYTHONPATH=build/lib.linux-x86_64-3.3/ python3 benchmark.py /usr/share -c $scandir || break; done
running build
running build_py
running build_ext

=== generic ===
Using very slow generic version of scandir
Comparing against builtin version of os.walk()
Benchmarking walks on /usr/share, repeat 1/3...
Benchmarking walks on /usr/share, repeat 2/3...
Benchmarking walks on /usr/share, repeat 3/3...
os.walk took 24.324s, scandir.walk took 24.215s -- 1.0x as fast

=== python ===
Using slower ctypes version of scandir
Comparing against builtin version of os.walk()
Benchmarking walks on /usr/share, repeat 1/3...
Benchmarking walks on /usr/share, repeat 2/3...
Benchmarking walks on /usr/share, repeat 3/3...
os.walk took 24.089s, scandir.walk took 21.374s -- 1.1x as fast

=== c ===
Using fast C version of scandir
Comparing against builtin version of os.walk()
Benchmarking walks on /usr/share, repeat 1/3...
Benchmarking walks on /usr/share, repeat 2/3...
Benchmarking walks on /usr/share, repeat 3/3...
os.walk took 24.225s, scandir.walk took 21.390s -- 1.1x as fast

----------
Added file: http://bugs.python.org/file36847/clear_system_cache.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22524>
_______________________________________