[New-bugs-announce] [issue34690] Store startup modules as C structures for 20%+ startup speed improvement

[Larry Hastings]

New submission from Larry Hastings <larry at hastings.org>:

This patch was sent to me privately by Jeethu Rao at Facebook.  It's a change they're working with internally to improve startup time.  What I've been told by Carl Shapiro at Facebook is that we have their blessing to post it publicly / merge it / build upon it for CPython.  Their patch was written for 3.6, I have massaged it to the point where it minimally works with 3.8.

What the patch does: it takes all the Python modules that are loaded as part of interpreter startup and deserializes the marshalled .pyc file into precreated objects stored as static C data.  You add this .C file to the Python build.  Then there's a patch to Python itself (about 250 lines iirc) that teaches it to load modules from these data structures.

I wrote a quick dumb test harness to compare this patch vs 3.8 stock.  It runs a command line 500 times and uses time.perf_counter to time the process.  On a fast quiescent laptop I observe a 21-22% improvement:

cmdline: ['./python', '-c', 'pass']
500 runs:

sm38
  average time 0.006302303705982922
          best 0.006055746000129147
         worst 0.00816565500008437

clean38
  average time 0.007969956444008858
          best 0.007829047999621253
         worst 0.008812210000542109

improvement 0.20924239043734505 %

cmdline: ['./python', '-c', 'import io']
500 runs:

sm38
  average time 0.006297688038004708
          best 0.005980765999993309
         worst 0.0072462130010535475

clean38
  average time 0.007996319670004595
          best 0.0078091849991324125
         worst 0.009175700999549008

improvement 0.21242667903482038 %


The downside of the patch: for these modules it ignores the Python files on disk--it doesn't even stat them.  If you add stat calls you lose half of the speed improvement.  I believe they added a work-around, where you can set a flag (command-line? environment variable? I don't know, I didn't go looking for it) that tells Python "don't use the frozen modules" and it loads all those files from disk.


I don't propose to merge the patch in its current state.  I think it would need a lot of work both in terms of "doing things the way Python does it" as well as just code smell (the serializer is implemented in both C and Python and jumps back and forth, also the build process for the serialized modules is pretty tiresome).

Is it worth working on?

----------
assignee: larry
components: Interpreter Core
messages: 325401
nosy: larry
priority: normal
severity: normal
stage: needs patch
status: open
title: Store startup modules as C structures for 20%+ startup speed improvement
type: enhancement
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34690>
_______________________________________