[New-bugs-announce] [issue26307] no PGO for built-in modules with `make profile-opt`

[Χρήστος Γεωργίου (Christos Georgiou)]

New submission from Χρήστος Γεωργίου (Christos Georgiou):

(related to issue #24915)
I discovered that `make profile-opt` does not use the profile information for the builtin-modules (e.g. arraymodule or _pickle) because in the `profile-opt` target there is the following sequence of actions:

    …
    $(MAKE) build_all_merge_profile
    @echo "Rebuilding with profile guided optimizations:"
    $(MAKE) clean
    $(MAKE) build_all_use_profile
    …

The action `$(MAKE) clean` performs an `rm -rf build`, destroying among other things all the *.gcda files generated for the built-in modules.
On my Linux system with gcc, a kludge to `Makefile.pre.in` that works is:

    …
    @echo "Rebuilding with profile guided optimizations:"
    find build -name \*.gcda -print | cpio -o >_modules.gcda.cpio # XXX
    $(MAKE) clean
    cpio -id <_modules.gcda.cpio # XXX
    $(MAKE) build_all_use_profile
    …

but, like I said, it's a kludge and it's POSIX-only (-print0 can be avoided since I believe it's guaranteed that there will be no whitespace-containing-filenames).

Now, if this road is to be taken, I believe the most cross-platform method available to save the profile-generated files is to use a custom python script (at this point, the local profile-generating python executable is functional) and make a tar file, which will be untared back. However, I don't like it much.

I'm willing to provide any necessary patches if someone gives me a "proper" roadmap as to how to handle this issue.

----------
components: Build
messages: 259842
nosy: tzot
priority: normal
severity: normal
status: open
title: no PGO for built-in modules with `make profile-opt`
type: performance
versions: Python 2.7, Python 3.5, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26307>
_______________________________________