[issue36694] Excessive memory use or memory fragmentation when unpickling many small objects

[Inada Naoki]

Inada Naoki <songofacandy at gmail.com> added the comment:

I'm using 1/10 version of dump.py
I removed total_size() because it creates some garbages.

sys._debugmallocstats() after load.py:

master:

# arenas allocated total           =                1,223
# arenas reclaimed                 =                    5
# arenas highwater mark            =                1,218
# arenas allocated current         =                1,218
1218 arenas * 262144 bytes/arena   =          319,291,392

# bytes in allocated blocks        =          218,026,128
# bytes in available blocks        =              149,024
23835 unused pools * 4096 bytes    =           97,628,160


PR 13036:

# arenas allocated total           =                  849
# arenas reclaimed                 =                    3
# arenas highwater mark            =                  846
# arenas allocated current         =                  846
846 arenas * 262144 bytes/arena    =          221,773,824

# bytes in allocated blocks        =          217,897,968
# bytes in available blocks        =              140,096
61 unused pools * 4096 bytes       =              249,856

Now "arena allocated current" is same to after dump.py:

# arenas allocated total           =                  847
# arenas reclaimed                 =                    1
# arenas highwater mark            =                  846
# arenas allocated current         =                  846
846 arenas * 262144 bytes/arena    =          221,773,824

# bytes in allocated blocks        =          217,998,792
# bytes in available blocks        =              131,112
38 unused pools * 4096 bytes       =              155,648


It looks nice.  Additionally, both of "time python dump.py"
and "time python load.py" become slightly faster.

master dump:
dump (Note that this time includes not only dump, but also constructing data)
real    0m3.539s
user    0m3.266s
sys     0m0.196s

master load:
real    0m1.408s
user    0m1.292s
sys     0m0.116s

PR-13036 dump:
real    0m2.758s
user    0m2.598s
sys     0m0.088s

PR-13036 load:
real    0m1.239s
user    0m1.183s
sys     0m0.056s


Would pickle experts review the PR?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36694>
_______________________________________