[issue31993] pickle.dump allocates unnecessary temporary bytes / str
Olivier Grisel
report at bugs.python.org
Thu Nov 9 17:21:24 EST 2017
Olivier Grisel <olivier.grisel at ensta.org> added the comment:
More benchmarks with the unix time command:
```
(py37) ogrisel at ici:~/code/cpython$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
(py37) ogrisel at ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle
Allocating source data...
=> peak memory usage: 2.014 GB
Dumping to disk...
done in 10.677s
=> peak memory usage: 5.936 GB
real 0m11.068s
user 0m0.940s
sys 0m5.204s
(py37) ogrisel at ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle
Allocating source data...
=> peak memory usage: 2.014 GB
Dumping to disk...
done in 5.089s
=> peak memory usage: 5.978 GB
real 0m5.367s
user 0m0.840s
sys 0m4.660s
(py37) ogrisel at ici:~/code/cpython$ git checkout issue-31993-pypickle-dump-mem-optim
Switched to branch 'issue-31993-pypickle-dump-mem-optim'
(py37) ogrisel at ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle
Allocating source data...
=> peak memory usage: 2.014 GB
Dumping to disk...
done in 6.974s
=> peak memory usage: 2.014 GB
real 0m7.300s
user 0m0.368s
sys 0m4.640s
(py37) ogrisel at ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle
Allocating source data...
=> peak memory usage: 2.014 GB
Dumping to disk...
done in 10.873s
=> peak memory usage: 2.014 GB
real 0m11.178s
user 0m0.324s
sys 0m5.100s
(py37) ogrisel at ici:~/code/cpython$ time python ~/tmp/large_pickle_dump.py --use-pypickle
Allocating source data...
=> peak memory usage: 2.014 GB
Dumping to disk...
done in 4.233s
=> peak memory usage: 2.014 GB
real 0m4.574s
user 0m0.396s
sys 0m4.368s
```
User time is always better in the PR than on master but is also much slower than system time (disk access) in any case. System time is much less deterministic.
----------
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31993>
_______________________________________
More information about the Python-bugs-list
mailing list