[Python-Dev] Python 3.0.1 (io-in-c)

[Antoine Pitrou]

Raymond Hettinger <python <at> rcn.com> writes:
> 
> What is involved in finishing io-in-c? 

Off the top of my head:
- fix the _ssl bug which prevents some tests from passing (issue #4967)
- clean up io.py (and decide what to do with the remaining Python code:
basically, the parts of StringIO which are implemented in Python)
- of course, test in various situations, review the code, suggest possible
improvements...

Now here are some performance figures. Text I/O is done in utf-8 with universal
newlines enabled:


=== I/O in C ===

** Binary input **

[ 400KB ] read one unit at a time...                   1.64 MB/s
[ 400KB ] read 20 units at a time...                   27.2 MB/s
[ 400KB ] read 4096 units at a time...                  845 MB/s

[  20KB ] read whole contents at once...                924 MB/s
[ 400KB ] read whole contents at once...                883 MB/s
[  10MB ] read whole contents at once...                980 MB/s

[ 400KB ] seek forward one unit at a time...          0.528 MB/s
[ 400KB ] seek forward 1000 units at a time...          516 MB/s
[ 400KB ] alternate read & seek one unit...            1.33 MB/s
[ 400KB ] alternate read & seek 1000 units...           490 MB/s

** Text input **

[ 400KB ] read one unit at a time...                   2.28 MB/s
[ 400KB ] read 20 units at a time...                   29.2 MB/s
[ 400KB ] read one line at a time...                   71.7 MB/s
[ 400KB ] read 4096 units at a time...                 97.4 MB/s

[  20KB ] read whole contents at once...                108 MB/s
[ 400KB ] read whole contents at once...                112 MB/s
[  10MB ] read whole contents at once...               89.7 MB/s

[ 400KB ] seek forward one unit at a time...         0.0904 MB/s
[ 400KB ] seek forward 1000 units at a time...         87.4 MB/s

** Binary append **

[  20KB ] write one unit at a time...                 0.668 MB/s
[ 400KB ] write 20 units at a time...                  12.2 MB/s
[ 400KB ] write 4096 units at a time...                 722 MB/s
[  10MB ] write 1e6 units at a time...                 1529 MB/s

** Text append **

[  20KB ] write one unit at a time...                 0.983 MB/s
[ 400KB ] write 20 units at a time...                    16 MB/s
[ 400KB ] write 4096 units at a time...                 236 MB/s
[  10MB ] write 1e6 units at a time...                  261 MB/s

** Binary overwrite **

[  20KB ] modify one unit at a time...                0.677 MB/s
[ 400KB ] modify 20 units at a time...                 12.1 MB/s
[ 400KB ] modify 4096 units at a time...                382 MB/s

[ 400KB ] alternate write & seek one unit...          0.212 MB/s
[ 400KB ] alternate write & seek 1000 units...          173 MB/s
[ 400KB ] alternate read & write one unit...          0.827 MB/s
[ 400KB ] alternate read & write 1000 units...          276 MB/s

** Text overwrite **

[  20KB ] modify one unit at a time...                0.296 MB/s
[ 400KB ] modify 20 units at a time...                 5.69 MB/s
[ 400KB ] modify 4096 units at a time...                151 MB/s


=== I/O in Python (branches/py3k) ===

** Binary input **

[ 400KB ] read one unit at a time...                  0.174 MB/s
[ 400KB ] read 20 units at a time...                   3.44 MB/s
[ 400KB ] read 4096 units at a time...                  246 MB/s

[  20KB ] read whole contents at once...                443 MB/s
[ 400KB ] read whole contents at once...                216 MB/s
[  10MB ] read whole contents at once...                274 MB/s

[ 400KB ] seek forward one unit at a time...          0.188 MB/s
[ 400KB ] seek forward 1000 units at a time...          182 MB/s
[ 400KB ] alternate read & seek one unit...          0.0821 MB/s
[ 400KB ] alternate read & seek 1000 units...          81.2 MB/s

** Text input **

[ 400KB ] read one unit at a time...                  0.218 MB/s
[ 400KB ] read 20 units at a time...                    3.8 MB/s
[ 400KB ] read one line at a time...                   3.69 MB/s
[ 400KB ] read 4096 units at a time...                 34.9 MB/s

[  20KB ] read whole contents at once...               70.5 MB/s
[ 400KB ] read whole contents at once...                 81 MB/s
[  10MB ] read whole contents at once...               68.7 MB/s

[ 400KB ] seek forward one unit at a time...         0.0709 MB/s
[ 400KB ] seek forward 1000 units at a time...         67.3 MB/s

** Binary append **

[  20KB ] write one unit at a time...                  0.15 MB/s
[ 400KB ] write 20 units at a time...                  2.88 MB/s
[ 400KB ] write 4096 units at a time...                 346 MB/s
[  10MB ] write 1e6 units at a time...                  728 MB/s

** Text append **

[  20KB ] write one unit at a time...                0.0814 MB/s
[ 400KB ] write 20 units at a time...                  1.51 MB/s
[ 400KB ] write 4096 units at a time...                 118 MB/s
[  10MB ] write 1e6 units at a time...                  218 MB/s

** Binary overwrite **

[  20KB ] modify one unit at a time...                0.123 MB/s
[ 400KB ] modify 20 units at a time...                 2.34 MB/s
[ 400KB ] modify 4096 units at a time...                213 MB/s

[ 400KB ] alternate write & seek one unit...         0.0816 MB/s
[ 400KB ] alternate write & seek 1000 units...         71.4 MB/s
[ 400KB ] alternate read & write one unit...         0.0448 MB/s
[ 400KB ] alternate read & write 1000 units...         41.1 MB/s

** Text overwrite **

[  20KB ] modify one unit at a time...               0.0723 MB/s
[ 400KB ] modify 20 units at a time...                 1.36 MB/s
[ 400KB ] modify 4096 units at a time...               88.3 MB/s

Regards

Antoine.