Python IO performance?

[Andrew MacIntyre]

On Tue, 3 Jun 2003, Rob Hall wrote:

> Yes, I agree that Python definitely needs some work done on IO.

3 things:-

- Perl has had lots of optimisation applied to its file I/O, as this is
  core to its intent; Python tries to stick close to standard C library
  calls for portability & maintainability.

- I've not built perl myself, but you should check whether Perl is
  configured for multithreading.  Python by default builds with threads
  enabled if the platform has threads and Python supports them.  This can
  make for a not inconsiderable difference in performance, due to the
  impact of thread locks (even if the script doesn't use threads).  I have
  seen an example where this affects performance of pystone by >25%, due
  to the multithreaded C library having locks around malloc() calls.  It
  can also affect I/O performance (I vaguely recall something like this
  being worked around on Windows for file reading).  If you can, try
  building Python without threads, and compare the results.

- regardless of threads, certain object creation/deletion scenarios in
  Python code can expose undesireable performance traits of platform
  malloc() implementations.  I've not heard of a platform on which Python
  hasn't been able to find a nasty side to malloc(), and the custom
  memory manager in Python 2.3 is a major step forward in minimising
  Python's exposure to such issues.

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen  ACT  2616
Web:    http://www.andymac.org/               |        Australia