How to write fast into a file in python?

[Carlos Nepomuceno]

I've got the following results on my desktop PC (Win7/Python2.7.5):

C:\src\Python>python -m timeit -cvn3 -r3 "execfile('fastwrite2.py')"
raw times: 123 126 125
3 loops, best of 3: 41 sec per loop

C:\src\Python>python -m timeit -cvn3 -r3 "execfile('fastwrite5.py')"
raw times: 34 34.3 34
3 loops, best of 3: 11.3 sec per loop

C:\src\Python>python -m timeit -cvn3 -r3 "execfile('fastwrite6.py')"
raw times: 0.4 0.447 0.391
3 loops, best of 3: 130 msec per loop


If you can just copy a preexisting file it will surely increase the speed to the levels you need, but doing the cStringIO operations can reduce the time in 72%.

Strangely I just realised that the time it takes to complete such scripts is the same no matter what hard drive I choose to run them. The results are the same for an SSD (main drive) and a HDD.

I think it's very strange to take 11.3s to write 50MB (4.4MB/s) sequentially on a SSD which is capable of 140MB/s.

Is that a Python problem? Why does it take the same time on the HDD?


### fastwrite2.py ###  <<< this is your code
size = 50*1024*1024
value = 0
filename = 'fastwrite2.dat'
with open(filename, "w") as f:
    while f.tell()< size:
        f.write("{0}\n".format(value))
        value += 1
    f.close()


### fastwrite5.py ###
import cStringIO
size = 50*1024*1024
value = 0
filename = 'fastwrite5.dat'
x = 0
b = cStringIO.StringIO()
while x < size:
    line = '{0}\n'.format(value)
    b.write(line)
    value += 1
    x += len(line)+1
f = open(filename, 'w')
f.write(b.getvalue())
f.close()
b.close()


### fastwrite6.py ###
import shutil
src = 'fastwrite.dat'
dst = 'fastwrite6.dat'
shutil.copyfile(src, dst)



----------------------------------------
> Date: Fri, 17 May 2013 07:58:43 -0400
> From: davea at davea.name
> To: python-list at python.org
> Subject: Re: How to write fast into a file in python?
>
> On 05/17/2013 12:35 AM, lokeshkoppaka at gmail.com wrote:
>> On Friday, May 17, 2013 8:50:26 AM UTC+5:30, lokesh... at gmail.com wrote:
>>> I need to write numbers into a file upto 50mb and it should be fast
>>>
>>> can any one help me how to do that?
>>>
>>> i had written the following code..
>>>
>>> <SNIP>
>>> value = 0
>>>
>>> with open(filename, "w") as f:
>>>
>>> while f.tell()< size:
>>>
>>> f.write("{0}\n".format(value))
>>> <SNIP more double-spaced nonsense from googlegroups>
> If you must use googlegroups, at least read this
> http://wiki.python.org/moin/GoogleGroupsPython.
>>>
>>>
>>> it takes about 20sec i need 5 to 10 times less than that.
>> size = 50mb
>>
>
> Most of the time is spent figuring out whether the file has reached its
> limit size. If you want Python to go fast, just specify the data. On
> my Linux system, it takes 11 seconds to write the first 6338888 values,
> which is just under 50mb. If I write the obvious loop, writing that
> many values takes .25 seconds.
>
> --
> DaveA
> --
> http://mail.python.org/mailman/listinfo/python-list