Python Fast I/o

[Chris Angelico]

On Wed, Jan 15, 2014 at 1:24 AM, Ayushi Dalmia
<ayushidalmia2604 at gmail.com> wrote:
> On Tuesday, January 14, 2014 7:33:08 PM UTC+5:30, Chris Angelico wrote:
>> On Wed, Jan 15, 2014 at 12:50 AM, Ayushi Dalmia
>>
>> <ayushidalmia2604 at gmail.com> wrote:
>>
>> > I need to write into a file for a project which will be evaluated on the basis of time. What is the fastest way to write 200 Mb of data, accumulated as a list into a file.
>>
>> >
>>
>> > Presently I am using this:
>>
>> >
>>
>> > with open('index.txt','w') as f:
>>
>> >       f.write("".join(data))
>>
>> >       f.close()
>>
>>
>>
>> with open('index.txt','w') as f:
>>
>>     for hunk in data:
>>
>>         f.write(hunk)
>>
>>
>>
>> You don't need to f.close() - that's what the 'with' block guarantees.
>>
>> Iterating over data and writing each block separately means you don't
>>
>> have to first build up a 200MB string. After that, your performance is
>>
>> going to be mainly tied to the speed of your disk, not anything that
>>
>> Python can affect.
>>
>>
>>
>> ChrisA

Your quoted text is becoming double spaced, because of bugs in the
Google Groups client. Please either edit this before posting, or
switch to a better newsreader, or use the mailing list:

https://mail.python.org/mailman/listinfo/python-list

Thanks!

> Which is more fast?
> Creating a 200 Mb string and then dumping into a file or dividing the 200 Mb string into chunks and then writing those chunks. Won't writing the chunks call more i/o operation?
>

When you're writing two hundred megabytes, the number of I/O
operations is dominated by that. You have to write that many sectors,
and nothing can change that. Joining your list of strings before
writing incurs the cost of building up a single 200MB string, but even
that is likely to be insignificant in the scheme of things (if you
have the memory available, it won't take very long compared to the
time it takes to write to the disk). Python will buffer its writes, so
you don't have to worry about the details. It's going to do the right
thing for you; you can concentrate on making your code look right.

ChrisA