How to write fast into a file in python?

[Steven D'Aprano]

On Thu, 16 May 2013 20:20:26 -0700, lokeshkoppaka wrote:

> I need to write numbers into a file upto 50mb and it should be fast can
> any one help me how to do that?
> i had written the following code..
> ----------------------------------------------------------------------
> def create_file_numbers_old(filename, size): start = time.clock()
> 
> value = 0
> with open(filename, "w") as f:
> while f.tell()< size:
> f.write("{0}\n".format(value))
> value += 1
> 
> end = time.clock()
> 
> print "time taken to write a file of size", size, " is ", (end -start),
> "seconds \n"
> --------------------------------------------------------------------
> it takes about 20sec i need 5 to 10 times less than that.


20 seconds to write how many numbers? If you are doing

create_file_numbers_old(filename, 5)

then 20 seconds is really slow. But if you are doing:

create_file_numbers_old(filename, 50000000000000)

then 20 seconds is amazingly fast.


Try this instead, it may be a little faster:


def create_file_numbers_old(filename, size):
    count = value = 0
    with open(filename, 'w') as f:
        while count < size:
            s = '%d\n' % value
            f.write(s)
            count += len(s)
            value += 1
            

If this is still too slow, you can try three other tactics:

1) pre-calculate the largest integer that will fit in `size` bytes, then 
use a for-loop instead of a while loop:

    maxn = calculation(...)
    with open(filename, 'w') as f:
        for i in xrange(maxn):
            f.write('%d\n' % i)


2) Write an extension module in C that writes to the file.

3) Get a faster hard drive, and avoid writing over a network.


-- 
Steven