How to write fast into a file in python?

[Carlos Nepomuceno]

Thanks Dan! I've never used CPython or PyPy. Will try them later.

I think the main difference between your create_file_numbers_file_like()
 and the fastwrite5.py I sent earlier is that I've used cStringIO 
instead of StringIO. It took 12s less using cStringIO.

My numbers are much greater, but I've used Python 2.7.5 instead:

C:\src\Python>python create_file_numbers.py
time taken to write a file of size 52428800  is  39.1199457743 seconds

time taken to write a file of size 52428800  is  14.8704800436 seconds

time taken to write a file of size 52428800  is  23.0011990985 seconds


I've downloaded bufsock.py and python2x3.py. The later one was hard to remove the source code from the web page.

Can I use them on my projects? I'm not used to the UCI license[1]. What's the difference to the GPL?




[1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html

________________________________
> Date: Sat, 18 May 2013 12:38:30 -0700 
> Subject: Re: How to write fast into a file in python? 
> From: drsalists at gmail.com 
> To: lokeshkoppaka at gmail.com 
> CC: python-list at python.org 
>  
>  
> With CPython 2.7.3: 
> ./t 
> time taken to write a file of size 52428800  is  15.86 seconds 
>  
> time taken to write a file of size 52428800  is  7.91 seconds 
>  
> time taken to write a file of size 52428800  is  9.64 seconds 
>  
>  
> With pypy-1.9: 
> ./t 
> time taken to write a file of size 52428800  is  3.708232 seconds 
>  
> time taken to write a file of size 52428800  is  4.868304 seconds 
>  
> time taken to write a file of size 52428800  is  1.93612 seconds 
>  

> Here's the code: 
> #!/usr/local/pypy-1.9/bin/pypy 
> #!/usr/bin/python 
>  
> import sys 
> import time 
> import StringIO 
>  
> sys.path.insert(0, '/usr/local/lib') 
> import bufsock 
>  
> def create_file_numbers_old(filename, size): 
>      start = time.clock() 
>  
>      value = 0 
>      with open(filename, "w") as f: 
>          while f.tell() < size: 
>              f.write("{0}\n".format(value)) 
>              value += 1 
>  
>      end = time.clock() 
>  
>      print "time taken to write a file of size", size, " is ", (end  
> -start), "seconds \n" 
>  
> def create_file_numbers_bufsock(filename, intended_size): 
>      start = time.clock() 
>  
>      value = 0 
>      with open(filename, "w") as f: 
>          bs = bufsock.bufsock(f) 
>          actual_size = 0 
>          while actual_size < intended_size: 
>              string = "{0}\n".format(value) 
>              actual_size += len(string) + 1 
>              bs.write(string) 
>              value += 1 
>          bs.flush() 
>  
>      end = time.clock() 
>  
>      print "time taken to write a file of size", intended_size, " is ",  
> (end -start), "seconds \n" 
>  
>  
> def create_file_numbers_file_like(filename, intended_size): 
>      start = time.clock() 
>  
>      value = 0 
>      with open(filename, "w") as f: 
>          file_like = StringIO.StringIO() 
>          actual_size = 0 
>          while actual_size < intended_size: 
>              string = "{0}\n".format(value) 
>              actual_size += len(string) + 1 
>              file_like.write(string) 
>              value += 1 
>          file_like.seek(0) 
>          f.write(file_like.read()) 
>  
>      end = time.clock() 
>  
>      print "time taken to write a file of size", intended_size, " is ",  
> (end -start), "seconds \n" 
>  
> create_file_numbers_old('output.txt', 50 * 2**20) 
> create_file_numbers_bufsock('output2.txt', 50 * 2**20) 
> create_file_numbers_file_like('output3.txt', 50 * 2**20) 
>  
>  
>  
>  
> On Thu, May 16, 2013 at 9:35 PM,  
> <lokeshkoppaka at gmail.com<mailto:lokeshkoppaka at gmail.com>> wrote: 
> On Friday, May 17, 2013 8:50:26 AM UTC+5:30,  
> lokesh... at gmail.com<mailto:lokesh... at gmail.com> wrote: 
> > I need to write numbers into a file upto 50mb and it should be fast 
> > 
> > can any one help me how to do that? 
> > 
> > i had written the following code.. 
> > 
> >  
> ----------------------------------------------------------------------------------------------------------- 
> > 
> > def create_file_numbers_old(filename, size): 
> > 
> > start = time.clock() 
> > 
> > 
> > 
> > value = 0 
> > 
> > with open(filename, "w") as f: 
> > 
> > while f.tell()< size: 
> > 
> > f.write("{0}\n".format(value)) 
> > 
> > value += 1 
> > 
> > 
> > 
> > end = time.clock() 
> > 
> > 
> > 
> > print "time taken to write a file of size", size, " is ", (end  
> -start), "seconds \n" 
> > 
> >  
> ------------------------------------------------------------------------------------------------------------------ 
> > 
> > it takes about 20sec i need 5 to 10 times less than that. 
> size = 50mb 
> -- 
> http://mail.python.org/mailman/listinfo/python-list 
>  
>  
> -- http://mail.python.org/mailman/listinfo/python-list