[Pythonmac-SIG] MemoryError Problem

[Richard Gordon]

--============_-1223865552==_ma============
Content-Type: text/plain; charset="us-ascii" ; format="flowed"

I finally got around to downloading the source code for "Python & 
Tkinter Programming" and observed that a) there are a lot of files 
and b) the line endings are \r\n. I have usually just done a batch 
Find/Replace in BBEdit to deal with things like this, but decided 
that it was time to sit down and write a python script to change the 
line endings to \r, so off we go.

The code is below and it works pretty good thru about half of the 
files, then dies with a memory error (using Python 2.1b2). I've tried 
jacking the interpreter's memory up to 48000 but it still tipped over 
at about the same point. I couldn't tell you exactly how many files 
are involved, but it's a lot and most are small. There is a 
chapter-based directory structure, but it isn't terribly deep and I 
don't think that this is some kind of recursion limit thing.

So does anyone see something here that may be causing the problem? 
I'm a little suspicious about .xreadlines() but it's really fast and 
I'd like to use it if possible. Any help will be appreciated.

As an aside, I had originally planned to modify the files in place by 
opening each in read mode, doing xreadlines, then opening it again in 
write mode and writing the processed lines back into the file. I 
didn't get any errors when I tried this, but nothing actually was 
written into the file either, so I wound up renaming the originals 
and writing a new file out under the old name. This is not an optimal 
solution and I'd be interested in any suggestions that you may have 
about editing files in place. Thanks.

#code starts here
import sys, os
from time import time

start = time()
fcount = 0

def strip(f):
	global fcount
	infh = open(f, 'rb').xreadlines()
	os.rename(f, f +'~')
	outfh = open(f, 'wb')
	for line in infh:
		outfh.write(line.replace('\n',''))
	outfh.close()
	fcount =+ 1
	return
def run(d):
	names = os.listdir(d)
	for name in names:
		f = d + ':' + name
		if os.path.isfile(f):
			if os.path.splitext(f)[1]=='.py':
				strip(f)
		else: run(f)


run('Conkie:python_scripts:PTP')
stop = time()
elapsed = stop-start
print "Changed %d files in %d seconds" % (fcount, elapsed)

#eof


Richard Gordon
--------------------
Gordon Design
Web Design/Database Development
http://www.richardgordon.net
--============_-1223865552==_ma============
Content-Type: text/enriched; charset="us-ascii"

I finally got around to downloading the source code for "Python &
Tkinter Programming" and observed that a) there are a lot of files and
b) the line endings are \r\n. I have usually just done a batch
Find/Replace in BBEdit to deal with things like this, but decided that
it was time to sit down and write a python script to change the line
endings to \r, so off we go.


The code is below and it works pretty good thru about half of the
files, then dies with a memory error (using Python 2.1b2). I've tried
jacking the interpreter's memory up to 48000 but it still tipped over
at about the same point. I couldn't tell you exactly how many files are
involved, but it's a lot and most are small. There is a chapter-based
directory structure, but it isn't terribly deep and I don't think that
this is some kind of recursion limit thing.


So does anyone see something here that may be causing the problem? I'm
a little suspicious about .xreadlines() but it's really fast and I'd
like to use it if possible. Any help will be appreciated.


As an aside, I had originally planned to modify the files in place by
opening each in read mode, doing xreadlines, then opening it again in
write mode and writing the processed lines back into the file. I didn't
get any errors when I tried this, but nothing actually was written into
the file either, so I wound up renaming the originals and writing a new
file out under the old name. This is not an optimal solution and I'd be
interested in any suggestions that you may have about editing files in
place. Thanks.


#code starts here

<fontfamily><param>Monaco</param><smaller>import sys, os

from time import time


start = time()

fcount = 0


def strip(f):

	global fcount

	infh = open(f, 'rb').xreadlines()

	os.rename(f, f +'~')

	outfh = open(f, 'wb')

	for line in infh:

		outfh.write(line.replace('\n',''))	

	outfh.close()

	fcount =+ 1

	return

def run(d):

	names = os.listdir(d)

	for name in names:

		f = d + ':' + name

		if os.path.isfile(f):

			if os.path.splitext(f)[1]=='.py':

				strip(f)

		else: run(f)	

	


run('Conkie:python_scripts:PTP')

stop = time()

elapsed = stop-start

print "Changed %d files in %d seconds" % (fcount, elapsed)


#eof
</smaller></fontfamily>



Richard Gordon

--------------------

Gordon Design

Web Design/Database Development

http://www.richardgordon.net

--============_-1223865552==_ma============--