How do I speedup this loop?

[george young]

On Tue, 13 Jul 2004 16:48:36 +1000
Steve <nospam at nopes> threw this fish to the penguins:
> 
> I'm getting some output by running a command using os.popen. I need to 
> parse the output and transform it in some sense so that it's 'DB 
> compatible', (i.e I need to store the output in a database (postgres) 
> after escaping some characters). Since I'm new to python, I wasn't sure 
> if there was a better way of doing this so this is what I did:
> 
>              # Parse the output returned by popen and return the script
> 	    out = os.popen('some command')
> 	    all_lines = out.readlines()
> 
>              script = []
>              for i in xrange(len(all_lines)):
>                  line = all_lines[i].replace("'", "\\'")[0:len(line)-1] 
> # replace ' with \'
>                  line_without_carriage = line[0:len(line)-1] # remove 
> carriage
>                  line_without_carriage = 
> line_without_carriage.replace("\\n", "$___n") # replace end of line with 
> $___n
>                  line_without_carriage += "@___n" # add a 'end of line' 
> character to the end
>                  script.append(line_without_carriage)
>              # end for
> 	
>              script = ''.join(script)

How about:

lines = []
out = os.popen('some command')
for l in out:
    lines.append(l.strip())
script = ''.join(lines)
out.close()

The "strip" actually removes white space from front and back of the string;
you could say l.strip('\n') if you only want the newlines removed (or '\r'
if they're really carriage return characters.)

Or if you want a clever (and most CPU efficient!) one-liner:

script = [l.strip() for l in os.popen('some command')]

I'm not advocating such a terse one-liner unless you are very comfortable
with it's meaning; will you easily know what it does when you see it
six months from now in the heat of battle?

Also, the one-liner does not allow you to explicitly close the file
descriptor from popen.  This could be a serious problem if it gets run
hundreds of times in a loop.

Have fun,
	-- George Young

-- 
"Are the gods not just?"  "Oh no, child.
What would become of us if they were?" (CSL)