How do I speedup this loop?

[Lonnie Princehouse]

Welcome to Python =)
Somebody else already mentioned checking out the DBI API's way of
escaping data; this is a good idea.  Besides that, here are some
general tips-

1. Consider using out.xreadlines() if you only need one line at a 
  time:
  
  for line in out.xreadlines():
    ...

  If you need all of the data at once, try out.read()

2. You can use negative numbers to index relative to the end of a 
  sequence:
  
  line[0:-1] is equivalent to line[0:len(line)-1] 
  (i.e. cut off the last character of a string)

  You can also use line.strip() to remove trailing whitespace, 
  including newlines.

3. If you omit the index on either side of a slice, Python will 
  default to the beginning and end of a sequence:

  line[:] is equivalent to line[0:len(line)] 
  
4. Check out the regular expression module.  Here's how to read all of
  your output and make multiple escape substitutions.  Smashing this 
  into one regular expression means you only need one pass over the 
  data. It also avoids string concatenation.

  import re, os

  out = os.popen('some command')
  data = out.read()
  
  substitution_map = {
    "'" : r"\'",
    "\n": "$___n",
  }
  
  def sub_func(match_object, smap=substitution_map):
    return smap[match_object.group(0)]

  escape_expr = re.compile('|'.join(substitution_map.keys())))

  escaped_data = escape_expr.sub(sub_func, data)

  # et voila... now you've got a big escaped string without even 
  # writing a single for loop.  Tastes great, less filling.
  
  (caveat: I didn't run this code.  It might have typos.)


Steve <nospam at nopes> wrote in message news:<40f385dc$1 at clarion.carno.net.au>...
> Hi,
> 
> I'm getting some output by running a command using os.popen. I need to 
> parse the output and transform it in some sense so that it's 'DB 
> compatible', (i.e I need to store the output in a database (postgres) 
> after escaping some characters). Since I'm new to python, I wasn't sure 
> if there was a better way of doing this so this is what I did:
> 
> 
>              # Parse the output returned by popen and return the script
> 	    out = os.popen('some command')
> 	    all_lines = out.readlines()
> 
>              script = []
>              for i in xrange(len(all_lines)):
>                  line = all_lines[i].replace("'", "\\'")[0:len(line)-1] 
> # replace ' with \'
>                  line_without_carriage = line[0:len(line)-1] # remove 
> carriage
>                  line_without_carriage = 
> line_without_carriage.replace("\\n", "$___n") # replace end of line with 
> $___n
>       

           line_without_carriage += "@___n" # add a 'end of line' 
> character to the end
>                  script.append(line_without_carriage)
>              # end for
> 	
>              script = ''.join(script)
> 
> Please help because I'm pretty sure I'm wasting a lot of cpu time in 
> this loop. Thanks
> 
> Steve