Memory error due to the huge/huge input file size

[James Mills]

On Tue, Nov 11, 2008 at 7:47 AM,  <tejsupra at gmail.com> wrote:
> refSeqIDsinTransPro = []
> promoterSequencesinTransPro = []
> reader2 = csv.reader(open(sys.argv[2],"rb"))
> reader2_list = []
> reader2_list.extend(reader2)

Without testing, this looks like you're reading the _ENTIRE_
input stream into memory! Try this:

def readCSV(file):

   if type(file) == str:
      fd = open(file, "rU")
   else:
      fd = file

   sniffer = csv.Sniffer()
   dialect = sniffer.sniff(fd.readline())
   fd.seek(0)

   reader = csv.reader(fd, dialect)
   for line in reader:
      yield line

for line in readCSV(open("foo.csv", "r")):
   ...

--JamesMills

-- 
--
-- "Problems are solved by method"