From pitam at deepspace.ucsb.edu  Tue Aug 31 02:59:41 2010
From: pitam at deepspace.ucsb.edu (Pitam Mitra)
Date: Mon, 30 Aug 2010 17:59:41 -0700
Subject: [Csv] read a csv file from StringIO
Message-ID: <web-4991453@deepspace.ucsb.edu>

Hi,

I am trying to read a CSV file from memory. Basically, what
I am trying to do is - I am downloading a CSV file from a
web site, and trying to parse it through CSV module,
without saving it to disk. However, CSV behaves in a
different way. For example:

import csv, urllib

url="http://......./xxxxx.csv"
webFile = urllib.urlopen(url)
localFile = open(fname, 'w')
localFile.write(webFile.read())
webFile.close()
localFile.close()

csvReader=csv.reader(open(fname))
for row in csvReader:
    print row


#gives output as: 
#['-2.0000e-03', '-0.0002']
#.......


Same thing, slightly different:

import csv, urllib

url="http://......./xxxxx.csv"
webFile = urllib.urlopen(url)

csvReader=csv.reader(open(webFile.read()))
webFile.close()
for row in csvReader:
    print row


#gives output:
#['-']
#....

How can I get rid of this behaviour?


Pitam

From sjmachin at lexicon.net  Tue Aug 31 03:49:58 2010
From: sjmachin at lexicon.net (John Machin)
Date: Tue, 31 Aug 2010 11:49:58 +1000
Subject: [Csv] read a csv file from StringIO
In-Reply-To: <web-4991453@deepspace.ucsb.edu>
References: <web-4991453@deepspace.ucsb.edu>
Message-ID: <4C7C5FC6.7070507@lexicon.net>

On 31/08/2010 10:59 AM, Pitam Mitra wrote:
> Hi,
>
> I am trying to read a CSV file from memory. Basically, what
> I am trying to do is - I am downloading a CSV file from a
> web site, and trying to parse it through CSV module,
> without saving it to disk. However, CSV behaves in a
> different way. For example:
>
> import csv, urllib
>
> url="http://......./xxxxx.csv"
> webFile = urllib.urlopen(url)
> localFile = open(fname, 'w')

For portablility, the above should be 'wb' ... otherwise on Windows 
you'd get a superflous `\r` for each `\n` in the file.

> localFile.write(webFile.read())
> webFile.close()
> localFile.close()
>
> csvReader=csv.reader(open(fname))
> for row in csvReader:
>      print row
>
>
> #gives output as:
> #['-2.0000e-03', '-0.0002']
> #.......
>
>
> Same thing, slightly different:
>
> import csv, urllib
>
> url="http://......./xxxxx.csv"
> webFile = urllib.urlopen(url)
>
> csvReader=csv.reader(open(webFile.read()))

webFile.read() surely must be a str object containing the unparsed csv 
data. I can't image why open(some_data) doesn't raise an exception ... 
open() expects a path to a file. Are you sure that's what you did? 
Better to copy/paste that to type from memory! The output that you got 
is consistent with
csvReader=csv.reader(webFile.read())

Your message has StringIO in the title but you don't use it in the code 
that you've shown. Try this:

data = webFile.read()
pseudofile = StringIO.StringIO(data)
csvReader = csv.reader(pseudofile)

> webFile.close()
> for row in csvReader:
>      print row
>
>
> #gives output:
> #['-']
> #....
>
> How can I get rid of this behaviour?
>
>
> Pitam