Reading *.json from URL - json.loads() versus urllib.urlopen.readlines()

[Dave Angel]

On 05/27/2013 04:47 PM, Bryan Britten wrote:
> Hey, everyone!
>
> I'm very new to Python and have only been using it for a couple of days, but have some experience in programming (albeit mostly statistical programming in SAS or R) so I'm hoping someone can answer this question in a technical way, but without using an abundant amount of jargon.
>
> The issue I'm having is that I'm trying to pull information from a website to practice Python with, but I'm having trouble getting the data in a timely fashion. If I use the following code:
>
> <code>
> import json
> import urllib
>
> urlStr = "https://stream.twitter.com/1/statuses/sample.json"
>
> twtrDict = [json.loads(line) for line in urllib.urlopen(urlStr)]
> </code>
>
> I get a memory issue. I'm running 32-bit Python 2.7 with 4 gigs of RAM if that helps at all.

Which OS?

The first question I'd ask is how big this file is.  I can't tell, since 
it needs a user name & password to actually get the file.  But it's not 
unusual to need at least double that space in memory, and in Windoze 
you're limited to two gig max, regardless of how big your hardware might be.

If you separately fetch the file, then you can experiment with it, 
including cutting it down to a dozen lines, and see if you can deal with 
that much.

How could you fetch it?  With wget, with a browser (and saveAs), with a 
simple loop which uses read(4096) repeatedly and writes each block to a 
local file.  Don't forget to use 'wb', as you don't know yet what line 
endings it might use.

Once you have an idea what the data looks like, you can answer such 
questions as whether it's json at all, whether the lines each contain a 
single json record, or what.

For all we know, the file might be a few terabytes in size.


-- 
DaveA