FW: Reading a remove csv file

[Terry Reedy]

On 11/2/2017 9:18 AM, ROGER GRAYDON CHRISTMAN wrote:
> I have a partial answer to my own question:
> This seems to work for me:
> 
> ---
> link = urllib.request.urlopen(urlpath)
> data = link.read().decode('utf-8').split('\n')
> 
> reader = csv.DictReader(data)
> for row in reader:
> ---
> 
> I think here my concern is that now 'data' is now a variable
> in my program's memory (all of the data),
> instead of streamlike.   I suppose I did read
> somewhere about setting a stream option.

csv.reader and csv.DictReader are transformation iterators whose first 
argument must be an iterable of string lines.  Given an iterator of 
bytes lines, you just need to interpose a bytes to string iterator  -- 
something like (untested)

def strgen(bytesource, encoding):
     for line in bytesource:
         yield line.decode(encoding)

with urllib.request.urlopen(urlpath) as link:
         lines = strgen(link, 'utf-8')
         reader = csv.DictReader(lines)  # plus any other args
         for row in reader:
             process(row)

Iterators are intended to be chained together like this.

-- 
Terry Jan Reedy