Load a CSV with different row lengths

[Peter Otten]

Miki Tebeka wrote:

> Greetings,
> 
>> I should've mentioned that I want to import my csv as a data frame or
>> numpy array or as a table.
> If you know the max length of a row, then you can do something like:
>     def gen_rows(stream, max_length):
>         for row in csv.reader(stream):
>             yield row + ([None] * (max_length - len(line))
> 
>     max_length = 10
>     with open('data.csv') as fo:
>         df = pd.DataFrame.from_records(gen_rows(fo, max_length))

With the help of the search engine that must not be named and some trial and 
error I also found a way to use pandas.read_csv():

$ cat data.csv
a,b
a,b,c,d
a,b,c
$ python3
Python 3.3.2+ (default, Feb 28 2014, 00:52:16) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.read_csv("data.csv", names=list(range(4)))
   0  1    2    3
0  a  b  NaN  NaN
1  a  b    c    d
2  a  b    c  NaN

And if the maximum row length is not known here's a modification of Miki's 
recipe:

def gen_rows(stream, max_length=None):
      rows = csv.reader(stream)
      if max_length is None:
          rows = list(rows)
          max_length = max(len(row) for row in rows)
      for row in rows:
          yield row + [None] * (max_length - len(row))

with open('data.csv') as f:
    df = pd.DataFrame.from_records(list(gen_rows(f))) # my version of pandas
                                                      # does not accept a
                                                      # generator