write csv to object and read into pandas

[Peter Otten]

Vincent Davis wrote:

> I have a csv file I have to make some changes to before I read it into
> pandas. Currently I open the csv read each row, make changes and save it
> to a new file. Then read it into pandas with pandas.read_csv(). How do I
> skip writing the file to disk? Using python3.5.
> 
> This is what I am doing now.
> 
> with open(infile,"r") as fin:
>         with open(outfile,"w") as fout:
>             writer=csv.writer(fout)
>             for row in csv.reader(fin):
>                 #do stuff to the row
>                 writer.writerow(row)
> 
> df = pandas.csv_reader(outfile)

If you can implement "do stuff to the row" in pandas I'd recommend that.
If not, the following should work (though I'm not a pandas expert):

def preprocess(filename):
    with open(filename) as f:
        for row in csv.reader(f):
            # do stuff
            yield row

rows = preprocess("pandas.csv")
df = pandas.DataFrame.from_records(rows, columns=next(rows))
df = df.convert_objects(convert_numeric=True)

If the csv file doesn't have a header row omit the

columns=next(rows)

argument.