[Tutor] Processing CSV files

[Sivaram Neelakantan]

On Wed, Oct 09 2013,Leena Gupta wrote:

> Hello,
>
> Looking for some inputs on Python's csv processing feature.
>
> I need to process a large csv file every 5-10 minutes. The file could
> contain 3mill to 10 mill rows and size could be 6MB to 10MB(+). As part of
> the processing, I need to sum up a number value by grouping on certain
> attributes and store the output in a datastore. I wanted to know if Python
> is recommended and can it be used for processing data in csv files of this
> size? Any issues that we need to be aware of? I believe Python has a csv
> library as well.

[snipped 6 lines]

I've found pandas to be very useful for this.  It provides good
functions to read CSVs and higher order functions to sum the generated
dataframes in pandas.


 sivaram
 --