Memory error while using pandas dataframe

naren narencr7 at gmail.com
Mon Jun 8 03:32:38 EDT 2015


Memory Error while working with pandas dataframe.

Description of Environment Windows 7 python 3.4.2 32-bit version pandas
0.16.0

We are running into the error described below. Any help provided will be
sincerely appreciated.

We are able to read a 300MB Csv file into a dataframe using the read_csv
function. While working with the dataframe we ran into memory error. We
used the pd.Concat function to concatenate two dataframes. So we decided to
use chunksize for lazy reading. Chunking returns an object of type
TextFileReader.

http://pandas.pydata.org/pandas-docs/stable/io.html#iterating-through-files-chunk-by-chunk

We are able to iterate over this object once as a debugging measure. The
iterator gets exhausted after iterating once. So we are not able to convert
the TextFileReader object back into a dataframe, using the pd.concat
function.

Error

Traceback (most recent call last):
  File "psindia.py", line 60, in <module>
    data=pd.concat(tp,ignore_index=True)
  File "C:\Python34\lib\site-packages\pandas\tools\merge.py", line 754, in conca
t
    copy=copy)
  File "C:\Python34\lib\site-packages\pandas\tools\merge.py", line 799, in __ini
t__
    raise ValueError('All objects passed were None')ValueError: All
objects passed were None

Thanks for your time.

-- 
Narendran Elango
B.Tech(2014)
Department of Computer Engineering
National Institute of Technology Karnataka, Surathkal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150608/2708ce36/attachment.html>


More information about the Python-list mailing list