[Tutor] memory errors with netCDF

[questions anon]

Hi Python Tutor,
I keep receiving a memory error when processing many netcdf files. I assume
it has something to do with how I loop things and maybe need to close things
off properly.???
In the code below I am looping through a bunch of netcdf files (each file is
hourly data for one month) and within each netcdf file I am outputting a
*png file every three hours. This works for one netcdf file (therefore one
month) but when it begins to process the next netcdf file I receive a memory
error (see below). Since I have tidied some of my code up it seems to
process partly into the second file but then I still receive the memory
error.
I have tried a few suggestions such as:
-Combining the dataset using MFDataset (using NETCDF4) is not an option
because the files do not have unlimited dimension.
- gc.collect() but that just results in a *GEOS_ERROR: bad allocation error*.

-only open LAT and LON once (which worked)

System Details:
Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul  3 2011, 15:13:59) [MSC
v.1500 32 bit (Intel)] on win32

Any feedback will be greatly appreciated as I seem to keep ending up with
memory errors when working with netcdf files this even happens if I am using
a much better computer.

Old error (before adding gc.collect() )
*Traceback (most recent call last):
  File "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py",
line 44, in <module>
    TSFC=ncfile.variables['T_SFC'][1::3]
  File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
(netCDF4.c:23094)
MemoryError*



from netCDF4 import Dataset
import numpy as N
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
from netcdftime import utime
from datetime import datetime
import os
import gc


shapefile1="E:/griddeddatasamples/GIS/DSE_REGIONS"
MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
'r+', 'NETCDF4')
LAT=fileforlatlon.variables['latitude'][:]
LON=fileforlatlon.variables['longitude'][:]

for (path, dirs, files) in os.walk(MainFolder):
    for dir in dirs:
        print dir
    path=path+'/'
    for ncfile in files:
        if ncfile[-3:]=='.nc':
            print "dealing with ncfiles:", ncfile
            ncfile=os.path.join(path,ncfile)
            ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
            TSFC=ncfile.variables['T_SFC'][1::3]
            TIME=ncfile.variables['time'][1::3]
            ncfile.close()
            gc.collect()

            for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
                cdftime=utime('seconds since 1970-01-01 00:00:00')
                ncfiletime=cdftime.num2date(TIME)
                print ncfiletime
                timestr=str(ncfiletime)
                d = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S')
                date_string = d.strftime('%Y%m%d_%H%M')

                map = Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,

llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
                x,y=map(*N.meshgrid(LON,LAT))
                map.drawcoastlines(linewidth=0.5)
                map.readshapefile(shapefile1, 'DSE_REGIONS')
                map.drawstates()

                plt.title('Surface temperature at %s UTC'%ncfiletime)
                ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
                CS = map.contourf(x,y,TSFC, ticks, cmap=plt.cm.jet)
                l,b,w,h =0.1,0.1,0.8,0.8
                cax = plt.axes([l+w+0.025, b, 0.025, h], )
                cbar=plt.colorbar(CS, cax=cax, drawedges=True)

                plt.savefig((os.path.join(OutputFolder,
'TSFC'+date_string+'UTC.png')))
                plt.close()
                gc.collect()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20111004/67b4b278/attachment.html>