Program very slow to finish

[Emile van Sebille]

"Fred" <fredNo at nospamco.com> wrote in message
news:3BE41CBE.94AED22 at nospamco.com...
> I'm dealing with about 100Gb of data that I first just need to
> characterize.  So, since the slowest part will be simply reading the
> data, I'm testing various languages and methods on a 10 line subset, a
> 32 Mb subset and a 750 Mb subset.
>
> The following python program prints out the results in about 30 seconds,
> however it doesn't finish for another minute with the 32 Mb set of
> data!  At first I thought it was stuck and killed it; however I finally
> let it run on the smaller data set and all was well.
>
> Is this a garbage collection issue?  Is there a better way to count the
> individual values than dictionaries?  I put the sys.exit call in while
> trying to figure out what was happening but it didn't make a difference.
>

I'd suspect that python is cleaning house before exiting.  Try exiting with
os._exit or os.abort.

--

Emile van Sebille
emile at fenx.com

---------