Program very slow to finish

[Fred]

Good call.  os_exit(0) gives:


$ time python unitest.py < 841123.dat
Total lines read: 316480
B: 158229 Value: 20191822207L Max/Min: 9101400 12
S: 158251 Value: 20193690457L Max/Min: 9101400 12
Number of acc: 56568
Number of tra: 64704
Number of stk: 388

real    0m24.68s
user    0m23.21s
sys     0m0.48s


Emile van Sebille wrote:

> "Fred" <fredNo at nospamco.com> wrote in message
> news:3BE41CBE.94AED22 at nospamco.com...
> > I'm dealing with about 100Gb of data that I first just need to
> > characterize.  So, since the slowest part will be simply reading the
> > data, I'm testing various languages and methods on a 10 line subset, a
> > 32 Mb subset and a 750 Mb subset.
> >
> > The following python program prints out the results in about 30 seconds,
> > however it doesn't finish for another minute with the 32 Mb set of
> > data!  At first I thought it was stuck and killed it; however I finally
> > let it run on the smaller data set and all was well.
> >
> > Is this a garbage collection issue?  Is there a better way to count the
> > individual values than dictionaries?  I put the sys.exit call in while
> > trying to figure out what was happening but it didn't make a difference.
> >
>
> I'd suspect that python is cleaning house before exiting.  Try exiting with
> os._exit or os.abort.
>
> --
>
> Emile van Sebille
> emile at fenx.com
>
> ---------