[SciPy-user] histogramdd question

[David Huard]

2007/8/16, Chris Lee <c.j.lee at tnw.utwente.nl>:
>
> Thanks David,
>
> That makes sense. How difficult would it be to put in an optional
> parameter to only use integers?


Not difficult at all, in fact, but I'm guessing there are not many people
around with the kind of needs you have ! I can't even begin to understand
how a computer can handle a 700G object. In any case, my suggestion would be
for you to copy and paste the histogramdd fonction from numpy and tweak it
to you heart's content. Better yet for such large arrays would be for you to
rewrite the function in fortran and wrap it using f2py. I'd also advise
using a "bloc" approach, where you feed the histogram function chunks of the
array instead of the whole thing. This avoids swapping memory to the hard
drive when you exceed you RAM capacity. If you want to give it a try, I'll
send you some drafts to get you started.

Cheers,
David

In some sense this doesn't matter too much since I can (in pathological
> cases) end up with a 700 GB histogram but normally It clocks in between
> 0.8-2 GB. In the end type casting isn't going to stop me from grabbing
> all the memory I can.
>
>
>
> David Huard wrote:
> > Hi Chris,
> >
> > histogramdd will consist of floats when it is normalized or when
> > non-integer weights are given. I thought it was preferable to return
> > floats in all cases so the output is consistent no matter what. It
> > could be changed if there is a compelling argument, but I'd rather not
> > risk the chance of breaking someone's code.
> >
> > Cheers,
> >
> > David
> >
> > 2007/8/15, Chris Lee <c.j.lee at tnw.utwente.nl
> > <mailto:c.j.lee at tnw.utwente.nl>>:
> >
> >     Hi All,
> >
> >     I have been using histogramdd to generate histograms of a 4-D data
> >     set.
> >     A histogram should, I believe, return integers but histogramdd
> returns
> >     doubles. Is there are reason for this?
> >
> >     Cheers
> >     Chris
> >
> >     --
> >     **********************************************
> >     *  Chris Lee                                 *
> >     *  Laser physics and nonlinear optics group  *
> >     *  MESA+ Institute                           *
> >     *  University of Twente                      *
> >     *  Phone: ++31 (0)53 489 3968                *
> >     *  fax: ++31 (0) 53 489 1102                 *
> >     **********************************************
> >
> >     _______________________________________________
> >     SciPy-user mailing list
> >     SciPy-user at scipy.org <mailto:SciPy-user at scipy.org>
> >     http://projects.scipy.org/mailman/listinfo/scipy-user
> >
> >
>
> --
> **********************************************
> *  Chris Lee                                 *
> *  Laser physics and nonlinear optics group  *
> *  MESA+ Institute                           *
> *  University of Twente                      *
> *  Phone: ++31 (0)53 489 3968                *
> *  fax: ++31 (0) 53 489 1102                 *
> **********************************************
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20070816/54512b42/attachment.html>