[Numpy-discussion] NumPy-Discussion Digest, Vol 87, Issue 35
Amira Chekir
chekir.amira at gmail.com
Wed Jan 1 10:45:09 EST 2014
Hi,
Thanks for your answer.
I use ubuntu 12.04 32 bits and python 2.7
I upgrade numpy to 1.8, but the error persists
I think that the problem is in gzip.py :
max_read_chunk = 10 * 1024 * 1024 # 10Mb
What do you think?
Best regards,
AMIRA
2013/12/31 <numpy-discussion-request at scipy.org>
> Send NumPy-Discussion mailing list submissions to
> numpy-discussion at scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> or, via email, send a message with subject or body 'help' to
> numpy-discussion-request at scipy.org
>
> You can reach the person managing the list at
> numpy-discussion-owner at scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of NumPy-Discussion digest..."
>
>
> Today's Topics:
>
> 1. Loading large NIfTI file -> MemoryError (Amira Chekir)
> 2. Re: Loading large NIfTI file -> MemoryError (Julian Taylor)
> 3. Re: proposal: min, max of complex should give warning (Cera,
> Tim)
> 4. Re: proposal: min, max of complex should give warning
> (Neal Becker)
> 5. Re: proposal: min, max of complex should give warning
> (Ralf Gommers)
> 6. Re: proposal: min, max of complex should give warning
> (Neal Becker)
> 7. ANN: NumPy 1.7.2 release (Julian Taylor)
> 8. Re: ANN: NumPy 1.7.2 release (Charles R Harris)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 31 Dec 2013 14:13:57 +0100
> From: Amira Chekir <chekir.amira at gmail.com>
> Subject: [Numpy-discussion] Loading large NIfTI file -> MemoryError
> To: numpy-discussion at scipy.org
> Message-ID:
> <CAB-foYgW4HiL37aPchrbYkgPNJLY1Dt55M8Ki30UaTg=
> EQ29Zw at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hello together,
>
> I try to load a (large) NIfTI file (DMRI from Human Connectome Project,
> about 1 GB) with NiBabel.
>
> import nibabel as nib
> img = nib.load("dmri.nii.gz")
> data = img.get_data()
>
> The program crashes during "img.get_data()" with an "MemoryError" (having 4
> GB of RAM in my machine).
>
> Any suggestions?
>
> Best regards,
> AMIRA
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/b13969b3/attachment-0001.html
>
> ------------------------------
>
> Message: 2
> Date: Tue, 31 Dec 2013 14:29:42 +0100
> From: Julian Taylor <jtaylor.debian at googlemail.com>
> Subject: Re: [Numpy-discussion] Loading large NIfTI file ->
> MemoryError
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID: <52C2C6C6.6070002 at googlemail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 31.12.2013 14:13, Amira Chekir wrote:
> > Hello together,
> >
> > I try to load a (large) NIfTI file (DMRI from Human Connectome Project,
> > about 1 GB) with NiBabel.
> >
> > import nibabel as nib
> > img = nib.load("dmri.nii.gz")
> > data = img.get_data()
> >
> > The program crashes during "img.get_data()" with an "MemoryError"
> > (having 4 GB of RAM in my machine).
> >
> > Any suggestions?
>
> are you using a 64 bit operating system?
> which version of numpy?
>
> assuming nibabel uses np.load under the hood you could try it with numpy
> 1.8 which reduces excess memory usage when loading compressed files.
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 31 Dec 2013 08:51:52 -0500
> From: "Cera, Tim" <tim at cerazone.net>
> Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> give warning
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> <
> CAO5s+D_m5N6SJgsKoV7O-+yHh5gPnB0_a-ozKgETGRwTgN_axg at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> I don't work with complex numbers, but just sampling what others do:
>
>
> Python: no ordering, results in TypeError
>
> Matlab: sorts by magnitude
> http://www.mathworks.com/help/matlab/ref/sort.html
>
> R: sorts first by real, then by imaginary
> http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
>
> Numpy: sorts first by real, then by imaginary (the documentation link
> below calls this sort 'lexicographical' which I don't think is
> correct)
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
>
>
> I would think that the Matlab sort might be more useful, but easy
> enough by using the absolute value.
>
> I think what Numpy does is normal enough to not justify a warning, but
> leave this to others because as I pointed out in the beginning I don't
> work with complex numbers.
>
> Kindest regards,
> Tim
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 31 Dec 2013 10:52:47 -0500
> From: Neal Becker <ndbecker2 at gmail.com>
> Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> give warning
> To: numpy-discussion at scipy.org
> Message-ID: <l9up83$549$1 at ger.gmane.org>
> Content-Type: text/plain; charset="ISO-8859-1"
>
> Cera, Tim wrote:
>
> > I don't work with complex numbers, but just sampling what others do:
> >
> >
> > Python: no ordering, results in TypeError
> >
> > Matlab: sorts by magnitude
> > http://www.mathworks.com/help/matlab/ref/sort.html
> >
> > R: sorts first by real, then by imaginary
> > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> >
> > Numpy: sorts first by real, then by imaginary (the documentation link
> > below calls this sort 'lexicographical' which I don't think is
> > correct)
> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> >
> >
> > I would think that the Matlab sort might be more useful, but easy
> > enough by using the absolute value.
> >
> > I think what Numpy does is normal enough to not justify a warning, but
> > leave this to others because as I pointed out in the beginning I don't
> > work with complex numbers.
> >
> > Kindest regards,
> > Tim
>
> But I'm not proposing to change numpy's result, which I'm sure would raise
> many
> objections. I'm just asking to give a warning, because I think in most
> cases
> this is actually a mistake on the user's part. Just like the warning
> currently
> given when complex data are truncated to real part.
>
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 31 Dec 2013 17:24:05 +0100
> From: Ralf Gommers <ralf.gommers at gmail.com>
> Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> give warning
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> <
> CABL7CQh9Fc0Uh36W9p16mzAR-oYjJ7_k7rU_Dwq+eZND6YrbDA at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Tue, Dec 31, 2013 at 4:52 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>
> > Cera, Tim wrote:
> >
> > > I don't work with complex numbers, but just sampling what others do:
> > >
> > >
> > > Python: no ordering, results in TypeError
> > >
> > > Matlab: sorts by magnitude
> > > http://www.mathworks.com/help/matlab/ref/sort.html
> > >
> > > R: sorts first by real, then by imaginary
> > > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> > >
> > > Numpy: sorts first by real, then by imaginary (the documentation link
> > > below calls this sort 'lexicographical' which I don't think is
> > > correct)
> > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> > >
> > >
> > > I would think that the Matlab sort might be more useful, but easy
> > > enough by using the absolute value.
> > >
> > > I think what Numpy does is normal enough to not justify a warning, but
> > > leave this to others because as I pointed out in the beginning I don't
> > > work with complex numbers.
> > >
> > > Kindest regards,
> > > Tim
> >
> > But I'm not proposing to change numpy's result, which I'm sure would
> raise
> > many
> > objections. I'm just asking to give a warning, because I think in most
> > cases
> > this is actually a mistake on the user's part. Just like the warning
> > currently
> > given when complex data are truncated to real part.
> >
>
> Keep in mind that warnings can be highly annoying. If you're a user who
> uses this functionality regularly (and you know what you're doing), then
> you're going to be very unhappy to have to wrap each function call in:
> olderr = np.seterr(all='ignore')
> max(...)
> np.seterr(**olderr)
> or in:
> with warnings.catch_warnings():
> warnings.filterwarnings('ignore', ...)
> max(...)
>
> The actual behavior isn't documented now it looks like, so that should be
> done. In the Notes section of max/min probably.
>
> As for your proposal, it would be good to know if adding a warning would
> actually catch any bugs. For the truncation warning it caught several in
> scipy and other libs IIRC.
>
> Ralf
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/add729d8/attachment-0001.html
>
> ------------------------------
>
> Message: 6
> Date: Tue, 31 Dec 2013 11:45:08 -0500
> From: Neal Becker <ndbecker2 at gmail.com>
> Subject: Re: [Numpy-discussion] proposal: min, max of complex should
> give warning
> To: numpy-discussion at scipy.org
> Message-ID: <l9usa8$7tr$1 at ger.gmane.org>
> Content-Type: text/plain; charset="ISO-8859-1"
>
> Ralf Gommers wrote:
>
> > On Tue, Dec 31, 2013 at 4:52 PM, Neal Becker <ndbecker2 at gmail.com>
> wrote:
> >
> >> Cera, Tim wrote:
> >>
> >> > I don't work with complex numbers, but just sampling what others do:
> >> >
> >> >
> >> > Python: no ordering, results in TypeError
> >> >
> >> > Matlab: sorts by magnitude
> >> > http://www.mathworks.com/help/matlab/ref/sort.html
> >> >
> >> > R: sorts first by real, then by imaginary
> >> > http://stat.ethz.ch/R-manual/R-patched/library/base/html/sort.html
> >> >
> >> > Numpy: sorts first by real, then by imaginary (the documentation link
> >> > below calls this sort 'lexicographical' which I don't think is
> >> > correct)
> >> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html
> >> >
> >> >
> >> > I would think that the Matlab sort might be more useful, but easy
> >> > enough by using the absolute value.
> >> >
> >> > I think what Numpy does is normal enough to not justify a warning, but
> >> > leave this to others because as I pointed out in the beginning I don't
> >> > work with complex numbers.
> >> >
> >> > Kindest regards,
> >> > Tim
> >>
> >> But I'm not proposing to change numpy's result, which I'm sure would
> raise
> >> many
> >> objections. I'm just asking to give a warning, because I think in most
> >> cases
> >> this is actually a mistake on the user's part. Just like the warning
> >> currently
> >> given when complex data are truncated to real part.
> >>
> >
> > Keep in mind that warnings can be highly annoying. If you're a user who
> > uses this functionality regularly (and you know what you're doing), then
> > you're going to be very unhappy to have to wrap each function call in:
> > olderr = np.seterr(all='ignore')
> > max(...)
> > np.seterr(**olderr)
> > or in:
> > with warnings.catch_warnings():
> > warnings.filterwarnings('ignore', ...)
> > max(...)
> >
> > The actual behavior isn't documented now it looks like, so that should be
> > done. In the Notes section of max/min probably.
> >
> > As for your proposal, it would be good to know if adding a warning would
> > actually catch any bugs. For the truncation warning it caught several in
> > scipy and other libs IIRC.
> >
> > Ralf
>
> I tripped over it yesterday, which is what prompted my suggestion.
>
>
>
> ------------------------------
>
> Message: 7
> Date: Tue, 31 Dec 2013 17:57:18 +0100
> From: Julian Taylor <jtaylor.debian at googlemail.com>
> Subject: [Numpy-discussion] ANN: NumPy 1.7.2 release
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>,
> SciPy Users List <scipy-user at scipy.org>, SciPy Developers
> List
> <scipy-dev at scipy.org>
> Message-ID: <52C2F76E.9010509 at googlemail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hello,
>
> I'm happy to announce the of Numpy 1.7.2.
> This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3.
>
> More than 42 issues were fixed, the most important issues are listed in
> the release notes:
> https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst
>
> Compared to the last release candidate four additional minor issues have
> been fixed and compatibility with python 3.4b1 improved.
>
> Source tarballs, installers and release notes can be found at
> https://sourceforge.net/projects/numpy/files/NumPy/1.7.2
>
> Cheers,
> Julian Taylor
>
>
> ------------------------------
>
> Message: 8
> Date: Tue, 31 Dec 2013 10:47:44 -0700
> From: Charles R Harris <charlesr.harris at gmail.com>
> Subject: Re: [Numpy-discussion] ANN: NumPy 1.7.2 release
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> <CAB6mnxJBWn+FM25=
> abrqm4DNRG7f6-1keU_hPd253O64d0-Yhw at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Tue, Dec 31, 2013 at 9:57 AM, Julian Taylor <
> jtaylor.debian at googlemail.com> wrote:
>
> > Hello,
> >
> > I'm happy to announce the of Numpy 1.7.2.
> > This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3.
> >
> > More than 42 issues were fixed, the most important issues are listed in
> > the release notes:
> > https://github.com/numpy/numpy/blob/v1.7.2/doc/release/1.7.2-notes.rst
> >
> > Compared to the last release candidate four additional minor issues have
> > been fixed and compatibility with python 3.4b1 improved.
> >
> > Source tarballs, installers and release notes can be found at
> > https://sourceforge.net/projects/numpy/files/NumPy/1.7.2
> >
> >
> Congrats on the release.
>
> Chuck
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/numpy-discussion/attachments/20131231/946abcb9/attachment.html
>
> ------------------------------
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> End of NumPy-Discussion Digest, Vol 87, Issue 35
> ************************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140101/279def51/attachment.html>
More information about the NumPy-Discussion
mailing list