[Numpy-discussion] The mu.py script will keep running and never end.

Andrea Gavana andrea.gavana at gmail.com
Mon Oct 12 10:41:08 EDT 2020 

Hi,

On Mon, 12 Oct 2020 at 16.22, Hongyi Zhao <hongyi.zhao at gmail.com> wrote:

> On Mon, Oct 12, 2020 at 9:33 PM Andrea Gavana <andrea.gavana at gmail.com>
> wrote:
> >
> > Hi,
> >
> > On Mon, 12 Oct 2020 at 14:38, Hongyi Zhao <hongyi.zhao at gmail.com> wrote:
> >>
> >> On Sun, Oct 11, 2020 at 3:42 PM Evgeni Burovski
> >> <evgeny.burovskiy at gmail.com> wrote:
> >> >
> >> > On Sun, Oct 11, 2020 at 9:55 AM Evgeni Burovski
> >> > <evgeny.burovskiy at gmail.com> wrote:
> >> > >
> >> > > The script seems to be computing the particle numbers for an array
> of chemical potentials.
> >> > >
> >> > > Two ways of speeding it up, both are likely simpler then using dask:
> >> > >
> >> > > First: use numpy
> >> > >
> >> > > 1. Move constructing mu_all out of the loop (np.linspace)
> >> > > 2. Arrange the integrands into a 2d array
> >> > > 3. np.trapz along an axis which corresponds to a single integrand
> array
> >> > > (Or avoid the overhead of trapz by just implementing the trapezoid
> formula manually)
> >> >
> >> >
> >> > Roughly like this:
> >> > https://gist.github.com/ev-br/0250e4eee461670cf489515ee427eb99
> >>
> >> I've done the comparison of the real execution time for your version
> >> I've compared the execution efficiency of your above method and the
> >> original method of the python script by directly using fermi() without
> >> executing vectorize() on it. Very surprisingly, the latter is more
> >> efficient than the former, see following for more info:
> >>
> >> $ time python fermi_integrate_np.py
> >> [[1.03000000e+01 4.55561775e+17]
> >>  [1.03001000e+01 4.55561780e+17]
> >>  [1.03002000e+01 4.55561786e+17]
> >>  ...
> >>  [1.08997000e+01 1.33654085e+21]
> >>  [1.08998000e+01 1.33818034e+21]
> >>  [1.08999000e+01 1.33982054e+21]]
> >>
> >> real    1m8.797s
> >> user    0m47.204s
> >> sys    0m27.105s
> >> $ time python mu.py
> >> [[1.03000000e+01 4.55561775e+17]
> >>  [1.03001000e+01 4.55561780e+17]
> >>  [1.03002000e+01 4.55561786e+17]
> >>  ...
> >>  [1.08997000e+01 1.33654085e+21]
> >>  [1.08998000e+01 1.33818034e+21]
> >>  [1.08999000e+01 1.33982054e+21]]
> >>
> >> real    0m38.829s
> >> user    0m41.541s
> >> sys    0m3.399s
> >>
> >> So, I think that the benchmark dataset used by you for testing code
> >> efficiency is not so appropriate. What's your point of view on this
> >> testing results?
> >
> >
> >
> >   Evgeni has provided an interesting example on how to speed up your
> code - granted, he used toy data but the improvement is real. As far as I
> can see, you haven't specified how big are your DOS etc... vectors, so it's
> not that obvious how to draw any conclusions. I find it highly puzzling
> that his implementation appears to be slower than your original code.
> >
> > In any case, if performance is so paramount for you, then I would
> suggest you to move in the direction Evgeni was proposing, i.e. shifting
> your implementation to C/Cython or Fortran/f2py.
>
> If so, I think that the C/Fortran based implementations should be more
> efficient than the ones using Cython/f2py.


That is not what I meant: what I meant is: write the time consuming part of
your code in C or Fortran and then bridge it to Python using Cython or
f2py.

Andrea.


>
>
> > I had much better results myself using Fortran/f2py than pure NumPy or
> C/Cython, but this is mostly because my knowledge of Cython is quite
> limited. That said, your problem should be fairly easy to implement in a
> compiled language.
> >
> > Andrea.
> >
> >
> >>
> >>
> >> Regards,
> >> HY
> >>
> >> >
> >> >
> >> >
> >> > > Second:
> >> > >
> >> > > Move the loop into cython.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > вс, 11 окт. 2020 г., 9:32 Hongyi Zhao <hongyi.zhao at gmail.com>:
> >> > >>
> >> > >> On Sun, Oct 11, 2020 at 2:02 PM Andrea Gavana <
> andrea.gavana at gmail.com> wrote:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Sun, 11 Oct 2020 at 07.52, Hongyi Zhao <hongyi.zhao at gmail.com>
> wrote:
> >> > >> >>
> >> > >> >> On Sun, Oct 11, 2020 at 1:33 PM Andrea Gavana <
> andrea.gavana at gmail.com> wrote:
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On Sun, 11 Oct 2020 at 07.14, Andrea Gavana <
> andrea.gavana at gmail.com> wrote:
> >> > >> >> >>
> >> > >> >> >> Hi,
> >> > >> >> >>
> >> > >> >> >> On Sun, 11 Oct 2020 at 00.27, Hongyi Zhao <
> hongyi.zhao at gmail.com> wrote:
> >> > >> >> >>>
> >> > >> >> >>> On Sun, Oct 11, 2020 at 1:48 AM Robert Kern <
> robert.kern at gmail.com> wrote:
> >> > >> >> >>> >
> >> > >> >> >>> > You don't need to use vectorize() on fermi(). fermi()
> will work just fine on arrays and should be much faster.
> >> > >> >> >>>
> >> > >> >> >>> Yes, it really does the trick. See the following for the
> benchmark
> >> > >> >> >>> based on your suggestion:
> >> > >> >> >>>
> >> > >> >> >>> $ time python mu.py
> >> > >> >> >>> [-10.999 -10.999 -10.999 ...  20.     20.     20.   ]
> [4.973e-84
> >> > >> >> >>> 4.973e-84 4.973e-84 ... 4.973e-84 4.973e-84 4.973e-84]
> >> > >> >> >>>
> >> > >> >> >>> real    0m41.056s
> >> > >> >> >>> user    0m43.970s
> >> > >> >> >>> sys    0m3.813s
> >> > >> >> >>>
> >> > >> >> >>>
> >> > >> >> >>> But are there any ways to further improve/increase
> efficiency?
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >> I believe it will get a bit better if you don’t column_stack
> an array 6000 times - maybe pre-allocate your output first?
> >> > >> >> >>
> >> > >> >> >> Andrea.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > I’m sorry, scratch that: I’ve seen a ghost white space in
> front of your column_stack call and made me think you were stacking your
> results very many times, which is not the case.
> >> > >> >>
> >> > >> >> Still not so clear on your solutions for this problem. Could you
> >> > >> >> please post here the corresponding snippet of your enhancement?
> >> > >> >
> >> > >> >
> >> > >> > I have no solution, I originally thought you were calling
> “column_stack” 6000 times in the loop, but that is not the case, I was
> mistaken. My apologies for that.
> >> > >> >
> >> > >> > The timings of your approach is highly dependent on the size of
> your “energy” and “DOS” array -
> >> > >>
> >> > >> The size of the “energy” and “DOS” array is Problem-related and
> >> > >> shouldn't be reduced arbitrarily.
> >> > >>
> >> > >> > not to mention calling trapz 6000 times in a loop.
> >> > >>
> >> > >> I'm currently thinking on parallelization the execution of the for
> >> > >> loop, say, with joblib <https://github.com/joblib/joblib>, but I
> still
> >> > >> haven't figured out the corresponding codes. If you have some
> >> > >> experience on this type of solution, could you please give me some
> >> > >> more hints?
> >> > >>
> >> > >> >  Maybe there’s a better way to do it with another approach, but
> at the moment I can’t think of one...
> >> > >> >
> >> > >> >>
> >> > >> >>
> >> > >> >> Regards,
> >> > >> >> HY
> >> > >> >> >
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >>>
> >> > >> >> >>>
> >> > >> >> >>> Regards,
> >> > >> >> >>> HY
> >> > >> >> >>>
> >> > >> >> >>> >
> >> > >> >> >>> > On Sat, Oct 10, 2020, 8:23 AM Hongyi Zhao <
> hongyi.zhao at gmail.com> wrote:
> >> > >> >> >>> >>
> >> > >> >> >>> >> Hi,
> >> > >> >> >>> >>
> >> > >> >> >>> >> My environment is Ubuntu 20.04 and python 3.8.3 managed
> by pyenv. I
> >> > >> >> >>> >> try to run the script
> >> > >> >> >>> >> <
> https://notebook.rcc.uchicago.edu/files/acs.chemmater.9b05047/Data/bulk/dft/mu.py
> >,
> >> > >> >> >>> >> but it will keep running and never end. When I use 'Ctrl
> + c' to
> >> > >> >> >>> >> terminate it, it will give the following output:
> >> > >> >> >>> >>
> >> > >> >> >>> >> $ python mu.py
> >> > >> >> >>> >> [-10.999 -10.999 -10.999 ...  20.     20.     20.   ]
> [4.973e-84
> >> > >> >> >>> >> 4.973e-84 4.973e-84 ... 4.973e-84 4.973e-84 4.973e-84]
> >> > >> >> >>> >>
> >> > >> >> >>> >> I have to terminate it and obtained the following
> information:
> >> > >> >> >>> >>
> >> > >> >> >>> >> ^CTraceback (most recent call last):
> >> > >> >> >>> >>   File "mu.py", line 38, in <module>
> >> > >> >> >>> >>     integrand=DOS*fermi_array(energy,mu,kT)
> >> > >> >> >>> >>   File
> "/home/werner/.pyenv/versions/datasci/lib/python3.8/site-packages/numpy/lib/function_base.py",
> >> > >> >> >>> >> line 2108, in __call__
> >> > >> >> >>> >>     return self._vectorize_call(func=func, args=vargs)
> >> > >> >> >>> >>   File
> "/home/werner/.pyenv/versions/datasci/lib/python3.8/site-packages/numpy/lib/function_base.py",
> >> > >> >> >>> >> line 2192, in _vectorize_call
> >> > >> >> >>> >>     outputs = ufunc(*inputs)
> >> > >> >> >>> >>   File "mu.py", line 8, in fermi
> >> > >> >> >>> >>     return 1./(exp((E-mu)/kT)+1)
> >> > >> >> >>> >> KeyboardInterrupt
> >> > >> >> >>> >>
> >> > >> >> >>> >>
> >> > >> >> >>> >> Any helps and hints for this problem will be highly
> appreciated?
> >> > >> >> >>> >>
> >> > >> >> >>> >> Regards,
> >> > >> >> >>> >> --
> >> > >> >> >>> >> Hongyi Zhao <hongyi.zhao at gmail.com>
> >> > >> >> >>> >> _______________________________________________
> >> > >> >> >>> >> NumPy-Discussion mailing list
> >> > >> >> >>> >> NumPy-Discussion at python.org
> >> > >> >> >>> >>
> https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > >> >> >>> >
> >> > >> >> >>> > _______________________________________________
> >> > >> >> >>> > NumPy-Discussion mailing list
> >> > >> >> >>> > NumPy-Discussion at python.org
> >> > >> >> >>> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > >> >> >>>
> >> > >> >> >>>
> >> > >> >> >>>
> >> > >> >> >>> --
> >> > >> >> >>> Hongyi Zhao <hongyi.zhao at gmail.com>
> >> > >> >> >>> _______________________________________________
> >> > >> >> >>> NumPy-Discussion mailing list
> >> > >> >> >>> NumPy-Discussion at python.org
> >> > >> >> >>> https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > >> >> >
> >> > >> >> > _______________________________________________
> >> > >> >> > NumPy-Discussion mailing list
> >> > >> >> > NumPy-Discussion at python.org
> >> > >> >> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >> --
> >> > >> >> Hongyi Zhao <hongyi.zhao at gmail.com>
> >> > >> >> _______________________________________________
> >> > >> >> NumPy-Discussion mailing list
> >> > >> >> NumPy-Discussion at python.org
> >> > >> >> https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > >> >
> >> > >> > _______________________________________________
> >> > >> > NumPy-Discussion mailing list
> >> > >> > NumPy-Discussion at python.org
> >> > >> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Hongyi Zhao <hongyi.zhao at gmail.com>
> >> > >> _______________________________________________
> >> > >> NumPy-Discussion mailing list
> >> > >> NumPy-Discussion at python.org
> >> > >> https://mail.python.org/mailman/listinfo/numpy-discussion
> >> > _______________________________________________
> >> > NumPy-Discussion mailing list
> >> > NumPy-Discussion at python.org
> >> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >>
> >>
> >>
> >> --
> >> Hongyi Zhao <hongyi.zhao at gmail.com>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at python.org
> >> https://mail.python.org/mailman/listinfo/numpy-discussion
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
>
> --
> Hongyi Zhao <hongyi.zhao at gmail.com>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20201012/69184291/attachment-0001.html>

More information about the NumPy-Discussion mailing list