[SciPy-Dev] Adding logsoftmax function to scipy.special

Fri Dec 6 23:48:27 EST 2019

On Fri, Dec 6, 2019 at 8:43 PM Dominik Stańczak <stanczakdominik at gmail.com>
wrote:

> Hey,
>
> your function may not show up if there's an __all__ variable in that
> module that lists importable functions and you didn't add your function to
> it. Sphinx goes by that one if it's available.
>
> Not sure that's relevant, but it's likely to be.
>
> Cheers,
> Dominik
>
> On Sat, Dec 7, 2019, 05:34 Takuya Koumura <koumura at cycentum.com> wrote:
>
>> Hi,
>>
>> I implemented log_softmax and I wrote a docstring for it. Can anyone tell
>> me how to add the function name to the scipy.special index page (
>> https://docs.scipy.org/doc/scipy/reference/special.html)?  Is it Okay to
>> directly edit the docstring in special/__init__.py?
>>
>
Yes, you need to add your function to __init__.py, right under `softmax`
looks like the correct place.

And indeed, as Dominic suggests, you also need to add the function name to
`__all__` in the file where you added it - `__all__` is a list of all
public names in each file.

> I wonder “.. autosummary::  :toctree: generated/” indicates it was
>> generated by Sphinx?
>>
>
No, that is a directive to tell Sphinx it should take the functions listed
below it, and generated links and doc pages for those in the html/pdf
output.

Cheers,
Ralf

>
>> Best regards,
>> Takuya
>>
>>
>> 2019年11月18日(月) 17:38 Takuya Koumura <koumura at cycentum.com>:
>>
>>> Hi Ralf, Josh, and all,
>>>
>>> Thank you for your positive comments and raising potential issues.
>>>
>>> Having read Josh’s comment, I conducted several experiments and found
>>> there indeed is a case where cancellation occurs in my original
>>> implementation, in which log_softmax(x) = x - logsumexp(x) = x - (m +
>>> log(sum(exp(x - m)))) where m = max(x).
>>>
>>> Calculating s = x - m first and using it to calculate log_softmax(x) = s
>>> - logsumexp(s) should decrease the chance of cancellation because it does
>>> not involve subtraction other than x - m. Note that s is always <=0, so s -
>>> log(sum(exp(s))) is actually addition of negative values.
>>> Also, reusing s may be a bit more efficient in computational time.
>>>
>>> Best,
>>> Takuya
>>>
>>>
>>> 2019年11月18日(月) 3:51 Joshua Wilson <josh.craig.wilson at gmail.com>:
>>>
>>>> > I would like to implement logsoftmax(x) as x-logsumexp(x)
>>>>
>>>> The function seems reasonable to add, but I am not so sure about that
>>>> implementation, as it appears that it could lead to cancellation.
>>>> Note, for example, that because of the scaling `logsumexp` does to
>>>> prevent overflow, with that implementation you will end up adding and
>>>> subtracting the component `x_i` of the largest magnitude when
>>>> computing the `i`th component of the results array, see e.g.
>>>>
>>>>
>>>> https://github.com/scipy/scipy/blob/master/scipy/special/_logsumexp.py#L124
>>>>
>>>> Off the top of my head, it is not obvious to me that situations
>>>> similar to that will not lead to cancellation errors.
>>>>
>>>> Of course, I might be wrong, but my overall point is that in order to
>>>> use a particular implementation we should have a reasonable
>>>> understanding of its theoretical properties, so I would like to see
>>>> some exploration of that before proceeding.
>>>>
>>>> - Josh
>>>>
>>>> On Fri, Nov 15, 2019 at 5:17 PM Ralf Gommers <ralf.gommers at gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi Takuya,
>>>> >
>>>> >
>>>> > On Tue, Nov 12, 2019 at 9:05 PM Takuya Koumura <koumura at cycentum.com>
>>>> wrote:
>>>> >>
>>>> >> Hello,
>>>> >>
>>>> >> I raised a GitHub issue (#11058) and was suggested to post it to
>>>> scipy-dev.
>>>> >>
>>>> >> I’m considering to send a PR to add logsoftmax function in
>>>> scipy.special. Before that, I would like to hear your opinion (partly
>>>> because it’s my first time to send a PR to scipy).
>>>> >
>>>> >
>>>> > Welcome! Thanks for proposing that. logsoftmax is fairly popular at
>>>> least in deep learning, so it makes sense I think, and we already have a
>>>> bunch of other log* functions in scipy.special.
>>>> >
>>>> > I noticed that both PyTorch and Tensorflow name this function
>>>> `log_softmax` rather than `logsoftmax`. The latter would be a little more
>>>> consistent with other functions (although we also have `special.log_ndtr`),
>>>> while the former is consistent with other implementations of the same
>>>> functionality. I'd be okay with either, with a slight preference for
>>>> `log_softmax`.
>>>> >
>>>> >>
>>>> >> I would like to implement logsoftmax(x) as x-logsumexp(x). Actually,
>>>> special.softmax(x) = np.exp(x-logsumexp(x)), so it is trivial for those who
>>>> read the source code of softmax, but I think including logsoftmax as a
>>>> separate function will be useful for other users. Logsoftmax is more
>>>> accurate with inputs that make softmax saturate, eg: When x=[1000, 0],
>>>> np.log(softmax(x))=[0, -Inf] (maybe depending on the floating point
>>>> precision), while logsoftmax(x)=[0, -1000].
>>>> >>
>>>> >> I am planning to add the new function at the bottom of
>>>> special/_logsumexp.py following the softmax function, and add some unit
>>>> tests in special/test/test_logsumexp.py. If you have comments, I’d
>>>> appreciate any.
>>>> >
>>>> >
>>>> > That seems like a good place.
>>>> >
>>>> > Cheers,
>>>> > Ralf
>>>> >
>>>> >>
>>>> >> Best wishes,
>>>> >> --
>>>> >> Takuya KOUMURA
>>>> >> koumura at cycentum.com
>>>> >> _______________________________________________
>>>> >> SciPy-Dev mailing list
>>>> >> SciPy-Dev at python.org
>>>> >> https://mail.python.org/mailman/listinfo/scipy-dev
>>>> >
>>>> > _______________________________________________
>>>> > SciPy-Dev mailing list
>>>> > SciPy-Dev at python.org
>>>> > https://mail.python.org/mailman/listinfo/scipy-dev
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>>
>>> --
>>> --
>>> Takuya KOUMURA
>>> koumura at cycentum.com
>>>
>>
>>
>> --
>> --
>> Takuya KOUMURA
>> koumura at cycentum.com
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20191206/3efb878e/attachment.html>