[Python-Dev] [numpy wishlist] PyMem_*Calloc

[Victor Stinner]

  Hi,

2014-04-14 1:39 GMT-04:00 Nathaniel Smith <njs at pobox.com>:
> The new tracemalloc infrastructure in python 3.4 is super-interesting
> to numerical folks, because we really like memory profiling.

Cool, thanks :-)

> calloc() is more awesome than malloc()+memset() (...)

I had a discussion with someone about tracemalloc and numpy at Pycon,
was it you? After this discussion, I realized that calloc() exists
because the operating system can have a very efficient implementation
for calloc() (as you described).

> SO, we'd like to route our allocations through PyMem_* in order to let
> tracemalloc "see" them, but because there is no PyMem_*Calloc, doing
> this would force us to give up on the calloc() optimizations.

It would also be useful in Python because in many places, Python uses
memset() to fill memory with zeros.

> The obvious solution is to add a PyMem_*Calloc to the API. Would this
> be possible? Unfortunately it would require adding a new field to the
> PyMemAllocator struct, which would be an ABI/API break; PyMemAllocator
> is exposed directly in the C API and passed by value:
>   https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator

I don't want to change the structure in Python 3.4, but I'm interested
to implement the change in Python 3.5.

Please open an issue and add me to the nosy list.

For Python 3.4, you can maybe add a compilation flag to use Python
allocators but reimplementing calloc() which will be slower as you
explained.

Victor