[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

2016-02-04 15:05 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
> Sometimes, yes, but we also do allocations for e.g.
> parsing values in Python argument tuples (e.g. using
> "es" or "et"):
>
> https://docs.python.org/3.6/c-api/arg.html
>
> We do document to use PyMem_Free() on those; not sure whether
> everyone does this though.

It's well documented. If programs start to crash, they must be fixed.

I don't propose to "break the API" for free, but to get a speedup on
the overall Python.

And I don't think that we can say that it's an API change, since we
already stated that PyMem_Free() must be used.

If your program has bugs, you can use a debug build of Python 3.5 to
detect misusage of the API.


> The Python test suite doesn't test Python C extensions,
> so it's not surprising that it passes :-)

What do you mean by "C extensions"? Which modules?

Many modules in the stdlib have "C accelerators" and the PEP 399 now
*require* to test the C and Python implementations.



>> Instead of teaching developers that well, in fact, PyObject_Malloc()
>> is unrelated to object programming, I think that it's simpler to
>> modify PyMem_Malloc() to reuse pymalloc ;-)
>
> Perhaps if you add some guards somewhere :-)

We have runtime checks but only implemented in debug mode for efficiency.

By the way, I proposed once to add an environment variable to allow to
enable these checks without having to recompile Python.  Since the PEP
445, it became easy to implement this. What do you think?
https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable

"This alternative was rejected because a new environment variable
would make Python initialization even more complex. PEP 432 tries to
simplify the CPython startup sequence."

The PEP 432 looks stuck, so I don't think that we should block
enhancements because of this PEP. Anyway, my idea should be easy to
implement.


> Seriously, this may work if C extensions use the APIs
> consistently, but in order to tell, we'd need to check
> few.

Can you suggest me names of projects that must be tested?


> I guess the main question then is whether pymalloc is good enough
> for general memory allocation needs; and the answer may well be
> "yes".

What do you mean by "good enough"? For the runtime performance,
pymalloc looks to be faster than malloc(). What are your other
criterias? Memory fragmentation?


Victor