[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

ping?

2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> 2016-02-04 15:05 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:
>> Sometimes, yes, but we also do allocations for e.g.
>> parsing values in Python argument tuples (e.g. using
>> "es" or "et"):
>>
>> https://docs.python.org/3.6/c-api/arg.html
>>
>> We do document to use PyMem_Free() on those; not sure whether
>> everyone does this though.
>
> It's well documented. If programs start to crash, they must be fixed.
>
> I don't propose to "break the API" for free, but to get a speedup on
> the overall Python.
>
> And I don't think that we can say that it's an API change, since we
> already stated that PyMem_Free() must be used.
>
> If your program has bugs, you can use a debug build of Python 3.5 to
> detect misusage of the API.
>
>
>> The Python test suite doesn't test Python C extensions,
>> so it's not surprising that it passes :-)
>
> What do you mean by "C extensions"? Which modules?
>
> Many modules in the stdlib have "C accelerators" and the PEP 399 now
> *require* to test the C and Python implementations.
>
>
>
>>> Instead of teaching developers that well, in fact, PyObject_Malloc()
>>> is unrelated to object programming, I think that it's simpler to
>>> modify PyMem_Malloc() to reuse pymalloc ;-)
>>
>> Perhaps if you add some guards somewhere :-)
>
> We have runtime checks but only implemented in debug mode for efficiency.
>
> By the way, I proposed once to add an environment variable to allow to
> enable these checks without having to recompile Python.  Since the PEP
> 445, it became easy to implement this. What do you think?
> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable
>
> "This alternative was rejected because a new environment variable
> would make Python initialization even more complex. PEP 432 tries to
> simplify the CPython startup sequence."
>
> The PEP 432 looks stuck, so I don't think that we should block
> enhancements because of this PEP. Anyway, my idea should be easy to
> implement.
>
>
>> Seriously, this may work if C extensions use the APIs
>> consistently, but in order to tell, we'd need to check
>> few.
>
> Can you suggest me names of projects that must be tested?
>
>
>> I guess the main question then is whether pymalloc is good enough
>> for general memory allocation needs; and the answer may well be
>> "yes".
>
> What do you mean by "good enough"? For the runtime performance,
> pymalloc looks to be faster than malloc(). What are your other
> criterias? Memory fragmentation?
>
>
> Victor