[Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Raymond Hettinger]

On Apr 6, 2011, at 10:24 AM, Maciej Fijalkowski wrote:
> 
> "I saw no need to complicate the pure python code for this."
> 
> if you complicate the C code for this, then please as well complicate
> python code for this since it's breaking stuff.


Do you really need a PEP for this one extraordinary and weird case?
The code is long since gone (never in 3.x).  If you disagreed with
the closing of the bug report, just re-open it and a patch can go
into a 2.7 point release.  The downside is that it would not be a
pretty piece of python.


> And this:
> 
> "FWIW, the C code is not guaranteed to be exactly the same in terms of
> implementation details, only the published API should be the same.
> And, for this module, a decision was made for the C code to support
> only lists eventhough the pure python version supports any sequence."
> 
> The idea of the PEP is for C code to be guaranteed to be the same as
> Python where it matters to people.


That is a good goal.  Unfortunately, people can choose to rely on
all manner of implementation details (whether in C or pure Python).

If we want a pure python version of map() for example, the straight-forward
way doesn't work very well because "map(chr, 3)" raises a TypeError
right away in C code, but a python version using a generator wouldn't
raise until next() is called.  Would this be considered a detail that
matters to people?  If so, it means that all the pure python equivalents
for itertools would be have to be written as classes, making them hard
to read and making them run slowly on all implementations except for PyPy.

The original of the bug report you mentioned arose because a major
tool relied on the pure python heapq code comparing "not b <= a"
rather than the equivalent "a < b".  So this was an implementation
detail that mattered to someone, but it went *far* beyond any guaranteed
behaviors.

Tracebacks are another area where C code and pure python code
can't be identical.  This may or may not matter to someone.

The example in the PEP focused on which particular exception,
a TypeError or AttributeError, was raised in response to an
oddly constructed Spam() class.  I don't know that that was
forseeable or that there would have been a reasonable way
to eliminate the difference.  It does sound like the difference
mattered to someone though.

C code tends to use direct internal calls such as Py_SIZE(obj)
rather than doing a lookup using obj.__len__().  This is often
a detail that matters to people because it prevents them from
hooking the call to  __len__.   The C code has to take this 
approach in order to protect its internal invariants and not crash.
If the pure python code tried to emulate this, then every call to
len(self) would need to be replaced by self.__internal_len()
where __internal_len is the real length method and __len__
is made equal to it.

In C to Python translations, do we want locks to be required
so that atomicity behaviors are matched?  That property
likely matters to some users.

ISTM that every person who sets out to translate code from
C to Python or vice versa is already trying their best to make them
behave as similarly as possible.  That is always the goal.

However, the PEP seems to be raising the bar by insisting
on the code being functionally identical.  I think we should
make some decisions about what that really means; otherwise,
every piece of code will be in violation of the PEP for someone
choosing to rely on an implementation detail that isn't the same.

In my opinion, a good outcome of this discussion would be
a list of implementation details that we want to guarantee
and ones that we explicitly say that are allowed to vary.

I would also like to see strong guidance on the use of the
concrete C API which can make it impossible for client code
to use subclasses of builtin types (issue 10977).  That is
another area where differences will arise that will matter
to some users.


Raymond


P.S.  It would be great if the PEP were to supply a
complete, real-word example of code that is considered
to be identical.  A pure python version of map() could
serve as a good example, trying to make it model all 
the C behaviors as exactly as possible (argument handling,
choice of exceptions, length estimation and presizing, etc).








-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110406/3c0cf00a/attachment.html>