[Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.

[Robert Bradshaw]

On Tue, Apr 12, 2011 at 8:33 AM, Sturla Molden <sturla at molden.no> wrote:
> Den 12.04.2011 14:59, skrev Arthur de Souza Ribeiro:
>>
>> 1 - Compile package modules - json module is inside a package (files:
>> __init__.py, decoder.py, encoder.py, decoder.py) is there a way to generate
>> the cython modules just like its get generated by cython?
>>
>
>
> I'll propose these 10 guidelines:
>
> 1. The major concern is to replace the manual use of Python C API with
> Cython.  We aim to improve correctness and readability, not speed.

Speed is a concern, otherwise many of these modules wouldn't have been
written in C in the first place (at least, not the ones with a pure
Python counterpart). Of course some of them are just wrapping C
libraries where speed doesn't matter as much.

> 2. Replacing plain C with Cython for readability is less important,
> sometimes even discourged.

Huh? I'd say this is a big point of the project. Maybe less so than
manual dependance on the C API, but certainly not discouraged.

> If you do, it's ok to leverage on Python
> container types if it makes the code concise and readable, even if it will
> sacrifice some speed.

That's true.

> 3. Use exceptions instead of C style error checks: It's better to ask
> forgiveness than permission.

Yep, this is natural in Cython.

> 4. Use exceptions correctly. All resourse C allocation belongs in __cinit__.
> All C resource deallocation belongs in __dealloc__. Remember that exceptions
> can cause resource leaks if you don't. Wrap all resource allocation in an
> extension type. Never use functions like malloc or fopen directly in your
> Cython code, except in a __cinit__ method.

This can be useful advice, but is not strictly necessary. Try..finally
can fill this need as well.

> 5. We should keep as much of the code in Python as we can. Replacing Python
> with Cython for speed is less important. Only the parts that will really
> benefit from static typing should be changed to Cython.

True. Of course, compiling the (unchanged) pure Python files with
Cython could also yield interesting results, but that's not part of
the project.

> 6. Leave the __init__.py file as it is. A Python package is allowed contain
> a mix of Python source files and Cython extension libraries.
>
> 7. Be careful to release the GIL whenever appropriate, and never release it
> otherwise. Don't yield the GIL just because you can, it does not come for
> free, even with a single thread.
>
> 8. Use the Python and C standard libraries whenever you can.  Don't
> re-invent the wheel. Don't use system dependent APIs when the standard
> libraries declare a common interface. Callbacks to Python are ok.
>
> 9. Write code that will work correctly on 32 and 64 bit systems, big- or
> little-endian. Know your C: Py_intptr_t can contain a pointer. Py_ssize_t
> can represent the largest array size allowed. Py_intptr_t and Py_ssize_t can
> have different size. The native array offset can be different from
> Py_ssize_t, for which a common example is AMD64.

It's rare to have to do pointer arithmetic in Cython, and rarer still
to have to store the pointer as an integer.

> 10. Don't clutter the namespace, use pxd includes. Short source files are
> preferred to long. Simple is better than complex. Keep the source nice and
> tidy.

Not sure what you mean by "pxd includes," but yes, you should use pxd
files and cimport just as you would in Python to keep things
manageable and modular.

- Robert