[Patches] [ python-Patches-432401 ] unicode encoding error callbacks
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 12 Jun 2001 11:00:10 -0700
Patches item #432401, was updated on 2001-06-12 06:43
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=432401&group_id=5470
Category: core (C code)
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Walter Dörwald (doerwalter)
Assigned to: M.-A. Lemburg (lemburg)
Summary: unicode encoding error callbacks
Initial Comment:
This patch adds unicode error handling callbacks to the
encode functionality. With this patch it's possible to
not only pass 'strict', 'ignore' or 'replace' as the
errors argument to encode, but also a callable
function, that will be called with the encoding name,
the original unicode object and the position of the
unencodable character. The callback must return a
replacement unicode object that will be encoded instead
of the original character.
For example replacing unencodable characters with XML
character references can be done in the following way.
u"aäoöuüß".encode(
"ascii",
lambda enc, uni, pos: u"&#x%x;" % ord(uni[pos])
)
----------------------------------------------------------------------
>Comment By: M.-A. Lemburg (lemburg)
Date: 2001-06-12 11:00
Message:
Logged In: YES
user_id=38388
About the Py_UNICODE*data, int size APIs:
Ok, point taken.
In general, I think we ought to keep the callback feature as
open as possible, so passing in pointers and sizes would not
be very useful.
BTW, could you summarize how the callback works in a few
lines ?
About _Encode121: I'd name this _EncodeUCS1 since that's
what it is ;-)
About the new functions: I was referring to the new static
functions which you gave PyUnicode_... names. If these are
not supposed to turn into non-static functions, I'd rather
have them use lower case names (since that's how the Python
internals work too -- most of the times).
----------------------------------------------------------------------
Comment By: Walter Dörwald (doerwalter)
Date: 2001-06-12 09:56
Message:
Logged In: YES
user_id=89016
> One thing which I don't like about your API change is that
> you removed the Py_UNICODE*data, int size style arguments
> --
> this makes it impossible to use the new APIs on non-Python
> data or data which is not available as Unicode object.
Another problem is, that the callback requires a Python
object, so in the PyObject *version, the refcount is
incref'd and the object is passed to the callback. The
Py_UNICODE*/int version would have to create a new Unicode
object from the data.
----------------------------------------------------------------------
Comment By: Walter Dörwald (doerwalter)
Date: 2001-06-12 09:32
Message:
Logged In: YES
user_id=89016
> * please don't place more than one C statement on one line
> like in:
> """
> + unicode = unicode2; unicodepos =
> unicode2pos;
> + unicode2 = NULL; unicode2pos = 0;
> """
OK, done!
> * Comments should start with a capital letter and be
> prepended
> to the section they apply to
Fixed!
> * There should be spaces between arguments in compares
> (a == b) not (a==b)
Fixed!
> * Where does the name "...Encode121" originate ?
encode one-to-one, it implements both ASCII and latin-1
encoding.
> * module internal APIs should use lower case names (you
> converted some of these to PyUnicode_...() -- this is
> normally reserved for APIs which are either marked as
> potential candidates for the public API or are very
> prominent in the code)
Which ones? I introduced a new function for every old one,
that had a "const char *errors" argument, and a few new ones
in codecs.h, of those PyCodec_EncodeHandlerForObject is
vital, because it is used to map for old string arguments to
the new function objects. PyCodec_RaiseEncodeErrors can be
used in the encoder implementation to raise an encode error,
but it could be made static in unicodeobject.h so only those
encoders implemented there have access to it.
> One thing which I don't like about your API change is that
> you removed the Py_UNICODE*data, int size style arguments > --
> this makes it impossible to use the new APIs on non-Python
> data or data which is not available as Unicode object.
I look through the code and found no situation where the
Py_UNICODE*/int version is really used and having two
(PyObject *)s (the original and the replacement string),
instead of UNICODE*/int and PyObject * made the
implementation a little easier, but I can fix that.
> Please separate the errors.c patch from this patch -- it
> seems totally unrelated to Unicode.
PyCodec_RaiseEncodeErrors uses this the have a \Uxxxx with
four hex digits. I removed it.
I'll upload a revised patch as soon as it's done.
----------------------------------------------------------------------
Comment By: M.-A. Lemburg (lemburg)
Date: 2001-06-12 07:29
Message:
Logged In: YES
user_id=38388
Thanks for the patch -- it looks very impressive !.
I'll give it a try later this week.
Some first cosmetic tidbits:
* please don't place more than one C statement on one line
like in:
"""
+ unicode = unicode2; unicodepos =
unicode2pos;
+ unicode2 = NULL; unicode2pos = 0;
"""
* Comments should start with a capital letter and be
prepended
to the section they apply to
* There should be spaces between arguments in compares
(a == b) not (a==b)
* Where does the name "...Encode121" originate ?
* module internal APIs should use lower case names (you
converted some of these to PyUnicode_...() -- this is
normally reserved for APIs which are either marked as
potential candidates for the public API or are very
prominent in the code)
One thing which I don't like about your API change is that
you removed the Py_UNICODE*data, int size style arguments --
this makes it impossible to use the new APIs on non-Python
data or data which is not available as Unicode object.
Please separate the errors.c patch from this patch -- it
seems totally unrelated to Unicode.
Thanks.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=432401&group_id=5470