[IPython-dev] Buffers

[Brian Granger]

On Mon, Jul 26, 2010 at 9:13 PM, Min RK <benjaminrk at gmail.com> wrote:

>
>
> On Jul 26, 2010, at 19:33, Fernando Perez <fperez.net at gmail.com> wrote:
>
> > Glad you worked it out, but I'm worried about one thing: I don't
> > believe you can send unicode strings as a buffer over the wire.  The
> > reason is that the two interpreters at both ends of the connection
> > could have been compiled with different internal unicode encodings.
> > Python can be compiled to store unicode internally either as UCS-2 or
> > UCS-4, you can check sys.maxunicode to find out how your particular
> > build was made:
> >
> > http://www.python.org/dev/peps/pep-0100/
> > http://www.python.org/dev/peps/pep-0261/
> >
> > If you send a unicode string as a buffer from a ucs2 python to a ucs4
> > one, you'll get  a mess at the other end, I think.
>
> I'm not sure that is our concern. If buffers are being sent, then it's not
> zmq whose job it is to interpret that buffer, it's the user's receiving
> code.
>
> zmq only sends bytes, and for most objects, unicode included, that's what
> the buffer interface is. But sometimes a unicode object is really just a
> simple str in a unicode package, and when that's the case we interpret it as
> a string. Otherwise it's treated like all other objects - a black box that
> provides a buffer interface. It's up to the user sending the data to send it
> in a form that they can understand on the other side.
>
>
Yes, I hadn't though about the fact that unicode objects are buffers as
well.  But, we could raise a TypeError when a user tries to send a unicode
object (str in python 3).  IOW, don't treat unicode as buffers and force
them to encode/de ode.  Does this make sense or should we allow unicode to
be sent as buffers.

Brian


> > Minor note:
> > On Mon, Jul 26, 2010 at 6:43 PM, Min RK <benjaminrk at gmail.com> wrote:
> >> isinstance(s,(str,unicode))
> >
> > this is equiv. to: isinstance(s, basestring)
> >
>
> ah, thanks, I hadn't seen that one. I'll use it.
>
> >
> >
> > Cheers,
> >
> > f
>
> your points have further clarified that I was mistaken to attempt to
> support unicode strings. We support basic strings and raw buffers. When
> faced with a unicode object, we effectively (but not literally) do:
> try: send(str(u))
> except: send(buffer(u))




-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100727/13cc532c/attachment.html>