[Python-Dev] Fw: Behavior of buffer()
Todd Miller
jmiller@stsci.edu
Thu, 18 Jul 2002 11:59:16 -0400
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title></title>
</head>
<body>
Scott Gilbert wrote:<br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap="">--- Todd Miller <a class="moz-txt-link-rfc2396E" href="mailto:jmiller@stsci.edu"><jmiller@stsci.edu></a> wrote:<br></pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">I don't understand what you say, but I believe you.<br><br></pre>
</blockquote>
<pre wrap="">I meant we call PyBuffer_FromReadWriteObject and the resulting buffer <br>lives longer than the extension function call that created it. I have <br>heard that it is possible for the original object to "move" leaving the <br>buffer object pointer to it dangling.<br></pre>
</blockquote>
<pre wrap=""><!----><br>Yes. The PyBufferObject grabs the pointer from the PyBufferProcs<br>supporting object when the PyBufferObject is created. If the PyBufferProcs<br>supporting object reallocates the memory (possibly from a resize) the</pre>
</blockquote>
Thanks for the example.<br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br>PyBufferObject can be left with a bad pointer. This is easily possible if<br>you try to use the array module arrays as a buffer.</pre>
</blockquote>
This is good to know.<br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br><br>I've submitted a patch to fix this particular problem (among others), but<br>there are still enough things that the buffer object can't do that<br>something new is needed.<br></pre>
</blockquote>
I understand. I saw your patches and they sounded good to me.<br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br></pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap=""><br></pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Maybe instead of the buffer() function/type, there should be a way to<br>allocate raw memory?<br><br></pre>
</blockquote>
</blockquote>
<blockquote type="cite">
<pre wrap="">Yes. It would also be nice to be able to:<br><br>1. Know (at the python level) that a type supports the buffer C-API.<br><br></pre>
</blockquote>
<pre wrap="">Good idea. (I guess right now you can see if calling buffer() with an<br>instance as argument works. :-)<br><br></pre>
<blockquote type="cite">
<pre wrap="">2. Copy bytes from one buffer to another (writeable buffer). <br><br></pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap=""><!----><br>And the copy operations shouldn't create any large temporaries:</pre>
</blockquote>
I agree with this completely. I could summarize my opinion by saying that
while<br>
I regard the current buffering system as pretty complete, the buffer object
places emphasis<br>
on the wrong behavior. In terms of modelling memory regions, strings are
the wrong way<br>
to go.
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br><br> buf1 = memory(50000)<br> buf2 = memory(50000)<br> # no 10K temporary should be created in the next line<br> buf1[10000:20000] = buf2[30000:40000] <br><br>The current buffer object could be used like this, but it would create a<br>temporary string. <br></pre>
</blockquote>
Looking at buffering most of this week, the fact that mmap slicing also returns
strings is one justification I've found for having a buffer object, i.e.,
mmap slicing is not a substitute for the buffer object. The buffer object
makes it possible to partition a mmap or any bufferable object into pseudo-independent,
possibly writable, pieces. <br>
<br>
One justification to have a new buffer object is pickling (one of Scott's
posts alerted me to this). I think the behavior we want for numarray is
to be able to pickle a view of a bufferable object more or less like a string
containing the buffer image, and to unpickle it as a memory object. The
prospect of adding pickling support makes me wonder if seperating the allocator
and view aspects of the buffer object is a good idea; I thought it was,
but now I wonder.<br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br>So getting an efficient copy operation seems to require that slices just<br>create new "views" to the same memory.</pre>
</blockquote>
Other justifications for a new buffer object might be:<br>
<br>
1. The ability to partition any bufferable object into regions which can
be passed around. These regions<br>
would themselves be buffers.<br>
<br>
2. The ability to efficiently pickle a view of any bufferable object.<br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br></pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Maybe you would like to work on a requirements gathering for a memory<br>object<br><br></pre>
</blockquote>
<pre wrap="">Sure. I'd be willing to poll comp.lang.python (python-list?) and <br>collate the results of any discussion that ensues. Is that what you had <br>in mind?<br><br></pre>
</blockquote>
<pre wrap=""><!----><br><br>In the PEP that I'm drafting, I've been calling the new object "bytes"<br>(since it is just a simple array of bytes). Now that you guys are<br>referring to it as the "memory object", should I change the name? Doesn't<br>really matter, but it might avoid confusion to know we're all talking about<br>the same thing.<br><br></pre>
</blockquote>
Calling this a memory type sounds the best to me. The question I have not
resolved for myself <br>
is whether there should be one type which "does it all" or two types, a memory
allocator and a bufferable<br>
object manipulator. <br>
<blockquote type="cite"
cite="mid20020715175256.5971.qmail@web40112.mail.yahoo.com">
<pre wrap=""><br><br><br>__________________________________________________<br>Do You Yahoo!?<br>Yahoo! Autos - Get free new car price quotes<br><a class="moz-txt-link-freetext" href="http://autos.yahoo.com">http://autos.yahoo.com</a><br></pre>
</blockquote>
<br>
<br>
</body>
</html>