[Python-checkins] r74336 - in python/trunk: Lib/test/test_io.py Misc/NEWS Modules/_io/bufferedio.c

[antoine.pitrou]

Author: antoine.pitrou
Date: Thu Aug  6 22:18:29 2009
New Revision: 74336

Log:
Issue #6629: Fix a data corruption issue in the new `io` package, which could
occur when writing to a BufferedRandom object (e.g. a file opened in "rb+" or
"wb+" mode) after having buffered a certain amount of data for reading. This
bug was not present in the pure Python implementation.

Yes, this is a serious issue.



Modified:
   python/trunk/Lib/test/test_io.py
   python/trunk/Misc/NEWS
   python/trunk/Modules/_io/bufferedio.c

Modified: python/trunk/Lib/test/test_io.py
==============================================================================
--- python/trunk/Lib/test/test_io.py	(original)
+++ python/trunk/Lib/test/test_io.py	Thu Aug  6 22:18:29 2009
@@ -1336,6 +1336,26 @@
             bufio.readinto(bytearray(1))
         self.check_writes(_read)
 
+    def test_write_after_readahead(self):
+        # Issue #6629: writing after the buffer was filled by readahead should
+        # first rewind the raw stream.
+        for overwrite_size in [1, 5]:
+            raw = self.BytesIO(b"A" * 10)
+            bufio = self.tp(raw, 4)
+            # Trigger readahead
+            self.assertEqual(bufio.read(1), b"A")
+            self.assertEqual(bufio.tell(), 1)
+            # Overwriting should rewind the raw stream if it needs so
+            bufio.write(b"B" * overwrite_size)
+            self.assertEqual(bufio.tell(), overwrite_size + 1)
+            # If the write size was smaller than the buffer size, flush() and
+            # check that rewind happens.
+            bufio.flush()
+            self.assertEqual(bufio.tell(), overwrite_size + 1)
+            s = raw.getvalue()
+            self.assertEqual(s,
+                b"A" + b"B" * overwrite_size + b"A" * (9 - overwrite_size))
+
     def test_misbehaved_io(self):
         BufferedReaderTest.test_misbehaved_io(self)
         BufferedWriterTest.test_misbehaved_io(self)

Modified: python/trunk/Misc/NEWS
==============================================================================
--- python/trunk/Misc/NEWS	(original)
+++ python/trunk/Misc/NEWS	Thu Aug  6 22:18:29 2009
@@ -354,6 +354,11 @@
 Library
 -------
 
+- Issue #6629: Fix a data corruption issue in the new `io` package, which could
+  occur when writing to a BufferedRandom object (e.g. a file opened in "rb+" or
+  "wb+" mode) after having buffered a certain amount of data for reading. This
+  bug was not present in the pure Python implementation.
+
 - Issue #4660: If a multiprocessing.JoinableQueue.put() was preempted, it was
   possible to get a spurious 'task_done() called too many times' error.
 

Modified: python/trunk/Modules/_io/bufferedio.c
==============================================================================
--- python/trunk/Modules/_io/bufferedio.c	(original)
+++ python/trunk/Modules/_io/bufferedio.c	Thu Aug  6 22:18:29 2009
@@ -1774,6 +1774,19 @@
     }
     Py_CLEAR(res);
 
+    /* Adjust the raw stream position if it is away from the logical stream
+       position. This happens if the read buffer has been filled but not
+       modified (and therefore _bufferedwriter_flush_unlocked() didn't rewind
+       the raw stream by itself).
+       Fixes issue #6629.
+    */
+    n = RAW_OFFSET(self);
+    if (n != 0) {
+        if (_buffered_raw_seek(self, -n, 1) < 0)
+            goto error;
+        self->raw_pos -= n;
+    }
+
     /* Then write buf itself. At this point the buffer has been emptied. */
     remaining = buf.len;
     written = 0;