[Patches] [ python-Patches-1630118 ] Patch to add tempfile.SpooledTemporaryFile (for #415692)

[SourceForge.net]

Patches item #1630118, was opened at 2007-01-07 14:36
Message generated for change (Comment added) made by djmitche
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1630118&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Library (Lib)
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Dustin J. Mitchell (djmitche)
Assigned to: Nobody/Anonymous (nobody)
Summary: Patch to add tempfile.SpooledTemporaryFile (for #415692)

Initial Comment:
Attached please find a patch that adds a SpooledTemporaryFile class to tempfile, along with the corresponding documentation (optimistically labeling the feature as added in Python 2.5) and some test cases.

----------------------------------------------------------------------

>Comment By: Dustin J. Mitchell (djmitche)
Date: 2007-02-03 11:22

Message:
Logged In: YES 
user_id=7446
Originator: YES

Sounds good -- here's a new version.
File Added: SpooledTemporaryFile.patch

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2007-01-11 01:55

Message:
Logged In: YES 
user_id=4771
Originator: NO

Reimplementing the whole file interface as a proxy functions might be the
safest route, yes.

I realized that the __getattr__() magic is also used to serve at least one
special method, namely the __iter__() of the file objects.  This only works
with old-style classes.  In the long-term future, when old-style classes
disappear and these classes become new-style, this is likely to introduce a
subtle bug.

----------------------------------------------------------------------

Comment By: Dustin J. Mitchell (djmitche)
Date: 2007-01-08 09:53

Message:
Logged In: YES 
user_id=7446
Originator: YES

I agree it would break in such a situation, but I'm not clear on which
direction your bias leads you (specifically, which do we get right -- don't
use bound methods, or don't use the __getattr__ magic?).

I could fix this by defining "proxy" functions (and some properties) for
the whole file interface, rather than just the methods that potentially
trigger rollover.  That would lose a little efficiency, but mostly only in
reading (calling e.g., f.read() will always result in two function
applications; in the current model, after the first call it runs at
"native" speed).  It would also lose forward compatibility if the file
protocol changes, although I'm not sure how likely that is.

Would you like me to do that?

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2007-01-08 02:26

Message:
Logged In: YES 
user_id=4771
Originator: NO

The __getattr__ magic makes the following kind of code fail with
SpooledTemporaryFile:

  f = SpooledTemporaryFile(max_size=something)
  rd = f.read
  wr = f.write
  for x in y:
      ...use rd(size) and wr(data)...

The problem is that the captured 'f.read' method is the one from the
StringIO instance, even after the write() rolled the file over to disk. 
Given that capturing bound methods is a semi-official speed hack advertised
in some respected places, we might have to be careful about it.  About such
matters I am biased towards first getting it right and then getting it
fast...

Also, Python 2.5 is already out, so this will probably be a 2.6 addition.

----------------------------------------------------------------------

Comment By: Dustin J. Mitchell (djmitche)
Date: 2007-01-07 14:37

Message:
Logged In: YES 
user_id=7446
Originator: YES

File Added: SpooledTemporaryFile.patch

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1630118&group_id=5470