[issue36129] io documentation unclear about flush() and close() semantics for wrapped streams

[Gregory Szorc]

Gregory Szorc <gregory.szorc at gmail.com> added the comment:

Thank you for the detailed reply, Josh.

I generally agree with what you are saying. However, I have some follow-ups.

In your answer to #2, you seem to distinguish between an "fd" (implying POSIX file descriptor) and "Python layers" (implying a file object). Is this correct? Should a "closefd" argument only apply when receiving an integer file descriptor? Or should a "closefd" argument also apply when receiving any io.RawIOBase object? If it doesn't apply for any generic file object, is there any recommended way to control whether close() cascades? (My opinion is that "closefd" should apply to any object with a close(), not just integer file descriptors.)

lzma.LZMAFile and gzip.GZIPFile do *not* cascade close() when a file object (as opposed to a named file) is passed into the constructor. (Presumably bz2.BZ2File behaves the same way but the documentation doesn't state this.) This is inconsistent with your answer that close() should always cascade. It's worth noting that the docs for GZIPFile explicitly call out reasons why close() does not cascade.

None of these compression *File constructors accept a "closefd" argument to control the behavior. If the goal is for close() to cascade by default, then it seems useful for these *File types to support automatic close() cascade. Although changing the default would be a backwards compatibility break and I'm not sure that's feasible. But at least you'd have the ability to opt in to the behavior.

It's also worth calling out the behavior of io.BytesIO and io.StringIO. When you close() these streams, you cannot call .getvalue() to get the buffer content. This is consistent with the io.RawIOBase behavior of not allowing I/O after a close(). However, the implication is that if you wrap a BytesIO/StringIO and then a close() on the outer stream cascades into the BytesIO/StringIO, you won't be able to access the written data! In fact, I encountered this problem when writing python-zstandard's unit tests! I had to implement a custom type that allowed access to getvalue() post close() (https://github.com/indygreg/python-zstandard/blob/735771961bc04f8f7de9372297921826a814fd12/tests/common.py#L82) to work around it.

Assuming a "closefd" argument applies to all file objects (not just file descriptors) and that all stream wrappers should accept a "closefd" argument to control close() cascade, I think I have a path forward for python-zstandard: I simply add a "closefd" argument to the stream wrapper constructors. But if "closefd" doesn't apply to generic file objects, then I'm still not sure how to proceed, as I don't want to implement behavior that conflicts with the standard library's.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36129>
_______________________________________