Bug 3.11.x behavioral, open file buffers not flushed til file closed.

[Cameron Simpson]

On 05Mar2023 10:38, aapost <aapost at idontexist.club> wrote:
>Additionally (not sure if this still applies):
>flush() does not necessarily write the file’s data to disk. Use flush() 
>followed by os.fsync() to ensure this behavior.

Yes. You almost _never_ need or want this behaviour. A database tends to 
fsync at the end of a transaction and at other critical points.

However, once you've `flush()`ed the file the data are then in the hands 
of the OS, to get to disc in a timely but efficient fashion. Calling 
fsync(), like calling flush(), affects writing _efficiency_ by depriving 
the OS (or for flush(), the Python I/O buffering system) the opportunity 
to bundle further data efficiency. It will degrade the overall 
performance.

Also, fsync() need not expedite the data getting to disc. It is equally 
valid that it just blocks your programme _until_ the data have gone to 
disc. I practice it probably does expedite things slightly, but the real 
world effect is that your pogramme will gratuitously block anyway, when 
it could just get on with its work, secure in the knowledge that the OS 
has its back.

flush() is for causality - ensuring the data are on their way so that 
some external party _will_ see them rather than waiting forever for data 
with are lurking in the buffer.  If that external party, for you, is an 
end user tailing a log file, then you might want to flush(0 at the end 
of every line.  Note that there is a presupplied line-buffering mode you 
can choose which will cause a file to flush like that for you 
automatically.

So when you flush is a policy decision which you can make either during 
the programme flow or to a less flexible degree when you open the file.

As an example of choosing-to-flush, here's a little bit of code in a 
module I use for writing packet data to a stream (eg a TCP connection):
https://github.com/cameron-simpson/css/blob/00ab1a8a64453dc8a39578b901cfa8d1c75c3de2/lib/python/cs/packetstream.py#L624

Starting at line 640: `if Q.empty():` it optionally pauses briefly to 
see if more packets are coming on the source queue. If another arrives, 
the flush() is _skipped_, and the decision to flush made again after the 
next packet is transcribed. In this way a busy source of packets can 
write maximally efficient data (full buffers) as long as there's new 
data coming from the queue, but if the queue is empty and stays empty 
for more that `grace` seconds we flush anyway so that the receiver 
_will_ still see the latest packet.

Cheers,
Cameron Simpson <cs at cskk.id.au>