File buffer flush issue in multi-process in Windows

[javen72 at gmail.com]

Hi,

I encountered a very strange issue in file flush operation in Windows.
Here's the scenario of my application:

    1. The main thread of my application will create makefiles
sequentially.
    2. Once a makefile is generated, launch a new process calling
nmake.exe to run it. The main thread then create another makefile
until no more makefiles to create.
    3. The number of new processes is limited by command line options.

My application has been running for almost a year without any problem.
But, after I made some changes recently to the content of makefile
generated, "nmake.exe" in separate process sometimes complains that
the makefile is not found. When I went into the directory, the
makefile did exist there.

Because I didn't change any thing related to file creation and the new
makefiles are a little bit less than before, I guessed that the
makefile just created hasn't been flushed to disk because of size
change so the new process could not see it in a short time.

So I decided add code to force flush file buffer after writing the
file content (I didn't flush the file explicitly before). I did it
like below:

    Fd = open(File, "w")
    Fd.write(Content)
    Fd.flush()
    os.fsync(Fd.fileno())
    Fd.close()

The strangest thing happened. The "makefile" missing happened more
frequently than no flush operation. I searched the web but no answer
there.

Finally I tried to use Windows file API to create the file via pywin32
extension. The problem's gone.

        import win32file
        Fd = win32file.CreateFile(
                        File,
                        win32file.GENERIC_WRITE,
                        0,
                        None,
                        win32file.CREATE_ALWAYS,
                        win32file.FILE_ATTRIBUTE_NORMAL,
                        None
                        )
        win32file.WriteFile(Fd, str(Content), None)
        win32file.FlushFileBuffers(Fd)
        win32file.CloseHandle(Fd)

I tried writing small python extension in C to make use Windows API to
create file like above. It also works well, even I removed the
FlushFileBuffers() calling.

Did anyone encounter such problem before? Did I do something wrong in
Python file operation? If not, I believe that there's a bug in Python
file buffer mechanism. I'd appreciate any help or hints.

Cheers,
Javen