[issue17560] problem using multiprocessing with really big objects?

[Richard Oudkerk]

Richard Oudkerk added the comment:

On 27/03/2013 8:14pm, Charles-François Natali wrote:
>
> Charles-François Natali added the comment:
>
>> Apart from creating, unlinking and resizing the file I don't think there
>> should be any disk I/O.
>>
>> On Linux disk I/O only occurs when fsync() or close() are called.
>
> What?
> Writeback occurs depending on the memory pressure, percentage of used
> pages, page modification time, etc. Try writing a large file without
> closing it, you'll see that there's disk activity (or use
> iostat/vmstat).

I meant when there is no memory pressure.

>> FreeBSD has a MAP_NOSYNC flag which gives Linux behaviour (otherwise
>> dirty pages are flushed every 30-60).
>
> It's the same on Linux, depending on your mount options, data will be
> committed to disk every 5 seconds or so, when the journal is
> committed.

Googling suggsests that MAP_SHARED on Linux is equivalent to MAP_SHARED 
| MAP_NOSYNC on FreeBSD.  I don't think it has anything to do with mount 
options.

The Linux man page refuses to specify

   MAP_SHARED
     Share this mapping. Updates to the mapping are visible to other
     processes that map this file, and are carried through to the
     underlying file. **The file may not actually be updated until
     msync(2) or munmap() is called.**

>> Once the file has been unlink()ed then any sensible operating system
>> should realize it does not need to sync the file.
>
> Why?
> Even if you delete the file right after open, if you write data to it,
> when the amount of data written fills your caches, the data has to go
> somewhere, even if only to make it available to the current process
> upon read()...

Can you demonstrate a slowdown with a benchmark?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17560>
_______________________________________