[OT] Lying hard drives [was Re: shutil.copyfile is incomplete (truncated)]

[Steven D'Aprano]

On Sat, 13 Apr 2013 03:33:29 +0100, Nobody wrote:

> On Fri, 12 Apr 2013 00:06:21 +0000, Steven D'Aprano wrote:
> 
>>> The close method is defined and flushing and closing a file, so it
>>> should not return until that's done.
>> 
>> But note that "done" in this case means "the file system thinks it is
>> done", not *actually* done. Hard drives, especially the cheaper ones,
>> lie. They can say the file is written when in fact the data is still in
>> the hard drive's internal cache and not written to the disk platter.
>> Also, in my experience, hardware RAID controllers will eat your data,
>> and then your brains when you try to diagnose the problem.
> 
> None of which is likely to be relevant here, 

Since we've actually identified the bug (the OP was using file.close 
without actually calling it), that's certainly the case :-)


[...]
> If you want to wait for the data written to be written to the physical
> disk (in order to obtain specific behaviour with respect to an unclean
> shutdown), use f.flush() followed by os.fsync(f.fileno()).

If only it were that simple. It has been documented that some disks will 
lie, even when told to sync. When I say "some", I mean *most*. There's 
probably nothing you can do about it, apart from not using that model or 
brand of disk, so you have to just live with the risk.

http://queue.acm.org/detail.cfm?id=2367378

USB sticks are especially nasty. I've got quite a few USB thumb drives 
where the "write" light keeps flickering for anything up to five minutes 
after the OS reports that the drive has been unmounted and is safe to 
unplug. I corrupted the data on these quite a few times until I noticed 
the light. And let's not even mention the drives that have no light at 
all...

But my favourite example of lying hard drives of all time is this:

http://blog.jitbit.com/2011/04/chinese-magic-drive.html

I want one of those!



-- 
Steven