Python 2.5.1 broken os.stat module
Joe Salmeri
JoeSalmeri at hotmail.com
Sat Jun 2 21:45:46 EDT 2007
> It's not clear whether it's an error, however, localtime() does
> something different from what dir does.
Hi Martin,
First off thank you for spending your time to investigate this bug with me.
Thanks for pointing out the issue with Windows XP caching the access
timestamps,
I was not aware of that.
The reality is that I am most interested in the last write time because that
is
the timestamp that Windows displays when you issue a dir command (WITHOUT a
/t option),
what Windows Explorer displays in detail view, and what every other program
I have
found so far references.
I re-reviewed all differences found by the program I posted for the
Python\lib directory
date comparison and if you eliminate access times all differences for
created and last
write that were reported were a difference of 1 hour.
Based on a bunch of testing that I did and some references listed below I
believe
that you are CORRECT, Python 2.5.1 is reporting the correct times and
Windows is
reporting the times incorrectly.
The short explaination of this issue is that the timestamp shown when you do
a dir
on a file that is on an NTFS volume changes by an hour when DST starts and
also when
DST ends, even though the file has NOT been modified!!! Note that this only
happens
if you have the setting turned on to automatically adjust for DST (this is
the default
in Windows).
Even though Python 2.5.1 is correct this presents a NEW problem because it
differs
from what XP SP 2 (wrongly) reports. I am sure this will cause a lot of
confusion
for people as it already appears that others have stumbled on this
difference.
Here are some tests I did which show the problem that Windows has.
Setup Test Environment Virtual machine
XP SP2
Set TimeZone to GMT with automatically adjust for DST turned on
Set Date and Time to 01/02/2007 12:00:00 AM
Create file with notepad named joe.txt
This establishes our baseline:
joe.txt last written 01/02/2007 12:00 AM GMT
Check what Windows, Python 2.4.2 and Python 2.5.1 Report
dir /tc
01/02/2007 12:00 AM 14 joe.txt
dir /tw
01/02/2007 12:00 AM 14 joe.txt
Python 2.4.2
Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:08
Python 2.5.1
Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:08
Great so far everybody is in agreement
Change Environment
Set TimeZone to GMT-5:00 Eastern Time (US & Canada) (where I am located)
DST still set to automatically adjust
No changes to date and time at this point
With these changes joe.txt should now report 01/01/2007 07:00 PM since I
am 5 hours behind GMT
Check what Windows, Python 2.4.2 and Python 2.5.1 Report
dir /tc
01/01/2007 07:00 PM 14 joe.txt
dir /tw
01/01/2007 07:00 PM 14 joe.txt
Python 2.4.2
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08
Python 2.5.1
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08
Great so far everyone is still in agreement
Change Environment
Leave TimeZone set to GMT-5:00 Eastern Time (US & Canada) (where I am
located)
DST still set to automatically adjust
Change date to todays date 06/02/2007 and time 02:00:00 PM
This test simulates my machine moving in time forward from 01/1/2007 to
06/02/2007
with no changes being made on the computer. Imagine I turned it off on
01/02/2007
and went on a long vacation and just came home on 06/02/2007 and turned
it back on.
Check what Windows, Python 2.4.2 and Python 2.5.1 Report
dir /tc
01/01/2007 08:00 PM 14 joe.txt
dir /tw
01/01/2007 08:00 PM 14 joe.txt
Python 2.4.2
Creation Time: 01/01/2007 20:00:04
Last Write Time: 01/01/2007 20:00:08
Python 2.5.1
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08
This is where the problem occurs.
This file was originally created 01/02/2007 12:00 AM GMT
which is the same locally as 01/01/2007 07:00 PM EST
but now everything EXCEPT Python 2.51
is reporting that the file changed on 01/01/2007 08:00 PM EST
Well that couldn't have happened since no changes were made (remember I
was on vacation :-)).
Everything was fine until our current date time enters DST.
At that point the file now shows that it was last written an hour later.
The last write timestamp for the file should not change or be affected
because of the start/end of DST has occurred on my computer.
To the end user it appears the file was modifed when in fact it was not.
Python 2.5.1 is CORRECT
Windows is WRONG (dir /t, Explorer)
Python 2.4.2 is WRONG
One Possible Work Around for the problem:
Turn OFF the automatically adjust for DST setting
NOTE: Turning this off causes issues with time sync. See details at the
end.
With it turned off the timestamps are all correct
dir /tc
01/01/2007 07:00 PM 14 joe.txt
dir /tw
01/01/2007 07:00 PM 14 joe.txt
Python 2.4.2
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08
Python 2.5.1
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08
I was concered that since I had the automatically adjust DST setting turned
on when
I originally started with the time set to GMT 12:00 AM that it might have
affected
my results so I reran the tests.
Setup Test Environment Virtual machine
XP SP2
Set TimeZone to GMT WITHOUT automatically adjust for DST
Set Date and Time to 01/02/2007 12:00:00 AM
Create file with notepad named joe.txt
This establishes our baseline:
joe.txt last written 01/02/2007 12:00 AM
Check what Windows, Python 2.4.2 and Python 2.5.1 Report
dir /tc
01/02/2007 12:00 AM 16 joe.txt
dir /tw
01/02/2007 12:00 AM 16 joe.txt
Python 2.4.2
Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:09
Python 2.5.1
Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:09
Just like the first time everone is in agreement
Change Environment
Set TimeZone to GMT-5:00 Eastern Time (US & Canada) (where I am located)
DST still NOT set to automatically adjust
No changes to date and time at this point
With these changes joe.txt should now report 01/01/2007 07:00 PM since I
am 5 hours behind GMT
Check what Windows, Python 2.4.2 and Python 2.5.1 Report
dir /tc
01/01/2007 07:00 PM 16 joe.txt
dir /tw
01/01/2007 07:00 PM 16 joe.txt
Python 2.4.2
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09
Python 2.5.1
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09
Just like the first time everyone still in agreement
Change Environment
Leave TimeZone set to GMT-5:00 Eastern Time (US & Canada) (where I am
located)
DST still NOT set to automatically adjust
Change date to todays date 06/02/2007 and time 02:00:00 PM
Check what Windows, Python 2.4.2 and Python 2.5.1 Report
dir /tc
01/01/2007 07:00 PM 16 joe.txt
dir /tw
01/01/2007 07:00 PM 16 joe.txt
Python 2.4.2
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09
Python 2.5.1
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09
Everyone still in agreement.
This confirms that the bug is in Windows when the automatically adjust
for DST
setting is turned on. Windows should NOT be using the DST setting to
adjust
the file times. It makes it look like the files were modified when in
fact
they have not.
If you turn it on after the last test then you get the following results
which
again prove that Python 2.5.1 is correct and that Windows and Python
2.4.2 are
wrong.
dir /tc
01/01/2007 08:00 PM 16 joe.txt
dir /tw
01/01/2007 08:00 PM 16 joe.txt
Python 2.4.2
Creation Time: 01/01/2007 20:00:04
Last Write Time: 01/01/2007 20:00:09
Python 2.5.1
Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09
The real issue now is that even though Python 2.5.1 is correct it is
confusing to
someone that is trying to compare their results to what Windows reports.
If you turn of automatically adjust DST and run the program I provided to
compare
the dates between windows and python in the other email message against the
Python\Lib directory then all create dates and all write dates between
Python and
Windows compare. Access dates do not match but you have already explained
why that
can occur (caching).
As shown above, one way to fix this is to turn OFF the automatically adjust
for
DST setting in Windows.
The problem with turning it OFF is that if you have automatic time sync
turned ON
when the time sync occurs and automatically adjust for DST is turned OFF and
it is DST
your time will be an hour off after the sync :-(.
Also any computers that are syncing their time with this computer that have
automatically adjust for DST turned ON will have their sync'd times one hour
off since this computer had automatically adjust for DST turned OFF.
The following link goes into even greater detail as to the source of the
problem
in Windows:
http://search.cpan.org/~shay/Win32-UTCFileTime-1.45/lib/Win32/UTCFileTime.pm
Based on the above article it does not seem like Microsoft will be fixing
this
which leaves us stuck because Python and Windows are not reporting the same
values when DST starts/ends and you have automatically adjust for DST turned
ON.
Having this difference between Python and Windows makes it difficult to
write
code that depends on those values being in sync.
It makes me wonder if there should be an option in localtime to "fudge" the
dates
so that they match what Windows reports....
Thanks to all that assisted in investigating this.
More information about the Python-list
mailing list