Python crashing when script is running against live system

[Chris Angelico]

On Mon, Aug 4, 2014 at 10:20 AM, Igor Korot <ikorot01 at gmail.com> wrote:
> I'm working on the script that should starting from the given
> directory enumerate all directories and files underneath and
> calculates the hash value of such file.
> It works fine when I start it from some particular directory, but when
> I give the "C:\" it crashes python.
>
> The last thing the script processes is the file in cmdcons directory,
> which is AFAIU from the ComboFix run.
> I have Windows XP SP3 with python 2.7.
>
> Now, I'm confused.
> My questions are:
>
> 1. Am I able to catch such situation and see where the crash occur?
> 2. If not, what is the proper way to debug such situation?

Great. A Heisenbug. These are so much fun...

Okay. Here's how I'd try to track it down.

1) Sort the directory listings before you start iterating, to ensure
that you're processing the files in a consistent order. Also, be sure
you're flushing to disk every time you produce a line of output - you
can probably do this by recording everything to stderr, which is
generally unbuffered.
2) Do a dry run. Run the script exactly the same as it does when it
crashes, but comment out the actual file reading and hash calculation.
This will almost certainly succeed.
3) Using the output of step 2, make yourself an enumerated text file
with all the files that the script would hash.
4) Do a half-dry run. Have the script alternate between doing the
actual hashing, and just skipping it and pretending it did.
5) Do the other-half-dry run - as above but switch whether it hashes
or skips the first one.

Ideally, your output from step 2 should be something like this:

1: c:\config.sys
2: c:\msdos.sys
3: c:\hiberfile.sys
4: c:\cmdcons\dpti20.sy_
... etc etc etc ...

Your step 4 and 5 runs should then be doing (either way around)
c:\config.sys and c:\hiberfile.sys, and c:\msdos.sys and
c:\cmdcons\dpti20.sy_. That'll give you valuable information. I can
imagine a few different crash scenarios from the half-and-half runs:

* One of them might crash on the same file that the full run crashed
on, which suggests that that file has an issue.
* They might both crash, roughly twice as far into the list, which
suggests that it's a memory leak or something that crashes after doing
a certain amount of hashing.
* They might crash at some completely different location, in which
case you'd have to look for clues there.

Another way to get some more info out of it would be to add a few
seconds' sleep after each file. You should be able to count off the
files by time, which will give additional hints. But try the other way
first, as this will potentially take a looooooong time.

There are a few possibilities to look out for - RAM fault, hard drive
issue, stuff like that - which would be a REAL pain, but if it's not
that kind of thing, it's probably a crash in the hashing code. That
would at least be a Mandelbug.

ChrisA