how to know if folder contents have changed

[Martin Marcher]

I think that without further information from the OP about the
requirements all we can do is guessing. So both of our solutions are
just theory after all (just my personal opinion)

2007/11/14, davisn90210 at gmail.com <davisn90210 at gmail.com>:
> On Nov 12, 11:27 am, "Martin Marcher" <mar... at marcher.name> wrote:
> > 2007/11/12, davisn90... at gmail.com <davisn90... at gmail.com>:
>>
> > a) create a thread that pulls all the time for changes or
>
> Given that it would only involve a check of one timestamp (the
> directory the files are located in), I don't think polling "from time
> to time" would be unreasonable.  The modification timestamp of the
> directory should be sufficient given the use case.  Even if it's not,
> tracking modification times for the files in the directory would not
> be unreasonable.

Not for the 400 Files but the OP asks about more files too. How about
40.000 files or 400.000 files? That could be a problem...

> > b) test everytime for changes
>
> Checking a timestamp should be a very quick operation.  Unless
> "everytime" occurs *very* frequently, it's certainly not unreasonable.

See above I think it also depends on the number of files

> > fam informs in a notification like way.
>
> FAM would work too.  However,
> 1) According to http://oss.sgi.com/projects/fam/faq.html#what_os_fam,
> FAM "should be fairly easy to port to ... Unix-like operating
> systems ....".  If the original poster is a user of a "Uniix-like
> operating system" he/she may actually be able to use it.  Regardless,
> it seems to me that you would lose a great deal of portability (i.e.,
> is there a Windows port?), which may or may not be important to the
> poster.

I don't use windows so speaking about portability you are right. It
may be a personal thing but I stopped providing solution (or trying to
think about them) for windows (another discussion probably best placed
in a forum about social interests or something....)

> 2) FAM undoubtedly uses some system resources.  Probably very little,
> but it's still an overhead that must be taken into account.

Both is true but most Linux distributions do use FAM at some point
anyway so the overhead is actually very little. Also I think that on
most OSs there is a similiar thing like FAM that could be used...

> 3) You still need to use another method for maintaining state across
> program invocations, do you not?

You need some method no matter wether your program is a long running
process or just invoked in irregular intervals.

After all I'm pretty sure that there is something FAM like that is
available on most OSs. FAM isn't probably available on OSX either but
I guess they provide some mechanism. If you want it really portable
I'd use an abstraction layer that tries to communicate with some
notification daemon which is probably available on the host os and if
all that fails provide a fallback implementation that does naive
tests. All accessible thru the same abstraction interface.

> Using timestamps are:
> 1) Portable.  Can you name one OS that does not provide timestamps?
> Last I checked, even Windows does :-)
> 2) Storage efficient.  I don't have to actually *store* the
> timestamps.  I can just check to see if a file/directory was modified
> after the last time I checked.

read below, a changed timestamp isn't necessarily a sign that a file
has indeed changed (backups, ....)

> 3) Easy to maintain persistent state -- just store the timestamp!

Well >>>I don't have to actually *store* the timestamps.<<< and
>>>just store the timestamp!<<< are a bit confusing. I think you
absolutely need to store the timestamp since between runs you won't
know what to check for anyway (new files, deleted files, changed files
- if these cases are important to you)

> > Personally I'd create a "hidden" cache file parsable by configparser
> > and have filename = $favorite_checksum_algo - key value pairs in it if
> > it's not a long running process.
>
> What is your reasoning for this?

because all I need to do to check for changes is getCache(configFile)
and compare the results to getActual(os.listdir) and those 2 methods
would give me the needed info (of course I'm just blindly guessing as
I don't know anything about the further requirements)

Of course with a lot of files this could be a problem. I wouldn't want
a configparser object with 40.000 (or even just a few thousand)
entries to be alive all the time. You'd probably have to create some
iterator for the file so that you can check thru the entries in a
memory efficient way...

>  It seems to me that it is
> inefficient and unreliable.  First of all you have to compute the
> checksum (which undoubtedly would involve reading every byte the file)
> -- not just once, but "everytime" (or however often you perform the
> check). Secondly, it is possible for the checksum to be the same even
> if the file has changed.  Unlikely?  Perhaps (depends on checksum
> algorithm used).  Impossible?  No.  So, in effect, you are using a
> "slow" algorithm that is known to give incorrect results in certain
> cases -- all to replace something as basic as timestamps?

It seems you are absolutely linking checksum with something like md5 or sha...

Maybe that was badly stated, depending on the use case of course a
timestamp could also be considered a valid checksum. However to be
safe some timestamp isn't really giving me the information. A lot of
backup tools do update the timestamp (atime in unix, dunno about
windows) and that could lead to even more wasting of resources.

Consider you are checking some CSV files with timestampt which upon
change initiate some real intensive number crunching. Now you do that
because you figured "Hey the timestamp has changed, I need to redo my
calculations..." while in fact just the backup programm was running.
But as I said it depends on the use case what you consider a valid to
know that a file changed...

So the checksum algo is something that should be chosen depending on

a) interval of checks (like you say)
b) need to be sure that 2 files don't actually have the same checksum

I guess a simple approach could be something like the Message-ID
header in emails, a bit adapted to local use cases.

> > Otherwise I'd probably go with fam (or hal i think that's the other
> > thing that does that)



-- 
http://noneisyours.marcher.name
http://feeds.feedburner.com/NoneIsYours