FAM and Python? (was Re: How To Do It Faster?!?)

[Jeremy Bowers]

On Sat, 02 Apr 2005 02:02:31 +0200, andrea_gavana wrote:

> Hello Jeremy & NG,
> Every user of thsi big directory works on big studies regarding oil fields.
> Knowing the amount of data (and number of files) we have to deal with (produced
> by simulators, visualization tools, and so on) and knowing that users are
> usually lazy in doing clean up of unused/old files, this is a way for one
> of us to "fast" scan all the directories and identify which files belong
> to him. Having them in an organized, size-sorted wxPython list, the user
> can decide if he want to delete some files (that almost surely he forgot
> even that they exist...) or not. It is easy as a button click (retrieve
> the data-->delete the files).

Got it. A good idea!

>>Here's an idea to sort of come at the problem from a different angle. Can
>>you run something on the file server itself, and use RPC to access it?
> 
> I don't even know what is RPC... I have to look at it.

RPC stands for "remote procedure call". The idea is that you do something
that looks like a normal function call, except it happens on a remote
server. Complexity varies widely.

Given your situation, and if running something on the UNIX server is a
possibility, I'd recommend downloading and playing with Pyro; it is Python
specific, so I think it would be the best thing for you, being powerful,
well integrated with Python, and easy to use.

Then, on your client machine in Windows, ultimately you'd make some sort
of call to your server like

    fileList = server.getFileList(user)

and you'd get the file list for that user, returning whatever you want for
your app; a list of tuples, objects, whatever you want. Pyro will add no
constraints to your app.

> I am not sure if my new explanation fits with your last information... as
> above, I didn't even know about fam... I've read a little, but probably
> I am too newbie to see a link between it and my scope. Do you think it exists?
> It would be nice to have something that tracks the file status on all the
> file system, but probably is a LOT of work wrt what my app should be able
> to do.

Maybe, maybe not. I've never used FAM. Perhaps someone who has can chime
in about the ease of use; I've changed the subject to try to attract such
a person. It also depends on if FAM works on your UNIX.

My point is that you can do one scan at startup (can't avoid this), but
then as the file system monitor tells you that a change has occurred, you
update your data structures to account for the change. That way, your data
is always in sync. (For safety's sake, you might set the server to
terminate itself and re-start every night.) Since it's always in sync, you
can send this data back instead of scanning the file system.

At this point, my suggestion would be to consider whether you want to
spend the effort to speed it up like this, which is something only you
(and presumably your managers) are in a position to know, given that you
have an existing tool (at least, you seem to speak like you have a
functional tool). If you do, then I'd take some time and work a bit with
Pyro and FAM, and *then* re-evaluate where you stand. By then you'll
probably be able to ask better questions, too, and like I said above,
perhaps someone will share their experiences with FAM.

Good luck, and have fun; seriously, that's important here.