[Tutor] Monitor directory Python3 script

[Mats Wichmann]

On 11/13/19 1:55 PM, Cameron Simpson wrote:
> On 13Nov2019 09:58, Nathan D'Elboux <nathan.delboux at gmail.com> wrote:
>> Im a student learning Python3. i have a small script id like to make
>> that would monitor some subdirectories for any new files that are
>> placed in there and move them to another directory on the same
>> partition
>>
>> Im looking around for modules to use and came across shutil. This
>> seems ok at first but seems more for batch processing so copy files
>> from dir1 to dir2 every time cron runs which could lead to the same
>> file being copied.
> 
> You make a point of saying "on the same partition" (which I'll take to 
> imply on the same filesystem). So you can just rename (mv) things 
> instead of copying them; very fast.
> 
> shutil is overkill for that, just use os.rename.
> 
>> Im thinking if i used pyinotify to monitor the
>> directory then only newly added files would be copied and not
>> everything before it every time it copies.
> 
> If you know you're on Linux that is reasonable. Again, possibly overkill.
> 
> Personally, I'd just have a loop while ran os.listdir on your target 
> directory with a 1 second sleep between loops. Notice anything new, move 
> it. You can keep a set() of names representing the "old" stuff, not to 
> be moved.
> 
> So for me: os.listdir, os.rename, os.path.{basename,dirname,join} would 
> cover everything required. And time.sleep for a short end of loop pause.

There's almost certainly code out there that does what you want, 
possibly buried in a framework of some sort. But as David just said 
(after I wrote this but before sending), it may not be easy to find.

I'm going to agree with Cameron that you can hack this together very 
simply. inotify is the "official" Linux API for what you want to do, but 
you don't necessarily need to use it. Use if it you expect to have a 
long-running process that wants to be very low-impact on the system.

polling is plenty good for a small simple case.  it doesn't scale 
particularly well. Sounded like small is all you were looking for. you 
can hack together something quick, and then if it needs to be used long 
term you can refactor it into a more robust tool with config options, 
able to run as a daemon process, whatever.

(sorry for the complete lack of Python content in this reply, it's all 
generic commentary)