[Tutor] why is os.path.walk so slow?

Garry Willgoose Garry.Willgoose at newcastle.edu.au
Thu Nov 5 00:36:15 CET 2009


> [snip]
>> I need to synchronize the files on my home and office machine and  
>> have
>> been using someone else's code for this to date but have been
>> frustrated by how slow it is in getting the information on files for
>> the mounted drive from my office machine...
> [/snip]
>
> Not to cut your coding project short, and it may not even be
> applicable, but have you looked into rsync? They kind of wrote the
> book on efficiency in regards to synchronization of files.
>
> Just a thought.
> -Modulok-

It looks like rsync is ideal for what I want ... thanks for pointing  
this out. One less project to work on ;-)

To the other replies:

1. Both local and remote systems for the testing are OSX 10.5 and CPU  
was not limiting during the test.  Python was the Enthought binary  
distribution. Internet service is ADSL 1500/512.
2. I neglected to mention that I did comment out the code inside  
visitfunc() with the same result so the issue is inside walk().

I guess now I know abut rsync the pressing need for this is gone (I  
might still write a nice GUI for rsync ;-) but I'm still intrigued by  
the problem and whether its a latency vs bandwidth problem. The idea  
of writing a script to run on the remote end is good, and something I  
might use for another project.


More information about the Tutor mailing list