long lists

[Merrigan]

On May 7, 10:21 pm, "Gabriel Genellina" <gagsl-... at yahoo.com.ar>
wrote:
> En Mon, 07 May 2007 09:14:34 -0300, Merrigan <charl.loub... at gmail.com>
> escribió:
>
> > The Script it available at this url :
> >http://www.lewendewoord.co.za/theScript.py
>
> I understand this as a learning exercise, since there are lot of utilities
> for remote syncing.
>
> Some comments:
> - use os.path.join to build file paths, instead of concatenating strings.
> - instead of reassigning sys.stdout before the call to retrlines, use the
> callback:
>
>      saveinfo = sys.stdout
>      fsock = open(tempDir + "remotelist.txt", "a")
>      sys.stdout = fsock
>      ftpconn.cwd(remotedir) #This changes to the remote directory
>      ftpconn.retrlines("LIST") #This gets a complete list of everything in
> the directory
>      sys.stdout = saveinfo
>      fsock.close()
>
> becomes:
>
>      fsock = open(os.path.join(tempDir,"remotelist.txt"), "a")
>      ftpconn.cwd(remotedir) #This changes to the remote directory
>      ftpconn.retrlines("LIST", fsock.write) #This gets a complete list of
> everything in the directory
>      fsock.close()
>      (Why mode="a"? Shouldn't it be "w"? Isn't the listing for a single
> directory?)
>
> - Saving both file lists may be useful, but why do you read them again? If
> you already have a list of local filenames and remote filenames, why read
> them from the saved copy?
> - It's very confusing having "filenames" ending with "\n" - strip that as
> you read it. You can use fname = fname.rstrip()
> - If you are interested on filenames with a certain extension, only
> process those files. That is, filter them *before* the processing begins.
>
> - The time-consuming part appears to be this:
>
> def comp_are():
>      global toup
>      temptoup = []
>      for file1 in remotefiles:
>          a = file1
>          for file2 in localfiles:
>              b = file2
>              if str(a) == str(b):
>                  pass
>              if str(b) != str(a):
>                  temptoup.append(str(str(b)))
>                  toup = list(sets.Set(temptoup))
>      for filename in remotefiles:
>          fn2up = filename
>          for item in toup:
>              if fn2up == item:
>                  toup.remove(item)
>              else:
>                  pass
>      toup.sort()
>
> (It's mostly nonsense... what do you expect from str(str(b)) different
>  from str(b)? and the next line is just a waste of time, can you see why?)
> I think you want to compare two lists of filenames, and keep the elements
> that are on one "localfiles" list but not on the other. As you appear to
> know about sets: it's the set difference between "localfiles" and
> "remotefiles". Keeping the same "globalish" thing:
>
> def comp_are():
>      global toup
>      toup = list(sets.Set(localfiles) - sets.Set(remotefiles))
>      toup.sort()
>
> Since Python 2.4, set is a builtin type, and you have sorted(), so you
> could write:
>
> def comp_are():
>      global toup
>      toup = sorted(set(localfiles) - set(remotefiles))
>
> - Functions may have parameters and return useful things :)
> That is, you may write, by example:
>
>    remotefiles = getRemoteFiles(host, remotedir)
>    localfiles = getLocalFiles(localdir)
>    newfiles = findNewFiles(localfiles, remotefiles)
>    uploadFiles(host, newfiles)
>
> --
> Gabriel Genellina

Hmmm, thanks a lot. This has really been helpful. I have tried putting
it in the set, and whoops, it workes. Now, I think I need to start
learning some more.

now the script is running a lot slower...
Now to get the rest of it up and running...

Thanx for the help!