Need help with shutils.copytree

[Chris Rebert]

On Sun, Feb 12, 2012 at 12:14 PM, Steven W. Orr <steveo at syslang.net> wrote:
> I have a 'master' directory and a collection of 'slave' dirs. I want the
> master to collect all of the stuff in the slave dirs.
>
> The slaves all look like this,
>
> .
> |-- slaveX
> |   `-- archI
> |   |   `-- distJ
> |   |   |   ` -- FILE
>
> Where the different slaveX dirs may contain multiple occurrences of archI
> and distJ, but across all slaveX dirs, there will only be one *unique*
> instance of FILE in archI and distJ.
>
> Here's an example: Given slave[1234], arch1 and arch2, and dist1 and dist2,
> I want master to end up looking like this:
>
> .
> |-- master
> |   `-- arch1
> |   |   ` -- dist1
> |   |   |    ` -- FILE
> |   `-- arch1
> |   |   ` -- dist2
> |   |   |    ` -- FILE
> |   `-- arch2
> |   |   ` -- dist1
> |   |   |    ` -- FILE
> |   `-- arch2
> |   |   ` -- dist2
> |   |   |    ` -- FILE
>
> etc...

You have multiple directories at the same level in the hierarchy with
identical names (e.g. two "arch1"s), which is invalid. I assume you
meant for them to be combined?

> In bash, I might use cpio passthrough mode and say something like:
>
> master=$path_to_master
> for slave in ${slaves}
> do
>    pushd $slave
>    find . -print | cpio -pdum $master
>    popd
> done
>
> but I'm having a hard time trying to get this functionality in python. (I'm
> trying to avoid writing a subprocess.)
>
> I tried using shutil.copytree with a try / except that does a pass on
> OSError (which is what gets raised when trying to create a dir that already
> exists). No joy there.

Right; the stack has already been unwound by the time your `except`
clause is reached.
You just need to recover by instead copying the children of the
subtree individually yourself when their parent already exists.
For example: master/arch1 already exists? Then copy the
slave/arch1/distN-s individually.

Or alternately, abandon copytree() entirely: you just LYBL and check
if the parent directories already exist; if not, you try to create the
directories yourself; and finally, you copy the individual files.
[Useful funcs: os.listdir(), os.path.exists(), os.mkdir() /
os.mkdirs()]

> I also tried an ignore function that always returns ().

That's an effective no-op which doesn't alter copytree()'s behavior whatsoever.

Cheers,
Chris
--
http://rebertia.com