[execnet-dev] RSync Issue

[Charles Solar]

Oh my, it appears in my haste to get a fail test case I mis-remembered
what you were actually asking for.  I had completely forgotten about
the original rsync on an old gateway does not work problem I was
having.  That file was meant to demonstrate the symlink error :P

Here is a python file that demonstrates the ORIGINAL rsync problem I
was having.  Turns out its not exactly 'an old gateway' like I said
earlier.  Its a gateway where a remote channel is still executing
code.  In my programs, I start a twisted reactor on the remote
machines, so the gateway's remote code never really 'completes.'  This
causes rsync to fail to use that gateway for the rsync channel.
In this demo file you can see this by first running it and watching it
freeze on rsync send.  Then, uncomment the chan.send and chan.receive
lines and it will work.

hopefully this attachment works.

Charles

On Thu, Aug 5, 2010 at 11:58 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi Charles,
>
> On Wed, Aug 04, 2010 at 15:34 -0500, Charles Solar wrote:
>> Ok I got that fail case for you, I uploaded it to rapidshare since my
>> last attachment did not work
>> http://rapidshare.com/files/411056579/rsync_fail.tar.gz.html
>
> with execnet-1.0.8 (containing the rsync fixes) this seems to work for me
> unless i am missing something?
>
>> Also, I am going to submit a change to you on bitbucket, I added a new
>> option for sshgateways to programatically use ssh options without
>> having to give a ssh config file.  I found sending an actual file was
>> not very platform independent with paths and all that.
>> I am using it to tell ssh to shutup about rsa identification like so
>> gw = execnet.makegateway(
>> "ssh=%s@%s//ssh_options=StrictHostKeyChecking=no;UserKnownHostsFile=/dev/null"
>> % ( host['user'], host['ip'] ))
>
> sounds good.  I wonder if it's also possible to append the option
> to the actual ssh=* spec. We'd need a safe separator but maybe
> ";" is reasonably safe so it could read:
>
>    ssh=user at host;StrictHostKeyChecking=no;UserKnownHostsFile=/dev/null
>
> looking forward to your patch :)
>
> best,
> holger
>
>>
>> Charles
>>
>> On Tue, Jul 27, 2010 at 5:14 PM, Charles Solar <charlessolar at gmail.com> wrote:
>> > Sorry could not get around to testing this out today, i will get this
>> > installed next time I have a chance to run my app.
>> > Oh yea, completely forgot about the test case, I will get a small
>> > package setup to illustrate too.
>> >
>> > Charles
>> >
>> > On Tue, Jul 27, 2010 at 12:47 PM, holger krekel <holger at merlinux.eu> wrote:
>> >> Hi Charles,
>> >>
>> >> so i think that rsyncing symlinks should now work reaonsably well.
>> >> I used your patches as a starting point but also added tests and made
>> >> sure permissions are also transferred for directories.  Could you verify
>> >> that after installing
>> >>
>> >>    http://hudson.testrun.org/view/pytest/job/execnet-sdist/lastSuccessfulBuild/artifact/.tox/dist/execnet-1.0.8a1.zip
>> >>
>> >> working with symlinks works for you?
>> >>
>> >> Also, i am still interested in a small failing test case for the other issue
>> >> where you could not use RSync later on.
>> >>
>> >> thanks & best,
>> >> holger
>> >>
>> >> On Mon, Jul 12, 2010 at 17:16 -0500, Charles Solar wrote:
>> >>> I'll make up a small test case but just so I do not forgot I found
>> >>> another bug dealing with symlinks.
>> >>> I think that all relative symlinks are broken, or else I do not
>> >>> understand how they could have worked before.
>> >>> I saw it be having a structure like this rsync:
>> >>>
>> >>> lib/
>> >>>    libnet.so -> libnet.so.1.0.0
>> >>>    libnet.so.1.0.0
>> >>>
>> >>> What it created on the remote side was
>> >>>
>> >>> lib/
>> >>>   libnet.so -> /tmp/rsynctemp/libnet.so.1.0.0
>> >>>   libnet.so.1.0.0
>> >>>
>> >>> The link on the remote side did not include the lib subdir in the
>> >>> link.  After looking at the code I fixed it up but I do not think they
>> >>> should be working at all since the code atm only works if the symlink
>> >>> is in the root dest directory.
>> >>>
>> >>> Ill send in a patch for you to look at.
>> >>>
>> >>> Charles
>> >>>
>> >>> On Sun, Jul 11, 2010 at 5:36 AM, holger krekel <holger at merlinux.eu> wrote:
>> >>> > Hi Charles,
>> >>> >
>> >>> > On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
>> >>> >> It would appear that RSync will only work with brand new gateways that
>> >>> >> have not had any other operations run on them.  Is this intentional?
>> >>> >
>> >>> > No.  Sounds like a bug.
>> >>> >
>> >>> >> For example, I do something like
>> >>> >>
>> >>> >> gw = execnet.makegateway( "ssh=foobar" )
>> >>> >> gw.remote_exec( mybootstrapper )
>> >>> >>
>> >>> >> ... then later I want to rsync ...
>> >>> >>
>> >>> >> rsync = execnet.RSync( "/tmp/foobar" )
>> >>> >> rsync.add_target( gw, "/tmp/barfoo" )
>> >>> >> rsync.send()
>> >>> >>
>> >>> >> I get remote EOFErrors
>> >>> >>
>> >>> >> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
>> >>> >>     self._end_of_channel(channel)
>> >>> >>   File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
>> >>> >>     channel.waitclose()
>> >>> >>   File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
>> >>> >>     raise error
>> >>> >> exceptions.EOFError: expected 1 bytes, got 0
>> >>> >>
>> >>> >> If I however create a brand new gateway to give to rsync it works fine.
>> >>> >>
>> >>> >> I would think that since rsync just creates a new channel on the
>> >>> >> gateway that it should work on existing 'old' gateways.
>> >>> >
>> >>> > yip. Myself i only used it for doing things at the beginning, it seems.
>> >>> > Do you have or could you try to find an repeatable minimal example?
>> >>> >
>> >>> > best,
>> >>> >
>> >>> > holger
>> >>> >
>> >>>
>> >>
>> >> --
>> >>
>> >
>>
>
> --
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fail.py
Type: text/x-python
Size: 340 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100805/ef74b5e5/attachment.py>