[execnet-dev] RSync Issue

[Charles Solar]

Ah, that makes sense. However, twisted's reactor requires that its run
in the main app loop, not a new thread.  Is there a way to get my
remote_exec module to run in the main app thread and still get a
thread for serving rync data?

( I tried using  that code and twisted actually spit back
"exceptions.ValueError: signal only works in main thread" at me )
Charles

On Thu, Aug 5, 2010 at 1:31 PM, holger krekel <holger at merlinux.eu> wrote:
> Hey Charles,
>
> On Thu, Aug 05, 2010 at 12:46 -0500, Charles Solar wrote:
>> Oh my, it appears in my haste to get a fail test case I mis-remembered
>> what you were actually asking for.  I had completely forgotten about
>> the original rsync on an old gateway does not work problem I was
>> having.  That file was meant to demonstrate the symlink error :P
>
> np, so we can consider the rsync symlink issue solved i guess :)
>
>> Here is a python file that demonstrates the ORIGINAL rsync problem I
>> was having.  Turns out its not exactly 'an old gateway' like I said
>> earlier.  Its a gateway where a remote channel is still executing
>> code.  In my programs, I start a twisted reactor on the remote
>> machines, so the gateway's remote code never really 'completes.'  This
>> causes rsync to fail to use that gateway for the rsync channel.
>> In this demo file you can see this by first running it and watching it
>> freeze on rsync send.  Then, uncomment the chan.send and chan.receive
>> lines and it will work.
>
> It's expected behaviour.  If you do:
>
>    gw.remote_init_threads(num=2)
>
> just after creating the gateway you can have two concurrent executions
> and the rsync will work.
>
> best,
> holger
>
>> hopefully this attachment works.
>>
>> Charles
>>
>> On Thu, Aug 5, 2010 at 11:58 AM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi Charles,
>> >
>> > On Wed, Aug 04, 2010 at 15:34 -0500, Charles Solar wrote:
>> >> Ok I got that fail case for you, I uploaded it to rapidshare since my
>> >> last attachment did not work
>> >> http://rapidshare.com/files/411056579/rsync_fail.tar.gz.html
>> >
>> > with execnet-1.0.8 (containing the rsync fixes) this seems to work for me
>> > unless i am missing something?
>> >
>> >> Also, I am going to submit a change to you on bitbucket, I added a new
>> >> option for sshgateways to programatically use ssh options without
>> >> having to give a ssh config file.  I found sending an actual file was
>> >> not very platform independent with paths and all that.
>> >> I am using it to tell ssh to shutup about rsa identification like so
>> >> gw = execnet.makegateway(
>> >> "ssh=%s@%s//ssh_options=StrictHostKeyChecking=no;UserKnownHostsFile=/dev/null"
>> >> % ( host['user'], host['ip'] ))
>> >
>> > sounds good.  I wonder if it's also possible to append the option
>> > to the actual ssh=* spec. We'd need a safe separator but maybe
>> > ";" is reasonably safe so it could read:
>> >
>> >    ssh=user at host;StrictHostKeyChecking=no;UserKnownHostsFile=/dev/null
>> >
>> > looking forward to your patch :)
>> >
>> > best,
>> > holger
>> >
>> >>
>> >> Charles
>> >>
>> >> On Tue, Jul 27, 2010 at 5:14 PM, Charles Solar <charlessolar at gmail.com> wrote:
>> >> > Sorry could not get around to testing this out today, i will get this
>> >> > installed next time I have a chance to run my app.
>> >> > Oh yea, completely forgot about the test case, I will get a small
>> >> > package setup to illustrate too.
>> >> >
>> >> > Charles
>> >> >
>> >> > On Tue, Jul 27, 2010 at 12:47 PM, holger krekel <holger at merlinux.eu> wrote:
>> >> >> Hi Charles,
>> >> >>
>> >> >> so i think that rsyncing symlinks should now work reaonsably well.
>> >> >> I used your patches as a starting point but also added tests and made
>> >> >> sure permissions are also transferred for directories.  Could you verify
>> >> >> that after installing
>> >> >>
>> >> >>    http://hudson.testrun.org/view/pytest/job/execnet-sdist/lastSuccessfulBuild/artifact/.tox/dist/execnet-1.0.8a1.zip
>> >> >>
>> >> >> working with symlinks works for you?
>> >> >>
>> >> >> Also, i am still interested in a small failing test case for the other issue
>> >> >> where you could not use RSync later on.
>> >> >>
>> >> >> thanks & best,
>> >> >> holger
>> >> >>
>> >> >> On Mon, Jul 12, 2010 at 17:16 -0500, Charles Solar wrote:
>> >> >>> I'll make up a small test case but just so I do not forgot I found
>> >> >>> another bug dealing with symlinks.
>> >> >>> I think that all relative symlinks are broken, or else I do not
>> >> >>> understand how they could have worked before.
>> >> >>> I saw it be having a structure like this rsync:
>> >> >>>
>> >> >>> lib/
>> >> >>>    libnet.so -> libnet.so.1.0.0
>> >> >>>    libnet.so.1.0.0
>> >> >>>
>> >> >>> What it created on the remote side was
>> >> >>>
>> >> >>> lib/
>> >> >>>   libnet.so -> /tmp/rsynctemp/libnet.so.1.0.0
>> >> >>>   libnet.so.1.0.0
>> >> >>>
>> >> >>> The link on the remote side did not include the lib subdir in the
>> >> >>> link.  After looking at the code I fixed it up but I do not think they
>> >> >>> should be working at all since the code atm only works if the symlink
>> >> >>> is in the root dest directory.
>> >> >>>
>> >> >>> Ill send in a patch for you to look at.
>> >> >>>
>> >> >>> Charles
>> >> >>>
>> >> >>> On Sun, Jul 11, 2010 at 5:36 AM, holger krekel <holger at merlinux.eu> wrote:
>> >> >>> > Hi Charles,
>> >> >>> >
>> >> >>> > On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
>> >> >>> >> It would appear that RSync will only work with brand new gateways that
>> >> >>> >> have not had any other operations run on them.  Is this intentional?
>> >> >>> >
>> >> >>> > No.  Sounds like a bug.
>> >> >>> >
>> >> >>> >> For example, I do something like
>> >> >>> >>
>> >> >>> >> gw = execnet.makegateway( "ssh=foobar" )
>> >> >>> >> gw.remote_exec( mybootstrapper )
>> >> >>> >>
>> >> >>> >> ... then later I want to rsync ...
>> >> >>> >>
>> >> >>> >> rsync = execnet.RSync( "/tmp/foobar" )
>> >> >>> >> rsync.add_target( gw, "/tmp/barfoo" )
>> >> >>> >> rsync.send()
>> >> >>> >>
>> >> >>> >> I get remote EOFErrors
>> >> >>> >>
>> >> >>> >> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
>> >> >>> >>     self._end_of_channel(channel)
>> >> >>> >>   File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
>> >> >>> >>     channel.waitclose()
>> >> >>> >>   File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
>> >> >>> >>     raise error
>> >> >>> >> exceptions.EOFError: expected 1 bytes, got 0
>> >> >>> >>
>> >> >>> >> If I however create a brand new gateway to give to rsync it works fine.
>> >> >>> >>
>> >> >>> >> I would think that since rsync just creates a new channel on the
>> >> >>> >> gateway that it should work on existing 'old' gateways.
>> >> >>> >
>> >> >>> > yip. Myself i only used it for doing things at the beginning, it seems.
>> >> >>> > Do you have or could you try to find an repeatable minimal example?
>> >> >>> >
>> >> >>> > best,
>> >> >>> >
>> >> >>> > holger
>> >> >>> >
>> >> >>>
>> >> >>
>> >> >> --
>> >> >>
>> >> >
>> >>
>> >
>> > --
>> >
>
>> import execnet
>>
>> print "New rsync"
>> rsync = execnet.RSync( "." )
>> print "Making gateway"
>> gw = execnet.makegateway()
>> print "Execute echo"
>> chan=gw.remote_exec( "channel.send( channel.receive() )" )
>> #chan.send( "Echo" )
>> #assert chan.receive() == "Echo"
>> print "Add rsync target"
>> rsync.add_target( gw, "/tmp/rsync" )
>> print "Rsync send"
>> rsync.send()
>
>
> --
>