From opensource at ronnypfannschmidt.de Fri Apr 24 20:17:08 2015 From: opensource at ronnypfannschmidt.de (Ronn Pfannschmidt) Date: Fri, 24 Apr 2015 20:17:08 +0200 Subject: [execnet-dev] [discuss] Tersely reimagining execnet gateways for persistent services Message-ID: <553A88A4.4020503@ronnypfannschmidt.de> Hello everybody, a while back i had an idea for getting execnet into shape for running persistent services for that decided to imagine the gateway itself in a new shape currently a gatway is something we have that has * communication channels * code execution now, what if a gatway itself has *only* communication channels so in order to have code execution one would have to install a service to the gateway, now even installation of services into a gatway could be a service so details like * servce installation * remote code execution * file delivery * virtualenv management * gateway io startup could all be services, one could pick and choose for somemthing like a persistent process suddenly new kinds of setups are thinkable for example * a ssl socket serving gateway that can only manage virtualenvs, open sub io's, and tell the supervisord that manages its connect socket to restart it or everything * having a predefined ssh command for users that only provides a service for socket connections to certain inner hosts without exposing ssh functionality or other execnet features * having a deployent process service that synchronizes a set of wheels to the server farm and manages virtualenvs for the last N deployed versions of an application and their migration state, however only providing logs and metadata to the outside users, while not allowing any commands or further gateways in order to facilitate this change, execnet needs to grow a few extra cappabilities as a first step, io and services on top of the io need to be more cleanly separated * code execition needs to be a service * rsync should b a service * gateway via opening needs to be a service the nice detail about those changes is, that even though they would require a larger internal change, the external facade would look the same once the base is created, we could then experiment with creating of gateways, delivery of services to gateways, as well as meta information about services. i believe tool like batou could also be interested in similar setups with this mail i hope i can spark some ideas, provoke thoughts and receive some start initial critics -- Ronny From holger at merlinux.eu Sat Apr 25 11:35:02 2015 From: holger at merlinux.eu (holger krekel) Date: Sat, 25 Apr 2015 09:35:02 +0000 Subject: [execnet-dev] [discuss] Tersely reimagining execnet gateways for persistent services In-Reply-To: <553A88A4.4020503@ronnypfannschmidt.de> References: <553A88A4.4020503@ronnypfannschmidt.de> Message-ID: <20150425093502.GA15735@merlinux.eu> Hi Ronny, On Fri, Apr 24, 2015 at 20:17 +0200, Ronn Pfannschmidt wrote: > Hello everybody, > > a while back i had an idea for getting execnet into shape for running > persistent services > > for that decided to imagine the gateway itself in a new shape > > currently a gatway is something we have that has > > * communication channels > * code execution > > now, what if a gatway itself has *only* communication channels > so in order to have code execution one would have to > install a service to the gateway, Here you are beginning to loose me. The basic gateway.remote_exec(somecode) call is kind of an installation of a service. How do you want to break it down further? > now even installation of services into a gatway could be a service > > so details like > > * servce installation > * remote code execution > * file delivery > * virtualenv management > * gateway io startup > could all be services, one could pick and choose for somemthing like a > persistent process What exactly do you mean with "persistent" process? holger > suddenly new kinds of setups are thinkable > > for example > > * a ssl socket serving gateway that can only manage virtualenvs, open > sub io's, > and tell the supervisord that manages its connect socket to restart it > or everything > > * having a predefined ssh command for users that only provides a service > for socket connections to certain inner hosts without exposing ssh > functionality or other execnet features > > * having a deployent process service that synchronizes > a set of wheels to the server farm and manages virtualenvs > for the last N deployed versions of an application and their migration > state, > however only providing logs and metadata to the outside users, > while not allowing any commands or further gateways > > > in order to facilitate this change, execnet needs to grow a few extra > cappabilities > > as a first step, io and services on top of the io need to be more > cleanly separated > > * code execition needs to be a service > * rsync should b a service > * gateway via opening needs to be a service > > the nice detail about those changes is, that even though they would > require a larger internal change, the external facade would look the same > > once the base is created, we could then experiment with creating of > gateways, > delivery of services to gateways, as well as meta information about > services. > > i believe tool like batou could also be interested in similar setups > > > with this mail i hope i can spark some ideas, provoke thoughts and > receive some start initial critics > > -- Ronny > > _______________________________________________ > execnet-dev mailing list > execnet-dev at python.org > https://mail.python.org/mailman/listinfo/execnet-dev > -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu From opensource at ronnypfannschmidt.de Sat Apr 25 13:31:57 2015 From: opensource at ronnypfannschmidt.de (Ronn Pfannschmidt) Date: Sat, 25 Apr 2015 13:31:57 +0200 Subject: [execnet-dev] [discuss] Tersely reimagining execnet gateways for persistent services In-Reply-To: <20150425093502.GA15735@merlinux.eu> References: <553A88A4.4020503@ronnypfannschmidt.de> <20150425093502.GA15735@merlinux.eu> Message-ID: <553B7B2D.3040201@ronnypfannschmidt.de> On 04/25/2015 11:35 AM, holger krekel wrote: > Hi Ronny, > > On Fri, Apr 24, 2015 at 20:17 +0200, Ronn Pfannschmidt wrote: >> Hello everybody, >> >> a while back i had an idea for getting execnet into shape for running >> persistent services >> >> for that decided to imagine the gateway itself in a new shape >> >> currently a gatway is something we have that has >> >> * communication channels >> * code execution >> >> now, what if a gatway itself has *only* communication channels >> so in order to have code execution one would have to >> install a service to the gateway, > Here you are beginning to loose me. The basic gateway.remote_exec(somecode) > call is kind of an installation of a service. How do you want to break > it down further? for something that manages virtualenvs, files and an starts other gateways, it seems like a good idea, if remote_exec was not actually possible - simply to make it more reliablee my definition of service is basically something the remote side that listens to command and acts on them, remote_exec by that definition is already a service that provides executing code remotely - for the manager processes of a more persistent network hover code exection is something one would like to avoid > >> now even installation of services into a gatway could be a service >> >> so details like >> >> * servce installation >> * remote code execution >> * file delivery >> * virtualenv management >> * gateway io startup >> could all be services, one could pick and choose for somemthing like a >> persistent process > What exactly do you mean with "persistent" process? > > holger > >> suddenly new kinds of setups are thinkable >> >> for example >> >> * a ssl socket serving gateway that can only manage virtualenvs, open >> sub io's, >> and tell the supervisord that manages its connect socket to restart it >> or everything >> >> * having a predefined ssh command for users that only provides a service >> for socket connections to certain inner hosts without exposing ssh >> functionality or other execnet features >> >> * having a deployent process service that synchronizes >> a set of wheels to the server farm and manages virtualenvs >> for the last N deployed versions of an application and their migration >> state, >> however only providing logs and metadata to the outside users, >> while not allowing any commands or further gateways >> >> >> in order to facilitate this change, execnet needs to grow a few extra >> cappabilities >> >> as a first step, io and services on top of the io need to be more >> cleanly separated >> >> * code execition needs to be a service >> * rsync should b a service >> * gateway via opening needs to be a service >> >> the nice detail about those changes is, that even though they would >> require a larger internal change, the external facade would look the same >> >> once the base is created, we could then experiment with creating of >> gateways, >> delivery of services to gateways, as well as meta information about >> services. >> >> i believe tool like batou could also be interested in similar setups >> >> >> with this mail i hope i can spark some ideas, provoke thoughts and >> receive some start initial critics >> >> -- Ronny >> >> _______________________________________________ >> execnet-dev mailing list >> execnet-dev at python.org >> https://mail.python.org/mailman/listinfo/execnet-dev >>