[py-dev] Remote Distribution

[holger krekel]

Hi Dj,

On Fri, Sep 25, 2009 at 19:56 -0600, Dj Gilcrease wrote:
> I know py.test has distributed testing already but I have been using
> Testswarm (a distributed javascript testing framework) for a while now
> and thought the idea would fit nicely with py.test. This is just me
> kind of brain storming the idea, so feel free to ignore me if you have
> no interest in this idea.

that's just fine!  I am using the opportunity to add some brainstorming
and fiction on a next-generation execnet/py.test :) 

> Ok basic premise is:
>     Central server for anyone to schedule test runs on
>     Clients connect to the central server and run tests as they come
> in for any and all apps they meet the requirements for
>
> 
> Goals:
>     Eliminate the need for everyone to establish their own distributed
> testing network by establishing a remote distributed network where
> clients connect and will run tests from any app that is registered on
> the server.

i like the goal!

> Brainstorm of requirements:
>     Server - Needs a way for applications / libraries to register
>     Server - Apps need a way to set dependencies, and a way to specify
> version for that dependency
>     Server - Apps need a way to set the environments to test on (os
> version and python version)
>     Server - Apps need a way to schedule test runs and upload the code
> to test against
>     Server - Need a UI to display a test metric like
> http://testswarm.com/user/jquery/
>     Server - Needs a method of tracking available clients, and their
> environments
>     Server - Needs a way to send tests to every available client that
> meets the environment specs for each app.
>         Needs to keep track of the test run id for results
>     Server - Needs to accept results back
>         Needs to identify the run and clients environment id to make
> sure we can mark an environment finished for the run
>     Server - Needs a fault catcher to reschedule a test run /
> environment if a remote client disconnects before it finished
>     Server - Needs a way of removing code once all tests have been
> executed for a particular run, or after X days
>     Server - Needs to ensure remote execution of apps is not possible
> for security
>     Client - Needs a way to connect to the central server and keep it
> notified that it is still alive
>     Client - Needs a way to receive Tests
>     Client - Needs a way to report results back

makes sense.  I'd like to allow to configure some aspects of 
remote environments, i.e. set up virtualenvs etc. See below. 

> Brainstorm implementation details:
>     modified? py.execnet for the server test dispatcher
>     modified? py.execnet for the client test receiver
>     Client should be command line only so it can be run as a service
>     Web app for managing Apps and result display
>         REsT API for scheduling test runs and uploading the required code?
>         "py.test schedule" command?

I am currently pondering imperative schemes of setting up execnet-style
networks.  Triggered by your post i eventually wrote up some bits in
my blog, see http://tinyurl.com/yco9aua 

Applied to testing distribution, here is what i would like to have for 
remote testing conceptually: 

    py.test --tx platform=darwin//packages=django1.1,nose,...

- would *discover* a darwin host from the above execnet network
- create/use a fitting virtualenv with the given packages/versions
- run tests in a new python process in that virtualenv 
- log results back to the invoking console 

A rather decentral system so far with no reporting.  Something like 

    py.test --runweb codespeak.net:9999

could dynamically fire up a web server at the specified 
host which then receives test results and it could also fire up 
a local browser window allowing to interact with the test run. 

The network so far is rather isolated.  To build larger 
test networks there needs to be a way to "connect" two 
execnets.  Maybe to make it publically available a 
command line like this would be nice: 

    execnet publish mynet https://testrun.org/012983102938123

so that from another place people could issue 

    execnet subscribe https://testrun.org/012983102938123

and the two execnets would be connected (whatever that
means exactly). 

    execnet list 

could list my local execnet's hosts and links to other execnets. 
So one could uniformly work with a multitude of dynamic 
user-contributed execnets at the fingertip of 
command line tools.

i am happy about any feedback and already grateful for
the opportunity to clarify my thoughts and plans :) 

cheers,
holger