[Baypiggies] A talk on testing frameworks

[Seth Friedman]

One topic I've been thinking about and been excited but haven't yet had the
opportunity to implement very fully is using metaprogramming for distributed
grid style testing, with some of the "fit" model.   I'd love to hear someone
talk on implementing this more fully.

Like I've been thinking, if my goal is to execute 1000 tests on each of 8
platforms, thing "A" is to abstract all the general technical stuff involved
in cross platform testing, like hide the differences in command line args
from the test so test writer can say "give me a process listing that outputs
the count of a process" and the library pipes it to the right ps command for
the platform.  Then if test executors can be determined on the fly - say 5
workers get 200 tests each.

One place we had this mostly implemented, running off of google
spreadsheets-as-a-DB with Cog, i think. It would take all the parameters of
the "test" (spreadsheet; test suite is a row) and it would generate the
actual executed code, which has all the specific implementations.    With
these things and a couple other pieces of data-driven stuff the tester could
stay focused on the business logic of what they're trying to test rather
than get bogged down in the details of running a large battery of tests
across 40, substantially different machines.

So, i'm curious if anyone has experience doing stuff like this, sort of
applying the "fit" model of automated testing, to big scale testing
challenges?

seth

On Mon, Jan 17, 2011 at 3:12 PM, Minesh B. Amin <mamin at mbasciences.com>wrote:

> Hi Keith,
>
> It is amazing how many basic things need to be revisited
> in the context of parallelism!
>
> Basic thing about setuptools is the design philosophy of
> "single-version, externally managed". So, one has to
> manipulate "sys.path" to pick the correct version ...
> something that gets out of hand very quickly when, say,
> trying to run multiple versions in parallel.
>
> Basic thing about our solution is "multiple-versions,
> internally managed". "sys.path" would point to the root
> where multiple versions reside. The actual path is
> determined by the version picked, or inferred (in case
> of default behavior).
>
> In other words, "sys.path" should be short and sweet :)
> Also, the system should tell me as early as possible when
> packages I am interested in are not present -- a really
> important feature when dealing with multiple cluster/cloud
> environments.
>
> Regards,
> Minesh
>
> On Mon, 2011-01-17 at 00:19 -0800, Keith Dart wrote:
> > That would be nice. We came to the same conclusions as you. The
> > Distribute/setuptools does offer some of that functionality already.
> > Did you use that? If so a good talk on the use of setuptools would be
> > good.
> >
> >
> >
> > -- Keith Dart
> >
>
> --
> MBA Sciences, Inc (www.mbasciences.com)
> Tel #: 650-938-4306
> Email: mamin at mbasciences.com
>
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20110117/2142f10a/attachment.html>