[OT] Fwd: [XP] Test run times needed

[Marco Bizzarri]

This is a little Off Topic, so, I ask forgiveness in advance.

I'm forwarding this message from Kent Beck from XP Mailing List. He's
asking for people to send execution times of unit tests. Even though
in the beginning of the mail he speaks about ant and xml formats,
actually, a simple text file with test name - execution time is more
than enough for doing some calculations.

Regards
Marco


---------- Forwarded message ----------
From: kentb <kentb at earthlink.net>
Date: Tue, Sep 9, 2008 at 1:53 AM
Subject: [XP] Test run times needed
To: extremeprogramming at yahoogroups.com


(Sorry for the cross post from JUnit.)

All,

Saff and I are working on ways in JUnit to shorten the validation phase of
the inner programming loop (feature -> test -> code -> validate). One of the
ideas is to find a general way to run suites faster by running tests in
parallel. To find effective parallelization strategies, we need data on test
run times. Rather than build big infrastructure to do this (which would
undoubtedly be cool), I'd like to start with the simplest thing that could
possibly work. So:

If you have a long-running JUnit test suite and
You run it using Ant and
You can use the XML formattter ("<formatter type="xml"/>") and
You don't mind sharing your test names with me confidentially

Would you please zip your reports and email them to me. If they're too big
for email, please let me know and we'll figure out a backup plan. I'd
appreciate any context you can provide--how long the suite has been in
development, the experience level of the developers, whatever else you think
we might need to know.

The first data set I looked at was from DevCreek-->90M test runs from
production coding representing more than 50 person-years of development. To
my surprise, the test runs exhibit a power law distribution (way lots of
fast tests, a few very long running tests, plot a histogram log-log and you
get a straight line). I have no idea what this means, but it brings to mind
the Asimov quote, "The most exciting phrase to hear in science, the one that
heralds new discoveries, is not 'Eureka!' (I found it!) but 'That's funny
...'" I've found many power law distributions in the static and dynamic
structure of code, but the mechanisms influencing test run times seem to be
completely different than those influencing code structure.

Anyway, I'd love to validate those findings. The Ant XML format seems like a
good place to start. Alternatively, you could send me one or more files with
test run times one per line. This would work for languages other than Java,
which would also be interesting to analyze.

Questions and comments appreciated.

Yours in science,

Kent Beck
Three Rivers Institute

__._,_.___
Messages in this topic (1) Reply (via web post) | Start a new topic
Messages | Files | Photos | Links | Database | Polls | Members
To Post a message, send it to:   extremeprogramming at eGroups.com

To Unsubscribe, send a blank message to:
extremeprogramming-unsubscribe at eGroups.com

ad-free courtesy of objectmentor.com
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch
format to Traditional
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe
Recent Activity

 27
New Members

Visit Your Group
Yahoo! Finance

It's Now Personal

Guides, news,

advice & more.

Wellness Spot

Embrace Change

Break the Yo-Yo

weight loss cycle.

Yahoo! Groups

Familyographer Zone

Learn how to capture

family moments.

.
__,_._,___


-- 
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/