[pypy-dev] Codespeed as on speed.pypy.org

[Christian Peel]

Sarah,

Thanks for the positive comments;  and again I should have been more clear
that we're thinking that the hosted service would be be free for
open-source users.  I guess in this case that you'd be more inclined to use
it; is that right?

Also, thank you so much for the other ideas and suggestions.   I like all
the plotting suggestions and your thoughts about flexibility.  If we make a
hosted service, there indeed are many issues with hardware that would need
to be addressed; we have considered comparing hardware architectures.

And yes, thanks for the pointer to the "Deploy to Heroku" button;  I agree
that for an open-source project this would be very useful.

I'm happy to talk with any of you further, but we should do it off the pypy
list.

Thanks again!

Chris

On Thu, Mar 12, 2015 at 2:29 AM, Sarah Mount <mount.sarah at gmail.com> wrote:

> Hi there,
>
> The twistedmatrix deployment is here BTW: http://speed.twistedmatrix.com/
>
> This is an interesting idea (and Codespeed is ace!). I have a few things
> that Codespeed could help with, although I'm, not sure I could justify
> paying for them, but certainly this is something that would interest me.
> One of my undergrads (Sam Giles) used it in his final year thesis and it
> worked very well for him.
>
> A few random thoughts for you:
>
> - To be more useful for academics Codespeed would need to generate more
> statistical information and much better quality charts. Statistical tests
> on improvements between versions, confidence intervals, etc. would be
> handy. This piece of work has some very nice ideas:
> https://github.com/jamesbornholt/plotty This would move Codespeed from
> being able to provide a basic "smoke test" for performance improvement, to
> something much more rigorous. I guess not everyone would necessarily need
> that, but I think it would not do anyone harm to have it.
>
> - Table output in LaTeX would also be useful for academic types, as would
> some way to make the charts more "configurable", or maybe just being able
> to dump the Python code (+ data) that creates the charts would do, then
> people could mess around offline. Another option would be to use
> https://plot.ly/ which gives you the code to generate each chart you
> create on its API (e.g. click on the "code" tab on the left of this:
> https://plot.ly/~snim2/43/ ). You could go a step further and have
> something like an "open my results in LaTeX / Overleaf [
> https://www.overleaf.com/devs ] or ShareLaTeX. The big thing here is
> workflow - how does the user go from pushing a changeset in their code to
> generating high quality analysis and output for a blog or paper publication
> or a business pitch? There are several ways to reduce friction there, you
> can probably think of more ideas than I have.
>
> - The style files could do with an update, which probably doesn't matter
> much to people on this list, but may matter more if you want to
> commercialise
>
> - For any user ease of use is really the key thing, but also flexibility.
> It is important to be able to get at the data in different formats, but
> people will probably find uses for Codespeed that you hadn't intended (and
> I guess not all future users would necessarily be full time software
> developers), and there are cases where the PyPy way of doing things isn't
> quite what you want. Here's a simple example use case: maybe instead of
> tracking the progress of some software project, I want to compare
> benchmarks for different pieces of hardware. When I push changes to the
> benchmark code and re-run the analysis I would not necessarily expect the
> results to change over time (although they might with updated firmware /
> SDKs, etc) but I might want to add more hardware platforms to the data and
> more benchmarks to the code. Codespeed doesn't quite match that use case at
> the moment, but I don't see why it shouldn't be possible to reconfigure it
> to be useful in that situation.
>
> - If you decide not to go down the commercial route, just having a "deploy
> to Heroku" button [ https://devcenter.heroku.com/articles/heroku-button ]
> might bring in many more users.
>
>
> Cheers,
>
> Sarah
>
>
>
> On Thu, Mar 12, 2015 at 7:37 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>
>> Hi Christian
>>
>> We as PyPy are not generally interested in freemium services (we are
>> poor to start with), however I think there is some potential with
>> codespeed being notoriously hard to deploy. I've seen
>> http://speed.pyston.org/ and http://speed.twistedmatrix.org/ (which is
>> under some URL, not clue where) deployed already and I do know about a
>> few commercial deployments.
>>
>> Cheers,
>> fijal
>>
>> On Thu, Mar 12, 2015 at 5:10 AM, Christian Peel <chris.peel at ieee.org>
>> wrote:
>> > Hi all,
>> >
>> > I'm working with the author of Codespeed (Miquel Torres) to determine
>> > whether there would be interest in a hosted  service which provided
>> > information similar to those from Codespeed as used at speed.pypy.org.
>> >
>> > Our belief is that pypy.org would be interested in using a freemium
>> hosted
>> > service with features similar to (or better than!) those in codespeed
>> and
>> > which was as easy to set up as Travis CI; can you confirm this?  What
>> > additional features would you want in such a service?    Are any of you
>> > using Codespeed commercially?
>> >
>> > My best
>> >
>> > Chris
>> >
>> > --
>> > chris.peel at ieee.org
>> >
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> Dr. Sarah Mount, Senior Lecturer, University of Wolverhampton
> website:  http://www.snim2.org/
> twitter: @snim2
>



-- 
chris.peel at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150312/950bdb5e/attachment-0001.html>