[AstroPy] Proliferating py-astro-libs

Mon Jun 13 15:11:13 EDT 2011

Hi all,

I'd be very interested in participating in a discussion of these
issues.  I am not an observer, but I am a member of the astrophysical
simulations community concerned with issues of interop, open science,
and developing community-focused packages for scientific analysis, and
I believe that  increasing cross-talk between the observation
community and the simulations community would be of great benefit to
all concerned.

I won't be at SciPy (haven't been able to make it since 2009,
unfortunately!) but I think this kind of discussion might warrant its
own mini-workshop.

-Matt

On Mon, Jun 13, 2011 at 12:07 PM, Tom Aldcroft
<aldcroft at head.cfa.harvard.edu> wrote:
> What about a splinter meeting at SciPy2011?  I guess the question is
> how many interested parties will NOT be there this year.
>
> - Tom
>
> On Mon, Jun 13, 2011 at 2:59 PM, Kelle Cruz <kellecruz at gmail.com> wrote:
>> I think a sit-down is desperately needed to resolve these issues, figure out
>> the mgmt structure (aka, pecking order), for the BDFL to emerge, and for
>> progress to occur.
>> I'd be happy to participate as a non-python/programming expert and maybe
>> provide the voice of the "users".
>> I propose there be a Splinter Meeting at AAS in Austin. (Splinter Meeting
>> deadline is Dec 1.) Or else someone will have to organize at CfA (Tom A?
>> Thom R?) or STScI (Marshall? Perry?) since, as far as I can tell, that seems
>> to be where most of the movers and shakers in this game are located.
>> kelle
>>
>> On Mon, Jun 13, 2011 at 2:00 PM, Perry Greenfield <perry at stsci.edu> wrote:
>>>
>>> That's a good idea.
>>>
>>> Perry
>>>
>>> On Jun 13, 2011, at 1:43 PM, James Turner wrote:
>>>
>>> > Maybe we should hold an AstroPy conference, where we can discuss
>>> > co-ordination, get to know each other better and even sit down and
>>> > work on libraries together (like at SciPy). That might help generate
>>> > a bit of momentum. Some of us have had meetings before that were
>>> > full of ideas that didn't go anywhere, but I don't think it has to
>>> > be that way if active people on the ground are talking to one another
>>> > rather than having institutions present their plans and try to
>>> > negotiate at a high level.
>>> >
>>> >
>>> > On 13/06/11 13:25, James Turner wrote:
>>> >> It seems that several of us would really like to improve
>>> >> collaboration on Python libraries but have been struggling to pull
>>> >> it off. I've raised the same issue on this list in past months, but
>>> >> my
>>> >> focus has unavoidably been on other things and since I'm wary of
>>> >> shouting a lot without contributing much, I haven't really been able
>>> >> to keep the discussion alive...
>>> >>
>>> >> I tend to agree with Mark and Stefan about the question of
>>> >> leadership.
>>> >> Perry & co. at Space Telescope deserve recognition for getting us
>>> >> this
>>> >> far with things like PyFITS and PyRAF. Others have taken the
>>> >> initiative
>>> >> with things like astronomical plotting and documentation sprints.
>>> >> We're
>>> >> still lacking a bit of coherence though, which (as Mark suggests) is
>>> >> likely to involve one or a few dedicated, energetic, knowledgeable,
>>> >> hands-on developer(s) who can glue things together. Those people need
>>> >> to be employed by someone, though, to ensure stability & continuity
>>> >> (fortunately there's already a bit of that going on at STScI, eg.
>>> >> with
>>> >> Mike and Matplotlib). Personally, I have the motivation but have not
>>> >> had the time/independence (and might not be assertive enough).
>>> >> Apparently we do have several energetic authors in the community now
>>> >> (like Thomas & Eli), but each with their own project.
>>> >>
>>> >> A couple of years ago, a number of us at the observatories
>>> >> submitted a
>>> >> white paper to the Decadal Survey, pointing out the need for more
>>> >> co-ordinated funding so that we can have people who focus on cross-
>>> >> institutional platform development & support. The report from the
>>> >> committee did give a nod to our concerns and their importance, but
>>> >> stopped short of making any recommendation, which basically means
>>> >> "good
>>> >> luck with that". Meanwhile, at Gemini we have had our own problems to
>>> >> deal with, which make it very difficult for me to propose something
>>> >> internally beyond working with STScI on the distribution of
>>> >> dependencies that Perry mentioned. Perhaps someone obtaining a grant
>>> >> for this is not out of the question though.
>>> >>
>>> >> I would like it if we could get together organically behind Astrolib,
>>> >> but sometimes it's difficult to get people away from their immediate
>>> >> priorities to focus on that, even within my own institution. If we
>>> >> could get people dedicated to it, though, it could become
>>> >> indispensable
>>> >> enough to attract and co-ordinate more effort. I'm just not sure
>>> >> how we
>>> >> get started at this point and my personal options for tackling the
>>> >> problem seem limited given the overarching funding transition at
>>> >> Gemini
>>> >> and the intense focus on projects that are needed to make that
>>> >> work...
>>> >>
>>> >> Cheers,
>>> >>
>>> >> James.
>>> >>
>>> >>
>>> >> On 10/06/11 09:48, Perry Greenfield wrote:
>>> >>>
>>> >>> On Jun 9, 2011, at 11:12 PM, Thomas Robitaille wrote:
>>> >>>
>>> >>>> I just wanted to also add that (in agreement with Marshall) I'm all
>>> >>>> in favor of many small modules that accomplish a particular task
>>> >>>> well, rather than packages that aim for a 'do-it-all' approach and
>>> >>>> fall short. It's always possible to bundle small packages together
>>> >>>> afterwards, and I don't mean merge development, but instead just
>>> >>>> bundling the packages for installation (kind of like EPD). I think
>>> >>>> that is the easiest approach for all of us.
>>> >>>>
>>> >>>> Maybe in the long run, a specific set of core packages will emerge
>>> >>>> as essential and we can then talk about truly merging them into a
>>> >>>> scipy-like package, but for now, I think the race is still on. And
>>> >>>> after all, there's nothing to say we *have* to achieve the same
>>> >>>> setup as in IDL.
>>> >>>
>>> >>> It sure seems to me that the time is ripe to start trying to
>>> >>> coalesce
>>> >>> some of the ongoing efforts.
>>> >>>
>>> >>> Mind you, I don't think it is necessarily good to start with only
>>> >>> one
>>> >>> version. Allowing a few different approaches initially has its
>>> >>> benefits. You get to see more approaches and ideas in play and
>>> >>> having
>>> >>> experience with them is very helpful in deciding which one is more
>>> >>> productive. And sometimes there is room for more than one in the
>>> >>> long
>>> >>> run. The different approaches may have their own niches. But it is
>>> >>> hard to imagine any long-term need for more than two or three
>>> >>> different approaches.
>>> >>>
>>> >>> Early on there are some pragmatic needs for different approaches.
>>> >>> For
>>> >>> example, having a fairly "literal" translation of IDL tools into
>>> >>> Python has its benefit. It is very useful for those that would
>>> >>> like to
>>> >>> migrate IDL code, and given the existing IDL versions, make it much
>>> >>> easier to test their correctness. But I don't see this as a
>>> >>> substitute
>>> >>> for a good set of modular tools that have a better object design and
>>> >>> consistent interfaces with other modules. Doing this is more work
>>> >>> and
>>> >>> will take more time. So a need for both approaches is likely. Some
>>> >>> could argue the same for replacing IRAF tasks.
>>> >>>
>>> >>> All this is much easier said than done of course.
>>> >>>
>>> >>> I wish STScI had more resources to devote to this than we've
>>> >>> actually
>>> >>> had. We've been planning to do more on this front than we've
>>> >>> actually
>>> >>> done. Things come up repeatedly that ruin these plans. But it may be
>>> >>> worth saying where some of our current efforts are going that may
>>> >>> overlap some of these other efforts.
>>> >>>
>>> >>> 1) We've been planning (along with Gemini, and particularly James
>>> >>> Turner), to try to develop some Sage-like installation package that
>>> >>> would make it easy to install all the basic tools for most
>>> >>> astronomers. We had hoped to have a beta version out, but one of the
>>> >>> people working on this left at the end of last year, and we've not
>>> >>> been able to replace that person. We are going to continue this
>>> >>> effort
>>> >>> with existing staff though. Hopefully in a few months we'll have
>>> >>> something to try out.
>>> >>>
>>> >>> 2) There is a recognition that a more serious effort needs to be
>>> >>> made
>>> >>> to replace IRAF functionality. Perhaps one of the benefits of a JWST
>>> >>> delay is that it will allow us to do some of that work more
>>> >>> explicitly. But we would not do it by replacing IRAF tasks one-for-
>>> >>> one
>>> >>> but coming up with an entirely different approach which has to start
>>> >>> from the bottom up (the end result could have applications that
>>> >>> mostly
>>> >>> emulate IRAF tasks, but also provide much more modular tools).
>>> >>>
>>> >>> 3) More specifically, we are currently focussing on how to handle
>>> >>> WCS
>>> >>> issues in a more general way than they are handled in FITS. If there
>>> >>> is interest, perhaps we should say more about the intended approach.
>>> >>> This is particularly important for replacing spectrographic tools in
>>> >>> IRAF, and this is where we are starting our effort.
>>> >>>
>>> >>> 4) We need a way of saving these WCS models, and FITS is not the
>>> >>> way.
>>> >>> We are looking at an alternate data format, not just for WCS models,
>>> >>> but for data in general [gasp!]. Work has begun on this too.
>>> >>>
>>> >>> 5) A lot of our recent work has been on pysynphot and ETCs. We
>>> >>> plan on
>>> >>> making the computational part of our ETCs a released tool. But I'm
>>> >>> also wondering if we can generalize the pysynphot spectral models
>>> >>> for
>>> >>> more general use in spectral tools.
>>> >>>
>>> >>> 6) We have been working on a framework for making pipelines easier
>>> >>> to
>>> >>> build and configure. That won't be ready for at least a few months,
>>> >>> but could well be of general interest and use.
>>> >>>
>>> >>> But besides these things, I would like to see if we can't begin the
>>> >>> effort of narrowing some of the underlying libraries being used.
>>> >>> FITS
>>> >>> WCS is one obvious area that seems ripe for that.
>>> >>>
>>> >>> But the community ought to identify one or two areas that are of the
>>> >>> most interest in consolidating (let's start small). What should we
>>> >>> start with? Focus is important in making any progress in this area.
>>> >>>
>>> >>> Perry
>>> >>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> AstroPy mailing list
>>> >>> AstroPy at scipy.org
>>> >>> http://mail.scipy.org/mailman/listinfo/astropy
>>> >
>>> >
>>> > --
>>> > James E.H. Turner
>>> > Gemini Observatory Southern Operations Centre,
>>> > Casilla 603,          Tel. (+56) 51 205609
>>> > La Serena, Chile.     Fax. (+56) 51 205650
>>> > _______________________________________________
>>> > AstroPy mailing list
>>> > AstroPy at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>>
>> --
>> Kelle Cruz, PhD — http://kellecruz.com/
>> 917.725.1334 — Hunter ext: 16486 — AMNH ext: 3404
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>