From anthonysmith80 at gmail.com Thu Jul 1 05:29:10 2010 From: anthonysmith80 at gmail.com (Anthony Smith) Date: Thu, 1 Jul 2010 10:29:10 +0100 Subject: [AstroPy] IDL to Python Switchers Guide In-Reply-To: References: Message-ID: <328B502B-2A97-46BD-B1CC-85C5AE9C599F@gmail.com> Hi Kelle, This looks great - very useful page. I don't know whether it's worth adding to that particular page, but it's worth pointing out to would-be IDL-to-Python switchers that is it possible to make the transition gradually, as there are ways of linking Python to IDL. I don't think these are perfect, but they avoid things like "I would use Python but I absolutely must use someObscureFunction.pro so I'll stick with IDL". Here are a couple: http://www.cacr.caltech.edu/~mmckerns/software.html = pyIDL - haven't been able to install it myself, but looks great http://astronomy.sussex.ac.uk/~anthonys/pidly/ = pIDLy - my own routine, perhaps easier to set up than pyIDL, but can be slow/inefficient when processing a lot of data. ... and there are others out there. All the best, Anthony On 1 Jul 2010, at 01:18, Kelle Cruz wrote: > Hi All, > > I've sketched out a table on the AstroBetter wiki to use a > "Switchers Guide" from IDL. But it will also for developers so that > they can see what's already been migrated. > > http://www.astrobetter.com/wiki/tiki-index.php?page=Python+Switchers+Guide > > My idea is to list the IDL Astrolib routines, their equivalent/ > similar functionality python module, followed by a brief > description. The table currently includes the sum total of my > Python knowledge so it's up to the community to fill it in. I also > included in the table a couple IDL routines that probably have > Python equivalents but that I just haven't figured out yet. (i > couldn't quite figure out what to make of the Starlink package.) > Also, feel free to add modules that are useful but that don't have > an IDL equivalent. It's a wiki, so edit away. You'll just have to > register.... > > I spent some time writing out the syntax for the pyfits header > manipulation stuff, but if you just want to include the package and > module, that'll be better than nothing. I also based the sections > and their order on the Astrolib page: http://idlastro.gsfc.nasa.gov/contents.html > > btw, it is not my intention to duplicate the content that's already > on the NumPy for IDL users site: http://mathesaurus.sourceforge.net/idl-numpy.html > > Anyway, for you python gurus out there...next time you're > procrastinating but still want to do something productive, fill in > some of the table! > > kelle > > -- > Kelle Cruz, PhD > Assistant Professor, Hunter College > Research Associate, AMNH > Visiting Associate, Caltech > http://kellecruz.com/ > http://astrobetter.com/ > 424.645.1334 > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From jh at physics.ucf.edu Fri Jul 2 14:31:08 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 02 Jul 2010 14:31:08 -0400 Subject: [AstroPy] SciPy docs: volunteers needed now! Message-ID: Dear SciPy users and developers, For the past two summers, the SciPy Documentation Project has run a concerted effort to write docstrings for all the NumPy objects. This has been very successful, with over 75 writers contributing. Nearly every "important" object now has a reviewable docstring, and we finally have someone working on doc wiki programming again! We accomplished this by writing 3000-5000 words a week, as tabulated on the NumPy doc wiki's stats page. Sometimes we wrote much more. This worked because 1) many volunteered and 2) there was a paid coordinator who managed the workflow, motivated the community, and chased down loose ends. This year we have taken on the SciPy docs, which are bigger and more technical. A handful of very dedicated people has taken up the challenge, and David Goldsmith is once again our able coordinator. However, at the rate we are going it will take something like a decade to finish. We're writing just 700-1200 words a week. I have had to question whether it is worthwhile paying a coordinator for the effort of just a few volunteers, as these people are fairly focused already and there just are not very many of them. I would hate to pull support for the coordinator now, but in the end this project lives or dies by the willingness of its users - US - to contribute our time. Each of us should consider the cost - thousands of dollars a year for many of us with big projects - of commercial numerical software. Extracting (or even requesting) payment in exchange for the right to use SciPy isn't our model, but we still need the labor. In the end, all labor is either volunteer or paid. So, this is an appeal for all knowledgeable SciPy users, including current code developers, to consider writing a little bit of documentation over this summer. For those in a position to assign people to work on docs or to pay people who do, now is the time to step forward as well. Specifically, we need to get the number of words per week up into the 3000 range to make a paid coordinator worthwhile (and indeed, to get anywhere in a reasonable time). Short of a significant pickup in participation and words produced, documenting SciPy will once again be a 100% volunteer effort. Please visit http://docs.scipy.org/numpy/ to learn about editing docs, then sign up, request edit rights, and visit http://docs.scipy.org/scipy/Milestones/ to get connected to the work going on. We meet on Skype every Friday at noon EDT. Contact David Goldsmith (d.l.goldsmith at gmail.com, d.l.goldsmith on Skype) to join those conversations. Email discussion of doc issues happens on scipy-dev. Thanks, --jh-- From rahman at astro.utoronto.ca Sat Jul 3 11:57:07 2010 From: rahman at astro.utoronto.ca (Mubdi Rahman) Date: Sat, 3 Jul 2010 15:57:07 +0000 (GMT) Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C2BBBA7.5060006@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <115583.37382.qm@web29012.mail.ird.yahoo.com> Hi James and AstroPy colleagues, Thanks for the note - coordinating the astronomical python community has been a long time coming, and I'm glad that someone's taken the initiative to start this line of communication. Last year, a number of us located here in Toronto started pyAstroLib, with the goal of converting the NASA IDL Astronomy Library into Python. We roadmapped the course of doing an initial conversion of the entire library, with the exception of scripts that already have python equivalents (i.e. those provided by pyFits, et cetera). In the case of existing functionality, the documentation of pyAstroLib would point users in the direction of the tools that do the equivalent. We further roadmapped a plan to go beyond the structure of the IDL AstroLib (we were calling the direct python conversion of these libraries the "legacy" library), and developing a more structured, streamlined and less redundant general purpose astronomy library, so as to make entry into the world of astronomical python as painless as possible. Beyond this stage, we had a number of dream-goals, including a lightweight python-based replacement to DS9 with a more intuitive UI (think Adobe Photoshop), which could be used as a standalone package or embedded within other python scripts. We had a number of design requirements for the library: - The library was to be fully open-source (we chose the LGPL as our model). - The library was to require as few external libraries as necessary (we've limited ourselves to NumPy, SciPy, PyFits, and MatPlotLib). - The library was intended to be seamlessly cross-platform. Personally, I'm a Linux/Windows user, where as some of the others in our collaboration are pure Mac or pure Linux users. We realized that at some point, the project would merge with a number of others out there, and our goal was to have a bit of a code base before we started expanding and merging. Needless to say, we were ambitious. That was a year ago. In that time, we had started the conversion process - a large number of the general IDL astronomical functions have been converted (as one can pull from our git repo on sourceforge), complete with docstrings, but without extensive testing. But as is always the case, each of us individually became more concerned about the functionality that we needed for our research. For instance, much of my reliance on IDL has been due to the coordinate conversion functions - so I had just ported that part of wcslib into python. (The other half of my reliance on IDL has come from the pixel to wcs coordinate transformations and the image manipulation and alignment scripts, which I'm currently working on). We've also been receiving a lot of contributed code for a variety of tools and functions that other users have needed. Up till now, we haven't had a place to put this code. Now considering the path that this project has taken, we're moving in a different direction, which is to round up the contributed code, even if it doesn't actually fit in the IDL AstroLib framework, give it a common API, and just get people using it as is for now, and slowly categorize and refine the code in subsequent releases. That's basically where we are with pyAstroLib. I hope this clears up some of the confusion and helps coordinating effort. Mubdi Rahman on behalf of the pyAstroLib crew. ----- Original Message ---- > From: James Turner > To: AstroPy ; SciPy Users List > Sent: Wed, June 30, 2010 5:48:23 PM > Subject: Co-ordinating Python astronomy libraries? > > Dear Python users in astronomy, At SciPy 2009, I arranged an astronomy > BoF where we discussed the fact that there are now a number of astronomy > libraries for Python floating around and maybe it would be good to collect > more code into a single place. People seemed receptive to this idea and > weren't sure why it hasn't already happened, given that there has been an > Astrolib page at SciPy for some years now, with an associated SVN > repository: > >http://scipy.org/AstroLib After the meeting last August, I was > supposed to contact the mailing list and some library authors I had talked to > previously, to discuss this further. My apologies for taking 10 months to do > that! I did draft an email the day after the BoF, but then we ran into a > hurdle with setting up new committers to the AstroLib repository (which > has taken a lot longer than expected to resolve), so it seemed a bad time > to suggest that new people start using it. To discuss these issues > further, we'd like to encourage everyone to sign up for the AstroPy mailing > list if you are not already on it. The traffic is just a few messages per > month. > href="http://lists.astropy.scipy.org/mailman/listinfo/astropy" target=_blank > >http://lists.astropy.scipy.org/mailman/listinfo/astropy We (the 2009 > BoF group) would also like to hear on the list about why people have decided > to host their own astronomy library (eg. not being aware of the one at > SciPy). Are you interested in contributing to Astrolib? Do you have any other > comments or concerns about co-ordinating tools? Our motivation is to make > libraries easy to find and install, allow sharing code easily, help > rationalize available functionality and fill in what's missing. A > standard astronomy library with a single set of documentation should be > more coherent and easier to maintain. The idea is not to limit > authors' flexibility of take ownership of their code -- the > sub-packages can still be maintained by different people. If you're at > SciPy this week, Perry Greenfield and I would be happy to talk to you. If you > would like to add your existing library to Astrolib, please contact Perry > Greenfield or Mark Sienkiewicz at STScI for access (contact details at > href="http://scipy.org/AstroLib" target=_blank > >http://scipy.org/AstroLib). Note that the repository is being moved to a > new server this week, after which the URLs will be updated at > scipy.org. Thanks! James Turner (Gemini). Bcc: various > library authors From brandon at rhodesmill.org Sat Jul 3 18:24:26 2010 From: brandon at rhodesmill.org (Brandon Craig Rhodes) Date: Sat, 03 Jul 2010 18:24:26 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C2BBBA7.5060006@gemini.edu> (James Turner's message of "Wed, 30 Jun 2010 17:48:23 -0400") References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <87mxu8fbl1.fsf@asaph.rhodesmill.org> James Turner writes: > At SciPy 2009 ... we discussed the fact that there are now a number of > astronomy libraries for Python floating around and maybe it would be > good to collect more code into a single place. People seemed > receptive to this idea and weren't sure why it hasn't already > happened... Hey, James! Thanks, I've now joined the AstroPy mailing list. The reason that I have never really considered trying to merge PyEphem with any other astronomy package is one of architecture. The "libastro" library from inside of XEphem, which PyEphem tweaks and then places behind a Pythonic interface, is designed to use all of its own types (either C "doubles" or structs built from doubles) for inputs like dates, locations, and orbital elements, and outputs like coordinates. It is its own little universe, software-wise, and PyEphem remains easiest to keep working when I leave the XEphem code in a nearly pristine state. This advantage would be lost if I gutted the pure-C astronomy routines to read and write data to, say, NumPy arrays instead, or any other third-party data format. The fun of using PyEphem is the efficiency of asking a question, getting a date back, then being able to plug that date right into your next calculation because it's already a floating-point number of the sort that "libastro" uses internally for dates. Another reason that I have never had dialog with another project about merging or sharing code is that most of the astronomy packages that I see appear for Python (I do go browse the other astronomy packages on PyPI from time to time) are doing entirely different things than figuring out where Jupiter is on a given night: they're doing things like image processing, which are entirely different problems that use entirely different data structures. Even an orbital mechanics simulation would probably use quite different data structures (it would hopefully include momentum, for example!) from the RA/declination-based "structs" that lie behind the Jupiter(), Mars(), and other objects in PyEphem. That's not to say that I've never thought of trying to start over with a pure-Python astronomy library. I even, recently, thought through how I would improve upon the PyEphem object set if I had the chance. It would also be nice if routines were implemented in Python - right now, PyEphem can only be used with C-Python but not easily with IronPython; that would not be the case if everything were written in Python. (Though, the routines would obviously be much slower until C-Python gets a JIT compiler!) But to take these steps would be to essentially start over with a new project, and the point of PyEphem was quite different: to take a robust, already-working, mature library for computing planet and comet positions, and make it efficiently callable from Python. -- Brandon Craig Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon From jturner at gemini.edu Sat Jul 3 18:25:56 2010 From: jturner at gemini.edu (James Turner) Date: Sat, 3 Jul 2010 18:25:56 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <4C2FB8F4.7090506@gemini.edu> Thanks for all the replies to my message about co-ordinating libraries. I've been wrapped up in SciPy code sprints for the past couple of days and now I'm going on vacation for 3 weeks, but I'll have a closer look at your libraries and documents and post more as soon as I can. Maybe Perry and Mark will also have comments. It sounds like there's some nice work going on, both filling in functionality and documenting it. It will be great if we can get some of that into AstroLib or the SciPy pages, or at least link to it from there so people can track it. Thanks a lot :-). James. From erik.tollerud at gmail.com Sat Jul 3 19:14:01 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 3 Jul 2010 16:14:01 -0700 Subject: [AstroPy] [SciPy-User] Co-ordinating Python astronomy libraries? In-Reply-To: <4C2BBBA7.5060006@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> Message-ID: Dear AstroPy list and James, Thanks for your efforts to do this coordination - clearly there's quite a bit out there doing similar work. For the last couple years I (and a few others) have been working on Astropysics (http://packages.python.org/Astropysics/ and source code at https://launchpad.net/astropysics). This initially spun out of my personal need for some of the utilities these other projects discussed (e.g. IDL astron-like functions), but I've been turning it in a slightly different direction. Astropysics now is being written as a more "pythonic" way of doing the same tasks instead of starting from cloning IDL procedural techniques. I'm trying to leverage all the existing tools that have been written in python for other domains (e.g. Enthought Traits to easily make good GUIs, sphinx to make useful documentation, distribute to make packaging easier). Further, where useful, I use object-oriented techniques instead of the procedural approach people familiar with IDL and IRAF are used to. The idea is to start doing things on "Spectrum" and "CCDImage" objects instead of passing arrays into functions and so on, and each astronomer can then sub-class these classes to do whatever they want. The aim here is to eliminate the tendancy for people to take public codes and change them internally, leading to some very painful efforts in disentangling versions. So far, Astropysics includes the following modules: * constants ? physical constants and cosmological calculations * coords ? coordinate classes and coordinate conversions * models ? model and data-fitting classes and tools * objcat ? flexible, dynamically updated object catalog * ccd ? image processing tools * spec ? spectrum and SED classes and tools * phot ? photometry/flux measurement classes and tools * obstools ? miscellaneous tools for observation * io ? data input/output classes and tools * plotting ? astronomy-oriented matplotlib and mayavi plotting tools * pipeline ? classes for data reduction and analysis pipelines * utils ? utility classes and functions And these GUI tools, which require Enthought Traits (and Enthought chaco for plots in the GUI): * Spylot ? Spectrum Plotter * Fitgui ? Interactive Curve Fitting * Spectarget ? MOS Targeting I'm making a concerted effort to document everything in a consistent manner using sphinx (http://sphinx.pocoo.org) - the resulting documents (http://packages.python.org/Astropysics/) end up being much more useful. I also try to bind in python wrappers around external tools like sextractor, Mike Blanton's kcorrect, MOS mask design tools, and galfit (planned) ... this happens mostly as I need them for my own science. But this cuts down on the time re-implementing standard programs that aren't necessarily worth re-working. There are two main things left before I consider it "releasable" in terms of the baseline functionality: First, it needs WCS support to be integrated in the coords framework, and second, the objcat package should have a web server module that lets you show/edit the catalog over the internet instead of within python. Another major goal I'd like to do when I or someone else gets a chance is something like ATV (http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI framework that the other gui tools are written in. I hope this explains the direction I have in mind for astropysics. As far as I can tell, I have a slightly different philosophy - I'm trying to set up something of a framework of my design, rather than a function library. This is why I have not been working on adding it to astropy, because astropy seems much more like the traditional library... That and I'm not a fan of the Trac project management system. At any rate, I think there's definitely room for all in the community. And if anyone likes what they see, feel free to drop me a line if you want to contribute. -- Erik Tollerud Graduate Student Center For Cosmology University of California, Irvine http://ps.uci.edu/~etolleru On Wed, Jun 30, 2010 at 2:48 PM, James Turner wrote: > Dear Python users in astronomy, > > At SciPy 2009, I arranged an astronomy BoF where we discussed the > fact that there are now a number of astronomy libraries for Python > floating around and maybe it would be good to collect more code into > a single place. People seemed receptive to this idea and weren't sure > why it hasn't already happened, given that there has been an Astrolib > page at SciPy for some years now, with an associated SVN repository: > > ? http://scipy.org/AstroLib > > After the meeting last August, I was supposed to contact the mailing > list and some library authors I had talked to previously, to discuss > this further. My apologies for taking 10 months to do that! I did > draft an email the day after the BoF, but then we ran into a hurdle > with setting up new committers to the AstroLib repository (which has > taken a lot longer than expected to resolve), so it seemed a bad > time to suggest that new people start using it. > > To discuss these issues further, we'd like to encourage everyone to > sign up for the AstroPy mailing list if you are not already on it. > The traffic is just a few messages per month. > > ? http://lists.astropy.scipy.org/mailman/listinfo/astropy > > We (the 2009 BoF group) would also like to hear on the list about > why people have decided to host their own astronomy library (eg. not > being aware of the one at SciPy). Are you interested in contributing > to Astrolib? Do you have any other comments or concerns about > co-ordinating tools? Our motivation is to make libraries easy to > find and install, allow sharing code easily, help rationalize > available functionality and fill in what's missing. A standard > astronomy library with a single set of documentation should be more > coherent and easier to maintain. The idea is not to limit authors' > flexibility of take ownership of their code -- the sub-packages > can still be maintained by different people. > > If you're at SciPy this week, Perry Greenfield and I would be happy > to talk to you. If you would like to add your existing library to > Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at > STScI for access (contact details at http://scipy.org/AstroLib). > Note that the repository is being moved to a new server this week, > after which the URLs will be updated at scipy.org. > > Thanks! > > James Turner (Gemini). > > Bcc: various library authors > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From cohen at lpta.in2p3.fr Sun Jul 4 03:30:14 2010 From: cohen at lpta.in2p3.fr (Johann Cohen-Tanugi) Date: Sun, 04 Jul 2010 09:30:14 +0200 Subject: [AstroPy] [SciPy-User] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <4C303886.6040206@lpta.in2p3.fr> Hi Erik, this is completely independent from the STS software, that shares some modules' name with you : https://www.stsci.edu/trac/ssb/astrolib ? Johann On 07/04/2010 01:14 AM, Erik Tollerud wrote: > Dear AstroPy list and James, > > Thanks for your efforts to do this coordination - clearly there's > quite a bit out there doing similar work. > > For the last couple years I (and a few others) have been working on > Astropysics (http://packages.python.org/Astropysics/ and source code > at https://launchpad.net/astropysics). This initially spun out of my > personal need for some of the utilities these other projects discussed > (e.g. IDL astron-like functions), but I've been turning it in a > slightly different direction. Astropysics now is being written as a > more "pythonic" way of doing the same tasks instead of starting from > cloning IDL procedural techniques. I'm trying to leverage all the > existing tools that have been written in python for other domains > (e.g. Enthought Traits to easily make good GUIs, sphinx to make useful > documentation, distribute to make packaging easier). Further, where > useful, I use object-oriented techniques instead of the procedural > approach people familiar with IDL and IRAF are used to. The idea is > to start doing things on "Spectrum" and "CCDImage" objects instead of > passing arrays into functions and so on, and each astronomer can then > sub-class these classes to do whatever they want. The aim here is to > eliminate the tendancy for people to take public codes and change them > internally, leading to some very painful efforts in disentangling > versions. > > So far, Astropysics includes the following modules: > * constants ? physical constants and cosmological calculations > * coords ? coordinate classes and coordinate conversions > * models ? model and data-fitting classes and tools > * objcat ? flexible, dynamically updated object catalog > * ccd ? image processing tools > * spec ? spectrum and SED classes and tools > * phot ? photometry/flux measurement classes and tools > * obstools ? miscellaneous tools for observation > * io ? data input/output classes and tools > * plotting ? astronomy-oriented matplotlib and mayavi plotting tools > * pipeline ? classes for data reduction and analysis pipelines > * utils ? utility classes and functions > > And these GUI tools, which require Enthought Traits (and Enthought > chaco for plots in the GUI): > * Spylot ? Spectrum Plotter > * Fitgui ? Interactive Curve Fitting > * Spectarget ? MOS Targeting > > I'm making a concerted effort to document everything in a consistent > manner using sphinx (http://sphinx.pocoo.org) - the resulting > documents (http://packages.python.org/Astropysics/) end up being much > more useful. > > I also try to bind in python wrappers around external tools like > sextractor, Mike Blanton's kcorrect, MOS mask design tools, and > galfit (planned) ... this happens mostly as I need them for my own > science. But this cuts down on the time re-implementing standard > programs that aren't necessarily worth re-working. > > There are two main things left before I consider it "releasable" in > terms of the baseline functionality: First, it needs WCS support to be > integrated in the coords framework, and second, the objcat package > should have a web server module that lets you show/edit the catalog > over the internet instead of within python. Another major goal I'd > like to do when I or someone else gets a chance is something like ATV > (http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI > framework that the other gui tools are written in. > > I hope this explains the direction I have in mind for astropysics. As > far as I can tell, I have a slightly different philosophy - I'm trying > to set up something of a framework of my design, rather than a > function library. This is why I have not been working on adding it to > astropy, because astropy seems much more like the traditional > library... That and I'm not a fan of the Trac project management > system. At any rate, I think there's definitely room for all in the > community. And if anyone likes what they see, feel free to drop me a > line if you want to contribute. > > From erik.tollerud at gmail.com Tue Jul 6 11:15:16 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Tue, 6 Jul 2010 08:15:16 -0700 Subject: [AstroPy] [SciPy-User] Co-ordinating Python astronomy libraries? In-Reply-To: <4C303886.6040206@lpta.in2p3.fr> References: <4C2BBBA7.5060006@gemini.edu> <4C303886.6040206@lpta.in2p3.fr> Message-ID: Well, the only module name I see that's the same is "coords" and as I'm sure you can imagine that's a pretty common name when dealing with astronomy. There's no namespace collision, though, because all of the astropysics modules are modules within the astropysics package, while the STSci stuff are each separate packages. So you never directly "import coords" when using astropysics - instead you "import astropysics.coords" or "from astropysics import coords" . On Sun, Jul 4, 2010 at 12:30 AM, Johann Cohen-Tanugi wrote: > Hi Erik, this is completely independent from the STS software, that shares > some modules' name with you : https://www.stsci.edu/trac/ssb/astrolib ?? > Johann > > On 07/04/2010 01:14 AM, Erik Tollerud wrote: >> >> Dear AstroPy list and James, >> >> Thanks for your efforts to do this coordination - clearly there's >> quite a bit out there doing similar work. >> >> For the last couple years I (and a few others) have been working on >> Astropysics (http://packages.python.org/Astropysics/ and source code >> at https://launchpad.net/astropysics). ?This initially spun out of my >> personal need for some of the utilities these other projects discussed >> (e.g. IDL astron-like functions), but I've been turning it in a >> slightly different direction. ?Astropysics now is being written as a >> more "pythonic" way of doing the same tasks instead of starting from >> cloning IDL procedural techniques. ?I'm trying to leverage all the >> existing tools that have been written in python for other domains >> (e.g. Enthought Traits to easily make good GUIs, sphinx to make useful >> documentation, distribute to make packaging easier). ?Further, where >> useful, I use object-oriented techniques instead of the procedural >> approach people familiar with IDL and IRAF are used to. ?The idea is >> to start doing things on "Spectrum" and "CCDImage" objects instead of >> passing arrays into functions and so on, and each astronomer can then >> sub-class these classes to do whatever they want. ?The aim here is to >> eliminate the tendancy for people to take public codes and change them >> internally, leading to some very painful efforts in disentangling >> versions. >> >> So far, Astropysics includes the following modules: >> * constants ? physical constants and cosmological calculations >> * coords ? coordinate classes and coordinate conversions >> * models ? model and data-fitting classes and tools >> * objcat ? flexible, dynamically updated object catalog >> * ccd ? image processing tools >> * spec ? spectrum and SED classes and tools >> * phot ? photometry/flux measurement classes and tools >> * obstools ? miscellaneous tools for observation >> * io ? data input/output classes and tools >> * plotting ? astronomy-oriented matplotlib and mayavi plotting tools >> * pipeline ? classes for data reduction and analysis pipelines >> * utils ? utility classes and functions >> >> And these GUI tools, which require Enthought Traits (and Enthought >> chaco for plots in the GUI): >> * Spylot ? Spectrum Plotter >> * Fitgui ? Interactive Curve Fitting >> * Spectarget ? MOS Targeting >> >> I'm making a concerted effort to document everything in a consistent >> manner using sphinx (http://sphinx.pocoo.org) - the resulting >> documents (http://packages.python.org/Astropysics/) end up being much >> more useful. >> >> I also try to bind in python wrappers around external tools like >> sextractor, Mike Blanton's kcorrect, MOS mask design tools, and >> galfit (planned) ... this happens mostly as I need them for my own >> science. ?But this cuts down on the time re-implementing standard >> programs that aren't necessarily worth re-working. >> >> There are two main things left before I consider it "releasable" in >> terms of the baseline functionality: First, it needs WCS support to be >> integrated in the coords framework, and second, the objcat package >> should have a web server module that lets you show/edit the catalog >> over the internet instead of within python. ?Another major goal I'd >> like to do when I or someone else gets a chance is something like ATV >> (http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI >> framework that the other gui tools are written in. >> >> I hope this explains the direction I have in mind for astropysics. ?As >> far as I can tell, I have a slightly different philosophy - I'm trying >> to set up something of a framework of my design, rather than a >> function library. ?This is why I have not been working on adding it to >> astropy, because astropy seems much more like the traditional >> library... That and I'm not a fan of the Trac project management >> system. ?At any rate, I think there's definitely room for all in the >> community. ?And if anyone likes what they see, feel free to drop me a >> line if you want to contribute. >> >> > From aarchiba at physics.mcgill.ca Tue Jul 6 12:39:14 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Tue, 6 Jul 2010 12:39:14 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C2FB8F4.7090506@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> <4C2FB8F4.7090506@gemini.edu> Message-ID: Hi, I use a couple of python astronomy libraries; the python bindings to the pulsar search library PRESTO, and the python bindings to the FORTRAN library SLALIB. Both were written by Scott Ransom; obviously I can't speak for his motivations in writing them. I think I follow a fairly common pattern, though: the libraries I use are fairly thin wrappers of existing compiled code, and in one case the code is very domain-specific. So combining them somehow with other python libraries would involve either linking together a huge mass of different compiled libraries with overlapping functionality, or modifying the underlying libraries. Either one would be a tremendous amount of work. That said, I use SLALIB for a simple coordinate conversion (in fact I use it to replicate exactly another coordinate conversion done in compiled code, so this is kind of theoretical), so in principle if there were a generic python astronomy library that covered this (I didn't find one) I might have used it instead. The PRESTO code, not so much; it includes some extremely specific pulsar searching code that can't reasonably be included in existing python libraries. What might be nice would be a "standard" list of data types for common objects, so that at least one could pass objects from one library to another. RA and Dec objects, perhaps, or time objects. Of course time objects would be a massive headache if you wanted them to be generically usable, since they'd need to be good to the nanosecond over a span of millenia (so doubles won't cut it), support UTC/TDB/TAI/etc. as well as MJD/sidereal/GMT/local time representations... Nevertheless, compiled code (and thin wrappers) tend to just take doubles and rely on the user to keep track of what they mean, while a more pythonic and less error-prone approach would use python's type system to catch many errors. Anne On 3 July 2010 18:25, James Turner wrote: > Thanks for all the replies to my message about co-ordinating libraries. > I've been wrapped up in SciPy code sprints for the past couple of days > and now I'm going on vacation for 3 weeks, but I'll have a closer look > at your libraries and documents and post more as soon as I can. Maybe > Perry and Mark will also have comments. > > It sounds like there's some nice work going on, both filling in > functionality and documenting it. It will be great if we can get some > of that into AstroLib or the SciPy pages, or at least link to it from > there so people can track it. > > Thanks a lot :-). > > James. > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From thomas.robitaille at gmail.com Tue Jul 6 14:36:22 2010 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 6 Jul 2010 14:36:22 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C2BBBA7.5060006@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <735692AF-E60A-439E-9E9C-C858EA28DB57@gmail.com> Hi all, Unlike several people who have replied so far, I have not been involved in general purpose astronomy libraries, but rather a few small packages, including: - APLpy (FITS image plotting built on matplotlib) at http://aplpy.sourceforge.net (co-developed with Eli Bressert) - ATpy (Seamless multi-format table handler) at http://atpy.sourceforge.net (also co-developed with Eli Bressert) - IDLSave (To read IDL save files into python) at http://idlsave.sourceforge.net - python-montage (Montage wrapper) at http://python-montage.sourceforgenet The main reason for keeping these separate rather than group them into a single package was that each of these packages accomplishes a well-defined task, and we were not prepared to develop all the other tools needed in a general-purpose library. However, I think that from a user point of view, it would be nice to have something more unified. The main point I want to make is that we need to distinguish between merging all development into a single repository, and bundling packages. There are cases where merging the development of packages does not make sense. For example, in the case of IDLSave, the module was originally developed for the astronomy community, but in fact can be used by any scientist that uses IDL. By developing it as part of a general astronomy libraries makes it less likely that non-astronomers will find it and use it. Another example is Tom Aldcroft's asciitable module (http://cxc.harvard.edu/contrib/asciitable/). This was developed to read in ASCII tables, but was not actually designed in an astronomy specific ways, since ASCII tables are of course not limited to astronomy. In the case of such a package, developing it as part of a general astronomy library would be detrimental, but bundling it as part of a general package might be desirable. So here is what I personally see as the ideal (hierarchical) setup: 1. A core library of essential routines, which handle for example FITS I/O, VO tools, WCS, coordinate transformations, etc. These could be held on a common development server. This would be a kind of 'numpy' or 'scipy' for astronomy, e.g. 'astropy' or 'astrocore'. Documentation for these would be merged and unified. 2. An astronomy 'bundle' which would include these essential routines, as well as extra astronomy packages (e.g. asciitable, IDLSave, ATpy, etc.). This bundle could be installable on top of the system python, or the enthought python distribution, e.g. 'astrolab'. Documentation for the extra packages would be left to the developers, but could be required to be of a consistent style (e.g. Sphinx) for inclusion in the bundle. 3. An 'all-in-one' package that would also include a full Python distribution, with numpy, scipy, matplotlib, etc - essentially an alternative to EPD for astronomy, e.g. 'APD (Astronomy Python Distribution)'. It could even come with a custom ipython prompt with all packges pre-loaded. All dependencies would be included. This would then cater to the different levels of python users in Astronomy. One quick note is that if we follow this or a similar model of bundling some packages without merging development repositories, is that developers of small packages will need to be more careful what license they release their code under, to ensure they can be bundled and re-released (but this should be trivial). Cheers, Thomas Robitaille On Jun 30, 2010, at 5:48 PM, James Turner wrote: > Dear Python users in astronomy, > > At SciPy 2009, I arranged an astronomy BoF where we discussed the > fact that there are now a number of astronomy libraries for Python > floating around and maybe it would be good to collect more code into > a single place. People seemed receptive to this idea and weren't sure > why it hasn't already happened, given that there has been an Astrolib > page at SciPy for some years now, with an associated SVN repository: > > http://scipy.org/AstroLib > > After the meeting last August, I was supposed to contact the mailing > list and some library authors I had talked to previously, to discuss > this further. My apologies for taking 10 months to do that! I did > draft an email the day after the BoF, but then we ran into a hurdle > with setting up new committers to the AstroLib repository (which has > taken a lot longer than expected to resolve), so it seemed a bad > time to suggest that new people start using it. > > To discuss these issues further, we'd like to encourage everyone to > sign up for the AstroPy mailing list if you are not already on it. > The traffic is just a few messages per month. > > http://lists.astropy.scipy.org/mailman/listinfo/astropy > > We (the 2009 BoF group) would also like to hear on the list about > why people have decided to host their own astronomy library (eg. not > being aware of the one at SciPy). Are you interested in contributing > to Astrolib? Do you have any other comments or concerns about > co-ordinating tools? Our motivation is to make libraries easy to > find and install, allow sharing code easily, help rationalize > available functionality and fill in what's missing. A standard > astronomy library with a single set of documentation should be more > coherent and easier to maintain. The idea is not to limit authors' > flexibility of take ownership of their code -- the sub-packages > can still be maintained by different people. > > If you're at SciPy this week, Perry Greenfield and I would be happy > to talk to you. If you would like to add your existing library to > Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at > STScI for access (contact details at http://scipy.org/AstroLib). > Note that the repository is being moved to a new server this week, > after which the URLs will be updated at scipy.org. > > Thanks! > > James Turner (Gemini). > > Bcc: various library authors > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Tue Jul 6 16:32:25 2010 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 6 Jul 2010 16:32:25 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <735692AF-E60A-439E-9E9C-C858EA28DB57@gmail.com> References: <4C2BBBA7.5060006@gemini.edu> <735692AF-E60A-439E-9E9C-C858EA28DB57@gmail.com> Message-ID: <9D3556DE-43C3-4047-B25F-CA6F67F07EC7@stsci.edu> On Jul 6, 2010, at 2:36 PM, Thomas Robitaille wrote: > Hi all, > > Unlike several people who have replied so far, I have not been > involved in general purpose astronomy libraries, but rather a few > small packages, including: > > - APLpy (FITS image plotting built on matplotlib) at http://aplpy.sourceforge.net > (co-developed with Eli Bressert) > - ATpy (Seamless multi-format table handler) at http://atpy.sourceforge.net > (also co-developed with Eli Bressert) > - IDLSave (To read IDL save files into python) at http://idlsave.sourceforge.net > - python-montage (Montage wrapper) at http://python-montage.sourceforgenet > > The main reason for keeping these separate rather than group them > into a single package was that each of these packages accomplishes a > well-defined task, and we were not prepared to develop all the other > tools needed in a general-purpose library. However, I think that > from a user point of view, it would be nice to have something more > unified. > > The main point I want to make is that we need to distinguish between > merging all development into a single repository, and bundling > packages. There are cases where merging the development of packages > does not make sense. For example, in the case of IDLSave, the module > was originally developed for the astronomy community, but in fact > can be used by any scientist that uses IDL. By developing it as part > of a general astronomy libraries makes it less likely that non- > astronomers will find it and use it. Another example is Tom > Aldcroft's asciitable module (http://cxc.harvard.edu/contrib/asciitable/ > ). This was developed to read in ASCII tables, but was not actually > designed in an astronomy specific ways, since ASCII tables are of > course not limited to astronomy. In the case of such a package, > developing it as part of a general astronomy library would be > detrimental, but bundling it as part of a general package might be > desirable. > In principle yes, some of these things are generic. But having libraries in a common repository doesn't preclude distributing them separately from astronomy. I also tend to thing that premature fragmentation is as bad a problem as being way too monolithic. I'd tend to wait until it was clear we had too much stuff in one repository before worrying about how to split things up. Sometimes it never becomes an issue. If some items grow non-astronomical contributors, they can go to scipy (or something similar). There are some advantages to a common repository: 1) We become more aware of areas of commonality and that foster greater sense of making things work better together. 2) Eventually it would help make for more consistent documentation and style, and ways of integrating the documentation. 3) Making the process of integrating common or redundant functionality easier later (I'll say a little about that in a separate email) And there are downsides of course: 1) Not everyone wants to use the same version control software (svn, git, hg...) or wiki, or issue tracker. 2) Coordination with other developers and projects is more work than doing things on your own. [...] > 3. An 'all-in-one' package that would also include a full Python > distribution, with numpy, scipy, matplotlib, etc - essentially an > alternative to EPD for astronomy, e.g. 'APD (Astronomy Python > Distribution)'. It could even come with a custom ipython prompt with > all packges pre-loaded. All dependencies would be included. > In this area STScI and Gemini are attempting to do something like this now as a joint project. We are just starting on this effort, but we hope to make a fairly easy to install distribution that includes most core tools astronomers would need (or something that would be easy to install optional items on). But we don't want to talk too much about this until we have something to have people try (I think Gemini would like to have something basic by the end of the year). We intend to include IRAF and the common external IRAF packages as well. > This would then cater to the different levels of python users in > Astronomy. One quick note is that if we follow this or a similar > model of bundling some packages without merging development > repositories, is that developers of small packages will need to be > more careful what license they release their code under, to ensure > they can be bundled and re-released (but this should be trivial). > Yes, licensing is one of the important issues to deal with... Perry From perry at stsci.edu Tue Jul 6 17:03:47 2010 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 6 Jul 2010 17:03:47 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <87mxu8fbl1.fsf@asaph.rhodesmill.org> References: <4C2BBBA7.5060006@gemini.edu> <87mxu8fbl1.fsf@asaph.rhodesmill.org> Message-ID: I'm heartened by all the responses so far in seeingg that there is interest at some level in the topic. We don't expect magic to happen and suddenly see coordination happen painlessly, nor simultaneously on all fronts. I (I'm speaking for myself on this) don't think that it is bad that there are multiple approaches tried for various things. Multiple approaches often show good ideas in each approach that others can learn from. Nevertheless, I can't but feel that in some areas the duplication of effort has gone too far. One example would be for FITS WCS modules. It would be good to understand if we could merge some of these efforts into a smaller number, or at least attempt to. There seem to be multiple parallel efforts to replicate the IDL astron library that may benefit from sharing of work and coordination. At this point it seems clear sphinx is going to be the documentation standard, and we might as all well begin using it or converting to it (we at STScI are in the process of doing that). We will be setting up the public astrolib repository this week (and I'll send an email when it is available). Are there any volunteers interested in migrating to it? It probably would be good only to have 1 or 2 to start with to see what issues arise. For those reluctant, we are curious about what the main obstacles are. E.g., 1) SVN? (what do you use?) 2) trac? (what do you use?) 3) security? (e.g., having other developers having the ability to commit changes to your code) 4) government regulations or laws (e.g., I have heard some are prevented from using some US hosts because of US laws) 5) worries about the safety of the code (we back up the repository nightly to disks at STScI) As to 3) I think it is reasonable to list those who are responsible for making decisions about what changes are accepted for any particular module (i.e., local control), and should anyone abuse that by repeatedly committing changes that aren't approved by the lead or leads on that module, they would lose their commit privileges. In other words, by moving your code to the astrolib repository, you aren't ceding control over changes to that code. Any other thoughts? Perry From jh at physics.ucf.edu Tue Jul 6 18:01:20 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Tue, 06 Jul 2010 18:01:20 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: (astropy-request@scipy.org) References: Message-ID: We've hit this topic (monolithic vs. Balkanized) several times over the past 5 years or so in various forms. Tom, you have hit many of the main issues in your posting (licenses, maintenance, management preferences, etc.). However, what you are contemplating doing at the end is unfortunately something that seems a disease: making Yet Another Specialized Python Distro(TM). There are already too many Sages and EPDs, and take it from one trying to get SciPy documented, managing a monolithic entity is a nightmare down the road when some of the packages go off maintenance, yet everyone depends on them. What I think we want from the users' perspective is an ability to say, *in the terms of their system's native package manager*, any of: give me this/these specific package(s) give me this/these group(s) of packages give me all the packages Then, we can publish metapackages that depend on groups of add-ons by topic (file format readers, coordinates/WCS/time, measurement extraction, spectral modeling, orbits, planets, galaxies, stars, etc.) and one that depends on all of the group packages, and viola, you can have Sage, EPD, or whatever fall out as a trivial consequence. What is critical here is something you did not mention in your posting, and that is solving the current difficulty of building a good Python stack from OS packages (e.g., debs), or even from package sources. The problem is that some of the packages do not play well with others. For example, HDF5 libraries were particularly problematic a year ago, and often compiled code complained because two different packages wanted different, specific versions of the same C or FORTRAN add-on library. Some of the code plain didn't work as advertized, or at all. In the end, it takes my very skilled system manager more than a week to do it, each time we do it, which is about once a year. The root of the problem is that there was no centralized build and test suite, nobody managing unified, integrated build testing and resolving the problems with the code maintainers. A year ago at SciPy'09, I pointed people to the build and test suite NSF requires for all their software projects. I think a few people looked at it at the time. Configured correctly (which takes work), it will build packages on umpteen different linuxes and a few other systems and package them in the native format. I think that the extended SciPy stack as a whole should be organized around such a system, but there seems to be little taste for it among the developer-heavy SciPy leadership. I think we can have a bit more practical vision, and at least for our own stuff (loosely defined) we should organize around principles that include these: - build and test everything frequently - manage the namespace so that all can play together - we really need to get everyone to agree to this or there will be conflicts, it's only a matter of time - produce LPUs (least packageable units) in binary format for all OSes - produce by topical meta-packages - produce one or more mega-packages that are equivalent to Sage or EPD - do it all for the native installers of all OSes - provide and enforce standards for docs and licenses - review the code - plan together and put out RFPs for needed codes - make sure procedures don't stifle innovators - document the whole, but lightly - provide a web community site for discussions, examples, reviews of code - locate and manage it so that it is owned by the community and survives long-term - ensure all jobs doable by at least 3 people - document procedures well - have formalized community governance and leadership - have a solid funding model - agree to hang together, even if you don't like something! With those (and perhaps other) goals in mind, we should then look at decisions like where/how to host it and what kind of wiki to use. Also, I keep thinking that this is best solved by joining forces with other scientific communities. The build-and-test part is hard, but once implemented, it scales fantastically. Again, all of SciPy should be doing this. We should at least build so that if others want to join, they have a place to fit in. This will need to be considered when doing package naming conventions. Generic names should be avoided so that two can play in the same sandbox. Even if we don't do the full thing from the start, we should plan it out and build as though that's where we're eventually going. AAS splinter meeting, anyone? --jh-- From mperrin at ucla.edu Tue Jul 6 19:55:30 2010 From: mperrin at ucla.edu (Marshall Perrin) Date: Tue, 6 Jul 2010 16:55:30 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: Message-ID: On Jul 6, 2010, at 3:01 PM, Joe Harrington wrote: > I think we can have a bit more > practical vision, and at least for our own stuff (loosely defined) we > should organize around principles that include these: > > - document the whole, but lightly > - provide a web community site for discussions, examples, reviews of code More than just documentation, what I would love to see is a set of tutorials or quick-start guides written at a roughly graduate-student level, that show a few examples of how to use all this code for common tasks. Perry and Robert's famous tutorial document (pydatatut.pdf) is an excellent start, but it's several years old now and misses all the new modules. I know it's a long way off until we have, say, an actual text in data analysis that we can point students to, but I think it's very important to keep new users (and IDL switchers, etc!) in mind when writing docs, not just other python gurus. For the web community site, there are already a few potential candidates (astropython.org, astrobetter.com, etc). My experience with these sorts of things in the past is that it takes a critical mass to keep a web community active and engaging. Seems like half the wikis I know get very little usage... So while everyone is thinking about monolithic vs. distributed package organization, it's probably worth considering that same question as applied to web sites too. - Marshall > - locate and manage it so that it is owned by the community and > survives long-term > - ensure all jobs doable by at least 3 people > - document procedures well > - have formalized community governance and leadership > - have a solid funding model > - agree to hang together, even if you don't like something! > > With those (and perhaps other) goals in mind, we should then look at > decisions like where/how to host it and what kind of wiki to use. > > Also, I keep thinking that this is best solved by joining forces with > other scientific communities. The build-and-test part is hard, but > once implemented, it scales fantastically. Again, all of SciPy should > be doing this. We should at least build so that if others want to > join, they have a place to fit in. This will need to be considered > when doing package naming conventions. Generic names should be > avoided so that two can play in the same sandbox. > > Even if we don't do the full thing from the start, we should plan it > out and build as though that's where we're eventually going. > > AAS splinter meeting, anyone? > > --jh-- > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From thomas.robitaille at gmail.com Tue Jul 6 23:15:30 2010 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 6 Jul 2010 23:15:30 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: Message-ID: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> > What I think we want from the users' perspective is an ability to say, > *in the terms of their system's native package manager*, any of: > > give me this/these specific package(s) > give me this/these group(s) of packages > give me all the packages > > Then, we can publish metapackages that depend on groups of add-ons by > topic (file format readers, coordinates/WCS/time, measurement > extraction, spectral modeling, orbits, planets, galaxies, stars, etc.) > and one that depends on all of the group packages, and viola, you can > have Sage, EPD, or whatever fall out as a trivial consequence. This is a very good idea - it is similar to what I was thinking of when I mentioned a hierarchical setup, but I went too far because - as you pointed out - a full Python distribution would be a nightmare to maintain. What you suggest is something more manageable, and in the end encourages better practices (and after all, who needs yet another full python distribution on their machine, besides e.g. the system python, scisoft, macports, etc.!) Is there a standard way of combining separate packages, potentially developed in different repositories, into a single meta-package for distribution to the user? Does distutils or distribute allow something like this by default? Cheers, Thomas > What is critical here is something you did not mention in your > posting, and that is solving the current difficulty of building a good > Python stack from OS packages (e.g., debs), or even from package > sources. The problem is that some of the packages do not play well > with others. For example, HDF5 libraries were particularly > problematic a year ago, and often compiled code complained because two > different packages wanted different, specific versions of the same C > or FORTRAN add-on library. Some of the code plain didn't work as > advertized, or at all. In the end, it takes my very skilled system > manager more than a week to do it, each time we do it, which is about > once a year. > > The root of the problem is that there was no centralized build and > test suite, nobody managing unified, integrated build testing and > resolving the problems with the code maintainers. > > A year ago at SciPy'09, I pointed people to the build and test suite > NSF requires for all their software projects. I think a few people > looked at it at the time. Configured correctly (which takes work), it > will build packages on umpteen different linuxes and a few other > systems and package them in the native format. > > I think that the extended SciPy stack as a whole should be organized > around such a system, but there seems to be little taste for it among > the developer-heavy SciPy leadership. I think we can have a bit more > practical vision, and at least for our own stuff (loosely defined) we > should organize around principles that include these: > > - build and test everything frequently > - manage the namespace so that all can play together > - we really need to get everyone to agree to this or there will be > conflicts, it's only a matter of time > - produce LPUs (least packageable units) in binary format for all OSes > - produce by topical meta-packages > - produce one or more mega-packages that are equivalent to Sage or EPD > - do it all for the native installers of all OSes > - provide and enforce standards for docs and licenses > - review the code > - plan together and put out RFPs for needed codes > - make sure procedures don't stifle innovators > - document the whole, but lightly > - provide a web community site for discussions, examples, reviews of code > - locate and manage it so that it is owned by the community and > survives long-term > - ensure all jobs doable by at least 3 people > - document procedures well > - have formalized community governance and leadership > - have a solid funding model > - agree to hang together, even if you don't like something! > > With those (and perhaps other) goals in mind, we should then look at > decisions like where/how to host it and what kind of wiki to use. > > Also, I keep thinking that this is best solved by joining forces with > other scientific communities. The build-and-test part is hard, but > once implemented, it scales fantastically. Again, all of SciPy should > be doing this. We should at least build so that if others want to > join, they have a place to fit in. This will need to be considered > when doing package naming conventions. Generic names should be > avoided so that two can play in the same sandbox. > > Even if we don't do the full thing from the start, we should plan it > out and build as though that's where we're eventually going. > > AAS splinter meeting, anyone? > > --jh-- > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Wed Jul 7 02:30:05 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Tue, 6 Jul 2010 23:30:05 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> References: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> Message-ID: > Is there a standard way of combining separate packages, potentially > developed in different repositories, into a single meta-package for > distribution to the user? Does distutils or distribute allow something like > this by default? This idea of a bunch of metapackages sounds great - see below for related thoughts. setuptools (and hence, distribute) has the option for "extras" - you add an extras_requires argument to the setup function with a dictionary that maps names of a extra features to a list of the associated packages. So you could define a mostly empty metapackage with just a bunch of extra_requires entries, and then users can do easy_install astrometapackage[optical] to install all the optical astronomy-related packages, or easy_install astrometapackage[all] to get all the contained packages. Having said this, I recall a discussion a while back on the distribute mailing list where they were talking about not including the "extras" feature in future releases, as it's difficult to maintain the dependencies or something like that. I don't know if this is actually going to happen, but apparently it's a little-used feature... I'm also not sure if it works with pip (the easy_install replacement to go with distribute) > The root of the problem is that there was no centralized build and > test suite, nobody managing unified, integrated build testing and > resolving the problems with the code maintainers. A well-organized centralized scheme is definitely a good idea (you're probably right that it'd be great for SciPy), but it seems like a large chunk of the astronomy community doesn't work that way (and probably never will). Better to have the *option* of a complex organizational/testing framework without the *requirement*. It should be easy to let projects naturally scale to a common organizational framework, but only as/if they need to > For those reluctant, we are curious about what the main obstacles are. I'm not exactly clear on what the repository you're speaking of encompasses (and hence what the obstacles are). Is this just a repository storing a whole bunch of seperate packages, or actually a common package that is distributed with a single "python setup.py install"? If it's the former, the main thing in my mind is that it seems highly redundant - there around exist repositories that are excellent (google code, launchpad, bitbucket, github, etc.), and It's not clear to me what is gained by making an "astronomy repository" when these other options are most likely to have anything we would need and probably will be more consistently maintained/upgraded given that they typically are focused on just being a hosting service. I also am definitely not a fan of Trac or SVN. The issue with Trac may be mostly just personal preference - it always just seems kind of clunky, behind-the-times, and difficult to use, although maybe I just didn't try hard enough. SVN, though, would be a non-starter -- distributed VCS like Mercurial, git, or Bazaar are much better suited to the rather common astronomy situation of a bunch of individuals hacking away at something when time permits, while still supporting the standard SVN/CVS-like centralized workflow. If it's the latter, the biggest problem to me is that all of a sudden it becomes *much* harder to manage distribution and have control over the "vision." I also can't help but think installation and dependencies would quickly become a huge hassle given the huge diversity of hardware and software needs that various parts of the astronomy community generally has. This is the main thing that would make me leery of a centralized build/testing scheme of the depth described above - lots of astronomers want to hack something quickly together that "just works" and will not bother if they have to jump through many hoops to contribute. In my mind, the most useful thing in the short-term would be a centralized (and actually close-to-complete!) listing of astronomy packages well-organized by topic, with the *option* of having that site also be the repository/host/bug tracker/wiki. There would be an expectation that listed packages clearly specify their requirements and post them to pypi.python.org in the standard setup.py way . All the mechanisms already exist within distutils and distribute - it there's an organized listing, it's trivial to write scripts that will use the already-existing pypi mechanisms to either generate metapackages as described above, or install various groupings of astronomy packages even if the extras option disappears. It's not quite as straitforward, but still probably only moderately difficult to combine these into system-specific packages. The could also be lots of neat ways to encourage people to combine closely related packages using this sort of scheme. It seems to me this can greatly simplify the distribution problems while still leaving a lot of flexibility for a community that's famously (notoriously?) independently-minded. > AAS splinter meeting, anyone? +1 -- Erik Tollerud From kdere at gmu.edu Wed Jul 7 09:04:58 2010 From: kdere at gmu.edu (Ken Dere) Date: Wed, 07 Jul 2010 09:04:58 -0400 Subject: [AstroPy] ChiantiPy - a Python interfact to the CHIANTI atomic database for astrophysical spectroscopy Message-ID: <4C347B7A.6080102@gmu.edu> We are pleased to announce the release of ChiantiPy, a Python package for calculating synthetic spectra from the CHIANTI atomic database for astrophysical spectroscopy. ChiantiPy provides the same functionality as the existing IDL package. One benefit of ChiantiPy is that it uses Python, a freely available programming language. ChiantiPy is constructed as an object-oriented package that provides a good match to the modular construction of the CHIANTI database itself. Our intention is to maintain both the IDL and Python packages for the indefinite future. The documentation for ChiantiPy can be found at http://chiantipy.sourceforge.net/ and the software package can be downloaded from http://sourceforge.net/projects/chiantipy/ Python is available for Windows, Mac OS, Linux and Unix. ChiantiPy itself has been developed on a Linux platform and has received most of its testing on Linux. In software development terminology, it is an 'alpha' release, meaning it has not really undergone sufficient testing in a variety of environments. We welcome reports of user's experiences on the chiantipy-users email list on Sourceforge: chiantipy-users at lists.sourceforge.net In addition, several discussion forums have been set up at http://sourceforge.net/projects/chiantipy/forums Ken Dere and the CHIANTI team -- Kenneth P. Dere Research Professor of Solar Physics Dept. of Computational and Data Sciences MS 6A2 George Mason University 4400 University Drive Fairfax VA 22030 703-993-4555 703-993-1992 FAX kdere at gmu.edu From perry at stsci.edu Wed Jul 7 13:42:02 2010 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 7 Jul 2010 13:42:02 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: Message-ID: <8AA546FF-B68E-4B94-987B-409C284524D4@stsci.edu> On Jul 6, 2010, at 6:01 PM, Joe Harrington wrote: > We've hit this topic (monolithic vs. Balkanized) several times over > the past 5 years or so in various forms. Tom, you have hit many of > the main issues in your posting (licenses, maintenance, management > preferences, etc.). However, what you are contemplating doing at the > end is unfortunately something that seems a disease: making Yet > Another Specialized Python Distro(TM). There are already too many > Sages and EPDs, and take it from one trying to get SciPy documented, > managing a monolithic entity is a nightmare down the road when some of > the packages go off maintenance, yet everyone depends on them. > > What I think we want from the users' perspective is an ability to say, > *in the terms of their system's native package manager*, any of: > > give me this/these specific package(s) > give me this/these group(s) of packages > give me all the packages > > Then, we can publish metapackages that depend on groups of add-ons by > topic (file format readers, coordinates/WCS/time, measurement > extraction, spectral modeling, orbits, planets, galaxies, stars, etc.) > and one that depends on all of the group packages, and viola, you can > have Sage, EPD, or whatever fall out as a trivial consequence. > I wish it were that simple. It isn't. There is a reason why there are "too many" Sages and EPDs; that's because there isn't a good alternate yet. In fact, packaging your own stack is what Guido himself has recommended as a reasonable solution to the distribution problem for significant applications: http://www.archive.org/details/ucb_py4science_2009_11_04_Guido_van_Rossum (about 90 minutes into the video is where Guido discusses this issue, and 95 minutes where he specifically recommends packaging Python with applications) Even RPMs aren't sufficient for all since they require root, and we do see a significant number of users who don't have root. RPMs also don't solve dependency conflicts. > What is critical here is something you did not mention in your > posting, and that is solving the current difficulty of building a good > Python stack from OS packages (e.g., debs), or even from package > sources. The problem is that some of the packages do not play well > with others. For example, HDF5 libraries were particularly > problematic a year ago, and often compiled code complained because two > different packages wanted different, specific versions of the same C > or FORTRAN add-on library. Some of the code plain didn't work as > advertized, or at all. In the end, it takes my very skilled system > manager more than a week to do it, each time we do it, which is about > once a year. > > The root of the problem is that there was no centralized build and > test suite, nobody managing unified, integrated build testing and > resolving the problems with the code maintainers. > I think that's only part of the problem, and the idealized solution you suggest is also part of the problem. It is important to realize that that the general dependency issue is a very difficult one (it has been shown to be NP-complete). So centralized at what level? All software in the world? Then the coordination, synchronization, and testing issues become impossible to manage. (A linux distribution is an illustration of what it takes to do that; it's a lot of work, and it means the software included is often well out of date as a result.) If not, where is the line drawn? Regardless of where you draw it, you will have people complain that it doesn't include what they need. Sage, EPD, and other stacks are different sets of centralized software sets that do this centralized building and testing. This is the essence of what they provide. It is natural that there has to be different versions to satisfy different communities if the problem is impossible to do globally. Sage includes much that most astronomers don't need, and doesn't include much that they do need. EPD is closer to what we need and we'll take a look at it as a possible basis for a distribution. > A year ago at SciPy'09, I pointed people to the build and test suite > NSF requires for all their software projects. I think a few people > looked at it at the time. Configured correctly (which takes work), it > will build packages on umpteen different linuxes and a few other > systems and package them in the native format. > We took a look at it after SciPy'09 and concluded that it wasn't suitable. > I think that the extended SciPy stack as a whole should be organized > around such a system, but there seems to be little taste for it among > the developer-heavy SciPy leadership. I think we can have a bit more > practical vision, and at least for our own stuff (loosely defined) we > should organize around principles that include these: > > - build and test everything frequently > - manage the namespace so that all can play together > - we really need to get everyone to agree to this or there will be > conflicts, it's only a matter of time > - produce LPUs (least packageable units) in binary format for all OSes > - produce by topical meta-packages > - produce one or more mega-packages that are equivalent to Sage or EPD > - do it all for the native installers of all OSes > - provide and enforce standards for docs and licenses > - review the code > - plan together and put out RFPs for needed codes > - make sure procedures don't stifle innovators > - document the whole, but lightly > - provide a web community site for discussions, examples, reviews of > code > - locate and manage it so that it is owned by the community and > survives long-term > - ensure all jobs doable by at least 3 people > - document procedures well > - have formalized community governance and leadership > - have a solid funding model > - agree to hang together, even if you don't like something! > > With those (and perhaps other) goals in mind, we should then look at > decisions like where/how to host it and what kind of wiki to use. > > Also, I keep thinking that this is best solved by joining forces with > other scientific communities. The build-and-test part is hard, but > once implemented, it scales fantastically. Again, all of SciPy should > be doing this. We should at least build so that if others want to > join, they have a place to fit in. This will need to be considered > when doing package naming conventions. Generic names should be > avoided so that two can play in the same sandbox. > > Even if we don't do the full thing from the start, we should plan it > out and build as though that's where we're eventually going. > > AAS splinter meeting, anyone? There is much in this that is Good. Yet doing all this requires a great deal of work for which resources currently aren't available. We can make a lot of progress and still not accomplish most of these things. We should be careful to prioritize these activities to those that are most important with the resources available. Perry From ebressert at cfa.harvard.edu Wed Jul 7 14:12:08 2010 From: ebressert at cfa.harvard.edu (Eli Bressert) Date: Wed, 7 Jul 2010 19:12:08 +0100 Subject: [AstroPy] AstroPy Digest, Vol 47, Issue 6 In-Reply-To: References: Message-ID: The interest in centralizing astronomy related Python modules has picked up a big interest on this mailing list. As a developer for ATpy and APLpy (co-developed with Thomas Robitaille and help from Adam Ginsburg), I like having the freedom or "vision" of which direction my packages will go. This appears to be what other developers prefer as well. The easiest or maybe most viable solution is a meta-package Python astrolibary. This goes along the lines of what Erik Tollerud suggested. It would be great to have the ability to type in 'easy_install astrolib' or 'pip astrolib' and have all the necessary packages installed. What may compliment the comprehensive easy_install idea is a website that allows users to select which packages they want from a list (e.g. http://www.astropython.org/resources). Then click a button that would generate a personalized package installer for easy_install. The user would copy a generated link and use easy_install as follows: 'easy_install http://astrolib-generator.org/unique-key/random_name.tar.gz' The tar.gz file would be small and only contain a dependency list which installs the user's selected packages. The sub-dependencies would be taken care of automatically. Making suggested packages, e.g. optical or X-ray bundles, from the package generator would be relatively simple in this case. > AAS splinter meeting, anyone? +2 Cheers, Eli Bressert From kellecruz at gmail.com Wed Jul 7 21:01:22 2010 From: kellecruz at gmail.com (Kelle Cruz) Date: Wed, 7 Jul 2010 21:01:22 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> Message-ID: Hi All, As a newbie to Python and a spoiled IDL user: Geez, what a mess! - I think a AAS Splinter is a great idea...looks like the deadline isn't until Dec 1 for Seattle, but we should get on it since they are assigned on a first-come, first-served basis based on room availability. It might make more sense to do Boston in May 2011 because people won't be as busy with other meeting things and it should be nice there in spring. I nominate Perry to lead the proposal. (It's not a big deal....main thing will be to figure out total cost/expected number of participants = fee...or find someone to pay for it. STScI, maybe?) On Wed, Jul 7, 2010 at 2:30 AM, Erik Tollerud wrote: > In my mind, the most useful thing in the short-term would be a centralized > (and actually close-to-complete!) listing of astronomy > packages well-organized by topic, with the *option* of having that site > also be the repository/host/bug tracker/wiki. > Well, Eli and Tom have already done the listing on AstroPython: http://www.astropython.org/resources or http://www.astropython.org/resources?sort=tag However, I find the entire AstroPython site to be very non-intuitive...and you can't upload files to it (currently) which you can on the AstroBetter wiki (as ianc did with his DAYCONV.py script). I'm going to work on filling in the Python wiki table I started and see how that works out: http://www.astrobetter.com/wiki/tiki-index.php?page=Python+Switchers+Guide It would be ideal of the authors of the multi-purpose packages -- Erik and Mubdi, I'm talking to you -- helped to fill it in. If you think I'm wasting my time with the table, please let me know. So far, it's at least been very helpful for *me* to get my head wrapped around everything that's out there.... kelle -- Kelle Cruz, PhD http://kellecruz.com/ 424.645.1334 -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Thu Jul 8 09:17:34 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 09:17:34 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> Message-ID: On Jul 7, 2010, at 9:01 PM, Kelle Cruz wrote: > Hi All, > > As a newbie to Python and a spoiled IDL user: Geez, what a mess! > > - I think a AAS Splinter is a great idea...looks like the deadline > isn't until Dec 1 for Seattle, but we should get on it since they > are assigned on a first-come, first-served basis based on room > availability. It might make more sense to do Boston in May 2011 > because people won't be as busy with other meeting things and it > should be nice there in spring. I nominate Perry to lead the > proposal. (It's not a big deal....main thing will be to figure out > total cost/expected number of participants = fee...or find someone > to pay for it. STScI, maybe?) > Sure if there is serious interest, I can look into having it funded. I didn't see anything specifically mentioning the fees involved. Anyone have an idea of that? (I'll email them as well asking). Perry From perry at stsci.edu Thu Jul 8 09:19:51 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 09:19:51 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> Message-ID: Spoke too soon. Apparently the cost is less than $500 for the room. That shouldn't be a problem I would think. On Jul 8, 2010, at 9:17 AM, Perry Greenfield wrote: > > On Jul 7, 2010, at 9:01 PM, Kelle Cruz wrote: > >> Hi All, >> >> As a newbie to Python and a spoiled IDL user: Geez, what a mess! >> >> - I think a AAS Splinter is a great idea...looks like the deadline >> isn't until Dec 1 for Seattle, but we should get on it since they >> are assigned on a first-come, first-served basis based on room >> availability. It might make more sense to do Boston in May 2011 >> because people won't be as busy with other meeting things and it >> should be nice there in spring. I nominate Perry to lead the >> proposal. (It's not a big deal....main thing will be to figure out >> total cost/expected number of participants = fee...or find someone >> to pay for it. STScI, maybe?) >> > Sure if there is serious interest, I can look into having it funded. > I didn't see anything specifically mentioning the fees involved. > Anyone have an idea of that? (I'll email them as well asking). > > Perry > From perry at stsci.edu Thu Jul 8 10:13:05 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 10:13:05 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> References: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> Message-ID: <0DD42A60-B14D-4B46-9BA0-8BE2896169C0@stsci.edu> On Jul 6, 2010, at 11:15 PM, Thomas Robitaille wrote: > > This is a very good idea - it is similar to what I was thinking of > when I mentioned a hierarchical setup, but I went too far because - > as you pointed out - a full Python distribution would be a nightmare > to maintain. What you suggest is something more manageable, and in > the end encourages better practices (and after all, who needs yet > another full python distribution on their machine, besides e.g. the > system python, scisoft, macports, etc.!) > > Is there a standard way of combining separate packages, potentially > developed in different repositories, into a single meta-package for > distribution to the user? Does distutils or distribute allow > something like this by default? Not reliably. Nor does easy_install work for many cases (the Python community is well aware of the current shortcomings of easy_install, (e.g., see: http://www.b-list.org/weblog/2008/dec/14/packaging/) These often don't handle fortran code well (scipy uses a modified distutils to handle those issues). To really ensure that everything works basically requires bundling the tricky stuff (particularly non-python library dependencies) into the core (e.g., not relying on it being present on the host system) and testing it routinely to make sure it works on the platforms you want it to work on. I don't think there is a purely modular system that works reliably. I can envision a lot of modular add-ons that don't present a problem (for example, pure python code that has no new library dependencies, or python/C code that uses distutils in a simple and reliable way). But there is a lot of stuff out there that doesn't fit that mode. So to elaborate a bit more on what we are thinking about is making a core distribution that contains things that are generally useful to most astronomers, things required for our software ("our" currently being Gemini and STScI, but that could grow), or libraries that are hard to install later but are used by possible modular add ons (VTK is a good example). Then there would be optional packages one can add to that list. The core system and optional packages would be regularly built and tested on many platforms. Finally, there would be instructions on how to install arbitrary software into that environment (not necessarily trivial work). While anyone will be able to update or add software to the environment (there are often good reasons to do so), given that arbitrary updates or additions could break other things in the environment, no guarantees on whether such things will work can be made (but you are free to do so). There is no perfect solution out there, or even one close to perfect. The idea is to make installing and using the basic tools easy, and possible to customize that (but with risk that must be borne by the user). The most flexible approach is to install everything yourself, piece by piece, each thing in its own way. If you have special needs, sometimes that is the only real solution. Perry From perry at stsci.edu Thu Jul 8 10:13:38 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 10:13:38 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? References: <041EBF0C-56C3-4243-A2B8-DE37D66EECBB@stsci.edu> Message-ID: <79C00147-6C17-47BB-B212-2CCBBD84C879@stsci.edu> A SLALIB wrapper would be great (and justifiably part of an astronomy toolset). Of course, the problem is with SLALIB itself since it doesn't have an open source license. (That's something that other would have to obtain and install separately). That's the big drawback. Perry On Jul 6, 2010, at 12:39 PM, Anne Archibald wrote: > Hi, > > I use a couple of python astronomy libraries; the python bindings to > the pulsar search library PRESTO, and the python bindings to the > FORTRAN library SLALIB. Both were written by Scott Ransom; obviously I > can't speak for his motivations in writing them. > > I think I follow a fairly common pattern, though: the libraries I use > are fairly thin wrappers of existing compiled code, and in one case > the code is very domain-specific. So combining them somehow with other > python libraries would involve either linking together a huge mass of > different compiled libraries with overlapping functionality, or > modifying the underlying libraries. Either one would be a tremendous > amount of work. > > That said, I use SLALIB for a simple coordinate conversion (in fact I > use it to replicate exactly another coordinate conversion done in > compiled code, so this is kind of theoretical), so in principle if > there were a generic python astronomy library that covered this (I > didn't find one) I might have used it instead. The PRESTO code, not so > much; it includes some extremely specific pulsar searching code that > can't reasonably be included in existing python libraries. > > What might be nice would be a "standard" list of data types for common > objects, so that at least one could pass objects from one library to > another. RA and Dec objects, perhaps, or time objects. Of course time > objects would be a massive headache if you wanted them to be > generically usable, since they'd need to be good to the nanosecond > over a span of millenia (so doubles won't cut it), support > UTC/TDB/TAI/etc. as well as MJD/sidereal/GMT/local time > representations... Nevertheless, compiled code (and thin wrappers) > tend to just take doubles and rely on the user to keep track of what > they mean, while a more pythonic and less error-prone approach would > use python's type system to catch many errors. > > Anne From perry at stsci.edu Thu Jul 8 10:14:03 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 10:14:03 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? References: Message-ID: <574936BD-2C04-43A8-9266-FFDB73BB40AE@stsci.edu> Again, I don't see dependencies on other software as a reason to keep packages out. This is a good example of this (sextractor, slalib), etc. Perry On Jul 3, 2010, at 6:24 PM, Brandon Craig Rhodes wrote: > James Turner writes: > >> At SciPy 2009 ... we discussed the fact that there are now a number >> of >> astronomy libraries for Python floating around and maybe it would be >> good to collect more code into a single place. People seemed >> receptive to this idea and weren't sure why it hasn't already >> happened... > > Hey, James! Thanks, I've now joined the AstroPy mailing list. > > The reason that I have never really considered trying to merge PyEphem > with any other astronomy package is one of architecture. The > "libastro" > library from inside of XEphem, which PyEphem tweaks and then places > behind a Pythonic interface, is designed to use all of its own types > (either C "doubles" or structs built from doubles) for inputs like > dates, locations, and orbital elements, and outputs like coordinates. > It is its own little universe, software-wise, and PyEphem remains > easiest to keep working when I leave the XEphem code in a nearly > pristine state. This advantage would be lost if I gutted the pure-C > astronomy routines to read and write data to, say, NumPy arrays > instead, > or any other third-party data format. The fun of using PyEphem is the > efficiency of asking a question, getting a date back, then being > able to > plug that date right into your next calculation because it's already a > floating-point number of the sort that "libastro" uses internally for > dates. > > Another reason that I have never had dialog with another project about > merging or sharing code is that most of the astronomy packages that I > see appear for Python (I do go browse the other astronomy packages on > PyPI from time to time) are doing entirely different things than > figuring out where Jupiter is on a given night: they're doing things > like image processing, which are entirely different problems that use > entirely different data structures. Even an orbital mechanics > simulation would probably use quite different data structures (it > would > hopefully include momentum, for example!) from the RA/declination- > based > "structs" that lie behind the Jupiter(), Mars(), and other objects in > PyEphem. > > That's not to say that I've never thought of trying to start over > with a > pure-Python astronomy library. I even, recently, thought through > how I > would improve upon the PyEphem object set if I had the chance. It > would > also be nice if routines were implemented in Python - right now, > PyEphem > can only be used with C-Python but not easily with IronPython; that > would not be the case if everything were written in Python. (Though, > the routines would obviously be much slower until C-Python gets a JIT > compiler!) > > But to take these steps would be to essentially start over with a new > project, and the point of PyEphem was quite different: to take a > robust, > already-working, mature library for computing planet and comet > positions, and make it efficiently callable from Python. > > -- > Brandon Craig Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From sransom at nrao.edu Thu Jul 8 10:17:58 2010 From: sransom at nrao.edu (Scott Ransom) Date: Thu, 8 Jul 2010 10:17:58 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <79C00147-6C17-47BB-B212-2CCBBD84C879@stsci.edu> References: <041EBF0C-56C3-4243-A2B8-DE37D66EECBB@stsci.edu> <79C00147-6C17-47BB-B212-2CCBBD84C879@stsci.edu> Message-ID: <201007081017.58358.sransom@nrao.edu> Hi Perry et al, The Fortran version of SLALIB that I wrapped was GPLd. You can get the code here (SLALIB + wrappers and tests): http://www.cv.nrao.edu/~sransom/pyslalib-1.0.tar.gz Scott On Thursday, July 08, 2010 10:13:38 am Perry Greenfield wrote: > A SLALIB wrapper would be great (and justifiably part of an > astronomy toolset). Of course, the problem is with SLALIB itself > since it doesn't have an open source license. (That's something that > other would have to obtain and install separately). That's the big > drawback. > > Perry > > On Jul 6, 2010, at 12:39 PM, Anne Archibald wrote: > > Hi, > > > > I use a couple of python astronomy libraries; the python bindings > > to the pulsar search library PRESTO, and the python bindings to > > the FORTRAN library SLALIB. Both were written by Scott Ransom; > > obviously I can't speak for his motivations in writing them. > > > > I think I follow a fairly common pattern, though: the libraries I > > use are fairly thin wrappers of existing compiled code, and in one > > case the code is very domain-specific. So combining them somehow > > with other python libraries would involve either linking together > > a huge mass of different compiled libraries with overlapping > > functionality, or modifying the underlying libraries. Either one > > would be a tremendous amount of work. > > > > That said, I use SLALIB for a simple coordinate conversion (in fact > > I use it to replicate exactly another coordinate conversion done > > in compiled code, so this is kind of theoretical), so in principle > > if there were a generic python astronomy library that covered this > > (I didn't find one) I might have used it instead. The PRESTO code, > > not so much; it includes some extremely specific pulsar searching > > code that can't reasonably be included in existing python > > libraries. > > > > What might be nice would be a "standard" list of data types for > > common objects, so that at least one could pass objects from one > > library to another. RA and Dec objects, perhaps, or time objects. > > Of course time objects would be a massive headache if you wanted > > them to be generically usable, since they'd need to be good to the > > nanosecond over a span of millenia (so doubles won't cut it), > > support UTC/TDB/TAI/etc. as well as MJD/sidereal/GMT/local time > > representations... Nevertheless, compiled code (and thin wrappers) > > tend to just take doubles and rely on the user to keep track of > > what they mean, while a more pythonic and less error-prone > > approach would use python's type system to catch many errors. > > > > Anne > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From perry at stsci.edu Thu Jul 8 10:20:48 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 10:20:48 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <201007081017.58358.sransom@nrao.edu> References: <041EBF0C-56C3-4243-A2B8-DE37D66EECBB@stsci.edu> <79C00147-6C17-47BB-B212-2CCBBD84C879@stsci.edu> <201007081017.58358.sransom@nrao.edu> Message-ID: On Jul 8, 2010, at 10:17 AM, Scott Ransom wrote: > Hi Perry et al, > > The Fortran version of SLALIB that I wrapped was GPLd. You can get > the > code here (SLALIB + wrappers and tests): > > http://www.cv.nrao.edu/~sransom/pyslalib-1.0.tar.gz > > Scott But the C version isn't (maybe I'm looking at the wrong web page), right? The fortran one would probably be more of a distribution hassle (but perhaps still the best solution). Have you had any trouble supporting this on different platforms? Perry From sransom at nrao.edu Thu Jul 8 10:23:28 2010 From: sransom at nrao.edu (Scott Ransom) Date: Thu, 8 Jul 2010 10:23:28 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <041EBF0C-56C3-4243-A2B8-DE37D66EECBB@stsci.edu> <201007081017.58358.sransom@nrao.edu> Message-ID: <201007081023.28686.sransom@nrao.edu> On Thursday, July 08, 2010 10:20:48 am Perry Greenfield wrote: > On Jul 8, 2010, at 10:17 AM, Scott Ransom wrote: > > Hi Perry et al, > > > > The Fortran version of SLALIB that I wrapped was GPLd. You can get > > the > > code here (SLALIB + wrappers and tests): > > > > http://www.cv.nrao.edu/~sransom/pyslalib-1.0.tar.gz > > > > Scott > > But the C version isn't (maybe I'm looking at the wrong web page), > right? I'm pretty sure that is correct. > The fortran one would probably be more of a distribution > hassle (but perhaps still the best solution). Have you had any > trouble supporting this on different platforms? The user base is probably pretty small. I think people are successfully using it on Mac and certainly on many flavors of Linux. Scott -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From jturner at gemini.edu Thu Jul 8 13:44:20 2010 From: jturner at gemini.edu (James Turner) Date: Thu, 8 Jul 2010 13:44:20 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <201007081017.58358.sransom@nrao.edu> References: <041EBF0C-56C3-4243-A2B8-DE37D66EECBB@stsci.edu> <79C00147-6C17-47BB-B212-2CCBBD84C879@stsci.edu> <201007081017.58358.sransom@nrao.edu> Message-ID: <4C360E74.7040402@gemini.edu> I'm still catching up on reading this monster thread between doing vacation things (thanks for all the comments, everyone)... I'll quickly comment on this first. > The Fortran version of SLALIB that I wrapped was GPLd. You can get the > code here (SLALIB + wrappers and tests): > > http://www.cv.nrao.edu/~sransom/pyslalib-1.0.tar.gz Yes, the latest SLALIB does have an open source licence, but it's GPL, which will put off a lot of the SciPy community (which may or may not matter in this case). Any application that links with SLALIB will also need to be GPL (at least its binaries) and the SciPy people are quite adamant about using BSD. I suspect that's largely because Enthought has contributed a lot to SciPy and their business model depends on selling the code in commercial apps. It's hard to argue with that, given how central the Enthought people have been in making SciPy and NumPy happen. There is, however, an older version of SLALIB distributed with IRAF, prior to JAC relicensing the library with their changes under the GPL. The licensing of that is not clearly stated though, as far as non-IRAF apps are concerned. Does anyone have comments on 1) whether people need the latest SLALIB (or is IRAF's version good enough?) and 2) whether we're concerned about licence-compatibility with SciPy for libraries that are so astronomy specific? For more general stuff (eg. deconvolution) I'd suggest that we should care about the licence, so we can share code with other disciplines and maintain goodwill in the SciPy community. Cheers, James. From rahman at astro.utoronto.ca Thu Jul 8 13:55:35 2010 From: rahman at astro.utoronto.ca (Mubdi Rahman) Date: Thu, 8 Jul 2010 17:55:35 +0000 (GMT) Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C360E74.7040402@gemini.edu> References: <041EBF0C-56C3-4243-A2B8-DE37D66EECBB@stsci.edu> <79C00147-6C17-47BB-B212-2CCBBD84C879@stsci.edu> <201007081017.58358.sransom@nrao.edu> <4C360E74.7040402@gemini.edu> Message-ID: <271111.89092.qm@web29012.mail.ird.yahoo.com> Just a quick question, how does the SciPy community feel about the LGPL license? -Mubdi ----- Original Message ---- > From: James Turner > To: Scott Ransom > Cc: astropy at scipy.org > Sent: Thu, July 8, 2010 1:44:20 PM > Subject: Re: [AstroPy] Co-ordinating Python astronomy libraries? > > I'm still catching up on reading this monster thread between doing > vacation things (thanks for all the comments, everyone)... I'll > quickly comment on this first. > > > The Fortran version of SLALIB that I wrapped was GPLd. You can get the > > code here (SLALIB + wrappers and tests): > > > > http://www.cv.nrao.edu/~sransom/pyslalib-1.0.tar.gz > > Yes, the latest SLALIB does have an open source licence, but it's GPL, > which will put off a lot of the SciPy community (which may or may not > matter in this case). Any application that links with SLALIB will also > need to be GPL (at least its binaries) and the SciPy people are quite > adamant about using BSD. I suspect that's largely because Enthought > has contributed a lot to SciPy and their business model depends on > selling the code in commercial apps. It's hard to argue with that, > given how central the Enthought people have been in making SciPy and > NumPy happen. > > There is, however, an older version of SLALIB distributed with IRAF, > prior to JAC relicensing the library with their changes under the GPL. > The licensing of that is not clearly stated though, as far as non-IRAF > apps are concerned. > > Does anyone have comments on 1) whether people need the latest SLALIB > (or is IRAF's version good enough?) and 2) whether we're concerned > about licence-compatibility with SciPy for libraries that are so > astronomy specific? For more general stuff (eg. deconvolution) I'd > suggest that we should care about the licence, so we can share code > with other disciplines and maintain goodwill in the SciPy community. > > Cheers, > > James. > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > From perry at stsci.edu Thu Jul 8 14:03:36 2010 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 8 Jul 2010 14:03:36 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <097CF2EC-C61E-4D83-8519-3A02C29E92DB@gmail.com> Message-ID: replying to the answer to the question I posed... On Jul 7, 2010, at 2:30 AM, Erik Tollerud wrote: > >> For those reluctant, we are curious about what the main obstacles >> are. > > I'm not exactly clear on what the repository you're speaking of > encompasses (and hence what the obstacles are). Is this just a > repository storing a whole bunch of seperate packages, or actually a > common package that is distributed with a single "python setup.py > install"? > Well, more the former, but also to enable something along the latter (though not necessarily part of a single install). When things are here and there, it is more difficult to package those together in an automatic way. Not impossible, just more work and more ways for things to go wrong. > If it's the former, the main thing in my mind is that it seems highly > redundant - there around exist repositories that are excellent (google > code, launchpad, bitbucket, github, etc.), and It's not clear to me > what is gained by making an "astronomy repository" when these other > options are most likely to have anything we would need and probably > will be more consistently maintained/upgraded given that they > typically are focused on just being a hosting service. I also am We are talking about using a managed hosting service (commercial in this instance). > definitely not a fan of Trac or SVN. The issue with Trac may be > mostly just personal preference - it always just seems kind of clunky, > behind-the-times, and difficult to use, although maybe I just didn't > try hard enough. SVN, though, would be a non-starter -- distributed > VCS like Mercurial, git, or Bazaar are much better suited to the > rather common astronomy situation of a bunch of individuals hacking > away at something when time permits, while still supporting the > standard SVN/CVS-like centralized workflow. > But trac is fairly portable since it is widely used. Migrating the info from some of the others can be a more difficult problem (lock-in issues). As for svn, git, hg, bazaar issues, nothing is going to satisfy everyone, I agree. > If it's the latter, the biggest problem to me is that all of a sudden > it becomes *much* harder to manage distribution and have control over > the "vision." I also can't help but think installation and > dependencies would quickly become a huge hassle given the huge > diversity of hardware and software needs that various parts of the > astronomy community generally has. This is the main thing that would > make me leery of a centralized build/testing scheme of the depth > described above - lots of astronomers want to hack something quickly > together that "just works" and will not bother if they have to jump > through many hoops to contribute. > If it is something hacked together quickly with little interest in making it generally useful or consistent, then I agree, it probably shouldn't be part of the repository (nor any standard distribution I argue). We're talking about things that seem to be beyond that. > > In my mind, the most useful thing in the short-term would be a > centralized (and actually close-to-complete!) listing of astronomy > packages well-organized by topic, with the *option* of having that > site also be the repository/host/bug tracker/wiki. There would be an > expectation that listed packages clearly specify their requirements > and post them to pypi.python.org in the standard setup.py way . All > the mechanisms already exist within distutils and distribute - it > there's an organized listing, it's trivial to write scripts that will > use the already-existing pypi mechanisms to either generate > metapackages as described above, or install various groupings of > astronomy packages even if the extras option disappears. It's not > quite as straitforward, but still probably only moderately difficult > to combine these into system-specific packages. The could also be > lots of neat ways to encourage people to combine closely related > packages using this sort of scheme. It seems to me this can greatly > simplify the distribution problems while still leaving a lot of > flexibility for a community that's famously (notoriously?) > independently-minded. > That works to a certain level. But before long, there are n flavors of representing this and that, and combining 10 packages like this to use in your own code can get to be a real chore. You'll spend a lot of time moving from one convention to another (and not catching some bugs). I don't think making everyone conform is a good solution, but eventually standardizing on core libraries is a very good thing in the long run. I think there is a reasonable middle ground on this kind of thing. > >> AAS splinter meeting, anyone? > +1 > So most of the contributors are more likely to be at AAS than ADASS? That says something interesting about ADASS methinks. Perry From callen at gemini.edu Thu Jul 8 16:52:14 2010 From: callen at gemini.edu (Craig Allen) Date: Thu, 8 Jul 2010 10:52:14 -1000 Subject: [AstroPy] pyfits extension naming issue Message-ID: <1278622334.29367.11.camel@ophiuchus.hi.gemini.edu> Pardon me if this is not the best list for this but I figured it's a good place to start. The issue is with pyfits... we want to rename the extension in memory, that is, assign or change the EXNAME and EXTVER keys for that extension. We can easilly manipulate the header to do so, but "pyfits.info()" does not reflect the change, further, picking out the extension with the HDULists' "[]" operator doesn't work, complaining of missing properties. After investigation we found that this was because two HDU members are not set, "name" and "_extver". The main use case, btw, is that we have raw data in which the SCI extensions are not named, and when loading this into our AstroData IO abstraction (based on pyfits but providing some higher level abstractions wrt the contained dataset), we want to automatically name these extensions. The following function therefore "works". def setExtname(self, name, ver = None): # trimmed some irrelevant code if True: hdu = self.hdulist[1] nheader = hdu.header nheader.update("extname", name, "added by AstroData") nheader.update("extver", ver, "added by AstroData") hdu.name = name hdu._extver = ver # print "AD553:", repr(hdu.__class__) The only other way I found to get the system to recognize this is creating new HDUs from scratch (or writing the change and reloading the file). I like this much better, but: (1) comments (2) directions to the proper way to do this which I missed (3) how should pyfits handle this if it does not already? thanks -craig callen at gemini.edu From aldcroft at head.cfa.harvard.edu Fri Jul 9 11:51:58 2010 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Fri, 9 Jul 2010 11:51:58 -0400 Subject: [AstroPy] asciitable 0.2.5 Message-ID: I'd like to announce the release of asciitable version 0.2.5. Asciitable is an extensible ASCII table reader with built-in support for a number of common table formats in astronomy. http://cxc.harvard.edu/contrib/asciitable/ This release is primarily a documentation update, including much more detail on the parameters for the read() function plus a bit more for other advanced features. There are a couple of minor code fixes / updates: - Gracefully read tables that have a header but no data lines. - Update the Tab reader so all data value spaces are preserved (including leading/trailing) By the way if there are feature requests or different table formats needing support I am happy to listen. For instance, is missing-value support a high priority for people? Regards, Tom Aldcroft From thomas.robitaille at gmail.com Fri Jul 9 17:02:23 2010 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Fri, 9 Jul 2010 17:02:23 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C2BBBA7.5060006@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> Message-ID: Hi all, After reading all the replies, I have the following suggestion to make. The model scipy follows is to have a 'core' scipy package, and 'scikit' packages which share a common namespace, and which are meant as addons to the scipy package, but have not yet (or might never) make it to the main scipy package: http://www.scipy.org/scipy/scikits/ "Scipy Toolkits are independent and seperately installable projects hosted under a common namespace. Packages that are distributed in this way are here (instead of in monolithic scipy) for at least one of three general reasons. Each of these reasons use the same high-level namespace (scikits)." I think we can use this model, and that the following approach can be used here, namely: - start with a very basic 'astropy' package, with e.g. support for FITS/WCS - agree to coordinate astronomy packages with a common namespace (e.g. 'astrokit', so for example, APLpy would become astrokit.aplpy). This can help us manage the namespace (as suggested in Joe Harrington's email) - as astrokit modules mature, they can (if the authors are willing) be merged into the main 'astropy' package, once they have met a number of conditions, including e.g. unit tests, sphinx documentation, limited dependencies (e.g. numpy/scipy/matplotlib, and any package in the 'astropy' package), and compatible license. The advantage of this model is that this encourages the growth from the bottom up of a core astronomy package, which is manageable, as well as the independent development of other packages. It also means that the core package will be quite stable, because it will only accrete 'astrokit' modules as they become stable and mature. At the same time, it encourages developers to make their own innovative astrokit, but without commitment from the maintainers of the core package to accrete it in future. In passing, this also leaves the possibility for those who want to develop meta-pacakges of astrokit modules. Also, it will make it easier for those who want to build fully fledged astronomy python distributions. There have been many ideas passed around in the mailing list, but I think that some are maybe too ambitious. I do think that the idea suggested above is much more manageable. The concrete steps would be: - setup a central repository for the core packages, as well as astrokit pacakges (although we should leave the option for people to develop astrokit packages outside this repository too, and rely on trust and communication to avoid namespace clashes) - start a core package with e.g. FITS and WCS support (e.g. pyfits + pywcs) - set up a list of 'registered' astrokit names to avoid conflict. - set up a list of recommendations and requirements for astrokit pacakges - encourage developers of existing packages to use this namespace and follow the guidelines. This would be very little work in some cases, which is why I think it could work. Sharing a namespace is (I think) the first step to showing that we are willing to work on something coherent, and users would only see two top-level namespaces - astropy and astrokit, which would be a great improvement over the current situation where it is not immediately obvious which packages are astronomy related. Comments/suggestions/criticism welcome! Cheers, Tom On Jun 30, 2010, at 5:48 PM, James Turner wrote: > Dear Python users in astronomy, > > At SciPy 2009, I arranged an astronomy BoF where we discussed the > fact that there are now a number of astronomy libraries for Python > floating around and maybe it would be good to collect more code into > a single place. People seemed receptive to this idea and weren't sure > why it hasn't already happened, given that there has been an Astrolib > page at SciPy for some years now, with an associated SVN repository: > > http://scipy.org/AstroLib > > After the meeting last August, I was supposed to contact the mailing > list and some library authors I had talked to previously, to discuss > this further. My apologies for taking 10 months to do that! I did > draft an email the day after the BoF, but then we ran into a hurdle > with setting up new committers to the AstroLib repository (which has > taken a lot longer than expected to resolve), so it seemed a bad > time to suggest that new people start using it. > > To discuss these issues further, we'd like to encourage everyone to > sign up for the AstroPy mailing list if you are not already on it. > The traffic is just a few messages per month. > > http://lists.astropy.scipy.org/mailman/listinfo/astropy > > We (the 2009 BoF group) would also like to hear on the list about > why people have decided to host their own astronomy library (eg. not > being aware of the one at SciPy). Are you interested in contributing > to Astrolib? Do you have any other comments or concerns about > co-ordinating tools? Our motivation is to make libraries easy to > find and install, allow sharing code easily, help rationalize > available functionality and fill in what's missing. A standard > astronomy library with a single set of documentation should be more > coherent and easier to maintain. The idea is not to limit authors' > flexibility of take ownership of their code -- the sub-packages > can still be maintained by different people. > > If you're at SciPy this week, Perry Greenfield and I would be happy > to talk to you. If you would like to add your existing library to > Astrolib, please contact Perry Greenfield or Mark Sienkiewicz at > STScI for access (contact details at http://scipy.org/AstroLib). > Note that the repository is being moved to a new server this week, > after which the URLs will be updated at scipy.org. > > Thanks! > > James Turner (Gemini). > > Bcc: various library authors > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From jturner at gemini.edu Sun Jul 11 15:10:14 2010 From: jturner at gemini.edu (James Turner) Date: Sun, 11 Jul 2010 15:10:14 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <4C3A1716.9060508@gemini.edu> Hi Thomas, I think this seems a good idea, even if it only solves part of the problem. Astrokits (or just scikits?) could be a good place for code to go whilst it's maturing, or for other reasons, with a low barrier to entry. We could encourage authors to follow the same standards/ guidelines as the core library, without enforcing them. A couple of limitations spring to mind. First, a proliferation of astrokits with overlapping functionality could perpetuate duplication and incoherence. As Perry said, there's no harm in doing something a couple of ways to see which is best, but if there are 5 different wrappers for WCSLIB, that's just confusing and makes it hard to focus on quality. WCS is not the best example, since you proposed putting that in the core, but in general this seems like something to keep in mind and it isn't solved just by associating different libraries. My second immediate concern would be distribution -- installing lots of astrokits could be a bigger problem for end users than a single library. Of course that could be solved by including most of the astrokits in our Gemini/STScI Python distribution once it's available, but these are early days and I don't think we're ready to promise the whole astronomy/science community a solution to its installation problems. Maybe others (eg. scisoft) would also pick them up though. Nevertheless, I think astrokits can only be better than lots of totally uncoordinated libraries and they could be a good path to accepting code into the core and/or encouraging contributions that might not happen otherwise. I think I'd back this proposal, unless others have objections I haven't considered yet -- but I'm not speaking for Perry's group, which ultimately manages Astrolib. Cheers, James. > Hi all, > > After reading all the replies, I have the following suggestion to make. > > The model scipy follows is to have a 'core' scipy package, and 'scikit' packages which share a common namespace, and which are meant as addons to the scipy package, but have not yet (or might never) make it to the main scipy package: > > http://www.scipy.org/scipy/scikits/ > > "Scipy Toolkits are independent and seperately installable projects hosted under a common namespace. Packages that are distributed in this way are here (instead of in monolithic scipy) for at least one of three general reasons. Each of these reasons use the same high-level namespace (scikits)." > > I think we can use this model, and that the following approach can be used here, namely: > > - start with a very basic 'astropy' package, with e.g. support for FITS/WCS > - agree to coordinate astronomy packages with a common namespace (e.g. 'astrokit', so for example, APLpy would become astrokit.aplpy). This can help us manage the namespace (as suggested in Joe Harrington's email) > - as astrokit modules mature, they can (if the authors are willing) be merged into the main 'astropy' package, once they have met a number of conditions, including e.g. unit tests, sphinx documentation, limited dependencies (e.g. numpy/scipy/matplotlib, and any package in the 'astropy' package), and compatible license. > > The advantage of this model is that this encourages the growth from the bottom up of a core astronomy package, which is manageable, as well as the independent development of other packages. It also means that the core package will be quite stable, because it will only accrete 'astrokit' modules as they become stable and mature. At the same time, it encourages developers to make their own innovative astrokit, but without commitment from the maintainers of the core package to accrete it in future. > > In passing, this also leaves the possibility for those who want to develop meta-pacakges of astrokit modules. Also, it will make it easier for those who want to build fully fledged astronomy python distributions. > > There have been many ideas passed around in the mailing list, but I think that some are maybe too ambitious. I do think that the idea suggested above is much more manageable. The concrete steps would be: > > - setup a central repository for the core packages, as well as astrokit pacakges (although we should leave the option for people to develop astrokit packages outside this repository too, and rely on trust and communication to avoid namespace clashes) > - start a core package with e.g. FITS and WCS support (e.g. pyfits + pywcs) > - set up a list of 'registered' astrokit names to avoid conflict. > - set up a list of recommendations and requirements for astrokit pacakges > - encourage developers of existing packages to use this namespace and follow the guidelines. This would be very little work in some cases, which is why I think it could work. > > Sharing a namespace is (I think) the first step to showing that we are willing to work on something coherent, and users would only see two top-level namespaces - astropy and astrokit, which would be a great improvement over the current situation where it is not immediately obvious which packages are astronomy related. > > Comments/suggestions/criticism welcome! > > Cheers, > > Tom From jturner at gemini.edu Sun Jul 11 16:33:43 2010 From: jturner at gemini.edu (James Turner) Date: Sun, 11 Jul 2010 16:33:43 -0400 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: References: Message-ID: <4C3A2AA7.8060907@gemini.edu> Hi Kelle et al., I'm not much of an IDL user personally, but I think this kind of documentation is great. I see there are a few different wiki sites cropping up, as well as what's on SciPy (eg. the topical software astro section and AstroLib): http://scipy.org/Topical_Software#head-4891fc4038123c6a923398225a728ec92ddb2780 http://scipy.org/AstroLib Clearly some of this old SciPy stuff could do with an update -- I might have a quick go at that. In the context of the "co-ordinating libraries" thread, how do people feel about the way information is distributed? Should more information go on the SciPy pages? What about Thomas's astrokits, if we do them? Should astropython.org be associated with scipy.org and Astrolib? I don't want to push the authors around, but it seems a good time to consider such questions. Whatever happens, I'd personally like to maintain a strong link between our AstroLib/AstroPy developments and SciPy -- but without limiting our flexibility too much (eg. avoiding some shortcomings that Joe pointed to). That way we combine forces with the wider scientific community and don't just get a bigger installation headache from moving to Python ;-). Cheers, James. > Hi All, > > I've sketched out a table on the AstroBetter wiki to use a "Switchers > Guide" from IDL. But it will also for developers so that they can see > what's already been migrated. > > http://www.astrobetter.com/wiki/tiki-index.php?page=Python+Switchers+Guide > > > My idea is to list the IDL Astrolib routines, their equivalent/similar > functionality python module, followed by a brief description. The table > currently includes the sum total of my Python knowledge so it's up to > the community to fill it in. I also included in the table a couple IDL > routines that probably have Python equivalents but that I just haven't > figured out yet. (i couldn't quite figure out what to make of the > Starlink package.) Also, feel free to add modules that are useful but > that don't have an IDL equivalent. It's a wiki, so edit away. You'll > just have to register.... > > I spent some time writing out the syntax for the pyfits header > manipulation stuff, but if you just want to include the package and > module, that'll be better than nothing. I also based the sections and > their order on the Astrolib > page: http://idlastro.gsfc.nasa.gov/contents.html > > btw, it is not my intention to duplicate the content that's already on > the NumPy for IDL users > site: http://mathesaurus.sourceforge.net/idl-numpy.html > > Anyway, for you python gurus out there...next time you're > procrastinating but still want to do something productive, fill in some > of the table! > > kelle > > -- > Kelle Cruz, PhD > Assistant Professor, Hunter College > Research Associate, AMNH > Visiting Associate, Caltech > http://kellecruz.com/ > http://astrobetter.com/ > 424.645.1334 From kellecruz at gmail.com Sun Jul 11 17:20:28 2010 From: kellecruz at gmail.com (Kelle Cruz) Date: Sun, 11 Jul 2010 17:20:28 -0400 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: <4C3A2AA7.8060907@gemini.edu> References: <4C3A2AA7.8060907@gemini.edu> Message-ID: Doesn't matter to me which wiki the Switchers Guide table lives on (AstroBetter /AstroLib/AstroPython) just as long as it continues to get filled in. These lists of packages just aren't doing it for me. Regarding the AstroLib site, I didn't even realize it was a wiki! I think if you really want people to edit it, i think it would help to make it more obvious that it's a wiki and how to register. Also, AstroLib is geared towards *developers* while I'm concerned with *users*. To add another question to the ones that James posed: Does anyone have any thoughts on which wiki the Switcher's Guide should live on? kelle On Sun, Jul 11, 2010 at 4:33 PM, James Turner wrote: > Hi Kelle et al., > > I'm not much of an IDL user personally, but I think this kind of > documentation is great. > > I see there are a few different wiki sites cropping up, as well as > what's on SciPy (eg. the topical software astro section and AstroLib): > > > http://scipy.org/Topical_Software#head-4891fc4038123c6a923398225a728ec92ddb2780 > http://scipy.org/AstroLib > > Clearly some of this old SciPy stuff could do with an update -- I > might have a quick go at that. > > In the context of the "co-ordinating libraries" thread, how do people > feel about the way information is distributed? Should more information > go on the SciPy pages? What about Thomas's astrokits, if we do them? > Should astropython.org be associated with scipy.org and Astrolib? I > don't want to push the authors around, but it seems a good time to > consider such questions. > > Whatever happens, I'd personally like to maintain a strong link > between our AstroLib/AstroPy developments and SciPy -- but without > limiting our flexibility too much (eg. avoiding some shortcomings that > Joe pointed to). That way we combine forces with the wider scientific > community and don't just get a bigger installation headache from > moving to Python ;-). > > Cheers, > > James. > > > Hi All, >> >> I've sketched out a table on the AstroBetter wiki to use a "Switchers >> Guide" from IDL. But it will also for developers so that they can see what's >> already been migrated. >> >> http://www.astrobetter.com/wiki/tiki-index.php?page=Python+Switchers+Guide< >> http://www.astrobetter.com/wiki/tiki-index.php?page=Python+Switchers+Guide >> > >> >> My idea is to list the IDL Astrolib routines, their equivalent/similar >> functionality python module, followed by a brief description. The table >> currently includes the sum total of my Python knowledge so it's up to the >> community to fill it in. I also included in the table a couple IDL routines >> that probably have Python equivalents but that I just haven't figured out >> yet. (i couldn't quite figure out what to make of the Starlink package.) >> Also, feel free to add modules that are useful but that don't have an IDL >> equivalent. It's a wiki, so edit away. You'll just have to register.... >> >> I spent some time writing out the syntax for the pyfits header >> manipulation stuff, but if you just want to include the package and module, >> that'll be better than nothing. I also based the sections and their order >> on the Astrolib page: http://idlastro.gsfc.nasa.gov/contents.html >> >> btw, it is not my intention to duplicate the content that's already on the >> NumPy for IDL users site: >> http://mathesaurus.sourceforge.net/idl-numpy.html >> >> Anyway, for you python gurus out there...next time you're procrastinating >> but still want to do something productive, fill in some of the table! >> >> kelle >> >> -- >> Kelle Cruz, PhD >> Assistant Professor, Hunter College >> Research Associate, AMNH >> Visiting Associate, Caltech >> http://kellecruz.com/ >> http://astrobetter.com/ >> 424.645.1334 >> > > -- Kelle Cruz, PhD http://kellecruz.com/ 424.645.1334 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jturner at gemini.edu Sun Jul 11 19:19:33 2010 From: jturner at gemini.edu (James Turner) Date: Sun, 11 Jul 2010 19:19:33 -0400 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: References: <4C3A2AA7.8060907@gemini.edu> Message-ID: <4C3A5185.8040609@gemini.edu> > Regarding the AstroLib site, I didn't even realize it was a wiki! I > think if you really want people to edit it, i think it would help to > make it more obvious that it's a wiki and how to register. Also, > AstroLib is geared towards *developers* while I'm concerned with *users*. Yes, there's some truth to that. Those sections could be more inviting or informative in general. I suspect that may have been one factor in people deciding to host their own libraries rather than contributing to AstroLib, though no-one here has said so. However, it's actually the WHOLE of scipy.org that's a wiki (I even just did a minor edit on the front page...). I think there's no reason not to put user documentation on the same pages; there just isn't any there yet. Cheers, James. From erik.tollerud at gmail.com Sun Jul 11 22:09:50 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sun, 11 Jul 2010 19:09:50 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C3A1716.9060508@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> <4C3A1716.9060508@gemini.edu> Message-ID: I think the "astrokits" idea is a good one - (definitely not melding into scikits, though - that's too big of an umbrella community for me to be comfortable with). I'm somewhat concerned about resulting long import statements("from astrokits.apackage.amodule.submodule import SomeReallyLongClassOrFunctionName"), but I guess that might be the necessary price to pay. I still think, though, that using that same sort of infrastructure, while relaxing the namespace requirement, is a better solution - that is, you can have an organizational structure like Thomas suggests without requiring that the packages start with "astrokits" - just strongly recommend the guidelines of consistent documentation (ideally, I think, using sphinx) and consistent use of the standard setup.py schemes (most important here is consistency in compiling any Fortran or C codes... or better yet have everyone use Cython for any interface layers), and you get the same advantages. As James rightly pointed out, if you go the astrokits route, you still don't really solve the installation problem because everything is installed separately - all that's really important is a central listing of the "sanctioned" astrokits. Another thing that would probably be a very good thing to do if possible would be to allow for a "proprietary time" - that is, allow people to develop something only they have access to until they publish the first paper or whatever using/describing that tool, and then afterwards it goes public as an astrokit for everyone to see. So in my mind it seems reasonable to set up a shared repository/web site with the option (but not requirement) of using an "astrokits" namespace, with the understanding that when possible, the core "astropy" modules would be used to avoid duplicating effort (and the code must follow the aforementioned rather loose standards). On Wed, Jul 7, 2010 at 6:01 PM, Kelle Cruz wrote: > Well, Eli and Tom have already done the listing on AstroPython: > http://www.astropython.org/resources > or > http://www.astropython.org/resources?sort=tag I hadn't realized how complete this listing is - it's a great start, but the one thing that's missing for what I had in mind is an access API or rigorous entry standard. That is, it has to be possible to write scripts that work like pip to grab download links and use them to automatically build and install packages. This isn't at all hard to do, it just requires either a secondary database with an API, or a slightly more standardized entry format with source download links (which don't have to be hosted on the same page), and some standardized description of the non-python requirements. With that, it'd be pretty easy to write something like an "astropip" that could be used to automate installation/upgrade of any of the listed packages. An "astropip" has the advantage of allowing us to just roll our own sorts of metapackages and get around the setuptools/distribute shortcomings. The more I think about it, that might be the way to go. > - I think a AAS Splinter is a great idea...looks like the deadline isn't > until Dec 1 for Seattle, but we should get on it since they are assigned on > a first-come, first-served basis based on room availability. ?It might make > more sense to do Boston in May 2011 because people won't be as busy with > other meeting things and it should be nice there in spring I would think Seattle might be better because a lot more people will be there... and anyway, a dreary Seattle winter day is much more likely to keep people in a room talking about python libraries, isn't it? (Also, for full disclosure, I'm on the west coast, so I personally would prefer it given that I'm not sure I can make it to Boston...) On Thu, Jul 8, 2010 at 11:03 AM, Perry Greenfield wrote: > Well, more the former, but also to enable something along the latter (though > not necessarily part of a single install). When things are here and there, > it is more difficult to package those together in an automatic way. Not > impossible, just more work and more ways for things to go wrong. You are right about this, but as I've described above, I think with better consistency in how all the packages are installed, it'll be far easier to deal with these problems. And in my mind, the gains are much greater because it give access to a much larger brain-trust of people who want a fair amount of freedom (at least initially) in how they're doing their project. > But trac is fairly portable since it is widely used. Migrating the info from > some of the others can be a ?more difficult problem (lock-in issues). As for > svn, git, hg, bazaar issues, nothing is going to satisfy everyone, I agree The Trac thing is a personal preference - I admit it might well be the best all-around compromise, although I still think allowing the option of external hosting might be a good idea, as long as they have standardized entries in whatever listing is used. I want to reiterate, though, that I think it's important to *not* use svn over the other options I mentioned - distributed version control systems (e.g. bzr, hg, or git) are far more suited to the kind of development I think most astronomers typically are used to, as they can make their own local copy, do their own thing without any access to the central repository, and later merge it back in. > That works to a certain level. But before long, there are n flavors of > representing this and that, and combining 10 packages like this to use in > your own ?code can get to be a real chore. You'll spend a lot of time moving > from one convention to another (and not catching some bugs). I don't think > making everyone conform is a good solution, but eventually standardizing on > core libraries is a very good thing in the long run. I think there is a > reasonable middle ground on this kind of thing. You're absolutely right about the problems of fragmentation and the virtues of later standardization. But right now, given that those core libraries don't exist, the freedom to experiment is also very important. Of course, I'm biased in that I'm working for a rather more pythonic/object-oriented flavor than the traditional IDL-like libraries so I'm leveraged a bit towards the experiment side... but I think the point is still valid, so as you say, a general middle ground is important. -- Erik Tollerud From perry at stsci.edu Mon Jul 12 09:03:20 2010 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 12 Jul 2010 09:03:20 -0400 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: <4C3A2AA7.8060907@gemini.edu> References: <4C3A2AA7.8060907@gemini.edu> Message-ID: On Jul 11, 2010, at 4:33 PM, James Turner wrote: > Hi Kelle et al., > > I'm not much of an IDL user personally, but I think this kind of > documentation is great. > > I see there are a few different wiki sites cropping up, as well as > what's on SciPy (eg. the topical software astro section and AstroLib): > > http://scipy.org/Topical_Software#head-4891fc4038123c6a923398225a728ec92ddb2780 > http://scipy.org/AstroLib > > Clearly some of this old SciPy stuff could do with an update -- I > might have a quick go at that. > > In the context of the "co-ordinating libraries" thread, how do people > feel about the way information is distributed? Should more information > go on the SciPy pages? What about Thomas's astrokits, if we do them? > Should astropython.org be associated with scipy.org and Astrolib? I > don't want to push the authors around, but it seems a good time to > consider such questions. > > Whatever happens, I'd personally like to maintain a strong link > between our AstroLib/AstroPy developments and SciPy -- but without > limiting our flexibility too much (eg. avoiding some shortcomings that > Joe pointed to). That way we combine forces with the wider scientific > community and don't just get a bigger installation headache from > moving to Python ;-). > > Cheers, > > James. > At the same time, if the astronomy effort is getting large enough, the main wiki doesn't have to be at scipy. I'm flexible as to where people wish to host it. We can use the trac wiki for our new astrolib repository (should be announced very soon now...) or some other place. From erik.tollerud at gmail.com Mon Jul 12 14:56:09 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 12 Jul 2010 11:56:09 -0700 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: <4C3A5185.8040609@gemini.edu> References: <4C3A2AA7.8060907@gemini.edu> <4C3A5185.8040609@gemini.edu> Message-ID: I would say it makes sense to have a single wiki that everyone is encouraged to use - in contrast to the software hosting, a wiki can be much more flexible standards-wise, but its more important that the information all be easily reachable from one place. But I do think it makes more sense to have that wiki be wherever the software repository ends up, rather than necessarily the scipy/astrolib site. The one that probably wins out content-wise is the one that the main package-builders actually contribute to, and putting it in the same place (ideally, with an individual wiki tied to each individual project's source, like bitbucket or trac does). Also, I personally think the astropython.org and astrobetter.com are a bit better-looking and easier to use than the scipy/astrolib site... and the possibility to alter such things rather than being locked into the scipy design scheme is a plus. > Yes, there's some truth to that. Those sections could be more inviting > or informative in general. I suspect that may have been one factor in > people deciding to host their own libraries rather than contributing > to AstroLib, though no-one here has said so. However, it's actually > the WHOLE of scipy.org that's a wiki (I even just did a minor edit on > the front page...). > > I think there's no reason not to put user documentation on the same > pages; there just isn't any there yet. Well, I think a wiki is definitely *not* the place for actual documentation at the developer level. In Python, that is much easier done within the source code itself (and/or with external files in the source tree) by way of sphinx. I know that was actually a reason why I couldn't make headway in astrolib when I first looked at it - the documentation at the developer level was difficult to follow. User-level tutorials, overviews, and guides (like the switching guide) are much better suited for the wiki, though, as they are more loosely-coupled to the source code and don't have to be tied to a particular project. Then again, we may be talking about different things regarding "developer" and "user." In astronomy, lots of "users" will be writing scripts that dig pretty deeply into what one might think of as "developer" territory, so I'm not clear on where the line is... -- Erik Tollerud From jturner at gemini.edu Mon Jul 12 15:53:12 2010 From: jturner at gemini.edu (James Turner) Date: Mon, 12 Jul 2010 15:53:12 -0400 Subject: [AstroPy] [SciPy-User] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> Message-ID: <4C3B72A8.5070404@gemini.edu> Hi Erik, I almost overlooked your first email in my SciPy folder... Astropysics is certainly looking interesting. As regards the code structure, I would not personally expect AstroLib to be limited to a collection of functional library routines. I'm particularly keen to use classes to present data processing code in a more accessible way, with the algorithmic structure exposed at the top level and the bookeeping details hidden (but accessible) underneath. Within Gemini, we already have a not-quite-released AstroData class on top of PyFITS that understands astronomical data at a higher level (currently it mainly acts as a generalized interface to metadata). I'd like us to extend that functionality to operators and transformations, but we're always a bit underresourced, so things get done on an as-needed basis. Cheers, James. > For the last couple years I (and a few others) have been working on > Astropysics (http://packages.python.org/Astropysics/ and source code > at https://launchpad.net/astropysics). This initially spun out of my > personal need for some of the utilities these other projects discussed > (e.g. IDL astron-like functions), but I've been turning it in a > slightly different direction. Astropysics now is being written as a > more "pythonic" way of doing the same tasks instead of starting from > cloning IDL procedural techniques. I'm trying to leverage all the > existing tools that have been written in python for other domains > (e.g. Enthought Traits to easily make good GUIs, sphinx to make useful > documentation, distribute to make packaging easier). Further, where > useful, I use object-oriented techniques instead of the procedural > approach people familiar with IDL and IRAF are used to. The idea is > to start doing things on "Spectrum" and "CCDImage" objects instead of > passing arrays into functions and so on, and each astronomer can then > sub-class these classes to do whatever they want. The aim here is to > eliminate the tendancy for people to take public codes and change them > internally, leading to some very painful efforts in disentangling > versions. > > So far, Astropysics includes the following modules: > * constants ? physical constants and cosmological calculations > * coords ? coordinate classes and coordinate conversions > * models ? model and data-fitting classes and tools > * objcat ? flexible, dynamically updated object catalog > * ccd ? image processing tools > * spec ? spectrum and SED classes and tools > * phot ? photometry/flux measurement classes and tools > * obstools ? miscellaneous tools for observation > * io ? data input/output classes and tools > * plotting ? astronomy-oriented matplotlib and mayavi plotting tools > * pipeline ? classes for data reduction and analysis pipelines > * utils ? utility classes and functions > > And these GUI tools, which require Enthought Traits (and Enthought > chaco for plots in the GUI): > * Spylot ? Spectrum Plotter > * Fitgui ? Interactive Curve Fitting > * Spectarget ? MOS Targeting > > I'm making a concerted effort to document everything in a consistent > manner using sphinx (http://sphinx.pocoo.org) - the resulting > documents (http://packages.python.org/Astropysics/) end up being much > more useful. > > I also try to bind in python wrappers around external tools like > sextractor, Mike Blanton's kcorrect, MOS mask design tools, and > galfit (planned) ... this happens mostly as I need them for my own > science. But this cuts down on the time re-implementing standard > programs that aren't necessarily worth re-working. > > There are two main things left before I consider it "releasable" in > terms of the baseline functionality: First, it needs WCS support to be > integrated in the coords framework, and second, the objcat package > should have a web server module that lets you show/edit the catalog > over the internet instead of within python. Another major goal I'd > like to do when I or someone else gets a chance is something like ATV > (http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI > framework that the other gui tools are written in. > > I hope this explains the direction I have in mind for astropysics. As > far as I can tell, I have a slightly different philosophy - I'm trying > to set up something of a framework of my design, rather than a > function library. This is why I have not been working on adding it to > astropy, because astropy seems much more like the traditional > library... That and I'm not a fan of the Trac project management > system. At any rate, I think there's definitely room for all in the > community. And if anyone likes what they see, feel free to drop me a > line if you want to contribute. From jturner at gemini.edu Mon Jul 12 16:04:37 2010 From: jturner at gemini.edu (James Turner) Date: Mon, 12 Jul 2010 16:04:37 -0400 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: References: <4C3A2AA7.8060907@gemini.edu> <4C3A5185.8040609@gemini.edu> Message-ID: <4C3B7555.60004@gemini.edu> > Well, I think a wiki is definitely *not* the place for actual > documentation at the developer level. In Python, that is much easier > done within the source code itself (and/or with external files in the > source tree) by way of sphinx. Yes, I think/hope everyone wants to use Sphinx for documentation (as opposed to general information). > In astronomy, lots of "users" will be writing scripts that dig pretty > deeply into what one might think of as "developer" territory, so I'm > not clear on where the line is... It's not very well defined, but I'd consider developers to be the people who care about structuring code, using version control, writing tests and the overall quality of the end product -- those who will take the time to understand developer documentation as far as they need it to do the job properly. For "users", the bar has to be set pretty low in terms of explaining the basics and focusing on getting things done quickly. Just my 11 pesos. Cheers, James. From kellecruz at gmail.com Mon Jul 12 16:43:11 2010 From: kellecruz at gmail.com (Kelle Cruz) Date: Mon, 12 Jul 2010 16:43:11 -0400 Subject: [AstroPy] Python documentation (was IDL to Python Switchers Guide) In-Reply-To: <4C3B7555.60004@gemini.edu> References: <4C3A2AA7.8060907@gemini.edu> <4C3A5185.8040609@gemini.edu> <4C3B7555.60004@gemini.edu> Message-ID: Yep, I agree with James. I think of "users" as folks who yes, will be writing scripts, but mostly calling on the AstroLib routines to do both the heavy lifting (interpolation, photometry, spectral extraction) and the mundane (fits header manipulation, coord conversion, reading ascii tables). Also, when I think of "users", I think of undergrads and grad students who don't need to/can't/won't dig into things too much and, as James said, just need to get things done quickly, efficiently, and hopefully effectively without getting bogged down in the guts of the code. So when I say users, i think the best translation for the folks on this list would be "newbies"...ppl who may have even never edited their .cshrc before.... Based on the responses so far, it seems to make sense for the Switchers Guide to stay on AstroBetter...especially since I'm the one doing most of the editing at this point! Which is fine...it's been a great way for me to learn. I have an upcoming observing run and two long plane rides (with WiFi) where I'll probably do another round of additions. Any comments/suggestions you have for the guide are welcome (off list). kelle On Mon, Jul 12, 2010 at 4:04 PM, James Turner wrote: > In astronomy, lots of "users" will be writing scripts that dig pretty >> deeply into what one might think of as "developer" territory, so I'm >> not clear on where the line is... >> > > It's not very well defined, but I'd consider developers to be the > people who care about structuring code, using version control, writing > tests and the overall quality of the end product -- those who will take > the time to understand developer documentation as far as they need it > to do the job properly. For "users", the bar has to be set pretty low > in terms of explaining the basics and focusing on getting things done > quickly. Just my 11 pesos. -- Kelle Cruz, PhD http://kellecruz.com/ 424.645.1334 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wkerzendorf at googlemail.com Wed Jul 14 11:02:52 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Wed, 14 Jul 2010 17:02:52 +0200 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C3B72A8.5070404@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> Message-ID: <4C3DD19C.1040609@gmail.com> Hello, So finally I also managed to put some of my libraries online at http://www.mso.anu.edu.au/~wkerzend/astropy. I have followed the discussion over the last few days and I think its great. Of course here my 2 cents ;-): I have read this once or twice already: I think there's a need for common base classes so that we can use these new python libraries together. This is very important especially now that this python astronomy library starts to take shape. Time and coordinate base classes were mentioned. I think pyephem might be the way to go here as Rhodesmill has already implemented quite a bit of both. The next thing is a 2D image class and I read from James's email that there's AstroData in preparation at Gemini . Is there any release ETA on AstroData? With pyspec I have a prototype for a 1D spectrum and echelle class. It can easily read spectra in a column seperated format: data=spectrum('myspec.dat'). You can easily slice spectra: data2=data[4000.:4050.] (if you give it a float for slicing it will use the wavelength instead of the array index). You can do arithmetic with spectra and it does a simple linear interpolate on the second spectrum to match the wavelength grid of the first: data3=data1/data2. I hope that we can get some base classes fairly soon, so that the development of new tools already includes them. Distribution wise I think there's many choices at the moment. I suggest trying something like scipy superpack for the moment. It's basically a shell script that will download the package tgzs and will run python setup.py install on them. It works well and is easy to write. The advantage of this system is that it is fairly easy to implement and the community can see what will come in the next year from distutil and the likes. I'm really excited about all these developments and hope that they will be a big improvement for astronomy. Cheers Wolfgang On 12/07/10 9:53 PM, James Turner wrote: > Hi Erik, > > I almost overlooked your first email in my SciPy folder... > > Astropysics is certainly looking interesting. As regards the code > structure, I would not personally expect AstroLib to be limited to a > collection of functional library routines. I'm particularly keen to > use classes to present data processing code in a more accessible way, > with the algorithmic structure exposed at the top level and the > bookeeping details hidden (but accessible) underneath. Within Gemini, > we already have a not-quite-released AstroData class on top of PyFITS > that understands astronomical data at a higher level (currently it > mainly acts as a generalized interface to metadata). I'd like us to > extend that functionality to operators and transformations, but we're > always a bit underresourced, so things get done on an as-needed basis. > > Cheers, > > James. > > >> For the last couple years I (and a few others) have been working on >> Astropysics (http://packages.python.org/Astropysics/ and source code >> at https://launchpad.net/astropysics). This initially spun out of my >> personal need for some of the utilities these other projects discussed >> (e.g. IDL astron-like functions), but I've been turning it in a >> slightly different direction. Astropysics now is being written as a >> more "pythonic" way of doing the same tasks instead of starting from >> cloning IDL procedural techniques. I'm trying to leverage all the >> existing tools that have been written in python for other domains >> (e.g. Enthought Traits to easily make good GUIs, sphinx to make useful >> documentation, distribute to make packaging easier). Further, where >> useful, I use object-oriented techniques instead of the procedural >> approach people familiar with IDL and IRAF are used to. The idea is >> to start doing things on "Spectrum" and "CCDImage" objects instead of >> passing arrays into functions and so on, and each astronomer can then >> sub-class these classes to do whatever they want. The aim here is to >> eliminate the tendancy for people to take public codes and change them >> internally, leading to some very painful efforts in disentangling >> versions. >> >> So far, Astropysics includes the following modules: >> * constants ? physical constants and cosmological calculations >> * coords ? coordinate classes and coordinate conversions >> * models ? model and data-fitting classes and tools >> * objcat ? flexible, dynamically updated object catalog >> * ccd ? image processing tools >> * spec ? spectrum and SED classes and tools >> * phot ? photometry/flux measurement classes and tools >> * obstools ? miscellaneous tools for observation >> * io ? data input/output classes and tools >> * plotting ? astronomy-oriented matplotlib and mayavi plotting tools >> * pipeline ? classes for data reduction and analysis pipelines >> * utils ? utility classes and functions >> >> And these GUI tools, which require Enthought Traits (and Enthought >> chaco for plots in the GUI): >> * Spylot ? Spectrum Plotter >> * Fitgui ? Interactive Curve Fitting >> * Spectarget ? MOS Targeting >> >> I'm making a concerted effort to document everything in a consistent >> manner using sphinx (http://sphinx.pocoo.org) - the resulting >> documents (http://packages.python.org/Astropysics/) end up being much >> more useful. >> >> I also try to bind in python wrappers around external tools like >> sextractor, Mike Blanton's kcorrect, MOS mask design tools, and >> galfit (planned) ... this happens mostly as I need them for my own >> science. But this cuts down on the time re-implementing standard >> programs that aren't necessarily worth re-working. >> >> There are two main things left before I consider it "releasable" in >> terms of the baseline functionality: First, it needs WCS support to be >> integrated in the coords framework, and second, the objcat package >> should have a web server module that lets you show/edit the catalog >> over the internet instead of within python. Another major goal I'd >> like to do when I or someone else gets a chance is something like ATV >> (http://www.physics.uci.edu/~barth/atv/) utilizing the same TraitsGUI >> framework that the other gui tools are written in. >> >> I hope this explains the direction I have in mind for astropysics. As >> far as I can tell, I have a slightly different philosophy - I'm trying >> to set up something of a framework of my design, rather than a >> function library. This is why I have not been working on adding it to >> astropy, because astropy seems much more like the traditional >> library... That and I'm not a fan of the Trac project management >> system. At any rate, I think there's definitely room for all in the >> community. And if anyone likes what they see, feel free to drop me a >> line if you want to contribute. > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From perry at stsci.edu Wed Jul 14 17:00:01 2010 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 14 Jul 2010 17:00:01 -0400 Subject: [AstroPy] Astrolib relocated to new hosting site References: Message-ID: <4A18249A-6717-4FC0-8F0B-55C361FDEAB9@stsci.edu> We've relocated the astrolib repository to a new hosting site. This site we hope will provide both reliable access and ease of giving access to people that are willing to make code contributions or modifications. Previously we were hosting it at STScI but regulations required by NASA for their computer systems made the process of adding access to repositories for non-employees cumbersome. Our hope is that this repository will become a center of astronomy community contributed code for Python. If you have written a module that you believe is useful for others and would like to make it available, please consider keeping the code in the astrolib repository. There are a number of benefits to doing so: - Having your code sitting close to other astronomy tools will help other contributors be aware of the tools contained with yours and that could very well influence their tools (cross-pollination, awareness of inconsistent interfaces, etc.), or provide very useful feedback or input for your tools (and perhaps even code modifications or enhancements for your tools). - It makes it easier for those packaging toolsets for astronomy to include your module since it is one less site to determine details of how to extract the right version for testing and distribution. - You will get some free testing and building infrastructure as part of hosting it there (we plan to run regular builds and regression tests on several popular platforms. - It will help in generating more consistent documentation (perhaps not useful to you, but certainly to users). We (initially anyway) wish to have a very lightweight process for contributed code. If the primary developer team for the site thinks the code is potentially useful, you will be given commit privileges to add the code. Although the commit privileges allow one to modify any code on astrolib, we expect people with privileges to only modify code on modules and packages they are involved with (and we will list the leaders of each package or module so that people interested in making contributions or changes can contact them for permission to make changes). We will encourage at least minimum level of documentation and tests (in order to be included with any distributed software), but we intend to err on the side of less restrictiveness. We would like these things to be driven mostly by social pressure rather than by bureaucratic processes. Many of the details of how this will work haven't really been specified, and we intend to go with what works best in practice (i.e., let's see how things work first before making rules). There are some minimal requirements. The software: - must be useful for astronomy (but we don't rule out general purpose tools that aren't available else where (we'll worry about contributing to scipy or other more general purpose sets when the software proves useful in other fields) - must have an open source license (preferably BSD, MIT, LGPL, but we can accept GPL) - must be usable from Python - may call other languages or even executables that are not part of the repository. - may include embedded C, C++, Fortran code called from Python To see what the repository currently contains: http://svn6.assembla.com/svn/astrolib/ (subversion URL) https://trac6.assembla.com/astrolib (trac URL) We are very interested in feedback about what would make this a more useful repository of astronomical software for Python. We'll be publishing on the site's wiki more details later about what we would like to see in contributed code (documentation, installation conventions, etc). To contribute a package or module, or to get commit privileges to existing modules, contact Perry Greenfield (perry at stsci.edu) or James Turner (jturner at gemini.edu) From jturner at gemini.edu Thu Jul 15 12:40:08 2010 From: jturner at gemini.edu (James Turner) Date: Thu, 15 Jul 2010 12:40:08 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C3DD19C.1040609@gmail.com> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> Message-ID: <4C3F39E8.7090309@gemini.edu> Hi Wolfgang, > I have read this once or twice already: I think there's a need for > common base classes so that we can use these new python libraries > together. This is very important especially now that this python > astronomy library starts to take shape. Classes and data structures seem to be another common theme in this thread that probably deserve more discussion. A few people have mentioned time and co-ordinates. I was wondering whether Anne or someone else has been following the NumPy datetime discussion and whether that's at all relevant (since I know she is more active on that list than I am)? > The next thing is a 2D image class and I read from James's email > that there's AstroData in preparation at Gemini. Is there any > release ETA on AstroData? It was very close, with the documentation just being tidied up before a first internal "release". I don't know exactly when Kathleen will be comfortable doing a public release, but I can check. It would be good to get it out sooner rather than later. > With pyspec I have a prototype for a 1D spectrum and echelle class. It > can easily read spectra in a column seperated format: > data=spectrum('myspec.dat'). You can easily slice spectra: > data2=data[4000.:4050.] (if you give it a float for slicing it will use > the wavelength instead of the array index). You can do arithmetic with > spectra and it does a simple linear interpolate on the second spectrum > to match the wavelength grid of the first: data3=data1/data2. That sounds convenient. Apparently several people/groups are working on these kinds of high-level objects and some standardization may indeed be helpful, otherwise I'll be using AstroData for part of my processing and then find I need to pick apart my object to construct your 1D spectrum from pyspec, then do something different again from Astropysics or PyEphem and so on. A few standard classes would probably help significantly in providing a good focal point. > I'm really excited about all these developments and hope that they will > be a big improvement for astronomy. Yes, I hope several projects are ready to come together to make a nice new platform for astronomy, after STScI and the SciPy people have figured out the basics for us. Cheers, James. From eemselle at eso.org Thu Jul 15 13:08:15 2010 From: eemselle at eso.org (Eric Emsellem) Date: Thu, 15 Jul 2010 19:08:15 +0200 Subject: [AstroPy] Fitting ellipses and higher order moments Message-ID: <4C3F407F.6080809@eso.org> Dear all, as I think this is rather specific "astro" tool: would anybody be aware of a python routine/module doing ellipse-fitting (with higher order Fourier moments) on photometric images? This obviously exists in IRAF, MIDAS, idl, ... but I am looking for an integrated python version. thanks and cheers Eric From erik.tollerud at gmail.com Thu Jul 15 22:49:25 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Thu, 15 Jul 2010 19:49:25 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C3F39E8.7090309@gemini.edu> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> Message-ID: Hi James and Wolfgang, > Classes and data structures seem to be another common theme in this > thread that probably deserve more discussion. A few people have > mentioned time and co-ordinates. I was wondering whether Anne or > someone else has been following the NumPy datetime discussion and > whether that's at all relevant (since I know she is more active on > that list than I am)? I definitely agree classes like this are a good way to make scientific codes much easier to work with and cleaner to manage. I would also like to note that I've already implemented a pretty complete and documented set of coordinate classes, transforms between them, and the related time-keeping in astropysics (see http://packages.python.org/Astropysics/coremods/coords.html and http://packages.python.org/Astropysics/coremods/obstools.html). This is in pure python (with numpy arrays, of course) and are ready to be subclassed right now if you want to use them. It's also dead-trivial to write transforms back to standard coordinate types. It's not as full-featured as pyephem right now, but is pure python, which makes installation/distribution easier, and crucially in my view, easier to extend. (To be clear this is not intended at all as a bash against pyephem - just a different design choice, as Brandon himself pointed out) As for the question, I'm pretty sure the datetime enhancements are supposed to go into numpy 2.0, because it requires an ABI change... I've never been clear on exactly when that's going to be released as it's been "soon" for some time, but I think they're shooting for end-of-summer or fall? (note that numpy 2.0 will also support python 3.0 along with a paralell scipy release). But I don't quite understand what the connection is between that and coordinate/time-keeping classes...? The datetime class in standard python works just fine for individual objects... > That sounds convenient. Apparently several people/groups are working > on these kinds of high-level objects and some standardization may > indeed be helpful, otherwise I'll be using AstroData for part of my > processing and then find I need to pick apart my object to construct > your 1D spectrum from pyspec, then do something different again from > Astropysics or PyEphem and so on. A few standard classes would > probably help significantly in providing a good focal point. Agreed completely on this, and that was my motivation behind making astropysics as an integrated set of packages... For example, the spec.Spectrum class in astropysics uses phot.Band objects (representing photometric bands) to do various calibrations (and optionally vice versa), and that's not possible if there isn't a well-documented API (hence I had to write them both along with their API). The main problem is how to integrate the base classes in a way that makes everybody happy and doesn't leave anyone thinking their work is ignored or uncredited, as perhaps shown by my perhaps not-so-subtle advertisements in the comments above... I can definitely see ways to integrate, say, pyspec with the astropysics Spectrum class, but it would be a fair bit of work given the rather different internals. Or would it be better to initially ignore all previous work, settle on a simple API, and then leave it to packages to implement the details? Although that seems problematic to me as the details are a lot of what matters in this sort of highly numerical work... -- Erik Tollerud From wkerzendorf at googlemail.com Fri Jul 16 05:01:42 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Fri, 16 Jul 2010 11:01:42 +0200 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> Message-ID: <4C401FF6.3090105@gmail.com> Hello guys, I think astropysics looks like a very good start for the coordinate class. As you said we should at the moment focus on having python-only classes for the base-level. That makes it easy to distribute. Once a good API has been established and there's complaints about speed, we can switch over to c or fortran implementations with the same API. A monolithic distribution is not so good. I think these baselevel classes should, be very modular. We can probably achieve the best exceptance when these base classes are lean and mean. Like the unix tools, each one of them should only provide a very limited set of functionality. A good start might be pyAstroTime and pyAstroCoords or so. That's where raiding and plundering the existing code base comes in. We can use some of Erik's and Brandon's stuff and others. I think we can easily make a working prototype and the build from there. As suggested, we need to be careful not to ignore anyone. But I think that's easily done by making groups from this community, that take care of a single implementation. Everyone who's interested in contributing, can join. That way the workload is shared and it is build by the community for the community. What do you guys think? Cheers Wolfgang On 16/07/10 4:49 AM, Erik Tollerud wrote: > Hi James and Wolfgang, > >> Classes and data structures seem to be another common theme in this >> thread that probably deserve more discussion. A few people have >> mentioned time and co-ordinates. I was wondering whether Anne or >> someone else has been following the NumPy datetime discussion and >> whether that's at all relevant (since I know she is more active on >> that list than I am)? > I definitely agree classes like this are a good way to make scientific > codes much easier to work with and cleaner to manage. I would also > like to note that I've already implemented a pretty complete and > documented set of coordinate classes, transforms between them, and the > related time-keeping in astropysics (see > http://packages.python.org/Astropysics/coremods/coords.html and > http://packages.python.org/Astropysics/coremods/obstools.html). This > is in pure python (with numpy arrays, of course) and are ready to be > subclassed right now if you want to use them. It's also dead-trivial > to write transforms back to standard coordinate types. It's not as > full-featured as pyephem right now, but is pure python, which makes > installation/distribution easier, and crucially in my view, easier to > extend. (To be clear this is not intended at all as a bash against > pyephem - just a different design choice, as Brandon himself pointed > out) > > As for the question, I'm pretty sure the datetime enhancements are > supposed to go into numpy 2.0, because it requires an ABI change... > I've never been clear on exactly when that's going to be released as > it's been "soon" for some time, but I think they're shooting for > end-of-summer or fall? (note that numpy 2.0 will also support python > 3.0 along with a paralell scipy release). But I don't quite > understand what the connection is between that and > coordinate/time-keeping classes...? The datetime class in standard > python works just fine for individual objects... > > >> That sounds convenient. Apparently several people/groups are working >> on these kinds of high-level objects and some standardization may >> indeed be helpful, otherwise I'll be using AstroData for part of my >> processing and then find I need to pick apart my object to construct >> your 1D spectrum from pyspec, then do something different again from >> Astropysics or PyEphem and so on. A few standard classes would >> probably help significantly in providing a good focal point. > Agreed completely on this, and that was my motivation behind making > astropysics as an integrated set of packages... For example, the > spec.Spectrum class in astropysics uses phot.Band objects > (representing photometric bands) to do various calibrations (and > optionally vice versa), and that's not possible if there isn't a > well-documented API (hence I had to write them both along with their > API). > > The main problem is how to integrate the base classes in a way that > makes everybody happy and doesn't leave anyone thinking their work is > ignored or uncredited, as perhaps shown by my perhaps not-so-subtle > advertisements in the comments above... I can definitely see ways to > integrate, say, pyspec with the astropysics Spectrum class, but it > would be a fair bit of work given the rather different internals. Or > would it be better to initially ignore all previous work, settle on a > simple API, and then leave it to packages to implement the details? > Although that seems problematic to me as the details are a lot of what > matters in this sort of highly numerical work... > > > From perry at stsci.edu Fri Jul 16 08:35:25 2010 From: perry at stsci.edu (Perry Greenfield) Date: Fri, 16 Jul 2010 08:35:25 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C401FF6.3090105@gmail.com> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> Message-ID: I'd like to tackle the WCS issue first since there are already several flavors out there and I really wonder if that is necessary. Mike Droettboom is going to take a look at the others and see what the differences are in the next week. It would be good to get some convergence on that. But I don't think that can stop others from surveying and comparing what is out there with regard to coordinates or time. As far as the pure python aspect goes, I don't know if I would be so definitive on that. If there is already a good time or coordinates library in C that has been very well tested, it might make sense to use that. It isn't usually a big deal to distribute C code if it has no other dependencies. Fortran is a different issue. And there are many tricky issues with regard to coordinate systems. If reimplemented in pure python I'd suggest that we do a exhaustive test comparison (mostly automatically generated if possible) with a well tested library to make sure that it was well validated. Perry On Jul 16, 2010, at 5:01 AM, Wolfgang Kerzendorf wrote: > Hello guys, > > I think astropysics looks like a very good start for the coordinate > class. > As you said we should at the moment focus on having python-only > classes > for the base-level. That makes it easy to distribute. Once a good API > has been established and there's complaints about speed, we can switch > over to c or fortran implementations with the same API. > > A monolithic distribution is not so good. I think these baselevel > classes should, be very modular. We can probably achieve the best > exceptance when these base classes are lean and mean. Like the unix > tools, each one of them should only provide a very limited set of > functionality. A good start might be pyAstroTime and pyAstroCoords or > so. That's where raiding and plundering the existing code base comes > in. > We can use some of Erik's and Brandon's stuff and others. I think we > can > easily make a working prototype and the build from there. > > As suggested, we need to be careful not to ignore anyone. But I think > that's easily done by making groups from this community, that take > care > of a single implementation. Everyone who's interested in contributing, > can join. That way the workload is shared and it is build by the > community for the community. > > What do you guys think? > > Cheers > Wolfgang From wkerzendorf at googlemail.com Fri Jul 16 09:38:50 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Fri, 16 Jul 2010 15:38:50 +0200 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> Message-ID: <4C4060EA.2010405@gmail.com> Looking forward to that. I'm wondering if PyWCS should be able to read wavelength solutions (I think they could be considered WCS as well). I have always wrapped wcstools (in a very crude way by calling them on the command line) and I found that they could handle nearly anything. I know they are under GPL and Mike D. regarded this as a problem. Is it still a problem? Cheers Wolfgang On 16/07/10 2:35 PM, Perry Greenfield wrote: > I'd like to tackle the WCS issue first since there are already several > flavors out there and I really wonder if that is necessary. Mike > Droettboom is going to take a look at the others and see what the > differences are in the next week. It would be good to get some > convergence on that. > > But I don't think that can stop others from surveying and comparing > what is out there with regard to coordinates or time. > > As far as the pure python aspect goes, I don't know if I would be so > definitive on that. If there is already a good time or coordinates > library in C that has been very well tested, it might make sense to > use that. It isn't usually a big deal to distribute C code if it has > no other dependencies. Fortran is a different issue. And there are > many tricky issues with regard to coordinate systems. If reimplemented > in pure python I'd suggest that we do a exhaustive test comparison > (mostly automatically generated if possible) with a well tested > library to make sure that it was well validated. > > Perry > > On Jul 16, 2010, at 5:01 AM, Wolfgang Kerzendorf wrote: > >> Hello guys, >> >> I think astropysics looks like a very good start for the coordinate >> class. >> As you said we should at the moment focus on having python-only classes >> for the base-level. That makes it easy to distribute. Once a good API >> has been established and there's complaints about speed, we can switch >> over to c or fortran implementations with the same API. >> >> A monolithic distribution is not so good. I think these baselevel >> classes should, be very modular. We can probably achieve the best >> exceptance when these base classes are lean and mean. Like the unix >> tools, each one of them should only provide a very limited set of >> functionality. A good start might be pyAstroTime and pyAstroCoords or >> so. That's where raiding and plundering the existing code base comes in. >> We can use some of Erik's and Brandon's stuff and others. I think we can >> easily make a working prototype and the build from there. >> >> As suggested, we need to be careful not to ignore anyone. But I think >> that's easily done by making groups from this community, that take care >> of a single implementation. Everyone who's interested in contributing, >> can join. That way the workload is shared and it is build by the >> community for the community. >> >> What do you guys think? >> >> Cheers >> Wolfgang > From jturner at gemini.edu Fri Jul 16 09:52:46 2010 From: jturner at gemini.edu (James Turner) Date: Fri, 16 Jul 2010 09:52:46 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> Message-ID: <4C40642E.8050806@gemini.edu> > But I don't think that can stop others from surveying and comparing > what is out there with regard to coordinates or time. Right, it's good to do WCS first, but I think there's quite a bit to discuss regarding classes and data structures before anything gets merged into the repository. Also, for anything regarding AstroData, I'd need to get other people at Gemini on board (which might be delicate given that we have almost no more time to work on it this year, but it must continue to serve our internal requirements). Cheers, James. From mdroe at stsci.edu Fri Jul 16 10:01:56 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Fri, 16 Jul 2010 10:01:56 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C4060EA.2010405@gmail.com> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> <4C4060EA.2010405@gmail.com> Message-ID: <4C406654.8040504@stsci.edu> On 07/16/2010 09:38 AM, Wolfgang Kerzendorf wrote: > Looking forward to that. I'm wondering if PyWCS should be able to read > wavelength solutions (I think they could be considered WCS as well). > I believe it does. As pywcs is just a wrapper around wcslib, it does most everything wcslib does. > I have always wrapped wcstools (in a very crude way by calling them on > the command line) and I found that they could handle nearly anything. I > know they are under GPL and Mike D. regarded this as a problem. Is it > still a problem? > Both wcslib is under the LGPL. wcstools appears to be under the GPL, though its libwcs is under the LGPL. We tend to prefer more open licenses. Mike > Cheers > Wolfgang > On 16/07/10 2:35 PM, Perry Greenfield wrote: > >> I'd like to tackle the WCS issue first since there are already several >> flavors out there and I really wonder if that is necessary. Mike >> Droettboom is going to take a look at the others and see what the >> differences are in the next week. It would be good to get some >> convergence on that. >> >> But I don't think that can stop others from surveying and comparing >> what is out there with regard to coordinates or time. >> >> As far as the pure python aspect goes, I don't know if I would be so >> definitive on that. If there is already a good time or coordinates >> library in C that has been very well tested, it might make sense to >> use that. It isn't usually a big deal to distribute C code if it has >> no other dependencies. Fortran is a different issue. And there are >> many tricky issues with regard to coordinate systems. If reimplemented >> in pure python I'd suggest that we do a exhaustive test comparison >> (mostly automatically generated if possible) with a well tested >> library to make sure that it was well validated. >> >> Perry >> >> On Jul 16, 2010, at 5:01 AM, Wolfgang Kerzendorf wrote: >> >> >>> Hello guys, >>> >>> I think astropysics looks like a very good start for the coordinate >>> class. >>> As you said we should at the moment focus on having python-only classes >>> for the base-level. That makes it easy to distribute. Once a good API >>> has been established and there's complaints about speed, we can switch >>> over to c or fortran implementations with the same API. >>> >>> A monolithic distribution is not so good. I think these baselevel >>> classes should, be very modular. We can probably achieve the best >>> exceptance when these base classes are lean and mean. Like the unix >>> tools, each one of them should only provide a very limited set of >>> functionality. A good start might be pyAstroTime and pyAstroCoords or >>> so. That's where raiding and plundering the existing code base comes in. >>> We can use some of Erik's and Brandon's stuff and others. I think we can >>> easily make a working prototype and the build from there. >>> >>> As suggested, we need to be careful not to ignore anyone. But I think >>> that's easily done by making groups from this community, that take care >>> of a single implementation. Everyone who's interested in contributing, >>> can join. That way the workload is shared and it is build by the >>> community for the community. >>> >>> What do you guys think? >>> >>> Cheers >>> Wolfgang >>> >> > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA From mdroe at stsci.edu Fri Jul 16 10:18:27 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Fri, 16 Jul 2010 10:18:27 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C406654.8040504@stsci.edu> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> <4C4060EA.2010405@gmail.com> <4C406654.8040504@stsci.edu> Message-ID: <4C406A33.9090905@stsci.edu> On 07/16/2010 10:01 AM, Michael Droettboom wrote: > On 07/16/2010 09:38 AM, Wolfgang Kerzendorf wrote: > >> Looking forward to that. I'm wondering if PyWCS should be able to read >> wavelength solutions (I think they could be considered WCS as well). >> >> > I believe it does. As pywcs is just a wrapper around wcslib, it does > most everything wcslib does. > I stand corrected -- it handles wavelength axes, but not, I believe, wavelength solutions. Mike -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA From erik.tollerud at gmail.com Sun Jul 18 15:16:08 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sun, 18 Jul 2010 12:16:08 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> Message-ID: I definitely agree that settling on a standard WCS library is a very good idea, and PyWCS seems the natural choice as the base to merge other work into. But I also think there's two levels to think about for a WCS library - there's the level that has already been implemented in a few different places as essentially python wrappings around other WCS libraries But then on a higher level, there's tying this into other classes so that, if you don't want to think about WCS, you shouldn't have to. You should be able to do something like wcs = WCS('myamazingdata.fits') im = CCDImage(wcs) im.plot() or just im = CCDImage('myamazingdata.fits') im.plot() or the like, and have the WCS class automatically take care of figuring out everything about the coordinates of the resulting plot. The user should be able to access the WCS details if they want to, but it shouldn't be necessary, assuming the file matches the WCS standard. Now this could well be two seperate independent packages (the ccd-related one, and the WCS library), but I think it's important to consider this as a major target for the interface at the WCS level. I also think (along with other cases) show why it is not necessarily good enough to think of these tools as unix-like/fully modular separate pieces that can operate independent of each other. To make a really useful class-based library, the classes need to all recognize each other when they naturally should. E.g. you should be able to use spectrophotometric data to calibrate photometry, if you want, or use photometric results to calibrate your spectra. Hence, they can't live in totally segmented worlds. Further, you get a lot of user-level gains by using consistently the same tools to do similar tasks. For example, I wrote a "models" module in astropysics to do all function-fitting, and a "fitgui" to go along with it, so whenever you need to fit a 1D function, it always uses the same class to do it, and the user in principle only needs to learn the interface to that class once. Now it might be that it's a better solution to write very simple interfaces for classes like spectra and images with essentially only low-level access to the data, and leave it to other packages to do all the thinks I'm talking about above that inter-link them. But at some level, the constructs are all going to need to be linked together for many astronomical applications. And as for the pure python vs. C wrapping question: I definitely agree that if there is a pre-existing non python module out there to do the task, wrapping is much easier and there's no reason not to use it if it can be reasonably easily wrapped in python. But if there exists a pure-python version that is comparable in speed for the typical applications, I would say that's a better choice, because it allows for much greater flexibility. If you want to subclass something based on a C library, the subclass will almost always be useless because it will be far too slow in python because the rest of the module is built in C and the translation to python constructs will either slow things down, or just not be available to optimize a python subclass. But the optimizations of a pure-python module have to be in place for it to work at all. Of course, it's absolutely right that there has to be careful checks that everything was done correctly in the python implementation before it can actually be released as official, but I think pure python is the goal, given enough time or effort. On Fri, Jul 16, 2010 at 5:35 AM, Perry Greenfield wrote: > I'd like to tackle the WCS issue first since there are already several > flavors out there and I really wonder if that is necessary. Mike Droettboom > is going to take a look at the others and see what the differences are in > the next week. It would be good to get some convergence on that. > > But I don't think that can stop others from surveying and comparing what is > out there with regard to coordinates or time. > > As far as the pure python aspect goes, I don't know if I would be so > definitive on that. If there is already a good time or coordinates library > in C that has been very well tested, it might make sense to use that. It > isn't usually a big deal to distribute C code if it has no other > dependencies. Fortran is a different issue. And there are many tricky issues > with regard to coordinate systems. If reimplemented in ?pure python I'd > suggest that we do a exhaustive test comparison (mostly automatically > generated if possible) with a well tested library to make sure that it was > well validated. > > Perry > > On Jul 16, 2010, at 5:01 AM, Wolfgang Kerzendorf wrote: > >> ?Hello guys, >> >> I think astropysics looks like a very good start for the coordinate class. >> As you said we should at the moment focus on having python-only classes >> for the base-level. That makes it easy to distribute. Once a good API >> has been established and there's complaints about speed, we can switch >> over to c or fortran implementations with the same API. >> >> A monolithic distribution is not so good. I think these baselevel >> classes should, be very modular. We can probably achieve the best >> exceptance when these base classes are lean and mean. Like the unix >> tools, each one of them should only provide a very limited set of >> functionality. A good start might be pyAstroTime and pyAstroCoords or >> so. That's where raiding and plundering the existing code base comes in. >> We can use some of Erik's and Brandon's stuff and others. I think we can >> easily make a working prototype and the build from there. >> >> As suggested, we need to be careful not to ignore anyone. But I think >> that's easily done by making groups from this community, that take care >> of a single implementation. Everyone who's interested in contributing, >> can join. That way the workload is shared and it is build by the >> community for the community. >> >> What do you guys think? >> >> Cheers >> ? ?Wolfgang > > -- Erik Tollerud From perry at stsci.edu Sun Jul 18 15:55:54 2010 From: perry at stsci.edu (Perry Greenfield) Date: Sun, 18 Jul 2010 15:55:54 -0400 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> Message-ID: I don't disagree with any of this. I think the idea that data should have an intrinsic means of mapping to real coordinates is a perfectly good abstraction. Of course FITS wcs can complicate that with multiple WCS representations and the like, but for most things, people wouldn't need to know the details. There are messy details behind the scenes though, like when taking slices and such and making sure the WCS is properly updated for such things. As for pure python vs wrapping libraries, I agree as well. Some C libraries are limiting in what they allow for good python object representations. I don't know if that is going to be an issue with WCS or not (and I've been making similar arguments very recently in a different setting, believe it or not). The existing WCSLIB isn't a bad place to start, but I wouldn't rule out eventually replacing it with a pure python implementation. It's probably not the highest priority thing to do unless we find some intrinsic limitation that really has to be removed though. Perry On Jul 18, 2010, at 3:16 PM, Erik Tollerud wrote: > I definitely agree that settling on a standard WCS library is a very > good idea, and PyWCS seems the natural choice as the base to merge > other work into. But I also think there's two levels to think about > for a WCS library - there's the level that has already been > implemented in a few different places as essentially python wrappings > around other WCS libraries But then on a higher level, there's tying > this into other classes so that, if you don't want to think about WCS, > you shouldn't have to. You should be able to do something like > > wcs = WCS('myamazingdata.fits') > im = CCDImage(wcs) > im.plot() > > or just > > im = CCDImage('myamazingdata.fits') > im.plot() > > or the like, and have the WCS class automatically take care of > figuring out everything about the coordinates of the resulting plot. > The user should be able to access the WCS details if they want to, but > it shouldn't be necessary, assuming the file matches the WCS standard. > Now this could well be two seperate independent packages (the > ccd-related one, and the WCS library), but I think it's important to > consider this as a major target for the interface at the WCS level. > > > I also think (along with other cases) show why it is not necessarily > good enough to think of these tools as unix-like/fully modular > separate pieces that can operate independent of each other. To make a > really useful class-based library, the classes need to all recognize > each other when they naturally should. E.g. you should be able to use > spectrophotometric data to calibrate photometry, if you want, or use > photometric results to calibrate your spectra. Hence, they can't live > in totally segmented worlds. Further, you get a lot of user-level > gains by using consistently the same tools to do similar tasks. For > example, I wrote a "models" module in astropysics to do all > function-fitting, and a "fitgui" to go along with it, so whenever you > need to fit a 1D function, it always uses the same class to do it, and > the user in principle only needs to learn the interface to that class > once. > > Now it might be that it's a better solution to write very simple > interfaces for classes like spectra and images with essentially only > low-level access to the data, and leave it to other packages to do all > the thinks I'm talking about above that inter-link them. But at some > level, the constructs are all going to need to be linked together for > many astronomical applications. > > And as for the pure python vs. C wrapping question: I definitely > agree that if there is a pre-existing non python module out there to > do the task, wrapping is much easier and there's no reason not to use > it if it can be reasonably easily wrapped in python. But if there > exists a pure-python version that is comparable in speed for the > typical applications, I would say that's a better choice, because it > allows for much greater flexibility. If you want to subclass > something based on a C library, the subclass will almost always be > useless because it will be far too slow in python because the rest of > the module is built in C and the translation to python constructs will > either slow things down, or just not be available to optimize a python > subclass. But the optimizations of a pure-python module have to be in > place for it to work at all. Of course, it's absolutely right that > there has to be careful checks that everything was done correctly in > the python implementation before it can actually be released as > official, but I think pure python is the goal, given enough time or > effort. > From wkerzendorf at googlemail.com Sun Jul 18 20:30:25 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Mon, 19 Jul 2010 02:30:25 +0200 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> Message-ID: <4C439CA1.4050009@gmail.com> Hey Guys, On 18/07/10 9:16 PM, Erik Tollerud wrote: > I definitely agree that settling on a standard WCS library is a very > good idea, and PyWCS seems the natural choice as the base to merge > other work into. But I also think there's two levels to think about > for a WCS library - there's the level that has already been > implemented in a few different places as essentially python wrappings > around other WCS libraries But then on a higher level, there's tying > this into other classes so that, if you don't want to think about WCS, > you shouldn't have to. You should be able to do something like > > wcs = WCS('myamazingdata.fits') > im = CCDImage(wcs) > im.plot() > > or just > > im = CCDImage('myamazingdata.fits') > im.plot() > > or the like, and have the WCS class automatically take care of > figuring out everything about the coordinates of the resulting plot. > The user should be able to access the WCS details if they want to, but > it shouldn't be necessary, assuming the file matches the WCS standard. > Now this could well be two seperate independent packages (the > ccd-related one, and the WCS library), but I think it's important to > consider this as a major target for the interface at the WCS level. > I agree with that. That's what I meant by baseclasses. At some stage we should think about metadata being embedded in these CCDImage objects. That would contain among other things a wcs object. > I also think (along with other cases) show why it is not necessarily > good enough to think of these tools as unix-like/fully modular > separate pieces that can operate independent of each other. To make a > really useful class-based library, the classes need to all recognize > each other when they naturally should. E.g. you should be able to use > spectrophotometric data to calibrate photometry, if you want, or use > photometric results to calibrate your spectra. Hence, they can't live > in totally segmented worlds. Further, you get a lot of user-level > gains by using consistently the same tools to do similar tasks. For > example, I wrote a "models" module in astropysics to do all > function-fitting, and a "fitgui" to go along with it, so whenever you > need to fit a 1D function, it always uses the same class to do it, and > the user in principle only needs to learn the interface to that class > once. > I agree and disagree. I think that we need to have a strict API, that assures compatibility between the different tools. Even if we improve the coordinate objects, then this should still work. On the other hand, I think we should strive for wide acceptance, as more users means more developers and less time for each individual. And thus a user wanting to use the astropy time objects, may want to only install this time object and not a big package. Also if someone has a better time object with the same API it is easy to replace the old one. I for one think that for this opensource and multiuser development small packages (that speak the same parameter language of course) is better suited for the task than monolithic development. > Now it might be that it's a better solution to write very simple > interfaces for classes like spectra and images with essentially only > low-level access to the data, and leave it to other packages to do all > the thinks I'm talking about above that inter-link them. But at some > level, the constructs are all going to need to be linked together for > many astronomical applications. > That's what I think as well. > And as for the pure python vs. C wrapping question: I definitely > agree that if there is a pre-existing non python module out there to > do the task, wrapping is much easier and there's no reason not to use > it if it can be reasonably easily wrapped in python. But if there > exists a pure-python version that is comparable in speed for the > typical applications, I would say that's a better choice, because it > allows for much greater flexibility. If you want to subclass > something based on a C library, the subclass will almost always be > useless because it will be far too slow in python because the rest of > the module is built in C and the translation to python constructs will > either slow things down, or just not be available to optimize a python > subclass. But the optimizations of a pure-python module have to be in > place for it to work at all. Of course, it's absolutely right that > there has to be careful checks that everything was done correctly in > the python implementation before it can actually be released as > official, but I think pure python is the goal, given enough time or > effort. > > Well as I said before for now pure python is probably a good idea. That sounds all very exciting let's see what happens. Cheers Wolfgang > On Fri, Jul 16, 2010 at 5:35 AM, Perry Greenfield wrote: >> I'd like to tackle the WCS issue first since there are already several >> flavors out there and I really wonder if that is necessary. Mike Droettboom >> is going to take a look at the others and see what the differences are in >> the next week. It would be good to get some convergence on that. >> >> But I don't think that can stop others from surveying and comparing what is >> out there with regard to coordinates or time. >> >> As far as the pure python aspect goes, I don't know if I would be so >> definitive on that. If there is already a good time or coordinates library >> in C that has been very well tested, it might make sense to use that. It >> isn't usually a big deal to distribute C code if it has no other >> dependencies. Fortran is a different issue. And there are many tricky issues >> with regard to coordinate systems. If reimplemented in pure python I'd >> suggest that we do a exhaustive test comparison (mostly automatically >> generated if possible) with a well tested library to make sure that it was >> well validated. >> >> Perry >> >> On Jul 16, 2010, at 5:01 AM, Wolfgang Kerzendorf wrote: >> >>> Hello guys, >>> >>> I think astropysics looks like a very good start for the coordinate class. >>> As you said we should at the moment focus on having python-only classes >>> for the base-level. That makes it easy to distribute. Once a good API >>> has been established and there's complaints about speed, we can switch >>> over to c or fortran implementations with the same API. >>> >>> A monolithic distribution is not so good. I think these baselevel >>> classes should, be very modular. We can probably achieve the best >>> exceptance when these base classes are lean and mean. Like the unix >>> tools, each one of them should only provide a very limited set of >>> functionality. A good start might be pyAstroTime and pyAstroCoords or >>> so. That's where raiding and plundering the existing code base comes in. >>> We can use some of Erik's and Brandon's stuff and others. I think we can >>> easily make a working prototype and the build from there. >>> >>> As suggested, we need to be careful not to ignore anyone. But I think >>> that's easily done by making groups from this community, that take care >>> of a single implementation. Everyone who's interested in contributing, >>> can join. That way the workload is shared and it is build by the >>> community for the community. >>> >>> What do you guys think? >>> >>> Cheers >>> Wolfgang >> > > From erik.tollerud at gmail.com Mon Jul 19 02:50:56 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sun, 18 Jul 2010 23:50:56 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C439CA1.4050009@gmail.com> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> <4C439CA1.4050009@gmail.com> Message-ID: Just one clarification I'd like to make - I would agree that the smaller targeted packages are probably a good idea in this situation, but the point I'm making is that coding standards for all of the packages under the umbrella project should be consistent, and that's easier to do from the group up. Things like the use of capital letters for the beginning of class names, consistent use of the *same* package to do tasks that are the same (like my model fitting example), identical style for documentation, etc. Without these consistencies it becomes very difficult for the user to understand anything (or the new developer to expand anything), so they'll just give up and use something else. But definitely good to see momentum in this direction! > I agree and disagree. I think that we need to have a strict API, that > assures compatibility between the different tools. Even if we improve the > coordinate objects, then this should still work. On the other hand, I think > we should strive for wide acceptance, as more users means more developers > and less time for each individual. And thus a user wanting to use the > astropy time objects, may want to only install this time object and not a > big package. Also if someone has a better time object with the same API it > is easy to replace the old one. > > I for one think that for this opensource and multiuser development small > packages (that speak the same parameter language of course) is better suited > for the task than monolithic development. -- Erik Tollerud From cohen at lpta.in2p3.fr Mon Jul 19 05:11:40 2010 From: cohen at lpta.in2p3.fr (Johann Cohen-Tanugi) Date: Mon, 19 Jul 2010 11:11:40 +0200 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> <4C439CA1.4050009@gmail.com> Message-ID: <4C4416CC.2020509@lpta.in2p3.fr> On 07/19/2010 08:50 AM, Erik Tollerud wrote: > Just one clarification I'd like to make - I would agree that the > smaller targeted packages are probably a good idea in this situation, > but the point I'm making is that coding standards for all of the > packages under the umbrella project should be consistent, and that's > easier to do from the group up. Things like the use of capital > letters for the beginning of class names, consistent use of the *same* > package to do tasks that are the same (like my model fitting example), > One package for any model fitting? That does not seem very reasonable. Even xspec does not come close to this! Model fitting is a ultra high level analysis tool. I do not think that we are at the stage that we can contemplate this kind of issues. I am doing single photon high-energy astrophysics (Fermi LAT data) : I can assure you that my spectral needs are totally different from a CCD based spectral analysis "a la" X-rays.... But I still need a seamless way to interface WCS and easily visualize astronomical data, with contour overlaid etc..., whatever the underlying system of coordinates used or the projection method requested. Johann > identical style for documentation, etc. Without these consistencies > it becomes very difficult for the user to understand anything (or the > new developer to expand anything), so they'll just give up and use > something else. > > But definitely good to see momentum in this direction! > > > >> I agree and disagree. I think that we need to have a strict API, that >> assures compatibility between the different tools. Even if we improve the >> coordinate objects, then this should still work. On the other hand, I think >> we should strive for wide acceptance, as more users means more developers >> and less time for each individual. And thus a user wanting to use the >> astropy time objects, may want to only install this time object and not a >> big package. Also if someone has a better time object with the same API it >> is easy to replace the old one. >> >> I for one think that for this opensource and multiuser development small >> packages (that speak the same parameter language of course) is better suited >> for the task than monolithic development. >> > > From mdroe at stsci.edu Wed Jul 21 10:06:45 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 21 Jul 2010 10:06:45 -0400 Subject: [AstroPy] ANN: pywcs 1.9 released Message-ID: <4C46FEF5.7070408@stsci.edu> I have made a new release of pywcs, version 1.9. It adds the following features and bugfixes since 1.8: - Support binary image arrays and pixel list format WCS by presenting a way to call wcslib's wcsbth() - Updated underlying wcslib to version 4.5, which fixes the following: - Fixed the interpretation of VELREF when translating AIPS-convention spectral types. Such translation is now handled by a new special- purpose function, spcaips(). The wcsprm struct has been augmented with an entry for velref which is filled by wcspih() and wcsbth(). Previously, selection by VELREF of the radio or optical velocity convention for type VELO was not properly handled. BUGS: - The "pc" member is now available with a default "raw" Wcsprm object. - Make properties that return arrays read-only, since modifying a (mutable) array could result in secondary values not being recomputed based on those changes. - float properties can now be set using int values Mike -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA From William.T.Bridgman at nasa.gov Wed Jul 21 10:27:37 2010 From: William.T.Bridgman at nasa.gov (Bridgman, William T.) Date: Wed, 21 Jul 2010 10:27:37 -0400 Subject: [AstroPy] ANN: pywcs 1.9 released In-Reply-To: <4C46FEF5.7070408@stsci.edu> References: <4C46FEF5.7070408@stsci.edu> Message-ID: <47BA2F4B-67A9-48F7-8012-CBEC6B9ADBBC@nasa.gov> Where is the top-level page for downloading this? Google seems to point to things out of date and searching for pywcs at www.stsci.edu leads to pages with documentation without download links, or "Environment not found". Tom On Jul 21, 2010, at 10:06 AM, Michael Droettboom wrote: > I have made a new release of pywcs, version 1.9. It adds the > following > features and bugfixes since 1.8: > > - Support binary image arrays and pixel list format WCS by presenting > a way to call wcslib's wcsbth() > > - Updated underlying wcslib to version 4.5, which fixes the following: > > - Fixed the interpretation of VELREF when translating > AIPS-convention spectral types. Such translation is now handled > by a new special- purpose function, spcaips(). The wcsprm > struct has been augmented with an entry for velref which is > filled by wcspih() and wcsbth(). Previously, selection by > VELREF of the radio or optical velocity convention for type VELO > was not properly handled. > > BUGS: > > - The "pc" member is now available with a default "raw" Wcsprm object. > > - Make properties that return arrays read-only, since modifying a > (mutable) array could result in secondary values not being > recomputed based on those changes. > > - float properties can now be set using int values > > Mike > > -- > Michael Droettboom > Science Software Branch > Space Telescope Science Institute > Baltimore, Maryland, USA > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- Dr. William T."Tom" Bridgman Scientific Visualization Studio Global Science & Technology, Inc. NASA/Goddard Space Flight Center Email: William.T.Bridgman at nasa.gov Code 610.3 Phone: 301-286-1346 Greenbelt, MD 20771 FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ From mdroe at stsci.edu Wed Jul 21 10:28:25 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 21 Jul 2010 10:28:25 -0400 Subject: [AstroPy] ANN: vo.table 0.6 released Message-ID: <4C470409.5000003@stsci.edu> I have made a new release of vo.table, a library for parsing, validating and generating VOTable XML format files. This release adds the following features since 0.5: Changes since 0.5 ================= - Initial support for conesearch, simple image access and simple spectrum access protocols. This support should be considered experimental and subject to drastic changes in the future. - Support for VOTable 1.2 - Improved compatibility with VOTable 1.0 - Support for Python 3.x - All vo-specific warnings and exceptions have a number. These warnings and exceptions are defined in the documentation. - Warnings and error messages are more detailed. - Only warnings indicating a violation of the specification are given type VOTableSpecWarning. - Improved compatibility with Python 2.6 - C extension for writing XML, resulting in 2x speedup - Generally faster - More workarounds for invalid VOTABLE files in the wild BUGS: - Loading binary tables without an explicit 'nrows' is now fixed. - pyfits is not required -- will only be used when necessary. Mike -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA From mdroe at stsci.edu Wed Jul 21 10:53:08 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 21 Jul 2010 10:53:08 -0400 Subject: [AstroPy] ANN: pywcs 1.9 released In-Reply-To: <47BA2F4B-67A9-48F7-8012-CBEC6B9ADBBC@nasa.gov> References: <4C46FEF5.7070408@stsci.edu> <47BA2F4B-67A9-48F7-8012-CBEC6B9ADBBC@nasa.gov> Message-ID: <4C4709D4.8010206@stsci.edu> http://trac6.assembla.com/astrolib/wiki/WikiStart Mike On 07/21/2010 10:27 AM, Bridgman, William T. wrote: > Where is the top-level page for downloading this? > > Google seems to point to things out of date and searching for pywcs at www.stsci.edu > leads to pages with documentation without download links, or > "Environment not found". > > Tom > On Jul 21, 2010, at 10:06 AM, Michael Droettboom wrote: > > >> I have made a new release of pywcs, version 1.9. It adds the >> following >> features and bugfixes since 1.8: >> >> - Support binary image arrays and pixel list format WCS by presenting >> a way to call wcslib's wcsbth() >> >> - Updated underlying wcslib to version 4.5, which fixes the following: >> >> - Fixed the interpretation of VELREF when translating >> AIPS-convention spectral types. Such translation is now handled >> by a new special- purpose function, spcaips(). The wcsprm >> struct has been augmented with an entry for velref which is >> filled by wcspih() and wcsbth(). Previously, selection by >> VELREF of the radio or optical velocity convention for type VELO >> was not properly handled. >> >> BUGS: >> >> - The "pc" member is now available with a default "raw" Wcsprm object. >> >> - Make properties that return arrays read-only, since modifying a >> (mutable) array could result in secondary values not being >> recomputed based on those changes. >> >> - float properties can now be set using int values >> >> Mike >> >> -- >> Michael Droettboom >> Science Software Branch >> Space Telescope Science Institute >> Baltimore, Maryland, USA >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > -- > Dr. William T."Tom" Bridgman Scientific Visualization > Studio > Global Science& Technology, Inc. NASA/Goddard Space Flight > Center > Email: William.T.Bridgman at nasa.gov Code 610.3 > Phone: 301-286-1346 Greenbelt, MD 20771 > FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From wkerzendorf at googlemail.com Wed Jul 21 10:58:43 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Wed, 21 Jul 2010 16:58:43 +0200 Subject: [AstroPy] converting from proper motion in ra, dec to proper motion in galactic long, lat Message-ID: <4C470B23.1070208@gmail.com> Hello, Does anyone know of something like this in any language? I also have errorbars that need converting. Cheers Wolfgang From aarchiba at physics.mcgill.ca Wed Jul 21 11:23:45 2010 From: aarchiba at physics.mcgill.ca (Anne Archibald) Date: Wed, 21 Jul 2010 11:23:45 -0400 Subject: [AstroPy] converting from proper motion in ra, dec to proper motion in galactic long, lat In-Reply-To: <4C470B23.1070208@gmail.com> References: <4C470B23.1070208@gmail.com> Message-ID: On 21 July 2010 10:58, Wolfgang Kerzendorf wrote: > ?Hello, > > Does anyone know of something like this in any language? I also have > errorbars that need converting. I believe that SLALIB (for which Scott Ransom wrote python bindings) can compute transformation matrices (i.e. the matrix of partial derivatives) for its coordinate conversions; this should allow you to convert proper motions. Error bars will take a little more work because after matrix multiplication you need to convert a non-axis-aligned error ellipse into some readily representable form. Failing this, if you've got a good conversion library, you can just compute the derivatives numerically (e.g. (to_galactic(x+vx*t,y+vy*t)-to_galactic(x,y))/t for small but not too small t). Anne > Cheers > ? ?Wolfgang > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From erik.tollerud at gmail.com Wed Jul 21 13:53:54 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Wed, 21 Jul 2010 10:53:54 -0700 Subject: [AstroPy] ANN: vo.table 0.6 released In-Reply-To: <4C470409.5000003@stsci.edu> References: <4C470409.5000003@stsci.edu> Message-ID: Do you have plans to submit vo.table as a package to pypi? It would be useful to be able to specify it as a requirement for packages that might want to access VOTables, but currently the python install tools need a package to have a pypi entry to resolve the dependencies (at least, automatically)... On Wed, Jul 21, 2010 at 7:28 AM, Michael Droettboom wrote: > I have made a new release of vo.table, a library for parsing, validating > and generating VOTable XML format files. > > This release adds the following features since 0.5: > > Changes since 0.5 > ================= > > - Initial support for conesearch, simple image access and simple > ? spectrum access protocols. ?This support should be considered > ? experimental and subject to drastic changes in the future. > > - Support for VOTable 1.2 > > - Improved compatibility with VOTable 1.0 > > - Support for Python 3.x > > - All vo-specific warnings and exceptions have a number. ?These > ? warnings and exceptions are defined in the documentation. > > - Warnings and error messages are more detailed. > > - Only warnings indicating a violation of the specification are given > ? type VOTableSpecWarning. > > - Improved compatibility with Python 2.6 > > - C extension for writing XML, resulting in 2x speedup > > - Generally faster > > - More workarounds for invalid VOTABLE files in the wild > > BUGS: > > - Loading binary tables without an explicit 'nrows' is now fixed. > > - pyfits is not required -- will only be used when necessary. > > Mike > > -- > Michael Droettboom > Science Software Branch > Space Telescope Science Institute > Baltimore, Maryland, USA > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud Graduate Student Center For Cosmology Department of Physics and Astronomy 2142 Frederick Reines Hall University of California, Irvine Cell: (651)307-9409 etolleru at uci.edu http://ps.uci.edu/~etolleru From mdroe at stsci.edu Wed Jul 21 17:36:54 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 21 Jul 2010 17:36:54 -0400 Subject: [AstroPy] ANN: vo.table 0.6 released In-Reply-To: <4C470409.5000003@stsci.edu> References: <4C470409.5000003@stsci.edu> Message-ID: <4C476876.60505@stsci.edu> I neglected to include the download link: https://trac6.assembla.com/astrolib Mike On 07/21/2010 10:28 AM, Michael Droettboom wrote: > I have made a new release of vo.table, a library for parsing, validating > and generating VOTable XML format files. > > This release adds the following features since 0.5: > > Changes since 0.5 > ================= > > - Initial support for conesearch, simple image access and simple > spectrum access protocols. This support should be considered > experimental and subject to drastic changes in the future. > > - Support for VOTable 1.2 > > - Improved compatibility with VOTable 1.0 > > - Support for Python 3.x > > - All vo-specific warnings and exceptions have a number. These > warnings and exceptions are defined in the documentation. > > - Warnings and error messages are more detailed. > > - Only warnings indicating a violation of the specification are given > type VOTableSpecWarning. > > - Improved compatibility with Python 2.6 > > - C extension for writing XML, resulting in 2x speedup > > - Generally faster > > - More workarounds for invalid VOTABLE files in the wild > > BUGS: > > - Loading binary tables without an explicit 'nrows' is now fixed. > > - pyfits is not required -- will only be used when necessary. > > Mike > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Wed Jul 21 17:44:05 2010 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 21 Jul 2010 17:44:05 -0400 Subject: [AstroPy] ANN: vo.table 0.6 released In-Reply-To: References: <4C470409.5000003@stsci.edu> Message-ID: <4C476A25.2050602@stsci.edu> I have no experience submitting to PyPI (I generally get annoyed at packages that go out to PyPI because there's a better than even chance it loads the wrong version of something). Since vo.table includes compiled code, does that mean I have to provide packages for all platforms? Mike On 07/21/2010 01:53 PM, Erik Tollerud wrote: > Do you have plans to submit vo.table as a package to pypi? It would be > useful to be able to specify it as a requirement for packages that > might want to access VOTables, but currently the python install tools > need a package to have a pypi entry to resolve the dependencies (at > least, automatically)... > > On Wed, Jul 21, 2010 at 7:28 AM, Michael Droettboom wrote: > >> I have made a new release of vo.table, a library for parsing, validating >> and generating VOTable XML format files. >> >> This release adds the following features since 0.5: >> >> Changes since 0.5 >> ================= >> >> - Initial support for conesearch, simple image access and simple >> spectrum access protocols. This support should be considered >> experimental and subject to drastic changes in the future. >> >> - Support for VOTable 1.2 >> >> - Improved compatibility with VOTable 1.0 >> >> - Support for Python 3.x >> >> - All vo-specific warnings and exceptions have a number. These >> warnings and exceptions are defined in the documentation. >> >> - Warnings and error messages are more detailed. >> >> - Only warnings indicating a violation of the specification are given >> type VOTableSpecWarning. >> >> - Improved compatibility with Python 2.6 >> >> - C extension for writing XML, resulting in 2x speedup >> >> - Generally faster >> >> - More workarounds for invalid VOTABLE files in the wild >> >> BUGS: >> >> - Loading binary tables without an explicit 'nrows' is now fixed. >> >> - pyfits is not required -- will only be used when necessary. >> >> Mike >> >> -- >> Michael Droettboom >> Science Software Branch >> Space Telescope Science Institute >> Baltimore, Maryland, USA >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> > > > From rowen at uw.edu Wed Jul 21 18:06:12 2010 From: rowen at uw.edu (Russell Owen) Date: Wed, 21 Jul 2010 15:06:12 -0700 Subject: [AstroPy] ANN: vo.table 0.6 released In-Reply-To: <4C476A25.2050602@stsci.edu> References: <4C470409.5000003@stsci.edu> <4C476A25.2050602@stsci.edu> Message-ID: <731BE068-4346-43D9-80DC-E919210F0BF0@uw.edu> I like PyPI for serving source code. I am not personally very impressed with binary eggs -- in my limited experience it's easy to install in an incompatible system or have easy_install get confused and download stuff you've already got. In fact I use pip instead of easy_install partly for this reason (also to avoid easy_install's mega-pth file). For Mac OS X I prefer to ordinary installers using bdist_mpkg. If you go that route you could serve the source code from PyPI and have a link to a separate site for downloading binary installers. -- Russell On Jul 21, 2010, at 2:44 PM, Michael Droettboom wrote: > I have no experience submitting to PyPI (I generally get annoyed at > packages that go out to PyPI because there's a better than even chance > it loads the wrong version of something). Since vo.table includes > compiled code, does that mean I have to provide packages for all > platforms? > > Mike > > On 07/21/2010 01:53 PM, Erik Tollerud wrote: >> Do you have plans to submit vo.table as a package to pypi? It would >> be >> useful to be able to specify it as a requirement for packages that >> might want to access VOTables, but currently the python install tools >> need a package to have a pypi entry to resolve the dependencies (at >> least, automatically)... >> >> On Wed, Jul 21, 2010 at 7:28 AM, Michael >> Droettboom wrote: >> >>> I have made a new release of vo.table, a library for parsing, >>> validating >>> and generating VOTable XML format files. >>> >>> This release adds the following features since 0.5: >>> >>> Changes since 0.5 >>> ================= >>> >>> - Initial support for conesearch, simple image access and simple >>> spectrum access protocols. This support should be considered >>> experimental and subject to drastic changes in the future. >>> >>> - Support for VOTable 1.2 >>> >>> - Improved compatibility with VOTable 1.0 >>> >>> - Support for Python 3.x >>> >>> - All vo-specific warnings and exceptions have a number. These >>> warnings and exceptions are defined in the documentation. >>> >>> - Warnings and error messages are more detailed. >>> >>> - Only warnings indicating a violation of the specification are >>> given >>> type VOTableSpecWarning. >>> >>> - Improved compatibility with Python 2.6 >>> >>> - C extension for writing XML, resulting in 2x speedup >>> >>> - Generally faster >>> >>> - More workarounds for invalid VOTABLE files in the wild >>> >>> BUGS: >>> >>> - Loading binary tables without an explicit 'nrows' is now fixed. >>> >>> - pyfits is not required -- will only be used when necessary. >>> >>> Mike >>> >>> -- >>> Michael Droettboom >>> Science Software Branch >>> Space Telescope Science Institute >>> Baltimore, Maryland, USA\ From erik.tollerud at gmail.com Wed Jul 21 23:57:11 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Wed, 21 Jul 2010 20:57:11 -0700 Subject: [AstroPy] Co-ordinating Python astronomy libraries? In-Reply-To: <4C4416CC.2020509@lpta.in2p3.fr> References: <4C2BBBA7.5060006@gemini.edu> <4C3B72A8.5070404@gemini.edu> <4C3DD19C.1040609@gmail.com> <4C3F39E8.7090309@gemini.edu> <4C401FF6.3090105@gmail.com> <4C439CA1.4050009@gmail.com> <4C4416CC.2020509@lpta.in2p3.fr> Message-ID: > One package for any model fitting? That does not seem very reasonable. Even > xspec does not come close to this! > Model fitting is a ultra high level analysis tool. I do not think that we > are at the stage that we can contemplate this kind of issues. > I am doing single photon high-energy astrophysics ?(Fermi LAT data) : I can > assure you that my spectral needs are totally different from a CCD based > spectral analysis "a la" X-rays.... Oh, you're absolutely right that the functional forms, parameter selection techniques, statistical analysis, and the like will be very different for different problem domains. But my point is that all of this can be encapsulated in classes with common interfaces. That is, if you write a parametric model that maps a scalar onto another scalar, it should be easy for you to stick that in a class that lets you specify a particular data fitting technique, but not have to write your own special GUI every time you design a new 1D parametric model. And it should be super trivial to write a model and have the basic fitting methods "just work" out of the box. That is, while model fitting (and for that matter, any given model) is indeed ultra high level, the general framework of a 1D parametric model with data (and errors) and the basic tasks of plotting and the like are actually fairly low level, and I would say that means they should be in the base classes so we can more easily leverage each others' work to get out more science. > But I still need a seamless way to > interface WCS and easily visualize astronomical data, with contour overlaid > etc..., whatever the underlying system of coordinates used or the projection > method requested. Agreed! At least about WCS - I think lots of people have pretty application specific ways of overlaying things on WCS, but the appropriate tools to do that fairly generically are the key. -- Erik Tollerud Graduate Student Center For Cosmology Department of Physics and Astronomy 2142 Frederick Reines Hall University of California, Irvine Cell: (651)307-9409 etolleru at uci.edu http://ps.uci.edu/~etolleru From jh at physics.ucf.edu Fri Jul 23 15:01:06 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 23 Jul 2010 15:01:06 -0400 Subject: [AstroPy] reviewers needed for NumPy Message-ID: Hi folks, We are (finally) about to begin reviewing and proofing the NumPy docstrings! This is the final step in producing professional-level docs for NumPy. What we need now are people willing to review docs. There are two types of reviewers: Technical reviewers should be developers or *very* experienced NumPy users. Technical review entails checking the source code (it's available on a click in the doc wiki) and reading the doc to ensure that the signature and description are both correct and complete. Presentation reviewers need to be modestly experienced with NumPy, and should have some experience either in technical writing or as educators. Their job is to make sure the docstring is understandable to the target audience (one level below the expected user of that item), including appropriate examples and references. Review entails reading each page, checking that it meets the review standards, and either approving it or saying how it doesn't meet them. All this takes place on the doc wiki, so the mechanics are easy. Please post a message on scipy-dev if you are interested in becoming a reviewer, or if you have questions about reviewing. As a volunteer reviewer, you can put as much or as little time into this as you like. Thanks! --jh-- for the SciPy Documentation Project team From jslavin at cfa.harvard.edu Fri Jul 23 16:07:53 2010 From: jslavin at cfa.harvard.edu (Jonathan Slavin) Date: Fri, 23 Jul 2010 16:07:53 -0400 Subject: [AstroPy] python equivalent to IDL routine smooth Message-ID: <1279915673.8463.9.camel@shevek> Does anyone know of a python equivalent to the IDL routine smooth? I know about the scipy routines convolve and convolve2d, but I need to do it 3-D. (Also, using convolve2d gave me strange results.) My goal is to smooth a noisy 3-D dataset. A simple boxcar type smoothing would be okay. Jon Slavin -- ______________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin at cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 cell: (781) 363-0035 USA ______________________________________________________________ From msk at astro.umd.edu Fri Jul 23 16:18:29 2010 From: msk at astro.umd.edu (Michael S. Kelley) Date: Fri, 23 Jul 2010 16:18:29 -0400 Subject: [AstroPy] python equivalent to IDL routine smooth In-Reply-To: <1279915673.8463.9.camel@shevek> References: <1279915673.8463.9.camel@shevek> Message-ID: <4C49F915.4070001@astro.umd.edu> Hi Jon, Although I haven't tried it with 3D smoothing, scipy.ndimage.uniform_filter() might be able to do it. Cheers, Mike Kelley On 7/23/10 4:07 PM, Jonathan Slavin wrote: > Does anyone know of a python equivalent to the IDL routine smooth? I > know about the scipy routines convolve and convolve2d, but I need to do > it 3-D. (Also, using convolve2d gave me strange results.) > > My goal is to smooth a noisy 3-D dataset. A simple boxcar type > smoothing would be okay. > > Jon Slavin -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Michael S Kelley University of Maryland -- College Park 301-405-3796 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 267 bytes Desc: OpenPGP digital signature URL: From erik.tollerud at gmail.com Fri Jul 23 20:48:52 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Fri, 23 Jul 2010 17:48:52 -0700 Subject: [AstroPy] ANN: vo.table 0.6 released In-Reply-To: <731BE068-4346-43D9-80DC-E919210F0BF0@uw.edu> References: <4C470409.5000003@stsci.edu> <4C476A25.2050602@stsci.edu> <731BE068-4346-43D9-80DC-E919210F0BF0@uw.edu> Message-ID: PyPI is perfectly happy with just getting the source code and no binary packages, although as Russell noted, it's also possible to upload compiled binaries (the distribute package has the tools to help the process). I've never really had any version trouble with PyPI as long as the package is actually using the proper distutils way of specifying version numbers - the rules are spelled out in http://docs.python.org/distutils/setupscript.html#relationships-between-distributions-and-packages , but if you use an incompatible numbering scheme, PyPI thinks you're uploading an older version, and that can indeed cause trouble... I agree PyPI can be trouble for binary eggs, though... some people seem to be able to make it work, but I always just compile it from source anyway, like pip does. I'd say the best approach is to just post the source code and if someone really cares about it, leave it to them to build binaries for their architecture as they want them. The main thing I think is important is to have some sort of download link on PyPI so the automated tools like pip or setuptools can resolve dependencies automatically. On Wed, Jul 21, 2010 at 3:06 PM, Russell Owen wrote: > I like PyPI for serving source code. > > I am not personally very impressed with binary eggs -- in my limited > experience it's easy to install in an incompatible system or have > easy_install get confused and download stuff you've already got. In > fact I use pip instead of easy_install partly for this reason (also to > avoid easy_install's mega-pth file). > > For Mac OS X I prefer to ordinary installers using bdist_mpkg. If you > go that route you could serve the source code from PyPI and have a > link to a separate site for downloading binary installers. > > -- Russell > > On Jul 21, 2010, at 2:44 PM, Michael Droettboom wrote: > >> I have no experience submitting to PyPI (I generally get annoyed at >> packages that go out to PyPI because there's a better than even chance >> it loads the wrong version of something). ?Since vo.table includes >> compiled code, does that mean I have to provide packages for all >> platforms? >> >> Mike >> >> On 07/21/2010 01:53 PM, Erik Tollerud wrote: >>> Do you have plans to submit vo.table as a package to pypi? It would >>> be >>> useful to be able to specify it as a requirement for packages that >>> might want to access VOTables, but currently the python install tools >>> need a package to have a pypi entry to resolve the dependencies (at >>> least, automatically)... >>> >>> On Wed, Jul 21, 2010 at 7:28 AM, Michael >>> Droettboom ?wrote: >>> >>>> I have made a new release of vo.table, a library for parsing, >>>> validating >>>> and generating VOTable XML format files. >>>> >>>> This release adds the following features since 0.5: >>>> >>>> Changes since 0.5 >>>> ================= >>>> >>>> - Initial support for conesearch, simple image access and simple >>>> ? spectrum access protocols. ?This support should be considered >>>> ? experimental and subject to drastic changes in the future. >>>> >>>> - Support for VOTable 1.2 >>>> >>>> - Improved compatibility with VOTable 1.0 >>>> >>>> - Support for Python 3.x >>>> >>>> - All vo-specific warnings and exceptions have a number. ?These >>>> ? warnings and exceptions are defined in the documentation. >>>> >>>> - Warnings and error messages are more detailed. >>>> >>>> - Only warnings indicating a violation of the specification are >>>> given >>>> ? type VOTableSpecWarning. >>>> >>>> - Improved compatibility with Python 2.6 >>>> >>>> - C extension for writing XML, resulting in 2x speedup >>>> >>>> - Generally faster >>>> >>>> - More workarounds for invalid VOTABLE files in the wild >>>> >>>> BUGS: >>>> >>>> - Loading binary tables without an explicit 'nrows' is now fixed. >>>> >>>> - pyfits is not required -- will only be used when necessary. >>>> >>>> Mike >>>> >>>> -- >>>> Michael Droettboom >>>> Science Software Branch >>>> Space Telescope Science Institute >>>> Baltimore, Maryland, USA\ > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud From fabricio at ferrari.pro.br Sat Jul 24 14:21:30 2010 From: fabricio at ferrari.pro.br (Fabricio Ferrari) Date: Sat, 24 Jul 2010 15:21:30 -0300 Subject: [AstroPy] python equivalent to IDL routine smooth Message-ID: Hello Jonathan it is much more efficient to do smoothering in Fourier space, since, as you know convolving in normal space is multiplication in Fourier space. So, just take the FFT of your data, the FFT of the window filter with the same size of your data FFT, multiply them and take the inverse FFT. The is n-dimensional FFT in Scipy or Numpy. Since you have the trouble of doing all this, you should not use a square window (boxcar) for it is the worst window in the frequency response sense, it has a lot of frequency leakage. Please try any of the other windows in scipy.signal, for example hamming window are a good one. please tell me if I can help any further, maybe off the list. Cheers, Fabricio ..-. ..-. Fabricio Ferrari? ? ?? [www.ferrari.pro.br] Universidade Federal do Pampa Bag? RS Brasil > ? 2. python equivalent to IDL routine smooth (Jonathan Slavin) > Message: 2 > Date: Fri, 23 Jul 2010 16:07:53 -0400 > From: Jonathan Slavin > Subject: [AstroPy] python equivalent to IDL routine smooth > To: astropy at scipy.org > Message-ID: <1279915673.8463.9.camel at shevek> > Content-Type: text/plain > > Does anyone know of a python equivalent to the IDL routine smooth? ?I > know about the scipy routines convolve and convolve2d, but I need to do > it 3-D. (Also, using convolve2d gave me strange results.) > > My goal is to smooth a noisy 3-D dataset. ?A simple boxcar type > smoothing would be okay. > > Jon Slavin > -- > ______________________________________________________________ > Jonathan D. Slavin ? ? ? ? ? ? ?Harvard-Smithsonian CfA > jslavin at cfa.harvard.edu ? ? ? ? 60 Garden Street, MS 83 > phone: (617) 496-7981 ? ? ? ? ? Cambridge, MA 02138-1516 From jslavin at cfa.harvard.edu Mon Jul 26 08:59:06 2010 From: jslavin at cfa.harvard.edu (Jonathan Slavin) Date: Mon, 26 Jul 2010 08:59:06 -0400 Subject: [AstroPy] AstroPy Digest, Vol 47, Issue 23 In-Reply-To: References: Message-ID: <1280149146.13448.12.camel@shevek> Fabricio, It is no doubt more efficient and more controllable to do smoothing in Fourier space, but all I need is a quick and simple method for display purposes. For that the method I found (and Michael Kelley also pointed out) is scipy.ndimage.filters.uniform_filter, or better scipy.ndimage.filters.gaussian_filter. These are very fast on my 160**3 grid and do the trick. Jon On Sun, 2010-07-25 at 12:00 -0500, astropy-request at scipy.org wrote: > Date: Sat, 24 Jul 2010 15:21:30 -0300 > From: Fabricio Ferrari > Subject: [AstroPy] python equivalent to IDL routine smooth > To: astropy at scipy.org > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > Hello Jonathan > > it is much more efficient to do smoothering in Fourier space, since, > as you know convolving in normal space is multiplication in Fourier > space. So, just take the FFT of your data, the FFT of the window > filter with the same size of your data FFT, multiply them and take the > inverse FFT. The is n-dimensional FFT in Scipy or Numpy. > > Since you have the trouble of doing all this, you should not use a > square window (boxcar) for it is the worst window in the frequency > response sense, it has a lot of frequency leakage. Please try any of > the other windows in scipy.signal, for example hamming window are a > good one. > > please tell me if I can help any further, maybe off the list. > > Cheers, > > Fabricio > ..-. ..-. > Fabricio Ferrari? ? ?? [www.ferrari.pro.br] > Universidade Federal do Pampa > Bag? RS > Brasil > > > ? 2. python equivalent to IDL routine smooth (Jonathan Slavin) > > Message: 2 > > Date: Fri, 23 Jul 2010 16:07:53 -0400 > > From: Jonathan Slavin > > Subject: [AstroPy] python equivalent to IDL routine smooth > > To: astropy at scipy.org > > Message-ID: <1279915673.8463.9.camel at shevek> > > Content-Type: text/plain > > > > Does anyone know of a python equivalent to the IDL routine smooth? ?I > > know about the scipy routines convolve and convolve2d, but I need to do > > it 3-D. (Also, using convolve2d gave me strange results.) > > > > My goal is to smooth a noisy 3-D dataset. ?A simple boxcar type > > smoothing would be okay. > > > > Jon Slavin > > -- > > ______________________________________________________________ > > Jonathan D. Slavin ? ? ? ? ? ? ?Harvard-Smithsonian CfA > > jslavin at cfa.harvard.edu ? ? ? ? 60 Garden Street, MS 83 > > phone: (617) 496-7981 ? ? ? ? ? Cambridge, MA 02138-1516 > > > ------------------------------ > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > End of AstroPy Digest, Vol 47, Issue 23 > *************************************** -- ______________________________________________________________ Jonathan D. Slavin Harvard-Smithsonian CfA jslavin at cfa.harvard.edu 60 Garden Street, MS 83 phone: (617) 496-7981 Cambridge, MA 02138-1516 cell: (781) 363-0035 USA ______________________________________________________________ From jh at physics.ucf.edu Mon Jul 26 09:20:01 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Mon, 26 Jul 2010 09:20:01 -0400 Subject: [AstroPy] python equivalent to IDL routine smooth In-Reply-To: (astropy-request@scipy.org) References: Message-ID: Recall that in Fourier space everything wraps, so be sure to subtract the mean and pad your data in a field of zeros large enough that your kernel won't average data on the left edge with data from the right, etc. Then transform, multiply, transform back, trim, and add back the mean. There are some very nice wavelet-based smoothing methods, as well, See paper by Torrence and Compo for painless introduction: http://paos.colorado.edu/research/wavelets/software.html I found the paper easier to learn from than the tutorial on the web site. No python code there but there are routines available in Python. Perhaps someone else has a suggestion on a good replacement for the code on that page in Python. --jh-- From erik.tollerud at gmail.com Mon Jul 26 19:49:06 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 26 Jul 2010 16:49:06 -0700 Subject: [AstroPy] python equivalent to IDL routine smooth In-Reply-To: References: Message-ID: It's not exactly a replacement for those particular codes, but I've heard good things about PyWavelets (although I haven't personally used it myself...) as an all-purpose wavelet transform tool: http://www.pybytes.com/pywavelets/ Although this might be a bit more in-depth than the original message was asking about... On Mon, Jul 26, 2010 at 6:20 AM, Joe Harrington wrote: > Recall that in Fourier space everything wraps, so be sure to subtract > the mean and pad your data in a field of zeros large enough that your > kernel won't average data on the left edge with data from the right, > etc. ?Then transform, multiply, transform back, trim, and add back the > mean. > > There are some very nice wavelet-based smoothing methods, as well, > See paper by Torrence and Compo for painless introduction: > > http://paos.colorado.edu/research/wavelets/software.html > > I found the paper easier to learn from than the tutorial on the web > site. ?No python code there but there are routines available in > Python. ?Perhaps someone else has a suggestion on a good replacement > for the code on that page in Python. > > --jh-- > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud From wkerzendorf at googlemail.com Fri Jul 30 00:16:32 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Fri, 30 Jul 2010 14:16:32 +1000 Subject: [AstroPy] Summer scholar Message-ID: <4C525220.1080508@gmail.com> Dear all, Here at Mt. Stromlo Observatory we will have summer scholars in November - February. There was a recent call for ideas for ~8 week summer projects. I can definitley co-supervise as this will all happen at Stromlo, but would need help with fleshing out an idea. Is there something we can offer for a physics undergrad to code for 8 weeks. Maybe build a coord base class? Is that too complex? Other ideas? Maybe we should also think about getting people in the next GSoC round? What are people's feeling on this? Cheers Wolfgang From jh at physics.ucf.edu Fri Jul 30 13:19:36 2010 From: jh at physics.ucf.edu (Joe Harrington) Date: Fri, 30 Jul 2010 13:19:36 -0400 Subject: [AstroPy] Summer scholar In-Reply-To: (astropy-request@scipy.org) References: Message-ID: Unless it's been done, I'd love to see this IDL/Matlab/FORTRAN code http://paos.colorado.edu/research/wavelets/ http://paos.colorado.edu/research/wavelets/bams_79_01_0061.pdf implemented in Python. *Lots* of people use it, and the lack of it is a factor in some people's decision to stick with IDL. There are a few Python wavelet codes and I haven't looked at any of them, so maybe all that needs to be done is to implement the high-level stuff there (tests and examples) in an existing Python wavelet code. An assessment of those and their suitability for tasks as outlined in the accompanying paper would be a great contribution for 8 weeks of undergrad work! I used this package, e.g., in my most recent ApJ paper: http://arxiv.org/abs/1005.3570 See especially the nearly-noiseless global transform! --jh-- Date: Fri, 30 Jul 2010 14:16:32 +1000 From: Wolfgang Kerzendorf Subject: [AstroPy] Summer scholar To: astropy at scipy.org Message-ID: <4C525220.1080508 at gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Dear all, Here at Mt. Stromlo Observatory we will have summer scholars in November - February. There was a recent call for ideas for ~8 week summer projects. I can definitley co-supervise as this will all happen at Stromlo, but would need help with fleshing out an idea. Is there something we can offer for a physics undergrad to code for 8 weeks. Maybe build a coord base class? Is that too complex? Other ideas? Maybe we should also think about getting people in the next GSoC round? What are people's feeling on this? Cheers Wolfgang From superluminique at gmail.com Fri Jul 30 13:36:43 2010 From: superluminique at gmail.com (Rene Breton) Date: Fri, 30 Jul 2010 13:36:43 -0400 Subject: [AstroPy] Summer scholar In-Reply-To: References: Message-ID: <4C530DAB.7090106@gmail.com> Hi Joe, That's an interesting suggestion and I just searched a bit and came across a scipy package called PyWavelets! Have you ever heard about it? I've never used it myself since last time I had to use wavelets stuff was a while ago, well before this came to existence. It certainly seems to me that working on a PyWavelets sub-package as part of the AstroPy library would be a useless duplicate of effort unless... I'd look into the next couple things: 1. Does PyWavelets have all the functionalities that one (you) is looking for? If so, problem solved. If not, maybe it would be a good idea to implement what's missing in that package in order to keeps things well organized, especially if the things you have in mind are general enough and not too specific to astronomers. 2. If needed, we could think about a specialized wavelets sub-package in AstroPy with that suits some very specific need of astronomers (not sure what it would be though). Again, to me the "requirement" would be to make it be some kind of front-end to PyWavelets. By doing things this way, we don't include further dependencies, which is certainly fundamental to AstroPy. If one wants to use the wavelets sub-package, then one can make sure that PyWavelets is working and that's all. But again, I think solution #1 would be better because it would avoid possible strong divergence in the future. Rene On 10-07-30 01:19 PM, Joe Harrington wrote: > Unless it's been done, I'd love to see this IDL/Matlab/FORTRAN code > > http://paos.colorado.edu/research/wavelets/ > http://paos.colorado.edu/research/wavelets/bams_79_01_0061.pdf > > implemented in Python. *Lots* of people use it, and the lack of it is > a factor in some people's decision to stick with IDL. There are a few > Python wavelet codes and I haven't looked at any of them, so maybe all > that needs to be done is to implement the high-level stuff there > (tests and examples) in an existing Python wavelet code. An > assessment of those and their suitability for tasks as outlined in the > accompanying paper would be a great contribution for 8 weeks of > undergrad work! I used this package, e.g., in my most recent ApJ > paper: http://arxiv.org/abs/1005.3570 > See especially the nearly-noiseless global transform! > > --jh-- > > Date: Fri, 30 Jul 2010 14:16:32 +1000 > From: Wolfgang Kerzendorf > Subject: [AstroPy] Summer scholar > To: astropy at scipy.org > Message-ID:<4C525220.1080508 at gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Dear all, > > Here at Mt. Stromlo Observatory we will have summer scholars in November > - February. There was a recent call for ideas for ~8 week summer > projects. I can definitley co-supervise as this will all happen at > Stromlo, but would need help with fleshing out an idea. Is there > something we can offer for a physics undergrad to code for 8 weeks. > Maybe build a coord base class? Is that too complex? Other ideas? Maybe > we should also think about getting people in the next GSoC round? > What are people's feeling on this? > > Cheers > Wolfgang > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From superluminique at gmail.com Fri Jul 30 13:44:58 2010 From: superluminique at gmail.com (Rene Breton) Date: Fri, 30 Jul 2010 13:44:58 -0400 Subject: [AstroPy] Summer scholar In-Reply-To: <4C525220.1080508@gmail.com> References: <4C525220.1080508@gmail.com> Message-ID: <4C530F9A.3010806@gmail.com> Hi Wolfgang, Coord base class might be interesting, though I'm not entirely sure to know what you have in mind. Is it stuff like planar projections, precession, coordinate transformations, etc.? Obviously, if that's something that you need and want to see, I'd say go for it. Something that I see as relatively important, in the view of what IDL's astrolib does, are tools to deal with photometry and spectroscopy. I know that some people are fans of sextractor. I don't know to what extent it is good and whether or not we should put efforts somewhere else but it's definitely worth thinking. PSF photometry, both from an analytical (arbitrary) function and from a template array, would be very appealing, I think. Same kind of thinking should go toward spectroscopy. Rene On 10-07-30 12:16 AM, Wolfgang Kerzendorf wrote: > Dear all, > > Here at Mt. Stromlo Observatory we will have summer scholars in November > - February. There was a recent call for ideas for ~8 week summer > projects. I can definitley co-supervise as this will all happen at > Stromlo, but would need help with fleshing out an idea. Is there > something we can offer for a physics undergrad to code for 8 weeks. > Maybe build a coord base class? Is that too complex? Other ideas? Maybe > we should also think about getting people in the next GSoC round? > What are people's feeling on this? > > Cheers > Wolfgang > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From erik.tollerud at gmail.com Fri Jul 30 19:21:59 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Fri, 30 Jul 2010 16:21:59 -0700 Subject: [AstroPy] Summer scholar In-Reply-To: <4C530F9A.3010806@gmail.com> References: <4C525220.1080508@gmail.com> <4C530F9A.3010806@gmail.com> Message-ID: Hello all, On Fri, Jul 30, 2010 at 10:19 AM, Joe Harrington wrote: > Unless it's been done, I'd love to see this IDL/Matlab/FORTRAN code As Rene said, PyWavelets does as much if not more as I think this does... but it's probably true that it would be worthwhile to write some wrappers or subclasses that deal with astronomy-specific techniques - as Jonathan Slavin was asking about last week on this list, there are some astronomy-centric tasks that could simply PyWavelets as the base package, but phrased in a way that astronomers are more likely to understand/use. > Coord base class might be interesting I'm not sure this is a useful way to go to start with, because there're already a number of these in place (astropysics.coords, pyephem, the astrolib coords module). While it might be a good idea to try to unify those systems somehow as has been discussed in previous threads, that's more of a community task than an actual summer project for an individual student. > Something that I see as relatively important, in the view of what IDL's > astrolib does, are tools to deal with photometry and spectroscopy. I > know that some people are fans of sextractor. I don't know to what > extent it is good and whether or not we should put efforts somewhere > else but it's definitely worth thinking. PSF photometry, both from an > analytical (arbitrary) function and from a template array, would be very > appealing, I think. Same kind of thinking should go toward spectroscopy. I agree - a well-document general purpose photometric reduction tool set seems to be lacking for python (except for pyraf, which I think doesn't really count as true python). I think SExtractor is clean (and complicated) enough that it's probably rather duplicated effort to try to actually re-implement it in python, at least at this stage. If anyone is interested, though, I have a pyton SExtractor wrapper (part of astropysics.phot) - obviously SExtractor has to be installed for it, but you can script all of the SExtractor operations in python using that module. Spectroscopy tools seem to be a bit more developed in python, though - pysynphot, astropysics.spec, PANDORA, perhaps others? - and I think there are a number of more domain-specific tools out there for spectra. So for the sake of completeness, I think photometry might be more the way to go between those. -- Erik Tollerud From wkerzendorf at googlemail.com Sat Jul 31 09:40:48 2010 From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf) Date: Sat, 31 Jul 2010 23:40:48 +1000 Subject: [AstroPy] Summer scholar In-Reply-To: <4C530F9A.3010806@gmail.com> References: <4C525220.1080508@gmail.com> <4C530F9A.3010806@gmail.com> Message-ID: <4C5427E0.6050901@gmail.com> Hi, So I think I might have to clarify the summer-scholar idea a little. When they arrive these students are often no or only novice programmers. They also often come from a Windows background and will spend two weeks or so getting used to the command-line. I understand that a wavelet analysis tool would be very useful for many astronomers, but I think far too complex for such a summer-scholarship. Maybe a good idea for a Google Summer of Code project. It is my belief that we need a project with phases. So that depending on the students skill, there's a finishing line. The student also needs a clear idea of the project path, which could be outlined in a wiki style document. The idea that Rene suggested with sextractor got me thinking. Phase I: We could have a function that finds stars in an image. It would not do photometry, but just find stars. We could model what sextractor does in this respect closely. This would also have nice byproducts like a background fitting routine. Phase II: If the student manages to do the star finding, we can try to identify stars in the image. We can use the Groth/Stetson algorithm using triangles: http://adsabs.harvard.edu/abs/1986AJ.....91.1244G Phase III: If we get this far, we could try to fit a wcs to it, maybe. I think that's very ambitious and so I am pretty sure the student would not get that far. So I think this is doable, but I'm not sure. What are people's thoughts on this? Cheers Wolfgang On 31/07/10 3:44 AM, Rene Breton wrote: > Hi Wolfgang, > > Coord base class might be interesting, though I'm not entirely sure to > know what you have in mind. Is it stuff like planar projections, > precession, coordinate transformations, etc.? Obviously, if that's > something that you need and want to see, I'd say go for it. > > Something that I see as relatively important, in the view of what IDL's > astrolib does, are tools to deal with photometry and spectroscopy. I > know that some people are fans of sextractor. I don't know to what > extent it is good and whether or not we should put efforts somewhere > else but it's definitely worth thinking. PSF photometry, both from an > analytical (arbitrary) function and from a template array, would be very > appealing, I think. Same kind of thinking should go toward spectroscopy. > > > Rene > > > On 10-07-30 12:16 AM, Wolfgang Kerzendorf wrote: >> Dear all, >> >> Here at Mt. Stromlo Observatory we will have summer scholars in November >> - February. There was a recent call for ideas for ~8 week summer >> projects. I can definitley co-supervise as this will all happen at >> Stromlo, but would need help with fleshing out an idea. Is there >> something we can offer for a physics undergrad to code for 8 weeks. >> Maybe build a coord base class? Is that too complex? Other ideas? Maybe >> we should also think about getting people in the next GSoC round? >> What are people's feeling on this? >> >> Cheers >> Wolfgang >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy