From thomas.robitaille at gmail.com Tue May 11 10:54:50 2021 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Tue, 11 May 2021 15:54:50 +0100 Subject: [AstroPy] Job ad: Software Developer at Aperio Software Message-ID: Hi everyone, This job may be of interest to some people on this list: *Aperio Software is a company that specialises in developing software and providing software training in scientific domains for a variety of organisations and open source projects. Past and current projects include the development of user tools and calibration software for the Daniel K. Inouye Solar Telescope (DKIST), work on the AAS WorldWide Telescope project, contributions to the Astropy project, and development of the glue visualisation tool.Aperio is seeking a software developer and experienced open-source contributor to work with us on our growing number of projects in solar physics and astronomy, primarily in established open-source projects. This will include working on the development, testing, documentation and maintenance of several different software packages. You will be required to liaise with clients, other staff and the open-source community. In addition, there will be the opportunity to get involved in designing and delivering programming workshops.For more details see the full job description at https://aperiosoftware.com/jobs * Please don't hesitate to email me if you have any questions about the position! Cheers, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhijeet.patil at xaviers.edu.in Thu May 13 03:22:50 2021 From: abhijeet.patil at xaviers.edu.in (Abhijeet Patil) Date: Thu, 13 May 2021 12:52:50 +0530 Subject: [AstroPy] Doubt regard astroquery.gaia Message-ID: Respected members of Astropy community, I am Abhijeet Patil, an MSc Physics student of St. Xaviers College, Mumbai, India. In my final semester I am working on an astronomy related project and I am extensively using the astropy package. I must mention that the developers have done a great job in making the software. It is a very useful package. I want to point out that, when I downloaded the GAIA data using the astroquery.gaia module with radius of 1 degree data around the center of NGC 2516, the plot of RA and DEC shows a different span for RA values. That is, from the center of RA (119.5 degree), the data spans till 121.5 deg; two times the radius selected. There is no problem for the DEC value. I tried with data encompassing stars within 2 degree and 5 degree from the center of NGC 2516. But the same problem persists. Can you please provide a proper explanation for this? I would be glad if you correct me. For your reference, I have attached the respective files. Thank you for your kind attention. Regards Abhijeet -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Data_download_gaia.py Type: text/x-python Size: 444 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 1deg.png Type: image/png Size: 516191 bytes Desc: not available URL: From pebarrett at gmail.com Thu May 13 09:07:09 2021 From: pebarrett at gmail.com (Paul Barrett) Date: Thu, 13 May 2021 09:07:09 -0400 Subject: [AstroPy] Reminder: mailing list messages are limited 100k bytes Message-ID: As AstroPy moderator, I'd like to remind everyone that messages submitted to the mailing list are limited to 100k bytes. Those over this limit must be approved by a moderator, which the moderator may decline or may not approve for several days. Please try to keep your messages below the 100k byte limit. -- Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrianmpw at gmail.com Thu May 13 09:14:07 2021 From: adrianmpw at gmail.com (Adrian Price-Whelan) Date: Thu, 13 May 2021 09:14:07 -0400 Subject: [AstroPy] Doubt regard astroquery.gaia In-Reply-To: References: Message-ID: Hi Abhijeet, The returned stars are within a separation of 1 degree: the fact that they span a larger range of longitude (RA) values is because at a given latitude (Dec), an angular displacement is ??=?RA cos(Dec), so here: ?RA ~ 2? @ Dec = 60.75? ?? ~ 2? * cos(60.75*pi/180) ~= 2? * 0.5 ~ 1? best, Adrian On Thu, May 13, 2021 at 8:53 AM Abhijeet Patil < abhijeet.patil at xaviers.edu.in> wrote: > Respected members of Astropy community, > > I am Abhijeet Patil, an MSc Physics student of St. Xaviers College, > Mumbai, India. In my final semester I am working on an astronomy related > project and I am extensively using the astropy package. I must mention that > the developers have done a great job in making the software. It is a very > useful package. > > I want to point out that, when I downloaded the GAIA data using the > astroquery.gaia module with radius of 1 degree data around the center of > NGC 2516, the plot of RA and DEC shows a different span for RA values. That > is, from the center of RA (119.5 degree), the data spans till 121.5 deg; > two times the radius selected. There is no problem for the DEC value. > > I tried with data encompassing stars within 2 degree and 5 degree from the > center of NGC 2516. But the same problem persists. > > Can you please provide a proper explanation for this? > > I would be glad if you correct me. > > For your reference, I have attached the respective files. > > Thank you for your kind attention. > > Regards > Abhijeet > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -- Adrian M. Price-Whelan Flatiron Institute, NYC http://adrn.github.io (he / him) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhijeet.patil at xaviers.edu.in Thu May 13 09:53:57 2021 From: abhijeet.patil at xaviers.edu.in (Abhijeet Patil) Date: Thu, 13 May 2021 19:23:57 +0530 Subject: [AstroPy] Doubt regard astroquery.gaia In-Reply-To: References: Message-ID: That clears all the doubts. Thank you very much. Regards Abhijeet On Thu, May 13, 2021 at 6:44 PM Adrian Price-Whelan wrote: > Hi Abhijeet, > > The returned stars are within a separation of 1 degree: the fact that they > span a larger range of longitude (RA) values is because at a given latitude > (Dec), an angular displacement is ??=?RA cos(Dec), so here: > > ?RA ~ 2? @ Dec = 60.75? > ?? ~ 2? * cos(60.75*pi/180) ~= 2? * 0.5 ~ 1? > > best, > Adrian > > On Thu, May 13, 2021 at 8:53 AM Abhijeet Patil < > abhijeet.patil at xaviers.edu.in> wrote: > >> Respected members of Astropy community, >> >> I am Abhijeet Patil, an MSc Physics student of St. Xaviers College, >> Mumbai, India. In my final semester I am working on an astronomy related >> project and I am extensively using the astropy package. I must mention that >> the developers have done a great job in making the software. It is a very >> useful package. >> >> I want to point out that, when I downloaded the GAIA data using the >> astroquery.gaia module with radius of 1 degree data around the center of >> NGC 2516, the plot of RA and DEC shows a different span for RA values. That >> is, from the center of RA (119.5 degree), the data spans till 121.5 deg; >> two times the radius selected. There is no problem for the DEC value. >> >> I tried with data encompassing stars within 2 degree and 5 degree from >> the center of NGC 2516. But the same problem persists. >> >> Can you please provide a proper explanation for this? >> >> I would be glad if you correct me. >> >> For your reference, I have attached the respective files. >> >> Thank you for your kind attention. >> >> Regards >> Abhijeet >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy >> > > > -- > Adrian M. Price-Whelan > Flatiron Institute, NYC > http://adrn.github.io > (he / him) > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhijeet.patil at xaviers.edu.in Thu May 13 09:57:25 2021 From: abhijeet.patil at xaviers.edu.in (Abhijeet Patil) Date: Thu, 13 May 2021 19:27:25 +0530 Subject: [AstroPy] Reminder: mailing list messages are limited 100k bytes In-Reply-To: References: Message-ID: Yes I will keep that in mind. Thank you for letting me know. Regards Abhijeet On Thu, May 13, 2021 at 6:37 PM Paul Barrett wrote: > As AstroPy moderator, I'd like to remind everyone that messages submitted > to the mailing list are limited to 100k bytes. Those over this limit must > be approved by a moderator, which the moderator may decline or may not > approve for several days. > > Please try to keep your messages below the 100k byte limit. > > -- Paul > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From npkuin at gmail.com Tue May 18 11:38:09 2021 From: npkuin at gmail.com (Paul Kuin) Date: Tue, 18 May 2021 16:38:09 +0100 Subject: [AstroPy] weird problem with table column operation Message-ID: Hi, I probably do something wrong. I have a Table with multiple columns, and one with a single colum. In [80]: type(tab),type(c1) Out[80]: (astropy.table.table.Table, astropy.table.table.Table) The data type in c1 is : tdb_minus_tt float64 ------------------- -33.9968477608636 -3.307830076664686 .... I add this c1 as aa new column using tab.add_column(c1,name='TDB_MINUS_TT') [if I do not add the name the name does not take) now the column is added to table TAB, BUT, the data type is now changed to void64 !? and the values have now been changed from float to (-33.9968477608636 ,) etc. I see no mention in the documentation anywhere of anything like this. Am I doing something stupid or is this a bug? In [86]: astropy.__version__ Out[86]: '4.0.2' Anny help will be appreciated! Thanks, Paul -- * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * * N.P.M. Kuin (n.kuin at ucl.ac.uk) -- mobile +44(0)7908715953 -- phone +44-(0)1483 (prefix) -204111 (when back to work) Mullard Space Science Laboratory ? University College London ? Holmbury St Mary ? Dorking ? Surrey RH5 6NT? U.K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrianmpw at gmail.com Tue May 18 11:46:49 2021 From: adrianmpw at gmail.com (Adrian Price-Whelan) Date: Tue, 18 May 2021 11:46:49 -0400 Subject: [AstroPy] weird problem with table column operation In-Reply-To: References: Message-ID: Hi Paul, This could be because your object c1 is actually a full astropy Table rather than just a column, so you are adding a column to "tab" that is a full table object. Try: tab.add_column(c1['TDB_MINUS_TT']) or tabl['TDB_MINUS_TT'] = c1['TDB_MINUS_TT'] instead? best, Adrian On Tue, May 18, 2021 at 11:38 AM Paul Kuin wrote: > Hi, > > I probably do something wrong. > > I have a Table with multiple columns, and one with a single colum. > In [80]: type(tab),type(c1) > Out[80]: (astropy.table.table.Table, astropy.table.table.Table) > The data type in c1 is : >
> tdb_minus_tt > float64 > ------------------- > -33.9968477608636 > -3.307830076664686 > .... > > I add this c1 as aa new column using > tab.add_column(c1,name='TDB_MINUS_TT') [if I do not add the name the name > does not take) > > now the column is added to table TAB, BUT, the data type is now changed to > void64 !? > and the values have now been changed from float to (-33.9968477608636 ,) > etc. > > I see no mention in the documentation anywhere of anything like this. > Am I doing something stupid or is this a bug? > > In [86]: astropy.__version__ > Out[86]: '4.0.2' > > Anny help will be appreciated! > > Thanks, > Paul > > -- > > * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * * > N.P.M. Kuin (n.kuin at ucl.ac.uk) -- mobile +44(0)7908715953 -- phone > +44-(0)1483 (prefix) -204111 (when back to work) > Mullard Space Science Laboratory ? University College London ? Holmbury > St Mary ? Dorking ? Surrey RH5 6NT? U.K. > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -- Adrian M. Price-Whelan Flatiron Institute, NYC http://adrn.github.io (he / him) -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Tue May 18 12:00:14 2021 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Tue, 18 May 2021 12:00:14 -0400 Subject: [AstroPy] weird problem with table column operation In-Reply-To: References: Message-ID: Hi Paul, You might also find the Table hstack() function useful. This will let you add all the columns of one table into another table. https://docs.astropy.org/en/stable/table/operations.html#id6 In your case this might look like: from astropy.table import hstack tab = hstack([tab, c1]) - Tom On Tue, May 18, 2021 at 11:47 AM Adrian Price-Whelan wrote: > Hi Paul, > > This could be because your object c1 is actually a full astropy Table > rather than just a column, so you are adding a column to "tab" that is a > full table object. Try: > > tab.add_column(c1['TDB_MINUS_TT']) > or > tabl['TDB_MINUS_TT'] = c1['TDB_MINUS_TT'] > > instead? > > best, > Adrian > > On Tue, May 18, 2021 at 11:38 AM Paul Kuin wrote: > >> Hi, >> >> I probably do something wrong. >> >> I have a Table with multiple columns, and one with a single colum. >> In [80]: type(tab),type(c1) >> Out[80]: (astropy.table.table.Table, astropy.table.table.Table) >> The data type in c1 is : >>
>> tdb_minus_tt >> float64 >> ------------------- >> -33.9968477608636 >> -3.307830076664686 >> .... >> >> I add this c1 as aa new column using >> tab.add_column(c1,name='TDB_MINUS_TT') [if I do not add the name the name >> does not take) >> >> now the column is added to table TAB, BUT, the data type is now changed >> to void64 !? >> and the values have now been changed from float to (-33.9968477608636 ,) >> etc. >> >> I see no mention in the documentation anywhere of anything like this. >> Am I doing something stupid or is this a bug? >> >> In [86]: astropy.__version__ >> Out[86]: '4.0.2' >> >> Anny help will be appreciated! >> >> Thanks, >> Paul >> >> -- >> >> * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * * >> N.P.M. Kuin (n.kuin at ucl.ac.uk) -- mobile +44(0)7908715953 -- phone >> +44-(0)1483 (prefix) -204111 (when back to work) >> Mullard Space Science Laboratory ? University College London ? Holmbury >> St Mary ? Dorking ? Surrey RH5 6NT? U.K. >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy >> > > > -- > Adrian M. Price-Whelan > Flatiron Institute, NYC > http://adrn.github.io > (he / him) > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From npkuin at gmail.com Tue May 18 12:07:02 2021 From: npkuin at gmail.com (Paul Kuin) Date: Tue, 18 May 2021 17:07:02 +0100 Subject: [AstroPy] weird problem with table column operation In-Reply-To: References: Message-ID: Thank you guys, both sound sensible to me. So it was me. lol. Paul On Tue, May 18, 2021 at 5:02 PM Aldcroft, Thomas < aldcroft at head.cfa.harvard.edu> wrote: > Hi Paul, > > You might also find the Table hstack() function useful. This will let you > add all the columns of one table into another table. > > https://docs.astropy.org/en/stable/table/operations.html#id6 > > In your case this might look like: > > from astropy.table import hstack > tab = hstack([tab, c1]) > > - Tom > > > On Tue, May 18, 2021 at 11:47 AM Adrian Price-Whelan > wrote: > >> Hi Paul, >> >> This could be because your object c1 is actually a full astropy Table >> rather than just a column, so you are adding a column to "tab" that is a >> full table object. Try: >> >> tab.add_column(c1['TDB_MINUS_TT']) >> or >> tabl['TDB_MINUS_TT'] = c1['TDB_MINUS_TT'] >> >> instead? >> >> best, >> Adrian >> >> On Tue, May 18, 2021 at 11:38 AM Paul Kuin wrote: >> >>> Hi, >>> >>> I probably do something wrong. >>> >>> I have a Table with multiple columns, and one with a single colum. >>> In [80]: type(tab),type(c1) >>> Out[80]: (astropy.table.table.Table, astropy.table.table.Table) >>> The data type in c1 is : >>>
>>> tdb_minus_tt >>> float64 >>> ------------------- >>> -33.9968477608636 >>> -3.307830076664686 >>> .... >>> >>> I add this c1 as aa new column using >>> tab.add_column(c1,name='TDB_MINUS_TT') [if I do not add the name the >>> name does not take) >>> >>> now the column is added to table TAB, BUT, the data type is now changed >>> to void64 !? >>> and the values have now been changed from float to (-33.9968477608636 >>> ,) etc. >>> >>> I see no mention in the documentation anywhere of anything like this. >>> Am I doing something stupid or is this a bug? >>> >>> In [86]: astropy.__version__ >>> Out[86]: '4.0.2' >>> >>> Anny help will be appreciated! >>> >>> Thanks, >>> Paul >>> >>> -- >>> >>> * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * * >>> N.P.M. Kuin (n.kuin at ucl.ac.uk) -- mobile +44(0)7908715953 -- phone >>> +44-(0)1483 (prefix) -204111 (when back to work) >>> Mullard Space Science Laboratory ? University College London >>> ? Holmbury St Mary ? Dorking ? Surrey RH5 6NT? U.K. >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at python.org >>> https://mail.python.org/mailman/listinfo/astropy >>> >> >> >> -- >> Adrian M. Price-Whelan >> Flatiron Institute, NYC >> http://adrn.github.io >> (he / him) >> _______________________________________________ >> AstroPy mailing list >> AstroPy at python.org >> https://mail.python.org/mailman/listinfo/astropy >> > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbaer25 at gmail.com Wed May 19 11:43:05 2021 From: rbaer25 at gmail.com (Rudolf Baer) Date: Wed, 19 May 2021 17:43:05 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy Message-ID: I am fitting the continuum of a NIR spectrum with a blackbody plus power law using lmfit and scipy. The plot of the fit looks good and the resulting parameters agree reasonably with previous results. T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) However the chi-square and reduced chi-square values (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) are clearly wrong. The problem is apparently known as the scipy guide says* Note that the calculation of chi-square and* *reduced chi-square assume that the returned residual function is scaled properly to the uncertainties in the data. * *For these statistics to be meaningful, the person writing the function to be minimized must scale them properly.* I do not know how to do this. Any advice will be appreciated. With kind regards Rudolf Baer Zurich The input data for the code are x: lambda in AA, y: flux density in erg/cm^{2}/s/{\AA) Code: x=x_AA/10000. #<---------conversion AA to um y=y*1e4 y=y*x #========================================================================================= def bb(x, T, const): from scipy.constants import h,k,c x = 1e-6 * x # convert to metres from um return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) #J/s/m2/m def powerlaw(x,A,p): return A*x**p mod= Model(bb) + Model(powerlaw) pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) result = mod.fit(y,pars,x=x) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivvv68 at gmail.com Wed May 19 16:19:01 2021 From: ivvv68 at gmail.com (Ivan Valtchanov) Date: Wed, 19 May 2021 22:19:01 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: References: Message-ID: Hi Rudolf, I would recommend to normalise the y-values within [0,1], I suspect the results you see are due to numerical accuracy limits. Cheers, Ivan V On Wed, 19 May 2021 at 17:43, Rudolf Baer wrote: > > I am fitting the continuum of a NIR spectrum with a blackbody plus power law using lmfit and scipy. The plot of the fit looks good > and the resulting parameters agree reasonably with previous results. > > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > However the chi-square and reduced chi-square values > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > are clearly wrong. The problem is apparently known as the scipy guide says Note that the calculation of chi-square and > reduced chi-square assume that the returned residual function is scaled properly to the uncertainties in the data. > For these statistics to be meaningful, the person writing the function to be minimized must scale them properly. > > I do not know how to do this. Any advice will be appreciated. > > With kind regards > Rudolf Baer > Zurich > > > > The input data for the code are x: lambda in AA, y: flux density in erg/cm^{2}/s/{\AA) > Code: > > x=x_AA/10000. #<---------conversion AA to um > > y=y*1e4 > > y=y*x > #========================================================================================= > > def bb(x, T, const): > > from scipy.constants import h,k,c > > x = 1e-6 * x # convert to metres from um > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) #J/s/m2/m > > > def powerlaw(x,A,p): > > return A*x**p > > > mod= Model(bb) + Model(powerlaw) > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > result = mod.fit(y,pars,x=x) > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy From peter.erwin at gmail.com Wed May 19 16:37:35 2021 From: peter.erwin at gmail.com (Peter Erwin) Date: Wed, 19 May 2021 22:37:35 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: References: Message-ID: <5170A1CC-ED9F-4D52-8D39-3F896310980E@gmail.com> Rudolf, It?s not clear to me what you are doing. You say you are using ?lmfit and scipy?. By ?lmfit? do you mean the lmfit package ? https://lmfit.github.io/lmfit-py/index.html ? This has nothing to do with scipy (though it assumes numpy exists), so maybe you are using something else instead? In any case, I?d guess the chi^2 values are ?wrong? because you don?t have good error estimates (?uncertainties?) for the data values. Providing examples of the actual Python commands you are using (and maybe a few sample data values from your NIR spectrum) would be helpful. cheers, Peter > On May 19, 2021, at 5:43 PM, Rudolf Baer wrote: > > I am fitting the continuum of a NIR spectrum with a blackbody plus power law using lmfit and scipy. The plot of the fit looks good > and the resulting parameters agree reasonably with previous results. > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > However the chi-square and reduced chi-square values > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > are clearly wrong. The problem is apparently known as the scipy guide says Note that the calculation of chi-square and > reduced chi-square assume that the returned residual function is scaled properly to the uncertainties in the data. > For these statistics to be meaningful, the person writing the function to be minimized must scale them properly. > > I do not know how to do this. Any advice will be appreciated. > > With kind regards > Rudolf Baer > Zurich > > > > The input data for the code are x: lambda in AA, y: flux density in erg/cm^{2}/s/{\AA) > Code: > > x=x_AA/10000. #<---------conversion AA to um > > y=y*1e4 > > y=y*x > #========================================================================================= > > def bb(x, T, const): > > from scipy.constants import h,k,c > > x = 1e-6 * x # convert to metres from um > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) #J/s/m2/m > > > def powerlaw(x,A,p): > > return A*x**p > > > mod= Model(bb) + Model(powerlaw) > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > result = mod.fit(y,pars,x=x) > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)176 2481 7713 85748 Garching, Germany fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin From rbaer25 at gmail.com Thu May 20 04:56:15 2021 From: rbaer25 at gmail.com (Rudolf Baer) Date: Thu, 20 May 2021 10:56:15 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: References: Message-ID: Hi Ivan brilliant ! it seems to work. Thank you very much Rudolf T: 1109.28599 +/- 3.08367329 (0.28%) (init = 1100) const: 1.0685e-10 +/- 1.4258e-12 (1.33%) (init = 1) A: 0.54709366 +/- 0.00142887 (0.26%) (init = 1) p: -1.06700052 +/- 0.01701320 (1.59%) (init = -1) # fitting method = leastsq # function evals = 36 # data points = 5360 # variables = 4 chi-square = 5.76471199 reduced chi-square = 0.00107631 On Wed, May 19, 2021 at 10:19 PM Ivan Valtchanov wrote: > Hi Rudolf, > > I would recommend to normalise the y-values within [0,1], I suspect > the results you see are due to numerical accuracy limits. > > Cheers, > Ivan V > > > On Wed, 19 May 2021 at 17:43, Rudolf Baer wrote: > > > > I am fitting the continuum of a NIR spectrum with a blackbody plus power > law using lmfit and scipy. The plot of the fit looks good > > and the resulting parameters agree reasonably with previous results. > > > > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > > > However the chi-square and reduced chi-square values > > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > > are clearly wrong. The problem is apparently known as the scipy guide > says Note that the calculation of chi-square and > > reduced chi-square assume that the returned residual function is scaled > properly to the uncertainties in the data. > > For these statistics to be meaningful, the person writing the function > to be minimized must scale them properly. > > > > I do not know how to do this. Any advice will be appreciated. > > > > With kind regards > > Rudolf Baer > > Zurich > > > > > > > > The input data for the code are x: lambda in AA, y: flux density in > erg/cm^{2}/s/{\AA) > > Code: > > > > x=x_AA/10000. #<---------conversion AA to um > > > > y=y*1e4 > > > > y=y*x > > > #========================================================================================= > > > > def bb(x, T, const): > > > > from scipy.constants import h,k,c > > > > x = 1e-6 * x # convert to metres from um > > > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) > #J/s/m2/m > > > > > > def powerlaw(x,A,p): > > > > return A*x**p > > > > > > mod= Model(bb) + Model(powerlaw) > > > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > > > result = mod.fit(y,pars,x=x) > > _______________________________________________ > > AstroPy mailing list > > AstroPy at python.org > > https://mail.python.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.erwin at gmail.com Thu May 20 06:02:05 2021 From: peter.erwin at gmail.com (Peter Erwin) Date: Thu, 20 May 2021 12:02:05 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: References: Message-ID: <4C519685-25FA-443E-A3E7-8EEA8EA41290@gmail.com> Hi Rudolf, My apologies -- I had failed to see that you *had* included the code you used in your original post. So, please ignore my previous email. That said (and having taken a closer look at lmfit), it's clear that the "chi^2" value returned by lmfit in your example is almost certainly what you *should* get, given that you have not provided any error values to go with your data. So it's almost certainly not some effect of "numerical accuracy limits". The Model.fit method of lmfit has an optional input called "weights". For standard chi^2 minimization, these should have values of 1/sigma, where sigma are the error values corresponding to the data values, *in the same units*. If you *don't* supply a "weights" vector, then lmfit effectively treats all data points have sigma = 1, which is of course not true for your data. What lmfit minimizes is the sum of (weights*(data - model))^2. If weights = 1/sigma, then this is the sum of ((data - model)/sigma)^2, which is the standard chi^2 formula for data with errors, and the "reduced" version of this will (assuming the errors are accurate and quasi-Gaussian) approximate a chi^2 distribution and can be used (with suitable caution) as a crude "goodness of fit" indicator. But if weights=None [the default if you don't supply a weights keyword], then internally lmfit minimizes the sum of (data - model)^2. The "reduced chi^2" version of this is simply [sum of (data - model)^2]/(N_data - N_var), which for large N_data is approximately the mean value of (data - model)^2. You don't provide any examples of data values, I'm guessing they're somewhere in the neighborhood of 10^11 or 10^-12. This is because that's the square root of your "reduced chi^2" value, and is something like the RMS value of (data - model). So it's important to understand that without providing the correct error values as input to the Model.fit method (via the "weights" keyword), the "chi^2" and "reduced chi^2" values returned by lmfit are *meaningless*. Rescaling your data so it has maximum values of 1 will give you different but *equally meaningless* values of chi^2 and reduced chi^2. cheers, Peter > On May 19, 2021, at 5:43 PM, Rudolf Baer wrote: > > I am fitting the continuum of a NIR spectrum with a blackbody plus power law using lmfit and scipy. The plot of the fit looks good > and the resulting parameters agree reasonably with previous results. > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > However the chi-square and reduced chi-square values > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > are clearly wrong. The problem is apparently known as the scipy guide says Note that the calculation of chi-square and > reduced chi-square assume that the returned residual function is scaled properly to the uncertainties in the data. > For these statistics to be meaningful, the person writing the function to be minimized must scale them properly. > > I do not know how to do this. Any advice will be appreciated. > > With kind regards > Rudolf Baer > Zurich > > > > The input data for the code are x: lambda in AA, y: flux density in erg/cm^{2}/s/{\AA) > Code: > > x=x_AA/10000. #<---------conversion AA to um > > y=y*1e4 > > y=y*x > #========================================================================================= > > def bb(x, T, const): > > from scipy.constants import h,k,c > > x = 1e-6 * x # convert to metres from um > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) #J/s/m2/m > > > def powerlaw(x,A,p): > > return A*x**p > > > mod= Model(bb) + Model(powerlaw) > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > result = mod.fit(y,pars,x=x) > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)176 2481 7713 85748 Garching, Germany fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin From rbaer25 at gmail.com Thu May 20 05:11:39 2021 From: rbaer25 at gmail.com (Rudolf Baer) Date: Thu, 20 May 2021 11:11:39 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: <5170A1CC-ED9F-4D52-8D39-3F896310980E@gmail.com> References: <5170A1CC-ED9F-4D52-8D39-3F896310980E@gmail.com> Message-ID: Hi Peter thanks for your reply. I have solved the problem by normalizing the y values as suggested by Ivan. With my first e mail I provided the python code. I attach the actual data file. Your comment on the errors is correct; the file has no errors. I have introduced a gaussian error, y_noise=(0.02*y * np.random.normal(size=len(x))) y=y+y_noise but it does not affect the results. with kind regards Rudolf On Wed, May 19, 2021 at 10:38 PM Peter Erwin wrote: > Rudolf, > > It?s not clear to me what you are doing. You say you are using ?lmfit and > scipy?. > By ?lmfit? do you mean the lmfit package ? > https://lmfit.github.io/lmfit-py/index.html ? > > This has nothing to do with scipy (though it assumes numpy exists), so > maybe you > are using something else instead? > > In any case, I?d guess the chi^2 values are ?wrong? because you don?t have > good error > estimates (?uncertainties?) for the data values. > > Providing examples of the actual Python commands you are using (and maybe > a few > sample data values from your NIR spectrum) would be helpful. > > cheers, > > Peter > > > On May 19, 2021, at 5:43 PM, Rudolf Baer wrote: > > > > I am fitting the continuum of a NIR spectrum with a blackbody plus power > law using lmfit and scipy. The plot of the fit looks good > > and the resulting parameters agree reasonably with previous results. > > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > > > However the chi-square and reduced chi-square values > > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > > are clearly wrong. The problem is apparently known as the scipy guide > says Note that the calculation of chi-square and > > reduced chi-square assume that the returned residual function is scaled > properly to the uncertainties in the data. > > For these statistics to be meaningful, the person writing the function > to be minimized must scale them properly. > > > > I do not know how to do this. Any advice will be appreciated. > > > > With kind regards > > Rudolf Baer > > Zurich > > > > > > > > The input data for the code are x: lambda in AA, y: flux density in > erg/cm^{2}/s/{\AA) > > Code: > > > > x=x_AA/10000. #<---------conversion AA to um > > > > y=y*1e4 > > > > y=y*x > > > #========================================================================================= > > > > def bb(x, T, const): > > > > from scipy.constants import h,k,c > > > > x = 1e-6 * x # convert to metres from um > > > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) > #J/s/m2/m > > > > > > def powerlaw(x,A,p): > > > > return A*x**p > > > > > > mod= Model(bb) + Model(powerlaw) > > > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > > > result = mod.fit(y,pars,x=x) > > _______________________________________________ > > AstroPy mailing list > > AstroPy at python.org > > https://mail.python.org/mailman/listinfo/astropy > > ============================================================= > Peter Erwin Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de Physics, Giessenbachstrasse > tel. +49 (0)176 2481 7713 85748 Garching, Germany > fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: baseline_BASS0006 0407.csv Type: text/csv Size: 204036 bytes Desc: not available URL: From pebarrett at gmail.com Thu May 20 07:28:08 2021 From: pebarrett at gmail.com (Paul Barrett) Date: Thu, 20 May 2021 07:28:08 -0400 Subject: [AstroPy] Reminder: mailing list messages are limited 100k bytes In-Reply-To: References: Message-ID: Note that the 100k byte limit for messages still holds. Please make every effort to comply with this limit. Respectfully, Paul AstroPy moderator On Thu, May 13, 2021 at 9:07 AM Paul Barrett wrote: > As AstroPy moderator, I'd like to remind everyone that messages submitted > to the mailing list are limited to 100k bytes. Those over this limit must > be approved by a moderator, which the moderator may decline or may not > approve for several days. > > Please try to keep your messages below the 100k byte limit. > > -- Paul > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbaer25 at gmail.com Thu May 20 08:22:49 2021 From: rbaer25 at gmail.com (Rudolf Baer) Date: Thu, 20 May 2021 14:22:49 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: <4C519685-25FA-443E-A3E7-8EEA8EA41290@gmail.com> References: <4C519685-25FA-443E-A3E7-8EEA8EA41290@gmail.com> Message-ID: Hi Peter, the original y values are about 4.26582479e-15 4.24795832e-15 4.13801217e-15 ... 1.88696367e-15 1.88558131e-15 1.88894658e-15 The data file which I attached previously was held up by th some basicse moderator as it is larger than the allowed size.; if you give me your e-mail address I will send it to you directly. As the original data do not have errors, what should I do? Can I introduce an artificial errort? Maybe I am missing some basics. I am fitting a continuum with a BB+power law. The chi^2 in my mind applies to this fit. Where does the error in original data come in? If I have to add weights (Model.fit method (via the "weights" keyword)) where exactly in my code can I do this? with kind regards Rudolf [image: Screenshot 2021-05-20 at 14.19.13.png] On Thu, May 20, 2021 at 12:02 PM Peter Erwin wrote: > Hi Rudolf, > > My apologies -- I had failed to see that you *had* included the code you > used in > your original post. > So, please ignore my previous email. > > > That said (and having taken a closer look at lmfit), it's clear that the > "chi^2" > value returned by lmfit in your example is almost certainly what you > *should* get, > given that you have not provided any error values to go with your data. So > it's > almost certainly not some effect of "numerical accuracy limits". > > The Model.fit method of lmfit has an optional input called "weights". For > standard chi^2 minimization, these should have values of 1/sigma, where > sigma > are the error values corresponding to the data values, *in the same units*. > If you *don't* supply a "weights" vector, then lmfit effectively treats all > data points have sigma = 1, which is of course not true for your data. > > What lmfit minimizes is the sum of (weights*(data - model))^2. If weights > = 1/sigma, > then this is the sum of ((data - model)/sigma)^2, which is the standard > chi^2 > formula for data with errors, and the "reduced" version of this will > (assuming the > errors are accurate and quasi-Gaussian) approximate a chi^2 distribution > and can be used (with suitable caution) as a crude "goodness of fit" > indicator. > > But if weights=None [the default if you don't supply a weights keyword], > then > internally lmfit minimizes the sum of (data - model)^2. > The "reduced chi^2" version of this is simply [sum of (data - > model)^2]/(N_data - N_var), > which for large N_data is approximately the mean value of (data - model)^2. > > You don't provide any examples of data values, I'm guessing they're > somewhere > in the neighborhood of 10^11 or 10^-12. This is because that's the square > root of your > "reduced chi^2" value, and is something like the RMS value of (data - > model). > > > So it's important to understand that without providing the correct error > values > as input to the Model.fit method (via the "weights" keyword), the "chi^2" > and > "reduced chi^2" values returned by lmfit are *meaningless*. Rescaling your > data > so it has maximum values of 1 will give you different but *equally > meaningless* > values of chi^2 and reduced chi^2. > > cheers, > > Peter > > > > On May 19, 2021, at 5:43 PM, Rudolf Baer wrote: > > > > I am fitting the continuum of a NIR spectrum with a blackbody plus power > law using lmfit and scipy. The plot of the fit looks good > > and the resulting parameters agree reasonably with previous results. > > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > > > However the chi-square and reduced chi-square values > > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > > are clearly wrong. The problem is apparently known as the scipy guide > says Note that the calculation of chi-square and > > reduced chi-square assume that the returned residual function is scaled > properly to the uncertainties in the data. > > For these statistics to be meaningful, the person writing the function > to be minimized must scale them properly. > > > > I do not know how to do this. Any advice will be appreciated. > > > > With kind regards > > Rudolf Baer > > Zurich > > > > > > > > The input data for the code are x: lambda in AA, y: flux density in > erg/cm^{2}/s/{\AA) > > Code: > > > > x=x_AA/10000. #<---------conversion AA to um > > > > y=y*1e4 > > > > y=y*x > > > #========================================================================================= > > > > def bb(x, T, const): > > > > from scipy.constants import h,k,c > > > > x = 1e-6 * x # convert to metres from um > > > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) > #J/s/m2/m > > > > > > def powerlaw(x,A,p): > > > > return A*x**p > > > > > > mod= Model(bb) + Model(powerlaw) > > > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > > > result = mod.fit(y,pars,x=x) > > _______________________________________________ > > AstroPy mailing list > > AstroPy at python.org > > https://mail.python.org/mailman/listinfo/astropy > > ============================================================= > Peter Erwin Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de Physics, Giessenbachstrasse > tel. +49 (0)176 2481 7713 85748 Garching, Germany > fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2021-05-20 at 14.19.13.png Type: image/png Size: 60335 bytes Desc: not available URL: From erwin at mpe.mpg.de Thu May 20 08:17:28 2021 From: erwin at mpe.mpg.de (Peter Erwin) Date: Thu, 20 May 2021 14:17:28 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: References: <5170A1CC-ED9F-4D52-8D39-3F896310980E@gmail.com> Message-ID: <1F0BC1C8-7753-41D9-9BF2-402EB99994BD@mpe.mpg.de> Hi Rudolf, As I suggested in my second comment, you haven?t actually ?solved? your problem by normalizing the y values; you?ve just transformed one (meaningless) chi^2 value into a different (meaningless) value. What I meant by ?missing errors? was not ?you need to invent some errors and add them to your data? ? which is what your ?y = y + y_noise? is doing. I meant that the data should ideally come with noise estimates (e.g., from whatever pipeline processed the original data). If all you have is the y values by themselves (in the form of e.g. flux values, as yours seem to be), then there?s really nothing to be done. Just be aware that the fit is not ideal (because all the points are being weighted equally, regardless of their actual S/N) and the reduced chi^2 value has no useful information about the ?goodness? of the fit. cheers, Peter > On May 20, 2021, at 11:11 AM, Rudolf Baer wrote: > > Hi Peter > thanks for your reply. > I have solved the problem by normalizing the y values as suggested by Ivan. > With my first e mail I provided the python code. I attach the actual data file. > Your comment on the errors is correct; the file has no errors. I have introduced a gaussian error, > y_noise=(0.02*y * np.random.normal(size=len(x))) > y=y+y_noise > but it does not affect the results. > > with kind regards > Rudolf > > > On Wed, May 19, 2021 at 10:38 PM Peter Erwin wrote: > Rudolf, > > It?s not clear to me what you are doing. You say you are using ?lmfit and scipy?. > By ?lmfit? do you mean the lmfit package ? https://lmfit.github.io/lmfit-py/index.html ? > > This has nothing to do with scipy (though it assumes numpy exists), so maybe you > are using something else instead? > > In any case, I?d guess the chi^2 values are ?wrong? because you don?t have good error > estimates (?uncertainties?) for the data values. > > Providing examples of the actual Python commands you are using (and maybe a few > sample data values from your NIR spectrum) would be helpful. > > cheers, > > Peter > > > On May 19, 2021, at 5:43 PM, Rudolf Baer wrote: > > > > I am fitting the continuum of a NIR spectrum with a blackbody plus power law using lmfit and scipy. The plot of the fit looks good > > and the resulting parameters agree reasonably with previous results. > > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > > > However the chi-square and reduced chi-square values > > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > > are clearly wrong. The problem is apparently known as the scipy guide says Note that the calculation of chi-square and > > reduced chi-square assume that the returned residual function is scaled properly to the uncertainties in the data. > > For these statistics to be meaningful, the person writing the function to be minimized must scale them properly. > > > > I do not know how to do this. Any advice will be appreciated. > > > > With kind regards > > Rudolf Baer > > Zurich > > > > > > > > The input data for the code are x: lambda in AA, y: flux density in erg/cm^{2}/s/{\AA) > > Code: > > > > x=x_AA/10000. #<---------conversion AA to um > > > > y=y*1e4 > > > > y=y*x > > #========================================================================================= > > > > def bb(x, T, const): > > > > from scipy.constants import h,k,c > > > > x = 1e-6 * x # convert to metres from um > > > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) #J/s/m2/m > > > > > > def powerlaw(x,A,p): > > > > return A*x**p > > > > > > mod= Model(bb) + Model(powerlaw) > > > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > > > result = mod.fit(y,pars,x=x) > > _______________________________________________ > > AstroPy mailing list > > AstroPy at python.org > > https://mail.python.org/mailman/listinfo/astropy > > ============================================================= > Peter Erwin Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de Physics, Giessenbachstrasse > tel. +49 (0)176 2481 7713 85748 Garching, Germany > fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)176 2481 7713 85748 Garching, Germany fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin From cbrasseur at stsci.edu Thu May 20 14:23:19 2021 From: cbrasseur at stsci.edu (Clara Brasseur) Date: Thu, 20 May 2021 18:23:19 +0000 Subject: [AstroPy] Soliciting Astropy Voting Member Nominations Message-ID: Dear colleagues, We are writing to encourage nominations for additional voting members on the Astropy project. You can nominate yourself or others. Following the governance process a set of initial voting members has been invited. The initial voting members who have thus far accepted are listed on the Astropy team page. The next step in the process is to solicit nominations from the community: you! You can nominate yourself or someone else. The initial set of voting members will vote on the nominations. Once the voting members are set, nominations for the coordinating committee will be opened. As per APE0, Voting members are expected to demonstrate: * A good grasp of the philosophy of the Astropy Project * A solid track record of being constructive and helpful * Significant contributions to the Project?s goals, in any form * A willingness to dedicate time to improving the Project * A willingness to recruit, train, and nominate new team members To nominate yourself or someone else please fill out the form at https://forms.gle/7Tbrague85n5mzjb6. (This form is anonymous) Nominations for those who have made significant contributions in the past, even if they are no longer actively involved in the project, are encouraged. All nominees will be asked whether they wish to accept the nomination, decline the nomination, or wish to be considered for emeritus. Following the process outlined in the governance process, the names of nominees will be known only to the CoCo, the Ombudsperson, and the voting members. The intended timeline for the nominations and elections is: * Nominations for this round close at midnight on June 7, 2021. To be clear, the nominations stay open as long as anywhere on Earth has not yet reached midnight on June 7, 2021. * The election will begin on June 21, 2021. * The election will close on July 9, 2021. Sincerely, Astropy governance implementation group -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbaer25 at gmail.com Fri May 21 04:22:13 2021 From: rbaer25 at gmail.com (Rudolf Baer) Date: Fri, 21 May 2021 10:22:13 +0200 Subject: [AstroPy] Chi-square problems with lmfit and scipy In-Reply-To: <1F0BC1C8-7753-41D9-9BF2-402EB99994BD@mpe.mpg.de> References: <5170A1CC-ED9F-4D52-8D39-3F896310980E@gmail.com> <1F0BC1C8-7753-41D9-9BF2-402EB99994BD@mpe.mpg.de> Message-ID: Hi Peter thank you very much for your help. Cheers Rudolf On Thu, May 20, 2021 at 2:37 PM Peter Erwin wrote: > Hi Rudolf, > > As I suggested in my second comment, you haven?t actually ?solved? your > problem > by normalizing the y values; you?ve just transformed one (meaningless) > chi^2 value > into a different (meaningless) value. > > What I meant by ?missing errors? was not ?you need to invent some errors > and add > them to your data? ? which is what your ?y = y + y_noise? is doing. I > meant that > the data should ideally come with noise estimates (e.g., from whatever > pipeline > processed the original data). > > If all you have is the y values by themselves (in the form of e.g. flux > values, as yours > seem to be), then there?s really nothing to be done. Just be aware that > the fit is not > ideal (because all the points are being weighted equally, regardless of > their actual > S/N) and the reduced chi^2 value has no useful information about the > ?goodness? > of the fit. > > cheers, > > Peter > > > On May 20, 2021, at 11:11 AM, Rudolf Baer wrote: > > > > Hi Peter > > thanks for your reply. > > I have solved the problem by normalizing the y values as suggested by > Ivan. > > With my first e mail I provided the python code. I attach the actual > data file. > > Your comment on the errors is correct; the file has no errors. I have > introduced a gaussian error, > > y_noise=(0.02*y * np.random.normal(size=len(x))) > > y=y+y_noise > > but it does not affect the results. > > > > with kind regards > > Rudolf > > > > > > On Wed, May 19, 2021 at 10:38 PM Peter Erwin > wrote: > > Rudolf, > > > > It?s not clear to me what you are doing. You say you are using ?lmfit > and scipy?. > > By ?lmfit? do you mean the lmfit package ? > https://lmfit.github.io/lmfit-py/index.html ? > > > > This has nothing to do with scipy (though it assumes numpy exists), so > maybe you > > are using something else instead? > > > > In any case, I?d guess the chi^2 values are ?wrong? because you don?t > have good error > > estimates (?uncertainties?) for the data values. > > > > Providing examples of the actual Python commands you are using (and > maybe a few > > sample data values from your NIR spectrum) would be helpful. > > > > cheers, > > > > Peter > > > > > On May 19, 2021, at 5:43 PM, Rudolf Baer wrote: > > > > > > I am fitting the continuum of a NIR spectrum with a blackbody plus > power law using lmfit and scipy. The plot of the fit looks good > > > and the resulting parameters agree reasonably with previous results. > > > T: 1109.29505 +/- 3.08340774 (0.28%) (init = 1100) > > > const: 4.7240e-21 +/- 6.3014e-23 (1.33%) (init = 2e-21) > > > A: 2.4188e-11 +/- 6.3181e-14 (0.26%) (init = 2e-11) > > > p: -1.06704890 +/- 0.01701493 (1.59%) (init = -1) > > > > > > However the chi-square and reduced chi-square values > > > (chi-square = 1.1268e-20 reduced chi-square = 2.1039e-24) > > > are clearly wrong. The problem is apparently known as the scipy guide > says Note that the calculation of chi-square and > > > reduced chi-square assume that the returned residual function is > scaled properly to the uncertainties in the data. > > > For these statistics to be meaningful, the person writing the function > to be minimized must scale them properly. > > > > > > I do not know how to do this. Any advice will be appreciated. > > > > > > With kind regards > > > Rudolf Baer > > > Zurich > > > > > > > > > > > > The input data for the code are x: lambda in AA, y: flux density in > erg/cm^{2}/s/{\AA) > > > Code: > > > > > > x=x_AA/10000. #<---------conversion AA to um > > > > > > y=y*1e4 > > > > > > y=y*x > > > > #========================================================================================= > > > > > > def bb(x, T, const): > > > > > > from scipy.constants import h,k,c > > > > > > x = 1e-6 * x # convert to metres from um > > > > > > return const*2*h*c**2 / (x**5 * (np.exp(h*c / (x*k*T)) - 1)) > #J/s/m2/m > > > > > > > > > def powerlaw(x,A,p): > > > > > > return A*x**p > > > > > > > > > mod= Model(bb) + Model(powerlaw) > > > > > > pars = mod.make_params(T=1100,const=2*1e-21,A=2*1e-11,p=-1.0) > > > > > > result = mod.fit(y,pars,x=x) > > > _______________________________________________ > > > AstroPy mailing list > > > AstroPy at python.org > > > https://mail.python.org/mailman/listinfo/astropy > > > > ============================================================= > > Peter Erwin Max-Planck-Insitute for Extraterrestrial > > erwin at mpe.mpg.de Physics, Giessenbachstrasse > > tel. +49 (0)176 2481 7713 85748 Garching, Germany > > fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin > > > > > > > > > > _______________________________________________ > > AstroPy mailing list > > AstroPy at python.org > > https://mail.python.org/mailman/listinfo/astropy > > 0407.csv>_______________________________________________ > > AstroPy mailing list > > AstroPy at python.org > > https://mail.python.org/mailman/listinfo/astropy > > ============================================================= > Peter Erwin Max-Planck-Insitute for Extraterrestrial > erwin at mpe.mpg.de Physics, Giessenbachstrasse > tel. +49 (0)176 2481 7713 85748 Garching, Germany > fax +49 (0)89 30000 3495 https://www.mpe.mpg.de/~erwin > > > > > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivvv68 at gmail.com Fri May 21 06:56:15 2021 From: ivvv68 at gmail.com (Ivan Valtchanov) Date: Fri, 21 May 2021 12:56:15 +0200 Subject: [AstroPy] Reading HTM table with astropy.table Message-ID: Dear all, I have a strange problem reading an HTML table with astropy.table. Some columns contain hyperlinks and even though I supply the necessary htmldict it still cannot read those properly. With the same htmldict I managed to write an HTML table keeping the columns with HTML tags. Here is a short code I did to illustrate the issue: ############# # Read HTML table with HTML tags # import io import bleach from astropy.table import Table, join html_table = """
col1col2col3
ABimage
""" # this one works as expected bleach.clean(html_table,tags=['img'],attributes=['src','alt','width']) html_dict = {"raw_html_cols":["col3"], "raw_html_clean_kwargs": {'tags': ['img'], 'attributes': ['src','alt','width']}} t = Table.read(io.BytesIO(bytes(html_table,encoding='utf-8')),format='ascii.html',htmldict=html_dict) # # this doesn't print (t) >>> print (t) col1 col2 col3 ---- ---- ---- A B -- ############### As you can see, col3 should be an but it's '--'. Using bleach.clean with whitelisted tags and attributes properly keeps the . Any advice on this? Of course I found a workaround (replace References: Message-ID: Hi Ivan, The current implementation is taking the "text" attribute of the tag. For your example the text in the tag is actually empty. So the short answer is that there is no built-in option that will make the reader do what you would expect. If you are familiar with beautifulsoup4 parsing, here is the code: data_elements = soup.find_all('td') if data_elements: yield [el.text.strip() for el in data_elements] One point that we should make more clear in the docs (sorry) is that "raw_html_.." options apply only to writing, and are ignored when reading. I could imagine a new option to allow taking whatever string is contained in the tag instead of just the text. If you agree then please go ahead and open an issue on GitHub. Cheers, Tom On Fri, May 21, 2021 at 6:56 AM Ivan Valtchanov wrote: > Dear all, > > I have a strange problem reading an HTML table with astropy.table. > Some columns contain hyperlinks and even though I supply the necessary > htmldict it still cannot read those properly. With the same htmldict I > managed to write an HTML table keeping the columns with HTML tags. > > Here is a short code I did to illustrate the issue: > > ############# > # Read HTML table with HTML tags > # > import io > import bleach > from astropy.table import Table, join > > html_table = """ > > > > > > > > > > > >
col1col2col3
ABimage width="300">
> > > """ > > # this one works as expected > bleach.clean(html_table,tags=['img'],attributes=['src','alt','width']) > > html_dict = {"raw_html_cols":["col3"], "raw_html_clean_kwargs": > {'tags': ['img'], 'attributes': ['src','alt','width']}} > > t = > Table.read(io.BytesIO(bytes(html_table,encoding='utf-8')),format='ascii.html',htmldict=html_dict) > # > # this doesn't > print (t) > >>> print (t) > col1 col2 col3 > ---- ---- ---- > A B -- > ############### > > As you can see, col3 should be an but it's '--'. > > Using bleach.clean with whitelisted tags and attributes properly keeps > the . > > Any advice on this? > > Of course I found a workaround (replace it, but it seems to me I've done all as explained in the docs. > > If it's a problem then I can raise a github issue, but I want to make > sure that I'm not missing something here. > > Cheers, > Ivan V > _______________________________________________ > AstroPy mailing list > AstroPy at python.org > https://mail.python.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: