From kevin at bud.ca Thu Oct 1 12:17:40 2009 From: kevin at bud.ca (Kevin Teague) Date: Thu, 1 Oct 2009 03:17:40 -0700 Subject: [stdlib-sig] standard metadata for the standard library (Was: Re: [Python-Dev] sharing stdlib across python implementations) In-Reply-To: <4AC36B2A.6020406@simplistix.co.uk> References: <4AB8D9F2.6050807@simplistix.co.uk> <4AB8E136.8040706@simplistix.co.uk> <5c6f2a5d0909220812u2319453axe72053bb5f736c81@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD04915F11@TK5EX14MBXC120.redmond.corp.microsoft.com> <19129.39295.970536.916767@montanaro.dyndns.org> <4dab5f760909230604j2d5cc2cam9bf4fdf492f57e31@mail.gmail.com> <4AC36B2A.6020406@simplistix.co.uk> Message-ID: <845298A4-C067-4CB7-93FD-9C0A1FD1D37F@bud.ca> On Sep 30, 2009, at 7:28 AM, Chris Withers wrote: > Frank Wierzbicki wrote: >> Talk has started up again on the stdlib-sig list about finding a core >> stdlib + tests that can be shared by all implementations, potentially >> living apart from CPython. I have volunteered to put together a PEP >> on the subject, with Jessie Noller and Brett Canon are helping me >> out. >> When I have something worth showing, I'll start the real PEP process. > > I'm on on stdlib-sig and I'm afraid I don't have the bandwidth to > start on it, but I'd just like to throw in (yet again) that it would > be great if the stdlib was actually a set of separate python > packages with their own version metadata so that packaging tools > could manage them, and upgrade them independently of python packages > when there are bug fixes. Amen! Currently there are a number of penalties for package-savy developer to use packages in the standard library, since they can't use their normal tool chains to work with the standard library. Instead it has to be treated as special cases. Aside from the annoying if-else statements used to build-up install_requires fields, a few other problems the lack of metadata for the standard library poses: * Install tools work differently with 3rd party packages that have been added to the standard library. For example, simplejson and easy_install. easy_install is not supposed to upgrade a distribution if it's already installed unless the -U switch is supplied. However, do an "easy_install simplejson" (no -U switch) with Python 2.6 and the distribution is unexpectedly upgraded. This is because the metadata has been tossed out once the distribution was incorporated to the standard lib. * Bug fixes are harder. If I'm working on a project which depends upon another project, and I find a bug in that dependant project, then the preferred route to solve that problem is contact the dependant project's author(s) and see if they'll provide a fix and do a new release. Then I just update the project so that its install_requries field specifies the minimum bug-free version. If it's in the standard library though, I file a bug report, but then instead of asking for a release for the package in question, I instead have to put a work- around into the project, even if the bug has been fixed, since there is no way to specify that I just need a fix for one particular package and that work-around needs to stay in-place in the project I was working on until the minimum required version of Python for that project is equal to the Python release which provides the fix. Bleh! * What metadata that does exist about the standard library is buried in non-standard formats and isn't programmatically accessible. The maintainers field is stored in Misc/maintainers.rst, author and version is stored as module attributes (__author__ and __version__). Ideally this metadata could be collected into setup.cfg files, and when installed would live in PEP 376 .egg-info directories, and you would replace __version__ attributes with something such as : import distutils __version__ = distutils.get_distribution('packagename').version > The big changes I can see from here would be moving the tests to the > packages from the central tests directory, and adding a setup.py > file or some other form of metadata providion for each package. Not > that big now that I've written it ;-) > Yeah, this is what I was thinking. It doesn't sound big, until you count up the number of packages in the standard library ... there's more distributions in there than Zope 3! :P However, if you are relying on Distutils to write-out the metadata, you run into a bootstrapping issue, where you need to use the Python interpreter you're installing to install the standard library, but the installation requires the standard library. Maybe there are some clever ways to solve this, by fiddling with PATHs and installing Distutils first or something ... But perhaps another way to solve the problem is to not use Distutils for installation of the working set of distributions that ships with a given release of a Python interpreter. You only need to ensure that the end-result is the same, and comply with the .egg-info metadata format. It really doesn't matter if a package is installed with Distutils or not. If the metadata consumed by setup.py files is in setup.cfg files (or perhaps some kind of .egg-info templated format that the standard lib setup.py files read), then those files could be munged by some shell commands, and written out as part of the makefile during "make install". (the only tricky bits in the new .egg-info format is computing the full-path to all installed files and computing the MD5 hash). Speaking of which, there is one .egg-info file in the standard library in the old-style format ... if PEP 376 is accepted then this line in CPython's Makefile will become a bug @for i in $(srcdir)/Lib/*.py $(srcdir)/Lib/*.doc $(srcdir)/Lib/*.egg- info ; \ Although wsgiref is the only project in the standard library with metadata, so it'd be easy enough to fix this by just removing it's metadata. But if the only package with standard metadata in the standard library had it's metadata removed, it would make me sad :( From vinay_sajip at yahoo.co.uk Mon Oct 12 00:04:14 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 11 Oct 2009 22:04:14 +0000 (UTC) Subject: [stdlib-sig] logging References: <4AB0D7D6.7030204@active-4.com> <4222a8490909160627v4da76637k39373fb1a2a6a788@mail.gmail.com> <4AB2B8E5.1070500@active-4.com> <1253263246.5710.6.camel@localhost> <4222a8490909180657i1abaddb7o1545e18d0a159d96@mail.gmail.com> <79990c6b0909180821y688af122qed5eaee18efc368c@mail.gmail.com> <878wfz4sxp.fsf@benfinney.id.au> Message-ID: Ben Finney writes: > I think you mean ?python_pep8_style_underscored_names? > > How would these names be provided? As simple ?fooName = foo_name?? Or > would the PEP-8 one be preferred, deprecating the non-conformant name? I don't see any need for deprecation, as it's a personal preference rather than anything else. And yes, just a 'foo_name = fooName' is what I was thinking of. Now, this has come up in the past and Guido has said he doesn't think it's really worth doing. Personally, it only makes sense to me if this is going to be done for the whole of the stdlib which isn't PEP-8 conformant. And you have to question whether the time to be spent on this couldn't be spent on some *real* improvements ;-) Regards, Vinay Sajip From brett at python.org Mon Oct 12 01:30:04 2009 From: brett at python.org (Brett Cannon) Date: Sun, 11 Oct 2009 16:30:04 -0700 Subject: [stdlib-sig] logging In-Reply-To: References: <4AB0D7D6.7030204@active-4.com> <4AB2B8E5.1070500@active-4.com> <1253263246.5710.6.camel@localhost> <4222a8490909180657i1abaddb7o1545e18d0a159d96@mail.gmail.com> <79990c6b0909180821y688af122qed5eaee18efc368c@mail.gmail.com> <878wfz4sxp.fsf@benfinney.id.au> Message-ID: On Sun, Oct 11, 2009 at 15:04, Vinay Sajip wrote: > Ben Finney writes: > > > I think you mean ?python_pep8_style_underscored_names? > > > > How would these names be provided? As simple ?fooName = foo_name?? Or > > would the PEP-8 one be preferred, deprecating the non-conformant name? > > I don't see any need for deprecation, as it's a personal preference rather > than > anything else. And yes, just a 'foo_name = fooName' is what I was thinking > of. > > I am not going to start up a discussion on moving logging or unittest over to PEP 8 standards, but I want to make clear it is not a personal preference thing but coding standards thing. Both the logging and unittest get a pass on not meeting our standard for historical reasons and that's it; nothing to do with personal preference. > Now, this has come up in the past and Guido has said he doesn't think it's > really worth doing. Personally, it only makes sense to me if this is going > to be > done for the whole of the stdlib which isn't PEP-8 conformant. And you have > to > question whether the time to be spent on this couldn't be spent on some > *real* > improvements ;-) > I agree. This would be more of a Py4k thing then bothering with it in Python 3 now. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Oct 12 07:49:21 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 12 Oct 2009 05:49:21 +0000 (UTC) Subject: [stdlib-sig] logging References: <4AB0D7D6.7030204@active-4.com> <4AB2B8E5.1070500@active-4.com> <1253263246.5710.6.camel@localhost> <4222a8490909180657i1abaddb7o1545e18d0a159d96@mail.gmail.com> <79990c6b0909180821y688af122qed5eaee18efc368c@mail.gmail.com> <878wfz4sxp.fsf@benfinney.id.au> Message-ID: Brett Cannon writes: > I am not going to start up a discussion on moving logging or unittest over > to PEP 8 standards, but I want to make clear it is not a personal preference > thing but coding standards thing. Both the logging and unittest get a pass > on not meeting our standard for historical reasons and that's it; nothing to > do with personal preference. Sorry, Brett, you're right. I wasn't trying to skirt around the importance of having a common coding style. When I said "personal preference", I was referring to the fact that some people get somewhat exercised about the issue, to the extent that the word "hate" comes up in the discussion. Typically it's not because they're PEP 8 zealots; more commonly, they're used to it from their Unix/C experience and find the Pascal, Java,and C# conventions obnoxious. I'm not personally tied to any convention, I normally work with the convention for the project. Logging's divergence from PEP 8 is, as you say, historical, and when the Py4K time comes around, I'll be rolling up my sleeves and getting on with the conversion :-) Regards, Vinay Sajip From jnoller at gmail.com Mon Oct 12 18:58:30 2009 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 12 Oct 2009 12:58:30 -0400 Subject: [stdlib-sig] logging In-Reply-To: References: <4AB0D7D6.7030204@active-4.com> <1253263246.5710.6.camel@localhost> <4222a8490909180657i1abaddb7o1545e18d0a159d96@mail.gmail.com> <79990c6b0909180821y688af122qed5eaee18efc368c@mail.gmail.com> <878wfz4sxp.fsf@benfinney.id.au> Message-ID: <4222a8490910120958t3f62646di979faee8543350a0@mail.gmail.com> > I'm not personally tied to any convention, I normally work with the convention > for the project. Logging's divergence from PEP 8 is, as you say, historical, > and when the Py4K time comes around, I'll be rolling up my sleeves and getting > on with the conversion :-) > > Regards, > > Vinay Sajip In all seriousness, why not do something like what we started with the threading module? We added pep8 aliases and started marking the old API deprecated. They'll follow the common deprecation guidelines we already have. That, plus the dict config option you mentioned plus a doc re-org would probably go a long way to improving the PR of the module. jesse From vinay_sajip at yahoo.co.uk Mon Oct 12 19:20:49 2009 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 12 Oct 2009 17:20:49 +0000 (UTC) Subject: [stdlib-sig] logging References: <4AB0D7D6.7030204@active-4.com> <1253263246.5710.6.camel@localhost> <4222a8490909180657i1abaddb7o1545e18d0a159d96@mail.gmail.com> <79990c6b0909180821y688af122qed5eaee18efc368c@mail.gmail.com> <878wfz4sxp.fsf@benfinney.id.au> <4222a8490910120958t3f62646di979faee8543350a0@mail.gmail.com> Message-ID: Jesse Noller writes: > In all seriousness, why not do something like what we started with the > threading module? We added pep8 aliases and started marking the old > API deprecated. They'll follow the common deprecation guidelines we > already have. > > That, plus the dict config option you mentioned plus a doc re-org > would probably go a long way to improving the PR of the module. It's just a question of priorities and timing. Clearly, the dict config option, other functional enhancements and documentation reorganisation should take priority. I'm waiting for a little while to see what feedback comes in from python-dev about the dict config functionality. I've been in touch with Doug Hellmann on the documentation reorganisation and am waiting for some feedback from him, and I've started to put together some diagrams which will hopefully help clarify some aspects of the documentation. Regards, Vinay Sajip From doug.hellmann at gmail.com Mon Oct 12 19:32:20 2009 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 12 Oct 2009 13:32:20 -0400 Subject: [stdlib-sig] logging In-Reply-To: References: <4AB0D7D6.7030204@active-4.com> <1253263246.5710.6.camel@localhost> <4222a8490909180657i1abaddb7o1545e18d0a159d96@mail.gmail.com> <79990c6b0909180821y688af122qed5eaee18efc368c@mail.gmail.com> <878wfz4sxp.fsf@benfinney.id.au> <4222a8490910120958t3f62646di979faee8543350a0@mail.gmail.com> Message-ID: On Oct 12, 2009, at 1:20 PM, Vinay Sajip wrote: > Jesse Noller writes: > >> In all seriousness, why not do something like what we started with >> the >> threading module? We added pep8 aliases and started marking the old >> API deprecated. They'll follow the common deprecation guidelines we >> already have. >> >> That, plus the dict config option you mentioned plus a doc re-org >> would probably go a long way to improving the PR of the module. > > It's just a question of priorities and timing. Clearly, the dict > config > option, other functional enhancements and documentation reorganisation > should take priority. > > I'm waiting for a little while to see what feedback comes in from > python-dev > about the dict config functionality. I've been in touch with Doug > Hellmann on > the documentation reorganisation and am waiting for some feedback > from him, and > I've started to put together some diagrams which will hopefully help > clarify > some aspects of the documentation. Jesse keeps bugging me to review PyCon proposals ;-), so I haven't had a chance to dig into the doc work, yet. It's still on my list, though. Doug From ubershmekel at gmail.com Sun Oct 18 20:37:30 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Sun, 18 Oct 2009 20:37:30 +0200 Subject: [stdlib-sig] ctypes or struct from an h file Message-ID: <9d153b7c0910181137u40cb63b5k8a758da33e445a67@mail.gmail.com> Is there a way that Python and C can have a shared definition for a binary data structure? It could be nice if: 1. struct or ctypes had a function that could parse a .h/.c/.cpp file to auto-generate constructors or 2. a ctypes definition could be exported to a .h file. So my question is - is there a way to do this in the std-lib or even pypi? --yuv ps If this doesn't exist, then I'm probably going to open a project and would like some tips/ideas. From benjamin at python.org Sun Oct 18 20:42:34 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 Oct 2009 13:42:34 -0500 Subject: [stdlib-sig] ctypes or struct from an h file In-Reply-To: <9d153b7c0910181137u40cb63b5k8a758da33e445a67@mail.gmail.com> References: <9d153b7c0910181137u40cb63b5k8a758da33e445a67@mail.gmail.com> Message-ID: <1afaf6160910181142k9ae19dx80cff5e09a5991a0@mail.gmail.com> 2009/10/18 Yuvgoog Greenle : > Is there a way that Python and C can have a shared definition for a > binary data structure? > > It could be nice if: > 1. struct or ctypes had a function that could parse a .h/.c/.cpp file > to auto-generate constructors > or > 2. a ctypes definition could be exported to a .h file. > > So my question is - is there a way to do this in the std-lib or even pypi? No, not in the stldlib. There is a tool which read .h files and converted them to ctypes code, but I'm not sure of its status. BTW, please ask questions like this on python-list next time. -- Regards, Benjamin From ubershmekel at gmail.com Sun Oct 18 20:43:41 2009 From: ubershmekel at gmail.com (Yuvgoog Greenle) Date: Sun, 18 Oct 2009 20:43:41 +0200 Subject: [stdlib-sig] ctypes or struct from an h file In-Reply-To: <1afaf6160910181142k9ae19dx80cff5e09a5991a0@mail.gmail.com> References: <9d153b7c0910181137u40cb63b5k8a758da33e445a67@mail.gmail.com> <1afaf6160910181142k9ae19dx80cff5e09a5991a0@mail.gmail.com> Message-ID: <9d153b7c0910181143j4c7a9720k3bf5722a48dbbbe7@mail.gmail.com> Please nobody reply to this thread, I'm forwarding to python-list. On Sun, Oct 18, 2009 at 8:42 PM, Benjamin Peterson wrote: > 2009/10/18 Yuvgoog Greenle : >> Is there a way that Python and C can have a shared definition for a >> binary data structure? >> >> It could be nice if: >> 1. struct or ctypes had a function that could parse a .h/.c/.cpp file >> to auto-generate constructors >> or >> 2. a ctypes definition could be exported to a .h file. >> >> So my question is - is there a way to do this in the std-lib or even pypi? > > No, not in the stldlib. There is a tool which read .h files and > converted them to ctypes code, but I'm not sure of its status. > > BTW, please ask questions like this on python-list next time. > > > > -- > Regards, > Benjamin >