From pje at telecommunity.com Wed Jan 1 01:32:47 2014 From: pje at telecommunity.com (PJ Eby) Date: Tue, 31 Dec 2013 19:32:47 -0500 Subject: [Distutils] Major update to the metadata 2.0 draft PEPs In-Reply-To: References: Message-ID: On Sat, Dec 21, 2013 at 9:46 AM, Nick Coghlan wrote: > I've just published the first draft of the metadata 2.0 spec that > moves all of the fields that aren't part of the core metadata or > potentially needed for dependency resolution out to a separate > "standard metadata extensions" PEP. Yay! > I think this makes PEP 426 itself substantially less overwhelming, and > also provides convenient names to refer to the various other subsets > of the metadata. Indeed! > Metadata 2.0: http://www.python.org/dev/peps/pep-0426/ > Versioning: http://www.python.org/dev/peps/pep-0440/ > Standard Extensions: http://www.python.org/dev/peps/pep-0459/ I have only been skimming these so far, will comment more later, but I just want to mention that for standard extensions, what is the rationale for not defining e.g. commands as exports? Or for that matter, defining hooks as exports? Both commands and hooks have a payload that's the same as an export payload, i.e., a reference to an importable object. I think it'd be good for them to be defined in terms of exports in order to reduce duplication of concepts and implementations, as well as providing a PEP-defined example of how to use export groups and export names effectively. (Also, noting the mapping of current script export namespaces to the new metadata would be helpful for people implementing translation tools from setuptools metadata.) Last but not least, some of the export examples should use dotted post-module names (e.g. "some.module:RandomClass.a_class_method") to highlight that qualified names can be used there. (Yes, the spec *says* qualified names, but it's the sort of thing that people overlook when the examples are all of unqualified simple names.) From p.f.moore at gmail.com Wed Jan 1 11:58:09 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jan 2014 10:58:09 +0000 Subject: [Distutils] migration path for entry_points? In-Reply-To: References: Message-ID: On 31 December 2013 22:53, Marcus Smith wrote: > for #1, during installation, I imagine the new setuptools would build the > sdist (and any entry_point declarations) using 2.0 metadata. So assuming the package consuming the entry point converts to using metadata 2.0 APIs (and does *not* try to have a compatibility mode that falls back to the old setuptools format) users would be required to use an up to date setuptools (installation with older setuptools would be unsupported). Also they would need to reinstall SomeExtension for it to be recognised by the new consumer. > for #2, the pip installer would be responsible for converting > "entry_points.txt" during the install (or maybe just forcing a "bdist_wheel" > rebuild when it detects metadata < 2.0) Again requiring users to be running a suitably recent pip and/or setuptools and reinstall SomeExtension. Presumably the new setuptools and pip would have to maintain both the Metadata 2.0 format data and the old-style entry_points.txt format for a period of compatibility. We should probably define the length of that transition period as part of the plan for Metadata 2.0 going live. So there is some user impact, but it sounds like it's manageable. Cool. Paul From ncoghlan at gmail.com Wed Jan 1 15:27:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Jan 2014 00:27:24 +1000 Subject: [Distutils] migration path for entry_points? In-Reply-To: References: Message-ID: On 1 Jan 2014 20:58, "Paul Moore" wrote: > > On 31 December 2013 22:53, Marcus Smith wrote: > > for #1, during installation, I imagine the new setuptools would build the > > sdist (and any entry_point declarations) using 2.0 metadata. > > So assuming the package consuming the entry point converts to using > metadata 2.0 APIs (and does *not* try to have a compatibility mode > that falls back to the old setuptools format) users would be required > to use an up to date setuptools (installation with older setuptools > would be unsupported). Also they would need to reinstall SomeExtension > for it to be recognised by the new consumer. > > > for #2, the pip installer would be responsible for converting > > "entry_points.txt" during the install (or maybe just forcing a "bdist_wheel" > > rebuild when it detects metadata < 2.0) > > Again requiring users to be running a suitably recent pip and/or > setuptools and reinstall SomeExtension. > > Presumably the new setuptools and pip would have to maintain both the > Metadata 2.0 format data and the old-style entry_points.txt format for > a period of compatibility. We should probably define the length of > that transition period as part of the plan for Metadata 2.0 going > live. > > So there is some user impact, but it sounds like it's manageable. Cool. Yeah, one of the big benefits of switching to the JSON format with a different filename was letting us distribute both the old and new metadata formats in parallel. The aim is to allow newer versions of pip and setuptools to be faster and more reliable, without breaking older versions. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 1 15:39:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Jan 2014 00:39:57 +1000 Subject: [Distutils] Major update to the metadata 2.0 draft PEPs In-Reply-To: References: Message-ID: On 1 Jan 2014 10:33, "PJ Eby" wrote: > > On Sat, Dec 21, 2013 at 9:46 AM, Nick Coghlan wrote: > > I've just published the first draft of the metadata 2.0 spec that > > moves all of the fields that aren't part of the core metadata or > > potentially needed for dependency resolution out to a separate > > "standard metadata extensions" PEP. > > Yay! > > > > I think this makes PEP 426 itself substantially less overwhelming, and > > also provides convenient names to refer to the various other subsets > > of the metadata. > > Indeed! > > > > Metadata 2.0: http://www.python.org/dev/peps/pep-0426/ > > Versioning: http://www.python.org/dev/peps/pep-0440/ > > Standard Extensions: http://www.python.org/dev/peps/pep-0459/ > > I have only been skimming these so far, will comment more later, but I > just want to mention that for standard extensions, what is the > rationale for not defining e.g. commands as exports? Or for that > matter, defining hooks as exports? Both commands and hooks have a > payload that's the same as an export payload, i.e., a reference to an > importable object. I think it'd be good for them to be defined in > terms of exports in order to reduce duplication of concepts and > implementations, as well as providing a PEP-defined example of how to > use export groups and export names effectively. I believe it was due to the extra layer of nesting they needed - using multiple parallel export groups with different final elements in their dotted names didn't feel right. I guess that indicates a flaw in my initial design for the export definitions though - I agree it would be better if commands and hooks could be cleanly defined within the exports mechanism, rather than needing separate custom metadata extensions. I'll take another look at using either parallel export groups, or else allowing subdivision of export groups as a general feature. > (Also, noting the > mapping of current script export namespaces to the new metadata would > be helpful for people implementing translation tools from setuptools > metadata.) > > Last but not least, some of the export examples should use dotted > post-module names (e.g. "some.module:RandomClass.a_class_method") to > highlight that qualified names can be used there. (Yes, the spec > *says* qualified names, but it's the sort of thing that people > overlook when the examples are all of unqualified simple names.) Yeah, those are both good ideas. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jan 1 16:21:01 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 1 Jan 2014 15:21:01 +0000 (GMT) Subject: [Distutils] Major update to the metadata 2.0 draft PEPs In-Reply-To: Message-ID: <1388589661.93266.YahooMailBasic@web171304.mail.ir2.yahoo.com> On Wed, 1/1/14, Nick Coghlan wrote: > I guess that indicates a flaw in my initial design for the export definitions though - I agree it would > be better if commands and hooks could be cleanly defined within the exports mechanism, rather > than needing separate custom metadata extensions. +1. Regards, Vinay Sajip From pje at telecommunity.com Wed Jan 1 18:58:45 2014 From: pje at telecommunity.com (PJ Eby) Date: Wed, 1 Jan 2014 12:58:45 -0500 Subject: [Distutils] Major update to the metadata 2.0 draft PEPs In-Reply-To: References: Message-ID: On Wed, Jan 1, 2014 at 9:39 AM, Nick Coghlan wrote: > On 1 Jan 2014 10:33, "PJ Eby" wrote: >> I have only been skimming these so far, will comment more later, but I >> just want to mention that for standard extensions, what is the >> rationale for not defining e.g. commands as exports? Or for that >> matter, defining hooks as exports? Both commands and hooks have a >> payload that's the same as an export payload, i.e., a reference to an >> importable object. I think it'd be good for them to be defined in >> terms of exports in order to reduce duplication of concepts and >> implementations, as well as providing a PEP-defined example of how to >> use export groups and export names effectively. > > I believe it was due to the extra layer of nesting they needed - using > multiple parallel export groups with different final elements in their > dotted names didn't feel right. > > I guess that indicates a flaw in my initial design for the export > definitions though - I agree it would be better if commands and hooks could > be cleanly defined within the exports mechanism, rather than needing > separate custom metadata extensions. If it's a flaw, I'd say it's in my original design of entry points. ;-) Basically, I wanted a way to do -- without XML -- what Eclipse does with its "extensions" and "extension points" machinery. I went with a "flat (with dots) is better than nested" concept. To me, though, this doesn't look terribly complicated (using entry points syntax): [python.exports.after_install] ComfyChair.plugins = ComfyChair.plugins:install_hook [python.exports.after_uninstall] ComfyChair.plugins = ComfyChair.plugins:install_hook Nor this: [python.extensions.after_install] python.exports = pip.export_group_hooks:run_install_hooks python.commands = pip.command_hook:install_wrapper_scripts [python.extensions.after_uninstall] python.exports = pip.export_group_hooks:run_uninstall_hooks (Also, adding hooks to *validate* extensions and exports at build and/or install time might be handy.) Finally, note that if the typical usecase is to define *both* an install and uninstall hook, then it might be simpler to just define the hook once, as an object with 'after_install' and 'after_uninstall' methods. This avoids the need to register a hook in two groups, and in the simplest case people can just make them both module-level functions and list the module as the export. From ncoghlan at gmail.com Thu Jan 2 00:40:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Jan 2014 09:40:01 +1000 Subject: [Distutils] Major update to the metadata 2.0 draft PEPs In-Reply-To: References: Message-ID: On 2 Jan 2014 03:59, "PJ Eby" wrote: > > On Wed, Jan 1, 2014 at 9:39 AM, Nick Coghlan wrote: > > On 1 Jan 2014 10:33, "PJ Eby" wrote: > >> I have only been skimming these so far, will comment more later, but I > >> just want to mention that for standard extensions, what is the > >> rationale for not defining e.g. commands as exports? Or for that > >> matter, defining hooks as exports? Both commands and hooks have a > >> payload that's the same as an export payload, i.e., a reference to an > >> importable object. I think it'd be good for them to be defined in > >> terms of exports in order to reduce duplication of concepts and > >> implementations, as well as providing a PEP-defined example of how to > >> use export groups and export names effectively. > > > > I believe it was due to the extra layer of nesting they needed - using > > multiple parallel export groups with different final elements in their > > dotted names didn't feel right. > > > > I guess that indicates a flaw in my initial design for the export > > definitions though - I agree it would be better if commands and hooks could > > be cleanly defined within the exports mechanism, rather than needing > > separate custom metadata extensions. > > If it's a flaw, I'd say it's in my original design of entry points. > ;-) Basically, I wanted a way to do -- without XML -- what Eclipse > does with its "extensions" and "extension points" machinery. I went > with a "flat (with dots) is better than nested" concept. > > To me, though, this doesn't look terribly complicated (using entry > points syntax): > > [python.exports.after_install] > ComfyChair.plugins = ComfyChair.plugins:install_hook > > [python.exports.after_uninstall] > ComfyChair.plugins = ComfyChair.plugins:install_hook > > Nor this: > > [python.extensions.after_install] > python.exports = pip.export_group_hooks:run_install_hooks > python.commands = pip.command_hook:install_wrapper_scripts > > [python.extensions.after_uninstall] > python.exports = pip.export_group_hooks:run_uninstall_hooks > > (Also, adding hooks to *validate* extensions and exports at build > and/or install time might be handy.) > > Finally, note that if the typical usecase is to define *both* an > install and uninstall hook, then it might be simpler to just define > the hook once, as an object with 'after_install' and 'after_uninstall' > methods. This avoids the need to register a hook in two groups, and > in the simplest case people can just make them both module-level > functions and list the module as the export. Ah, true - people could provide them as either functions or class methods, and if we add a new one, only the list of permitted names (and their signatures) needs to change. And splitting commands into python.commands and python.commands.gui export groups would cover the export compatible cases. Since identifying "prebuilt commands" in the metadata is a new feature without any particularly compelling justification at this point, I'll likely just leave it out for 2.0. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 2 14:30:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Jan 2014 23:30:29 +1000 Subject: [Distutils] PyPI classifier for standard library modules/backports? Message-ID: I started putting together a list of standard library modules that are also available on PyPI for the https://wiki.python.org/moin/Python2orPython3 wiki page. Dariusz Suchojad suggested it might be easier if there was a suitable classifier to mark such modules rather than trying to keep a list on the wiki up to date by hand. Perhaps something like "Progamming Language :: Python :: Standard Library"? Thoughts? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Thu Jan 2 15:37:55 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 2 Jan 2014 14:37:55 +0000 (GMT) Subject: [Distutils] PyPI classifier for standard library modules/backports? In-Reply-To: Message-ID: <1388673475.13265.YahooMailBasic@web171301.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 2/1/14, Nick Coghlan wrote: > Dariusz Suchojad suggested it might be easier if there was a > suitable classifier to mark such modules rather than trying to keep a > list on the wiki up to date by hand. Perhaps something like > "Progamming Language :: Python :: Standard Library"? I think it makes sense to have "Progamming Language :: Python :: Backport" and "Progamming Language :: Python :: Backport :: Standard Library" While we're on the subject of classifiers, IMO it would also be helpful to be able to identify (the growing number of) projects which are 2 and 3 compatible without the need for 2to3 to be run. So perhaps "Progamming Language :: Python :: Python2and3" or some such. Regards, Vinay Sajip From brett at python.org Thu Jan 2 16:07:15 2014 From: brett at python.org (Brett Cannon) Date: Thu, 2 Jan 2014 10:07:15 -0500 Subject: [Distutils] PyPI classifier for standard library modules/backports? In-Reply-To: <1388673475.13265.YahooMailBasic@web171301.mail.ir2.yahoo.com> References: <1388673475.13265.YahooMailBasic@web171301.mail.ir2.yahoo.com> Message-ID: On Thu, Jan 2, 2014 at 9:37 AM, Vinay Sajip wrote: > > -------------------------------------------- > On Thu, 2/1/14, Nick Coghlan wrote: > > > Dariusz Suchojad suggested it might be easier if there was a > > suitable classifier to mark such modules rather than trying to keep a > > list on the wiki up to date by hand. Perhaps something like > > "Progamming Language :: Python :: Standard Library"? > > I think it makes sense to have > > "Progamming Language :: Python :: Backport" > > and > > "Progamming Language :: Python :: Backport :: Standard Library" > > While we're on the subject of classifiers, IMO it would also be helpful to > be able to identify (the growing number of) projects which are 2 and 3 > compatible without the need for 2to3 to be run. So perhaps > > "Progamming Language :: Python :: Python2and3" > > or some such. > How is that any better than specifying the Python 2 classifier and the Python 3 one? One should specify every individual version of Python that a project is supposed to be compatible with anyway so I don't see the benefit of knowing through a classifier if 2to3 is needed or not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Thu Jan 2 16:58:01 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 02 Jan 2014 15:58:01 +0000 Subject: [Distutils] Can pypi python 3 packages be sorted please? Message-ID: So can they be sorted in the same way as the normal index? To my eye the unsorted list looks horrible. OTOH it's too much hassle or this is covered by the Warehouse project I won't lose any sleep over it. Also for pypi issues https://mail.python.org/mailman/listinfo/pydotorg-www points you to https://mail.python.org/mailman/listinfo/catalog-sig where it states it's retired and points you here. How do we get this changed? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From vinay_sajip at yahoo.co.uk Thu Jan 2 17:12:55 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 2 Jan 2014 16:12:55 +0000 (GMT) Subject: [Distutils] PyPI classifier for standard library modules/backports? In-Reply-To: Message-ID: <1388679175.11012.YahooMailBasic@web171303.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 2/1/14, Brett Cannon wrote: >> "Progamming Language :: Python :: Python2and3" >> or some such. > How is that any better than specifying the Python > 2 classifier and the Python 3 one? One should specify every > individual version of Python that a project is supposed to > be compatible with anyway so I don't see the benefit of > knowing through a classifier if 2to3 is needed or > not. At the moment, if a classifier indicates that a distribution is both 2 and 3 compatible, there is no declarative way to indicate whether or not 2to3 is to be run - only via a setuptools.setup argument. I'm not hung up on whether a classifier is used, but I think there should be *some* declarative mechanism. A classifier has the advantage that it could be applied to a specific release in the index without changing its code. Regards, Vinay Sajip From brett at python.org Thu Jan 2 19:40:05 2014 From: brett at python.org (Brett Cannon) Date: Thu, 2 Jan 2014 13:40:05 -0500 Subject: [Distutils] PyPI classifier for standard library modules/backports? In-Reply-To: <1388679175.11012.YahooMailBasic@web171303.mail.ir2.yahoo.com> References: <1388679175.11012.YahooMailBasic@web171303.mail.ir2.yahoo.com> Message-ID: On Thu, Jan 2, 2014 at 11:12 AM, Vinay Sajip wrote: > > -------------------------------------------- > On Thu, 2/1/14, Brett Cannon wrote: > > >> "Progamming Language :: Python :: Python2and3" > >> or some such. > > > How is that any better than specifying the Python > > 2 classifier and the Python 3 one? One should specify every > > individual version of Python that a project is supposed to > > be compatible with anyway so I don't see the benefit of > > knowing through a classifier if 2to3 is needed or > > not. > > At the moment, if a classifier indicates that a distribution is both 2 and > 3 compatible, there is no declarative way to indicate whether or not 2to3 > is to be run - only via a setuptools.setup argument. I'm not hung up on > whether a classifier is used, but I think there should be *some* > declarative mechanism. A classifier has the advantage that it could be > applied to a specific release in the index without changing its code. > I guess my question then is "why do you care?" If 2to3 is run at install time then it's a cost, but it's one-time and if you really care you can always create your own wheel of the translated code or something. I guess I just don't view the overhead of some packages using 2to3 to be enough to warrant a classifier that only some people will use (it's hard enough to try and get people to use proper classifiers for what version of Python they support as it is). -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jan 2 23:20:55 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 2 Jan 2014 22:20:55 +0000 (GMT) Subject: [Distutils] PyPI classifier for standard library modules/backports? In-Reply-To: Message-ID: <1388701255.42495.YahooMailBasic@web171302.mail.ir2.yahoo.com> > I guess my question then is "why do you > care?" If 2to3 is run at install time then it's a > cost, but it's one-time and if you really care you can > always create your own wheel of the translated code or > something. Because I've written a packaging tool that doesn't use setup.py but instead uses a fully declarative format for building and installing packages, and I don't want that tool to run what can be a time-consuming step every time it installs certain packages, when that step isn't necessary. It's not a common use case, I grant you :-) Regards, Vinay Sajip From ncoghlan at gmail.com Fri Jan 3 03:13:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Jan 2014 12:13:02 +1000 Subject: [Distutils] PyPI classifier for standard library modules/backports? In-Reply-To: <1388701255.42495.YahooMailBasic@web171302.mail.ir2.yahoo.com> References: <1388701255.42495.YahooMailBasic@web171302.mail.ir2.yahoo.com> Message-ID: On 3 Jan 2014 08:20, "Vinay Sajip" wrote: > > > I guess my question then is "why do you > > care?" If 2to3 is run at install time then it's a > > cost, but it's one-time and if you really care you can > > always create your own wheel of the translated code or > > something. > > Because I've written a packaging tool that doesn't use setup.py but instead uses a fully declarative format for building and installing packages, and I don't want that tool to run what can be a time-consuming step every time it installs certain packages, when that step isn't necessary. It's not a common use case, I grant you :-) Also getting into the metabuild system design. The idea of classifiers potentially having programmatic effects is one that hadn't previously occurred to me - I've always just viewed them as advisories for human readers. However, it would only apply when installing from an sdist (not a wheel), so I don't believe the extra complexity is worth it at this point - for the moment, setup.py remains the only "defined" build system. However, we should definitely make sure that it is covered in the development of the metabuild system. Cheers, Nick. > > Regards, > > Vinay Sajip -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Fri Jan 3 17:30:34 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 03 Jan 2014 16:30:34 +0000 Subject: [Distutils] Can pypi python 3 packages be sorted please? In-Reply-To: References: Message-ID: On 02/01/2014 15:58, Mark Lawrence wrote: > So can they be sorted in the same way as the normal index? To my eye > the unsorted list looks horrible. OTOH it's too much hassle or this is > covered by the Warehouse project I won't lose any sleep over it. > > Also for pypi issues > https://mail.python.org/mailman/listinfo/pydotorg-www points you to > https://mail.python.org/mailman/listinfo/catalog-sig where it states > it's retired and points you here. How do we get this changed? > Anybody? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From richard at python.org Fri Jan 3 23:29:44 2014 From: richard at python.org (Richard Jones) Date: Sat, 4 Jan 2014 09:29:44 +1100 Subject: [Distutils] Can pypi python 3 packages be sorted please? In-Reply-To: References: Message-ID: Hi, (just FYI I live in UTC+10 so immediate responses sometimes aren't reasonable to expect), could you clarify which list of Python 3 packages you're referring to? I presume it's this one: https://pypi.python.org/pypi?:action=browse&c=533&show=all In which case I think I might be able to quickly hack an ordering in there, yes. I'm really not sure what the point of that page is any longer though :) As for the mailing list stuff: you'd need to contact the owner(s) of the pydotorg-www mailing list to have that change made. Richard On 3 January 2014 02:58, Mark Lawrence wrote: > So can they be sorted in the same way as the normal index? To my eye the > unsorted list looks horrible. OTOH it's too much hassle or this is covered > by the Warehouse project I won't lose any sleep over it. > > Also for pypi issues https://mail.python.org/mailman/listinfo/pydotorg-wwwpoints you to > https://mail.python.org/mailman/listinfo/catalog-sig where it states it's > retired and points you here. How do we get this changed? > > -- > My fellow Pythonistas, ask not what our language can do for you, ask what > you can do for our language. > > Mark Lawrence > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Sat Jan 4 04:39:35 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 04 Jan 2014 03:39:35 +0000 Subject: [Distutils] Can pypi python 3 packages be sorted please? In-Reply-To: References: Message-ID: On 03/01/2014 22:29, Richard Jones wrote: > Hi, (just FYI I live in UTC+10 so immediate responses sometimes aren't > reasonable to expect), could you clarify which list of Python 3 packages > you're referring to? I presume it's this one: Why I gave it 24 hours, and sorry I don't do UTC, I'm still on GMT :) > > https://pypi.python.org/pypi?:action=browse&c=533&show=all Correct. > > In which case I think I might be able to quickly hack an ordering in > there, yes. I'm really not sure what the point of that page is any > longer though :) Bin it then if you like, as I've said I won't lose any sleep over it. > > As for the mailing list stuff: you'd need to contact the owner(s) of the > pydotorg-www mailing list to have that change made. Noted. > > > Richard > > > On 3 January 2014 02:58, Mark Lawrence > wrote: > > So can they be sorted in the same way as the normal index? To my > eye the unsorted list looks horrible. OTOH it's too much hassle or > this is covered by the Warehouse project I won't lose any sleep over it. > > Also for pypi issues > https://mail.python.org/__mailman/listinfo/pydotorg-www > points you > to https://mail.python.org/__mailman/listinfo/catalog-sig > where it > states it's retired and points you here. How do we get this changed? > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From chris at chrullrich.net Sun Jan 5 12:31:41 2014 From: chris at chrullrich.net (Christian Ullrich) Date: Sun, 05 Jan 2014 12:31:41 +0100 Subject: [Distutils] Buildout index URL override Message-ID: Hello all, does buildout ever take its default index URL from the distutils configuration, i.e. [py]distutils.cfg? Given that it uses easy_install for most installation tasks, that could make sense, and I'm not sure whether it is actually supposed to happen already, but does not happen on my systems for some reason. Thanks for any info you can give me. -- Christian From malik.a.rumi at gmail.com Sun Jan 5 13:38:18 2014 From: malik.a.rumi at gmail.com (Malik Rumi) Date: Sun, 5 Jan 2014 06:38:18 -0600 Subject: [Distutils] errors trying to get setuptools 2.0.2 Message-ID: Here is what I did and the response: c:\> c:\>ez_setup.py Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-2.0. 2.tar.gz Exception calling "DownloadFile" with "2" argument(s): "An exception occurred during a WebClient request." At line:1 char:1 + (new-object System.Net.WebClient).DownloadFile('https://pypi.python.org/packages ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~ ~~~ + CategoryInfo : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : WebException Traceback (most recent call last): File "C:\Python27\Scripts\ez_setup.py", line 361, in sys.exit(main()) File "C:\Python27\Scripts\ez_setup.py", line 357, in main downloader_factory=options.downloader_factory) File "C:\Python27\Scripts\ez_setup.py", line 282, in download_setuptools downloader(url, saveto) File "C:\Python27\Scripts\ez_setup.py", line 169, in download_file_powershell _clean_check(cmd, target) File "C:\Python27\Scripts\ez_setup.py", line 152, in _clean_check subprocess.check_call(cmd) File "C:\Python27\lib\subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['powershell', '-Command', "(new-object System.Net.WebClient).DownloadFile ('https://pypi.python.org/packages/source/s/se tuptools/setuptools-2.0.2.tar.gz', 'c:\\\\setuptools-2.0.2.tar.gz')"]' returned non-zero exit status 1 c:\> The parts in red were in red in my terminal window. I am on windows 8.1 and I am brand new. Less than a week ago, I installed Distribute because someone on Django Google Groups recommended it to me, along with virtualenv and pip. Then today - well, last night since it was a few hours ago - I found out Distribute is deprecated. So I am trying to follow the uninstall instructions but this is what I got. Just for kicks I ran it again just now - same result. Please help. I took time off from work Thursday and Friday to get all this set up have been at this all weekend only to be stumped again. And no, I do not have Django up and running yet. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Sun Jan 5 23:50:12 2014 From: jim at zope.com (Jim Fulton) Date: Sun, 5 Jan 2014 17:50:12 -0500 Subject: [Distutils] Buildout index URL override In-Reply-To: References: Message-ID: On Sun, Jan 5, 2014 at 6:31 AM, Christian Ullrich wrote: > Hello all, > > does buildout ever take its default index URL from the distutils > configuration, i.e. [py]distutils.cfg? > No. It has it's own configuration for the index url ... which sadly isn't documented. Weird. You can control the index URL using the buildout index option. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Mon Jan 6 21:17:02 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 6 Jan 2014 15:17:02 -0500 Subject: [Distutils] errors trying to get setuptools 2.0.2 In-Reply-To: References: Message-ID: I'm not sure why your powershell is blowing up, but try running python ez_setup.py --insecure instead. On Sun, Jan 5, 2014 at 7:38 AM, Malik Rumi wrote: > Here is what I did and the response: > > c:\> > > c:\>ez_setup.py > > Downloading > > > https://pypi.python.org/packages/source/s/setuptools/setuptools-2.0. > > 2.tar.gz > > Exception calling "DownloadFile" with "2" argument(s): "An exception > > > occurred > > during a WebClient request." > > At line:1 char:1 > > + (new-object > > System.Net.WebClient).DownloadFile('https://pypi.python.org/packages > > > ... > > + > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > ~~~~~~ > > ~~~ > > + CategoryInfo : NotSpecified: (:) [], > > > MethodInvocationException > > + FullyQualifiedErrorId : WebException > > > Traceback (most recent call last): > > File "C:\Python27\Scripts\ez_setup.py", line 361, in > > sys.exit(main()) > > File "C:\Python27\Scripts\ez_setup.py", line 357, in main > > downloader_factory=options.downloader_factory) > > File "C:\Python27\Scripts\ez_setup.py", line 282, in > > > download_setuptools > > downloader(url, saveto) > > File "C:\Python27\Scripts\ez_setup.py", line 169, in > > > download_file_powershell > > _clean_check(cmd, target) > > File "C:\Python27\Scripts\ez_setup.py", line 152, in _clean_check > > subprocess.check_call(cmd) > > File "C:\Python27\lib\subprocess.py", line 511, in check_call > > raise CalledProcessError(retcode, cmd) > > subprocess.CalledProcessError: Command '['powershell', '-Command', > > > "(new-object > > System.Net.WebClient).DownloadFile > > > ('https://pypi.python.org/packages/source/s/se > > tuptools/setuptools-2.0.2.tar.gz', 'c:\\\\setuptools-2.0.2.tar.gz')"]' > > > returned > > non-zero exit status 1 > > > c:\> > > The parts in red were in red in my terminal window. I am on windows 8.1 and > I am brand new. Less than a week ago, I installed Distribute because someone > on Django Google Groups recommended it to me, along with virtualenv and pip. > Then today - well, last night since it was a few hours ago - I found out > Distribute is deprecated. So I am trying to follow the uninstall > instructions but this is what I got. Just for kicks I ran it again just now > - same result. Please help. I took time off from work Thursday and Friday to > get all this set up have been at this all weekend only to be stumped again. > And no, I do not have Django up and running yet. Thank you. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From steve at spvi.com Mon Jan 6 21:26:49 2014 From: steve at spvi.com (Steve Spicklemire) Date: Mon, 6 Jan 2014 15:26:49 -0500 Subject: [Distutils] Packaging today Message-ID: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Hi Distutils folks, I'm kind of a lurker here, due primarily to the fact that I'm too swamped with various other things to materially contribute to the effort, but I've been lurking for some time hoping to learn enough to avoid troubling anyone with pesky questions. In that respect I've apparently failed, because here comes the question! Background: I'm helping out with a python package: vpython and I'm also teaching an intro scientific computing class this spring. I'm mostly a Mac/Linux user, but my students are often windows users. I would love to permit my students to use enthought/canopy and/or continuum analytics (C.A.) along with vpython. At the moment we're creating binary releases of vpython for windows and mac and posting them on sourceforge . Bruce has been building the windows binary using VC (no setup.py) in a way that's compatible withpython.org python for windows. I've been building the mac version using a setup.py script I cobbled together that works on MacOSX and Linux. I've noticed that the anaconda system that C.A. installs uses MinGW on windows to build extensions. I'd love to figure out how to build vpython under this system so that my windows users could use them together transparently. (BTW is this the same 'anaconda' that has been discussed on this list recently, or is that something different?) I'm pretty sure I could work out how to build vpython with continuum analytics on the mac (which means building boost + wxPython using the C.A. python). Questions: Is there any way, *today*, to incorporate dependencies on external libraries (e.g., boost) in setup.py? (I've noticed the recent conversation about binary dependencies, but from the discussion it seemed to be mostly about the future...) Where should I go to get the 'latest' advise/documentation on distribution for a package that I want to distribute today (rather than pestering you folks)? thanks! -steve (P.S., I tried to send this earlier today, but it appears to have failed. Apologies if you've gotten this twice) Steve Spicklemire University of Indianapolis Dept. of Physics and Earth Space Sciences spicklemire at uindy.edu (317) 788-3313 From chris.barker at noaa.gov Mon Jan 6 22:55:08 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 6 Jan 2014 13:55:08 -0800 Subject: [Distutils] Packaging today In-Reply-To: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: On Mon, Jan 6, 2014 at 12:26 PM, Steve Spicklemire wrote: > avoid troubling anyone with pesky questions. In that respect I've > apparently failed, because here comes the question! > I think this is a use case that is quite useful for us all to chew on a bit... 1st -- yes "Anaconda" refers to the distribution from Continuum. Note that "conda" is the packaging system that Anaconda uses, but it can also be used independently of the distribution. > I'm helping out with a python package: vpython [side note: fairly recently a por tof VPython to wxPython was done -- is that what you are using? Notable because I think wxPython is still not available for Anaconda...) > and I'm also teaching an intro scientific computing class this spring. I'm > mostly a Mac/Linux user, but my students are often windows users. I would > love to permit my students to use enthought/canopy and/or continuum > analytics (C.A.) along with vpython. Either/or? As an instructor, I'd recommend you pick one and go with it -- if you need wx, that means Canopy for now. Alternatively, you suggest the python.org builds, and point your users to binaries they can get elsewhere (Chris Gohlke's site for Windows...) At the moment we're creating binary releases of vpython for windows and mac > and posting them on sourceforge < > https://sourceforge.net/projects/vpythonwx/>. Are these for the python.org builds? good for you! Bruce has been building the windows binary using VC (no setup.py) in a way > that's compatible withpython.org python for windows. I've been building > the mac version using a setup.py script I cobbled together that works on > MacOSX and Linux. Why not distutils for building Windows? I find it really helpful. > I've noticed that the anaconda system that C.A. installs uses MinGW on > windows to build extensions. I think Canopy does that too -- at least it did a few years ago. but I _think_ you can build extensions with either MinGW or MSVC for the same binary python -- if it's set up right ;-) I'd love to figure out how to build vpython under this system so that my > windows users could use them together transparently. You want to take a look at conda: https://github.com/pydata/conda If you can build a conda recipe then you are set to go... That being said, it is supposed to be a goal for Anaconda to be binary compatible with the python.org binaries -- so you may well be able to build the way you are, and give the users a way to install it into Anaconda. In theory, binary wheels are the way to do this. I'm pretty sure I could work out how to build vpython with continuum > analytics on the mac (which means building boost + wxPython using the C.A. > python). > ahh -- you are using wx -- I'd check with the CA folks and see where they are at -- they said they were working on a wxPython package....and I'm sure they'd like help and testing... > Is there any way, *today*, to incorporate dependencies on external > libraries (e.g., boost) in setup.py? > no -- except by hand with custom code. I'm still a little unclear on your goals here. If you want to simple be able to tell your students to use Anaconda then look into conda and the CA help lists -- and conda is more or less designed to solve these sorts of problems. also, the odds are good that Anaconda already has boost, and if not someone has done a conda recipe for it: https://github.com/faircloth-lab/conda-recipes/tree/master/boost If you want your users to be able to use any of: Anaconda Python.org python Canopy Then I'd look into building binary wheels, and see if you can get them to work with Anaconda and Canopy. Note: distutils does not address the third-party non-python dependency problem -- conda tries does address it (though still not clear to me if it helps you build them...) You may also want to look at Gattai, which aims to help you build them: http://sourceforge.net/projects/gattai/ HTH, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at spvi.com Tue Jan 7 00:20:09 2014 From: steve at spvi.com (Steve Spicklemire) Date: Mon, 6 Jan 2014 18:20:09 -0500 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: Thanks Chris for the detailed reply. On Jan 6, 2014, at 4:55 PM, Chris Barker wrote: > Either/or? As an instructor, I'd recommend you pick one and go with it -- if you need wx, that means Canopy for now. Alternatively, you suggest the python.org builds, and point your users to binaries they can get elsewhere (Chris Gohlke's site for Windows...) Right. My impression is/was that python.org/CA/Canopy were all different builds of python that were *not* interoperable. So that a binary built with one could not generally be expected to work with another. If that's not true, then maybe this is a non-problem. I guess I should just "try it" and see what happens. > > At the moment we're creating binary releases of vpython for windows and mac and posting them on sourceforge . > > Are these for the python.org builds? good for you! Yes they are for the python.org builds. > > Bruce has been building the windows binary using VC (no setup.py) in a way that's compatible withpython.org python for windows. I've been building the mac version using a setup.py script I cobbled together that works on MacOSX and Linux. > > Why not distutils for building Windows? I find it really helpful. I don't even have a 'real' windows system (only VirtualBox) and I don't have VC Studio, so Bruce has been doing all the windows work (he's got code that creates an installer that's been working for years). My attempt at a setup.py for windows didn't work for him, and I have limited ability to debug it myself. If I could get MinGW to work with my setup.py, maybe I could get things going. I just wanted to get some confirmation that it was a reasonable plan before I wasted a lot of time on what might turn out to be a dead end. Also, I've built wxPython on the mac from source multiple times so that I could turn on debug symbols. Do you think the build-wxpython.py script would work under windows with MinGW? I guess that's probably kind of a naive hope. ;-) > > I've noticed that the anaconda system that C.A. installs uses MinGW on windows to build extensions. > > I think Canopy does that too -- at least it did a few years ago. but I _think_ you can build extensions with either MinGW or MSVC for the same binary python -- if it's set up right ;-) As before, maybe this is a non-problem that I should just try it. That'll be step 1. ;-) > > I'd love to figure out how to build vpython under this system so that my windows users could use them together transparently. > > You want to take a look at conda: > > https://github.com/pydata/conda > > If you can build a conda recipe then you are set to go... > > That being said, it is supposed to be a goal for Anaconda to be binary compatible with the python.org binaries -- so you may well be able to build the way you are, and give the users a way to install it into Anaconda. In theory, binary wheels are the way to do this. > > I'm pretty sure I could work out how to build vpython with continuum analytics on the mac (which means building boost + wxPython using the C.A. python). > > ahh -- you are using wx -- I'd check with the CA folks and see where they are at -- they said they were working on a wxPython package....and I'm sure they'd like help and testing... > > Is there any way, *today*, to incorporate dependencies on external libraries (e.g., boost) in setup.py? > > no -- except by hand with custom code. > > > I'm still a little unclear on your goals here. Well I guess I am too. I was impressed with CAs ability to use 'pip' on windows to install plotly right away. It's almost like working in unix. I liked that! It would be a dream if a student, after installing the CA distribution could type: /anaconda/bin/pip install vpython and it would "just work". I understand that's impossible at the moment. But if I could create instructions and/or build a set of binary files a student could easily install that would give them: 1) vpython 2) matplotlib 3) ipython 4) scipy I'd be ecstatic. I'll also check Chris Gohlke's site. Maybe I don't need all the bells and whistles of Canopy/CA etc. > If you want to simple be able to tell your students to use Anaconda then look into conda and the CA help lists -- and conda is more or less designed to solve these sorts of problems. also, the odds are good that Anaconda already has boost, and if not someone has done a conda recipe for it: > > https://github.com/faircloth-lab/conda-recipes/tree/master/boost > > If you want your users to be able to use any of: > > Anaconda > Python.org python > Canopy > > Then I'd look into building binary wheels, and see if you can get them to work with Anaconda and Canopy. > > Note: distutils does not address the third-party non-python dependency problem -- conda tries does address it (though still not clear to me if it helps you build them...) You may also want to look at Gattai, which aims to help you build them: > > http://sourceforge.net/projects/gattai/ OK. Thanks so much! I appreciate your pointers. I'll let you know what I discover. Ultimately I'd like to help Bruce package vpython in such a way that folks can use 'pip' to include wx and vpython in whichever python distribution they happen to choose without a lot of fuss. thanks, -steve > > HTH, > -Chris > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov From maciej at opencsw.org Tue Jan 7 10:07:56 2014 From: maciej at opencsw.org (=?UTF-8?Q?Maciej_=28Matchek=29_Blizi=C5=84ski?=) Date: Tue, 7 Jan 2014 09:07:56 +0000 Subject: [Distutils] File names with spaces In-Reply-To: References: Message-ID: 2013/10/1 Maciej (Matchek) Blizi?ski > Hello setuptools developers, > > I'm attempting to package the newest setuptools version (1.1.6) > on Solaris 9 and 10. One of the limitations of the Solaris package manager > (the old one, pkgadd/pkgrm), is that it is unable to handle file names > with > spaces. Would you mind renaming "script template.py" to something like > "script_template.py"? > No replies. Did I write to a wrong list? I've created a pull request with the change, do you have any advice who should I talk to? https://bitbucket.org/pypa/setuptools/pull-request/33/rename-script-templatepy-to/diff Maciej -------------- next part -------------- An HTML attachment was scrubbed... URL: From spicklemire at uindy.edu Mon Jan 6 12:35:05 2014 From: spicklemire at uindy.edu (Steve Spicklemire) Date: Mon, 6 Jan 2014 06:35:05 -0500 Subject: [Distutils] Packaging today... Message-ID: Hi Distutils folks, I'm kind of a lurker here, due primarily to the fact that I'm too swamped with various other things to materially contribute to the effort, but I've been lurking for some time hoping to learn enough to avoid troubling anyone with pesky questions. In that respect I've apparently failed, because here comes the question! Background: I'm helping out with a python package: vpython and I'm also teaching an intro scientific computing class this spring. I'm mostly a Mac/Linux user, but my students are often windows users. I would love to permit my students to use enthought/canopy and/or continuum analytics (C.A.) along with vpython. At the moment we're creating binary releases of vpython for windows and mac and posting them on sourceforge . Bruce has been building the windows binary using VC (no setup.py) in a way that's compatible with python.org python for windows. I've been building the mac version using a setup.py script I cobbled together that works on MacOSX and Linux. I've noticed that the anaconda system that C.A. installs uses MinGW on windows to build extensions. I'd love to figure out how to build vpython under this system so that my windows users could use them together transparently. (BTW is this the same 'anaconda' that has been discussed on this list recently, or is that something different?) I'm pretty sure I could work out how to build vpython with continuum analytics on the mac (which means building boost + wxPython using the C.A. python). Questions: Is there any way, *today*, to incorporate dependencies on external libraries (e.g., boost) in setup.py? (I've noticed the recent conversion about binary dependencies, but from the discussion it seemed to be mostly about the future...) Where should I go to get the 'latest' advise/documentation on distribution for a package that I want to distribute today (rather than pestering you folks)? thanks! -steve Steve Spicklemire University of Indianapolis Dept. of Physics and Earth Space Sciences spicklemire at uindy.edu (317) 788-3313 From donald at stufft.io Wed Jan 8 00:19:03 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Jan 2014 18:19:03 -0500 Subject: [Distutils] Virtualenv 1.11.1rc1 Available for Testing Message-ID: <726F4AE5-CBF5-4B0C-82B8-3DB3A3DC7C36@stufft.io> virtualenv 1.11.1rc1 has been released, which solves a serious issue with the ``?system-site-packages`` flag. In order to test this copy of virtualenv: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.11.1rc1.tar.gz $ echo "e517f28a31792f00699b369932a9b66a 1.11.1rc1.tar.gz" | md5sum -c 1.11.1rc1.tar.gz: OK $ tar zxf 1.11.1rc1.tar.gz $ python virtualenv-1.11.1rc1/virtualenv.py myVE $ myVE/bin/pip install SomePackage ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From pje at telecommunity.com Wed Jan 8 00:39:29 2014 From: pje at telecommunity.com (PJ Eby) Date: Tue, 7 Jan 2014 18:39:29 -0500 Subject: [Distutils] File names with spaces In-Reply-To: References: Message-ID: On Tue, Jan 7, 2014 at 4:07 AM, Maciej (Matchek) Blizi?ski wrote: > 2013/10/1 Maciej (Matchek) Blizi?ski >> >> Hello setuptools developers, >> >> I'm attempting to package the newest setuptools version (1.1.6) >> on Solaris 9 and 10. One of the limitations of the Solaris package manager >> (the old one, pkgadd/pkgrm), is that it is unable to handle file names >> with >> spaces. Would you mind renaming "script template.py" to something like >> "script_template.py"? Probably yes, they would mind. ;-) I believe the reason there's a space there is so that the file is not importable. (It should never be imported; it's a data file rather than a python module.) I suspect you'll have better luck with a suggestion like 'script-template.py', since that would still not be importable. > I've created a pull request with the change, do you have any advice who > should I talk to? > > https://bitbucket.org/pypa/setuptools/pull-request/33/rename-script-templatepy-to/diff I've added a comment there about the underscore vs. dash; it looks from the other pull requests like Jason wants issues filed describing the problems that a pull request is intended to fix, and that may mean he hasn't seen the pull request yet. From chris.barker at noaa.gov Wed Jan 8 02:06:24 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 7 Jan 2014 17:06:24 -0800 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: On Mon, Jan 6, 2014 at 3:20 PM, Steve Spicklemire wrote: > Thanks Chris for the detailed reply. Well, I'm trying to sort out similar issues myself.... Right. My impression is/was that python.org/CA/Canopy were all different > builds of python that were *not* interoperable. well, in the case of Anaconda, Travis told me the intent was that it would be -- the reality, I'm not sure about. > So that a binary built with one could not generally be expected to work > with another. If that's not true, then maybe this is a non-problem. I guess > I should just "try it" and see what happens. it's still tricky to get things to install correctly -- the Windows (and Mac) installers expect python to be in a particular location -- Anaconda is not there. > Why not distutils for building Windows? I find it really helpful. > > I don't even have a 'real' windows system (only VirtualBox) and I don't > have VC Studio, you should be able to do it with the free visual studio express 2008. A bit hard to find an installer these days, but I think it's still there. I've had much better luck with that than MinGW. Do you think the build-wxpython.py script would work under windows with > MinGW? I guess that's probably kind of a naive hope. ;-) I doubt it --- but again VS2008 Express might build it OK -- but then Robin provides installers for wx anyway. > Well I guess I am too. I was impressed with CAs ability to use 'pip' on > windows to install plotly right away. It's almost like working in unix. I > liked that! > is plotly pure python? in that case, then it's pretty easy, really. /anaconda/bin/pip install vpython > pip install with compiled binaries is a different beast -- I _think_ pypi is now set up to find binary wheels that match the python.org python. I have no idea if those will install under Anaconda. But you probably want "conda install vpython" if you want Anaconda anyway. > and it would "just work". > > I understand that's impossible at the moment. But if I could create > instructions and/or build a set of binary files a student could easily > install that would give them: > > 1) vpython > 2) matplotlib > 3) ipython > 4) scipy > we're pretty close to having all these as binary wheels now. There isn't much stopping it. vpyton is up to you. But wx is not there -- though if you can get it to build on Windows, making a wheel of it should be easy. I imagine Robin would be happy to put them up in PyPi. I'd be ecstatic. I'll also check Chris Gohlke's site. Maybe I don't need > all the bells and whistles of Canopy/CA etc. > > That is a GREAT resource! Ultimately I'd like to help Bruce package vpython in such a way that folks > can use 'pip' to include wx and vpython in whichever python distribution > they happen to choose without a lot of fuss. > It's not really a full-on goal for Anaconda or Canopy to be fully pip compatible -- so that may be a bit of a fantasy... Also: you can point pip at a custom "wheelhouse" -- i.e. a collection of wheels that you put together. In your position, I"d be tempted to provide a full set of wheels for the python.org build for everything that you need that isn't already pip-installable. Then point your students to that. If you're lucky, those some wheels may work with Anaconda, or even Canopy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Jan 8 10:48:25 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Jan 2014 09:48:25 +0000 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: On Mon, Jan 6, 2014 at 9:55 PM, Chris Barker wrote: > On Mon, Jan 6, 2014 at 12:26 PM, Steve Spicklemire wrote: > > >> avoid troubling anyone with pesky questions. In that respect I've >> apparently failed, because here comes the question! >> > > I think this is a use case that is quite useful for us all to chew on a > bit... > > 1st -- yes "Anaconda" refers to the distribution from Continuum. Note that > "conda" is the packaging system that Anaconda uses, but it can also be used > independently of the distribution. > > >> I'm helping out with a python package: vpython > > > [side note: fairly recently a por tof VPython to wxPython was done -- is > that what you are using? Notable because I think wxPython is still not > available for Anaconda...) > > >> and I'm also teaching an intro scientific computing class this spring. >> I'm mostly a Mac/Linux user, but my students are often windows users. I >> would love to permit my students to use enthought/canopy and/or continuum >> analytics (C.A.) along with vpython. > > > Either/or? As an instructor, I'd recommend you pick one and go with it -- > if you need wx, that means Canopy for now. Alternatively, you suggest the > python.org builds, and point your users to binaries they can get > elsewhere (Chris Gohlke's site for Windows...) > > At the moment we're creating binary releases of vpython for windows and >> mac and posting them on sourceforge < >> https://sourceforge.net/projects/vpythonwx/>. > > > Are these for the python.org builds? good for you! > > Bruce has been building the windows binary using VC (no setup.py) in a way >> that's compatible withpython.org python for windows. I've been building >> the mac version using a setup.py script I cobbled together that works on >> MacOSX and Linux. > > > Why not distutils for building Windows? I find it really helpful. > > >> I've noticed that the anaconda system that C.A. installs uses MinGW on >> windows to build extensions. > > > I think Canopy does that too -- at least it did a few years ago. but I > _think_ you can build extensions with either MinGW or MSVC for the same > binary python -- if it's set up right ;-) > We don't use mingw to build packages distributed within canopy (at least not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC often causes trouble. We do provide mingw so that people can build extensions without depending on MS toolchain, but that's not a setup I would recommend in general. It works well for extensions that don't depend much on the C runtime (numpy is a good example), but as soon as you manipulate IO, etc... you encounter issues linked to ABI incompatibilities between what you get from the python runtime and mingw. David > > I'd love to figure out how to build vpython under this system so that my >> windows users could use them together transparently. > > > You want to take a look at conda: > > https://github.com/pydata/conda > > If you can build a conda recipe then you are set to go... > > That being said, it is supposed to be a goal for Anaconda to be binary > compatible with the python.org binaries -- so you may well be able to > build the way you are, and give the users a way to install it into > Anaconda. In theory, binary wheels are the way to do this. > > I'm pretty sure I could work out how to build vpython with continuum >> analytics on the mac (which means building boost + wxPython using the C.A. >> python). >> > > ahh -- you are using wx -- I'd check with the CA folks and see where they > are at -- they said they were working on a wxPython package....and I'm sure > they'd like help and testing... > > >> Is there any way, *today*, to incorporate dependencies on external >> libraries (e.g., boost) in setup.py? >> > > no -- except by hand with custom code. > > > I'm still a little unclear on your goals here. If you want to simple be > able to tell your students to use Anaconda then look into conda and the CA > help lists -- and conda is more or less designed to solve these sorts of > problems. also, the odds are good that Anaconda already has boost, and if > not someone has done a conda recipe for it: > > https://github.com/faircloth-lab/conda-recipes/tree/master/boost > > If you want your users to be able to use any of: > > Anaconda > Python.org python > Canopy > > Then I'd look into building binary wheels, and see if you can get them to > work with Anaconda and Canopy. > > Note: distutils does not address the third-party non-python dependency > problem -- conda tries does address it (though still not clear to me if it > helps you build them...) You may also want to look at Gattai, which aims to > help you build them: > > http://sourceforge.net/projects/gattai/ > > HTH, > -Chris > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maciej at opencsw.org Wed Jan 8 13:08:35 2014 From: maciej at opencsw.org (=?UTF-8?Q?Maciej_=28Matchek=29_Blizi=C5=84ski?=) Date: Wed, 8 Jan 2014 12:08:35 +0000 Subject: [Distutils] File names with spaces In-Reply-To: References: Message-ID: 2014/1/7 PJ Eby > Probably yes, they would mind. ;-) I believe the reason there's a > space there is so that the file is not importable. (It should never > be imported; it's a data file rather than a python module.) I suspect > you'll have better luck with a suggestion like 'script-template.py', > since that would still not be importable. The setuptools code did not make it clear at all. Thanks for explaining! > > I've created a pull request with the change, do you have any advice who > > should I talk to? > > > > https://bitbucket.org/pypa/setuptools/pull-request/33/rename-script-templatepy-to/diff > > I've added a comment there about the underscore vs. dash; it looks > from the other pull requests like Jason wants issues filed describing > the problems that a pull request is intended to fix, and that may mean > he hasn't seen the pull request yet. Thanks for the suggestion, I've created an issue: https://bitbucket.org/pypa/setuptools/issue/134/spaces-in-file-names-make-it-impossible-to I'm not sure if there's a bitbucket-internal linking between issues and pull requests? I've just pasted the link in a comment to the pull request. Maciej From cournape at gmail.com Wed Jan 8 19:48:23 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Jan 2014 18:48:23 +0000 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: On Wed, Jan 8, 2014 at 6:44 PM, Chris Barker wrote: > On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau wrote: > >> We don't use mingw to build packages distributed within canopy (at least >> not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC >> often causes trouble. >> >> > so is Canopy binary-compatible with the python.org builds? > > i.e would the same binary wheel work for both? > It should, and we will fix issues if they come up. David > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at spvi.com Wed Jan 8 20:09:49 2014 From: steve at spvi.com (Steve Spicklemire) Date: Wed, 8 Jan 2014 14:09:49 -0500 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: <50E8D742-2A44-4CD5-A318-1B3B7BA64D43@spvi.com> This is good news. I'll experiment! thanks, -steve On Jan 8, 2014, at 1:48 PM, David Cournapeau wrote: > > > > On Wed, Jan 8, 2014 at 6:44 PM, Chris Barker wrote: > On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau wrote: > We don't use mingw to build packages distributed within canopy (at least not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC often causes trouble. > > > so is Canopy binary-compatible with the python.org builds? > > i.e would the same binary wheel work for both? > > It should, and we will fix issues if they come up. > > David > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > From chris.barker at noaa.gov Wed Jan 8 19:44:59 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 8 Jan 2014 10:44:59 -0800 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau wrote: > We don't use mingw to build packages distributed within canopy (at least > not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC > often causes trouble. > > so is Canopy binary-compatible with the python.org builds? i.e would the same binary wheel work for both? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 9 07:57:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jan 2014 16:57:09 +1000 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> Message-ID: On 9 January 2014 04:44, Chris Barker wrote: > On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau wrote: >> >> We don't use mingw to build packages distributed within canopy (at least >> not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC often >> causes trouble. >> > > so is Canopy binary-compatible with the python.org builds? > > i.e would the same binary wheel work for both? The same should be true for Anaconda (they had a bug where they used the wrong runtime for Python 3.3 on Windows but that should have been fixed a few months ago). Cheers, Nick. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bkabrda at redhat.com Thu Jan 9 10:29:54 2014 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Thu, 9 Jan 2014 04:29:54 -0500 (EST) Subject: [Distutils] Wheel "build tag" In-Reply-To: <611830167.27762689.1389257997052.JavaMail.root@redhat.com> Message-ID: <1925649004.27796148.1389259794569.JavaMail.root@redhat.com> Hi all, the wheel package format, as defined in pep 427 specifies, that wheel name can contain an optional "build tag" [1]. This build tag is supposed to be a tie-breaker if two wheels have the same version. I decided that I would try to send a patch to upstream wheel project [2] to support adding this tag. I have a few questions and maybe someone here (Nick?) can help me with this: - Where is the "build tag" supposed to be stored in a built distribution? The pep doesn't seem to specify this. Should it be somewhere in METADATA or pydist.json? - How should the bdist_wheel command be told to use a "build tag"? By using some commandline option, e.g. "--build-tag 3"? My motivation for doing this is: For Fedora's Python 3.4 packaging, I'll have to unbundle the bundled setuptools and pip wheels and make ensurepip work differently. The best approach seems to be to make python3 require setuptools and pip (in other words, they'll always install along), so if user runs "python3 -m ensurepip", ensurepip won't need to do anything. This gets tricky if the same command is run in a virtualenv, since I don't have the bundled wheels. I've discussed this with Nick and we've agreed that the best solution would be to recreate the wheels from the system installed RPMs dynamically and then installing these wheels into virtualenv (I've already done some progress on this front with a small project that I call rewheel [3]). I'm now working on a downstream patch of ensurepip, that would use rewheel to do this. Now let me explain a bit about how we package setuptools/pip in Fedora. When there is a security issue or just a bug in a package, we don't upgrade it to latest upstream version in stable Fedora releases, since as a distribution we try to maintain API/ABI stability. We rather bump the "release" (which is sort of downstream version of the package). E.g. if there is a bug in package pip-%{version}-%{release}, we fix it and bump it to pip-%{version}-%{release+1}. But that means that if we fix a package and someone runs "python -m ensurepip --upgrade", he won't get the fix into virtualenv because the version is the same. That's why I'd like to use Fedora's release as the "build tag", so that this use case would work. I hope this all makes some sense :) Thanks! -- Regards, Bohuslav "Slavek" Kabrda. [1] http://www.python.org/dev/peps/pep-0427/#file-name-convention [2] https://bitbucket.org/dholth/wheel/ [3] https://github.com/bkabrda/rewheel From donald at stufft.io Thu Jan 9 13:35:32 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 9 Jan 2014 07:35:32 -0500 Subject: [Distutils] Wheel "build tag" In-Reply-To: <1925649004.27796148.1389259794569.JavaMail.root@redhat.com> References: <1925649004.27796148.1389259794569.JavaMail.root@redhat.com> Message-ID: <1E6A02BF-DCF7-463C-BA05-0A3FE70730D1@stufft.io> On Jan 9, 2014, at 4:29 AM, Bohuslav Kabrda wrote: > Hi all, > the wheel package format, as defined in pep 427 specifies, that wheel name can contain an optional "build tag" [1]. This build tag is supposed to be a tie-breaker if two wheels have the same version. I decided that I would try to send a patch to upstream wheel project [2] to support adding this tag. > I have a few questions and maybe someone here (Nick?) can help me with this: > - Where is the "build tag" supposed to be stored in a built distribution? The pep doesn't seem to specify this. Should it be somewhere in METADATA or pydist.json? > - How should the bdist_wheel command be told to use a "build tag"? By using some commandline option, e.g. "--build-tag 3"? > > My motivation for doing this is: > For Fedora's Python 3.4 packaging, I'll have to unbundle the bundled setuptools and pip wheels and make ensurepip work differently. The best approach seems to be to make python3 require setuptools and pip (in other words, they'll always install along), so if user runs "python3 -m ensurepip", ensurepip won't need to do anything. This gets tricky if the same command is run in a virtualenv, since I don't have the bundled wheels. I've discussed this with Nick and we've agreed that the best solution would be to recreate the wheels from the system installed RPMs dynamically and then installing these wheels into virtualenv (I've already done some progress on this front with a small project that I call rewheel [3]). I'm now working on a downstream patch of ensurepip, that would use rewheel to do this. > Now let me explain a bit about how we package setuptools/pip in Fedora. When there is a security issue or just a bug in a package, we don't upgrade it to latest upstream version in stable Fedora releases, since as a distribution we try to maintain API/ABI stability. We rather bump the "release" (which is sort of downstream version of the package). E.g. if there is a bug in package pip-%{version}-%{release}, we fix it and bump it to pip-%{version}-%{release+1}. But that means that if we fix a package and someone runs "python -m ensurepip --upgrade", he won't get the fix into virtualenv because the version is the same. That's why I'd like to use Fedora's release as the "build tag", so that this use case would work. I hope this all makes some sense :) > > > Thanks! > > -- > Regards, > Bohuslav "Slavek" Kabrda. > > [1] http://www.python.org/dev/peps/pep-0427/#file-name-convention > [2] https://bitbucket.org/dholth/wheel/ > [3] https://github.com/bkabrda/rewheel > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I?m not entirely sure the build tag actually works at this point in time. It might though. Another option would be to use the local version number as specified in PEP440. So if you have pip 1.5 bundled, and you change something, you make it pip 1.5-1, then if another change comes out you make it pip 1.5-2, etc. Again this might not work exactly right at the moment as this is a case that hasn?t been well tested yet and it might be seen as a pre-release. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Thu Jan 9 14:31:31 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 9 Jan 2014 08:31:31 -0500 Subject: [Distutils] Wheel "build tag" In-Reply-To: <1E6A02BF-DCF7-463C-BA05-0A3FE70730D1@stufft.io> References: <1925649004.27796148.1389259794569.JavaMail.root@redhat.com> <1E6A02BF-DCF7-463C-BA05-0A3FE70730D1@stufft.io> Message-ID: The wheel build tag is only stored in the filename. The feature is not well tested. On Thu, Jan 9, 2014 at 7:35 AM, Donald Stufft wrote: > > On Jan 9, 2014, at 4:29 AM, Bohuslav Kabrda wrote: > >> Hi all, >> the wheel package format, as defined in pep 427 specifies, that wheel name can contain an optional "build tag" [1]. This build tag is supposed to be a tie-breaker if two wheels have the same version. I decided that I would try to send a patch to upstream wheel project [2] to support adding this tag. >> I have a few questions and maybe someone here (Nick?) can help me with this: >> - Where is the "build tag" supposed to be stored in a built distribution? The pep doesn't seem to specify this. Should it be somewhere in METADATA or pydist.json? >> - How should the bdist_wheel command be told to use a "build tag"? By using some commandline option, e.g. "--build-tag 3"? >> >> My motivation for doing this is: >> For Fedora's Python 3.4 packaging, I'll have to unbundle the bundled setuptools and pip wheels and make ensurepip work differently. The best approach seems to be to make python3 require setuptools and pip (in other words, they'll always install along), so if user runs "python3 -m ensurepip", ensurepip won't need to do anything. This gets tricky if the same command is run in a virtualenv, since I don't have the bundled wheels. I've discussed this with Nick and we've agreed that the best solution would be to recreate the wheels from the system installed RPMs dynamically and then installing these wheels into virtualenv (I've already done some progress on this front with a small project that I call rewheel [3]). I'm now working on a downstream patch of ensurepip, that would use rewheel to do this. >> Now let me explain a bit about how we package setuptools/pip in Fedora. When there is a security issue or just a bug in a package, we don't upgrade it to latest upstream version in stable Fedora releases, since as a distribution we try to maintain API/ABI stability. We rather bump the "release" (which is sort of downstream version of the package). E.g. if there is a bug in package pip-%{version}-%{release}, we fix it and bump it to pip-%{version}-%{release+1}. But that means that if we fix a package and someone runs "python -m ensurepip --upgrade", he won't get the fix into virtualenv because the version is the same. That's why I'd like to use Fedora's release as the "build tag", so that this use case would work. I hope this all makes some sense :) >> >> >> Thanks! >> >> -- >> Regards, >> Bohuslav "Slavek" Kabrda. >> >> [1] http://www.python.org/dev/peps/pep-0427/#file-name-convention >> [2] https://bitbucket.org/dholth/wheel/ >> [3] https://github.com/bkabrda/rewheel >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > I?m not entirely sure the build tag actually works at this point in time. It might though. > > Another option would be to use the local version number as specified in PEP440. So > if you have pip 1.5 bundled, and you change something, you make it pip 1.5-1, then if > another change comes out you make it pip 1.5-2, etc. Again this might not work exactly > right at the moment as this is a case that hasn?t been well tested yet and it might be > seen as a pre-release. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From ncoghlan at gmail.com Thu Jan 9 19:46:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Jan 2014 04:46:14 +1000 Subject: [Distutils] Wheel "build tag" In-Reply-To: <1E6A02BF-DCF7-463C-BA05-0A3FE70730D1@stufft.io> References: <1925649004.27796148.1389259794569.JavaMail.root@redhat.com> <1E6A02BF-DCF7-463C-BA05-0A3FE70730D1@stufft.io> Message-ID: On 9 Jan 2014 20:35, "Donald Stufft" wrote: > > > On Jan 9, 2014, at 4:29 AM, Bohuslav Kabrda wrote: > > > Hi all, > > the wheel package format, as defined in pep 427 specifies, that wheel name can contain an optional "build tag" [1]. This build tag is supposed to be a tie-breaker if two wheels have the same version. I decided that I would try to send a patch to upstream wheel project [2] to support adding this tag. > > I have a few questions and maybe someone here (Nick?) can help me with this: > > - Where is the "build tag" supposed to be stored in a built distribution? The pep doesn't seem to specify this. Should it be somewhere in METADATA or pydist.json? > > - How should the bdist_wheel command be told to use a "build tag"? By using some commandline option, e.g. "--build-tag 3"? > > > > My motivation for doing this is: > > For Fedora's Python 3.4 packaging, I'll have to unbundle the bundled setuptools and pip wheels and make ensurepip work differently. The best approach seems to be to make python3 require setuptools and pip (in other words, they'll always install along), so if user runs "python3 -m ensurepip", ensurepip won't need to do anything. This gets tricky if the same command is run in a virtualenv, since I don't have the bundled wheels. I've discussed this with Nick and we've agreed that the best solution would be to recreate the wheels from the system installed RPMs dynamically and then installing these wheels into virtualenv (I've already done some progress on this front with a small project that I call rewheel [3]). I'm now working on a downstream patch of ensurepip, that would use rewheel to do this. > > Now let me explain a bit about how we package setuptools/pip in Fedora. When there is a security issue or just a bug in a package, we don't upgrade it to latest upstream version in stable Fedora releases, since as a distribution we try to maintain API/ABI stability. We rather bump the "release" (which is sort of downstream version of the package). E.g. if there is a bug in package pip-%{version}-%{release}, we fix it and bump it to pip-%{version}-%{release+1}. But that means that if we fix a package and someone runs "python -m ensurepip --upgrade", he won't get the fix into virtualenv because the version is the same. That's why I'd like to use Fedora's release as the "build tag", so that this use case would work. I hope this all makes some sense :) > > > > > > Thanks! > > > > -- > > Regards, > > Bohuslav "Slavek" Kabrda. > > > > [1] http://www.python.org/dev/peps/pep-0427/#file-name-convention > > [2] https://bitbucket.org/dholth/wheel/ > > [3] https://github.com/bkabrda/rewheel > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > I?m not entirely sure the build tag actually works at this point in time. It might though. > > Another option would be to use the local version number as specified in PEP440. So > if you have pip 1.5 bundled, and you change something, you make it pip 1.5-1, then if > another change comes out you make it pip 1.5-2, etc. Again this might not work exactly > right at the moment as this is a case that hasn?t been well tested yet and it might be > seen as a pre-release. Aye, this is currently "path less travelled" stuff, but the integrator suffix is indeed intended to be the solution. Between that and the wheel build tag, hopefully we can find something that either works already, or can be made to work with a straightforward patch. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at spvi.com Thu Jan 9 23:43:02 2014 From: steve at spvi.com (Steve Spicklemire) Date: Thu, 9 Jan 2014 17:43:02 -0500 Subject: [Distutils] Packaging today In-Reply-To: <50E8D742-2A44-4CD5-A318-1B3B7BA64D43@spvi.com> References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> <50E8D742-2A44-4CD5-A318-1B3B7BA64D43@spvi.com> Message-ID: So, related question: Should the Mac binaries also work with the python.org mac build? Regarding my quest: I just punted for now (out of time, need something working by Monday). I'm using Christoph Gohlke's binaries for windows (seem to work well!) and cooked up my own custom installer for Mac that loads on top of python.org python and our current VPython installer. All working, so crisis averted. ;-) I'll look into binary wheels when the dust settles. thanks for all the suggestions! -steve On Jan 8, 2014, at 2:09 PM, Steve Spicklemire wrote: > This is good news. I'll experiment! > > thanks, > -steve > > On Jan 8, 2014, at 1:48 PM, David Cournapeau wrote: > >> >> >> >> On Wed, Jan 8, 2014 at 6:44 PM, Chris Barker wrote: >> On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau wrote: >> We don't use mingw to build packages distributed within canopy (at least not anymore). We build everything with MSVC 2008, as mixing mingw/MSVC often causes trouble. >> >> >> so is Canopy binary-compatible with the python.org builds? >> >> i.e would the same binary wheel work for both? >> >> It should, and we will fix issues if they come up. >> >> David >> >> -Chris >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > From cournape at gmail.com Fri Jan 10 02:23:37 2014 From: cournape at gmail.com (David Cournapeau) Date: Fri, 10 Jan 2014 01:23:37 +0000 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> <50E8D742-2A44-4CD5-A318-1B3B7BA64D43@spvi.com> Message-ID: On Thu, Jan 9, 2014 at 10:43 PM, Steve Spicklemire wrote: > So, related question: Should the Mac binaries also work with the > python.org mac build? > Binary compatibility is best achieved on windows: the platform makes it easier, and it is so painful to build stuff there that it would be insane for vendors not to be compatible. Mac is a whole different story. First, you have frameworks vs non frameworks builds, and compilers/dev environments keep changing in incompatible ways: you have gcc, gcc-llvm, clang, SDK which are tied to xcode versions themselves tied to OS versions, it is a mess :) Case in point, the python.org binaries are generally not compatible with the system python provided by Apple (that is you can't use an installer made for one with the other). At least in the scientific community, we generally provide binaries for python.org, and I think it is good practice to avoid touching the system python. David > > Regarding my quest: I just punted for now (out of time, need something > working by Monday). > > I'm using Christoph Gohlke's binaries for windows (seem to work well!) and > cooked up my own custom installer for Mac that loads on top of python.orgpython and our current VPython installer. All working, so crisis averted. > ;-) > > I'll look into binary wheels when the dust settles. > > thanks for all the suggestions! > -steve > > > On Jan 8, 2014, at 2:09 PM, Steve Spicklemire wrote: > > > This is good news. I'll experiment! > > > > thanks, > > -steve > > > > On Jan 8, 2014, at 1:48 PM, David Cournapeau wrote: > > > >> > >> > >> > >> On Wed, Jan 8, 2014 at 6:44 PM, Chris Barker > wrote: > >> On Wed, Jan 8, 2014 at 1:48 AM, David Cournapeau > wrote: > >> We don't use mingw to build packages distributed within canopy (at > least not anymore). We build everything with MSVC 2008, as mixing > mingw/MSVC often causes trouble. > >> > >> > >> so is Canopy binary-compatible with the python.org builds? > >> > >> i.e would the same binary wheel work for both? > >> > >> It should, and we will fix issues if they come up. > >> > >> David > >> > >> -Chris > >> > >> > >> -- > >> > >> Christopher Barker, Ph.D. > >> Oceanographer > >> > >> Emergency Response Division > >> NOAA/NOS/OR&R (206) 526-6959 voice > >> 7600 Sand Point Way NE (206) 526-6329 fax > >> Seattle, WA 98115 (206) 526-6317 main reception > >> > >> Chris.Barker at noaa.gov > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Jan 10 22:37:30 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 10 Jan 2014 13:37:30 -0800 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> <50E8D742-2A44-4CD5-A318-1B3B7BA64D43@spvi.com> Message-ID: What David said, plus: On Thu, Jan 9, 2014 at 10:43 PM, Steve Spicklemire wrote: > >> So, related question: Should the Mac binaries also work with the >> python.org mac build? > > Not sure what "also" is respect to, but the pyton.org builds are a good common denominator: The Apple builds have their issues: - Apple never upgrades them - You can't re-distribute them (build stuff with Py2app) .. at least without violating copyright. While it's appealing for people to not have to install anything, if they are installing 3rd party packages they are installing stuff, son one extra install at the start is not a big deal. So: if you are going to support one binary -- it should be the python.orgone. It is a bit of a pain to build binaries for the Python.org builds, as they are universal and support older OS versions. Personally, I think we should address this by: 1) having a centralized project for building varios binary dependencies that are compatible with the python.org builds -- why should multiple package distributors all have to figure out how to build, e.g. freetype correctly? I've created a repo for this, but haven't gotten far: https://github.com/MacPython/osxinst let me know if you are interested in contributing. 2) Maybe it's time to put out an "official" python.org build that's simpler: perhaps 10.7+ 64bit Intel only. But I'm not sure how many folks still need 32 bit. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at spvi.com Sat Jan 11 13:04:53 2014 From: steve at spvi.com (Steve Spicklemire) Date: Sat, 11 Jan 2014 07:04:53 -0500 Subject: [Distutils] Packaging today In-Reply-To: References: <1C58DB6F-A9F6-44B9-ACFA-A72C42586115@spvi.com> <50E8D742-2A44-4CD5-A318-1B3B7BA64D43@spvi.com> Message-ID: <703954BD-8D4D-4664-B0AA-9D8716AB8DD4@spvi.com> Thanks Chris, From earlier in the thread the statement was made: > i.e would the same binary wheel work for both? > > It should, and we will fix issues if they come up. in reference to Windows builds against MSVC 2008. I was asking if this "binary compatibility with python.org python" also applied to mac builds of canopy. Anyway, I'll check the repo! I'd love it it we could get vpython install strategies that worked with all the major distributions. thanks for the help! -steve On Jan 10, 2014, at 4:37 PM, Chris Barker wrote: > Not sure what "also" is respect to, but the pyton.org builds are a good common denominator: > > The Apple builds have their issues: > - Apple never upgrades them > - You can't re-distribute them (build stuff with Py2app) .. at least without violating copyright. > > While it's appealing for people to not have to install anything, if they are installing 3rd party packages they are installing stuff, son one extra install at the start is not a big deal. > > So: if you are going to support one binary -- it should be the python.org one. > > It is a bit of a pain to build binaries for the Python.org builds, as they are universal and support older OS versions. > > Personally, I think we should address this by: > > 1) having a centralized project for building varios binary dependencies that are compatible with the python.org builds -- why should multiple package distributors all have to figure out how to build, e.g. freetype correctly? > > I've created a repo for this, but haven't gotten far: > > https://github.com/MacPython/osxinst > > let me know if you are interested in contributing. > > 2) Maybe it's time to put out an "official" python.org build that's simpler: perhaps 10.7+ 64bit Intel only. But I'm not sure how many folks still need 32 bit. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov From p.f.moore at gmail.com Sun Jan 12 17:55:51 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Jan 2014 16:55:51 +0000 Subject: [Distutils] A couple of questions on the PyPI API Message-ID: I'm writing an application to maintain a mirror of PyPI package metadata, and I'm trying to understand the changelog data. In particular, I've not been able to find anything that lists the valid values for the "action" element of a change record. Looking at what's available if I just query the database, I get: pkg rename from XXX pkg, version remove pkg remove Owner XXX pkg remove Maintainer XXX pkg update hosting_mode pkg, version update list,of,fields pkg, version new release pkg add Owner XXX pkg add Maintainer XXX pkg, version add url XXX pkg, version add XXX file YYY But I don't think that can be all (for a start, there's no "add a new package" - unless the "new release" action can have a version of None, which I haven't seen) Is there a full list anywhere? I had a brief look at the Warehouse code, but short of tracking down all the places journal entries are stored, I couldn't see a way of finding out. Actually, given that the XMLRPC API only lets you get the releases of a package, the release data for a given release or a list of the url data for a given release, there's not really much need for me to worry beyond that level (so, for example, add url and add file both just mean "grab the url data and replace what's there", and "update" just means "replace the package data but don't bother refreshing the urls"). Does that sound reasonable? Thanks, Paul From marius at pov.lt Tue Jan 14 12:55:46 2014 From: marius at pov.lt (Marius Gedminas) Date: Tue, 14 Jan 2014 13:55:46 +0200 Subject: [Distutils] PyPI download issues from Rackspace Cloud In-Reply-To: <20131125083809.GA13083@fridge.pov.lt> References: <20131123100536.GA30416@fridge.pov.lt> <20131125064805.GA19537@fridge.pov.lt> <20131125083809.GA13083@fridge.pov.lt> Message-ID: <20140114115546.GA5296@fridge.pov.lt> On Mon, Nov 25, 2013 at 10:38:09AM +0200, Marius Gedminas wrote: > My experimental code: https://gist.github.com/mgedmin/7637559 > > I now have two Wireshark pcap files of two SSL conversations: one > failed, one successful. It's a bit beyond my skills to analyze them. > I think the server did send everything (I see a TLSv1 Application Data > record of size 5654, which looks like the final chunk), but Wireshark > marks it as [TCP Out-Of-Order], and then there are some > desperate-looking [TCP Retransmission] packets at the end. > > (Incidentally, the captured download was truncated at 1303669 > bytes -- i.e. it was missing the last three TLS application data chunks > of 8000, 8000 and 5634 bytes.) > > Given that Firefox/curl seem to be able to download PyPI packages over > HTTPS without problems, I'd be inclined to blame it on a bug in the > CPython's ssl module, maybe. But doesn't Requests use it too? I never got to the bottom of this. I upgraded pip to 1.5 (painfully -- had to download the .tar.gz with curl, then use python -m pip install -U pip*.tar.gz to upgrade) and these SSL download errors disappeared. Marius Gedminas -- "Actually, the Singularity seems rather useful in the entire work avoidance field. "I _could_ write up that report now but if I put it off, I may well become a weakly godlike entity, at which point not only will I be able to type faster but my comments will be more on-target." - James Nicoll -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From ethan at stoneleaf.us Tue Jan 14 22:32:01 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 14 Jan 2014 13:32:01 -0800 Subject: [Distutils] setup.py issue - some files are included as intended, but one is not In-Reply-To: References: Message-ID: <52D5ACD1.3000604@stoneleaf.us> On 01/14/2014 01:26 PM, Dan Stromberg wrote: > On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg wrote: >> Hi folks. >> >> I have a setup.py problem that's driving me nuts. > > Anyone? I've received 0 responses. I have no answer, but forwarding to Distutils (hopefully it's an appropriate topic ;) >> I have a treap.py file that tries to "import * from pyx_treap.so", and >> failing that, it'll "import * from py_treap.py" (sans extensions of >> course). Naturally, all 3 of these should be included - although in >> the case of pyx_treap.so, it probably should be a .c that's included. >> >> It is also intended to include nest.py, which has a simple form. >> >> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >> seem to get py_treap.py to be included for some reason. >> >> I get no errors during "python setup.py sdist upload", but upon >> "python $(which pip) install treap", I get: >> $ sudo /usr/bin/python $(which pip) install treap >> Downloading/unpacking treap >> Running setup.py egg_info for package treap >> >> file py_treap.py (for module py_treap) not found >> Installing collected packages: treap >> Running setup.py install for treap >> file py_treap.py (for module py_treap) not found >> file py_treap.py (for module py_treap) not found >> >> file py_treap.py (for module py_treap) not found >> file py_treap.py (for module py_treap) not found >> Successfully installed treap >> Cleaning up... >> >> And it's not successfully installed - py_treap.py is missing. The pyx >> code does its job, so the problem is masked, other than the messages >> above, and the absence of py_treap.py from >> /usr/local/lib/python2.7/dist-packages >> >> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >> least getting packaged up that much. It's not listed in >> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >> , but I can see it in >> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >> . >> >> My setup.py is at: >> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >> >> I've tried that setup.py and several variations on that theme, but >> none seem to include py_treap.py . >> >> Please make some suggestions? How can I get py_treap.py included in >> the pip install? >> >> Thanks! From qwcode at gmail.com Wed Jan 15 00:37:54 2014 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 14 Jan 2014 15:37:54 -0800 Subject: [Distutils] setup.py issue - some files are included as intended, but one is not In-Reply-To: <52D5ACD1.3000604@stoneleaf.us> References: <52D5ACD1.3000604@stoneleaf.us> Message-ID: hello, I checked out your project from svn. looks like it requires some make commands, before it can be packaged? I'm getting tripped up on that. also confused in that your src tree is at 1.30, but the tar you mention is 1.35, and then your install dir was 1.31. maybe get your src tree up to date with where your working copy, and post any make commands that need to be run. On Tue, Jan 14, 2014 at 1:32 PM, Ethan Furman wrote: > On 01/14/2014 01:26 PM, Dan Stromberg wrote: > >> On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg >> wrote: >> >>> Hi folks. >>> >>> I have a setup.py problem that's driving me nuts. >>> >> >> Anyone? I've received 0 responses. >> > > I have no answer, but forwarding to Distutils (hopefully it's an > appropriate topic ;) > > > I have a treap.py file that tries to "import * from pyx_treap.so", and >>> failing that, it'll "import * from py_treap.py" (sans extensions of >>> course). Naturally, all 3 of these should be included - although in >>> the case of pyx_treap.so, it probably should be a .c that's included. >>> >>> It is also intended to include nest.py, which has a simple form. >>> >>> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >>> seem to get py_treap.py to be included for some reason. >>> >>> I get no errors during "python setup.py sdist upload", but upon >>> "python $(which pip) install treap", I get: >>> $ sudo /usr/bin/python $(which pip) install treap >>> Downloading/unpacking treap >>> Running setup.py egg_info for package treap >>> >>> file py_treap.py (for module py_treap) not found >>> Installing collected packages: treap >>> Running setup.py install for treap >>> file py_treap.py (for module py_treap) not found >>> file py_treap.py (for module py_treap) not found >>> >>> file py_treap.py (for module py_treap) not found >>> file py_treap.py (for module py_treap) not found >>> Successfully installed treap >>> Cleaning up... >>> >>> And it's not successfully installed - py_treap.py is missing. The pyx >>> code does its job, so the problem is masked, other than the messages >>> above, and the absence of py_treap.py from >>> /usr/local/lib/python2.7/dist-packages >>> >>> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >>> least getting packaged up that much. It's not listed in >>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >>> , but I can see it in >>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >>> . >>> >>> My setup.py is at: >>> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >>> >>> I've tried that setup.py and several variations on that theme, but >>> none seem to include py_treap.py . >>> >>> Please make some suggestions? How can I get py_treap.py included in >>> the pip install? >>> >>> Thanks! >>> >> _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Jan 15 08:14:59 2014 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 14 Jan 2014 23:14:59 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved Message-ID: Fyi, the "Python Packaging User Guide" has moved from bitbucket to github. The new project home is here: https://github.com/pypa/python-packaging-user-guide and the built site is still here: https://python-packaging-user-guide.readthedocs.org/en/latest/ For those who are interested in helping, a quick survey of the text will show many sections marked up with "FIXME" comments. Although general editing is helpful, I think we mostly need "packaging experts" at this point who are willing to invest the time in filling out and correcting the sections they feel strong on. As stated in the README, "The guide is part of a larger effort to improve all of the packaging and installation docs, including pip, setuptools, virtualenv, and wheel. Ultimately, users need more than a "guide" to feel confident about the current tools. They need complete, accurate and inter-consistent documentation across all the projects" As such, a lot of my effort will be going into improving and normalizing the pip/setuptools/virtualenv/wheel docs directly so the Packaging User Guide doesn't have to do all the work on top of those docs. I have issues open for each of the four projects: https://github.com/pypa/python-packaging-user-guide/issues My goal is to have something that's sufficient for a public announcement by PyCon. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 15 10:30:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Jan 2014 19:30:55 +1000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 15 Jan 2014 17:15, "Marcus Smith" wrote: > > Fyi, the "Python Packaging User Guide" has moved from bitbucket to github. > > The new project home is here: https://github.com/pypa/python-packaging-user-guide > and the built site is still here: https://python-packaging-user-guide.readthedocs.org/en/latest/ Guess I better make a new clone :) > For those who are interested in helping, a quick survey of the text will show many sections marked up with "FIXME" comments. Although general editing is helpful, I think we mostly need "packaging experts" at this point who are willing to invest the time in filling out and correcting the sections they feel strong on. > > As stated in the README, "The guide is part of a larger effort to improve all of the packaging and installation docs, including pip, setuptools, virtualenv, and wheel. Ultimately, users need more than a "guide" to feel confident about the current tools. They need complete, accurate and inter-consistent documentation across all the projects" I would include docs.python.org in that list. I've already added links from the relevant pages to the user's guide (as even in its current incomplete state it is a better end user resource than the stdlib docs), but ultimately we need to cull the current outdated information from the stdlib docs and actively promote the development of cross-version compatible packages. > As such, a lot of my effort will be going into improving and normalizing the pip/setuptools/virtualenv/wheel docs directly so the Packaging User Guide doesn't have to do all the work on top of those docs. I have issues open for each of the four projects: https://github.com/pypa/python-packaging-user-guide/issues > > My goal is to have something that's sufficient for a public announcement by PyCon. There's also the Python 3.4 release in March - I haven't decided yet if the new install docs will point directly at pip's docs or at the user guide, but I'm currently leaning towards the latter. Cheers, Nick. > > Marcus > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshepang at gmail.com Wed Jan 15 17:18:41 2014 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Wed, 15 Jan 2014 18:18:41 +0200 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On Wed, Jan 15, 2014 at 9:14 AM, Marcus Smith wrote: > Fyi, the "Python Packaging User Guide" has moved from bitbucket to github. Why the move? From p.f.moore at gmail.com Wed Jan 15 17:24:03 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jan 2014 16:24:03 +0000 Subject: [Distutils] Metadata 2.0 and single-component version numbers Message-ID: Metadata 2.0 doesn't support versions that are a single number. This has recently been flagged as a pip issue (https://github.com/pypa/pip/pull/1445) because pip won't handle wheels with a single-component version. We can obviously accept the patch for pip as a backward-compatibility measure, but I'm not sure if that's a good idea. wheels, and much of the new pip functionality, is based around Metadata 2.0 and the new packaging PEPs, at least in principle, so I'd rather get a better view on why Metadata 2.0 doesn't accept these versions. Based on that, we can decide what to do in pip. As a data point, pywin32 uses a single-digit version and can't be built with the free MS compilers, so it would be an ideal candidate for distribution as a wheel. But I've seen nothing from the project indicating that they are even aware of wheels yet, and there may be good reasons why wheels won't work for them (registration of WIndows COM components, for example) So - should Metadata 2.0 allow single-number versions? Should we be advocating a version format change to projects like pywin32? If neither, what will be the impact (and do people have views on whether it's a serious enough issue to warrant pip special-casing this)? Thanks, Paul From qwcode at gmail.com Wed Jan 15 18:25:34 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 15 Jan 2014 09:25:34 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: fair question. partly due to personal comfort level with git/github, but also was hoping we'd get more involvement using github, since it's more popular. On Wed, Jan 15, 2014 at 8:18 AM, Tshepang Lekhonkhobe wrote: > On Wed, Jan 15, 2014 at 9:14 AM, Marcus Smith wrote: > > Fyi, the "Python Packaging User Guide" has moved from bitbucket to > github. > > Why the move? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jan 15 20:27:47 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 15 Jan 2014 19:27:47 +0000 (GMT) Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: Message-ID: <1389814067.80756.YahooMailBasic@web172406.mail.ir2.yahoo.com> > fair question. partly due to personal comfort level with git/github, but also was > hoping we'd get more involvement using github, since it's more popular. Your choice, of course, and I understand about the personal preference part. I do wonder how much involvement you will get from people who would be put off getting involved by the fact that a project is on BitBucket (which fully supports Git as well as Mercurial). There's so little difference in functionality between the two sites that I can't see any barrier to a real contributor (real = someone willing to spend a reasonable amount of time on a project, larger than the time required to learn a new but similar site). I suppose time will tell :-) Regards, Vinay Sajip From donald at stufft.io Wed Jan 15 20:53:59 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 15 Jan 2014 14:53:59 -0500 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: <1389814067.80756.YahooMailBasic@web172406.mail.ir2.yahoo.com> References: <1389814067.80756.YahooMailBasic@web172406.mail.ir2.yahoo.com> Message-ID: <066C1D43-2A3C-4367-AD10-465702BAA065@stufft.io> On Jan 15, 2014, at 2:27 PM, Vinay Sajip wrote: >> fair question. partly due to personal comfort level with git/github, but also was >> hoping we'd get more involvement using github, since it's more popular. > > Your choice, of course, and I understand about the personal preference part. I do > wonder how much involvement you will get from people who would be put off > getting involved by the fact that a project is on BitBucket (which fully supports Git > as well as Mercurial). There's so little difference in functionality between the two > sites that I can't see any barrier to a real contributor (real = someone willing to > spend a reasonable amount of time on a project, larger than the time required > to learn a new but similar site). I suppose time will tell :-) > > Regards, > > Vinay Sajip > Not that I won?t contribute to something on bitbucket, but it factors into my decision and I?m less likely to contribute to something on bitbucket, and even more less likely to contribute to something self hosted or on another host. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed Jan 15 20:56:47 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 15 Jan 2014 14:56:47 -0500 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: <066C1D43-2A3C-4367-AD10-465702BAA065@stufft.io> References: <1389814067.80756.YahooMailBasic@web172406.mail.ir2.yahoo.com> <066C1D43-2A3C-4367-AD10-465702BAA065@stufft.io> Message-ID: It's not just part of a joint PSU / PYPA plan to go "full Rackspace"? :-) On Wed, Jan 15, 2014 at 2:53 PM, Donald Stufft wrote: > > On Jan 15, 2014, at 2:27 PM, Vinay Sajip wrote: > >>> fair question. partly due to personal comfort level with git/github, but also was >>> hoping we'd get more involvement using github, since it's more popular. >> >> Your choice, of course, and I understand about the personal preference part. I do >> wonder how much involvement you will get from people who would be put off >> getting involved by the fact that a project is on BitBucket (which fully supports Git >> as well as Mercurial). There's so little difference in functionality between the two >> sites that I can't see any barrier to a real contributor (real = someone willing to >> spend a reasonable amount of time on a project, larger than the time required >> to learn a new but similar site). I suppose time will tell :-) >> >> Regards, >> >> Vinay Sajip >> > > Not that I won?t contribute to something on bitbucket, but it factors into my decision > and I?m less likely to contribute to something on bitbucket, and even more less likely > to contribute to something self hosted or on another host. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From ncoghlan at gmail.com Thu Jan 16 00:13:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Jan 2014 09:13:15 +1000 Subject: [Distutils] Metadata 2.0 and single-component version numbers In-Reply-To: References: Message-ID: On 16 Jan 2014 02:24, "Paul Moore" wrote: > > Metadata 2.0 doesn't support versions that are a single number. Yes it does - that was one of the compatibility changes I made quite some time ago. It's also listed as one of the differences relative to PEP 346. The only things preventing me from accepting PEP 440 at this point are: - needing an updated version parsing regex - needing updated compatibility stats - needing to discuss the idea of supporting version epochs, which are an interesting way of allowing a shift from date based to semantic versioning without order confusion (you include a leading "1:" to indicate a new epoch, with "0:" as the implied default). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 16 00:23:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Jan 2014 09:23:46 +1000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: <1389814067.80756.YahooMailBasic@web172406.mail.ir2.yahoo.com> References: <1389814067.80756.YahooMailBasic@web172406.mail.ir2.yahoo.com> Message-ID: On 16 Jan 2014 05:30, "Vinay Sajip" wrote: > > > fair question. partly due to personal comfort level with git/github, but also was > > hoping we'd get more involvement using github, since it's more popular. > > Your choice, of course, and I understand about the personal preference part. I do > wonder how much involvement you will get from people who would be put off > getting involved by the fact that a project is on BitBucket (which fully supports Git > as well as Mercurial). There's so little difference in functionality between the two > sites that I can't see any barrier to a real contributor (real = someone willing to > spend a reasonable amount of time on a project, larger than the time required > to learn a new but similar site). I suppose time will tell :-) Lowering the barrier to participation for pip, virtualenv and warehouse developers was also a specific consideration. Marcus didn't make the decision to move unilaterally - we discussed it on the user guide issue tracker and decided that on balance it made sense to move, since I didn't mind either way as the originator of the idea, it was easier for him as the author of most of the current content, the overall PyPA dev balance favours GitHub, and most Python community workshops for new users also express a preference for GitHub. While I have concerns about that situation in general, in this case, lowering overall barriers to contribution seemed the most important consideration. Cheers, Nick. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 00:29:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jan 2014 23:29:45 +0000 Subject: [Distutils] Metadata 2.0 and single-component version numbers In-Reply-To: References: Message-ID: On 15 January 2014 23:13, Nick Coghlan wrote: > On 16 Jan 2014 02:24, "Paul Moore" wrote: >> >> Metadata 2.0 doesn't support versions that are a single number. > > Yes it does - that was one of the compatibility changes I made quite some > time ago. It's also listed as one of the differences relative to PEP 346. My apologies. I misread N[.N]+ as the '+' meaning "one or more" (as in regular expressions). Marcus pointed my mistake out to me. Paul From vinay_sajip at yahoo.co.uk Thu Jan 16 00:42:11 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 15 Jan 2014 23:42:11 +0000 (GMT) Subject: [Distutils] Metadata 2.0 and single-component version numbers In-Reply-To: Message-ID: <1389829331.27253.YahooMailBasic@web172406.mail.ir2.yahoo.com> > My apologies. I misread N[.N]+ as the '+' meaning "one or more" (as in > regular expressions). Marcus pointed my mistake out to me. It reads the same way to me - I would expect N[.N]* to indicate that a single numeric component is valid. Still, as long as the regex is right ... distlib has added some support for recent version related changes (e.g. local versions), but this isn't yet in. Regards, Vinay Sajip From qwcode at gmail.com Thu Jan 16 01:03:15 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 15 Jan 2014 16:03:15 -0800 Subject: [Distutils] Metadata 2.0 and single-component version numbers In-Reply-To: References: Message-ID: > > > My apologies. I misread N[.N]+ as the '+' meaning "one or more" (as in > maybe N[(.N)+] would be clearer. all of (.N)+ is optional but when it's present it can be .N or .N.N or .N.N.N etc... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 16 01:12:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Jan 2014 10:12:39 +1000 Subject: [Distutils] Metadata 2.0 and single-component version numbers In-Reply-To: References: Message-ID: On 16 Jan 2014 10:03, "Marcus Smith" wrote: >> >> >> My apologies. I misread N[.N]+ as the '+' meaning "one or more" (as in > > > maybe N[(.N)+] would be clearer. > > all of (.N)+ is optional > but when it's present it can be .N or .N.N or .N.N.N etc... Yeah, that's a good idea. Simply switching to * might encourage the idea it's a regex, when it's more the kind of pseudo grammar people use to describe command line parameters. Cheers, Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Jan 16 01:58:42 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 15 Jan 2014 16:58:42 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > > > As stated in the README, "The guide is part of a larger effort to > improve all of the packaging and installation docs, including pip, > setuptools, virtualenv, and wheel. Ultimately, users need more than a > "guide" to feel confident about the current tools. They need complete, > accurate and inter-consistent documentation across all the projects" > > I would include docs.python.org in that list. I've already added links > from the relevant pages to the user's guide (as even in its current > incomplete state it is a better end user resource than the stdlib docs), > but ultimately we need to cull the current outdated information from the > stdlib docs and actively promote the development of cross-version > compatible packages. > I just added docs.python.org to the "larger effort" list in the README and the intro. I also created an issue for "docs.python.org" work, and brought over some of the comments from bitbucket. https://github.com/pypa/python-packaging-user-guide/issues/12 -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Thu Jan 16 02:25:33 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 15 Jan 2014 17:25:33 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On Wed, Jan 15, 2014 at 1:30 AM, Nick Coghlan wrote: > > On 15 Jan 2014 17:15, "Marcus Smith" wrote: >> >> Fyi, the "Python Packaging User Guide" has moved from bitbucket to github. >> >> The new project home is here: >> https://github.com/pypa/python-packaging-user-guide >> and the built site is still here: >> https://python-packaging-user-guide.readthedocs.org/en/latest/ > > Guess I better make a new clone :) > >> For those who are interested in helping, a quick survey of the text will >> show many sections marked up with "FIXME" comments. Although general >> editing is helpful, I think we mostly need "packaging experts" at this point >> who are willing to invest the time in filling out and correcting the >> sections they feel strong on. >> >> As stated in the README, "The guide is part of a larger effort to improve >> all of the packaging and installation docs, including pip, setuptools, >> virtualenv, and wheel. Ultimately, users need more than a "guide" to feel >> confident about the current tools. They need complete, accurate and >> inter-consistent documentation across all the projects" > > I would include docs.python.org in that list. I've already added links from > the relevant pages to the user's guide (as even in its current incomplete > state it is a better end user resource than the stdlib docs), but ultimately > we need to cull the current outdated information from the stdlib docs and > actively promote the development of cross-version compatible packages. Is there a description somewhere of the plan for what packaging-related information will be covered in docs.python.org proper (and the stages for getting there), and which information will be off-loaded to the documentation for the other projects that Marcus mentioned? Will there be any planned overlap? I seem to remember or got the impression from earlier e-mails on the Distutils list that a lot of the information currently in docs.python.org was going to be removed at some point. (Some of it still seems useful by the way, for example by covering things that are still stubs in the PPUG. Also by the way, is that the right acronym for quick reference? :) ). --Chris From qwcode at gmail.com Thu Jan 16 03:32:32 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 15 Jan 2014 18:32:32 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > PPUG. Also by the way, is that the right acronym for quick reference? :) > I guess so, or maybe "PUG". It's not pretty though. (btw, HHGTP was the acronym for the older "Hitchhiker's Guide to Packaging") > which information will > be off-loaded to the documentation for the other projects that Marcus > mentioned? Will there be any planned overlap? as for PUG overlap to other Pypa projects, the inline FIXME comments mention linking vs recreating content in a number of places. E.g. it seem wasteful to me to recreate yet another setuptools tutorial, when the setuptools docs already has something close to that, but it just need improvements. If you don't see mention of linking in the FIXME, then you can assume the plan is for the content to be in the PUG. I can go thru and make sure that's clear everywhere. as for making the links less "jumpy" to users, that's where my idea of refactoring the pip/setuptools/virtualenv/wheel docs to be more consistent comes in. I have 4 issues open for that, one for each project, and they all reference this outline: https://gist.github.com/qwcode/8431828 I've already started on pip and setuptools, but virtualenv and wheel are open at the moment. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 09:08:22 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 08:08:22 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 01:25, Chris Jerdonek wrote: > Is there a description somewhere of the plan for what > packaging-related information will be covered in docs.python.org > proper (and the stages for getting there), and which information will > be off-loaded to the documentation for the other projects that Marcus > mentioned? I don't have a clear picture of the split myself but it seems to me that docs.python.org should be the master reference data for *distutils*. That's somewhat screwed, though, as we're recommending use of setuptools, and setuptools messes round so invasively with distutils that the usefulness of pure distutils documentation is limited :-( I still think that the best resource available would be a basic "best practice" project template for a simple pure-python package with a few tests. I started putting one together myself (https://github.com/pfmoore/sampleproject). It's basically done, but I'm not sure how it (or something like it) could be incorporated into the docs. At the moment, I tend to use it as a cut-and-paste starting point, and I suspect that most people have something similar. Also, it makes me a bit sad how complicated the setup.py ended up looking (given that it was supposed to be a "minimum starting point" example... Maybe if pip gets support for plugin commands, I could write a "pip new-project" command that generates an empty project template. That might be a good place for it - although it takes pip away from being a pure package management tool and into more of a project development tool, which maybe isn't right. I'd rather not add yet another tool to my "essential staring point for building a project" toolset, though... Paul From ncoghlan at gmail.com Thu Jan 16 10:40:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Jan 2014 19:40:24 +1000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 Jan 2014 18:08, "Paul Moore" wrote: > > On 16 January 2014 01:25, Chris Jerdonek wrote: > > Is there a description somewhere of the plan for what > > packaging-related information will be covered in docs.python.org > > proper (and the stages for getting there), and which information will > > be off-loaded to the documentation for the other projects that Marcus > > mentioned? > > I don't have a clear picture of the split myself but it seems to me > that docs.python.org should be the master reference data for > *distutils*. That's somewhat screwed, though, as we're recommending > use of setuptools, and setuptools messes round so invasively with > distutils that the usefulness of pure distutils documentation is > limited :-( I'm currently thinking the stdlib docs should aim to be an always available reminder/quick reference for end users, and a full reference for packaging and build tool developers. For the end user reference, tutorials, etc, it would defer to the cross version guide. > I still think that the best resource available would be a basic "best > practice" project template for a simple pure-python package with a few > tests. I started putting one together myself > (https://github.com/pfmoore/sampleproject). It's basically done, but > I'm not sure how it (or something like it) could be incorporated into > the docs. At the moment, I tend to use it as a cut-and-paste starting > point, and I suspect that most people have something similar. Also, it > makes me a bit sad how complicated the setup.py ended up looking > (given that it was supposed to be a "minimum starting point" > example... > > Maybe if pip gets support for plugin commands, I could write a "pip > new-project" command that generates an empty project template. That > might be a good place for it - although it takes pip away from being a > pure package management tool and into more of a project development > tool, which maybe isn't right. I'd rather not add yet another tool to > my "essential staring point for building a project" toolset, though... Audrey Roy's "cookiecutter" project is just such a tool (although her default config is far from minimal - setup.py, GitHub, ReadTheDocs, tox, Travis, documentation skeleton, etc. It's all reasonable recommendations, though) Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 11:17:17 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 10:17:17 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 09:40, Nick Coghlan wrote: > Audrey Roy's "cookiecutter" project is just such a tool (although her > default config is far from minimal - setup.py, GitHub, ReadTheDocs, tox, > Travis, documentation skeleton, etc. It's all reasonable recommendations, > though) Agreed, I was pointed to that at one point. For me, it suffers from the problem of being yet another tool, as well as the default template available being very complex, as you said. But yes, it looks useful for more complex projects. I do think the PUG needs a "this is how you start" basic template though, that includes good practices like the right classifiers, using entry points for scripts, basic unit tests, a readme etc. The current guide doesn't even have a "Basic Project Template" section in the index :-( I'll try to find some time to add one. On that note, the PUG index currently feels a little overwhelming from the point of view of someone who just wants to get a job done. it's got a lot of "Getting started with X" and "What is" sections, but nothing like "Installing a package from PyPI" or "Starting a new project". Maybe the index should be reorganised to be more task-oriented, and less concerned with providing history and background to the confusion that led us to where we are now? I'd be tempted to actually remove a lot of what's currently in the contents, and relegate it to some sort of "background" section (which would probably need to be organised into subsections of its own, but that level of detail doesn't need to be on the front page). Paul From donald at stufft.io Thu Jan 16 15:22:59 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 16 Jan 2014 09:22:59 -0500 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On Jan 16, 2014, at 9:20 AM, Daniel Holth wrote: > Has anyone ever written a setup.py that was *not* copy-and-pasted from > somewhere else? > > On Thu, Jan 16, 2014 at 5:17 AM, Paul Moore wrote: >> On 16 January 2014 09:40, Nick Coghlan wrote: >>> Audrey Roy's "cookiecutter" project is just such a tool (although her >>> default config is far from minimal - setup.py, GitHub, ReadTheDocs, tox, >>> Travis, documentation skeleton, etc. It's all reasonable recommendations, >>> though) >> >> Agreed, I was pointed to that at one point. For me, it suffers from >> the problem of being yet another tool, as well as the default template >> available being very complex, as you said. But yes, it looks useful >> for more complex projects. >> >> I do think the PUG needs a "this is how you start" basic template >> though, that includes good practices like the right classifiers, using >> entry points for scripts, basic unit tests, a readme etc. The current >> guide doesn't even have a "Basic Project Template" section in the >> index :-( I'll try to find some time to add one. >> >> On that note, the PUG index currently feels a little overwhelming from >> the point of view of someone who just wants to get a job done. it's >> got a lot of "Getting started with X" and "What is" sections, but >> nothing like "Installing a package from PyPI" or "Starting a new >> project". Maybe the index should be reorganised to be more >> task-oriented, and less concerned with providing history and >> background to the confusion that led us to where we are now? I'd be >> tempted to actually remove a lot of what's currently in the contents, >> and relegate it to some sort of "background" section (which would >> probably need to be organised into subsections of its own, but that >> level of detail doesn't need to be on the front page). >> >> Paul I write my setup.py?s from scratch :D ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Thu Jan 16 15:20:43 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 16 Jan 2014 09:20:43 -0500 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: Has anyone ever written a setup.py that was *not* copy-and-pasted from somewhere else? On Thu, Jan 16, 2014 at 5:17 AM, Paul Moore wrote: > On 16 January 2014 09:40, Nick Coghlan wrote: >> Audrey Roy's "cookiecutter" project is just such a tool (although her >> default config is far from minimal - setup.py, GitHub, ReadTheDocs, tox, >> Travis, documentation skeleton, etc. It's all reasonable recommendations, >> though) > > Agreed, I was pointed to that at one point. For me, it suffers from > the problem of being yet another tool, as well as the default template > available being very complex, as you said. But yes, it looks useful > for more complex projects. > > I do think the PUG needs a "this is how you start" basic template > though, that includes good practices like the right classifiers, using > entry points for scripts, basic unit tests, a readme etc. The current > guide doesn't even have a "Basic Project Template" section in the > index :-( I'll try to find some time to add one. > > On that note, the PUG index currently feels a little overwhelming from > the point of view of someone who just wants to get a job done. it's > got a lot of "Getting started with X" and "What is" sections, but > nothing like "Installing a package from PyPI" or "Starting a new > project". Maybe the index should be reorganised to be more > task-oriented, and less concerned with providing history and > background to the confusion that led us to where we are now? I'd be > tempted to actually remove a lot of what's currently in the contents, > and relegate it to some sort of "background" section (which would > probably need to be organised into subsections of its own, but that > level of detail doesn't need to be on the front page). > > Paul From marius at pov.lt Thu Jan 16 15:34:06 2014 From: marius at pov.lt (Marius Gedminas) Date: Thu, 16 Jan 2014 16:34:06 +0200 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: <20140116143406.GA11068@fridge.pov.lt> On Thu, Jan 16, 2014 at 09:20:43AM -0500, Daniel Holth wrote: > Has anyone ever written a setup.py that was *not* copy-and-pasted from > somewhere else? Presumably the 1st setup.py had to have been written by hand somehow. Marius Gedminas -- Five words to strike fear into the heart of any software developer: 'How long will it take?' -- Sean McGrath From qwcode at gmail.com Thu Jan 16 16:02:36 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 07:02:36 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > > I still think that the best resource available would be a basic "best > practice" project template for a simple pure-python package with a few > tests. I started putting one together myself > (https://github.com/pfmoore/sampleproject). It's basically done, but > I'm not sure how it (or something like it) could be incorporated into > the docs. I like this idea. how about move it to the "pypa" org to make it more official, and then it can be mentioned here: https://python-packaging-user-guide.readthedocs.org/en/latest/packaging.html#getting-started-with-setuptools -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 16:14:01 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 15:14:01 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 15:02, Marcus Smith wrote: >> >> I still think that the best resource available would be a basic "best >> practice" project template for a simple pure-python package with a few >> tests. I started putting one together myself >> (https://github.com/pfmoore/sampleproject). It's basically done, but >> I'm not sure how it (or something like it) could be incorporated into >> the docs. > > > I like this idea. how about move it to the "pypa" org to make it more > official, and then it can be mentioned here: > https://python-packaging-user-guide.readthedocs.org/en/latest/packaging.html#getting-started-with-setuptools I'm happy to do that, if people think it's sufficiently representative of "best practice" (and if they don't, they can always improve it :-)) Paul. From p.f.moore at gmail.com Thu Jan 16 16:16:06 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 15:16:06 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 15:14, Paul Moore wrote: > I'm happy to do that, if people think it's sufficiently representative > of "best practice" (and if they don't, they can always improve it :-)) Done. From qwcode at gmail.com Thu Jan 16 17:42:45 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 08:42:45 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: Like you said, it's more elaborate than you might expect at first, but all the comments make it very clear. I like it. On Thu, Jan 16, 2014 at 7:14 AM, Paul Moore wrote: > On 16 January 2014 15:02, Marcus Smith wrote: > >> > >> I still think that the best resource available would be a basic "best > >> practice" project template for a simple pure-python package with a few > >> tests. I started putting one together myself > >> (https://github.com/pfmoore/sampleproject). It's basically done, but > >> I'm not sure how it (or something like it) could be incorporated into > >> the docs. > > > > > > I like this idea. how about move it to the "pypa" org to make it more > > official, and then it can be mentioned here: > > > https://python-packaging-user-guide.readthedocs.org/en/latest/packaging.html#getting-started-with-setuptools > > I'm happy to do that, if people think it's sufficiently representative > of "best practice" (and if they don't, they can always improve it :-)) > > Paul. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Jan 16 18:24:52 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 09:24:52 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > > On that note, the PUG index currently feels a little overwhelming The main index is 2 levels deep currently. Let me drop it to one level and see how that looks. I personally like a deep index as my main entry point to docs, but I can see it being a bit much. > the point of view of someone who just wants to get a job done. it's > got a lot of "Getting started with X" and "What is" sections, but > nothing like "Installing a package from PyPI" or "Starting a new > project". Maybe the index should be reorganised to be more > task-oriented, and less concerned with providing history and > background to the confusion that led us to where we are now? 2 things: 1) In my opinion, a big part of the PUG is about "deconfusing" people, and giving confidence that there's consensus on what to do now (with a justification), and there's plans to move forward. That's what many people need and will come to the PUG for. They already know the basics of pip and setuptools. 2) As for providing basic pip/setuptools tutorials, I've taken the position that the PUG shouldn't do it, but instead defer with links. The project docs themselves should have tutorials, so let's just do it once, and right, and link to it. Supposing you still thought it was a good idea for both to maintain tutorials, practically speaking, I think it will fail. One, or both will end up being not maintained or inconsistent with each other in the long run. I want to construct something that has a good chance of persisting and staying up to date. We have access to the 4 main projects, and can get PRs through. There seems to be an unspoken idea that we can't update the setuptools docs. We rely on setuptools heavily, and it has years of life left in it. The docs can be first rate, and consistent with what's currently going on. But in saying that, I'm still open to other TOC ideas. It is an open project. But with any idea, I'd like to see a full breakdown of the alternative TOC and how it covers everything we have now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Jan 16 18:46:27 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 09:46:27 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > > I don't have a clear picture of the split myself but it seems to me > that docs.python.org should be the master reference data for > *distutils*. That's somewhat screwed, though, as we're recommending > use of setuptools, and setuptools messes round so invasively with > distutils that the usefulness of pure distutils documentation is > limited :-( > > docs.python.org should provide a distutils reference, and drop the "Installing/Distributing Python Modules" guides in deference to the PUG. and setuptools (as an extension of distutils) should provide a *complete* index of commands/options that covers everything, not just what it changed or added, because it's too hard for users to keep track of what's pure vs modified. That's my plan in the refactor of the setuptools docs I'm working on. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 18:47:43 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 17:47:43 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 17:24, Marcus Smith wrote: >> On that note, the PUG index currently feels a little overwhelming > > The main index is 2 levels deep currently. Let me drop it to one level and > see how that looks. > I personally like a deep index as my main entry point to docs, but I can see > it being a bit much. It's not so much the length/depth, as the content. But maybe our views of what the PUG is for are different. See my next email for details. Paul From carl at oddbird.net Thu Jan 16 18:49:59 2014 From: carl at oddbird.net (Carl Meyer) Date: Thu, 16 Jan 2014 10:49:59 -0700 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: <52D81BC7.6010908@oddbird.net> Could we stop cross-posting this thread to both pypa-dev and distutils-sig? Seems like it belongs on pypa-dev to me, but I don't really care so long as it isn't cross-posted to both :-) Carl From qwcode at gmail.com Thu Jan 16 18:56:07 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 09:56:07 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: <52D81BC7.6010908@oddbird.net> References: <52D81BC7.6010908@oddbird.net> Message-ID: ok, let's do distutils-sig. true, it is a pypa project, but non-pypa people are already in this thread, and I imagine all pypa-dev people are on distutils-sig. On Thu, Jan 16, 2014 at 9:49 AM, Carl Meyer wrote: > Could we stop cross-posting this thread to both pypa-dev and > distutils-sig? Seems like it belongs on pypa-dev to me, but I don't > really care so long as it isn't cross-posted to both :-) > > Carl > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Jan 16 19:06:40 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 10:06:40 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > > docs.python.org should provide a distutils reference, and drop the > "Installing/Distributing Python Modules" guides in deference to the PUG. > to be clearer, the 2 "* Python Modules" guides have some good content, but I think it needs to be pushed into the distutils reference, not presented in a "guide" to installation and distribution. > and setuptools (as an extension of distutils) should provide a *complete* > index of commands/options that covers everything > this index could just link to the distutils reference docs (in the case where setuptools hasn't modifed anything), if those sections were filled out sufficiently -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 19:19:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 18:19:49 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 17:46, Marcus Smith wrote: >> >> I don't have a clear picture of the split myself but it seems to me >> that docs.python.org should be the master reference data for >> *distutils*. That's somewhat screwed, though, as we're recommending >> use of setuptools, and setuptools messes round so invasively with >> distutils that the usefulness of pure distutils documentation is >> limited :-( >> > > docs.python.org should provide a distutils reference, and drop the > "Installing/Distributing Python Modules" guides in deference to the PUG. > and setuptools (as an extension of distutils) should provide a *complete* > index of commands/options that covers everything, not just what it changed > or added, because it's too hard for users to keep track of what's pure vs > modified. That's my plan in the refactor of the setuptools docs I'm working > on. That makes sense. But I think it's a case of getting people to what they need as quickly as possible (and having the details available further on, if they need them). The pip documentation is great for this. Go to the front page, first thing you see is "Quickstart". First thing that hits your eye on that page is "pip install SomePackage==1.0". Maybe that's what the PUG should have - the very first index entry is "Quick Start". That points to a page that goes: Installing a package ============== Use `pip` to install from `PyPI`: pip install SomePackage Creating a project ============= Install `pip`, `setuptools` and `wheel`. [Question: In the system python? Introduce virtualenv here?] Create a project structure like this:: myproject myproject __init__.py setup.py setup.cfg tests README.txt `Here` is a basic setup.py you can copy and edit. ... etc. (Not too long, but cover the essentials). In the above, I'm using `...` to mark links to the detailed documentation. After that "Quick Start" link, the rest of the index can go into the details however people prefer. It's just that quick start that needs to be up front. Maybe I can do a PR for this - would that be useful? I'm lousy at writing documentation, but if I put a shell in that people can work from would that help? Paul From qwcode at gmail.com Thu Jan 16 19:39:01 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 10:39:01 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > > > That makes sense. But I think it's a case of getting people to what > they need as quickly as possible (and having the details available > further on, if they need them). The pip documentation is great for > this. Go to the front page, first thing you see is "Quickstart". First > thing that hits your eye on that page is "pip install > SomePackage==1.0". > > Maybe that's what the PUG should have - the very first index entry is > "Quick Start". That points to a page that goes: > given my previous arguments, if we have a "Quickstart", I would just have it contain links to the pip & setuptools quickstarts (and wheel too I guess). setuptools doesn't have a "quickstart" now, but that's part of what I'm doing in my setuptools PR, or you could work on adding one too (that includes reference to your sample project) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 16 20:19:40 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Jan 2014 19:19:40 +0000 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: On 16 January 2014 18:39, Marcus Smith wrote: > given my previous arguments, if we have a "Quickstart", I would just have it > contain links to the pip & setuptools quickstarts (and wheel too I guess). Yeah... Although I question the value of setting up a chain of 3 links to get someone to a pip one-liner. That seems to me to be taking DRY too far. Can't we at least put the basic "pip install" into the PUG? Here's an example of what I was meaning (please excuse the fact that I know nearly nothing about restructured text, and completely nothing about Sphinx...): https://github.com/pfmoore/python-packaging-user-guide/tree/quickstart Paul From qwcode at gmail.com Thu Jan 16 20:43:29 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 16 Jan 2014 11:43:29 -0800 Subject: [Distutils] "Python Packaging User Guide" has moved In-Reply-To: References: Message-ID: > Here's an example of what I was meaning (please excuse the fact that I > know nearly nothing about restructured text, and completely nothing > about Sphinx...): > https://github.com/pfmoore/python-packaging-user-guide/tree/quickstart well, ok, if we keep it really tight (like 2 pages or less), then I can bear it. : ) but I don't want it to grow and duplicate other parts of the guide (or other docs).... and it starts explaining and justifying things, and tries to be exhaustive and self-sufficient. it's a slippery slope. put some line at the top that makes it clear that it's the "super-lite no-explanation quickstart". -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jan 16 22:25:07 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 16 Jan 2014 21:25:07 +0000 (GMT) Subject: [Distutils] distlib 0.1.7 released on PyPI Message-ID: <1389907507.78484.YahooMailBasic@web172401.mail.ir2.yahoo.com> I've just released version 0.1.7 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Fixed issue #39: Changed to work in a situation where the user's home directory is unavailable. Fixed issue #40: Resources code manipulates paths differently for file system and zip resources. Script generation now uses correct logic to determine if running in a venv. Native strings are used in path computations to avoid errors on Windows with Unicode paths. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.7 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new From donald at stufft.io Fri Jan 17 02:46:41 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 16 Jan 2014 20:46:41 -0500 Subject: [Distutils] pip 1.5.1rc1 and virtualenv 1.11.1rc2 released Message-ID: <624FA3EE-3266-48F3-BB26-6F6AB9578F70@stufft.io> I?ve just released virtualenv 1.11.1rc2 and pip 1.5.1rc1, it fixes a number of issues with the latest releases of each one, but most importantly: * virtualenvs created with ?system-site-packages not getting pip or setuptools installed into it * Errors about setuptools being too old * Bug fixes that affected certain third party servers hosting packages with Content-Encoding headers * Have ?allow-unverified imply ?allow-external * Bug fixes that affected cases when a DNS name failed to resolve * Errors with older versions of Python 2.6, most notably from OSX system Pythons In order to try this out in an isolated manner simply: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.11.1rc2.tar.gz $ echo "b23fd95ef06effa2fde9f06fdb86ec38 1.11.1rc2.tar.gz" | md5sum -c 1.11.1rc2.tar.gz: OK $ tar zxf 1.11.1rc2.tar.gz $ python virtualenv-1.11.1rc2/virtualenv.py myVE $ myVE/bin/pip install SomePackage Report any bugs to the pip or virtualenv issues trackers located at: https://github.com/pypa/pip/issues https://github.com/pypa/virtualenv/issues Please do try these out and let us know of any issues! Thanks, ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hannes at ucsc.edu Fri Jan 17 03:52:46 2014 From: hannes at ucsc.edu (Hannes Schmidt) Date: Thu, 16 Jan 2014 18:52:46 -0800 Subject: [Distutils] Please do not remove dependency_links Message-ID: I read through the "Removing dependency_links" thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach. The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature. Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there. I understand the motivation for this change ? security ? but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two? [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes at ucsc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jan 17 13:44:27 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 17 Jan 2014 07:44:27 -0500 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: Message-ID: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> On Jan 16, 2014, at 9:52 PM, Hannes Schmidt wrote: > I read through the "Removing dependency_links" thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach. > > The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature. > > Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there. > > I understand the motivation for this change ? security ? but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two? > > [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html Pip 1.5 has already been released which turns dependency links off by default and adds a flag ?process-dependency-links and they are scheduled to be removed in 1.6. It?s possible we can ?undeprecate? them and leave them off by default however my question to you would be why you can?t use a requirements.txt file to handle installing things from bitbucket? Fundamentally a setup.py should not contain ?concrete? locations but instead should contain abstract names/version specifiers whereas a requirements.txt (or the pip invocation itself) takes those abstract names and makes them concrete by pairing them with an index. So to be more explicit, is there any reason why one of: 1) Create a requirements.txt and point to bitbucket in that and install your environment using ``pip install -r requirements.txt`` 2) Create tar balls of your packages and host them in a directory somewhere behind nginx and use ?find-links https://example.com/that-directory/ as your install location. Won?t work just as fine for you? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From teh.ivo at gmail.com Fri Jan 17 14:08:42 2014 From: teh.ivo at gmail.com (Matthew Iversen) Date: Sat, 18 Jan 2014 00:08:42 +1100 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: Message-ID: <52D92B5A.9050500@notevencode.com> On 17/01/14 13:52, Hannes Schmidt wrote: > I read through the "Removing dependency_links" thread [1] and I beg > you not follow through with the deprecation and removal > of dependency_links and to rethink your approach. > > The mentioned thread indicates that research was done to gauge the > popularity of the dependency_links in publicly hosted Python projects. > That approach is fundamentally flawed: Publicly hosted projects are > much more likely to also be available on PyPI than private, > closed-source projects. Consequently, their dependencies are also more > likely to be hosted on PyPI as well. Because of that, they are much > less likely to rely on the dependency_links feature. > > Another misconception seem to be that dependency_links is > predominantly used for installing patched or customized versions of > dependencies hosted on PyPI. I'm pretty sure the predominant use case > for dependency_links is with projects that are hosted privately, e.g. > for an organization's internal use. I represent such an organization > and removing dependency_links would impact us negatively. We host a > set of internal projects and their dependencies on Bitbucket and we > rely on dependency_links to install them directly from there. > > I understand the motivation for this change -- security -- but there > must be smarter way to handle it. Could we fallback to > dependency_links if a PyPI lookup isn't successful? Could we restrict > dependency_links to links that share a prefix with the link from which > the package is currently being installed? A combination of the two? > > [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html > > -- > Hannes Schmidt > Software Application Developer > Data Migration Engineer > Cancer Genomics Hub > University of California, Santa Cruz > > (206) 696-2316 (cell) > hannes at ucsc.edu > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Are you aware that you can also provide a solid infrastructure for a "proprietary environment of packages" by hosting your own private pypi mirror? There are quite a few projects that enable this, such as https://pypi.python.org/pypi/pypiserver https://pypi.python.org/pypi/mypypi https://github.com/steiza/simplepypi http://djangopypi2.readthedocs.org/en/latest/ You can use the -i flag, or a .pypirc config file to tell pip to look at your own private index instead of the official one; or direct it to use the official pypi only after looking at your private pypi. You can also host a network-available folder of wheels for pip to find, or simply a http accessible folder of packages as Donald suggested. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Fri Jan 17 16:08:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 18 Jan 2014 01:08:39 +1000 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: <52D92B5A.9050500@notevencode.com> References: <52D92B5A.9050500@notevencode.com> Message-ID: On 17 Jan 2014 23:22, "Matthew Iversen" wrote: > > On 17/01/14 13:52, Hannes Schmidt wrote: >> >> I read through the "Removing dependency_links" thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach. >> >> The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature. >> >> Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there. >> >> I understand the motivation for this change ? security ? but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two? >> >> [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html >> >> -- >> Hannes Schmidt >> Software Application Developer >> Data Migration Engineer >> Cancer Genomics Hub >> University of California, Santa Cruz >> >> (206) 696-2316 (cell) >> hannes at ucsc.edu >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > > Are you aware that you can also provide a solid infrastructure for a "proprietary environment of packages" by hosting your own private pypi mirror? > > There are quite a few projects that enable this, such as > > https://pypi.python.org/pypi/pypiserver > > https://pypi.python.org/pypi/mypypi > > https://github.com/steiza/simplepypi > > http://djangopypi2.readthedocs.org/en/latest/ > > You can use the -i flag, or a .pypirc config file to tell pip to look at your own private index instead of the official one; or direct it to use the official pypi only after looking at your private pypi. > > You can also host a network-available folder of wheels for pip to find, or simply a http accessible folder of packages as Donald suggested. Private PyPI mirrors and internal hosting options would be a suitable advanced topic for the PUG. I don't recall whether or not Marcus already added a placeholder for it, though. Cheers, Nick. > > -- > Matt Iversen > PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hannes at ucsc.edu Fri Jan 17 19:45:37 2014 From: hannes at ucsc.edu (Hannes Schmidt) Date: Fri, 17 Jan 2014 10:45:37 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: Thank you for your suggestions, Donald. They are both feasible but they feel like workarounds to me. Manually listing transitive dependencies seems like a step backwards to me. Isn't the whole point of setuptools/distutils that each component manages its own list of dependencies? How will this scale to dozens of transitive dependencies and busy development teams where dependencies are revved frequently. I really liked what Maven did for Java and therefore liked what pip is trying to do for Python. I haven't used --find-links yet so I may be wrong but tarring up packages and serving them by a web server is additional work that I'd rather avoid. It just creates yet another copy of the package and requires maintaining an additional server component. I loved the fact that I could point pip directly at my source repos for both my top-level projects as well as their dependencies. That seemed like a perfect fit for an interpreted language like Python where packages are distributed in source form. Removing dependency_links makes that feature useless to me because it doesn't extend to dependencies, only top level components. On Fri, Jan 17, 2014 at 4:44 AM, Donald Stufft wrote: > > On Jan 16, 2014, at 9:52 PM, Hannes Schmidt wrote: > > I read through the "Removing dependency_links" thread [1] and I beg you > not follow through with the deprecation and removal of dependency_links and > to rethink your approach. > > The mentioned thread indicates that research was done to gauge the > popularity of the dependency_links in publicly hosted Python projects. That > approach is fundamentally flawed: Publicly hosted projects are much more > likely to also be available on PyPI than private, closed-source projects. > Consequently, their dependencies are also more likely to be hosted on PyPI > as well. Because of that, they are much less likely to rely on the > dependency_links feature. > > Another misconception seem to be that dependency_links is predominantly > used for installing patched or customized versions of dependencies hosted > on PyPI. I'm pretty sure the predominant use case for dependency_links is > with projects that are hosted privately, e.g. for an organization's > internal use. I represent such an organization and removing > dependency_links would impact us negatively. We host a set of internal > projects and their dependencies on Bitbucket and we rely on > dependency_links to install them directly from there. > > I understand the motivation for this change ? security ? but there must be > smarter way to handle it. Could we fallback to dependency_links if a PyPI > lookup isn't successful? Could we restrict dependency_links to links that > share a prefix with the link from which the package is currently being > installed? A combination of the two? > > [1]: > https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html > > > > Pip 1.5 has already been released which turns dependency links off by > default and adds a flag ?process-dependency-links and they are > scheduled to be removed in 1.6. It?s possible we can ?undeprecate? them > and leave them off by default however my question to you would be why > you can?t use a requirements.txt file to handle installing things from > bitbucket? > > Fundamentally a setup.py should not contain ?concrete? locations but > instead > should contain abstract names/version specifiers whereas a requirements.txt > (or the pip invocation itself) takes those abstract names and makes them > concrete by pairing them with an index. > > So to be more explicit, is there any reason why one of: > > 1) Create a requirements.txt and point to bitbucket in that and install > your > environment using ``pip install -r requirements.txt`` > 2) Create tar balls of your packages and host them in a directory somewhere > behind nginx and use ?find-links https://example.com/that-directory/as > your install location. > > Won?t work just as fine for you? > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes at ucsc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Jan 17 20:12:28 2014 From: dholth at gmail.com (Daniel Holth) Date: Fri, 17 Jan 2014 14:12:28 -0500 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On Fri, Jan 17, 2014 at 1:45 PM, Hannes Schmidt wrote: > Thank you for your suggestions, Donald. They are both feasible but they feel > like workarounds to me. > > Manually listing transitive dependencies seems like a step backwards to me. > Isn't the whole point of setuptools/distutils that each component manages > its own list of dependencies? How will this scale to dozens of transitive > dependencies and busy development teams where dependencies are revved > frequently. I really liked what Maven did for Java and therefore liked what > pip is trying to do for Python. > > I haven't used --find-links yet so I may be wrong but tarring up packages > and serving them by a web server is additional work that I'd rather avoid. > It just creates yet another copy of the package and requires maintaining an > additional server component. > > I loved the fact that I could point pip directly at my source repos for both > my top-level projects as well as their dependencies. That seemed like a > perfect fit for an interpreted language like Python where packages are > distributed in source form. Removing dependency_links makes that feature > useless to me because it doesn't extend to dependencies, only top level > components. IIUC a dependency link is the same as passing --find-links=x for that URL. I understand this doesn't exactly solve your problem. Your internal projects are in private bitbucket repositories? From ncoghlan at gmail.com Sat Jan 18 01:09:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 18 Jan 2014 10:09:59 +1000 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On 18 Jan 2014 04:45, "Hannes Schmidt" wrote: > > Thank you for your suggestions, Donald. They are both feasible but they feel like workarounds to me. > > Manually listing transitive dependencies seems like a step backwards to me. Isn't the whole point of setuptools/distutils that each component manages its own list of dependencies? How will this scale to dozens of transitive dependencies and busy development teams where dependencies are revved frequently. I really liked what Maven did for Java and therefore liked what pip is trying to do for Python. Hannes, this is the role we consider to be best served by running a private index server and using the full dependency resolution mechanism. You may want to review PEP 426 to see the intended mode of operation. However, the question at this point is whether there is a legitimate model of use where dependency links (or an equivalent) are used to create a more concrete dependency tree in a closed environment where the software developer/publisher and integrator/user are the *same* organisation. PEP 426 focuses on the case where those are different organisations (since that case poses the greatest security risk, allowing publishers to compromise the installations of integrators and users, beyond just running the software they provide). While it does include the "meta requirements" field to handle cases of (effective) dependency bundling rather than normal versioning, that model is deliberately more restrictive than dependency links. It seems to me that overall, it's much, much safer to simply avoid supporting this degree of transitive trust directly in the core installation toolchain. A more appropriate solution to handling the trusted internal distribution use case may be a dependency scanner that processes the dependency links metadata to generate a suitable requirements.txt file. >From pip's perspective, trusting that generated file is the same as trusting any other explicitly specified set of packages, while it also provides an opportunity for the resolved dependency links to be audited before trusting them. Unfortunately, we can't actually do that easily in a secure fashion, because obtaining the metadata currently requires executing an untrusted setup.py. So trusting dependency links "just" to generate a requirements.txt file is almost as bad as trusting them for installing software. The main advantage is that you can always generate the dependency list as an unprivileged user or in a sandbox, even if operating in an untrusted environment. Installation, by contrast, is often going to involve elevated privileges. Regardless, perhaps there's a case to be made that such a requirements.txt generator, with opt-in dependency links support, should remain available, with the feature being removed entirely only for actual package installation. That approach is still far more secure by default than the status quo, strongly encourages the use of a private index to better align with the upstream distribution model, but restores support for users like Hannes that are currently relying on dependency links as a dependency resolution solution that doesn't require additional server components in a trusted environment. > I haven't used --find-links yet so I may be wrong but tarring up packages and serving them by a web server is additional work that I'd rather avoid. It just creates yet another copy of the package and requires maintaining an additional server component. > > I loved the fact that I could point pip directly at my source repos for both my top-level projects as well as their dependencies. That seemed like a perfect fit for an interpreted language like Python where packages are distributed in source form. In a trusted environment, it can be, but on the internet, it is intolerably insecure. However, making it easy to generate a transitive requirements.txt for a trusted environment may be a usage model we can support without leaving upstream distribution and usage open to compromise. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jan 18 01:29:04 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 17 Jan 2014 19:29:04 -0500 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On Jan 17, 2014, at 7:09 PM, Nick Coghlan wrote: > > On 18 Jan 2014 04:45, "Hannes Schmidt" wrote: > > > > Thank you for your suggestions, Donald. They are both feasible but they feel like workarounds to me. > > > > Manually listing transitive dependencies seems like a step backwards to me. Isn't the whole point of setuptools/distutils that each component manages its own list of dependencies? How will this scale to dozens of transitive dependencies and busy development teams where dependencies are revved frequently. I really liked what Maven did for Java and therefore liked what pip is trying to do for Python. > > I?ve never used Maven, but doesn?t it require you to run a Maven repository or otherwise point it at each individual location if they aren?t in a Maven repo? > Hannes, this is the role we consider to be best served by running a private index server and using the full dependency resolution mechanism. You may want to review PEP 426 to see the intended mode of operation. > > However, the question at this point is whether there is a legitimate model of use where dependency links (or an equivalent) are used to create a more concrete dependency tree in a closed environment where the software developer/publisher and integrator/user are the *same* organisation. > > PEP 426 focuses on the case where those are different organisations (since that case poses the greatest security risk, allowing publishers to compromise the installations of integrators and users, beyond just running the software they provide). While it does include the "meta requirements" field to handle cases of (effective) dependency bundling rather than normal versioning, that model is deliberately more restrictive than dependency links. > > It seems to me that overall, it's much, much safer to simply avoid supporting this degree of transitive trust directly in the core installation toolchain. A more appropriate solution to handling the trusted internal distribution use case may be a dependency scanner that processes the dependency links metadata to generate a suitable requirements.txt file. From pip's perspective, trusting that generated file is the same as trusting any other explicitly specified set of packages, while it also provides an opportunity for the resolved dependency links to be audited before trusting them. > > Unfortunately, we can't actually do that easily in a secure fashion, because obtaining the metadata currently requires executing an untrusted setup.py. So trusting dependency links "just" to generate a requirements.txt file is almost as bad as trusting them for installing software. The main advantage is that you can always generate the dependency list as an unprivileged user or in a sandbox, even if operating in an untrusted environment. Installation, by contrast, is often going to involve elevated privileges. > > Regardless, perhaps there's a case to be made that such a requirements.txt generator, with opt-in dependency links support, should remain available, with the feature being removed entirely only for actual package installation. > > That approach is still far more secure by default than the status quo, strongly encourages the use of a private index to better align with the upstream distribution model, but restores support for users like Hannes that are currently relying on dependency links as a dependency resolution solution that doesn't require additional server components in a trusted environment. > > > I haven't used --find-links yet so I may be wrong but tarring up packages and serving them by a web server is additional work that I'd rather avoid. It just creates yet another copy of the package and requires maintaining an additional server component. > > > > I loved the fact that I could point pip directly at my source repos for both my top-level projects as well as their dependencies. That seemed like a perfect fit for an interpreted language like Python where packages are distributed in source form. > > In a trusted environment, it can be, but on the internet, it is intolerably insecure. > > However, making it easy to generate a transitive requirements.txt for a trusted environment may be a usage model we can support without leaving upstream distribution and usage open to compromise. > > Regards, > Nick. Let me just be explicit that I totally get the fact that this is breaking your workflow. Nobody *ever* likes being told that the way they are doing something is no longer going to be supported. Processing dependency links by default is a security risk however in your constrained use case it?s probably Ok, but only because you have a tightly constrained use case. One thing that pip *could* do is leave the ?process-dependency-links flag in place instead of removing it as per the deprecation cycle, however I think that ultimately this will be a disservice to users. The new formats that we?re designing and implementing are not going to support things like dependency links so while this would allow things to continue for awhile as we further improve the system it will likely be increasingly difficult to maintain the functionality of dependency links. So again I?m really really sorry that this is affecting you, but I can?t see a good way to keep this concept around that doesn?t hurt the security for the other use case. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sat Jan 18 02:21:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 18 Jan 2014 11:21:55 +1000 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On 18 Jan 2014 10:29, "Donald Stufft" wrote: >> >> > I haven't used --find-links yet so I may be wrong but tarring up packages and serving them by a web server is additional work that I'd rather avoid. It just creates yet another copy of the package and requires maintaining an additional server component. >> > >> > I loved the fact that I could point pip directly at my source repos for both my top-level projects as well as their dependencies. That seemed like a perfect fit for an interpreted language like Python where packages are distributed in source form. >> >> In a trusted environment, it can be, but on the internet, it is intolerably insecure. >> >> However, making it easy to generate a transitive requirements.txt for a trusted environment may be a usage model we can support without leaving upstream distribution and usage open to compromise. >> >> Regards, >> Nick. > > Let me just be explicit that I totally get the fact that this is breaking your workflow. Nobody *ever* likes being told that the way they are doing something is no longer going to be supported. Processing dependency links by default is a security risk however in your constrained use case it?s probably Ok, but only because you have a tightly constrained use case. > > One thing that pip *could* do is leave the ?process-dependency-links flag in place instead of removing it as per the deprecation cycle, however I think that ultimately this will be a disservice to users. The new formats that we?re designing and implementing are not going to support things like dependency links so while this would allow things to continue for awhile as we further improve the system it will likely be increasingly difficult to maintain the functionality of dependency links. We don't know that for sure - dependency links is entirely supportable even under PEP 426 through an appropriate metadata extension, it would just require someone to write the code to generate and process it. Alternatively, affected users could just stay with setuptools style metadata for internal use. > So again I?m really really sorry that this is affecting you, but I can?t see a good way to keep this concept around that doesn?t hurt the security for the other use case. That's where I see a separate "scan-dependency-links" tool potentially fitting in. The concept would still be completely gone from the core installation toolchain in pip so it isn't available by default, but users in a situation like Hannes could use it to generate a suitable requirements.txt file (perhaps even dynamically) and feed *that* to pip. PyPI itself, however, could still change to block uploading of *new* packages that define dependency links (similar to what we did for external link scanning). Imposing backwards incompatible changes like this one, however well justified, always brings with it a certain responsibility to help users in managing the transition. In this case, I suspect a *separate* link scanning tool (which we can put as many security warnings on as you like) that generates a requirements.txt file may be a more appropriate approach than just telling users to use built packages on a private HTTP or PyPI server, since spinning up and maintaining new server infrastructure is often a much more challenging prospect in user environments than installing and using a new client tool. Cheers, Nick. > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jan 18 02:58:27 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 17 Jan 2014 17:58:27 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: > > > I haven't used --find-links yet so I may be wrong but tarring up packages > and serving them by a web server is additional work that I'd rather avoid. > It just creates yet another copy of the package and requires maintaining an > additional server component. > --find-links can point to a local directory of packages, if that helps you. also, in case it's not clear, you don't manually tar up packages, you run "python setup.py sdist" to generate them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jan 18 03:25:26 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 17 Jan 2014 18:25:26 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: > > > Manually listing transitive dependencies seems like a step backwards to > me. Isn't the whole point of setuptools/distutils that each component > manages its own list of dependencies? How will this scale to dozens of > transitive dependencies and busy development teams where dependencies are > revved frequently. I really liked what Maven did for Java and therefore > liked what pip is trying to do for Python. > just in case it's not clear, if you package everything and dump the tars into a directory, then there's no listing everything, just maintain the install_requires listings for each project. it's just this: pip install --find-links=/local/packages my-top-level-app==X.Y but if you don't package, and pip stops processing dependency links, then yes, you will end up putting all the vcs urls into one requirements file per top-level app. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hannes at ucsc.edu Sat Jan 18 03:25:36 2014 From: hannes at ucsc.edu (Hannes Schmidt) Date: Fri, 17 Jan 2014 18:25:36 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On Fri, Jan 17, 2014 at 5:58 PM, Marcus Smith wrote: > >> I haven't used --find-links yet so I may be wrong but tarring up packages >> and serving them by a web server is additional work that I'd rather avoid. >> It just creates yet another copy of the package and requires maintaining an >> additional server component. >> > > --find-links can point to a local directory of packages, if that helps you. > also, in case it's not clear, you don't manually tar up packages, you run > "python setup.py sdist" to generate them. > Sure, scratch the "manual" part. But don't I'd still have to run this for every package after every commit? I really like the fact that I can install bleeding edge using @master (or @default) as well as a tag, e.g. @1.5 or a stable branch, e.g. @stable. -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes at ucsc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From hannes at ucsc.edu Sat Jan 18 03:21:34 2014 From: hannes at ucsc.edu (Hannes Schmidt) Date: Fri, 17 Jan 2014 18:21:34 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On Fri, Jan 17, 2014 at 11:12 AM, Daniel Holth wrote: > On Fri, Jan 17, 2014 at 1:45 PM, Hannes Schmidt wrote: > > Thank you for your suggestions, Donald. They are both feasible but they > feel > > like workarounds to me. > > > > Manually listing transitive dependencies seems like a step backwards to > me. > > Isn't the whole point of setuptools/distutils that each component manages > > its own list of dependencies? How will this scale to dozens of transitive > > dependencies and busy development teams where dependencies are revved > > frequently. I really liked what Maven did for Java and therefore liked > what > > pip is trying to do for Python. > > > > I haven't used --find-links yet so I may be wrong but tarring up packages > > and serving them by a web server is additional work that I'd rather > avoid. > > It just creates yet another copy of the package and requires maintaining > an > > additional server component. > > > > I loved the fact that I could point pip directly at my source repos for > both > > my top-level projects as well as their dependencies. That seemed like a > > perfect fit for an interpreted language like Python where packages are > > distributed in source form. Removing dependency_links makes that feature > > useless to me because it doesn't extend to dependencies, only top level > > components. > > IIUC a dependency link is the same as passing --find-links=x for that > URL. I understand this doesn't exactly solve your problem. > I don't know. How do I translate pip install hg+ssh://hg at bitbucket.org/cghub/cghub-cloud-agent at default with cghub-cloud-agent's setup.py being setup( name="cghub-cloud-agent", version="1.0.dev1", install_requires=[ 'cghub-cloud-lib>=1.0.dev1', 'cghub-python-lib>=1.4.dev1', ... ], dependency_links=[ 'hg+ssh:// hg at bitbucket.org/cghub/cghub-python-lib at default#egg=cghub-python-lib-1.4.dev1 ', 'hg+ssh:// hg at bitbucket.org/cghub/cghub-cloud-lib at default#egg=cghub-cloud-lib-1.0.dev1' ], ) into a use of --find-links? > > Your internal projects are in private bitbucket repositories? > Yep -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes at ucsc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jan 18 03:33:15 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 17 Jan 2014 18:33:15 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: > > > I don't know. How do I translate > > pip install hg+ssh://hg at bitbucket.org/cghub/cghub-cloud-agent at default > > with cghub-cloud-agent's setup.py being > > setup( > name="cghub-cloud-agent", > version="1.0.dev1", > install_requires=[ > 'cghub-cloud-lib>=1.0.dev1', > 'cghub-python-lib>=1.4.dev1', ... > ], > dependency_links=[ > 'hg+ssh:// > hg at bitbucket.org/cghub/cghub-python-lib at default#egg=cghub-python-lib-1.4.dev1 > ', > 'hg+ssh:// > hg at bitbucket.org/cghub/cghub-cloud-lib at default#egg=cghub-cloud-lib-1.0.dev1 > ' > ], > ) > > into a use of --find-links? > --find-links assumes packaging. if you don't package, it doesn't help you. you're left putting all the vcs urls into a requirements file for your top-level app. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hannes at ucsc.edu Sat Jan 18 03:34:26 2014 From: hannes at ucsc.edu (Hannes Schmidt) Date: Fri, 17 Jan 2014 18:34:26 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: On Fri, Jan 17, 2014 at 4:29 PM, Donald Stufft wrote: > > On Jan 17, 2014, at 7:09 PM, Nick Coghlan wrote: > > > On 18 Jan 2014 04:45, "Hannes Schmidt" wrote: > > > > Thank you for your suggestions, Donald. They are both feasible but they > feel like workarounds to me. > > > > Manually listing transitive dependencies seems like a step backwards to > me. Isn't the whole point of setuptools/distutils that each component > manages its own list of dependencies? How will this scale to dozens of > transitive dependencies and busy development teams where dependencies are > revved frequently. I really liked what Maven did for Java and therefore > liked what pip is trying to do for Python. > > > I?ve never used Maven, but doesn?t it require you to run a Maven > repository or otherwise point it at each individual location if they aren?t > in a Maven repo? > Yes. But Java is a compiled language so you *need* to host the compiled artifacts somewhere. Python's interpreted nature is what makes the incredible convenience of installing from sources at hg:// and git:// URLs possible. > > Hannes, this is the role we consider to be best served by running a > private index server and using the full dependency resolution mechanism. > You may want to review PEP 426 to see the intended mode of operation. > > However, the question at this point is whether there is a legitimate model > of use where dependency links (or an equivalent) are used to create a more > concrete dependency tree in a closed environment where the software > developer/publisher and integrator/user are the *same* organisation. > > PEP 426 focuses on the case where those are different organisations (since > that case poses the greatest security risk, allowing publishers to > compromise the installations of integrators and users, beyond just running > the software they provide). While it does include the "meta requirements" > field to handle cases of (effective) dependency bundling rather than normal > versioning, that model is deliberately more restrictive than dependency > links. > > It seems to me that overall, it's much, much safer to simply avoid > supporting this degree of transitive trust directly in the core > installation toolchain. A more appropriate solution to handling the trusted > internal distribution use case may be a dependency scanner that processes > the dependency links metadata to generate a suitable requirements.txt file. > From pip's perspective, trusting that generated file is the same as > trusting any other explicitly specified set of packages, while it also > provides an opportunity for the resolved dependency links to be audited > before trusting them. > > Unfortunately, we can't actually do that easily in a secure fashion, > because obtaining the metadata currently requires executing an untrusted > setup.py. So trusting dependency links "just" to generate a > requirements.txt file is almost as bad as trusting them for installing > software. The main advantage is that you can always generate the dependency > list as an unprivileged user or in a sandbox, even if operating in an > untrusted environment. Installation, by contrast, is often going to involve > elevated privileges. > > Regardless, perhaps there's a case to be made that such a requirements.txt > generator, with opt-in dependency links support, should remain available, > with the feature being removed entirely only for actual package > installation. > > That approach is still far more secure by default than the status quo, > strongly encourages the use of a private index to better align with the > upstream distribution model, but restores support for users like Hannes > that are currently relying on dependency links as a dependency resolution > solution that doesn't require additional server components in a trusted > environment. > > > I haven't used --find-links yet so I may be wrong but tarring up > packages and serving them by a web server is additional work that I'd > rather avoid. It just creates yet another copy of the package and requires > maintaining an additional server component. > > > > I loved the fact that I could point pip directly at my source repos for > both my top-level projects as well as their dependencies. That seemed like > a perfect fit for an interpreted language like Python where packages are > distributed in source form. > > In a trusted environment, it can be, but on the internet, it is > intolerably insecure. > > However, making it easy to generate a transitive requirements.txt for a > trusted environment may be a usage model we can support without leaving > upstream distribution and usage open to compromise. > > Regards, > Nick. > > Let me just be explicit that I totally get the fact that this is breaking > your workflow. Nobody *ever* likes being told that the way they are doing > something is no longer going to be supported. Processing dependency links > by default is a security risk however in your constrained use case it?s > probably Ok, but only because you have a tightly constrained use case. > > One thing that pip *could* do is leave the ?process-dependency-links flag > in place instead of removing it as per the deprecation cycle, however I > think that ultimately this will be a disservice to users. The new formats > that we?re designing and implementing are not going to support things like > dependency links so while this would allow things to continue for awhile as > we further improve the system it will likely be increasingly difficult to > maintain the functionality of dependency links. > > So again I?m really really sorry that this is affecting you, but I can?t > see a good way to keep this concept around that doesn?t hurt the security > for the other use case. > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes at ucsc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jan 18 03:46:26 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 17 Jan 2014 18:46:26 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: > > > Imposing backwards incompatible changes like this one, however well > justified, always brings with it a certain responsibility to help users in > managing the transition. In this case, I suspect a *separate* link scanning > tool (which we can put as many security warnings on as you like) that > generates a requirements.txt file may be a more appropriate approach than > just telling users to use built packages on a private HTTP or PyPI server, > since spinning up and maintaining new server infrastructure is often a much > more challenging prospect in user environments than installing and using a > new client tool. > so, instead of pip install git+https://myrepo at master#egg=toplevelapp it's this: deplinks_scantool git+https://myrepo at master#egg=toplevelapp > requirements.txt pip install -r requirements.txt the scan tool step is pretty hefty in that it's downloading and running egg_info, but yes, I can see that being easier for some users than dealing with packaging -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Sat Jan 18 02:06:23 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 17 Jan 2014 17:06:23 -0800 Subject: [Distutils] setup.py issue - some files are included as intended, but one is not Message-ID: Hi all. Yes, it does currently require make. I'm very comfortable with make, but I'm open to redoing the make rules in setup.py if that'll help get this problem solved. It'd probably be less puzzling to Python people anyway. I kept increasing the version number as I tried different ways of getting setup.py to do what I wanted. That's why the version numbers don't appear to match up. If there's a better way of testing setup.py's, please let me know! Thanks! >hello, I checked out your project from svn. >looks like it requires some make commands, before it can be packaged? >I'm getting tripped up on that. >also confused in that your src tree is at 1.30, but the tar you mention is >1.35, and then your install dir was 1.31. >maybe get your src tree up to date with where your working copy, and post >any make commands that need to be run. > >On Tue, Jan 14, 2014 at 1:32 PM, Ethan Furman wrote: > >> On 01/14/2014 01:26 PM, Dan Stromberg wrote: >> >>> On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg >>> wrote: >>> >>>> Hi folks. >>>> >>>> I have a setup.py problem that's driving me nuts. >>>> >>> >>> Anyone? I've received 0 responses. >>> >> >> I have no answer, but forwarding to Distutils (hopefully it's an >> appropriate topic ;) >> >> >> I have a treap.py file that tries to "import * from pyx_treap.so", and >>>> failing that, it'll "import * from py_treap.py" (sans extensions of >>>> course). Naturally, all 3 of these should be included - although in >>>> the case of pyx_treap.so, it probably should be a .c that's included. >>>> >>>> It is also intended to include nest.py, which has a simple form. >>>> >>>> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >>>> seem to get py_treap.py to be included for some reason. >>>> >>>> I get no errors during "python setup.py sdist upload", but upon >>>> "python $(which pip) install treap", I get: >>>> $ sudo /usr/bin/python $(which pip) install treap >>>> Downloading/unpacking treap >>>> Running setup.py egg_info for package treap >>>> >>>> file py_treap.py (for module py_treap) not found >>>> Installing collected packages: treap >>>> Running setup.py install for treap >>>> file py_treap.py (for module py_treap) not found >>>> file py_treap.py (for module py_treap) not found >>>> >>>> file py_treap.py (for module py_treap) not found >>>> file py_treap.py (for module py_treap) not found >>>> Successfully installed treap >>>> Cleaning up... >>>> >>>> And it's not successfully installed - py_treap.py is missing. The pyx >>>> code does its job, so the problem is masked, other than the messages >>>> above, and the absence of py_treap.py from >>>> /usr/local/lib/python2.7/dist-packages >>>> >>>> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >>>> least getting packaged up that much. It's not listed in >>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >>>> , but I can see it in >>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >>>> . >>>> >>>> My setup.py is at: >>>> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >>>> >>>> I've tried that setup.py and several variations on that theme, but >>>> none seem to include py_treap.py . >>>> >>>> Please make some suggestions? How can I get py_treap.py included in >>>> the pip install? >>>> >>>> Thanks! >>>> >>> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >-------------- next part -------------- >An HTML attachment was scrubbed... >URL: From hannes at ucsc.edu Sat Jan 18 04:40:09 2014 From: hannes at ucsc.edu (Hannes Schmidt) Date: Fri, 17 Jan 2014 19:40:09 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: Drawing from my Maven experience, there is no direct equivalent to dependency_links in Maven. In pom.xml (the equivalent of setup.py) one specifies dependencies and version constraints. This is the "what". In ~/.m2/settings.xml one specifies the repositories to be scanned for those dependencies. That's the "where". You *can* specify repositories in pom.xml, too, but I always considered that an artificial use case, and if I remember correctly, there were controversies about that. I do actually think that dependency_links meddles the "what" and the "where" concerns which is one reason I sort of agree with its removal. I'm just not convinced it's a good idea to remove it without offering an equally convenient alternative. By convenient I mean that it facilitates installation directly from source repository without the need of going through a separate and mostly redundant (due to Python being an interpreted language) package repository. I'd be completely happy with specifying the mapping from "what" to "where" in a configuration file, outside of setup.py. That would be even more convenient since it would be more DRY than having dependency_links in every package's setup.py. The mapping could be Python code consisting of lines like: map( 'foo<=1.5', git+https://myrepo/foo at master#egg=1.5 ) or with patterns map_re( '(bar-.*)<=(.*)', git+https://myrepo/$1 at master#egg=$1-$2 ) This is probably a bit naive, especially with respect to how it translates the version constraint into a concrete version. On Fri, Jan 17, 2014 at 6:46 PM, Marcus Smith wrote: > >> Imposing backwards incompatible changes like this one, however well >> justified, always brings with it a certain responsibility to help users in >> managing the transition. In this case, I suspect a *separate* link scanning >> tool (which we can put as many security warnings on as you like) that >> generates a requirements.txt file may be a more appropriate approach than >> just telling users to use built packages on a private HTTP or PyPI server, >> since spinning up and maintaining new server infrastructure is often a much >> more challenging prospect in user environments than installing and using a >> new client tool. >> > > so, instead of > > pip install git+https://myrepo at master#egg=toplevelapp > > it's this: > > deplinks_scantool git+https://myrepo at master#egg=toplevelapp > > requirements.txt > pip install -r requirements.txt > > > the scan tool step is pretty hefty in that it's downloading and running > egg_info, but yes, I can see that being easier for some users than dealing > with packaging > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes at ucsc.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Jan 18 05:58:17 2014 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 17 Jan 2014 20:58:17 -0800 Subject: [Distutils] Please do not remove dependency_links In-Reply-To: References: <0F207F06-BCA0-472B-8A23-23D5741C168C@stufft.io> Message-ID: > > > I'd be completely happy with specifying the mapping from "what" to "where" > in a configuration file, outside of setup.py. That would be even more > convenient since it would be more DRY than having dependency_links in every > package's setup.py. The mapping could be Python code consisting of lines > like: > > map( 'foo<=1.5', git+https://myrepo/foo at master#egg=1.5 ) > > or with patterns > > map_re( '(bar-.*)<=(.*)', git+https://myrepo/$1 at master#egg=$1-$2 ) > > This is probably a bit naive, especially with respect to how it translates > the version constraint into a concrete version. > I think that's an interesting idea, although I think your $2 would need to alter the @ part as well, which is the tag or branch your cloning If the mappings could be constructed as option values, then the mappings could just live in the requirements file. e.g. a mapping for "myproject" that has myproject==version map to a tag in the clone equal to "version" --vcs-req-map myproject:git+ https://myrepo/myproject@${version}#egg=myproject -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf at systemexit.de Mon Jan 20 23:25:56 2014 From: ralf at systemexit.de (Ralf Schmitt) Date: Mon, 20 Jan 2014 23:25:56 +0100 Subject: [Distutils] [ANN] pypiserver 1.1.5 - minimal private pypi server Message-ID: <87ha8y70jv.fsf@myhost.lan> Hi, I've just uploaded pypiserver 1.1.5 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. https://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver I didn't announce the previous version, so here's the changelog for the last two versions: 1.1.5 (2014-01-20) ------------------ - only list devpi-server and proxypypi as alternatives - fix wheel file handling for certain wheels - serve wheel files as application/octet-stream - make pypiserver executable from wheel file - build universal wheel - remove scripts subdirectory - add --index-url cli parameter 1.1.4 (2014-01-03) ------------------ - make pypiserver compatible with pip 1.5 (https://github.com/schmir/pypiserver/pull/42) -- Cheers Ralf From donald at stufft.io Tue Jan 21 05:07:29 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 20 Jan 2014 23:07:29 -0500 Subject: [Distutils] Pip 1.5.1 and Virtualenv 1.11.1 Released Message-ID: <54B77387-E95F-4450-A039-62364C65CBE9@stufft.io> I?ve just released pip 1.5.1 and virtualenv 1.11.1 which fixes a number of issues with 1.5 and 1.11 respectively. It is recommend that all users upgrade to these releases. 1.5.1 (2014-01-20) ------------------ * pip now only requires setuptools (any setuptools, not a certain version) when installing distributions from src (i.e. not from wheel). (Pull #1434). * `get-pip.py` now installs setuptools, when it's not already installed (Pull #1475) * Don't decode downloaded files that have a ``Content-Encoding`` header. (Pull #1435) * Fix to correctly parse wheel filenames with single digit versions. (Pull #1445) * If `--allow-unverified` is used assume it also means `--allow-external`. (Pull #1457) 1.11.1 (2014-01-20) ~~~~~~~~~~~~~~~~~~~ * Fixed an issue where pip and setuptools were not getting installed when using the ``--system-site-packages`` flag. * Updated setuptools to fix an issue when installed with easy_install * Fixed an issue with Python 3.4 and sys.stdout encoding being set to ascii * Upgraded pip to v1.5.1 * Upgraded setuptools to v2.1 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue Jan 21 16:36:30 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 Jan 2014 15:36:30 +0000 Subject: [Distutils] Local version identifiers and wheels Message-ID: PEP 440 allows for local version identifiers, to mark local patches of a distribution. As this stands, local versions cannot be used in wheels, because the identifier is delimited with a dash, which clashes with the separator for the different parts in a wheel filename. What needs to be done to address this? Should we use a different delimiter in PEP 440? Or should we specify a translation for the dash to be used in wheel filenames? This is directly relevant at the moment, because there is a discussion on the distlib tracker of an API to allow locally modifying wheels, and I'm advocating that if this is done, the modified wheel should be forced to have a local version identifier, to avoid confusion with "official" versions. Any thoughts? Thanks, Paul From wickman at gmail.com Mon Jan 20 06:28:08 2014 From: wickman at gmail.com (Brian Wickman) Date: Sun, 19 Jan 2014 21:28:08 -0800 Subject: [Distutils] inconsistency between PEP425 and wheel tags Message-ID: For interpreter-specific tags PEP425 says: "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18, pp19." But in practice if you build with an alternate interpreter you just get the equivalent CPython version level, e.g. pycrypto-2.6.1-pp27-none-macosx_10_7_x86_64.whl for PyPy 2.2.1, when the PEP indicates it should be pp22. As an implementer, should I follow the spec or follow reality? If the latter, should PEP425 be amended? thanks! brian twitter.com/wickman -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Jan 21 17:30:10 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 21 Jan 2014 11:30:10 -0500 Subject: [Distutils] inconsistency between PEP425 and wheel tags In-Reply-To: References: Message-ID: That should probably be considered a bug in bdist_wheel. I don't think there has been any movement on alternate implementations choosing what they want their PEP 425 tags to be. Are you implementing your own wheel generator? (neat) On Mon, Jan 20, 2014 at 12:28 AM, Brian Wickman wrote: > For interpreter-specific tags PEP425 says: > > "The version is py_version_nodot. CPython gets away with no dot, but if one > is needed the underscore _ is used instead. PyPy should probably use its own > versions here pp18, pp19." > > But in practice if you build with an alternate interpreter you just get the > equivalent CPython version level, e.g. > pycrypto-2.6.1-pp27-none-macosx_10_7_x86_64.whl for PyPy 2.2.1, when the PEP > indicates it should be pp22. > > As an implementer, should I follow the spec or follow reality? If the > latter, should PEP425 be amended? > > thanks! > brian > twitter.com/wickman > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From wickman at gmail.com Tue Jan 21 19:09:04 2014 From: wickman at gmail.com (Brian Wickman) Date: Tue, 21 Jan 2014 10:09:04 -0800 Subject: [Distutils] inconsistency between PEP425 and wheel tags In-Reply-To: References: Message-ID: For now I'm just using bdist_wheel, but the question is more focused on the dependency resolution side where I need to select from a list of compatible wheels on a remote server. As it stands, I will be very unsuccessful if bdist_wheel produces pp27 tags and my downloader only looks for distributions tagged with pp22. :-) Happy to send out some pull requests. pip, wheel and distlib all get this wrong. AFAICT pip/wheel use the same pep425tags code. Does that have a shared home? ~brian On Tue, Jan 21, 2014 at 8:30 AM, Daniel Holth wrote: > That should probably be considered a bug in bdist_wheel. I don't think > there has been any movement on alternate implementations choosing what > they want their PEP 425 tags to be. > > Are you implementing your own wheel generator? (neat) > > > > On Mon, Jan 20, 2014 at 12:28 AM, Brian Wickman wrote: > > For interpreter-specific tags PEP425 says: > > > > "The version is py_version_nodot. CPython gets away with no dot, but if > one > > is needed the underscore _ is used instead. PyPy should probably use its > own > > versions here pp18, pp19." > > > > But in practice if you build with an alternate interpreter you just get > the > > equivalent CPython version level, e.g. > > pycrypto-2.6.1-pp27-none-macosx_10_7_x86_64.whl for PyPy 2.2.1, when the > PEP > > indicates it should be pp22. > > > > As an implementer, should I follow the spec or follow reality? If the > > latter, should PEP425 be amended? > > > > thanks! > > brian > > twitter.com/wickman > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Jan 21 19:18:34 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 21 Jan 2014 13:18:34 -0500 Subject: [Distutils] inconsistency between PEP425 and wheel tags In-Reply-To: References: Message-ID: On Tue, Jan 21, 2014 at 1:09 PM, Brian Wickman wrote: > For now I'm just using bdist_wheel, but the question is more focused on the > dependency resolution side where I need to select from a list of compatible > wheels on a remote server. As it stands, I will be very unsuccessful if > bdist_wheel produces pp27 tags and my downloader only looks for > distributions tagged with pp22. :-) > > Happy to send out some pull requests. pip, wheel and distlib all get this > wrong. AFAICT pip/wheel use the same pep425tags code. Does that have a > shared home? > > ~brian Best to update the bdist_wheel one. From vinay_sajip at yahoo.co.uk Tue Jan 21 19:51:40 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 21 Jan 2014 18:51:40 +0000 (GMT) Subject: [Distutils] inconsistency between PEP425 and wheel tags In-Reply-To: Message-ID: <1390330300.22720.YahooMailBasic@web172404.mail.ir2.yahoo.com> It's easy enough to change in distlib, but the use of "probably" in that PyPy-specific sentence in PEP 425 makes that statement as a whole less than definitive. Indeed, the clear specification is to use py_version_nodot, which on PyPy 1.9 at least gives me '27'. So, if we should be using the PyPy version number, I think PEP 425 should explicitly say something like On PyPy, use ''.join(sys.pypy_version_info[:2]) Or whichever PyPy-specific API is supposed to be used to get this information, if not the one I've suggested. Regards, Vinay Sajip -------------------------------------------- On Tue, 21/1/14, Brian Wickman wrote: Subject: Re: [Distutils] inconsistency between PEP425 and wheel tags To: "Daniel Holth" Cc: "DistUtils mailing list" Date: Tuesday, 21 January, 2014, 18:09 For now I'm just using bdist_wheel, but the question is more focused on the dependency resolution side where I need to select from a list of compatible wheels on a remote server. ?As it stands, I will be very unsuccessful if bdist_wheel produces pp27 tags and my downloader only looks for distributions tagged with pp22. :-) Happy to send out some pull requests. ?pip, wheel and distlib all get this wrong. ?AFAICT pip/wheel use the same pep425tags code. ?Does that have a shared home? ~brian On Tue, Jan 21, 2014 at 8:30 AM, Daniel Holth wrote: That should probably be considered a bug in bdist_wheel. I don't think there has been any movement on alternate implementations choosing what they want their PEP 425 tags to be. Are you implementing your own wheel generator? (neat) On Mon, Jan 20, 2014 at 12:28 AM, Brian Wickman wrote: > For interpreter-specific tags PEP425 says: > > "The version is py_version_nodot. CPython gets away with no dot, but if one > is needed the underscore _ is used instead. PyPy should probably use its own > versions here pp18, pp19." > > But in practice if you build with an alternate interpreter you just get the > equivalent CPython version level, e.g. > pycrypto-2.6.1-pp27-none-macosx_10_7_x86_64.whl for PyPy 2.2.1, when the PEP > indicates it should be pp22. > > As an implementer, should I follow the spec or follow reality? ?If the > latter, should PEP425 be amended? > > thanks! > brian > twitter.com/wickman > > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -----Inline Attachment Follows----- _______________________________________________ Distutils-SIG maillist? -? Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Tue Jan 21 23:48:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Jan 2014 08:48:11 +1000 Subject: [Distutils] inconsistency between PEP425 and wheel tags In-Reply-To: <1390330300.22720.YahooMailBasic@web172404.mail.ir2.yahoo.com> References: <1390330300.22720.YahooMailBasic@web172404.mail.ir2.yahoo.com> Message-ID: The problem is that sys.implementation is relatively new, so there's no standard way to retrieve the implementation version prior to that addition to the language spec. I agree the PEP should be explicit that that tag needs to reflect the implementation's *ABI* stability, which may or may not align with the Python language version. We're going to need an update to the compatibility tagging spec to sort out the Linux distro compatibility issues anyway, so perhaps someone is prepared to tackle a revision to the compatibility tagging PEP? (The relevant repo for draft specs is https://bitbucket.org/pypa/pypi-metadata-formats - although it occurs to me that could use some tags to reflect published versions of accepted PEPs) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.postma at gmail.com Wed Jan 22 15:29:55 2014 From: warren.postma at gmail.com (Warren Postma) Date: Wed, 22 Jan 2014 09:29:55 -0500 Subject: [Distutils] (no subject) Message-ID: Today I ran across something that made me go "wat?". There is a package named wx in PyPI and pip happily installs it. It's completely useless, the module doc string is not in english, and it's definitely not wxPython. This has been noticed by others: https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-package What is the proper channel for asking for the renaming or removal of a package? If this package is indeed useless, then it should be removed. If it is useful, but completely undocumented, and appears that import wx followed by dir(wx) shows absolutely nothing useful inside it. This is the entire content of the unfortunately named wx.py module: # -*- coding:utf-8 -*- """ ?? PHP ?? file_get_content ??Python???? """ def file_get_contents(file): file = open(file) data = '' for line in file: data = data + line file.close() return data The last part translated from Translate.google.com is "My Python test module". How did this get into PyPI? It's pypi page url is: https://pypi.python.org/pypi/wx/1.0.0 Does anyone have to approve these things before they get into PyPI or can any post their hello world programs to PyPi? Can I write three lines, call it pyQt, and upload it? that particular four letter name is unclaimed, yet I bet most pip users would like it if it gave them an error rather than installing some package which is not PyQt, it is just called that. Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jan 22 18:50:28 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 22 Jan 2014 12:50:28 -0500 Subject: [Distutils] (no subject) In-Reply-To: References: Message-ID: <048A7760-761D-4789-9330-A75970F57E57@stufft.io> On Jan 22, 2014, at 9:29 AM, Warren Postma wrote: > > Today I ran across something that made me go "wat?". There is a package named wx in PyPI and pip happily installs it. > It's completely useless, the module doc string is not in english, and it's definitely not wxPython. > > This has been noticed by others: > > https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-package > > What is the proper channel for asking for the renaming or removal of a package? If this package is indeed useless, then it should be removed. If it is useful, but completely undocumented, and appears that import wx followed by dir(wx) shows absolutely nothing useful inside it. This is the entire content of the unfortunately named wx.py module: > > # -*- coding:utf-8 -*- > > """ > ?? PHP ?? file_get_content > ??Python???? > """ > def file_get_contents(file): > file = open(file) > data = '' > for line in file: > data = data + line > > file.close() > return data > > > The last part translated from Translate.google.com is "My Python test module". > > How did this get into PyPI? It's pypi page url is: > https://pypi.python.org/pypi/wx/1.0.0 > > Does anyone have to approve these things before they get into PyPI or can any post their hello world programs to PyPi? > > Can I write three lines, call it pyQt, and upload it? that particular four letter name is unclaimed, yet I bet most pip users would like it if it gave them an error rather than installing some package which is not PyQt, it is just called that. > > Warren > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Names on PyPI are on a first come first serve basis although we do exercise administrative control when deemed appropriate. It?s possible this name should be taken over, I?d want to leave that up to Richard. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Wed Jan 22 21:22:21 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 22 Jan 2014 12:22:21 -0800 Subject: [Distutils] (no subject) In-Reply-To: <048A7760-761D-4789-9330-A75970F57E57@stufft.io> References: <048A7760-761D-4789-9330-A75970F57E57@stufft.io> Message-ID: btw, see this thread from last year about squatters and naming... https://mail.python.org/pipermail/distutils-sig/2013-June/021010.html On Wed, Jan 22, 2014 at 9:50 AM, Donald Stufft wrote: > > On Jan 22, 2014, at 9:29 AM, Warren Postma > wrote: > > > Today I ran across something that made me go "wat?". There is a package > named wx in PyPI and pip happily installs it. > It's completely useless, the module doc string is not in english, and it's > definitely not wxPython. > > This has been noticed by others: > > > https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-package > > What is the proper channel for asking for the renaming or removal of a > package? If this package is indeed useless, then it should be removed. If > it is useful, but completely undocumented, and appears that import wx > followed by dir(wx) shows absolutely nothing useful inside it. This is the > entire content of the unfortunately named wx.py module: > > # -*- coding:utf-8 -*- > > """ > ?? PHP ?? file_get_content > ??Python???? > """ > def file_get_contents(file): > file = open(file) > data = '' > for line in file: > data = data + line > > file.close() > return data > > > The last part translated from Translate.google.comis " > My Python test module". > > How did this get into PyPI? It's pypi page url is: > https://pypi.python.org/pypi/wx/1.0.0 > > Does anyone have to approve these things before they get into PyPI or can > any post their hello world programs to PyPi? > > Can I write three lines, call it pyQt, and upload it? that particular four > letter name is unclaimed, yet I bet most pip users would like it if it gave > them an error rather than installing some package which is not PyQt, it is > just called that. > > Warren > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > Names on PyPI are on a first come first serve basis although we do > exercise administrative control when deemed appropriate. It?s possible this > name should be taken over, I?d want to leave that up to Richard. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed Jan 22 21:46:56 2014 From: richard at python.org (Richard Jones) Date: Thu, 23 Jan 2014 07:46:56 +1100 Subject: [Distutils] (no subject) In-Reply-To: References: <048A7760-761D-4789-9330-A75970F57E57@stufft.io> Message-ID: [Oops. I keep forgetting to check the bitbucket tracker for support issues which are intended to (and mostly do) go to the sourceforge tracker.] As the PyPI admin, I deal with naming issues that come up, which happens not too frequently thankfully. Usually they're resolved reasonably easily when I contact the author of the offending project. Indeed, yesterday I resolved an issue with someone uploading their own personal fork "kafka" which was confusing users of the project "python-kafka". I suspect Warehouse might have to grow a name blacklist or something though the administrative burden would probably become too much. Richard On 23 January 2014 07:22, Marcus Smith wrote: > btw, see this thread from last year about squatters and naming... > https://mail.python.org/pipermail/distutils-sig/2013-June/021010.html > > > On Wed, Jan 22, 2014 at 9:50 AM, Donald Stufft wrote: > >> >> On Jan 22, 2014, at 9:29 AM, Warren Postma >> wrote: >> >> >> Today I ran across something that made me go "wat?". There is a package >> named wx in PyPI and pip happily installs it. >> It's completely useless, the module doc string is not in english, and >> it's definitely not wxPython. >> >> This has been noticed by others: >> >> >> https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-package >> >> What is the proper channel for asking for the renaming or removal of a >> package? If this package is indeed useless, then it should be removed. If >> it is useful, but completely undocumented, and appears that import wx >> followed by dir(wx) shows absolutely nothing useful inside it. This is the >> entire content of the unfortunately named wx.py module: >> >> # -*- coding:utf-8 -*- >> >> """ >> ?? PHP ?? file_get_content >> ??Python???? >> """ >> def file_get_contents(file): >> file = open(file) >> data = '' >> for line in file: >> data = data + line >> >> file.close() >> return data >> >> >> The last part translated from Translate.google.comis " >> My Python test module". >> >> How did this get into PyPI? It's pypi page url is: >> https://pypi.python.org/pypi/wx/1.0.0 >> >> Does anyone have to approve these things before they get into PyPI or can >> any post their hello world programs to PyPi? >> >> Can I write three lines, call it pyQt, and upload it? that particular >> four letter name is unclaimed, yet I bet most pip users would like it if it >> gave them an error rather than installing some package which is not PyQt, >> it is just called that. >> >> Warren >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> Names on PyPI are on a first come first serve basis although we do >> exercise administrative control when deemed appropriate. It?s possible this >> name should be taken over, I?d want to leave that up to Richard. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jan 22 22:14:34 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 22 Jan 2014 21:14:34 +0000 (GMT) Subject: [Distutils] (no subject) In-Reply-To: Message-ID: <1390425274.61503.YahooMailBasic@web172406.mail.ir2.yahoo.com> Come to think of it, my project "python-gnupg" (which I forked from some work that Richard and others did) has itself been forked, and the derivative project uploaded to PyPI as "gnupg". It's not a drop-in replacement, and I know of one instance where it has caused confusion. I have nothing against the other project, it's trying to do the same job - just not in a compatible way - which is what can lead to confusion. Regards, Vinay Sajip -------------------------------------------- On Wed, 22/1/14, Richard Jones wrote: Subject: Re: [Distutils] (no subject) To: "Marcus Smith" Cc: "Warren Postma" , "DistUtils mailing list" Date: Wednesday, 22 January, 2014, 20:46 [Oops. I keep forgetting to check the bitbucket tracker for support issues which are intended to (and mostly do) go to the sourceforge tracker.] As the PyPI admin, I deal with naming issues that come up, which happens not too frequently thankfully. Usually they're resolved reasonably easily when I contact the author of the offending project. Indeed, yesterday I resolved an issue with someone uploading their own personal fork "kafka" which was confusing users of the project "python-kafka". I suspect Warehouse might have to grow a name blacklist or something though the administrative burden would probably become too much. ? ? ?Richard On 23 January 2014 07:22, Marcus Smith wrote: btw, see this thread from last year about squatters and naming...https://mail.python.org/pipermail/distutils-sig/2013-June/021010.html On Wed, Jan 22, 2014 at 9:50 AM, Donald Stufft wrote: On Jan 22, 2014, at 9:29 AM, Warren Postma wrote: Today I ran across something that made me go "wat?".?? There is a package named wx in PyPI and pip happily installs it. It's completely useless, the module doc string is not in english, and it's definitely not wxPython. This has been noticed by others: https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-package What is the proper channel for asking for the renaming or removal of a package?? If this package is indeed useless, then it should be removed. If it is useful, but completely undocumented, and appears that import wx followed by dir(wx) shows absolutely nothing useful inside it.? This is the entire content of the unfortunately named wx.py module: # -*- coding:utf-8 -*- """ ?? PHP ?? file_get_content ??Python???? """ def file_get_contents(file): ??? file = open(file) ??? data = '' ??? for line in file: ????? data = data + line ??? file.close() ??? return data The last part translated from Translate.google.com is "My Python test module". How did this get into PyPI? It's pypi page url is: https://pypi.python.org/pypi/wx/1.0.0 Does anyone have to approve these things before they get into PyPI or can any post their hello world programs to PyPi? Can I write three lines, call it pyQt, and upload it? that particular four letter name is unclaimed, yet I bet most pip users would like it if it gave them an error rather than installing some package which is not PyQt, it is just called that. Warren _______________________________________________ Distutils-SIG maillist ?- ?Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig Names on PyPI are on a first come first serve basis although we do exercise administrative control when deemed appropriate. It?s possible this name should be taken over, I?d want to leave that up to Richard. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B?7C5D 5E2B 6356 A926 F04F 6E3C?BCE9 3372 DCFA _______________________________________________ Distutils-SIG maillist ?- ?Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig _______________________________________________ Distutils-SIG maillist ?- ?Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -----Inline Attachment Follows----- _______________________________________________ Distutils-SIG maillist? -? Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig From theller at ctypes.org Thu Jan 23 09:28:07 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 09:28:07 +0100 Subject: [Distutils] error message from testpypi.python.org Message-ID: I'm trying out to submit a wheel to testpypi.python.org and then install it from there with pip. It doesn't work; what does the following error message mean? c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/ Exception: Traceback (most recent call last): File "C:\Python34-64\lib\site-packages\pip\basecommand.py", line 122, in main status = self.run(options, args) File "C:\Python34-64\lib\site-packages\pip\commands\install.py", line 258, in run for req in parse_requirements(filename, finder=finder, options=options, session=session): File "C:\Python34-64\lib\site-packages\pip\req.py", line 1572, in parse_requirements req = InstallRequirement.from_line(line, comes_from, prereleases=getattr(options, "pre", None)) File "C:\Python34-64\lib\site-packages\pip\req.py", line 172, in from_line return cls(req, comes_from, url=url, prereleases=prereleases) File "C:\Python34-64\lib\site-packages\pip\req.py", line 70, in __init__ req = pkg_resources.Requirement.parse(req) File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", line 2612, in parse reqs = list(parse_requirements(s)) File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", line 2538, in parse_requirements raise ValueError("Missing distribution spec", line) ValueError: ('Missing distribution spec', '') Storing debug log for failure in C:\Users\thomas\AppData\Roaming\pip\pip.log Thanks, Thomas From p.f.moore at gmail.com Thu Jan 23 10:05:14 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Jan 2014 09:05:14 +0000 Subject: [Distutils] error message from testpypi.python.org In-Reply-To: References: Message-ID: On 23 January 2014 08:28, Thomas Heller wrote: > I'm trying out to submit a wheel to testpypi.python.org and then > install it from there with pip. It doesn't work; what does the > following error message mean? > > c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/ The flag -r is for a requirements file. Did you mean -i (--index-url)? Paul From theller at ctypes.org Thu Jan 23 10:42:30 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 10:42:30 +0100 Subject: [Distutils] error message from testpypi.python.org In-Reply-To: References: Message-ID: Am 23.01.2014 10:05, schrieb Paul Moore: > On 23 January 2014 08:28, Thomas Heller wrote: >> I'm trying out to submit a wheel to testpypi.python.org and then >> install it from there with pip. It doesn't work; what does the >> following error message mean? >> >> c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/ > > The flag -r is for a requirements file. Did you mean -i (--index-url)? Sure - thanks for that, now it works. My only excuse is that setup.py uses the -r flag for that purpose... Thomas From chris.jerdonek at gmail.com Thu Jan 23 10:50:03 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 23 Jan 2014 01:50:03 -0800 Subject: [Distutils] error message from testpypi.python.org In-Reply-To: References: Message-ID: On Thu, Jan 23, 2014 at 1:42 AM, Thomas Heller wrote: > Am 23.01.2014 10:05, schrieb Paul Moore: > >> On 23 January 2014 08:28, Thomas Heller wrote: >>> >>> I'm trying out to submit a wheel to testpypi.python.org and then >>> install it from there with pip. It doesn't work; what does the >>> following error message mean? >>> >>> c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/ >> >> >> The flag -r is for a requirements file. Did you mean -i (--index-url)? > > > Sure - thanks for that, now it works. > My only excuse is that setup.py uses the -r flag for that purpose... Well, that and the fact that the resulting error didn't seem to be of much help in pointing you in the right direction. I wonder if it could have failed earlier with a more helpful message. --Chris From theller at ctypes.org Thu Jan 23 12:48:10 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 12:48:10 +0100 Subject: [Distutils] pip on windows experience Message-ID: After having used and built bdist_wininst installers for years (well, I should say decades) I have recently started playing with pip. In theory, it is a grat solution. Respects virtual environments (although I haven't tested them yet), allows to freeze to a requirements file, and so on. It would allow to freeze and then commit a requirements file into our version control system, add a 'pip install -r requirements.txt' to our build scripts, and any user in out company as well as the build server would automatically have installed all the required stuff. (We have 8 developers working on our source code, plus a build server that does automatic builds). However, does it work in real life? There are no wheels for important packages like pywin32, numpy, lxml, at least not on PyPi. 'pip install' using the source packages fails to build for most but the simplest stuff; probably because distutils has been extended too much, or because header files or libraries are missing, or whatever... Is there a solution to this? I've seen that the wheel tool can convert bdist_wininst installers into wheels - does this work for the packages I mentioned above? Do we have to build or convert to wheel those packages, and setup a central place where we store them and make them available to our developers? BTW: It seems the pip 1.4.1 doesn't download the wheel built with the most recent setuptools/wheel version, instead it downloads and tries to build from the source package. Thanks, Thomas From p.f.moore at gmail.com Thu Jan 23 13:16:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Jan 2014 12:16:02 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: Message-ID: On 23 January 2014 11:48, Thomas Heller wrote: > After having used and built bdist_wininst installers for years (well, I > should say decades) I have recently started playing with pip. Thanks for trying this - it's really useful to have a newcomer's perspective to the pip experience. > However, does it work in real life? > > There are no wheels for important packages like pywin32, numpy, lxml, > at least not on PyPi. 'pip install' using the source packages fails to > build for most but the simplest stuff; probably because distutils has > been extended too much, or because header files or libraries are > missing, or whatever... This is the key point - pip install from source fails on Windows for pretty much any package with C code and non-trivial dependencies. Building any of the above packages is murder on Windows, and it's 100% critical to the user experience that you shouldn't have to. > Is there a solution to this? I've seen that the wheel tool can convert > bdist_wininst installers into wheels - does this work for the packages > I mentioned above? Do we have to build or convert to wheel those > packages, and setup a central place where we store them and make them > available to our developers? The good news is that "wheel convert XXX.exe" works on all the above for creating a wheel from the wininst installer. The official numpy installer uses some complex magic to select the right binaries based on your CPU, and this means that the official numpy "superpack" wininst files don't convert (at least I don't think they do, it's a while since I tried). But happily, Christoph Gohlke hosts a huge list of readymade wininst installers for hard-to-build projects, and the 3 you mention are all there. He's very good about building for latest Pythons, too (3.4 is already there for many packages). Anyone working on Windows who doesn't know his site (http://www.lfd.uci.edu/~gohlke/pythonlibs/) should check it out. So, to summarise, yes, you can get wheels for pretty much everything you need by using wheel convert on wininst installers. You do need to manually download, run wheel convert, and host the wheels locally (a simple directory is fine though). I'd love to see some or all of these projects host wheels themselves on PyPI, or someone like Christoph host a pypi-style index of wheels, so we could just point pip at that index as well as PyPI (note: I don't know if pip works when a package is split across 2 index sites, that's something I should check, but if it doesn't, I'd like to see that added). But that's in the future for now - we probably need more uptake of wheels before the demand becomes sufficient to persuade anyone to do this. > BTW: It seems the pip 1.4.1 doesn't download the wheel built with the > most recent setuptools/wheel version, instead it downloads and tries > to build from the source package. Pip 1.4 doesn't use wheels by default - you should add "--use-wheel" to the install command (or put it in your pip.ini). "--use-wheel is the default from pip 1.5 onwards. Hope this helps, Paul. From drsalists at gmail.com Thu Jan 23 06:44:13 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 22 Jan 2014 21:44:13 -0800 Subject: [Distutils] setup.py issue - some files are included as intended, but one is not In-Reply-To: References: Message-ID: I've redone it so it doesn't require make, and renamed the Makefile. However, it's quite a bit slower now. Also, "python setup.py clean" doesn't appear to remove all the build artifacts. Is there some way I can speed it up? Is there some way I can automatically remove the build artifacts without relying on make? When they're left lying around, "svn status" isn't quite as quick and easy. Thanks! PS: The link is http://stromberg.dnsalias.org/svn/treap/trunk/ On Fri, Jan 17, 2014 at 5:06 PM, Dan Stromberg wrote: > Hi all. > > Yes, it does currently require make. I'm very comfortable with make, > but I'm open to redoing the make rules in setup.py if that'll help get > this problem solved. It'd probably be less puzzling to Python people > anyway. > > I kept increasing the version number as I tried different ways of > getting setup.py to do what I wanted. That's why the version numbers > don't appear to match up. If there's a better way of testing > setup.py's, please let me know! > > Thanks! > >>hello, I checked out your project from svn. >>looks like it requires some make commands, before it can be packaged? >>I'm getting tripped up on that. >>also confused in that your src tree is at 1.30, but the tar you mention is >>1.35, and then your install dir was 1.31. >>maybe get your src tree up to date with where your working copy, and post >>any make commands that need to be run. >> >>On Tue, Jan 14, 2014 at 1:32 PM, Ethan Furman wrote: >> >>> On 01/14/2014 01:26 PM, Dan Stromberg wrote: >>> >>>> On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg >>>> wrote: >>>> >>>>> Hi folks. >>>>> >>>>> I have a setup.py problem that's driving me nuts. >>>>> >>>> >>>> Anyone? I've received 0 responses. >>>> >>> >>> I have no answer, but forwarding to Distutils (hopefully it's an >>> appropriate topic ;) >>> >>> >>> I have a treap.py file that tries to "import * from pyx_treap.so", and >>>>> failing that, it'll "import * from py_treap.py" (sans extensions of >>>>> course). Naturally, all 3 of these should be included - although in >>>>> the case of pyx_treap.so, it probably should be a .c that's included. >>>>> >>>>> It is also intended to include nest.py, which has a simple form. >>>>> >>>>> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >>>>> seem to get py_treap.py to be included for some reason. >>>>> >>>>> I get no errors during "python setup.py sdist upload", but upon >>>>> "python $(which pip) install treap", I get: >>>>> $ sudo /usr/bin/python $(which pip) install treap >>>>> Downloading/unpacking treap >>>>> Running setup.py egg_info for package treap >>>>> >>>>> file py_treap.py (for module py_treap) not found >>>>> Installing collected packages: treap >>>>> Running setup.py install for treap >>>>> file py_treap.py (for module py_treap) not found >>>>> file py_treap.py (for module py_treap) not found >>>>> >>>>> file py_treap.py (for module py_treap) not found >>>>> file py_treap.py (for module py_treap) not found >>>>> Successfully installed treap >>>>> Cleaning up... >>>>> >>>>> And it's not successfully installed - py_treap.py is missing. The pyx >>>>> code does its job, so the problem is masked, other than the messages >>>>> above, and the absence of py_treap.py from >>>>> /usr/local/lib/python2.7/dist-packages >>>>> >>>>> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >>>>> least getting packaged up that much. It's not listed in >>>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >>>>> , but I can see it in >>>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >>>>> . >>>>> >>>>> My setup.py is at: >>>>> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >>>>> >>>>> I've tried that setup.py and several variations on that theme, but >>>>> none seem to include py_treap.py . >>>>> >>>>> Please make some suggestions? How can I get py_treap.py included in >>>>> the pip install? >>>>> >>>>> Thanks! >>>>> >>>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>-------------- next part -------------- >>An HTML attachment was scrubbed... >>URL: From theller at ctypes.org Thu Jan 23 15:06:39 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 15:06:39 +0100 Subject: [Distutils] pip on windows experience In-Reply-To: References: Message-ID: Am 23.01.2014 13:16, schrieb Paul Moore: > On 23 January 2014 11:48, Thomas Heller wrote: >> Is there a solution to this? I've seen that the wheel tool can convert >> bdist_wininst installers into wheels - does this work for the packages >> I mentioned above? Do we have to build or convert to wheel those >> packages, and setup a central place where we store them and make them >> available to our developers? > > The good news is that "wheel convert XXX.exe" works on all the above > for creating a wheel from the wininst installer. The official numpy > installer uses some complex magic to select the right binaries based > on your CPU, and this means that the official numpy "superpack" > wininst files don't convert (at least I don't think they do, it's a > while since I tried). But happily, Christoph Gohlke hosts a huge list > of readymade wininst installers for hard-to-build projects, and the 3 > you mention are all there. He's very good about building for latest > Pythons, too (3.4 is already there for many packages). Anyone working > on Windows who doesn't know his site > (http://www.lfd.uci.edu/~gohlke/pythonlibs/) should check it out. > > So, to summarise, yes, you can get wheels for pretty much everything > you need by using wheel convert on wininst installers. You do need to > manually download, run wheel convert, and host the wheels locally (a > simple directory is fine though). Thanks Paul, for this info. Thomas (over to the next topic) From theller at ctypes.org Thu Jan 23 15:23:26 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 15:23:26 +0100 Subject: [Distutils] wheels, metadata Message-ID: I'm reading up and done all those peps: 425 - compaibility tags 426 - metadata 2.0 427 - wheel binary format 1.0 440 - version identification and dependency specifications and trying to make sense from them. Well, they make sense to me, but to what do they apply? Example: I can now build, upload, and install a wheel for my distribution. I have imported setuptools in my setup.py script, and have run 'setup.py bdist_wheel'. The resulting file is named 'py2exe-0.9.0-py34-none-any.whl', which means: python tag 'py34', abi tag 'none', platform tag 'any' What this file really is: it is a library that only works on CPython 3.3 and 3.4, win32 and win_amd64. So I would have expected a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar. Hoever, the only thing that I found which did change the filename is this section in the setup.cfg file (which is not appopriate for this distribution): [wheel] universal=1 So, the question is: Are the specifications in the PEPs above implemented somewhere or are they only 'specifications'? There are very detailed descriptions how to specify requirements in the metadata (in pep426), something like "supports_environments": ["python_version >= '3.3' and 'win' in sys_platform"] Where can I specify this? Does distutils support it? Do I have to manually edit some pdist.json file, or do I have to rename the created .whl? Thomas From oscar.j.benjamin at gmail.com Thu Jan 23 15:42:29 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 23 Jan 2014 14:42:29 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: Message-ID: <20140123144227.GB3553@gmail.com> On Thu, Jan 23, 2014 at 12:16:02PM +0000, Paul Moore wrote: > > The official numpy installer uses some complex magic to select the > right binaries based on your CPU, and this means that the official > numpy "superpack" wininst files don't convert (at least I don't think > they do, it's a while since I tried). It's probably worth noting that numpy are toying around with wheels and have uploaded a number of them to PyPI for testing: http://sourceforge.net/projects/numpy/files/wheels_to_test/ Currently there are only OSX wheels there (excluding the puer Python ones) and they're not available on PyPI. I assume that they're waiting for a solution for the Windows installer (a post-install script for wheels). That would give a lot more impetus to put wheels up on PyPI. The Sourceforge OSX wheels are presumably not getting that much use right now. The OSX-specific numpy wheel has been downloaded 4 times in the last week: twice on Windows and twice on Linux! > But happily, Christoph Gohlke hosts a huge list of readymade wininst > installers for hard-to-build projects, and the 3 you mention are all > there. He's very good about building for latest Pythons, too (3.4 is > already there for many packages). Anyone working on Windows who > doesn't know his site > (http://www.lfd.uci.edu/~gohlke/pythonlibs/) should check it out. Also I've seen Cristoph mention on the numpy-discussion list that he was at leasting testing building wheels although none seem to be available on his site at the moment. Oscar From dholth at gmail.com Thu Jan 23 16:08:30 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 23 Jan 2014 10:08:30 -0500 Subject: [Distutils] wheels, metadata In-Reply-To: References: Message-ID: On Thu, Jan 23, 2014 at 9:23 AM, Thomas Heller wrote: > I'm reading up and done all those peps: > > 425 - compaibility tags > 426 - metadata 2.0 > 427 - wheel binary format 1.0 > 440 - version identification and dependency specifications > > and trying to make sense from them. Well, they make sense to me, > but to what do they apply? > > Example: I can now build, upload, and install a wheel for my distribution. > > I have imported setuptools in my setup.py script, > and have run 'setup.py bdist_wheel'. > > The resulting file is named 'py2exe-0.9.0-py34-none-any.whl', > which means: > > python tag 'py34', abi tag 'none', platform tag 'any' > > What this file really is: it is a library that only works on > CPython 3.3 and 3.4, win32 and win_amd64. So I would have expected > a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar. > > Hoever, the only thing that I found which did change the filename > is this section in the setup.cfg file (which is not appopriate for > this distribution): > > [wheel] > universal=1 > > So, the question is: > > Are the specifications in the PEPs above implemented somewhere > or are they only 'specifications'? > > There are very detailed descriptions how to specify requirements > in the metadata (in pep426), something like > > "supports_environments": ["python_version >= '3.3' > and 'win' in sys_platform"] > > Where can I specify this? Does distutils support it? Do > I have to manually edit some pdist.json file, or do I have > to rename the created .whl? > > Thomas supports_environments is not implemented. Currently you should be able to just rename the created .whl to change the tags. In the future the wheel metadata will have to match the filename so you would need a 'retag' command or just better setup.cfg options. bdist_wheel supports this syntax for replacing the requirements metadata and including the full text of a license in the metadata, in setup.cfg. One drawback to this approach is that the wheel can have different dependencies than the non-wheel because vanilla setuptools doesn't generate the .dist-info style metadata. distutils sucks, but I haven't had the opportunity to write a replacement. [metadata] provides-extra = tool signatures faster-signatures requires-dist = argparse; python_version == '2.6' keyring; extra == 'signatures' dirspec; sys.platform != 'win32' and extra == 'signatures' ed25519ll; extra == 'faster-signatures' license-file = LICENSE.txt [wheel] # use py2.py3 tag for pure-python dist: universal=1 From vinay_sajip at yahoo.co.uk Thu Jan 23 16:49:05 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 23 Jan 2014 15:49:05 +0000 (GMT) Subject: [Distutils] wheels, metadata In-Reply-To: Message-ID: <1390492145.12306.YahooMailBasic@web172402.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 23/1/14, Daniel Holth wrote: supports_environments is not implemented. Perhaps not in pip and wheel, but there is support in distlib and distil. Specifically, distil should present a list of failing clauses in supports_environments for any distributions to be installed (including dependent ones). Of course, there's not much on PyPI one can test with, since that metadata is absent, so testing of this feature has been limited to local test data. Regards, Vinay Sajip From qwcode at gmail.com Thu Jan 23 19:30:34 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 23 Jan 2014 10:30:34 -0800 Subject: [Distutils] wheels, metadata In-Reply-To: References: Message-ID: > > > Are the specifications in the PEPs above implemented somewhere > or are they only 'specifications'? > The Packaging User Guide is maintaining a summary of the relevant PEPs that gives a brief description of the "user impact" and "implementation" (i.e. whether projects are actually implementing any of this). https://python-packaging-user-guide.readthedocs.org/en/latest/peps.html For those working on the various projects, please help keep this up to date. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jan 23 19:52:22 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 23 Jan 2014 19:52:22 +0100 Subject: [Distutils] pip on windows experience In-Reply-To: <20140123144227.GB3553@gmail.com> References: <20140123144227.GB3553@gmail.com> Message-ID: On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin wrote: > On Thu, Jan 23, 2014 at 12:16:02PM +0000, Paul Moore wrote: > > > > The official numpy installer uses some complex magic to select the > > right binaries based on your CPU, and this means that the official > > numpy "superpack" wininst files don't convert (at least I don't think > > they do, it's a while since I tried). > > It's probably worth noting that numpy are toying around with wheels and > have uploaded a number of them to PyPI for testing: > http://sourceforge.net/projects/numpy/files/wheels_to_test/ > > Currently there are only OSX wheels there (excluding the puer Python > ones) and they're not available on PyPI. I assume that they're waiting > for a solution for the Windows installer (a post-install script for > wheels). That would give a lot more impetus to put wheels up on PyPI. > Indeed. We discussed just picking the SSE2 or SSE3 build and putting that up as a wheel, but that was deemed a not so great idea: http://article.gmane.org/gmane.comp.python.numeric.general/56072 The Sourceforge OSX wheels are presumably not getting that much use > right now. The OSX-specific numpy wheel has been downloaded 4 times in > the last week: twice on Windows and twice on Linux! > Some feedback from the people who did try those wheels would help. I asked for that on the numpy list after creating them, but didn't get much. So I haven't been in a hurry to move them over to PyPi. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jan 23 20:02:33 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 23 Jan 2014 19:02:33 +0000 (GMT) Subject: [Distutils] wheels, metadata In-Reply-To: Message-ID: <1390503753.59722.YahooMailBasic@web172401.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 23/1/14, Marcus Smith wrote: > The Packaging User Guide is maintaining a summary > of the relevant PEPs that gives a brief description of the > "user impact" and "implementation" (i.e. > whether projects are actually implementing any of > this). > https://python-packaging-user-guide.readthedocs.org/en/latest/peps.html > For those working on the various projects, please > help keep this up to date. A lot of this stuff is implemented in distlib and distil (distlib itself is just a library that provides the underpinnings, and some of the functionality required, such as actually installing stuff, is implemented in distil. I assume it's OK to add references to these? For example, distlib implements PEP 376, PEP 425, PEP 426, PEP 427, PEP 440, but only the last of these is mentioned on the PEP summaries page. Regards, Vinay Sajip From qwcode at gmail.com Thu Jan 23 20:12:21 2014 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 23 Jan 2014 11:12:21 -0800 Subject: [Distutils] wheels, metadata In-Reply-To: <1390503753.59722.YahooMailBasic@web172401.mail.ir2.yahoo.com> References: <1390503753.59722.YahooMailBasic@web172401.mail.ir2.yahoo.com> Message-ID: > > > A lot of this stuff is implemented in distlib and distil (distlib itself > is just a library that provides the underpinnings, and some of the > functionality required, such as actually installing stuff, is implemented > in distil. I assume it's OK to add references to these? For example, > distlib implements PEP 376, PEP 425, PEP 426, PEP 427, PEP 440, but only > the last of these is mentioned on the PEP summaries page. > certainly mention the distlib implementations, but also let's be clear if there is a pypa-recommend tool that is user-facing (like pip), that is using those parts of distlib. In most cases, that is not true currently. As for mentioning distil, I'm inclined to say no. Up to this point, you've presented it as a proof of concept. If you're wanting to mention "distil" as a real option for users, I'm concerned about fracturing the mind of users, but it's something to discuss I guess. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jan 23 20:41:52 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 23 Jan 2014 14:41:52 -0500 Subject: [Distutils] wheels, metadata In-Reply-To: References: <1390503753.59722.YahooMailBasic@web172401.mail.ir2.yahoo.com> Message-ID: We need pip's build to become more like conda build. http://docs.continuum.io/conda/build.html A simple recipe, not necessarily part of the to-be-built package source, invokes build.sh to create a package. In that kind of system things like distil can be used at will. On Thu, Jan 23, 2014 at 2:12 PM, Marcus Smith wrote: >> >> A lot of this stuff is implemented in distlib and distil (distlib itself >> is just a library that provides the underpinnings, and some of the >> functionality required, such as actually installing stuff, is implemented in >> distil. I assume it's OK to add references to these? For example, distlib >> implements PEP 376, PEP 425, PEP 426, PEP 427, PEP 440, but only the last of >> these is mentioned on the PEP summaries page. > > > certainly mention the distlib implementations, but also let's be clear if > there is a pypa-recommend tool that is user-facing (like pip), that is using > those parts of distlib. > In most cases, that is not true currently. > > As for mentioning distil, I'm inclined to say no. Up to this point, > you've presented it as a proof of concept. > If you're wanting to mention "distil" as a real option for users, I'm > concerned about fracturing the mind of users, but it's something to discuss > I guess. > > Marcus > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From vinay_sajip at yahoo.co.uk Thu Jan 23 21:18:48 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 23 Jan 2014 20:18:48 +0000 (GMT) Subject: [Distutils] wheels, metadata In-Reply-To: Message-ID: <1390508328.55711.YahooMailBasic@web172406.mail.ir2.yahoo.com> -------------------------------------------- > certainly mention the distlib implementations, > but also let's be clear if there is a pypa-recommend > tool that is user-facing (like pip), that is using those > parts of distlib. ? > In most cases, that is not true currently. > As for mentioning distil, ?I'm inclined to > say no. ? Up to this point, you've presented it as a > proof of concept.If you're wanting to mention > "distil" as a real option for users, I'm > concerned about fracturing the mind of users, but it's > something to discuss I guess. I hear what you're saying. I've positioned distil as a proof of concept purely because it hasn't had widespread use, but I certainly expect it to fulfil the same role as pip functionally (which it must do to be an effective test-bed for distlib). I understand that pip is the officially recommended tool, and don't want to muddy the waters, there being enough confusion about packaging in the wider community. It seems a shame that some of the improvements over pip won't be more widely available, but such is life. I have the use of them, so there's that :-) Regards, Vinay Sajip From theller at ctypes.org Thu Jan 23 21:25:56 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 21:25:56 +0100 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: Am 23.01.2014 19:52, schrieb Ralf Gommers: > > > > On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin > > wrote: > > On Thu, Jan 23, 2014 at 12:16:02PM +0000, Paul Moore wrote: > > > > The official numpy installer uses some complex magic to select the > > right binaries based on your CPU, and this means that the official > > numpy "superpack" wininst files don't convert (at least I don't think > > they do, it's a while since I tried). > > It's probably worth noting that numpy are toying around with wheels and > have uploaded a number of them to PyPI for testing: > http://sourceforge.net/projects/numpy/files/wheels_to_test/ > > Currently there are only OSX wheels there (excluding the puer Python > ones) and they're not available on PyPI. I assume that they're waiting > for a solution for the Windows installer (a post-install script for > wheels). That would give a lot more impetus to put wheels up on PyPI. > > > Indeed. We discussed just picking the SSE2 or SSE3 build and putting > that up as a wheel, but that was deemed a not so great idea: > http://article.gmane.org/gmane.comp.python.numeric.general/56072 Did I say this before? I would suggest that numpy develops a way where all the SSE binary variations would be installed, and the appropriate ones be loaded at runtime, depending on the user's CPU capabilities. This would also allow py2exe'd distributions to include them all. > The Sourceforge OSX wheels are presumably not getting that much use > right now. The OSX-specific numpy wheel has been downloaded 4 times in > the last week: twice on Windows and twice on Linux! > > > Some feedback from the people who did try those wheels would help. I > asked for that on the numpy list after creating them, but didn't get > much. So I haven't been in a hurry to move them over to PyPi. I would have tried wheels for windows, python 3.3 or 3.4, but there aren't any. Thomas From theller at ctypes.org Thu Jan 23 21:27:27 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jan 2014 21:27:27 +0100 Subject: [Distutils] wheels, metadata In-Reply-To: References: Message-ID: > > Are the specifications in the PEPs above implemented somewhere > or are they only 'specifications'? > > > The Packaging User Guide is maintaining a summary of the relevant PEPs > that gives a brief description of the "user impact" and "implementation" > (i.e. whether projects are actually implementing any of this). > https://python-packaging-user-guide.readthedocs.org/en/latest/peps.html > For those working on the various projects, please help keep this up to date. Thanks for the pointer; bookmarked. Thomas From ncoghlan at gmail.com Thu Jan 23 22:48:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Jan 2014 07:48:40 +1000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 January 2014 06:25, Thomas Heller wrote: > Am 23.01.2014 19:52, schrieb Ralf Gommers: >> On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin >> > wrote: >> >> On Thu, Jan 23, 2014 at 12:16:02PM +0000, Paul Moore wrote: >> > >> > The official numpy installer uses some complex magic to select the >> > right binaries based on your CPU, and this means that the official >> > numpy "superpack" wininst files don't convert (at least I don't >> think >> > they do, it's a while since I tried). >> >> It's probably worth noting that numpy are toying around with wheels >> and >> have uploaded a number of them to PyPI for testing: >> http://sourceforge.net/projects/numpy/files/wheels_to_test/ >> >> Currently there are only OSX wheels there (excluding the puer Python >> ones) and they're not available on PyPI. I assume that they're waiting >> for a solution for the Windows installer (a post-install script for >> wheels). That would give a lot more impetus to put wheels up on PyPI. >> >> >> Indeed. We discussed just picking the SSE2 or SSE3 build and putting >> that up as a wheel, but that was deemed a not so great idea: >> http://article.gmane.org/gmane.comp.python.numeric.general/56072 > > > Did I say this before? I would suggest that numpy develops a way > where all the SSE binary variations would be installed, and the > appropriate ones be loaded at runtime, depending on the user's CPU > capabilities. This would also allow py2exe'd distributions to include > them all. I believe I suggested that at one point, but the dependencies were scattered throughout NumPy rather than being in one place (so you had to toggle several modules, rather than just one), and it made for some interesting build problems further up the stack. Agreed that runtime selection would be better, though - the current approach not only makes it difficult to do things like create universal py2exe and wheel binaries, it also makes it difficult to run NumPy from portable media like a USB key, because different systems may need different SSE modules. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Thu Jan 23 22:56:51 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 23 Jan 2014 13:56:51 -0800 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Thu, Jan 23, 2014 at 12:25 PM, Thomas Heller wrote: > Did I say this before? I would suggest that numpy develops a way > where all the SSE binary variations would be installed, and the > appropriate ones be loaded at runtime, depending on the user's CPU > capabilities. This would also allow py2exe'd distributions to include > them all. That was discussed on the numpy list, and would be really nice, but it may also be really difficult. OS-X has built-in support for multi-architecture binaries, but Windows does not, and while selecting a particular .dll (or .pyd) to load at run-time would be fairly straightforward numpy has more than one, and then there is the whole scipy stack, and all the third-party stuff compiled against it. I suspect this wold have to be built-in to the python importing and distutils build system to be workable. But maybe someone smarter than me will figure it out. Some feedback from the people who did try those wheels would help. I >> asked for that on the numpy list after creating them, but didn't get >> much. So I haven't been in a hurry to move them over to PyPi. > > Serious chicken-egg problem there.... > I would have tried wheels for windows, python 3.3 or 3.4, but there > aren't any. Yeah we need to get those up -- SSE2 only ones would work for MOST people. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 24 00:58:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Jan 2014 09:58:29 +1000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 Jan 2014 08:03, "Chris Barker" wrote: > > On Thu, Jan 23, 2014 at 12:25 PM, Thomas Heller wrote: >> >> Did I say this before? I would suggest that numpy develops a way >> where all the SSE binary variations would be installed, and the >> appropriate ones be loaded at runtime, depending on the user's CPU >> capabilities. This would also allow py2exe'd distributions to include >> them all. > > > That was discussed on the numpy list, and would be really nice, but it may also be really difficult. OS-X has built-in support for multi-architecture binaries, but Windows does not, and while selecting a particular .dll (or .pyd) to load at run-time would be fairly straightforward numpy has more than one, and then there is the whole scipy stack, and all the third-party stuff compiled against it. > > I suspect this wold have to be built-in to the python importing and distutils build system to be workable. But maybe someone smarter than me will figure it out. > > >>> Some feedback from the people who did try those wheels would help. I >>> asked for that on the numpy list after creating them, but didn't get >>> much. So I haven't been in a hurry to move them over to PyPi. > > > Serious chicken-egg problem there.... > >> >> I would have tried wheels for windows, python 3.3 or 3.4, but there aren't any. > > > Yeah we need to get those up -- SSE2 only ones would work for MOST people. I really think that's our best near term workaround - still room for improvement, but " pip install numpy assumes SSE2" is a much better situation than "pip install numpy doesn't work on Windows". Such a change would help a lot of people *right now*, while still leaving room to eventually figure out something more sophisticated (like postinstall hooks or simpler runtime multi-build support or NumPy changing to a dependency that internally makes this decision at runtime). Cheers, Nick. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Fri Jan 24 01:17:53 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 24 Jan 2014 00:17:53 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 23 January 2014 23:58, Nick Coghlan wrote: > > I really think that's our best near term workaround - still room for > improvement, but " pip install numpy assumes SSE2" is a much better > situation than "pip install numpy doesn't work on Windows". Is it? Do you have any idea what proportion of (the relevant) people would be using Windows with hardware that doesn't support SSE2? I feel confident that it's less than 10% but I don't know how to justify a tighter bound than that. You need to bear in mind that people currently have a variety of ways to install numpy on Windows that do work already without limitations on CPU instruction set. Most numpy users will not get any immediate benefit from the fact that "it works using pip" rather than "it works using the .exe installer" (or any of a number of other options). It's the unfortunate end users and the numpy folks who would have to pick up the pieces if/when the SSE2 assumption fails. > Such a change would help a lot of people *right now*, while still leaving > room to eventually figure out something more sophisticated (like postinstall > hooks or simpler runtime multi-build support or NumPy changing to a > dependency that internally makes this decision at runtime). Postinstall hooks are not that sophisticated and most packaging systems have them. You're advocating for numpy to take a dodgy compromise here but can it not be the other way round? Wheel, pip etc. could quickly agree on and implement a postinstall hook that would work for numpy and then that could be made more sophisticated later on. Oscar From noah at coderanger.net Fri Jan 24 01:21:26 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 23 Jan 2014 16:21:26 -0800 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: <4067C78B-F5D2-45F8-B321-EA2C9EB92F18@coderanger.net> On Jan 23, 2014, at 4:17 PM, Oscar Benjamin wrote: > On 23 January 2014 23:58, Nick Coghlan wrote: >> >> I really think that's our best near term workaround - still room for >> improvement, but " pip install numpy assumes SSE2" is a much better >> situation than "pip install numpy doesn't work on Windows". > > Is it? Do you have any idea what proportion of (the relevant) people > would be using Windows with hardware that doesn't support SSE2? I feel > confident that it's less than 10% but I don't know how to justify a > tighter bound than that. > > You need to bear in mind that people currently have a variety of ways > to install numpy on Windows that do work already without limitations > on CPU instruction set. Most numpy users will not get any immediate > benefit from the fact that "it works using pip" rather than "it works > using the .exe installer" (or any of a number of other options). It's > the unfortunate end users and the numpy folks who would have to pick > up the pieces if/when the SSE2 assumption fails. > This all sounds very similar to the issues with Linux binary wheels and varying system ABIs. Should probably keep in mind for any solution that might apply to both. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri Jan 24 10:41:57 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 24 Jan 2014 09:41:57 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 January 2014 00:17, Oscar Benjamin wrote: > You need to bear in mind that people currently have a variety of ways > to install numpy on Windows that do work already without limitations > on CPU instruction set. Most numpy users will not get any immediate > benefit from the fact that "it works using pip" rather than "it works > using the .exe installer" (or any of a number of other options). It's > the unfortunate end users and the numpy folks who would have to pick > up the pieces if/when the SSE2 assumption fails. The people who would benefit are those who (like me!) don't have a core requirement for numpy, but who just want to "try it out" casually, or for experimenting or one-off specialised scripts. These are the people who won't be using one of the curated distributions, and quite possibly will be using a virtualenv, so the exe installers won't work. Giving these people a means to try numpy could introduce a wider audience to it. Having said that, I can understand the reluctance to have to deal with non-specialist users hitting obscure "your CPU is too old" errors - that's *not* a good initial experience. And your point that it's just as reasonable for pip to adopt a partial solution in the short term is also fair - although it would be harder for pip to replace an API we added and which people are using, than it would be for numpy to switch to deploying better wheels when the facilities become available. So the comparison isn't entirely equal. Paul From ncoghlan at gmail.com Fri Jan 24 11:18:47 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Jan 2014 20:18:47 +1000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 Jan 2014 19:41, "Paul Moore" wrote: > > On 24 January 2014 00:17, Oscar Benjamin wrote: > > You need to bear in mind that people currently have a variety of ways > > to install numpy on Windows that do work already without limitations > > on CPU instruction set. Most numpy users will not get any immediate > > benefit from the fact that "it works using pip" rather than "it works > > using the .exe installer" (or any of a number of other options). It's > > the unfortunate end users and the numpy folks who would have to pick > > up the pieces if/when the SSE2 assumption fails. > > The people who would benefit are those who (like me!) don't have a > core requirement for numpy, but who just want to "try it out" > casually, or for experimenting or one-off specialised scripts. These > are the people who won't be using one of the curated distributions, > and quite possibly will be using a virtualenv, so the exe installers > won't work. Giving these people a means to try numpy could introduce a > wider audience to it. > > Having said that, I can understand the reluctance to have to deal with > non-specialist users hitting obscure "your CPU is too old" errors - > that's *not* a good initial experience. > > And your point that it's just as reasonable for pip to adopt a partial > solution in the short term is also fair - although it would be harder > for pip to replace an API we added and which people are using, than it > would be for numpy to switch to deploying better wheels when the > facilities become available. So the comparison isn't entirely equal. There's also the fact that we're still trying to recover from the setup.py situation (which was a "quick and easy" alternative to a declarative build system), so quick hacks in the core metadata specs that will then be locked in for years by backwards compatibility requirements are definitely *not* acceptable. We already have more than enough of those in the legacy metadata we're aiming to replace :P All NumPy should need to reduce end user confusion to tolerable levels is an import time CPU check that raises an error that includes a link to stable URL explaining the limitations of the published wheel file, and alternative ways of obtaining NumPy (like Christophe's installers, or a science & data analysis focused distribution like Anaconda or EPD, or bootstrapping conda). In return, as Paul points out, it becomes substantially easier for people that *aren't* wholly invested in the scientific Python stack to try it out with their regular tools, rather than having to completely change how they work with Python. Also consider that, given the status quo, any users that might see that new error instead get even *more* incomprehensible errors as pip attempts to build NumPy from source and fails at doing so. The choice given the current metadata standards isn't between confusing Windows users or not, it's between confusing 100% of those that try "pip install numpy" with cryptic errors from a failed build at install time and confusing a much smaller percentage of those with a CPU compatibility error at runtime. Is the latter a desirable *final* state? No, and metadata 2.0 will aim to address that. It is, however, substantially better than the status quo and doesn't run the risk of compromising interoperability standards we're going to have to live with indefinitely. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at notevencode.com Fri Jan 24 11:09:57 2014 From: matt at notevencode.com (Matthew Iversen) Date: Fri, 24 Jan 2014 21:09:57 +1100 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: Might I suggest you could upload some wheels (both windows and linux) to testpypi, which afaik is pretty much made for this purpose? https://wiki.python.org/moin/TestPyPI People can easily install then with e.g. `pip install --index-url https://testpypi.python.org/pypi numpy`, and see what tends to break or what doesn't. On 24 January 2014 05:52, Ralf Gommers wrote: > > > > On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin < > oscar.j.benjamin at gmail.com> wrote: > >> On Thu, Jan 23, 2014 at 12:16:02PM +0000, Paul Moore wrote: >> > >> > The official numpy installer uses some complex magic to select the >> > right binaries based on your CPU, and this means that the official >> > numpy "superpack" wininst files don't convert (at least I don't think >> > they do, it's a while since I tried). >> >> It's probably worth noting that numpy are toying around with wheels and >> have uploaded a number of them to PyPI for testing: >> http://sourceforge.net/projects/numpy/files/wheels_to_test/ >> >> Currently there are only OSX wheels there (excluding the puer Python >> ones) and they're not available on PyPI. I assume that they're waiting >> for a solution for the Windows installer (a post-install script for >> wheels). That would give a lot more impetus to put wheels up on PyPI. >> > > Indeed. We discussed just picking the SSE2 or SSE3 build and putting that > up as a wheel, but that was deemed a not so great idea: > http://article.gmane.org/gmane.comp.python.numeric.general/56072 > > The Sourceforge OSX wheels are presumably not getting that much use >> right now. The OSX-specific numpy wheel has been downloaded 4 times in >> the last week: twice on Windows and twice on Linux! >> > > Some feedback from the people who did try those wheels would help. I asked > for that on the numpy list after creating them, but didn't get much. So I > haven't been in a hurry to move them over to PyPi. > > Ralf > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 24 14:34:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Jan 2014 23:34:00 +1000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 January 2014 20:09, Matthew Iversen wrote: > Might I suggest you could upload some wheels (both windows and linux) to > testpypi, which afaik is pretty much made for this purpose? Well, Windows and Mac OS X - we don't allow PyPI wheels for Linux at the moment (since it turns out the compatibility tagging needs to be updated to distinguish distros as well before that's a good idea), although Armin Ronacher pointed out our documentation of that limitation is rather lacking at this point in time :( > https://wiki.python.org/moin/TestPyPI > > People can easily install then with e.g. `pip install --index-url > https://testpypi.python.org/pypi numpy`, and see what tends to break or what > doesn't. And yes, using testpypi to experiment with wheels is a good idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Fri Jan 24 23:21:54 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 24 Jan 2014 14:21:54 -0800 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Fri, Jan 24, 2014 at 2:18 AM, Nick Coghlan wrote: > In return, as Paul points out, it becomes substantially easier for people > that *aren't* wholly invested in the scientific Python stack to try it out > with their regular tools, rather than having to completely change how they > work with Python. > This is a really important constituency, actually. And one that has been neglected for a while. > Also consider that, given the status quo, any users that might see that > new error instead get even *more* incomprehensible errors as pip attempts > to build NumPy from source and fails at doing so. > well, numpy _should_ build out of the box with nothing special if you are set up to build regular extensions. I understand that a lto f Windows users are not set up to build extensions at all, but tehy ar presumably used to getting "compiler not found" errors (or whatever the message is). But you won't get an optimized numpy and much of the rest of the "stack" is harder to build: scipy, matplotlib. So a set of working binary wheels would be great. And while we in the numpy commmunity don't really want a lot of "numpy is slower than MATLAB" FUD out there, I still think it's better to get a sub-optimum, but working build out there. The "should I use python instead of MATLAB?" crowd would be better served by one of the other options anyway... So how rare are non-SSE2 systems? Any w ay to find out? I"m guessing rare enough that we can a) not worry about it, and b) those users will know they have an old system and may expect issue,s particularly with something billed as being for high-performance computation. So I say SSE2 -- but if we do think there ar a lot of non-SSE2 users out there, then do SSE1-only , it would still work just fine for casual use. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Jan 24 23:40:11 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 24 Jan 2014 22:40:11 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 January 2014 22:21, Chris Barker wrote: > well, numpy _should_ build out of the box with nothing special if you are > set up to build regular extensions. I understand that a lto f Windows users > are not set up to build extensions at all, but tehy ar presumably used to > getting "compiler not found" errors (or whatever the message is). But you > won't get an optimized numpy and much of the rest of the "stack" is harder > to build: scipy, matplotlib. Seriously? If I have MSVC 2010 installed, pip install numpy will correctly build numpy from source? It's a *long* time since I tried this, but I really thought building numpy was harder than that. A quick test later: No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the rest of the error stack (which I'm frankly not interested in investing the time to diagnose and fix) I get "RuntimeError: Broken toolchain: cannot link a simple C program". Which is utter rubbish - I routinely build extensions with this installation. So no, numpy does not build out of the box. Ah well. Paul From chris.barker at noaa.gov Fri Jan 24 23:51:14 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 24 Jan 2014 14:51:14 -0800 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Fri, Jan 24, 2014 at 2:40 PM, Paul Moore wrote: > So no, numpy does not build out of the box. Ah well. > Darn -- it used to, and it should. It has shipped for years with an "LAPACK light", and shouldn't need any fortran. It used to not even look for LAPACK with a default configuration. But I haven't done if or years, so who know when this might have been broken? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Sat Jan 25 22:33:15 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 25 Jan 2014 21:33:15 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 January 2014 22:40, Paul Moore wrote: > On 24 January 2014 22:21, Chris Barker wrote: >> well, numpy _should_ build out of the box with nothing special if you are >> set up to build regular extensions. I understand that a lto f Windows users >> are not set up to build extensions at all, but tehy ar presumably used to >> getting "compiler not found" errors (or whatever the message is). But you >> won't get an optimized numpy and much of the rest of the "stack" is harder >> to build: scipy, matplotlib. > > Seriously? If I have MSVC 2010 installed, pip install numpy will > correctly build numpy from source? It's a *long* time since I tried > this, but I really thought building numpy was harder than that. > > A quick test later: > No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the > rest of the error stack (which I'm frankly not interested in investing > the time to diagnose and fix) I get "RuntimeError: Broken toolchain: > cannot link a simple C program". Which is utter rubbish - I routinely > build extensions with this installation. > > So no, numpy does not build out of the box. Ah well. Last time I tried with mingw it worked (I've since departed the Windows world). I think official numpy binaries for Windows are built with mingw (Christoph uses MSVC though). Oscar From p.f.moore at gmail.com Sat Jan 25 22:46:41 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 25 Jan 2014 21:46:41 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 25 January 2014 21:33, Oscar Benjamin wrote: > Last time I tried with mingw it worked (I've since departed the > Windows world). I think official numpy binaries for Windows are built > with mingw (Christoph uses MSVC though). That may well be the case, but MSVC (Express or full) is the "standard" for building extensions - it's reasonable to expect that that is what the "casual user" we're talking about here would have (anyone who has taken the time to set up mingw for builds can be assumed to be "experienced" to some level or other...) Paul From oscar.j.benjamin at gmail.com Sat Jan 25 22:56:09 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 25 Jan 2014 21:56:09 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 24 January 2014 10:18, Nick Coghlan wrote: > > On 24 Jan 2014 19:41, "Paul Moore" wrote: >> >> On 24 January 2014 00:17, Oscar Benjamin >> wrote: >> > You need to bear in mind that people currently have a variety of ways >> > to install numpy on Windows that do work already without limitations >> > on CPU instruction set. Most numpy users will not get any immediate >> > benefit from the fact that "it works using pip" rather than "it works >> > using the .exe installer" (or any of a number of other options). It's >> > the unfortunate end users and the numpy folks who would have to pick >> > up the pieces if/when the SSE2 assumption fails. >> >> The people who would benefit are those who (like me!) don't have a >> core requirement for numpy, but who just want to "try it out" >> casually, or for experimenting or one-off specialised scripts. These >> are the people who won't be using one of the curated distributions, >> and quite possibly will be using a virtualenv, so the exe installers >> won't work. Giving these people a means to try numpy could introduce a >> wider audience to it. >> >> Having said that, I can understand the reluctance to have to deal with >> non-specialist users hitting obscure "your CPU is too old" errors - >> that's *not* a good initial experience. >> >> And your point that it's just as reasonable for pip to adopt a partial >> solution in the short term is also fair - although it would be harder >> for pip to replace an API we added and which people are using, than it >> would be for numpy to switch to deploying better wheels when the >> facilities become available. So the comparison isn't entirely equal. > > There's also the fact that we're still trying to recover from the setup.py > situation (which was a "quick and easy" alternative to a declarative build > system), so quick hacks in the core metadata specs that will then be locked > in for years by backwards compatibility requirements are definitely *not* > acceptable. We already have more than enough of those in the legacy metadata > we're aiming to replace :P It wasn't a totally serious suggestion: I knew what your response would be. ;) I'll try to summarise your take on this: You would like to take the time to ensure that Python packaging is done properly. That may mean that some functionality isn't available for some time, but you think that it's better to "get it right" than rush something out the door just to "get it working fast". That's not an unreasonable position to take but I wanted to contrast that with your advice to numpy: Just rush something out of the door even if it has obvious problems. Don't worry about getting it right; we'll do that later... We all want a solution that definitely works so that you can advise any old noob to use it. So if you could say 'just use pip' then that would be great. But if you say... ''' 'just use pip... unless your CPU doesn't support SSE2. Don't worry if you've never heard of SSE2 just do 'pip install numpy' and then 'python -c "import numpy"'. If you see an error like "your CPU doesn't support SSE2 please install the non-SSE version of numpy." then you'll need to install numpy using one of the other options listed below and make sure that you do that before trying to use pip to install any of these other packages and if you use Cristoph's .exe for numpy then the you can't use pip for scipy and some other set of packages (I'm not totally sure which) so you shouldn't use pip for anything after that. Unless it's a pure Python package. Don't worry if you don't know what a pure Python package is, just try it with pip and if it doesn't work just try something else... """ ... then putting the wheel on PyPI becomes substantially less attractive. Just having to explain that pip might not work and then trying to describe when it will and won't and what to do about it is a pain. I wouldn't want to recommend to my students that they do this unless I was confident that it would work. Also, note that I don't really think a post-install script is the best solution for something like this. It would be better to have an extensible system for querying things like CPU capability. It would also be better to have an extensible system for detecting things like Fortran ABI compatibility - this can also be handled with a post-install script but it's not the best solution. Are there any plans to solve these problems? Also is there a roadmap describing the expected timeline for future packaging features? Oscar From p.f.moore at gmail.com Sun Jan 26 00:02:10 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 25 Jan 2014 23:02:10 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 25 January 2014 21:56, Oscar Benjamin wrote: > I'll try to summarise your take on this: You would like to take the > time to ensure that Python packaging is done properly. That may mean > that some functionality isn't available for some time, but you think > that it's better to "get it right" than rush something out the door > just to "get it working fast". > > That's not an unreasonable position to take but I wanted to contrast > that with your advice to numpy: Just rush something out of the door > even if it has obvious problems. Don't worry about getting it right; > we'll do that later... Just to be clear, that's *not* my position. (It may be the position of some of the other pip developers). My view is that the sooner "pip install X" works out of the box, for as many of the cases where it didn't in the past, the faster we'll get adoption and the sooner people will start reporting any remaining issues. It won't be a perfect solution, but it will be better than the current status quo (at least, I'm looking for solutions that *are* better than the status quo :-)). I'd also like to see some visible impact from wheels - at the moment, even though (for example) pip and setuptools publish wheels, I doubt anyone sees any difference. I sort of wish we hadn't managed to make wheels quite so transparent in use, ironically :-) Wheels are an improvement over the wininst status quo because they support virtualenvs. So I want to see numpy wheels available, because that would be a significant step in making people aware of some of the improvements we're making. I do *not* want to see existing numpy users, or the specialists already well-served by the scientific python community, being harmed by the existence of wheels - but my impression was that the advice given to people who want to use numpy/scipy seriously, is to use one of the curated stacks like conda or enthought. So I'm looking at people who at the moment don't use numpy, but are somewhat interested in trying it out (not enough to install a whole new Python stack, though). Those people currently either use the wininst installers (which I'm *not* advocating that we remove) or they use virtualenvs, and have the view that they have to at a minimum jump through some hoops to get numpy to work. It's purely the people who can't use the wininst installers, and don't want a curated stack, that are my focus at the moment. It seems to me that there are a number of solutions for them: 1. Ignore them. It's a small enough group as to not matter. I have a personal dislike of this option, because I'm in this group :-) But it *is* a fair position to take. 2. The pip developers add facilities to pip to allow the numpy folks to generate multi-architecture wheels 3. The numpy folks put up interim wheels that work for some, but not all, users. Option 2 has an issue because developing a "proper" solution is still a long way off. But a "quick fix" postinstall script solution is possible. However, even this requires some development work, and once that has been done, a new pip release is needed, and the numpy folks then need to add the relevant postinstall scripts and update their build process to incorporate the multi-architecture DLLs into their wheels. So it's a non-trivial amount of work and time, even if it's better than a "full" solution. Option 3 has a problem because there's a support problem with people who try to use the wheels and get obscure runtime errors. But that's the only cost I can see - the numpy people can build wheels by just adding "setup.py bdist_wheel" to their build process alongside the current "setup.py bdist_wininst", and then upload them. Maybe to testpypi if it's important to make using the wheels an opt-in process. Also, maybe we could distributw a script that allowed people to check in advance if the numpy wheels would work on their PC. I'm not giving conflicting advice here - all I'm doing is looking for the short-term action that gives the most benefit for the least cost. It seems to me that option 2 takes longer and involves more effort than option 3. I'm not even convinced that option 2 is less effort for the numpy developers (if we ignore the pip development work). But I know nothing about the numpy build process, so my assumptions could be way off here - I'd be more than happy for someone to clarify what effort is involved in publishing wheels for numpy (which already exist, is that right?). As general information, it would be very valuable, because I'm pretty sure most people round here assume it's little more than adding "bdist_wheel" to an existing binary distribution production process. My apologies for this email being so long, but hopefully it's explained my position more clearly than I seem to have done previously. Paul From donald at stufft.io Sun Jan 26 00:38:53 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 25 Jan 2014 18:38:53 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage Message-ID: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new infrastructure. The old infrastructure was: - a single database server managed by OSUOSL - a pair of load balancers shared by all of the python.org services hosted on OSUOSL - a single backend VM that served as everything else for PyPI The VM that was acting as the backend server from PyPI was partially hand configured and partially setup to be managed by chef. Additionally it had an issue that caused it to kernel panic every so often which had been the cause of a number of downtimes in the last few months. Because it was primarily configured and administered by hand and because the way it was set up it was not feasible to have any sort of failover or spare. The new infrastructure is: - 2 Web VMs - 2 Database servers in Master/Slave Configuration - 2 PgPool Servers pooling connections to the database servers and load balancing reads across them. - 2 GlusterFS servers backed by Cloud Block Storage acting as the file storage for package and package docs - 1 metrics server to handle updating the download counts as they come in from Fastly All of the VMs are hosted on Rackspace?s Public Cloud and have their configuration completely controlled and managed using Salt. Going forward this will allow us to easily scale out as required or kill malfunctioning servers and spin up new ones easily. Additionally the setup has been setup so that where possible there is two servers performing the same role, ideally in an Active/Active configuration but at least in a Master/Slave configuration. This should allow PyPI to be far more stable moving forward and make downtimes much easier to recover from. The services are still fronted by Fasty?s CDN and in the new infrastructure we?ve removed our load balancer and have replaced it by having Fastly handle the load balancing for us. Additionally we?ve recently setup a static mirror of PyPI that is updated once every minute. This is hosted on Rackspace cloud as well but in a separate data center from the rest of PyPI. Fastly is configured to fall back to this static mirror in the case that neither of the two web heads are functioning. This should ensure that even in the event of a catastrophic failure of the PyPI service that the bulk of package installations should hopefully remain working. The bad news, (and the ?Breakage? from the subject) is that while the new infrastructure was being planned out, built, and migrated to the ?pypissh? package was forgotten. The pypissh package is an alternative way to upload packages to PyPI however it is very difficult, because of the way it works, to provide HA support for it as we?ve set up for everything else. We don?t have any numbers for how many people are actively using this package but looking at a roughly 2 week chunk of time in PyPI?s download history, the pypissh package was downloaded 7 times by a browser, and 7 times by pip. All other downloads were caused by the mirroring system. As of right now pypissh is non functional and due to the difficulty in HAing and monitoring the current setup and because it is apparently has a very small set of users we would like to effectively kill off this particular service. Additionally the benefits of pypissh have been reduced now that PyPI is available over a TLS connection with a well trusted certificate. My question to you is, is this something that distutils-sig is willing to have happen? If we are to re-enable pypissh we?ll need to write a new solution to doing it that can be properly HA?d and we?d prefer to put our efforts into improving things for a much larger set of people. So yea, PyPI should be loads more stable and more reliable now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Sun Jan 26 00:36:42 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 25 Jan 2014 23:36:42 +0000 (GMT) Subject: [Distutils] pip on windows experience In-Reply-To: Message-ID: <1390693002.71907.YahooMailBasic@web172403.mail.ir2.yahoo.com> -------------------------------------------- On Sat, 25/1/14, Paul Moore wrote: > Wheels are an improvement over the wininst status quo because they > support virtualenvs. So I want to see numpy wheels available, because > that would be a significant step in making people aware of some of the > improvements we're making. ... > Those people currently either use the wininst installers (which I'm > *not* advocating that we remove) or they use virtualenvs, and have the > view that they have to at a minimum jump through some hoops to get > numpy to work. I don't know if you're aware that recent versions of distil can convert bdist_wininst installers to wheels. The feature has not been extensively tested, but AFAIK it basically works. For example, with one of Christoph Gohlke's installers (I just happened to run the conversion on POSIX, it should work the same way on Windows): $ distil package --format=wheel /tmp/numpy-MKL-1.8.0.win32-py3.3.exe The following packages were built: /tmp/numpy-1.8.0-cp33-none-win32.whl Installing on Windows in a freshly created venv created with python -m venv:: C:\Documents and Settings\Vinay>distil -e \temp\venv33 install \temp\numpy-1.8.0-cp33-none-win32.whl Checking requirements for numpy (1.8.0) ... done. The following new packages will be installed from a local location: numpy (1.8.0) Installing numpy (1.8.0) ... Installation completed. C:\Documents and Settings\Vinay>\temp\venv33\Scripts\python ActivePython 3.3.2.0 (ActiveState Software Inc.) based on Python 3.3.2 (default, Sep 16 2013, 23:10:06) [MSC v.1600 32 bit (Intel)] on win 32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__file__ 'C:\\temp\\venv33\\lib\\site-packages\\numpy\\__init__.py' >>> Of course, that doesn't prove much, other than that the conversion and installation processes can be fairly painless. I'd be interested in any feedback about these distil features. Regards, Vinay Sajip From richard at python.org Sun Jan 26 00:50:26 2014 From: richard at python.org (Richard Jones) Date: Sun, 26 Jan 2014 10:50:26 +1100 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: Thanks everyone who helped make this happen. >From my perspective* I believe the ssh upload mechanism was added to address security issues around the basic-auth-over-http method used historically. Now uploads *may* be done over https, and those using the ssh method can move over to using twine or pip upload, I think that it's reasonable to discontinue support for ssh uploads. Richard * as the guy who will be hassled if its loss is noticed ;) On 26 January 2014 10:38, Donald Stufft wrote: > Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new > infrastructure. > > The old infrastructure was: > > - a single database server managed by OSUOSL > - a pair of load balancers shared by all of the python.org services > hosted on > OSUOSL > - a single backend VM that served as everything else for PyPI > > The VM that was acting as the backend server from PyPI was partially hand > configured and partially setup to be managed by chef. Additionally it had > an > issue that caused it to kernel panic every so often which had been the > cause of > a number of downtimes in the last few months. Because it was primarily > configured and administered by hand and because the way it was set up it > was > not feasible to have any sort of failover or spare. > > The new infrastructure is: > > - 2 Web VMs > - 2 Database servers in Master/Slave Configuration > - 2 PgPool Servers pooling connections to the database servers and load > balancing reads across them. > - 2 GlusterFS servers backed by Cloud Block Storage acting as the file > storage > for package and package docs > - 1 metrics server to handle updating the download counts as they come in > from > Fastly > > All of the VMs are hosted on Rackspace?s Public Cloud and have their > configuration completely controlled and managed using Salt. Going forward > this > will allow us to easily scale out as required or kill malfunctioning > servers > and spin up new ones easily. Additionally the setup has been setup so that > where possible there is two servers performing the same role, ideally in an > Active/Active configuration but at least in a Master/Slave configuration. > This > should allow PyPI to be far more stable moving forward and make downtimes > much > easier to recover from. > > The services are still fronted by Fasty?s CDN and in the new infrastructure > we?ve removed our load balancer and have replaced it by having Fastly > handle > the load balancing for us. Additionally we?ve recently setup a static > mirror of > PyPI that is updated once every minute. This is hosted on Rackspace cloud > as > well but in a separate data center from the rest of PyPI. Fastly is > configured > to fall back to this static mirror in the case that neither of the two web > heads are functioning. This should ensure that even in the event of a > catastrophic failure of the PyPI service that the bulk of package > installations > should hopefully remain working. > > The bad news, (and the ?Breakage? from the subject) is that while the new > infrastructure was being planned out, built, and migrated to the ?pypissh? > package was forgotten. The pypissh package is an alternative way to upload > packages to PyPI however it is very difficult, because of the way it > works, to > provide HA support for it as we?ve set up for everything else. We don?t > have > any numbers for how many people are actively using this package but looking > at a roughly 2 week chunk of time in PyPI?s download history, the pypissh > package was downloaded 7 times by a browser, and 7 times by pip. All other > downloads were caused by the mirroring system. > > As of right now pypissh is non functional and due to the difficulty in > HAing > and monitoring the current setup and because it is apparently has a very > small set of users we would like to effectively kill off this particular > service. Additionally the benefits of pypissh have been reduced now that > PyPI > is available over a TLS connection with a well trusted certificate. My > question > to you is, is this something that distutils-sig is willing to have happen? > If > we are to re-enable pypissh we?ll need to write a new solution to doing it > that > can be properly HA?d and we?d prefer to put our efforts into improving > things > for a much larger set of people. > > So yea, PyPI should be loads more stable and more reliable now. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jan 26 00:55:30 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 25 Jan 2014 23:55:30 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: <1390693002.71907.YahooMailBasic@web172403.mail.ir2.yahoo.com> References: <1390693002.71907.YahooMailBasic@web172403.mail.ir2.yahoo.com> Message-ID: On 25 January 2014 23:36, Vinay Sajip wrote: >> Those people currently either use the wininst installers (which I'm >> *not* advocating that we remove) or they use virtualenvs, and have the >> view that they have to at a minimum jump through some hoops to get >> numpy to work. > > I don't know if you're aware that recent versions of distil can convert bdist_wininst installers to wheels. The feature has not been extensively tested, but AFAIK it basically works. For example, with one of Christoph Gohlke's installers (I just happened to run the conversion on POSIX, it should work the same way on Windows): Wheel convert also has that ability pretty much since the start - I added the code for it . But yes, it's what I meant by "jumping through hoops" Just for information, you can also convert the official numpy superpack exes, but they are multiple wininsts combined in a single exe, and you have to open them up in a zip manager, grab the individual wininsts out, and convert the one you need. (If I recall the process, it was some time ago and may have changed). Paul From chris.jerdonek at gmail.com Sun Jan 26 01:04:20 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 25 Jan 2014 16:04:20 -0800 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: On Sat, Jan 25, 2014 at 3:38 PM, Donald Stufft wrote: > Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new > infrastructure. > > The old infrastructure was: > > - a single database server managed by OSUOSL > - a pair of load balancers shared by all of the python.org services hosted on > OSUOSL > - a single backend VM that served as everything else for PyPI > > The VM that was acting as the backend server from PyPI was partially hand > configured and partially setup to be managed by chef. Additionally it had an > issue that caused it to kernel panic every so often which had been the cause of > a number of downtimes in the last few months. Because it was primarily > configured and administered by hand and because the way it was set up it was > not feasible to have any sort of failover or spare. > > The new infrastructure is: > > - 2 Web VMs > - 2 Database servers in Master/Slave Configuration > - 2 PgPool Servers pooling connections to the database servers and load > balancing reads across them. > - 2 GlusterFS servers backed by Cloud Block Storage acting as the file storage > for package and package docs > - 1 metrics server to handle updating the download counts as they come in from > Fastly > > All of the VMs are hosted on Rackspace?s Public Cloud and have their > configuration completely controlled and managed using Salt. Going forward this Can you say a little about the choice to use Salt instead of Chef? I don't really care either way, but am just curious. Is it because Salt is written in Python, or were there other reasons (functionality, etc)? --Chris > will allow us to easily scale out as required or kill malfunctioning servers > and spin up new ones easily. Additionally the setup has been setup so that > where possible there is two servers performing the same role, ideally in an > Active/Active configuration but at least in a Master/Slave configuration. This > should allow PyPI to be far more stable moving forward and make downtimes much > easier to recover from. > > The services are still fronted by Fasty?s CDN and in the new infrastructure > we?ve removed our load balancer and have replaced it by having Fastly handle > the load balancing for us. Additionally we?ve recently setup a static mirror of > PyPI that is updated once every minute. This is hosted on Rackspace cloud as > well but in a separate data center from the rest of PyPI. Fastly is configured > to fall back to this static mirror in the case that neither of the two web > heads are functioning. This should ensure that even in the event of a > catastrophic failure of the PyPI service that the bulk of package installations > should hopefully remain working. > > The bad news, (and the ?Breakage? from the subject) is that while the new > infrastructure was being planned out, built, and migrated to the ?pypissh? > package was forgotten. The pypissh package is an alternative way to upload > packages to PyPI however it is very difficult, because of the way it works, to > provide HA support for it as we?ve set up for everything else. We don?t have > any numbers for how many people are actively using this package but looking > at a roughly 2 week chunk of time in PyPI?s download history, the pypissh > package was downloaded 7 times by a browser, and 7 times by pip. All other > downloads were caused by the mirroring system. > > As of right now pypissh is non functional and due to the difficulty in HAing > and monitoring the current setup and because it is apparently has a very > small set of users we would like to effectively kill off this particular > service. Additionally the benefits of pypissh have been reduced now that PyPI > is available over a TLS connection with a well trusted certificate. My question > to you is, is this something that distutils-sig is willing to have happen? If > we are to re-enable pypissh we?ll need to write a new solution to doing it that > can be properly HA?d and we?d prefer to put our efforts into improving things > for a much larger set of people. > > So yea, PyPI should be loads more stable and more reliable now. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From donald at stufft.io Sun Jan 26 01:18:12 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 25 Jan 2014 19:18:12 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: <60552257-3130-4B19-9F28-E6C2716C68C8@stufft.io> On Jan 25, 2014, at 7:04 PM, Chris Jerdonek wrote: > On Sat, Jan 25, 2014 at 3:38 PM, Donald Stufft wrote: >> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new >> infrastructure. >> >> The old infrastructure was: >> >> - a single database server managed by OSUOSL >> - a pair of load balancers shared by all of the python.org services hosted on >> OSUOSL >> - a single backend VM that served as everything else for PyPI >> >> The VM that was acting as the backend server from PyPI was partially hand >> configured and partially setup to be managed by chef. Additionally it had an >> issue that caused it to kernel panic every so often which had been the cause of >> a number of downtimes in the last few months. Because it was primarily >> configured and administered by hand and because the way it was set up it was >> not feasible to have any sort of failover or spare. >> >> The new infrastructure is: >> >> - 2 Web VMs >> - 2 Database servers in Master/Slave Configuration >> - 2 PgPool Servers pooling connections to the database servers and load >> balancing reads across them. >> - 2 GlusterFS servers backed by Cloud Block Storage acting as the file storage >> for package and package docs >> - 1 metrics server to handle updating the download counts as they come in from >> Fastly >> >> All of the VMs are hosted on Rackspace?s Public Cloud and have their >> configuration completely controlled and managed using Salt. Going forward this > > Can you say a little about the choice to use Salt instead of Chef? I > don't really care either way, but am just curious. Is it because Salt > is written in Python, or were there other reasons (functionality, > etc)? > > --Chris I?d need to ask Ernest to be sure, but I believe it was mostly that he was more familiar with it. The fact that it was written in Python was a bonus as well ;) I don?t think that there was anything that Chef was missing or that Salt had over Chef, just familiarity of the person who did most of the work. I?ll double check with Ernest to make sure there wasn?t another reason :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Jan 26 01:29:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Jan 2014 10:29:20 +1000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: Paul's position exactly mirrors my own - I an perfectly fine with the recommended advice to scientific users continuing to be "NumPy doesn't officially support pip and virtualenv because of the way it is built and installed, so you will have to get one of the curated scientific stacks, bootstrap conda, use the Windows installers or use the version provided by your Linux distro vendor." The metadata 2.0 standards *will not* be accepted until the pip 1.6 or 1.7 time frame, and it's more likely the latter, since I don't want to distract anyone from the current security work (I know I said otherwise recently, but I managed to temporarily forget that the Warehouse transition and implementing PEP 458 was next on the to do list when I said that). So, if the NumPy community choose to wait for general post-install script support in wheel files, they're likely to be waiting at least until the release of pip 1.7. In the meantime, the failure mode for people attempting to try out the Scientific Python stack via "pip install numpy" in an existing Python installation or virtualenv will remain a failure to build with a likely cryptic error. I do see a few possible workarounds, but none of them would change the metadata 2.0 standards: 1. Add explicit NumPy support *directly* to pip. This would be the quick hack, private API support that Oscar is requesting, since it would be a special arrangement between the pip devs and the numpy devs, and eventually replaced by a general purpose post-install mechanism in metadata 2.0. 2. Add support to pip to request the conversion of available wininst installers (and bdist_dumb?) to wheels for installation with pip. Vinay has this working from a technical perspective, so it may be something the pip devs are interested in exploring for pip 1.6. 3. Both of the above options require waiting for pip 1.6 (at the earliest), which means neither will improve the behaviour in CPython 3.4 (which will ship pip 1.5.1). The only folks with the power to improve *that* situation are the NumPy devs, who have the ability to choose between the "doesn't work for anyone except experienced build engineers" status quo and "works for a large percentage of users, but will still fail at runtime for users on hardware without SSE2 support". To put the "but what if the user doesn't have SSE2 support?" concern in context, it should only affect Intel users with CPUs older than a Pentium 4 (released 2001), and AMD users with a CPU older than an Opteron or Athlon 64 (both released 2003). All x86/x86_64 CPUs released in the past decade should be able to handle SSE2 binaries, so our caveat can be "if your computer is more than a decade old, 'pip install numpy' may not work for you, but it should do the right thing on newer systems". Now, the NumPy devs may feel that persisting with the status quo for another 6 to 12 months while waiting for still hypothetical additional changes in pip specifically to accommodate NumPy's current installation practices is a better alternative than taking option 3. However, from my perspective, having NumPy readily available to users using the python.orgWindows installers for Python 3.4 would *significantly* lower the barrier to entry to the Scientific Python stack for new users on relatively modern systems when compared to the 4 current options (while we accept the Linux distro problem is on distutils-sig to deal with, that's far from being a NumPy specific problem). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jan 26 01:40:45 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Jan 2014 10:40:45 +1000 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: On 26 Jan 2014 09:51, "Richard Jones" wrote: > > Thanks everyone who helped make this happen. Indeed - fine work! :) > From my perspective* I believe the ssh upload mechanism was added to address security issues around the basic-auth-over-http method used historically. Now uploads *may* be done over https, and those using the ssh method can move over to using twine or pip upload, I think that it's reasonable to discontinue support for ssh uploads. Yes, I agree that pointing the (very few) pypissh users towards twine as a replacement is the most reasonable option at this point - we should chat to MvL about putting a notice to that effect in the pypissh README (IIRC, MvL is the creator of that upload option). Cheers, Nick. > > > Richard > > * as the guy who will be hassled if its loss is noticed ;) > > > On 26 January 2014 10:38, Donald Stufft wrote: >> >> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new >> infrastructure. >> >> The old infrastructure was: >> >> - a single database server managed by OSUOSL >> - a pair of load balancers shared by all of the python.org services hosted on >> OSUOSL >> - a single backend VM that served as everything else for PyPI >> >> The VM that was acting as the backend server from PyPI was partially hand >> configured and partially setup to be managed by chef. Additionally it had an >> issue that caused it to kernel panic every so often which had been the cause of >> a number of downtimes in the last few months. Because it was primarily >> configured and administered by hand and because the way it was set up it was >> not feasible to have any sort of failover or spare. >> >> The new infrastructure is: >> >> - 2 Web VMs >> - 2 Database servers in Master/Slave Configuration >> - 2 PgPool Servers pooling connections to the database servers and load >> balancing reads across them. >> - 2 GlusterFS servers backed by Cloud Block Storage acting as the file storage >> for package and package docs >> - 1 metrics server to handle updating the download counts as they come in from >> Fastly >> >> All of the VMs are hosted on Rackspace?s Public Cloud and have their >> configuration completely controlled and managed using Salt. Going forward this >> will allow us to easily scale out as required or kill malfunctioning servers >> and spin up new ones easily. Additionally the setup has been setup so that >> where possible there is two servers performing the same role, ideally in an >> Active/Active configuration but at least in a Master/Slave configuration. This >> should allow PyPI to be far more stable moving forward and make downtimes much >> easier to recover from. >> >> The services are still fronted by Fasty?s CDN and in the new infrastructure >> we?ve removed our load balancer and have replaced it by having Fastly handle >> the load balancing for us. Additionally we?ve recently setup a static mirror of >> PyPI that is updated once every minute. This is hosted on Rackspace cloud as >> well but in a separate data center from the rest of PyPI. Fastly is configured >> to fall back to this static mirror in the case that neither of the two web >> heads are functioning. This should ensure that even in the event of a >> catastrophic failure of the PyPI service that the bulk of package installations >> should hopefully remain working. >> >> The bad news, (and the ?Breakage? from the subject) is that while the new >> infrastructure was being planned out, built, and migrated to the ?pypissh? >> package was forgotten. The pypissh package is an alternative way to upload >> packages to PyPI however it is very difficult, because of the way it works, to >> provide HA support for it as we?ve set up for everything else. We don?t have >> any numbers for how many people are actively using this package but looking >> at a roughly 2 week chunk of time in PyPI?s download history, the pypissh >> package was downloaded 7 times by a browser, and 7 times by pip. All other >> downloads were caused by the mirroring system. >> >> As of right now pypissh is non functional and due to the difficulty in HAing >> and monitoring the current setup and because it is apparently has a very >> small set of users we would like to effectively kill off this particular >> service. Additionally the benefits of pypissh have been reduced now that PyPI >> is available over a TLS connection with a well trusted certificate. My question >> to you is, is this something that distutils-sig is willing to have happen? If >> we are to re-enable pypissh we?ll need to write a new solution to doing it that >> can be properly HA?d and we?d prefer to put our efforts into improving things >> for a much larger set of people. >> >> So yea, PyPI should be loads more stable and more reliable now. >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Sun Jan 26 03:07:15 2014 From: richard at python.org (Richard Jones) Date: Sun, 26 Jan 2014 13:07:15 +1100 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: I'll get in touch with Martin. On 26 January 2014 11:40, Nick Coghlan wrote: > > On 26 Jan 2014 09:51, "Richard Jones" wrote: > > > > Thanks everyone who helped make this happen. > > Indeed - fine work! :) > > > From my perspective* I believe the ssh upload mechanism was added to > address security issues around the basic-auth-over-http method used > historically. Now uploads *may* be done over https, and those using the ssh > method can move over to using twine or pip upload, I think that it's > reasonable to discontinue support for ssh uploads. > > Yes, I agree that pointing the (very few) pypissh users towards twine as a > replacement is the most reasonable option at this point - we should chat to > MvL about putting a notice to that effect in the pypissh README (IIRC, MvL > is the creator of that upload option). > > Cheers, > Nick. > > > > > > > Richard > > > > * as the guy who will be hassled if its loss is noticed ;) > > > > > > On 26 January 2014 10:38, Donald Stufft wrote: > >> > >> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to > new > >> infrastructure. > >> > >> The old infrastructure was: > >> > >> - a single database server managed by OSUOSL > >> - a pair of load balancers shared by all of the python.org services > hosted on > >> OSUOSL > >> - a single backend VM that served as everything else for PyPI > >> > >> The VM that was acting as the backend server from PyPI was partially > hand > >> configured and partially setup to be managed by chef. Additionally it > had an > >> issue that caused it to kernel panic every so often which had been the > cause of > >> a number of downtimes in the last few months. Because it was primarily > >> configured and administered by hand and because the way it was set up > it was > >> not feasible to have any sort of failover or spare. > >> > >> The new infrastructure is: > >> > >> - 2 Web VMs > >> - 2 Database servers in Master/Slave Configuration > >> - 2 PgPool Servers pooling connections to the database servers and load > >> balancing reads across them. > >> - 2 GlusterFS servers backed by Cloud Block Storage acting as the file > storage > >> for package and package docs > >> - 1 metrics server to handle updating the download counts as they come > in from > >> Fastly > >> > >> All of the VMs are hosted on Rackspace?s Public Cloud and have their > >> configuration completely controlled and managed using Salt. Going > forward this > >> will allow us to easily scale out as required or kill malfunctioning > servers > >> and spin up new ones easily. Additionally the setup has been setup so > that > >> where possible there is two servers performing the same role, ideally > in an > >> Active/Active configuration but at least in a Master/Slave > configuration. This > >> should allow PyPI to be far more stable moving forward and make > downtimes much > >> easier to recover from. > >> > >> The services are still fronted by Fasty?s CDN and in the new > infrastructure > >> we?ve removed our load balancer and have replaced it by having Fastly > handle > >> the load balancing for us. Additionally we?ve recently setup a static > mirror of > >> PyPI that is updated once every minute. This is hosted on Rackspace > cloud as > >> well but in a separate data center from the rest of PyPI. Fastly is > configured > >> to fall back to this static mirror in the case that neither of the two > web > >> heads are functioning. This should ensure that even in the event of a > >> catastrophic failure of the PyPI service that the bulk of package > installations > >> should hopefully remain working. > >> > >> The bad news, (and the ?Breakage? from the subject) is that while the > new > >> infrastructure was being planned out, built, and migrated to the > ?pypissh? > >> package was forgotten. The pypissh package is an alternative way to > upload > >> packages to PyPI however it is very difficult, because of the way it > works, to > >> provide HA support for it as we?ve set up for everything else. We don?t > have > >> any numbers for how many people are actively using this package but > looking > >> at a roughly 2 week chunk of time in PyPI?s download history, the > pypissh > >> package was downloaded 7 times by a browser, and 7 times by pip. All > other > >> downloads were caused by the mirroring system. > >> > >> As of right now pypissh is non functional and due to the difficulty in > HAing > >> and monitoring the current setup and because it is apparently has a very > >> small set of users we would like to effectively kill off this particular > >> service. Additionally the benefits of pypissh have been reduced now > that PyPI > >> is available over a TLS connection with a well trusted certificate. My > question > >> to you is, is this something that distutils-sig is willing to have > happen? If > >> we are to re-enable pypissh we?ll need to write a new solution to doing > it that > >> can be properly HA?d and we?d prefer to put our efforts into improving > things > >> for a much larger set of people. > >> > >> So yea, PyPI should be loads more stable and more reliable now. > >> > >> ----------------- > >> Donald Stufft > >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > >> > >> > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> https://mail.python.org/mailman/listinfo/distutils-sig > >> > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Jan 26 05:18:31 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 25 Jan 2014 23:18:31 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <60552257-3130-4B19-9F28-E6C2716C68C8@stufft.io> Message-ID: <0AC916E8-E63C-46EF-BFE1-9AF6EBAC5C9D@stufft.io> On Jan 25, 2014, at 11:15 PM, Kyle Kelley wrote: > Where are the states stored? https://github.com/python/pypi-salt ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Jan 26 06:18:44 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 26 Jan 2014 00:18:44 -0500 Subject: [Distutils] pip 1.5.2 and virtualenv 1.11.2 released Message-ID: I?ve just released pip 1.5.2 and virtualenv 1.11.2 1.5.2 (2014-01-26) ------------------ * Upgraded the vendored ``pkg_resources`` and ``_markerlib`` to setuptools 2.1. * Fixed an error that prevented accessing PyPI when pyopenssl, ndg-httpsclient, and pyasn1 are installed * Fixed an issue that caused trailing comments to be incorrectly included as part of the URL in a requirements file 1.11.2 (2014-01-26) ~~~~~~~~~~~~~~~~~~~ * Fixed easy_install installed virtualenvs by updated pip to 1.5.2 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Sun Jan 26 12:03:31 2014 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 26 Jan 2014 12:03:31 +0100 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: <20140126120331.Horde.CX1A4twLCxV2Gy84goa5aA7@webmail.df.eu> Quoting Richard Jones : > Thanks everyone who helped make this happen. > > From my perspective* I believe the ssh upload mechanism was added to > address security issues around the basic-auth-over-http method used > historically. Now uploads *may* be done over https, and those using the ssh > method can move over to using twine or pip upload, I think that it's > reasonable to discontinue support for ssh uploads. There is one usecase that still isn't addressed by any of the alternatives: Automated uploads still require the password to be stored on disk. So if the laptop is stolen, the password may get stolen as well. With SSH upload, the authentication comes from the ssh-agent, which protects the credentials better (i.e. if the laptop is powered-down, or requires the user to enter a password on access, the key is protected). It has been suggested to resolve this using the keyring library (which would give the same protection to the password as ssh-agent to the private key), but a) I don't think it actually *has* been implemented, and b) to properly implement it (i.e. without monkey-patching register/upload), it would have to be done in CPython, and c) that would require to put keyring into CPython, which could happen in Python 3.5 at the earliest. So I suggest that somebody does a), and then provides a package that works around b) and c) by monkeypatching distutils (just like pypissh does). In any case, if you really chose to discontinue SSH access, I suggest that you also change the UI to drop registration of SSH keys, and then ultimately remove them from the schema. BTW, you can get an indication of how many users this might affect by checking how many users have keys registered. Regards, Martin From rgbkrk at gmail.com Sun Jan 26 05:15:15 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Sat, 25 Jan 2014 21:15:15 -0700 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: <60552257-3130-4B19-9F28-E6C2716C68C8@stufft.io> References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <60552257-3130-4B19-9F28-E6C2716C68C8@stufft.io> Message-ID: Congrats! Thanks for always making the PyPI infrastructure better and better. Where are the states stored? On Sat, Jan 25, 2014 at 5:18 PM, Donald Stufft wrote: > > On Jan 25, 2014, at 7:04 PM, Chris Jerdonek > wrote: > > > On Sat, Jan 25, 2014 at 3:38 PM, Donald Stufft wrote: > >> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to > new > >> infrastructure. > >> > >> The old infrastructure was: > >> > >> - a single database server managed by OSUOSL > >> - a pair of load balancers shared by all of the python.org services > hosted on > >> OSUOSL > >> - a single backend VM that served as everything else for PyPI > >> > >> The VM that was acting as the backend server from PyPI was partially > hand > >> configured and partially setup to be managed by chef. Additionally it > had an > >> issue that caused it to kernel panic every so often which had been the > cause of > >> a number of downtimes in the last few months. Because it was primarily > >> configured and administered by hand and because the way it was set up > it was > >> not feasible to have any sort of failover or spare. > >> > >> The new infrastructure is: > >> > >> - 2 Web VMs > >> - 2 Database servers in Master/Slave Configuration > >> - 2 PgPool Servers pooling connections to the database servers and load > >> balancing reads across them. > >> - 2 GlusterFS servers backed by Cloud Block Storage acting as the file > storage > >> for package and package docs > >> - 1 metrics server to handle updating the download counts as they come > in from > >> Fastly > >> > >> All of the VMs are hosted on Rackspace?s Public Cloud and have their > >> configuration completely controlled and managed using Salt. Going > forward this > > > > Can you say a little about the choice to use Salt instead of Chef? I > > don't really care either way, but am just curious. Is it because Salt > > is written in Python, or were there other reasons (functionality, > > etc)? > > > > --Chris > > I?d need to ask Ernest to be sure, but I believe it was mostly that he was > more familiar > with it. The fact that it was written in Python was a bonus as well ;) I > don?t think that > there was anything that Chef was missing or that Salt had over Chef, just > familiarity > of the person who did most of the work. I?ll double check with Ernest to > make sure there > wasn?t another reason :) > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Sun Jan 26 05:15:36 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Sat, 25 Jan 2014 21:15:36 -0700 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <60552257-3130-4B19-9F28-E6C2716C68C8@stufft.io> Message-ID: s/states/salt states/ On Sat, Jan 25, 2014 at 9:15 PM, Kyle Kelley wrote: > Congrats! Thanks for always making the PyPI infrastructure better and > better. > > Where are the states stored? > > > On Sat, Jan 25, 2014 at 5:18 PM, Donald Stufft wrote: > >> >> On Jan 25, 2014, at 7:04 PM, Chris Jerdonek >> wrote: >> >> > On Sat, Jan 25, 2014 at 3:38 PM, Donald Stufft >> wrote: >> >> Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to >> new >> >> infrastructure. >> >> >> >> The old infrastructure was: >> >> >> >> - a single database server managed by OSUOSL >> >> - a pair of load balancers shared by all of the python.org services >> hosted on >> >> OSUOSL >> >> - a single backend VM that served as everything else for PyPI >> >> >> >> The VM that was acting as the backend server from PyPI was partially >> hand >> >> configured and partially setup to be managed by chef. Additionally it >> had an >> >> issue that caused it to kernel panic every so often which had been the >> cause of >> >> a number of downtimes in the last few months. Because it was primarily >> >> configured and administered by hand and because the way it was set up >> it was >> >> not feasible to have any sort of failover or spare. >> >> >> >> The new infrastructure is: >> >> >> >> - 2 Web VMs >> >> - 2 Database servers in Master/Slave Configuration >> >> - 2 PgPool Servers pooling connections to the database servers and load >> >> balancing reads across them. >> >> - 2 GlusterFS servers backed by Cloud Block Storage acting as the file >> storage >> >> for package and package docs >> >> - 1 metrics server to handle updating the download counts as they come >> in from >> >> Fastly >> >> >> >> All of the VMs are hosted on Rackspace?s Public Cloud and have their >> >> configuration completely controlled and managed using Salt. Going >> forward this >> > >> > Can you say a little about the choice to use Salt instead of Chef? I >> > don't really care either way, but am just curious. Is it because Salt >> > is written in Python, or were there other reasons (functionality, >> > etc)? >> > >> > --Chris >> >> I?d need to ask Ernest to be sure, but I believe it was mostly that he >> was more familiar >> with it. The fact that it was written in Python was a bonus as well ;) I >> don?t think that >> there was anything that Chef was missing or that Salt had over Chef, just >> familiarity >> of the person who did most of the work. I?ll double check with Ernest to >> make sure there >> wasn?t another reason :) >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aclark at aclark.net Sun Jan 26 14:26:55 2014 From: aclark at aclark.net (Alex Clark) Date: Sun, 26 Jan 2014 08:26:55 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: On 1/25/14, 6:38 PM, Donald Stufft wrote: > My question > to you is, is this something that distutils-sig is willing to have happen? If > we are to re-enable pypissh we?ll need to write a new solution to doing it that > can be properly HA?d and we?d prefer to put our efforts into improving things > for a much larger set of people. +0 re: pypissh, but I am still interested in seeing OAuth support come back: - https://bitbucket.org/pypa/pypi/issue/85/oauth-authorise-not-found-https-must-be Any idea if this can come back as part of PyPI or if we have to wait for warehouse? Nice work on the infrastructure, thank you! -- Alex Clark ? http://about.me/alex.clark From richard at python.org Sun Jan 26 22:40:12 2014 From: richard at python.org (Richard Jones) Date: Mon, 27 Jan 2014 08:40:12 +1100 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: It definitely looks like we've got some issues introduced in recent server migrations and reconfigurations. Things I'm aware of: - OAuth is busted - OpenID is confused and/or busted - password reset is possibly busted - pypissh is busted Richard On 27 January 2014 00:26, Alex Clark wrote: > On 1/25/14, 6:38 PM, Donald Stufft wrote: > >> My question >> to you is, is this something that distutils-sig is willing to have >> happen? If >> we are to re-enable pypissh we?ll need to write a new solution to doing >> it that >> can be properly HA?d and we?d prefer to put our efforts into improving >> things >> for a much larger set of people. >> > > +0 re: pypissh, but I am still interested in seeing OAuth support come > back: > > - https://bitbucket.org/pypa/pypi/issue/85/oauth-authorise- > not-found-https-must-be > > Any idea if this can come back as part of PyPI or if we have to wait for > warehouse? Nice work on the infrastructure, thank you! > > -- > Alex Clark ? http://about.me/alex.clark > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jan 27 13:14:27 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 Jan 2014 07:14:27 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: Just a follow up. On Jan 26, 2014, at 4:40 PM, Richard Jones wrote: > It definitely looks like we've got some issues introduced in recent server migrations and reconfigurations. Things I'm aware of: > > - OAuth is busted > - OpenID is confused and/or busted These two issues existed prior to the migration as far as I can tell. > - password reset is possibly busted This has been fixed. > - pypissh is busted This is still the case and was known from the first announcement. Additionally it seems there is an issue with the health checks when connecting from Australia that cause issues where folks will intermittently get the static page instead of the real PyPI. > > > Richard > > > On 27 January 2014 00:26, Alex Clark wrote: > On 1/25/14, 6:38 PM, Donald Stufft wrote: > My question > to you is, is this something that distutils-sig is willing to have happen? If > we are to re-enable pypissh we?ll need to write a new solution to doing it that > can be properly HA?d and we?d prefer to put our efforts into improving things > for a much larger set of people. > > +0 re: pypissh, but I am still interested in seeing OAuth support come back: > > - https://bitbucket.org/pypa/pypi/issue/85/oauth-authorise-not-found-https-must-be > > Any idea if this can come back as part of PyPI or if we have to wait for warehouse? Nice work on the infrastructure, thank you! > > -- > Alex Clark ? http://about.me/alex.clark > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From aclark at aclark.net Mon Jan 27 13:28:21 2014 From: aclark at aclark.net (Alex Clark) Date: Mon, 27 Jan 2014 12:28:21 +0000 (UTC) Subject: [Distutils] =?utf-8?q?PyPI_Migrated_to_New_Infrastructure_with_so?= =?utf-8?q?me=09Breakage?= References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > > > > Just a follow up. > - OAuth is busted > > > These two issues existed prior to the migration as far as I can tell. Correct. We've discussed Oauth in IRC and this ticket has existed since late last year: - https://bitbucket.org/pypa/pypi/issue/85/oauth-authorise-not-found-https-must-be I'm bringing it up now because I'm still interested in seeing it fixed. IIUC MvL correctly, it happened around the time of the CDN switch. In any event, there is a portion of traffic going to/from PyPI unencrypted and PyPI needs it to be encrypted. This leads to the confusing error message when trying to do OAuth over "https". You talk https to the end point, and the end point (seemingly) responds "I need this to be https". From donald at stufft.io Mon Jan 27 13:31:52 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 Jan 2014 07:31:52 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> Message-ID: <43128140-B47F-44EC-9BAA-A3520EFCDC22@stufft.io> On Jan 27, 2014, at 7:28 AM, Alex Clark wrote: > Donald Stufft stufft.io> writes: > >> >> >> >> Just a follow up. >> - OAuth is busted >> >> >> These two issues existed prior to the migration as far as I can tell. > > Correct. We've discussed Oauth in IRC and this ticket has existed since late > last year: > > - > https://bitbucket.org/pypa/pypi/issue/85/oauth-authorise-not-found-https-must-be > > I'm bringing it up now because I'm still interested in seeing it fixed. IIUC > MvL correctly, it happened around the time of the CDN switch. > > In any event, there is a portion of traffic going to/from PyPI unencrypted > and PyPI needs it to be encrypted. This leads to the confusing error message > when trying to do OAuth over "https". You talk https to the end point, and > the end point (seemingly) responds "I need this to be https?. It?s very unlikely for something to happen over not HTTPS now. The backend servers for PyPI do not offer a non HTTPS port, and Fastly has a blanket HTTP -> HTTPS redirect. Most likely the issue is just that PyPI isn?t realizing that it?s being accessed via HTTPS. > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From aclark at aclark.net Mon Jan 27 13:39:14 2014 From: aclark at aclark.net (Alex Clark) Date: Mon, 27 Jan 2014 12:39:14 +0000 (UTC) Subject: [Distutils] =?utf-8?q?PyPI_Migrated_to_New_Infrastructure_with=09?= =?utf-8?q?some=09Breakage?= References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <43128140-B47F-44EC-9BAA-A3520EFCDC22@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > Most likely the issue is just that PyPI isn?t realizing > that it?s being accessed via HTTPS. Can you say more about this? If that's the case, then it sounds like someone can send you a pull request to remove the (bogus?) https check and we are done. From donald at stufft.io Mon Jan 27 13:42:11 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 Jan 2014 07:42:11 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <43128140-B47F-44EC-9BAA-A3520EFCDC22@stufft.io> Message-ID: <83789005-FFF5-4280-9032-C18431B75F67@stufft.io> On Jan 27, 2014, at 7:39 AM, Alex Clark wrote: > Donald Stufft stufft.io> writes: > >> Most likely the issue is just that PyPI isn?t realizing >> that it?s being accessed via HTTPS. > > Can you say more about this? If that's the case, then it sounds like someone > can send you a pull request to remove the (bogus?) https check and we are done. > I haven?t looked into it yet simply out of a function of time. Obviously PyPI is checking if it?s being accessed via HTTPS somehow, and obviously (due to the nature of the error) it doesn?t believe it is being accessed via HTTPS. Since I know that it shouldn?t be possible to access PyPI via non HTTPS I can only deduce that however it?s determining that it?s running behind HTTPS isn?t working for one reason or another. I don?t really know more than that at the moment. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Mon Jan 27 14:31:07 2014 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Jan 2014 14:31:07 +0100 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: <83789005-FFF5-4280-9032-C18431B75F67@stufft.io> References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <43128140-B47F-44EC-9BAA-A3520EFCDC22@stufft.io> <83789005-FFF5-4280-9032-C18431B75F67@stufft.io> Message-ID: <52E65F9B.9080402@v.loewis.de> > I haven?t looked into it yet simply out of a function of time. Obviously PyPI is checking > if it?s being accessed via HTTPS somehow, and obviously (due to the nature of the > error) it doesn?t believe it is being accessed via HTTPS. Since I know that it shouldn?t > be possible to access PyPI via non HTTPS I can only deduce that however it?s > determining that it?s running behind HTTPS isn?t working for one reason or another. > > I don?t really know more than that at the moment. See the line in the ticket. https://bitbucket.org/pypa/pypi/issue/85/oauth-authorise-not-found-https-must-be https://bitbucket.org/pypa/pypi/src/099a6bb6e4f23f61d2dc2117d36f86fd3dfd57e2/webui.py?at=default#cl-3592 The "HTTPS" environment variable is not set, which ought to have the value "on" if access came through https. The issue is probably this: http://lists.unbit.it/pipermail/uwsgi/2010-August/000561.html So a line uwsgi_param HTTPS on; in https://github.com/python/pypi-salt/blob/master/provisioning/salt/roots/salt/nginx/config/nginx.ssl.conf.jinja should help, or alternatively a conditional line in pypi's app.conf, condition on the scheme being https. On dinsdale, the old nginx configuration had the line uwsgi_param HTTPS $https if_not_empty; so I'm pretty sure it worked when I moved the service to OSL. Regards, Martin From merwok at netwok.org Mon Jan 27 18:29:49 2014 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 27 Jan 2014 12:29:49 -0500 Subject: [Distutils] PyPI Migrated to New Infrastructure with some Breakage In-Reply-To: <20140126120331.Horde.CX1A4twLCxV2Gy84goa5aA7@webmail.df.eu> References: <6C21B61E-BE91-4C73-A96A-F7B0FEBD4339@stufft.io> <20140126120331.Horde.CX1A4twLCxV2Gy84goa5aA7@webmail.df.eu> Message-ID: <52E6978D.8050008@netwok.org> Hello, Le 26/01/2014 06:03, martin at v.loewis.de a ?crit : > There is one usecase that still isn't addressed by any of the alternatives: > Automated uploads still require the password to be stored on disk. So if > the laptop is stolen, the password may get stolen as well. > > With SSH upload, the authentication comes from the ssh-agent, which > protects the credentials better (i.e. if the laptop is powered-down, or > requires the user to enter a password on access, the key is protected). > > It has been suggested to resolve this using the keyring library (which > would give the same protection to the password as ssh-agent to the private > key) [...] distutils can?t depend on keyring, but twine could. Regards From ncoghlan at gmail.com Tue Jan 28 11:40:31 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jan 2014 20:40:31 +1000 Subject: [Distutils] Explicitly constructing SOABI for Python 2.7 Message-ID: Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x - the SOABI symbol it expects to find isn't present. PEP 425 also disclaims dealing with the problem. However, PEP 3149 (which defined SOABI) explains how to define that based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC >>> sysconfig.get_config_var("SOABI") >>> sysconfig.get_config_var("Py_DEBUG") 0 >>> sysconfig.get_config_var("Py_UNICODE_SIZE") 4 >>> sysconfig.get_config_var("WITH_PYMALLOC") 1 With those settings and the platform module (to get the current implementation), it is possible to figure out that the wheel ABI tag should be: cp27mu If Py_DEBUG was true, then it would be "cp27dmu" If WITH_PYMALLOC was false for some reason: "cp27u" And for a narrow Unicode build: "cp27m" That way, the ABI tagging would work properly in 2.x as well, rather than people getting the impression that wheels can't mark the ABI compatibility requirements properly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jan 28 14:02:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jan 2014 23:02:02 +1000 Subject: [Distutils] PEP 440 (versioning) nearing completion (I hope) Message-ID: I just pushed a new draft of PEP 440 live: http://www.python.org/dev/peps/pep-0440/ Relative to the version I posted just before Christmas, the main significant change is the inclusion of version epochs as part of the spec. In 99.9% of cases, those won't be needed and can be ignored. However, I'm swayed by the fact that at least Debian and Fedora found them useful enough to add to their versioning schemes, and they're a relatively elegant solution to the "How do I switch from date based versioning to semantic versioning without renaming my project?" problem. There are also two clarifications: - I changed the pseudo-syntax for the release segment from N[.N]+ to N(.N)* to make it clearer that single value version numbers are acceptable - I point out in the rationale section that "-" makes local version identifiers more readable in isolation, and the existing escaping rules in PEP 427 take care of converting them to hyphens to avoid ambiguity in wheel filenames. (Paul, you can consider this a belated reply to your post from last week...) With this update, I believe PEP 440 is functionally complete. However, there are three additional requirements that need to be met before I'm willing to declare it accepted: - a reference implementation (I'm hoping Vinay will be willing to step up to the plate with distlib again on that front) - rerunning the compatibility analysis against the PyPI of early 2014 - surviving distutils-sig review without too much grumbling or virtual tomato throwing ;) I know it's a rather complex spec, but robust and comprehensive versioning is a rather complex problem :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Tue Jan 28 14:29:09 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 28 Jan 2014 08:29:09 -0500 Subject: [Distutils] Explicitly constructing SOABI for Python 2.7 In-Reply-To: References: Message-ID: +1 However, on Windows, I don't see most of the config vars. At least the Unicode width is easy to get at in other ways. We also need to update the PEP/implementation to allow/add to the list of supported tags "py2.py3-none-(arch)" for cffi-type "doesn't use the ABI" extensions. This might just be a bug in the implementation... On Tue, Jan 28, 2014 at 5:40 AM, Nick Coghlan wrote: > Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x > - the SOABI symbol it expects to find isn't present. PEP 425 also > disclaims dealing with the problem. > > However, PEP 3149 (which defined SOABI) explains how to define that > based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and > WITH_PYMALLOC > >>>> sysconfig.get_config_var("SOABI") >>>> sysconfig.get_config_var("Py_DEBUG") > 0 >>>> sysconfig.get_config_var("Py_UNICODE_SIZE") > 4 >>>> sysconfig.get_config_var("WITH_PYMALLOC") > 1 > > With those settings and the platform module (to get the current > implementation), it is possible to figure out that the wheel ABI tag > should be: > > cp27mu > > If Py_DEBUG was true, then it would be "cp27dmu" > If WITH_PYMALLOC was false for some reason: "cp27u" > And for a narrow Unicode build: "cp27m" > > That way, the ABI tagging would work properly in 2.x as well, rather > than people getting the impression that wheels can't mark the ABI > compatibility requirements properly. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Tue Jan 28 23:45:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 08:45:48 +1000 Subject: [Distutils] Explicitly constructing SOABI for Python 2.7 In-Reply-To: References: Message-ID: On 28 Jan 2014 23:29, "Daniel Holth" wrote: > > +1 > > However, on Windows, I don't see most of the config vars. At least the > Unicode width is easy to get at in other ways. It's less of an issue on Windows, since we're already assuming python.orgbinary compatibility in most cases anyway. However, I'll have a look and see if this info is exposed any other way. > We also need to update the PEP/implementation to allow/add to the list > of supported tags "py2.py3-none-(arch)" for cffi-type "doesn't use the > ABI" extensions. This might just be a bug in the implementation... That's definitely supposed to be allowed by the spec, specifically for that ctypes/cffi style use case. Now, whether the tools currently provide a way to *generate* a tag like that is a different question :) Cheers, Nick. > > On Tue, Jan 28, 2014 at 5:40 AM, Nick Coghlan wrote: > > Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x > > - the SOABI symbol it expects to find isn't present. PEP 425 also > > disclaims dealing with the problem. > > > > However, PEP 3149 (which defined SOABI) explains how to define that > > based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and > > WITH_PYMALLOC > > > >>>> sysconfig.get_config_var("SOABI") > >>>> sysconfig.get_config_var("Py_DEBUG") > > 0 > >>>> sysconfig.get_config_var("Py_UNICODE_SIZE") > > 4 > >>>> sysconfig.get_config_var("WITH_PYMALLOC") > > 1 > > > > With those settings and the platform module (to get the current > > implementation), it is possible to figure out that the wheel ABI tag > > should be: > > > > cp27mu > > > > If Py_DEBUG was true, then it would be "cp27dmu" > > If WITH_PYMALLOC was false for some reason: "cp27u" > > And for a narrow Unicode build: "cp27m" > > > > That way, the ABI tagging would work properly in 2.x as well, rather > > than people getting the impression that wheels can't mark the ABI > > compatibility requirements properly. > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jan 29 00:14:08 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 Jan 2014 18:14:08 -0500 Subject: [Distutils] Using Wheel with zipimport Message-ID: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed to be compatible with zipimport so that you could directly add them to sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that this was the case, and had thought otherwise, and have been telling people that although Wheels could technically be used that way that doing so was considered unsupported. I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. One of the major problems with Egg was that it confused the difference between a distribution format and a runtime installation format. I believe that if we have this as a supported option, that ultimately it will lead towards the same kind of problems that Egg had trying to fulfill two roles at once. Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). If a single file importable item for a project is something that end users want, then an additional format should be developed that does not concern itself with the ability to unpack and install the project and instead solves the problems that come with trying to stick arbitrary things in a zipped file and import them. So yea, I think that the goals of an importable format and an distribution format are not completely overlapped, and if Wheel tries to be both, then both sides of that will suffer with a less useful and less suitable ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 00:38:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 09:38:22 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jan 29 01:18:41 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 Jan 2014 19:18:41 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). > > The key problems with eggs in relation to this were: > - easy_install preferring to install as eggs by default > - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation > > Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. > > All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories). > > Cheers, > Nick. > Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed Jan 29 02:19:51 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 28 Jan 2014 20:19:51 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> Message-ID: On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft wrote: > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > > I think you're reading too much into that comment. Putting a wheel file > directly on sys.path is no different from putting any other zipfile directly > on sys.path - whether or not it will work depends on the context, but it's a > useful capability if used responsibly (as we do in the ensurepip > implementation). > > The key problems with eggs in relation to this were: > - easy_install preferring to install as eggs by default > - setuptools install a global site hook that added every installed egg to > sys.path for every application run in that Python installation > > Neither of those applies to wheels - pip always unpacks them when > installing, and if you want to add one to sys.path you have to do it > manually, it doesn't happen automatically. > > All the new note in the PEP is clarifying is that it *isn't* an accident > that the wheel format is zipimport compatible for pure Python wheels, we > deliberately designed it that way (hence the "Root-is-purelib" setting, > rather than requiring separate purelib and platlib subdirectories). > > Cheers, > Nick. > > > Regardless if it was or wasn't an accident, I believe it was a mistake. > Supporting it officially at all means that we have limitations on what we > can > do to make Wheel a better format. I had hopes that Wheel could be made more > generic than it currently is, but because of the fact that directly adding > them to sys.path is supported that makes it much much more awkward to do so. Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? I'm opposed to making wheel generic as in "package PostgreSQL itself" generic. There are other ways to do that. From eugene at sazhin.us Wed Jan 29 02:04:58 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Tue, 28 Jan 2014 20:04:58 -0500 Subject: [Distutils] PEP 427 Message-ID: Hi, I saw that people from this list are responsible for Wheel related PEP. I?m comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. I?m trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... So, i would like to ask few questions about it : 1. I?m coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn?t it be way easier to deal with the dependencies and imports? 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can?t be used the same exact way? I would appreciate any insight about the development infrastructure the wheel designers have in mind. Thanks in advance, Eugene From dholth at gmail.com Wed Jan 29 03:54:41 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 28 Jan 2014 21:54:41 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: Message-ID: On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin wrote: > Hi, > > I saw that people from this list are responsible for Wheel related PEP. > I'm comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. > > I'm trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... > > So, i would like to ask few questions about it : > > 1. I'm coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn't it be way easier to deal with the dependencies and imports? > 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can't be used the same exact way? > > > I would appreciate any insight about the development infrastructure the wheel designers have in mind. > > Thanks in advance, > Eugene The main reason the packaging format is not explicitly importable is simply because Python hasn't had "jar-like" deployment for as long or as consistently as Java. So while Java code universally uses a "get resource" API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. From vinay_sajip at yahoo.co.uk Wed Jan 29 05:14:22 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 04:14:22 +0000 (UTC) Subject: [Distutils] Using Wheel with zipimport References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > I think we need to reconsider this. I cannot stress how badly an idea > I think it is for Wheels to concern itself with the ability to add the > Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format. When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified. > Further more it won't even work accurately for all Wheels (as Nick's > edit says) so we require that people wanting to do this figure out if > their Wheel can or can not be added to the sys.path (which isn't as > simple as looking at the tags because a platform specific Wheel may only > contain optional C modules). I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a "mount" method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no "figuring out" that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it. Regards, Vinay Sajip From ncoghlan at gmail.com Wed Jan 29 05:54:45 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 14:54:45 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> Message-ID: On 29 Jan 2014 11:19, "Daniel Holth" wrote: > > On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft wrote: > > > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > > > > I think you're reading too much into that comment. Putting a wheel file > > directly on sys.path is no different from putting any other zipfile directly > > on sys.path - whether or not it will work depends on the context, but it's a > > useful capability if used responsibly (as we do in the ensurepip > > implementation). > > > > The key problems with eggs in relation to this were: > > - easy_install preferring to install as eggs by default > > - setuptools install a global site hook that added every installed egg to > > sys.path for every application run in that Python installation > > > > Neither of those applies to wheels - pip always unpacks them when > > installing, and if you want to add one to sys.path you have to do it > > manually, it doesn't happen automatically. > > > > All the new note in the PEP is clarifying is that it *isn't* an accident > > that the wheel format is zipimport compatible for pure Python wheels, we > > deliberately designed it that way (hence the "Root-is-purelib" setting, > > rather than requiring separate purelib and platlib subdirectories). > > > > Cheers, > > Nick. > > > > > > Regardless if it was or wasn't an accident, I believe it was a mistake. > > Supporting it officially at all means that we have limitations on what we > > can > > do to make Wheel a better format. I had hopes that Wheel could be made more > > generic than it currently is, but because of the fact that directly adding > > them to sys.path is supported that makes it much much more awkward to do so. > > Hey, as long as they are zipfiles that don't totally scramble the > layout of the internal Python code you can add them to sys.path. Did > you know you can even add subpaths of a zipfile to sys.path? blown> Yep, the only requirement is "there will be a non empty subset of valid wheel files that can be used directly with zipimport". Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - "supported when practical" isn't the same thing as "recommended". Cheers, Nick. > > I'm opposed to making wheel generic as in "package PostgreSQL itself" > generic. There are other ways to do that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jan 29 06:01:33 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 05:01:33 +0000 (UTC) Subject: [Distutils] PEP 440 (versioning) nearing completion (I hope) References: Message-ID: Nick Coghlan gmail.com> writes: > - a reference implementation (I'm hoping Vinay will be willing to step > up to the plate with distlib again on that front) First cut, done :-) I updated distlib's implementation to: - include the local component in NormalizedVersion - allow single numeric component versions in NormalizedVersion - update the NormalizedMatcher to include it in matches when required - convert between - and _ in the version for the wheel filename I've added/updated tests for these changes, but additional reviews/test results are always welcome. Regards, Vinay Sajip From donald at stufft.io Wed Jan 29 06:52:23 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 00:52:23 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> Message-ID: <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> On Jan 28, 2014, at 11:54 PM, Nick Coghlan wrote: > > On 29 Jan 2014 11:19, "Daniel Holth" wrote: > > > > On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft wrote: > > > > > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > > > > > > I think you're reading too much into that comment. Putting a wheel file > > > directly on sys.path is no different from putting any other zipfile directly > > > on sys.path - whether or not it will work depends on the context, but it's a > > > useful capability if used responsibly (as we do in the ensurepip > > > implementation). > > > > > > The key problems with eggs in relation to this were: > > > - easy_install preferring to install as eggs by default > > > - setuptools install a global site hook that added every installed egg to > > > sys.path for every application run in that Python installation > > > > > > Neither of those applies to wheels - pip always unpacks them when > > > installing, and if you want to add one to sys.path you have to do it > > > manually, it doesn't happen automatically. > > > > > > All the new note in the PEP is clarifying is that it *isn't* an accident > > > that the wheel format is zipimport compatible for pure Python wheels, we > > > deliberately designed it that way (hence the "Root-is-purelib" setting, > > > rather than requiring separate purelib and platlib subdirectories). > > > > > > Cheers, > > > Nick. > > > > > > > > > Regardless if it was or wasn't an accident, I believe it was a mistake. > > > Supporting it officially at all means that we have limitations on what we > > > can > > > do to make Wheel a better format. I had hopes that Wheel could be made more > > > generic than it currently is, but because of the fact that directly adding > > > them to sys.path is supported that makes it much much more awkward to do so. > > > > Hey, as long as they are zipfiles that don't totally scramble the > > layout of the internal Python code you can add them to sys.path. Did > > you know you can even add subpaths of a zipfile to sys.path? > blown> > > Yep, the only requirement is "there will be a non empty subset of valid wheel files that can be used directly with zipimport". > > Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - "supported when practical" isn't the same thing as "recommended". > > Cheers, > Nick. > > > > > I'm opposed to making wheel generic as in "package PostgreSQL itself" > > generic. There are other ways to do that. > And yet on another read through of PEP427 the first item in ?Comparison to Egg? is "Wheel is an installation format; egg is importable.?. The only other mention of importing any of them in that PEP is the section you just added saying that Wheels are designed to be importable. The original text of the PEP does not provide any evidence that Wheels were meant to be importable and instead makes it a point to call that a difference from Eggs. Further more there is a comment from Daniel on python-dev [1] which states that ?Jim Fulton is right that weird failures are a characteristic of zipped eggs, so one of the #1 requests for setuptools is how to prohibit zipping from ever happening. This is an important reason why wheel is billed as an installation format -- fewer users with pitchforks. It's very cool that it works though. Debugging is slightly easier than it was in the old days because pdb can now read the source code from the zip.? Which further shows that at the time it was ?cool that it worked? but that the ?weird failures? being a reason that Wheel was an installation format. I believe that adding this ?feature? to the PEP ex post facto is a bad idea. Had the PEP contained anything to indicate that Wheels were intended to be importable as part of the design philosophy I would have spoken out about it then instead it?s been added after it was accepted with no discussion that I can see. If I missed where this discussion happened during the PEP please direct me to it because I?ve just spent 15 minutes googling attempting to find any information on it, and all I?ve been able to find is Vinay?s experiment with Wheel.mount(), You mentioning the possibility of using Wheels in the sense of Multi Version Installs (which ends up talking about a Wheel inspired directory layout instead), and the thread I mentioned above. Not that it?s entirely relevant because really regardless of if it was discussed if it wasn?t encoded in the PEP than people following along by watching the PEPs had no ability to step in to speak for or against it. [1] https://groups.google.com/d/msg/dev-python/BS28mF7mb6c/o8jOo1NcousJ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Jan 29 07:09:03 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 01:09:03 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> On Jan 28, 2014, at 11:14 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I think we need to reconsider this. I cannot stress how badly an idea >> I think it is for Wheels to concern itself with the ability to add the >> Wheel to sys.path and import it. > > I know that people have had bad experiences in the past with eggs, for the > reasons Nick outlined in his response. However, you don't say *with > specifics* why you think putting wheels on sys.path is a bad idea - I don't > think it's convincing enough just to hand-wavingly reference problems with > the egg format. > > When this topic came up before, I asked for specific failure modes which > were causing concern, but I never got a response IIRC. > > One can't say that having the same packaging and importable formats is > inherently bad, since Java shows otherwise. So if there are specific > problems, they should be identified. Last time this topic came up Jim Fulton did point things out. 1. That unpacking into the home directory is problematic because users that run services often don?t have home directories. This you waved away by saying that the home directory isn?t a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and ?generally cause pain?. Which I wholly agree with, intact Mitre has assigned CVE?s to applications that did switch their Egg Cache to directories other than HOME. 2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a ?deployment format? (which I don?t believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him. 3. Zipped imports (In Jim?s experience) are slower than unpacked imports, which you replied (quite literally) "Caveat emptor?. Jim?s response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words: "It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony.? He closes up with, "Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs.? So you have pip that doesn?t use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it). Why do we insist upon repeating the mistakes of the past? > >> Further more it won't even work accurately for all Wheels (as Nick's >> edit says) so we require that people wanting to do this figure out if >> their Wheel can or can not be added to the sys.path (which isn't as >> simple as looking at the tags because a platform specific Wheel may only >> contain optional C modules). > > I don't yet see a technical impediment here. For example, distlib's wheel > implementation uses a "mount" method to add a wheel to sys.path. This uses > the tags to check for compatibility - there's no "figuring out" that the > user has to do. If additional metadata about C extensions is available > (which it is, if the wheel is built from source using distil), then those > extensions are made available for import, too. I realise that's an extension > to the current spec, but then no one is forcing any one to use it. Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn?t using __file__ shenanigans instead of pkgutil.get_data()? easy_install tried to do this with varying degrees of success, I?ve never seen much except pain from zipped Eggs. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 08:32:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 17:32:14 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic. However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to. Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file. Running directly from a wheel is a power tool - that's a reason to put "handle with care" warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments. We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is "You probably don't want to use them". However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists. I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way. Cheers, Nick. From p.f.moore at gmail.com Wed Jan 29 09:06:07 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 08:06:07 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 29 January 2014 05:06, Chris Barker wrote: > This isn't just for users of the SciPy Stack -- there are LOT of use-cases > for just numpy by itself. Not that I don't want folks to have easy access of > the rest of the stack as well -- just sayin' Agreed - my main use for NumPy is with Pandas for data analysis tasks, which is a truly great combination. Paul From ncoghlan at gmail.com Wed Jan 29 09:15:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 18:15:14 +1000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 29 January 2014 18:06, Paul Moore wrote: > On 29 January 2014 05:06, Chris Barker wrote: >> This isn't just for users of the SciPy Stack -- there are LOT of use-cases >> for just numpy by itself. Not that I don't want folks to have easy access of >> the rest of the stack as well -- just sayin' > > Agreed - my main use for NumPy is with Pandas for data analysis tasks, > which is a truly great combination. I confess I tend to lump the many and varied data analysis tools that depend on NumPy under the phrase "Scientific Python Stack", rather than intending to refer specifically to SciPy :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Jan 29 09:48:08 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 08:48:08 +0000 (UTC) Subject: [Distutils] Using Wheel with zipimport References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > 1. That unpacking into the home directory is problematic because users > that run services often don?t have home directories. This you waved > away by saying that the home directory isn?t a required place, to > which Jim responded that unpacking them anywhere else was likely to > have access control / security issues and ?generally cause pain?. $HOME is just a sensible default location for caches, but as long as other locations can be used in situations where there's no HOME directory, I don't see how that's "waving away". If a service has no access to writeable disk locations at all, I can see how that might pose a problem, but that hardly seems like a use case which should dictate policies for every other scenario. OTOH "anywhere else was likely to have access control / security issues and 'generally cause pain'" is light on details. > wholly agree with, intact Mitre has assigned CVE?s to applications that > did switch their Egg Cache to directories other than HOME. Are you referring to the situation where users of setuptools set PYTHON_EGG_CACHE to a world-writeable location? If so, the problem would appear to be with the using code, not setuptools itself. AFAICT it was felt that while CVEs could be assigned against individual applications that did this, and setuptools itself was not assigned a CVE. If a HOME-less application uses an alternative cache location which isn't world-writeable, does a security issue arise? > 2. Zipped Eggs are more difficult to work with for debugging purposes, which > you responded saying that Wheels are a ?deployment format? (which I > don?t believe is supported by the PEP at all) and Jim responded with his > experience in developing against things that got installed as zipped Eggs > and the pain that has personally caused him. > > 3. Zipped imports (In Jim?s experience) are slower than unpacked imports, > which you replied (quite literally) "Caveat emptor?. Jim?s response was that > this had been tried with Egg and had been found to be more pain than it > was worth. In his exact words: > > "It's been tried with eggs. This is not new ground. Encouraging people to > do this is going to cause pain and resentment. > > I think one of the reasons there's so much (IMO mostly irrational) hate for > eggs is that people think you can only used zipped eggs, and zipped eggs > cause pain and agony.? > > He closes up with, > > "Importing from zipped eggs has proved itself to be an anti pattern. > > Buildout (as of buildout 2) always unzips eggs.? > > So you have pip that doesn?t use zipped Eggs, buildout which made sure > in buildout 2 (presumably drawing from the lessons of buildout 1) to always > unzip eggs, and easy_install which does support zipped eggs (sometimes, > if it thinks that particular Egg will support it). > > Why do we insist upon repeating the mistakes of the past? Because there are scenarios where that functionality is desirable, and the downsides (such as greater difficulty in debugging, no caching of byte code, lower performance of imports) are tolerable. Just because something can cause problems in some scenarios is no reason for a wholesale ban. C makes segfaulting easy, but that doesn't mean that C extensions are right out ;-) Python is generally a "consenting adults" kind of system. An important difference from the setuptools/easy_install/zipped eggs experience is that nothing happens automatically - a specific action has to be taken to add wheels to sys.path, whereas in the zipped eggs case, sys.path manipulation happens under the hood. Since you don't know it's happening, failures are bound to seem weird - "it's nothing I've done, I'm sure!" Making something possible != recommending its use wholesale. > > Zipped Eggs have a long long history of causing very weird failures conditions, > you say your Wheel.mount checks for compatibility, does it also check that > the library to be installed isn?t using __file__ shenanigans instead of > pkgutil.get_data()? easy_install tried to do this with varying degrees of success, > I?ve never seen much except pain from zipped Eggs. Weird failure conditions are only weird because they haven't been investigated to find the exact cause. There's a lot of software around that can't run from zips because of __file__ shenanigans - that doesn't negate the very real usefulness of zipimport for other software. If I want to write software that runs from zip, I could take care to use pkgutil rather than __file__. Presumably, others can do the same. Regards, Vinay Sajip From chris.barker at noaa.gov Wed Jan 29 06:06:53 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 28 Jan 2014 21:06:53 -0800 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Sat, Jan 25, 2014 at 4:29 PM, Nick Coghlan wrote: > To put the "but what if the user doesn't have SSE2 support?" concern in > context, it should only affect Intel users with CPUs older than a Pentium 4 > (released 2001), and AMD users with a CPU older than an Opteron or Athlon > 64 (both released 2003). All x86/x86_64 CPUs released in the past decade > should be able to handle SSE2 binaries, so our caveat can be "if your > computer is more than a decade old, 'pip install numpy' may not work for > you, but it should do the right thing on newer systems". > Exactly > However, from my perspective, having NumPy readily available to users > using the python.org Windows installers for Python 3.4 would > *significantly* lower the barrier to entry to the Scientific Python stack > for new users on relatively modern systems when compared to the 4 current > options > +1 with a note: This isn't just for users of the SciPy Stack -- there are LOT of use-cases for just numpy by itself. Not that I don't want folks to have easy access of the rest of the stack as well -- just sayin' -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jan 29 10:23:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 09:23:35 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> Message-ID: On 29 January 2014 05:52, Donald Stufft wrote: > Which further shows that at the time it was "cool that it worked" but that > the "weird failures" being a reason that Wheel was an installation format. Having "cool" features in a format is not a bad thing. Zip imports are the "cool " feature here - they have been used for lots of really impressive things (like py2exe, for a less controversial example). But they have limitations (which become "weird failures" when people try to use the feature without knowing or catering for the limitations) and it's the responsibility of the user to understand and address those limitations - or not use the feature if the limitations are too great for them. > I believe that adding this "feature" to the PEP ex post facto is a bad idea. > Had the PEP contained anything to indicate that Wheels were intended > to be importable as part of the design philosophy I would have spoken out > about it then instead it's been added after it was accepted with no > discussion that I can see. As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command. I do *not* recommend using wheels as a runtime format if the package in the wheel hits any of the limitations of zipimport (C extensions, data files that are not or cannot be accessed via pkgutil.package_data, etc). I also would not recommend using wheels except in *very* specialised circumstances (where you have the wheel already and cannot easily just install it before use - pip, ensurepip, virtualenv and similar distribution-oriented environments are the key use cases here). But I do think the feature is useful and should be supported. Paul. From p.f.moore at gmail.com Wed Jan 29 10:40:14 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 09:40:14 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: On 29 January 2014 04:14, Vinay Sajip wrote: > When this topic came up before, I asked for specific failure modes which > were causing concern, but I never got a response IIRC. >From what I recall, the topic came up before in relation to distlib's wheel mount functionality. My specific concerns about that API (none of which have been addressed, AFAIK, but most of which are somewhat non-specific, I concede, for reasons see below) are: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit. 2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The "attractive nuisance" argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected. 3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view. I can't give specific examples of "failure modes" because I don't use the wheel mount functions, nor do I typically add wheels to sys.path. When I did (in virtualenv) I hit a number of issues, but all of these were ones I fixed in user code (either in virtualenv itself, or in pip, which was the wheel I was importing). So you could reasonably dismiss these as "not related to the mount API" to which all I can say is that if I'd been able to do wheel.mount(wheelname) I would likely have put less thought into whether what I was doing was a good idea - and *that's* what I think is the bad aspect of the API. Ultimately, I'll just never use the distlib mount functionality, and I'll recommend not using it when (if) people ask. But I'd rather it were not there to prompt the question. I hope this helps, Paul From vinay_sajip at yahoo.co.uk Wed Jan 29 11:36:33 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 10:36:33 +0000 (UTC) Subject: [Distutils] Using Wheel with zipimport References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: Paul Moore gmail.com> writes: > 1. It does *not* just use the fact that wheels are importable. It goes > beyond that and *extracts* C extensions to make them importable, too. > That is a workaround for a known and accepted limitation of zipimport, > and as a workaround, it has issues. If C extensions in zipfiles could > work reliably, this should go into zipimport itself, and *not* into > 3rd party code. Then everyone would benefit. (a) The mount method does not do the extraction of C extensions *unconditionally* - only when there is suitable indication in the metadata. (b) What are these issues to which you refer? As I said in my other response, I don't see the zipped-egg problem as the same, because that sys.path manipulation is under the hood/not under the developer's control. (c) Why can't third-party code break new ground, which may or may not prove fertile? If considered beneficial, it could always be added to zipimport at a later date. That's how a lot of functionality has entered the stdlib, after all. > 2. It makes what should be a rare use case, to be used only when the > code in the wheel has been carefully checked to work from a zipfile, > seem like a common and straightforward operation. (The "attractive > nuisance" argument). I believe that people using this API will > typically *not* check the code, and will blame the wheel format, or > distlib, when their application does not work as expected. Doesn't the same argument apply to zipimport? Any code which is zipimported shouldn't use __file__ manipulation to access package resources, for example. Are you saying that you also feel wheels should never be importable, since one can't guarantee whether any code in general will work correctly from a zip? And yet, we have PEP 441, which is intended to encourage use of zipimport. Do we say, "no point - there's bound to be people out there who'll use __file__ instead of pkgutil"? > 3. It is no easier than sys.path.insert(0, wheelname). All it adds > over that is compatibility checking and the ability to import C > extensions (see above on why I think that's a bad thing). As for > compatibility checking, I'd prefer a distlib.wheel.check_compatibility > API that people could call *before* manually inserting the wheel onto > sys.path. That's a better separation of concerns, in my view. There's no problem with providing a compatibility check API, but the other POV is that people might forget to call that API, so it seems to make sense to call it from the mount method too. And I don't see why you say "all it adds" - surely the check is important to prevent a certain class of "weird things" happening. > I can't give specific examples of "failure modes" because I don't use > the wheel mount functions, nor do I typically add wheels to sys.path. > When I did (in virtualenv) I hit a number of issues, but all of these > were ones I fixed in user code (either in virtualenv itself, or in > pip, which was the wheel I was importing). So you could reasonably When I looked into running pip from a zip, it was clear that the issues were related to pip's use of __file__ and such, and nothing to do with shortcomings in the wheel format or zipimport. > dismiss these as "not related to the mount API" to which all I can say > is that if I'd been able to do wheel.mount(wheelname) I would likely > have put less thought into whether what I was doing was a good idea - > and *that's* what I think is the bad aspect of the API. Again, this feels like saying "zipimport is bad because people use __file__ instead of pkgutil". Having any API available doesn't absolve people from the responsibility of thinking about what they're doing and what the implications of using that API are. > Ultimately, I'll just never use the distlib mount functionality, and > I'll recommend not using it when (if) people ask. But I'd rather it > were not there to prompt the question. You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone. Regards, Vinay Sajip From ncoghlan at gmail.com Wed Jan 29 12:19:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 21:19:20 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: On 29 January 2014 20:36, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > > You don't want it to be there, even if it might be useful to others, > just because it isn't useful to you? It's not as if distlib is forcing > that functionality on anyone. I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that isn't easy for a quick scan of the files in the zip archive to detect (especially since the PEP 376 installation database PEP doesn't include any support for arbitrary metapath importers). By contrast PEP 441 is a *distribution* utility - the creator of the application is expected to ensure that doing so actually works correctly before publishing their app that way, just as we would expect py2exe, py2app and cx-freeze users to do. With the "reference implementation" position that distlib is likely to occupy in a post-PEP-426/440/459 world, though, there's an additional legitimate concern about allowing end users to easily distinguish between "this API is fully supported by the PyPA as part of the reference implementation for metadata 2.0" and "this is an experimental packaging related API that may or may not be useful in general, and some members of the PyPA may still have grave reservations about it". At the moment, distlib contains both kinds of API, and it confuses *us*, let alone anyone else that isn't closely following along on distutils-sig. As long as distlib is serving the dual role of providing both "the reference implementation for metadata 2.0" and "some experimental packaging related APIs", we're going to get concerns like this one arising. If there was a clear way to distinguish them (ideally with a separate project for either the reference implementation or the experimental stuff, but even a distinct namespace within the distlib project would help a great deal), I suspect there would be less concern. In the specific case of distlib.mount, if it's eventually combined with a metadata extension like "distlib.mount" which packages must export in order for the command to allow them to be automatically used that way, then I don't see anything wrong with it *in general* - it's a natural extension of the setuptools "zip_safe" flag, but with the ability to include additional details (like whether or not there are C extensions that need to be automatically extracted). Note that this goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools "attempt to guess": if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like "assume_zip_safe=True" or "force=True"). However, like Paul, I have some concerns about a still experimental API like that being in the metadata 2.0 reference implementation, since that will likely end up having to deal with stdlib-like levels of backwards compatibility requirements, and removing experimental APIs that we later decided we weren't happy with could prove problematic. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jan 29 12:26:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 21:26:06 +1000 Subject: [Distutils] wheels, metadata In-Reply-To: <1390508328.55711.YahooMailBasic@web172406.mail.ir2.yahoo.com> References: <1390508328.55711.YahooMailBasic@web172406.mail.ir2.yahoo.com> Message-ID: On 24 January 2014 06:18, Vinay Sajip wrote: > > -------------------------------------------- > >> certainly mention the distlib implementations, >> but also let's be clear if there is a pypa-recommend >> tool that is user-facing (like pip), that is using those >> parts of distlib. >> In most cases, that is not true currently. >> As for mentioning distil, I'm inclined to >> say no. Up to this point, you've presented it as a >> proof of concept.If you're wanting to mention >> "distil" as a real option for users, I'm >> concerned about fracturing the mind of users, but it's >> something to discuss I guess. > > I hear what you're saying. I've positioned distil as a proof of concept purely because it hasn't had widespread use, but I certainly expect it to fulfil the same role as pip functionally (which it must do to be an effective test-bed for distlib). I understand that pip is the officially recommended tool, and don't want to muddy the waters, there being enough confusion about packaging in the wider community. It seems a shame that some of the improvements over pip won't be more widely available, but such is life. I have the use of them, so there's that :-) The fact we're still working on PEP 426/440/459 so distlib and distil are chasing a moving target also makes it a little difficult to recommend them to end users at this point :) I actually expect that we'll see many of the internals of pip significantly refactored in the next 12 months or so - in addition to metadata 2.0, there's also The Update Framework support as a result of PEP 458, and once we get proper metadata publication on PyPI, then adopting a real dependency solver becomes a far more viable option than it is today (and both conda and Fedora's hawkey have tackled the problem of making a depsolver available to a Python based installation tool). pip, ultimately, is just a CLI - so long as that remains the case, then the internals can change radically (which is why I agree with the idea of it *not* having a public Python API, and instead leaving that to distlib). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 12:41:41 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 06:41:41 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> Message-ID: On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: > As I recall, it was in the original version of the PEP/spec and it was > always an intended feature. The wheel file for the wheel project > itself is (deliberately) runnable as a zip file, so that you can > bootstrap into the wheel world using the "wheel install" command. I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick?s most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Wed Jan 29 12:46:51 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 11:46:51 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: On 29 January 2014 10:36, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> 1. It does *not* just use the fact that wheels are importable. It goes >> beyond that and *extracts* C extensions to make them importable, too. >> That is a workaround for a known and accepted limitation of zipimport, >> and as a workaround, it has issues. If C extensions in zipfiles could >> work reliably, this should go into zipimport itself, and *not* into >> 3rd party code. Then everyone would benefit. > > (a) The mount method does not do the extraction of C extensions > *unconditionally* - only when there is suitable indication in the > metadata. Suitable indication of what, exactly? It's not possible (at an OS level) to link to a DLL in a zipfile, so if there's a DLL in there and it's used, it needs to be extracted. I don't really see much point hairsplitting over whether that counts as "unconditional" or not. > (b) What are these issues to which you refer? As I said in my other > response, I don't see the zipped-egg problem as the same, because that > sys.path manipulation is under the hood/not under the developer's > control. I did point out that I knew I was being non-specific. But I know people have reported problems in the past with attempts (and there have been many) to do this. Getting cleanup right is one area. Picking a suitable location to extract to is another. But conceded, this is anecdotal. Feel free to ignore it if that's what you want to do. > (c) Why can't third-party code break new ground, which may or may not > prove fertile? If considered beneficial, it could always be added > to zipimport at a later date. That's how a lot of functionality has > entered the stdlib, after all. This is *not* new ground. Gordon McMillan's importer did it years ago, back when zipimport was first implemented. Eggs did it in pkg_resources. I think py2exe explicitly decided *not* to do it because of known issues with Gordon McMillan's implementation (feel free to call this anecdotal again if you want - but you may be able to find the details with some research). If this was really something that had never been tried before, I'd be happy with it as an "experimental" API. But it's neither marked as experimental, nor is it new. Your implementation *may* have found some new approach - but from your descriptions, I'm not sure it has, and you seem not to have been aware of or researched the previous attempts, so I'm guessing you're not aware of the prior art here. >> 2. It makes what should be a rare use case, to be used only when the >> code in the wheel has been carefully checked to work from a zipfile, >> seem like a common and straightforward operation. (The "attractive >> nuisance" argument). I believe that people using this API will >> typically *not* check the code, and will blame the wheel format, or >> distlib, when their application does not work as expected. > > Doesn't the same argument apply to zipimport? Any code which is zipimported > shouldn't use __file__ manipulation to access package resources, for > example. Are you saying that you also feel wheels should never be > importable, since one can't guarantee whether any code in general will > work correctly from a zip? And yet, we have PEP 441, which is intended to > encourage use of zipimport. Do we say, "no point - there's bound to be > people out there who'll use __file__ instead of pkgutil"? No, I'm saying that hiding the subtleties may be doing people a disservice. That's why I described it as an "attractive nuisance". I'm fully aware that this is a judgement call, and one person's convenience API is another's disaster waiting to happen. Again, I'me fine with your opinion not matching mine. >> 3. It is no easier than sys.path.insert(0, wheelname). All it adds >> over that is compatibility checking and the ability to import C >> extensions (see above on why I think that's a bad thing). As for >> compatibility checking, I'd prefer a distlib.wheel.check_compatibility >> API that people could call *before* manually inserting the wheel onto >> sys.path. That's a better separation of concerns, in my view. > > There's no problem with providing a compatibility check API, but the > other POV is that people might forget to call that API, so it seems to > make sense to call it from the mount method too. And I don't see why you > say "all it adds" - surely the check is important to prevent a certain > class of "weird things" happening. OK, I misspoke. All that mount adds over a compatibility API is a sys.path.insert call (*if* you agree with my point that the C extension stuff shouldn't be used - and I know you don't). Breaking mount down: 1. Compatibility check - I support this and would like a separate API 2. sys.path modification - trivial for anyone who knows enough to be using this API 3. C extension support - we'll have to agree to disagree here >> I can't give specific examples of "failure modes" because I don't use >> the wheel mount functions, nor do I typically add wheels to sys.path. >> When I did (in virtualenv) I hit a number of issues, but all of these >> were ones I fixed in user code (either in virtualenv itself, or in >> pip, which was the wheel I was importing). So you could reasonably > > When I looked into running pip from a zip, it was clear that the issues > were related to pip's use of __file__ and such, and nothing to do > with shortcomings in the wheel format or zipimport. You're agreeing with me here - I just *said* you can reasonably dismiss these as not related to the mount API. >> dismiss these as "not related to the mount API" to which all I can say >> is that if I'd been able to do wheel.mount(wheelname) I would likely >> have put less thought into whether what I was doing was a good idea - >> and *that's* what I think is the bad aspect of the API. > > Again, this feels like saying "zipimport is bad because people use > __file__ instead of pkgutil". Having any API available doesn't absolve > people from the responsibility of thinking about what they're doing and > what the implications of using that API are. You're missing my point - zipimport is not bad because people use __file__ - pkgutil is *good* because it gives people a means of getting round some of the limitations of zipimport. But it's all layers. Convenience APIs are *abstractions*. They make things *convenience* - and one aspect of that convenience is being able to avoid bothering about some of the details. My concern is that the mount API makes it too easy for people not to think about whether what they are mounting is mountable. If the API documentation clearly stated what was mountable then this would not be an issue (but the docs would be huge and scary). But the API wouldn't look as attractive, either. >> Ultimately, I'll just never use the distlib mount functionality, and >> I'll recommend not using it when (if) people ask. But I'd rather it >> were not there to prompt the question. > > You don't want it to be there, even if it might be useful to others, > just because it isn't useful to you? It's not as if distlib is forcing > that functionality on anyone. No, no, no. Maybe you asked for facts and I gave opinions. If so I apologise. But what I'm saying is that I feel that the API might encourage people to make mistakes they would not otherwise have considered, and that's a shame. If the API is there, personally I'll just never use it. So I don't *care* as such. Let's just agree to disagree. It's a pretty small point in the overall picture. Cheers, Paul. From donald at stufft.io Wed Jan 29 12:47:12 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 06:47:12 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: On Jan 29, 2014, at 2:32 AM, Nick Coghlan wrote: > All of those arguments are against *recommending* directly importing > from wheels. Yes, there are lots of problems with running directly > from a zip archive, which is why the fact that easy_install *actively > encourages* potentially inexperienced users to run things that way is > so problematic. > > However, for people that are comfortable with the import system and > the limitations of direct zip imports, it's a very useful feature. I > wouldn't have accepted PEP 427 without the zipimport compatibility > that meant a developer *could* use it as a direct replacement for eggs > if they really wanted to. > > Otherwise we'd have to define a whole new format for something that > can be adequately handled by a wheel that meets certain restrictions, > and that would be pointless (we already have too many formats, and we > wanted the wheel format to offer a strict superset of the egg format's > capabilities). > > I clarified PEP 427 specifically because Armin Ronacher's wheel > article showed that he was unaware of this *deliberately included* > feature of the wheel format. People are free not to like it - the > default tools deliberately make it less convenient to run things that > way, and that's as it should be. However, it's not a super-secret > capability only to be used by us to implement things like ensurepip - > it's a defined capability of the format that if your software is > capable of running correctly from a zip archive in the first place, > then that archive can be also be a valid wheel file. > > Running directly from a wheel is a power tool - that's a reason to put > "handle with care" warnings on it, not to refuse to support a feature > that was deliberately designed into the format. You can do a lot more > damage with a badly written meta-importer, yet we have no intention of > deprecating that capability either. Even *.pth files have turned out > to have a valid use case for sharing packages between virtual > environments. > > We have lots of features like that elsewhere in Python - when people > ask about metaclasses, the first reaction is "You probably don't want > to use them". However, sometimes developers *do* need them, and that's > why the feature exists. Most of the time developers won't want to make > use of the zipimport compatibility of wheel files, either, but > advanced use cases like ensurepip are exactly why the capability > exists. > > I can make the new note in the PEP more explicit that while this is a > supported use case that ensures the feature set provided by wheels is > a strict superset of that provided by eggs, that's not the same thing > as *recommending* that wheels be used that way. > > Cheers, > Nick. I don?t think it particularly matters wether you would have accepted the PEP without that ability or not. You *did* accept the PEP when the text of the PEP directly pointed out that one of the differences of Wheel and Egg were that ?Wheel is an installation format, Egg is importable? which points to that fact that Wheel was not designed to be importable. As far as I can tell you?ve added this ?feature? to the PEP by fiat without any chance for anyone to be able to discuss it after it?s already been accepted. I?m well aware that we cannot prevent Wheels from being imported, but that?s another thing all together from directly supporting it. For one thing by supporting it we forever lock ourselves into dumping what would be in site-packages directly in the root of the Wheel because adding the .whl file directly to sys.path is now a documented feature. This behavior has personally caused me annoyance and one of the things I was hoping on doing in a Wheel 1.2 was revise the layout a bit to make installation simpler and make inspecting a Wheel file easier. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Jan 29 12:50:36 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 06:50:36 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: On Jan 29, 2014, at 6:47 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 2:32 AM, Nick Coghlan wrote: > >> All of those arguments are against *recommending* directly importing >> from wheels. Yes, there are lots of problems with running directly >> from a zip archive, which is why the fact that easy_install *actively >> encourages* potentially inexperienced users to run things that way is >> so problematic. >> >> However, for people that are comfortable with the import system and >> the limitations of direct zip imports, it's a very useful feature. I >> wouldn't have accepted PEP 427 without the zipimport compatibility >> that meant a developer *could* use it as a direct replacement for eggs >> if they really wanted to. >> >> Otherwise we'd have to define a whole new format for something that >> can be adequately handled by a wheel that meets certain restrictions, >> and that would be pointless (we already have too many formats, and we >> wanted the wheel format to offer a strict superset of the egg format's >> capabilities). >> >> I clarified PEP 427 specifically because Armin Ronacher's wheel >> article showed that he was unaware of this *deliberately included* >> feature of the wheel format. People are free not to like it - the >> default tools deliberately make it less convenient to run things that >> way, and that's as it should be. However, it's not a super-secret >> capability only to be used by us to implement things like ensurepip - >> it's a defined capability of the format that if your software is >> capable of running correctly from a zip archive in the first place, >> then that archive can be also be a valid wheel file. >> >> Running directly from a wheel is a power tool - that's a reason to put >> "handle with care" warnings on it, not to refuse to support a feature >> that was deliberately designed into the format. You can do a lot more >> damage with a badly written meta-importer, yet we have no intention of >> deprecating that capability either. Even *.pth files have turned out >> to have a valid use case for sharing packages between virtual >> environments. >> >> We have lots of features like that elsewhere in Python - when people >> ask about metaclasses, the first reaction is "You probably don't want >> to use them". However, sometimes developers *do* need them, and that's >> why the feature exists. Most of the time developers won't want to make >> use of the zipimport compatibility of wheel files, either, but >> advanced use cases like ensurepip are exactly why the capability >> exists. >> >> I can make the new note in the PEP more explicit that while this is a >> supported use case that ensures the feature set provided by wheels is >> a strict superset of that provided by eggs, that's not the same thing >> as *recommending* that wheels be used that way. >> >> Cheers, >> Nick. > > I don?t think it particularly matters wether you would have accepted the > PEP without that ability or not. You *did* accept the PEP when the text > of the PEP directly pointed out that one of the differences of Wheel > and Egg were that ?Wheel is an installation format, Egg is importable? > which points to that fact that Wheel was not designed to be importable. As > far as I can tell you?ve added this ?feature? to the PEP by fiat without any > chance for anyone to be able to discuss it after it?s already been accepted. > > I?m well aware that we cannot prevent Wheels from being imported, but > that?s another thing all together from directly supporting it. For one thing > by supporting it we forever lock ourselves into dumping what would be > in site-packages directly in the root of the Wheel because adding the .whl > file directly to sys.path is now a documented feature. This behavior has > personally caused me annoyance and one of the things I was hoping on > doing in a Wheel 1.2 was revise the layout a bit to make installation simpler > and make inspecting a Wheel file easier. > As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.jerdonek at gmail.com Wed Jan 29 12:52:17 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 29 Jan 2014 03:52:17 -0800 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> Message-ID: On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: > >> As I recall, it was in the original version of the PEP/spec and it was >> always an intended feature. The wheel file for the wheel project >> itself is (deliberately) runnable as a zip file, so that you can >> bootstrap into the wheel world using the "wheel install" command. > > I just read every version of the PEP that has ever existed in Mercurial > and no version besides Nick's most recent contains any text about > the importability of Wheels besides that one of the differences of > Wheel and Egg is that Wheel is an installation format and Egg is > importable. FWIW, Nick clarified this aspect of wheels on this list at least once -- back in Sep 2012 when the PEP was first introduced. See this email: https://mail.python.org/pipermail/distutils-sig/2012-September/018959.html "It's also not quite true that the contents of a wheel aren't importable - since they're still just a zipfile, they can still be added to an import path in the same manner as egg files. It's just not *recommended* to use them that way, as the format is designed primarily for use in distribution rather than runtime imports." --Chris From donald at stufft.io Wed Jan 29 12:58:51 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 06:58:51 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> Message-ID: <1C85E701-242F-4175-8F48-4C58D269F293@stufft.io> That doesn't really speak to the fact they were designed for that. If I read that it looks like some commenting that they are importable (which as it stands they are) and not someone calling it a supported feature of the format. It even states that the format is designed primarily for distribution. Given that the PEPs text had always been contrary to the position that wheels were designed to be importable I don't think you can draw any other conclusion. > On Jan 29, 2014, at 6:52 AM, Chris Jerdonek wrote: > >> On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft wrote: >> >>> On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: >>> >>> As I recall, it was in the original version of the PEP/spec and it was >>> always an intended feature. The wheel file for the wheel project >>> itself is (deliberately) runnable as a zip file, so that you can >>> bootstrap into the wheel world using the "wheel install" command. >> >> I just read every version of the PEP that has ever existed in Mercurial >> and no version besides Nick's most recent contains any text about >> the importability of Wheels besides that one of the differences of >> Wheel and Egg is that Wheel is an installation format and Egg is >> importable. > > FWIW, Nick clarified this aspect of wheels on this list at least once > -- back in Sep 2012 when the PEP was first introduced. See this > email: > > https://mail.python.org/pipermail/distutils-sig/2012-September/018959.html > > "It's also not quite true that the contents of a wheel aren't > importable - since they're still just a zipfile, they can still be > added to an import path in the same manner as egg files. It's just not > *recommended* to use them that way, as the format is designed > primarily for use in distribution rather than runtime imports." > > --Chris From p.f.moore at gmail.com Wed Jan 29 12:59:58 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 11:59:58 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> Message-ID: On 29 January 2014 11:41, Donald Stufft wrote: > > On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: > >> As I recall, it was in the original version of the PEP/spec and it was >> always an intended feature. The wheel file for the wheel project >> itself is (deliberately) runnable as a zip file, so that you can >> bootstrap into the wheel world using the "wheel install" command. > > I just read every version of the PEP that has ever existed in Mercurial > and no version besides Nick's most recent contains any text about > the importability of Wheels besides that one of the differences of > Wheel and Egg is that Wheel is an installation format and Egg is > importable. Apologies. It was something Daniel pointed out a few times very early on - I hadn't realised it wasn't in the spec directly. What is in the spec - which effectively constrains the format to *allowing* (rather than encouraging) direct import - are the facts that wheels are zip format, and that one of purelib/platlib is at the root. The concept of separating "unpack" and "spread" and the comment "Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages" doesn't leave any room for wheels *not* to be importable in the majority of pure-python, no package data, cases. Debating how we present this in later versions of the wheel spec is fine. But deliberately making wheels not importable would break backward compatibility in a way that would have other, likely more serious, implications. Nevertheless, I understand your concerns, and I think we should be very careful not to let people get the impression that this is in any way similar to "importable eggs", which had a very bad press. Paul From donald at stufft.io Wed Jan 29 13:18:28 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 07:18:28 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> Message-ID: <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> On Jan 29, 2014, at 6:59 AM, Paul Moore wrote: > On 29 January 2014 11:41, Donald Stufft wrote: >> >> On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: >> >>> As I recall, it was in the original version of the PEP/spec and it was >>> always an intended feature. The wheel file for the wheel project >>> itself is (deliberately) runnable as a zip file, so that you can >>> bootstrap into the wheel world using the "wheel install" command. >> >> I just read every version of the PEP that has ever existed in Mercurial >> and no version besides Nick's most recent contains any text about >> the importability of Wheels besides that one of the differences of >> Wheel and Egg is that Wheel is an installation format and Egg is >> importable. > > Apologies. It was something Daniel pointed out a few times very early > on - I hadn't realised it wasn't in the spec directly. > > What is in the spec - which effectively constrains the format to > *allowing* (rather than encouraging) direct import - are the facts > that wheels are zip format, and that one of purelib/platlib is at the > root. The concept of separating "unpack" and "spread" and the comment > "Although a specialized installer is recommended, a wheel file may be > installed by simply unpacking into site-packages" doesn't leave any > room for wheels *not* to be importable in the majority of pure-python, > no package data, cases. > > Debating how we present this in later versions of the wheel spec is > fine. But deliberately making wheels not importable would break > backward compatibility in a way that would have other, likely more > serious, implications. > > Nevertheless, I understand your concerns, and I think we should be > very careful not to let people get the impression that this is in any > way similar to "importable eggs", which had a very bad press. > > Paul I read that statement differently, in that it doesn?t guarantee that the files would be at the *root* of the Wheel, just that you could unpack a Wheel into a site-packages directory using unzip and have it be ?installed?. I can see how it could be read otherwise though. In either case this is something that is more able to be removed in later versions of Wheel because unpacking by hand is something I don?t believe will ever be commonplace, and any installer that doesn?t check it?s Wheel version before doing things with the Wheel is wrong. But more importantly, while I?m against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion. I don?t think anyone can call me a casual observer of distutils-sig or the various PEP processes and this was the first time that I had heard of Wheels being importable as anything other than a sometimes useful hack that wasn?t a design goal but instead just an accident of implementation. I followed the PEP closely trying to make sure that we weren?t going to add a supported feature that locked us into any corners while the format was still new and this is something I would have fought against had it been in the original PEP. Nick may have, in his head, considered this to be a feature of Wheel all along and not an implementation detail, however that is not what the PEP stated. I don?t believe that as BDFL-Delegate for packaging issues Nick should be able to add supported features to an already accepted PEP without discussion especially when the existing text of that PEP is directly contrary to that feature being part of the PEP. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Wed Jan 29 13:43:00 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 12:43:00 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> Message-ID: On 29 January 2014 12:18, Donald Stufft wrote: > But more importantly, while I'm against officially supporting importable Wheels > because of various reasons, my biggest problem is that this was added in > without any chance for discussion. Fair point. I'm sure Nick did just think of it as clarification - I know I did - but the process point is reasonable. I'm happy to leave off arguing with your position till the point comes for proper debate ;-) Paul From donald at stufft.io Wed Jan 29 13:57:27 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 07:57:27 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> Message-ID: On Jan 29, 2014, at 7:43 AM, Paul Moore wrote: > On 29 January 2014 12:18, Donald Stufft wrote: >> But more importantly, while I'm against officially supporting importable Wheels >> because of various reasons, my biggest problem is that this was added in >> without any chance for discussion. > > Fair point. I'm sure Nick did just think of it as clarification - I > know I did - but the process point is reasonable. I'm happy to leave > off arguing with your position till the point comes for proper debate > ;-) > > Paul Let me be explicit that I don?t think Nick is doing anything nefarious here, I do believe that he probably did think of it as a clarification. I?m concerned with the fact that the PEP did not support this when it was accepted and contained language that, to me at least, is contrary to the fact that this was supported, and then the new text was addd to an already accepted PEP giving it a new feature without any chance for discussion or disagreement. If I wasn?t anal about reading every email that hits my inbox *and* I wasn?t on python-checkins I never would have noticed this addition until it was likely too late to do anything about it because it had been documented as part of the PEP long enough that people started to depend on it. So hopefully nobody thinks I?m calling Nick a bad actor in this case. Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 13:58:23 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 22:58:23 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: On 29 January 2014 21:50, Donald Stufft wrote: > > As a follow up, even if this becomes something we document and actually > support, then it should be done in the next version of Wheel when people > have the chance to discuss it. It should not be added ex post facto by > a committer to the PEP repo just because they want Wheel to have it. It was discussed during the approval process for the wheel format, I approved it on the basis that wheels provided a superset of the functionality of eggs. I merely missed the fact it wasn't properly recorded in the PEP (which is now rectified). You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 13:59:03 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 07:59:03 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: <83BC4D25-2143-4C24-991A-C28A21D0A86F@stufft.io> On Jan 29, 2014, at 7:58 AM, Nick Coghlan wrote: > On 29 January 2014 21:50, Donald Stufft wrote: >> >> As a follow up, even if this becomes something we document and actually >> support, then it should be done in the next version of Wheel when people >> have the chance to discuss it. It should not be added ex post facto by >> a committer to the PEP repo just because they want Wheel to have it. > > It was discussed during the approval process for the wheel format, I > approved it on the basis that wheels provided a superset of the > functionality of eggs. I merely missed the fact it wasn't properly > recorded in the PEP (which is now rectified). > > You can campaign to deprecate that feature *if* we ever want to make a > change to the wheel format that is incompatible with it. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia Can you point to this discussion? Because I cannot find it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Wed Jan 29 14:10:32 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 13:10:32 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: On 29 January 2014 12:58, Nick Coghlan wrote: > You can campaign to deprecate that feature *if* we ever want to make a > change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul From ncoghlan at gmail.com Wed Jan 29 14:14:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 23:14:37 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> Message-ID: On 29 January 2014 22:57, Donald Stufft wrote: > Also to be specific, what I believe should be done is that the change should be > reverted, and if it is desired that Wheels officially support zip import then in the > next version of the Wheel spec it should be added when everyone has a chance > to properly discuss it. Whether you like it or not, the wheel spec *does* already support being used that way, and changing it *would* be a backwards incompatible change to the format (and hence not acceptable without a compelling justification, and there isn't one that wouldn't equally well apply to other powerful but potentially confusing features like metaclasses and monkeypatching). I am also not willing to allow the perception that the wheel format does not offer a strict superset of the feature of the egg format to persist until we have a defined wheel 1.1 spec (which we don't even have a volunteer to work on yet). So, rather than removing it, I have instead updated the note in PEP 427 to read: ============================= Is it possible to import Python code directly from a wheel file? Yes, the wheel format is deliberately designed to be compatible with Python's support for importing from zip files, ensuring that it provides a superset of the functionality provided by the preceding egg format. However, this is generally not a *recommended* approach to using wheel files, as importing from a zip file rather than an ordinary filesystem directory imposes a number of additional constraints that will often break the assumptions of Python developers (for example, C extensions cannot be imported directly from a zip archive, and the ``__file__`` attribute no longer refers to an ordinary filesystem path, but to a combination path that includes both the location of the zip archive on the filesystem and the relative path to the module inside the archive). Like metaclasses and metapath importers, if you're not sure if you need to take advantage of this feature, you almost certainly don't need it. ============================= Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 14:16:37 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:16:37 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> Message-ID: <9B3C40F5-1129-4E0B-B6AF-161A0C80B729@stufft.io> On Jan 29, 2014, at 8:14 AM, Nick Coghlan wrote: > On 29 January 2014 22:57, Donald Stufft wrote: >> Also to be specific, what I believe should be done is that the change should be >> reverted, and if it is desired that Wheels officially support zip import then in the >> next version of the Wheel spec it should be added when everyone has a chance >> to properly discuss it. > > Whether you like it or not, the wheel spec *does* already support > being used that way, and changing it *would* be a backwards > incompatible change to the format (and hence not acceptable without a > compelling justification, and there isn't one that wouldn't equally > well apply to other powerful but potentially confusing features like > metaclasses and monkeypatching). > > I am also not willing to allow the perception that the wheel format > does not offer a strict superset of the feature of the egg format to > persist until we have a defined wheel 1.1 spec (which we don't even > have a volunteer to work on yet). > > So, rather than removing it, I have instead updated the note in PEP 427 to read: > > ============================= > Is it possible to import Python code directly from a wheel file? > Yes, the wheel format is deliberately designed to be compatible with > Python's support for importing from zip files, ensuring that it > provides a superset of the functionality provided by the preceding > egg format. > > However, this is generally not a *recommended* approach to using wheel > files, as importing from a zip file rather than an ordinary filesystem > directory imposes a number of additional constraints that will often > break the assumptions of Python developers (for example, C extensions > cannot be imported directly from a zip archive, and the ``__file__`` > attribute no longer refers to an ordinary filesystem path, but to > a combination path that includes both the location of the zip archive > on the filesystem and the relative path to the module inside the > archive). > > Like metaclasses and metapath importers, if you're not sure if you need > to take advantage of this feature, you almost certainly don't need it. > ============================= > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that? Can you point to a single line in the PEP that supports this besides the ones you?ve added? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 14:17:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 23:17:16 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: On 29 January 2014 23:10, Paul Moore wrote: > On 29 January 2014 12:58, Nick Coghlan wrote: >> You can campaign to deprecate that feature *if* we ever want to make a >> change to the wheel format that is incompatible with it. > > Note that virtualenv uses the ability to run wheels from sys.path. I > don't believe that virtualenv should be a "special exception" here, > nor do I want it to rely on accidents of the implementation. > > So from my POV, pypa is already shipping code that relies on this > behaviour being by design. Not just PyPA - this feature is at the heart of how the ensurepip module in the standard library works. This approach is explicitly documented in PEP 453: http://www.python.org/dev/peps/pep-0453/#implementation-strategy Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 14:20:43 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:20:43 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: <7D01C594-B52F-4A10-A664-E547556F2728@stufft.io> On Jan 29, 2014, at 8:17 AM, Nick Coghlan wrote: > On 29 January 2014 23:10, Paul Moore wrote: >> On 29 January 2014 12:58, Nick Coghlan wrote: >>> You can campaign to deprecate that feature *if* we ever want to make a >>> change to the wheel format that is incompatible with it. >> >> Note that virtualenv uses the ability to run wheels from sys.path. I >> don't believe that virtualenv should be a "special exception" here, >> nor do I want it to rely on accidents of the implementation. >> >> So from my POV, pypa is already shipping code that relies on this >> behaviour being by design. > > Not just PyPA - this feature is at the heart of how the ensurepip > module in the standard library works. > > This approach is explicitly documented in PEP 453: > http://www.python.org/dev/peps/pep-0453/#implementation-strategy > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia There is a very important distinction. Both of those are designed to work with very specific Wheels and relied on, until this change, private, undocumented, and undiscussed features of Wheel. It?s not unusual for the Python standard library to make use of internal APIs to make it?s features work nor is it unusual for Virtualenv to do that either because of the nature of it bolting things on to Python that Python wasn?t designed to do. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 14:31:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 23:31:22 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <9B3C40F5-1129-4E0B-B6AF-161A0C80B729@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> <9B3C40F5-1129-4E0B-B6AF-161A0C80B729@stufft.io> Message-ID: On 29 January 2014 23:16, Donald Stufft wrote: > So basically even though the text of the PEP specifically points out that a difference > of Wheel and Egg is that Eggs are importable it somehow supports that? Can you > point to a single line in the PEP that supports this besides the ones you've added? I added the clarification based on the facts that: 1. We discussed this extensively before PEP 427 was approved, and this was an accepted feature of the design 2. Root-is-purelib only makes sense in the context of supporting the feature 3. Both ensurepip and virtualenv rely on the feature 4. PEP 453 explicitly documents ensurepip's reliance on the feature, with no caveat about this being unsupported in the spec 5. I wouldn't have accepted PEP 427 if wheels didn't provide a strict superset of the features provided by eggs We make mistakes, and things that were discussed and agreed don't always get properly captured in the corresponding PEPs. When that happens, it's a judgement call as to whether properly documenting that is a new feature requiring a new PEP, or merely a clarification to the existing one. For standard library PEPs, we often don't do either - we just fix the implementation without going back for another round of PEP discussions (for smaller tweaks, sometimes we don't even go back to python-dev and instead just resolve things on the tracker). In this case, as BDFL-Delegate, I decided it was a case that merely called for clarification, because I *know* what spec I accepted, and it was the one where wheels offer all the features that eggs do and more. I added the new text specifically because people like Armin Ronacher and yourself had gained an idea from the PEP text that emphatically does *not* align with the design discussions that occurred prior to the acceptance of the PEP. Now, if you'd like to campaign to *remove* this support, then explain your rationale, and make the case for why you think providing the feature is so dangerous that removing it is worth breaking backwards compatibility over. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 14:31:33 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:31:33 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: > On 29 January 2014 12:58, Nick Coghlan wrote: >> You can campaign to deprecate that feature *if* we ever want to make a >> change to the wheel format that is incompatible with it. > > Note that virtualenv uses the ability to run wheels from sys.path. I > don't believe that virtualenv should be a "special exception" here, > nor do I want it to rely on accidents of the implementation. > > So from my POV, pypa is already shipping code that relies on this > behaviour being by design. > > Paul Here?s Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 14:32:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 23:32:53 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On 29 January 2014 23:31, Donald Stufft wrote: > > Here's Paul explicitly mentioning that Wheels being used with zip import is > an incidental benefit and not a core feature. > > https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 14:33:30 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:33:30 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <1864C52F-782F-4F6A-ADF0-352E7B9168A8@stufft.io> <992A7ABE-DC2A-4EF5-B29E-189365E5F7FE@stufft.io> <855E7600-F3B2-49FE-B24F-A447B34D323C@stufft.io> <9B3C40F5-1129-4E0B-B6AF-161A0C80B729@stufft.io> Message-ID: On Jan 29, 2014, at 8:31 AM, Nick Coghlan wrote: > On 29 January 2014 23:16, Donald Stufft wrote: >> So basically even though the text of the PEP specifically points out that a difference >> of Wheel and Egg is that Eggs are importable it somehow supports that? Can you >> point to a single line in the PEP that supports this besides the ones you've added? > > I added the clarification based on the facts that: > > 1. We discussed this extensively before PEP 427 was approved, and this > was an accepted feature of the design > 2. Root-is-purelib only makes sense in the context of supporting the feature > 3. Both ensurepip and virtualenv rely on the feature > 4. PEP 453 explicitly documents ensurepip's reliance on the feature, > with no caveat about this being unsupported in the spec > 5. I wouldn't have accepted PEP 427 if wheels didn't provide a strict > superset of the features provided by eggs > > We make mistakes, and things that were discussed and agreed don't > always get properly captured in the corresponding PEPs. > > When that happens, it's a judgement call as to whether properly > documenting that is a new feature requiring a new PEP, or merely a > clarification to the existing one. For standard library PEPs, we often > don't do either - we just fix the implementation without going back > for another round of PEP discussions (for smaller tweaks, sometimes we > don't even go back to python-dev and instead just resolve things on > the tracker). > > In this case, as BDFL-Delegate, I decided it was a case that merely > called for clarification, because I *know* what spec I accepted, and > it was the one where wheels offer all the features that eggs do and > more. I added the new text specifically because people like Armin > Ronacher and yourself had gained an idea from the PEP text that > emphatically does *not* align with the design discussions that > occurred prior to the acceptance of the PEP. > > Now, if you'd like to campaign to *remove* this support, then explain > your rationale, and make the case for why you think providing the > feature is so dangerous that removing it is worth breaking backwards > compatibility over. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia This was supposedly extensively discussed prior to PEP427 being accepted yet I have no memory of this being discussed, am unable to find any discussion of it (other than one offs saying it?s possible but not a core feature), and you?ve been apparently unwilling to point to any discussion. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Jan 29 14:34:12 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:34:12 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On Jan 29, 2014, at 8:32 AM, Nick Coghlan wrote: > On 29 January 2014 23:31, Donald Stufft wrote: >> >> Here's Paul explicitly mentioning that Wheels being used with zip import is >> an incidental benefit and not a core feature. >> >> https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html > > Donald, I was *there*. I know what we discussed, and I know what PEP I > approved. Trying to play gotcha with links *isn't going to work*. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia If I?m wrong, then by all means show me where it was discussed so I can admit I was wrong. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Wed Jan 29 04:41:32 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Tue, 28 Jan 2014 22:41:32 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: Message-ID: On Jan 28, 2014, at 9:54 PM, Daniel Holth wrote: > On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin wrote: >> Hi, >> >> I saw that people from this list are responsible for Wheel related PEP. >> I'm comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. >> >> I'm trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... >> >> So, i would like to ask few questions about it : >> >> 1. I'm coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn't it be way easier to deal with the dependencies and imports? >> 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can't be used the same exact way? >> >> >> I would appreciate any insight about the development infrastructure the wheel designers have in mind. >> >> Thanks in advance, >> Eugene > > The main reason the packaging format is not explicitly importable is > simply because Python hasn't had "jar-like" deployment for as long or > as consistently as Java. Thanks a lot for taking time to respond! Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? If wheel is promoted as the ultimate solution to the packaging problems (at least that?s my impression ) aren?t the abilities to import packages and classes from it as well as make it runnable a kind of top priority and something that is right on the surface? > So while Java code universally uses a "get > resource" API to get stuff on the classpath, a a lot of Python code > will try to open a file. C extensions can't be loaded from inside zip > files. I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? I?d think wheel should do everything in its power to provide pain free workflow for pure python, and if there is a need to add some C extension then solve that problem separately? Is the possibility to add C extension into the wheel so critical, that it should prohibit useful feature for pure python modules? > And it's a lot harder to edit .py files once they are zipped > up, unlike Java where only the compiled and non-editable .class files > are zipped. Why would i want to edit the zipped .py file from a wheel? I want to import a package or a class from it, reuse a function. Nothing more. > > ZIP import is a great feature but wheels aren't really designed for > it. It's more reliable that way. More reliable in what regard? My impression is that the whole thing about pip, virtualenv and wheels is great - it takes into consideration a lot of end user problems, aka getting proper module and installing it to RUN applications. But for whatever reason it misses some important problems on the DEVELOPMENT side. I?m trying to find a sane way for developer to work, so if wheel is not an answer - what is? Deprecated Egg? Or may be then egg is not going to go anywhere and pip has to learn to deal with them as well as it does with wheels? Thanks, Eugene From matt at notevencode.com Wed Jan 29 07:48:22 2014 From: matt at notevencode.com (Matthew Iversen) Date: Wed, 29 Jan 2014 17:48:22 +1100 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: On the one hand, it's easy to see the allure of a zipimport-able format. Grab a file, import it, use functionality straight away. It has an attraction of self-containedness. On the other hand, python's dynamic nature means that things are not as simple as Java jar's as Donald points out. I agree that wheels shouldn't officially support this feature wholesale. However, perhaps they could do so conditionally? E.g. designate some mechanism by which it is possible to inspect a wheel and determine that it intends to explicitly be zipimport compatible. A prime candidate would be that the wheel is of a purelib in nature, another would be to add a top-level file or metadata indicating such (e.g. '.zipimportable'). But other wise I agree that we should explicitly seek to avoid any language that could lead users to expect this functionality of wheels wholesale in any way, even though they happen to be regular zip files. On 29 Jan 2014 17:09, "Donald Stufft" wrote: > > On Jan 28, 2014, at 11:14 PM, Vinay Sajip wrote: > > > Donald Stufft stufft.io> writes: > > > >> I think we need to reconsider this. I cannot stress how badly an idea > >> I think it is for Wheels to concern itself with the ability to add the > >> Wheel to sys.path and import it. > > > > I know that people have had bad experiences in the past with eggs, for > the > > reasons Nick outlined in his response. However, you don't say *with > > specifics* why you think putting wheels on sys.path is a bad idea - I > don't > > think it's convincing enough just to hand-wavingly reference problems > with > > the egg format. > > > > When this topic came up before, I asked for specific failure modes which > > were causing concern, but I never got a response IIRC. > > > > One can't say that having the same packaging and importable formats is > > inherently bad, since Java shows otherwise. So if there are specific > > problems, they should be identified. > > Last time this topic came up Jim Fulton did point things out. > > 1. That unpacking into the home directory is problematic because users > that run services often don't have home directories. This you waved > away by saying that the home directory isn't a required place, to which > Jim responded that unpacking them anywhere else was likely to have > access control / security issues and "generally cause pain". Which I > wholly agree with, intact Mitre has assigned CVE's to applications that > did switch their Egg Cache to directories other than HOME. > > 2. Zipped Eggs are more difficult to work with for debugging purposes, > which > you responded saying that Wheels are a "deployment format" (which I > don't believe is supported by the PEP at all) and Jim responded with > his > experience in developing against things that got installed as zipped > Eggs > and the pain that has personally caused him. > > 3. Zipped imports (In Jim's experience) are slower than unpacked imports, > which you replied (quite literally) "Caveat emptor". Jim's response > was that > this had been tried with Egg and had been found to be more pain than it > was worth. In his exact words: > > "It's been tried with eggs. This is not new ground. Encouraging people > to > do this is going to cause pain and resentment. > > I think one of the reasons there's so much (IMO mostly irrational) > hate for > eggs is that people think you can only used zipped eggs, and zipped > eggs > cause pain and agony." > > He closes up with, > > "Importing from zipped eggs has proved itself to be an anti pattern. > > Buildout (as of buildout 2) always unzips eggs." > > So you have pip that doesn't use zipped Eggs, buildout which made sure > in buildout 2 (presumably drawing from the lessons of buildout 1) to always > unzip eggs, and easy_install which does support zipped eggs (sometimes, > if it thinks that particular Egg will support it). > > Why do we insist upon repeating the mistakes of the past? > > > > >> Further more it won't even work accurately for all Wheels (as Nick's > >> edit says) so we require that people wanting to do this figure out if > >> their Wheel can or can not be added to the sys.path (which isn't as > >> simple as looking at the tags because a platform specific Wheel may only > >> contain optional C modules). > > > > I don't yet see a technical impediment here. For example, distlib's wheel > > implementation uses a "mount" method to add a wheel to sys.path. This > uses > > the tags to check for compatibility - there's no "figuring out" that the > > user has to do. If additional metadata about C extensions is available > > (which it is, if the wheel is built from source using distil), then those > > extensions are made available for import, too. I realise that's an > extension > > to the current spec, but then no one is forcing any one to use it. > > Zipped Eggs have a long long history of causing very weird failures > conditions, > you say your Wheel.mount checks for compatibility, does it also check that > the library to be installed isn't using __file__ shenanigans instead of > pkgutil.get_data()? easy_install tried to do this with varying degrees of > success, > I've never seen much except pain from zipped Eggs. > > > > > Regards, > > > > Vinay Sajip > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jan 29 14:39:50 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:39:50 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On Jan 29, 2014, at 8:34 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 8:32 AM, Nick Coghlan wrote: > >> On 29 January 2014 23:31, Donald Stufft wrote: >>> >>> Here's Paul explicitly mentioning that Wheels being used with zip import is >>> an incidental benefit and not a core feature. >>> >>> https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html >> >> Donald, I was *there*. I know what we discussed, and I know what PEP I >> approved. Trying to play gotcha with links *isn't going to work*. >> >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > If I?m wrong, then by all means show me where it was discussed so I can admit > I was wrong. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Ironically, in another thread [1] on distutils-sig we have Daniel Holth, the author of the PEP stating: The main reason the packaging format is not explicitly importable is simply because Python hasn't had "jar-like" deployment for as long or as consistently as Java. So while Java code universally uses a "get resource" API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. [1] https://mail.python.org/pipermail/distutils-sig/2014-January/023554.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Jan 29 14:44:57 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:44:57 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On Jan 29, 2014, at 8:43 AM, Daniel Holth wrote: > On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft wrote: >> >> On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: >> >>> On 29 January 2014 12:58, Nick Coghlan wrote: >>>> You can campaign to deprecate that feature *if* we ever want to make a >>>> change to the wheel format that is incompatible with it. >>> >>> Note that virtualenv uses the ability to run wheels from sys.path. I >>> don't believe that virtualenv should be a "special exception" here, >>> nor do I want it to rely on accidents of the implementation. >>> >>> So from my POV, pypa is already shipping code that relies on this >>> behaviour being by design. >>> >>> Paul >> >> >> Here's Paul explicitly mentioning that Wheels being used with zip import is >> an incidental benefit and not a core feature. >> >> https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html > > I designed it very intentionally to be compatible with zip import. > Otherwise wheels would probably not even be zip files (like pypm > packages which are tar) and would achieve greater compression. > > It may be useful to understand that wheel has *political features* or > if you prefer *setting the defaults based on what we have learned from > eggs*. I don't recommend that they be zip-imported generally but if > you are a consenting adult who understands the caveats you may do so. Then why did you tell someone on this very list a few hours ago that Wheels were not really designed to be zip imported. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed Jan 29 14:46:01 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 Jan 2014 08:46:01 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On Wed, Jan 29, 2014 at 8:44 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 8:43 AM, Daniel Holth wrote: > >> On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft wrote: >>> >>> On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: >>> >>>> On 29 January 2014 12:58, Nick Coghlan wrote: >>>>> You can campaign to deprecate that feature *if* we ever want to make a >>>>> change to the wheel format that is incompatible with it. >>>> >>>> Note that virtualenv uses the ability to run wheels from sys.path. I >>>> don't believe that virtualenv should be a "special exception" here, >>>> nor do I want it to rely on accidents of the implementation. >>>> >>>> So from my POV, pypa is already shipping code that relies on this >>>> behaviour being by design. >>>> >>>> Paul >>> >>> >>> Here's Paul explicitly mentioning that Wheels being used with zip import is >>> an incidental benefit and not a core feature. >>> >>> https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html >> >> I designed it very intentionally to be compatible with zip import. >> Otherwise wheels would probably not even be zip files (like pypm >> packages which are tar) and would achieve greater compression. >> >> It may be useful to understand that wheel has *political features* or >> if you prefer *setting the defaults based on what we have learned from >> eggs*. I don't recommend that they be zip-imported generally but if >> you are a consenting adult who understands the caveats you may do so. > > Then why did you tell someone on this very list a few hours ago that Wheels > were not really designed to be zip imported. It's shorter. From donald at stufft.io Wed Jan 29 14:48:02 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 08:48:02 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: <1542A1F9-75AF-4121-806A-074E63AADAC5@stufft.io> On Jan 29, 2014, at 8:46 AM, Nick Coghlan wrote: > On 29 January 2014 23:34, Donald Stufft wrote: >> >> If I'm wrong, then by all means show me where it was discussed so I can admit >> I was wrong. > > You have the burden of proof backwards there. You're the one asking me > to break backwards compatibility, and to let people continue to > believe wheels are only a partial replacement for eggs - it's up to > you to make the case that waiting until wheel 1.1 (which, again, has > nobody committed to writing it and a completely unspecified timeline) > is a superior approach to clarifying something I thought was already > documented when I accepted the PEP (and is certainly inherent in the > design of the format). > > The zipimport compatibility didn't need to be discussed much because > it was there in Daniel's original wheel design - there was never any > proposal to use a zipimport *incompatible* approach, so nobody had to > campaign in favour of zipimport compatibility. > > Cheers, > Nick. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia So what did you mean when you said ?We discussed it extensively before PEP 427 was approved? if you?re now saying that it wasn?t discussed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Wed Jan 29 14:43:25 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 Jan 2014 08:43:25 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: > >> On 29 January 2014 12:58, Nick Coghlan wrote: >>> You can campaign to deprecate that feature *if* we ever want to make a >>> change to the wheel format that is incompatible with it. >> >> Note that virtualenv uses the ability to run wheels from sys.path. I >> don't believe that virtualenv should be a "special exception" here, >> nor do I want it to rely on accidents of the implementation. >> >> So from my POV, pypa is already shipping code that relies on this >> behaviour being by design. >> >> Paul > > > Here's Paul explicitly mentioning that Wheels being used with zip import is > an incidental benefit and not a core feature. > > https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression. It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. From ncoghlan at gmail.com Wed Jan 29 14:46:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 23:46:07 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On 29 January 2014 23:34, Donald Stufft wrote: > > If I'm wrong, then by all means show me where it was discussed so I can admit > I was wrong. You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to believe wheels are only a partial replacement for eggs - it's up to you to make the case that waiting until wheel 1.1 (which, again, has nobody committed to writing it and a completely unspecified timeline) is a superior approach to clarifying something I thought was already documented when I accepted the PEP (and is certainly inherent in the design of the format). The zipimport compatibility didn't need to be discussed much because it was there in Daniel's original wheel design - there was never any proposal to use a zipimport *incompatible* approach, so nobody had to campaign in favour of zipimport compatibility. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jan 29 14:59:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jan 2014 23:59:55 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1542A1F9-75AF-4121-806A-074E63AADAC5@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> <1542A1F9-75AF-4121-806A-074E63AADAC5@stufft.io> Message-ID: On 29 January 2014 23:48, Donald Stufft wrote: > > So what did you mean when you said "We discussed it extensively before > PEP 427 was approved" if you're now saying that it wasn't discussed. "Explicitly" would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Jan 29 14:59:33 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 13:59:33 +0000 (UTC) Subject: [Distutils] Using Wheel with zipimport References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > I believe Paul's concern is with anything that suggests that arbitrary > *third party* code can be run from wheel files, when the reality is > that it is fairly easy to accidentally write code that assumes it is > installed on the filesystem in a way that isn't easy for a quick scan > of the files in the zip archive to detect (especially since the PEP > 376 installation database PEP doesn't include any support for > arbitrary metapath importers). That is a valid concern, but no one is suggesting that arbitrary third party code can run from wheel files, just as zipimport makes no guarantees about zipped code working. > By contrast PEP 441 is a *distribution* utility - the creator of the > application is expected to ensure that doing so actually works > correctly before publishing their app that way, just as we would > expect py2exe, py2app and cx-freeze users to do. True, but there's no reason why wheels couldn't have some metadata indicating that this diligence has been exercised by the wheel creator. > With the "reference implementation" position that distlib is likely to > occupy in a post-PEP-426/440/459 world, though, there's an additional > legitimate concern about allowing end users to easily distinguish > between "this API is fully supported by the PyPA as part of the > reference implementation for metadata 2.0" and "this is an > experimental packaging related API that may or may not be useful in > general, and some members of the PyPA may still have grave > reservations about it". > > At the moment, distlib contains both kinds of API, and it confuses > *us*, let alone anyone else that isn't closely following along on > distutils-sig. As long as distlib is serving the dual role of > providing both "the reference implementation for metadata 2.0" and > "some experimental packaging related APIs", we're going to get > concerns like this one arising. If there was a clear way to > distinguish them (ideally with a separate project for either the > reference implementation or the experimental stuff, but even a > distinct namespace within the distlib project would help a great > deal), I suspect there would be less concern. These are social concerns perhaps more than technical concerns, and to me they lack specificity. Of course some of the APIs in distlib are new and untried-except-by-me, but the way to allay concerns is to focus on specifics, force out the details of the concerns and then see how best they can be addressed. This is not doable with "zipped-eggs-were-bad" rhetoric. Details generally help to identify what the real problem is. For example, Donald raised the spectre of security vulnerabilities with his mention of Mitre and CVEs, but there were no specifics beyond that. I found a discussion where someone had set PYTHON_EGG_CACHE to /tmp. I can certainly see the negative security implications of that, but the finger was pointed at the using applications rather than setuptools. Even though setuptools specifically added code as a remedy to warn when the env var pointed to a world-writeable directory, this was seen as trying to be helpful rather than patching a vulnerability. Of course, if I've misunderstood something in that discussion or missed some other security issue, then some pointers would help move the discussion along. > In the specific case of distlib.mount, if it's eventually combined > with a metadata extension like "distlib.mount" which packages must > export in order for the command to allow them to be automatically used > that way, then I don't see anything wrong with it *in general* - it's > a natural extension of the setuptools "zip_safe" flag, but with the > ability to include additional details (like whether or not there are C > extensions that need to be automatically extracted). Are you talking just about adding wheels to sys.path, or do you mean the extension-extraction stuff? Note that distlib's Wheel.mount does a compatibility check and addition to sys.path, which I feel is not especially controversial and better than just adding to sys.path, which user code can now do, anyway. But nothing else happens, unless specific metadata is provided in the wheel to enable it. While it's not specifically a "distlib.mount" export, there is a facility to ask for extensions to be extracted, and the in absence of metadata asking for this, no extraction occurs. > goes further than the current EXTENSIONS approach - this proposal > would be akin to *requiring* an empty EXTENSIONS file, and/or the > setuptools zip_safe flag in order to allow mounting of even the pure > Python wheel. Such a conservative approach is also the antithesis of > the setuptools "attempt to guess": if the package publisher doesn't > explicitly opt in to zip support, then distlib.mount would assume that > it is *not* supported (but may provide an API for the caller to > override that, like "assume_zip_safe=True" or "force=True"). I have no problem with adding wheel metadata to allow/disallow even adding to sys.path - it's effectively just like another step in the compatibility check. It would make most sense to place this in the WHEEL metadata, rather than pydist.json or similar, since it relates to the contents of a particular wheel rather than the distribution in general. > However, like Paul, I have some concerns about a still experimental > API like that being in the metadata 2.0 reference implementation, > since that will likely end up having to deal with stdlib-like levels > of backwards compatibility requirements, and removing experimental > APIs that we later decided we weren't happy with could prove > problematic. But we're talking about the Python 3.5 time-frame here, and 3.4 isn't even out yet. ISTM there is plenty of time to get these sorts of issues ironed out. While I tend to favour backward compatibility wherever possible, distlib is nowhere near 1.0, and so distlib users (a small number, from what I can see) could expect some API breakage if there's no sensible alternative. Regards, Vinay Sajip From dholth at gmail.com Wed Jan 29 15:01:52 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 Jan 2014 09:01:52 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: Message-ID: On Tue, Jan 28, 2014 at 10:41 PM, Evgeny Sazhin wrote: > > On Jan 28, 2014, at 9:54 PM, Daniel Holth wrote: > >> On Tue, Jan 28, 2014 at 8:04 PM, Evgeny Sazhin wrote: >>> Hi, >>> >>> I saw that people from this list are responsible for Wheel related PEP. >>> I'm comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. >>> >>> I'm trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... >>> >>> So, i would like to ask few questions about it : >>> >>> 1. I'm coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn't it be way easier to deal with the dependencies and imports? >>> 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can't be used the same exact way? >>> >>> >>> I would appreciate any insight about the development infrastructure the wheel designers have in mind. >>> >>> Thanks in advance, >>> Eugene >> > > >> The main reason the packaging format is not explicitly importable is >> simply because Python hasn't had "jar-like" deployment for as long or >> as consistently as Java. > > Thanks a lot for taking time to respond! > > Does it mean that it actually makes sense to look into that direction and make wheel usage closer to jar? > If wheel is promoted as the ultimate solution to the packaging problems (at least that's my impression ) aren't the abilities to import packages and classes from it as well as make it runnable a kind of top priority and something that is right on the surface? > >> So while Java code universally uses a "get >> resource" API to get stuff on the classpath, a a lot of Python code >> will try to open a file. C extensions can't be loaded from inside zip >> files. > > I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? > I'd think wheel should do everything in its power to provide pain free workflow for pure python, and if there is a need to add some C extension then solve that problem separately? Is the possibility to add C extension into the wheel so critical, that it should prohibit useful feature for pure python modules? > >> And it's a lot harder to edit .py files once they are zipped >> up, unlike Java where only the compiled and non-editable .class files >> are zipped. > > Why would i want to edit the zipped .py file from a wheel? I want to import a package or a class from it, reuse a function. Nothing more. > > >> >> ZIP import is a great feature but wheels aren't really designed for >> it. It's more reliable that way. > > More reliable in what regard? > > My impression is that the whole thing about pip, virtualenv and wheels is great - it takes into consideration a lot of end user problems, aka getting proper module and installing it to RUN applications. But for whatever reason it misses some important problems on the DEVELOPMENT side. > > I'm trying to find a sane way for developer to work, so if wheel is not an answer - what is? Deprecated Egg? Or may be then egg is not going to go anywhere and pip has to learn to deal with them as well as it does with wheels? It's true that we have not put a lot of effort into trying to support jar-style development in this system. If you want that then you will have to do additional work. However a lot of people do find developing on the filesystem in virtualenvs to be quite sane. All the source code for the whole application remains easily editable for debugging. (By the way, these days the debugger does know how to load .py source from within zip files.) You should also give buildout a try. It installs each dependency into its own directory as an unzipped egg and does not use virtualenv. Egg will be around for a while longer, it's just not written down in a PEP. Wheel solves a coupling problem between the build system and the installer, allowing those two steps to be performed in separate processes or even on separate machines. It does not solve all problems. From ncoghlan at gmail.com Wed Jan 29 15:06:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 00:06:40 +1000 Subject: [Distutils] PEP 427 In-Reply-To: References: Message-ID: On 29 January 2014 13:41, Evgeny Sazhin wrote: > > I have no knowledge about c extensions scope, but i feel like it might be of less importance then pure python packaging issues? Am I wrong? Most interesting Python projects will end up with C dependencies through things like NumPy, SQL Alchemy, GUI toolkits, image libraries, etc. You can do a lot with pure Python and the standard library, but interfacing with C is the norm rather than the exception the way it is in Java. > I'd think wheel should do everything in its power to provide pain free workflow for pure python, and if there is a need to add some C extension then solve that problem separately? Is the possibility to add C extension into the wheel so critical, that it should prohibit useful feature for pure python modules? The Python culture and the Java culture is very different in that regard - the JVM allows Java developers to almost completely ignore the C runtime on a system. CPython, by contrast, integrates with that C runtime directly, and this is then reflected in the ecosystem that has built up around it. So while there is some support for running without unpacking things to disk, we generally prefer approaches that work seamlessly with C extensions as well. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Jan 29 15:11:11 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 14:11:11 +0000 (GMT) Subject: [Distutils] PEP 427 In-Reply-To: Message-ID: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> > Does it mean that it actually makes sense to look into that > direction and make wheel usage closer to jar? There is a parallel discussion going on, with the title "Using Wheel with zipimport", which is relevant to this question, and other questions you raised (e.g. about supporting C extensions/pure-Python modules. > I have no knowledge about c extensions scope, but i feel > like it might be of less importance then pure python > packaging issues? Am I wrong? A lot of Python users depend on C extensions - and while it is a subset of all Python users, it is a large (and important) subset. Example: any usage of Python in numerical analysis or scientific applications involves use of C extensions. Regards, Vinay Sajip From donald at stufft.io Wed Jan 29 15:12:02 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 09:12:02 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> <1542A1F9-75AF-4121-806A-074E63AADAC5@stufft.io> Message-ID: <0A7ED519-6C0B-482A-BE5B-7D76B8EB1C05@stufft.io> On Jan 29, 2014, at 8:59 AM, Nick Coghlan wrote: > On 29 January 2014 23:48, Donald Stufft wrote: >> >> So what did you mean when you said "We discussed it extensively before >> PEP 427 was approved" if you're now saying that it wasn't discussed. > > "Explicitly" would be a better word: > https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html > > Like I said, that particular aspect wasn't controversial, so while it > was noted a few times (a few other examples of which you found), it > was the overall discussions that were extensive. Both Daniel and I > knew the zipimport compatibility for packages that were themselves zip > compatible was a deliberate feature, so it was a surprise to me when > Armin Ronacher said in his recent article that it wasn't supported > (and hence the clarification). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia That also mentions the fact that he was considering allowing other compression algorithms which would make it incompatible with zip import. I don?t agree that this points to any promise that Wheels are themselves importable as I can guarantee I poured over that spec over and over making sure we weren?t adding something from Egg that had been problematic in the past. I have no plans to support adding Wheels to sys.path in a way that is supported for end users to do that, in any tool or project I work on. But whatever, I feel like you?ve placed yourself above the PEP process and are adding features by fiat instead of through the process. Obviously you feel different and this discussion has gone toxic. So carry on. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Jan 29 15:13:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 00:13:04 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: On 29 January 2014 23:59, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> goes further than the current EXTENSIONS approach - this proposal >> would be akin to *requiring* an empty EXTENSIONS file, and/or the >> setuptools zip_safe flag in order to allow mounting of even the pure >> Python wheel. Such a conservative approach is also the antithesis of >> the setuptools "attempt to guess": if the package publisher doesn't >> explicitly opt in to zip support, then distlib.mount would assume that >> it is *not* supported (but may provide an API for the caller to >> override that, like "assume_zip_safe=True" or "force=True"). > > I have no problem with adding wheel metadata to allow/disallow even > adding to sys.path - it's effectively just like another step in the > compatibility check. It would make most sense to place this in the > WHEEL metadata, rather than pydist.json or similar, since it relates to > the contents of a particular wheel rather than the distribution in > general. Oh, an "Import-compatible" flag in the WHEEL metadata? Yeah, I like it. Noted as an issue for wheel 1.1: https://bitbucket.org/pypa/pypi-metadata-formats/issues?component=Wheel >> However, like Paul, I have some concerns about a still experimental >> API like that being in the metadata 2.0 reference implementation, >> since that will likely end up having to deal with stdlib-like levels >> of backwards compatibility requirements, and removing experimental >> APIs that we later decided we weren't happy with could prove >> problematic. > > But we're talking about the Python 3.5 time-frame here, and 3.4 isn't even > out yet. ISTM there is plenty of time to get these sorts of issues ironed > out. While I tend to favour backward compatibility wherever possible, > distlib is nowhere near 1.0, and so distlib users (a small number, from > what I can see) could expect some API breakage if there's no sensible > alternative. Yeah, I think I'm just nervous because sitting down and properly reviewing the distlib API is somewhere in the middle of a gigantic todo list :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Jan 29 15:22:21 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 09:22:21 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> Message-ID: <1383370A-2BB0-498F-B874-E703D3BD63B5@stufft.io> On Jan 29, 2014, at 8:59 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> I believe Paul's concern is with anything that suggests that arbitrary >> *third party* code can be run from wheel files, when the reality is >> that it is fairly easy to accidentally write code that assumes it is >> installed on the filesystem in a way that isn't easy for a quick scan >> of the files in the zip archive to detect (especially since the PEP >> 376 installation database PEP doesn't include any support for >> arbitrary metapath importers). > > That is a valid concern, but no one is suggesting that arbitrary third > party code can run from wheel files, just as zipimport makes no guarantees > about zipped code working. > >> By contrast PEP 441 is a *distribution* utility - the creator of the >> application is expected to ensure that doing so actually works >> correctly before publishing their app that way, just as we would >> expect py2exe, py2app and cx-freeze users to do. > > True, but there's no reason why wheels couldn't have some metadata > indicating that this diligence has been exercised by the wheel creator. > >> With the "reference implementation" position that distlib is likely to >> occupy in a post-PEP-426/440/459 world, though, there's an additional >> legitimate concern about allowing end users to easily distinguish >> between "this API is fully supported by the PyPA as part of the >> reference implementation for metadata 2.0" and "this is an >> experimental packaging related API that may or may not be useful in >> general, and some members of the PyPA may still have grave >> reservations about it". >> >> At the moment, distlib contains both kinds of API, and it confuses >> *us*, let alone anyone else that isn't closely following along on >> distutils-sig. As long as distlib is serving the dual role of >> providing both "the reference implementation for metadata 2.0" and >> "some experimental packaging related APIs", we're going to get >> concerns like this one arising. If there was a clear way to >> distinguish them (ideally with a separate project for either the >> reference implementation or the experimental stuff, but even a >> distinct namespace within the distlib project would help a great >> deal), I suspect there would be less concern. > > These are social concerns perhaps more than technical concerns, and to > me they lack specificity. Of course some of the APIs in distlib are > new and untried-except-by-me, but the way to allay concerns is to focus > on specifics, force out the details of the concerns and then see how best > they can be addressed. This is not doable with "zipped-eggs-were-bad" > rhetoric. Details generally help to identify what the real problem is. For > example, Donald raised the spectre of security vulnerabilities with his > mention of Mitre and CVEs, but there were no specifics beyond that. > I found a discussion where someone had set PYTHON_EGG_CACHE to /tmp. I can > certainly see the negative security implications of that, but the finger > was pointed at the using applications rather than setuptools. Even > though setuptools specifically added code as a remedy to warn when the > env var pointed to a world-writeable directory, this was seen as trying to > be helpful rather than patching a vulnerability. Of course, if I've > misunderstood something in that discussion or missed some other > security issue, then some pointers would help move the discussion along. Mitre?s rules for CVEs are not entirely obvious to people who are not familiar with them. Generally if the feature *can* be used securely or there was no evidence that the author intended that the code be secure they will not issue a CVE. The issue is that the feature makes a very attractive footgun for people using it to do the wrong thing and have it be a very bad idea. > >> In the specific case of distlib.mount, if it's eventually combined >> with a metadata extension like "distlib.mount" which packages must >> export in order for the command to allow them to be automatically used >> that way, then I don't see anything wrong with it *in general* - it's >> a natural extension of the setuptools "zip_safe" flag, but with the >> ability to include additional details (like whether or not there are C >> extensions that need to be automatically extracted). > > Are you talking just about adding wheels to sys.path, or do you mean > the extension-extraction stuff? Note that distlib's Wheel.mount does a > compatibility check and addition to sys.path, which I feel is not > especially controversial and better than just adding to sys.path, > which user code can now do, anyway. But nothing else happens, unless > specific metadata is provided in the wheel to enable it. While it's not > specifically a "distlib.mount" export, there is a facility to ask for > extensions to be extracted, and the in absence of metadata asking for this, > no extraction occurs. > >> goes further than the current EXTENSIONS approach - this proposal >> would be akin to *requiring* an empty EXTENSIONS file, and/or the >> setuptools zip_safe flag in order to allow mounting of even the pure >> Python wheel. Such a conservative approach is also the antithesis of >> the setuptools "attempt to guess": if the package publisher doesn't >> explicitly opt in to zip support, then distlib.mount would assume that >> it is *not* supported (but may provide an API for the caller to >> override that, like "assume_zip_safe=True" or "force=True"). > > I have no problem with adding wheel metadata to allow/disallow even > adding to sys.path - it's effectively just like another step in the > compatibility check. It would make most sense to place this in the > WHEEL metadata, rather than pydist.json or similar, since it relates to > the contents of a particular wheel rather than the distribution in > general. > >> However, like Paul, I have some concerns about a still experimental >> API like that being in the metadata 2.0 reference implementation, >> since that will likely end up having to deal with stdlib-like levels >> of backwards compatibility requirements, and removing experimental >> APIs that we later decided we weren't happy with could prove >> problematic. > > But we're talking about the Python 3.5 time-frame here, and 3.4 isn't even > out yet. ISTM there is plenty of time to get these sorts of issues ironed > out. While I tend to favour backward compatibility wherever possible, > distlib is nowhere near 1.0, and so distlib users (a small number, from > what I can see) could expect some API breakage if there's no sensible > alternative. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Wed Jan 29 15:25:36 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 14:25:36 +0000 (GMT) Subject: [Distutils] Using Wheel with zipimport In-Reply-To: Message-ID: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> > It may be useful to understand that wheel has *political features* or > if you prefer *setting the defaults based on what we have learned from > eggs*. I don't recommend that they be zip-imported generally > but if you are a consenting adult who understands the caveats you > may do so. What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable. But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way. What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about "pain" and "weirdness". I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound na?ve to some - but there it is. Regards, Vinay Sajip From donald at stufft.io Wed Jan 29 15:40:07 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 09:40:07 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> References: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> Message-ID: On Jan 29, 2014, at 9:25 AM, Vinay Sajip wrote: >> It may be useful to understand that wheel has *political features* or >> if you prefer *setting the defaults based on what we have learned from >> eggs*. I don't recommend that they be zip-imported generally >> but if you are a consenting adult who understands the caveats you >> may do so. > > What *exactly* have we learned from eggs? We've learned that sys.path > manipulation under the hood is bad, because sys.path is important to system > behaviour and because the ramifications of changing it under the hood are unpredictable. > > But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital > in many situations to be able to manipulate sys.path in a controlled way. > > What *else* have we learned from eggs? If we leave to one side the sys.path manipulation > issues, what else has been problematic about the egg format? Details seem hard to find, > though there's lots of talk in general about "pain" and "weirdness". I'm asking because I > really want to understand at a more detailed level what those problems are, because I > would be interested in trying to solve them, or minimise their impact. That might sound > na?ve to some - but there it is. > > Regards, > > Vinay Sajip > It?s hard to pin down because the failure modes of zipped eggs are nebulous themselves. For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it?s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it?s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory. That?s really the biggest problem with strictly pure python Zipped Eggs. That the failure modes are ill defined and depend greatly on the application/library itself. For some you?ll get exceptions, for others some features will just stop working, for even others you?ll just get subtle bugs that only happen under zip import. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Wed Jan 29 15:42:00 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 14:42:00 +0000 (GMT) Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1383370A-2BB0-498F-B874-E703D3BD63B5@stufft.io> Message-ID: <1391006520.48107.YahooMailBasic@web172403.mail.ir2.yahoo.com> -------------------------------------------- On Wed, 29/1/14, Donald Stufft wrote: > Mitre?s rules for CVEs are not entirely obvious to people who are not > familiar with them. Generally if the feature *can* be used securely or > there was no evidence that the author intended that the code be secure > they will not issue a CVE. The issue is that the feature makes a very > attractive footgun for people using it to do the wrong thing and have it > be a very bad idea. So, was a CVE issued against setuptools? My understanding is that it wasn't - have I misunderstood? tool = setuptools footgun = configurability of egg cache using PYTHON_EGG_CACHE trigger = setuptools user sets PYTHON_EGG_CACHE to a world writeable directory shot = malicious user replaces eggs in the cache with malicious code BTW when no HOME directory is available, distlib uses tempfile.mkdtemp() which IIUC provides a directory with permissions of 0700, which should be safe from tampering. Do you see a security problem with this? Regards, Vinay Sajip From brett at python.org Wed Jan 29 15:46:39 2014 From: brett at python.org (Brett Cannon) Date: Wed, 29 Jan 2014 09:46:39 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> Message-ID: On Wed, Jan 29, 2014 at 9:40 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 9:25 AM, Vinay Sajip wrote: > > >> It may be useful to understand that wheel has *political features* or > >> if you prefer *setting the defaults based on what we have learned from > >> eggs*. I don't recommend that they be zip-imported generally > >> but if you are a consenting adult who understands the caveats you > >> may do so. > > > > What *exactly* have we learned from eggs? We've learned that sys.path > > manipulation under the hood is bad, because sys.path is important to > system > > behaviour and because the ramifications of changing it under the hood > are unpredictable. > > > > But we don't let that lead to a mindset that prohibits *any* sys.path > manipulation - it's vital > > in many situations to be able to manipulate sys.path in a controlled way. > > > > What *else* have we learned from eggs? If we leave to one side the > sys.path manipulation > > issues, what else has been problematic about the egg format? Details > seem hard to find, > > though there's lots of talk in general about "pain" and "weirdness". I'm > asking because I > > really want to understand at a more detailed level what those problems > are, because I > > would be interested in trying to solve them, or minimise their impact. > That might sound > > na?ve to some - but there it is. > > > > Regards, > > > > Vinay Sajip > > > > It?s hard to pin down because the failure modes of zipped eggs are > nebulous themselves. > > For instance take pip. I just recently redid the get-pip.py installer to > use a zip file (not a > Wheel or Egg) that contained pip and add that zip file straight to > sys.path instead of > unzipping it. One of the failure modes was that it was suddenly unable to > validate TLS > certificates. The reason why, was because it bundles it?s own cacerts.pem > which it > passes to the ssl library to validate the connection. In this case it?s > impossible as far as > I can tell to use pkgutil.get_data directly because the ssl library does > not directly > support pkgutil.get_data nor does it support passing the certificates as a > string or other > in memory. > > That?s really the biggest problem with strictly pure python Zipped Eggs. > That the failure > modes are ill defined and depend greatly on the application/library > itself. For some you?ll > get exceptions, for others some features will just stop working, for even > others you?ll > just get subtle bugs that only happen under zip import. Just a heads-up that if I ever get around to re-implementing zipfile importing using importlib (http://bugs.python.org/issue17630) I will ask this list for feature feedback to make sure I don't miss something that's critical. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jan 29 15:55:56 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 Jan 2014 09:55:56 -0500 Subject: [Distutils] problems with eggs? Message-ID: Here are some of the problems with eggs that I tried to solve: - Cannot be unzipped on top of each other due to the EGG-INFO directory. Wheels repeat the package name in the dist-info directory and are more like their installed layout, hopefully taking all the mystery out of the format. - The egg naming scheme doesn't let you distinguish between Python interpreter implementations (not an important issue at the time). - They include .pyc and so are always [at least somewhat] Python version specific. Wheels don't. - Eggs do not have categories of files. Wheel uses subdirectories of the .data/ directory, and are therefore a complete replacement for most other kinds of bdist_x - And of course wheels use .dist-info which is also something that is newer than eggs, and generally try to bring the useful packaging PEP work into reality. I remember dealing with the zipped eggs hassle, and I remember having to wait too long for binary packages to be uploaded to pypi as eggs when a new Python version was released. From ncoghlan at gmail.com Wed Jan 29 15:58:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 00:58:15 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> References: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> Message-ID: On 30 Jan 2014 00:28, "Vinay Sajip" wrote: > > > It may be useful to understand that wheel has *political features* or > > if you prefer *setting the defaults based on what we have learned from > > eggs*. I don't recommend that they be zip-imported generally > > but if you are a consenting adult who understands the caveats you > > may do so. > > What *exactly* have we learned from eggs? We've learned that sys.path > manipulation under the hood is bad, because sys.path is important to system > behaviour and because the ramifications of changing it under the hood are unpredictable. > > But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital > in many situations to be able to manipulate sys.path in a controlled way. > > What *else* have we learned from eggs? If we leave to one side the sys.path manipulation > issues, what else has been problematic about the egg format? Details seem hard to find, > though there's lots of talk in general about "pain" and "weirdness". I'm asking because I > really want to understand at a more detailed level what those problems are, because I > would be interested in trying to solve them, or minimise their impact. That might sound > na?ve to some - but there it is. I went through this with Chris McDonough back when packaging was dropped from 3.3, and he really helped me focus on what I found to be the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default That was due to easy_install defaults being chosen for the Chandler use case rather than installing into a shared environment, but the egg format (and zip imports in general) still ended up being tainted by association. More recently, I've become concerned about the way setuptools/easy_install assumes that an egg is zip import compatible in the absence of clear markers that it isn't. pip's wheel support avoids those issues, because the normal workflows just install things normally. It takes special effort to go to the trouble of keeping a wheel file *as* a wheel and then putting it on sys.path later. distlib.mount already has the virtue of always being opt-in from the user point of view, and the C extension support is opt-in from the publisher point of view, but it would be improved by needing the publisher to opt in to declaring support for zipimport (or meta imports in general), rather than letting end users deal with debugging obscure errors from code that assumes it has been fully installed. Cheers, Nick. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Jan 29 16:00:05 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 15:00:05 +0000 (GMT) Subject: [Distutils] Using Wheel with zipimport In-Reply-To: Message-ID: <1391007605.47275.YahooMailBasic@web172405.mail.ir2.yahoo.com> -------------------------------------------- On Wed, 29/1/14, Donald Stufft wrote: > It?s hard to pin down because the failure modes of zipped > eggs are nebulous themselves. > For instance take pip. I just recently redid the get-pip.py > installer to use a zip file (not a Wheel or Egg) that contained pip > and add that zip file straight to sys.path instead of unzipping it. > One of the failure modes was that it was suddenly unable to validate > TLS certificates. The reason why, was because it bundles it?s > own cacerts.pem which it passes to the ssl library to validate the > connection. In this case it?s impossible as far as I can tell to use > pkgutil.get_data directly because the ssl library does not directly > support pkgutil.get_data nor does it support passing the certificates > as a string or other in memory. > That?s really the biggest problem with strictly pure python Zipped Eggs. If that's all, I'm not overly worried as long as there's a mountability flag on wheels. I looked at running pip from a zip a while ago, the cacerts thing was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to come across as a show-off.) Here's the code I added as a fix in pip/locations.py (no doubt it could be improved): def _check_for_zip(): import zipimport global default_cert_path try: if isinstance(__loader__, zipimport.zipimporter): # running from a zip file. Save data to a handy location. data = __loader__.get_data(default_cert_path) destdir = os.path.expanduser('~/.pip') if not os.path.isdir(destdir): os.mkdir(destdir) default_cert_path = os.path.join(destdir, 'cacert.pem') with open(default_cert_path, 'wb') as f: f.write(data) except NameError: # __loader__ not defined, so not in zip pass _check_for_zip() del _check_for_zip Of course, pip wasn't originally designed to run from zip, so one would expect to find these kinds of issues. OTOH, if importable zips were more commonplace, then presumably people would think about these kinds of issues more when writing their code. Of course, in the above case, using pkgutil wouldn't have worked, and there are bound to be other similar cases, but I still don't see any real reason to fear the consequences of importable wheels. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Jan 29 16:07:08 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 15:07:08 +0000 (GMT) Subject: [Distutils] problems with eggs? In-Reply-To: Message-ID: <1391008028.38003.YahooMailBasic@web172402.mail.ir2.yahoo.com> -------------------------------------------- On Wed, 29/1/14, Daniel Holth wrote: > Here are some of the problems with eggs that I tried to solve: [snipped] Right, and the wheel design has addressed those issues. I suppose what I was after was the kinds of problems that would arise from the importability of wheels, which seems to be causing so much fear, uncertainty and doubt :-) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Jan 29 16:15:50 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 15:15:50 +0000 (GMT) Subject: [Distutils] Using Wheel with zipimport In-Reply-To: Message-ID: <1391008550.69442.YahooMailBasic@web172406.mail.ir2.yahoo.com> > I went through this with Chris McDonough back when packaging was > dropped from 3.3, and he really helped me focus on what I found to be > the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default > That was due to easy_install defaults being > chosen for the Chandler use case rather than installing into > a shared environment, but the egg format (and zip imports in > general) still ended up being tainted by association. But now that we have identified that it was "by association", we can hopefully make more informed decisions and not throw the baby out with the bath-water. > pip's wheel support avoids those issues, because the normal workflows > just install things normally. It takes special effort to go to the trouble of > keeping a wheel file *as* a wheel and then putting it on sys.path later. > distlib.mount already has the virtue of always being opt-in from the user > point of view, and the C extension support is opt-in from the publisher > point of view, but it would be improved by needing the publisher to > opt in to declaring support for zipimport (or meta imports in general), > rather than letting end users deal with debugging obscure errors from > code that assumes it has been fully installed. That's a measured response to a specific concern: it makes sense to codify the publisher's intent in the wheel (with a sensible default) and to have distlib's Wheel.mount() take into account the publisher's intent when considering whether to allow mounting of a wheel to take place. Regards, Vinay Sajip From p.f.moore at gmail.com Wed Jan 29 16:37:27 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 15:37:27 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> Message-ID: On 29 January 2014 13:31, Donald Stufft wrote: > Here's Paul explicitly mentioning that Wheels being used with zip import is > an incidental benefit and not a core feature. > > https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html That's quoted out of context. It was in a thread about enabling importing C extensions from zipfiles, which I'm against. I don't always write emails as if they were documents to be saved for posterity. Sometimes I even say things that are contradictory, or not right. Heck, I'm not the authority on wheels, so what I'm saying is at best opinion. Sorry if I confused you or gave you the wrong impression, but honestly I don't have the time to review and vet my emails any more than I currently do :-( Paul From donald at stufft.io Wed Jan 29 16:40:48 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 10:40:48 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1391007605.47275.YahooMailBasic@web172405.mail.ir2.yahoo.com> References: <1391007605.47275.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: <2F79F486-52AA-4326-84CF-318629C4E36C@stufft.io> On Jan 29, 2014, at 10:00 AM, Vinay Sajip wrote: > > -------------------------------------------- > On Wed, 29/1/14, Donald Stufft wrote: > >> It?s hard to pin down because the failure modes of zipped >> eggs are nebulous themselves. > >> For instance take pip. I just recently redid the get-pip.py >> installer to use a zip file (not a Wheel or Egg) that contained pip >> and add that zip file straight to sys.path instead of unzipping it. >> One of the failure modes was that it was suddenly unable to validate >> TLS certificates. The reason why, was because it bundles it?s >> own cacerts.pem which it passes to the ssl library to validate the >> connection. In this case it?s impossible as far as I can tell to use >> pkgutil.get_data directly because the ssl library does not directly >> support pkgutil.get_data nor does it support passing the certificates >> as a string or other in memory. > >> That?s really the biggest problem with strictly pure python Zipped Eggs. > > If that's all, I'm not overly worried as long as there's a mountability flag on > wheels. I looked at running pip from a zip a while ago, the cacerts thing > was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to > come across as a show-off.) Here's the code I added as a fix in > pip/locations.py (no doubt it could be improved): > > def _check_for_zip(): > import zipimport > global default_cert_path > try: > if isinstance(__loader__, zipimport.zipimporter): > # running from a zip file. Save data to a handy location. > data = __loader__.get_data(default_cert_path) > destdir = os.path.expanduser('~/.pip') > if not os.path.isdir(destdir): > os.mkdir(destdir) > default_cert_path = os.path.join(destdir, 'cacert.pem') > with open(default_cert_path, 'wb') as f: > f.write(data) > except NameError: > # __loader__ not defined, so not in zip > pass > > _check_for_zip() > del _check_for_zip > > Of course, pip wasn't originally designed to run from zip, so one would > expect to find these kinds of issues. OTOH, if importable zips were more > commonplace, then presumably people would think about these kinds of > issues more when writing their code. Of course, in the above case, using > pkgutil wouldn't have worked, and there are bound to be other similar cases, > but I still don't see any real reason to fear the consequences of importable > wheels. > > Regards, > > Vinay Sajip FWIW I'm not particularly that upset about this feature itself. I think it's a bad idea and I think the way it locks the format into a particular method is a poor trade off and that there is possibly a better form for this feature, but if I had engaged the process argued my side, and then lost I could deal with that. What I'm upset about is that I can find no support that the *PEP* text (not the intentions of the PEP) contains explicit support for this feature (as in, it exists because we want it to work and we think people should use it) and little to no evidence that the fact that this was (apparently) a design goal was properly communicated (which includes multiple statements by various people including Daniel that it wasn't really designed for this, though Daniel is apparently recanting that meaning). According to PEP1 a PEP must "be a clear and complete description of the proposed enhancement". I believe that what was proposed for discussion, if it was indeed intended that Wheels should be generally importable by adding them to sys path, was not clear about the fact that this was part of the proposal (and in fact contains text that reads contrary to that intent), and because the proposal and the resulting discussion was unclear, that people, such as myself, were not given the chance to discuss this particular aspect of the proposal. I feel that as BDFL delegate Nick is abusing the fact that he ultimately accepted the PEP to circumvent the lack of a clear proposal of a particular design goal (albeit likely unwittingly) to include this part of the proposal without discussion. *This* is what I have a problem with. The PEP process exists to gather everyone's input on a proposal and that ability was not afforded to people who do not believe that this particular feature should be supported. This is why I started this thread (which has someone gone off the rails trying to discuss whether or not this is a good feature or not which is immaterial to the actual issue), because I believe that this change should be reverted, and if this is something that Nick, Daniel, Vinay, or anyone else feels is important then it should be proposed as part of Wheel 1.1 and everyone should be given a fair chance to discuss it before it is accepted. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Wed Jan 29 16:58:51 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Jan 2014 15:58:51 +0000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> References: <1391005536.26976.YahooMailBasic@web172404.mail.ir2.yahoo.com> Message-ID: On 29 January 2014 14:25, Vinay Sajip wrote: > What *else* have we learned from eggs? That designing modules to support zipimport correctly is non-trivial. And that assuming that things work unless told otherwise is a bad default behaviour. That packaging solutions should not depend on runtime support modules. How you package your code should not imply runtime dependencies. That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility. Paul From vinay_sajip at yahoo.co.uk Wed Jan 29 17:22:09 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jan 2014 16:22:09 +0000 (GMT) Subject: [Distutils] Using Wheel with zipimport In-Reply-To: Message-ID: <1391012529.12131.YahooMailBasic@web172405.mail.ir2.yahoo.com> -------------------------------------------- On Wed, 29/1/14, Paul Moore wrote: > That designing modules to support zipimport correctly is non-trivial. It's not trivial, but it's not especially hard, either. Mostly, it's about remembering to consider zipimport, since that hasn't been a mainstream way of deploying Python software. > And that assuming that things work unless told otherwise is a bad > default behaviour. Assumptions in general are bad, if they are made without due consideration. That's not especially egg-related, is it? > That packaging solutions should not depend on runtime support modules. > How you package your code should not imply runtime dependencies. Do you mean pkg_resources here? Certainly, distlib/distil don't expect to be present in stuff that they install. > That package installation utilities should not dabble in sys.path manipulation. > The import path is the user's responsibility. User as in developer (rather than end user). Right, and distlib's wheel code does no sys.path manipulation unless explicitly asked to. Regards, Vinay Sajip From matthew at opentechschool.org Wed Jan 29 16:29:59 2014 From: matthew at opentechschool.org (Matthew Iversen) Date: Thu, 30 Jan 2014 02:29:59 +1100 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <43378985-D7C5-42AB-AAE3-6FC9CC48DE97@stufft.io> <1542A1F9-75AF-4121-806A-074E63AADAC5@stufft.io> Message-ID: <52E91E77.9050706@notevencode.com> On 30/01/14 00:59, Nick Coghlan wrote: > On 29 January 2014 23:48, Donald Stufft wrote: >> So what did you mean when you said "We discussed it extensively before >> PEP 427 was approved" if you're now saying that it wasn't discussed. > "Explicitly" would be a better word: > https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html > > Like I said, that particular aspect wasn't controversial, so while it > was noted a few times (a few other examples of which you found), it > was the overall discussions that were extensive. Both Daniel and I > knew the zipimport compatibility for packages that were themselves zip > compatible was a deliberate feature, so it was a surprise to me when > Armin Ronacher said in his recent article that it wasn't supported > (and hence the clarification). > > Cheers, > Nick. > In My Humble Opinion, that just isn't good enough. I am sure that everyone on this list has a passionate interest in making sure that the decisions for "python packaging" are as right as possible, and we know there are processes to follow that help us achieve that. One of those is that one should always be explicit about what features and functionality anything supports, which means that all features and functionality can be discussed, agreed upon, improved, etc, before it becomes something that everyone wilfully agrees to support. Wheels are either *documented* to be a binary format that *officially supports* zipimport-ability (fully, conditionally, or whatever), or the default answer is, like any other functionality not explicitly mentioned, no they are not. That's how you get a standard where issues can be discussed without tension. Two people agreeing to themselves that this is an uncontroversial implicit feature is not part of the documentation process, and whether or not they are the creators and acceptors of such standards, it leaves others out of the process until this late stage of the game. The unfortunate disagreements are exactly what can happen when things aren't made explicit from the start and are what are preventable when they are. "Docs or it doesn't exist" is popular developer mantra for a reason. Even as mostly an kibitzer on this group I hope it can learn from and avoid this sort of discussion into the future. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From qwcode at gmail.com Wed Jan 29 18:27:31 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 29 Jan 2014 09:27:31 -0800 Subject: [Distutils] problems with eggs? In-Reply-To: <1391008028.38003.YahooMailBasic@web172402.mail.ir2.yahoo.com> References: <1391008028.38003.YahooMailBasic@web172402.mail.ir2.yahoo.com> Message-ID: btw, there's a stub in the PUG for "Wheel vs Egg", if anyone wants to help fill that out. https://python-packaging-user-guide.readthedocs.org/en/latest/technical.html#wheel-vs-egg On Wed, Jan 29, 2014 at 7:07 AM, Vinay Sajip wrote: > > -------------------------------------------- > On Wed, 29/1/14, Daniel Holth wrote: > > > Here are some of the problems with eggs that I tried to solve: > [snipped] > Right, and the wheel design has addressed those issues. I suppose > what I was after was the kinds of problems that would arise from the > importability of wheels, which seems to be causing so much fear, > uncertainty and doubt :-) > > Regards, > > Vinay Sajip > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Jan 29 22:50:21 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Jan 2014 10:50:21 +1300 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> Message-ID: <52E9779D.7020704@canterbury.ac.nz> Nick Coghlan wrote: > Otherwise we'd have to define a whole new format for something that > can be adequately handled by a wheel that meets certain restrictions, > and that would be pointless (we already have too many formats, and we > wanted the wheel format to offer a strict superset of the egg format's > capabilities). Would it help if zipimportable wheels were given a different extension? The format could be exactly the same (except perhaps for a flag in the metadata) but users would be able to immediately tell whether a particular wheel was zipimportable without having to run a tool. We would need a suitably catchy name for them too. Zippy wheels? Fast wheels? Hotwheels? -- Greg From ncoghlan at gmail.com Wed Jan 29 22:54:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 07:54:48 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <52E9779D.7020704@canterbury.ac.nz> References: <4948BE1F-2FC2-4C33-AF35-9D8DEC8D96A5@stufft.io> <9F21C03A-DED2-431C-BADF-6EA41E910851@stufft.io> <52E9779D.7020704@canterbury.ac.nz> Message-ID: On 30 Jan 2014 07:50, "Greg Ewing" wrote: > > Nick Coghlan wrote: >> >> Otherwise we'd have to define a whole new format for something that >> can be adequately handled by a wheel that meets certain restrictions, >> and that would be pointless (we already have too many formats, and we >> wanted the wheel format to offer a strict superset of the egg format's >> capabilities). > > > Would it help if zipimportable wheels were given a > different extension? The format could be exactly the > same (except perhaps for a flag in the metadata) but > users would be able to immediately tell whether a > particular wheel was zipimportable without having to > run a tool. > > We would need a suitably catchy name for them too. > Zippy wheels? Fast wheels? Hotwheels? 1.1 will likely have a flag in WHEEL (off by default), and distlib.mount will then refuse to add wheels that don't have it set (including 1.0 wheels) to sys.path. Cheers, Nick. > > -- > Greg > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Jan 29 23:04:54 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Jan 2014 22:04:54 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Sun, Jan 26, 2014 at 12:29 AM, Nick Coghlan wrote: > Paul's position exactly mirrors my own - I an perfectly fine with the > recommended advice to scientific users continuing to be "NumPy doesn't > officially support pip and virtualenv because of the way it is built and > installed, so you will have to get one of the curated scientific stacks, > bootstrap conda, use the Windows installers or use the version provided by > your Linux distro vendor." > > The metadata 2.0 standards *will not* be accepted until the pip 1.6 or 1.7 > time frame, and it's more likely the latter, since I don't want to distract > anyone from the current security work (I know I said otherwise recently, > but I managed to temporarily forget that the Warehouse transition and > implementing PEP 458 was next on the to do list when I said that). > > So, if the NumPy community choose to wait for general post-install script > support in wheel files, they're likely to be waiting at least until the > release of pip 1.7. > > In the meantime, the failure mode for people attempting to try out the > Scientific Python stack via "pip install numpy" in an existing Python > installation or virtualenv will remain a failure to build with a likely > cryptic error. > > I do see a few possible workarounds, but none of them would change the > metadata 2.0 standards: > > 1. Add explicit NumPy support *directly* to pip. This would be the quick > hack, private API support that Oscar is requesting, since it would be a > special arrangement between the pip devs and the numpy devs, and eventually > replaced by a general purpose post-install mechanism in metadata 2.0. > I am not speaking for the whole numpy team, but as the former maintainer of numpy.distutils, I think this will be more hurtful than helpful. I think the SSE issue is a bit of a side discussion: most people who care about performance already know how to install numpy. What we care about here are people who don't care so much about fast eigenvalue decomposition, but want to use e.g. pandas. Building numpy in a way that supports every architecture is both doable and acceptable IMO. 2. Add support to pip to request the conversion of available wininst > installers (and bdist_dumb?) to wheels for installation with pip. Vinay has > this working from a technical perspective, so it may be something the pip > devs are interested in exploring for pip 1.6. > Building numpy wheels is not hard, we can do that fairly easily (I have already done so several times, the hard parts have nothing to do with wheel or even python, and are related to mingw issues on win 64 bits). > 3. Both of the above options require waiting for pip 1.6 (at the > earliest), which means neither will improve the behaviour in CPython 3.4 > (which will ship pip 1.5.1). The only folks with the power to improve > *that* situation are the NumPy devs, who have the ability to choose between > the "doesn't work for anyone except experienced build engineers" status quo > and "works for a large percentage of users, but will still fail at runtime > for users on hardware without SSE2 support". > > To put the "but what if the user doesn't have SSE2 support?" concern in > context, it should only affect Intel users with CPUs older than a Pentium 4 > (released 2001), and AMD users with a CPU older than an Opteron or Athlon > 64 (both released 2003). All x86/x86_64 CPUs released in the past decade > should be able to handle SSE2 binaries, so our caveat can be "if your > computer is more than a decade old, 'pip install numpy' may not work for > you, but it should do the right thing on newer systems". > > Now, the NumPy devs may feel that persisting with the status quo for > another 6 to 12 months while waiting for still hypothetical additional > changes in pip specifically to accommodate NumPy's current installation > practices is a better alternative than taking option 3. However, from my > perspective, having NumPy readily available to users using the python.orgWindows installers for Python 3.4 would *significantly* lower the barrier > to entry to the Scientific Python stack for new users on relatively modern > systems when compared to the 4 current options (while we accept the Linux > distro problem is on distutils-sig to deal with, that's far from being a > NumPy specific problem). > Just to clarify: you actually can install numpy on windows with python.orginstallers fairly easily by using easy_install already (we upload a bdist_wininst compatible binary which should not use any CPU-specific instructions). It looks like those are missing for 1.8.0, but we can fix this fairly easily. David > Cheers, > Nick. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 29 23:24:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 08:24:30 +1000 Subject: [Distutils] wheels on sys.path clarification (reboot) Message-ID: I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. Here is the key quote from PEP 427: """Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time.""" That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. Once you have that "can be installed just by unzipping the archive" feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the "no specialised installer needed" feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Jan 29 23:27:49 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 29 Jan 2014 14:27:49 -0800 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau wrote: > I think the SSE issue is a bit of a side discussion: most people who care > about performance already know how to install numpy. What we care about > here are people who don't care so much about fast eigenvalue decomposition, > but want to use e.g. pandas. Building numpy in a way that supports every > architecture is both doable and acceptable IMO. > Exactly -- I'm pretty sure SSE2 is being suggested because that's the lowest common denominator that we expect to see a lot of -- if their really are a lot of non-SSE-2 machines out there we could leave that off, too. > Building numpy wheels is not hard, we can do that fairly easily (I have > already done so several times, the hard parts have nothing to do with wheel > or even python, and are related to mingw issues on win 64 bits). > David, Where is numpy as with building "out of the box" with the python.org binary for Windows, and the "standard" MS compilers that are used with those builds. That used to be an easy "python setup.py install" away -- has that changed? If so, is this a known bug, or a known we-aren't-supporting-that? i.e. it would be nice if anyone setup to build C extensions could "just build numpy". -Chris Just to clarify: you actually can install numpy on windows with python.orginstallers fairly easily by using easy_install already (we upload a > bdist_wininst compatible binary which should not use any CPU-specific > instructions). It looks like those are missing for 1.8.0, but we can fix > this fairly easily. > presumably just as easy to do a binary wheel then -- I vote for that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jan 29 23:39:55 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 17:39:55 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: On Jan 29, 2014, at 5:24 PM, Nick Coghlan wrote: > I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. > > Here is the key quote from PEP 427: > > """Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time.""" > > That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. > > Once you have that "can be installed just by unzipping the archive" feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. > > Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. > > I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the "no specialised installer needed" feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. > > Regards, > Nick. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I understand what you're saying. My point is there is a difference between supported because we didn't go out of our way to break it, and supported because it's an API contract we intend to honor. The first one is fine, because it's an implementation detail and anyone relying on it cannot expect it to have backwards compatibility concerns. It's the second one that concerned me. By including this in the PEP we've moved the fact that Wheel's are directly importable by adding them to the sys.path from an implementation detail to a documented feature (maybe documented is a better word than supported here) which means we've made some level of promise that this is going to continue working in the future. It's that second part that I have a problem with. It means that any change to Wheels in the future must be weighed against the fact that we can't break compatibility with zip import without incrementing to a new major rev of the Wheel format. It means that we're promising that this feature will work. It's essentially the difference between naming a function "_foo" instead of "foo". Yes you can use a function named "_foo" just fine, but there's no promise that it won't break in the future if you rely on that. Again let me stress that this is separate from the issue of *if* we should officially support zip importing Wheels as a feature. I have reservations about that but I would not be angry if that made it into the spec *after* discussion. What has me upset is that this "API" promise is being added without discussion with the rationale of "We didn't actively attempt to break it so now it's an officially supported feature?. I'm upset that the ability to discuss and home and perhaps exclude a certain API promise was kept from the population at large because of the fact that the format didn't go out it's way to prevent itself from being used that way. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From cournape at gmail.com Wed Jan 29 23:50:30 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Jan 2014 22:50:30 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Wed, Jan 29, 2014 at 10:27 PM, Chris Barker wrote: > On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau wrote: > >> I think the SSE issue is a bit of a side discussion: most people who care >> about performance already know how to install numpy. What we care about >> here are people who don't care so much about fast eigenvalue decomposition, >> but want to use e.g. pandas. Building numpy in a way that supports every >> architecture is both doable and acceptable IMO. >> > > Exactly -- I'm pretty sure SSE2 is being suggested because that's the > lowest common denominator that we expect to see a lot of -- if their really > are a lot of non-SSE-2 machines out there we could leave that off, too. > The failure mode is fairly horrible though, and the gain is not that substantial anyway compared to really optimized installation (MKL, etc... as provided by Continuum or us). > Building numpy wheels is not hard, we can do that fairly easily (I have >> already done so several times, the hard parts have nothing to do with wheel >> or even python, and are related to mingw issues on win 64 bits). >> > > David, > > Where is numpy as with building "out of the box" with the python.orgbinary for Windows, and the "standard" MS compilers that are used with > those builds. That used to be an easy "python setup.py install" away -- has > that changed? If so, is this a known bug, or a known > we-aren't-supporting-that? > > i.e. it would be nice if anyone setup to build C extensions could "just > build numpy". > This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). Numpy is actually fairly easy to build if you have a C Compiler (which is the obvious pain point on windows). Scipy, and fortran is where things fall apart. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jan 29 23:52:06 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 17:52:06 -0500 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: I don?t see any reason why SSE couldn?t be added as tags in the Wheel filename fwiw. That doesn?t help for things like MKL though. On Jan 29, 2014, at 5:50 PM, David Cournapeau wrote: > > > > On Wed, Jan 29, 2014 at 10:27 PM, Chris Barker wrote: > On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau wrote: > I think the SSE issue is a bit of a side discussion: most people who care about performance already know how to install numpy. What we care about here are people who don't care so much about fast eigenvalue decomposition, but want to use e.g. pandas. Building numpy in a way that supports every architecture is both doable and acceptable IMO. > > Exactly -- I'm pretty sure SSE2 is being suggested because that's the lowest common denominator that we expect to see a lot of -- if their really are a lot of non-SSE-2 machines out there we could leave that off, too. > > The failure mode is fairly horrible though, and the gain is not that substantial anyway compared to really optimized installation (MKL, etc... as provided by Continuum or us). > > Building numpy wheels is not hard, we can do that fairly easily (I have already done so several times, the hard parts have nothing to do with wheel or even python, and are related to mingw issues on win 64 bits). > > David, > > Where is numpy as with building "out of the box" with the python.org binary for Windows, and the "standard" MS compilers that are used with those builds. That used to be an easy "python setup.py install" away -- has that changed? If so, is this a known bug, or a known we-aren't-supporting-that? > > i.e. it would be nice if anyone setup to build C extensions could "just build numpy". > > This has always been possible, and if not, that's certainly considered as a bug (I would be eager to fix). > > Numpy is actually fairly easy to build if you have a C Compiler (which is the obvious pain point on windows). Scipy, and fortran is where things fall apart. > > David > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From cournape at gmail.com Wed Jan 29 23:58:44 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Jan 2014 22:58:44 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Wed, Jan 29, 2014 at 10:52 PM, Donald Stufft wrote: > I don?t see any reason why SSE couldn?t be added as tags in the Wheel > filename fwiw. > You still need to decide when to install what, but I would be interested in talking more about that part. > > That doesn?t help for things like MKL though. > Nope, but MKL is actually easy in the sense that it deals with architectures at runtime. OSS numerical libraries generally don't (lots of work, and often a non issue when you can build stuff by yourself :) ). David > > On Jan 29, 2014, at 5:50 PM, David Cournapeau wrote: > > > > > On Wed, Jan 29, 2014 at 10:27 PM, Chris Barker wrote: > >> On Wed, Jan 29, 2014 at 2:04 PM, David Cournapeau wrote: >> >>> I think the SSE issue is a bit of a side discussion: most people who >>> care about performance already know how to install numpy. What we care >>> about here are people who don't care so much about fast eigenvalue >>> decomposition, but want to use e.g. pandas. Building numpy in a way that >>> supports every architecture is both doable and acceptable IMO. >>> >> >> Exactly -- I'm pretty sure SSE2 is being suggested because that's the >> lowest common denominator that we expect to see a lot of -- if their really >> are a lot of non-SSE-2 machines out there we could leave that off, too. >> > > The failure mode is fairly horrible though, and the gain is not that > substantial anyway compared to really optimized installation (MKL, etc... > as provided by Continuum or us). > > >> Building numpy wheels is not hard, we can do that fairly easily (I have >>> already done so several times, the hard parts have nothing to do with wheel >>> or even python, and are related to mingw issues on win 64 bits). >>> >> >> David, >> >> Where is numpy as with building "out of the box" with the python.orgbinary for Windows, and the "standard" MS compilers that are used with >> those builds. That used to be an easy "python setup.py install" away -- has >> that changed? If so, is this a known bug, or a known >> we-aren't-supporting-that? >> >> i.e. it would be nice if anyone setup to build C extensions could "just >> build numpy". >> > > This has always been possible, and if not, that's certainly considered as > a bug (I would be eager to fix). > > Numpy is actually fairly easy to build if you have a C Compiler (which is > the obvious pain point on windows). Scipy, and fortran is where things fall > apart. > > David > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 29 23:59:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 08:59:15 +1000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: But that's what I'm saying, there are only three ways to break this behaviour: 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme The first two aren't going to happen, which leaves only the third. You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature. People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jan 30 00:45:57 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 18:45:57 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: On Jan 29, 2014, at 5:59 PM, Nick Coghlan wrote: > But that's what I'm saying, there are only three ways to break this behaviour: > > 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer > 2. Break zipimport itself to explicitly disallow wheel files > 3. Switch to a zipimport incompatible compression scheme > > The first two aren't going to happen, which leaves only the third. > > > You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature. > Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility. > People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. > > Regards, > Nick. > I completely realized that the current design implied zip import compatibility, but implicit features do not make features that we intend to support going into the future. You're making the decision for us that we're not going to be making these kinds of changes and even then it's really not about whether or not this change can be removed or not. For instance, pip supports installing from Wheels how long until we get a bug report asking for the ability to install zipped Wheels? If we refuse to accept it are we violating the spec? If we implement the spec as it stands are we handing the users potential footguns? It's more than just backwards compatibility concerns as well. I care about providing the end users of these tools comprehensive solutions to the problems they solve. By simply tacking a documented feature on after the PEP has been accepted you're also removing the ability of this sig to arrive at such a comprehensive solution. Instead what you get is "Well you can add a Wheel directly to sys.path, except when you can't". This is user hostile. Essentially a feature is more than the raw ability to do something, it also includes documentation, support features, support infrastructure, etc. You've removed the chance for this sig to participate in defining how we're going to present this feature, what APIs around it we want to provide etc. Python is a very dynamic language, just because you *can* do something doesn't mean it's supported to actually do it. The PEP process exists so that important decisions can be discussed and honed. I want this change reverted and deferred to 1.1 so that we can follow the PEP process and have this discussion. These seems like the most reasonable thing to do and the thing that is most in spirit of the PEP process. It may be that ultimately zip importable Wheels are added as a feature, hopefully with the proper infrastructure to handle it sanely, but it is *wrong* to make the decision that this a feature that Wheels officially support with first having that discussion. Finally I don't understand why reverting is such a controversial change with you. As far as I can tell your rationale for adding this change is that it would be difficult to remove support for it. In that case, and if it's truly a thing that this sig wants to support as a desired feature, then I'm sure the proponents of this particular feature will have no trouble getting it formally accepted as part of Wheel 1.1 whereas inversely if it ends up being a thing that this sig does *not* want to support, "removing" that feature is much harder than "adding" it. The dialog is essentially being forced to be "Why should we break this for people who depended on our documented feature" which is a much harder thing to justify than "Why should we formally define this feature". ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From noah at coderanger.net Thu Jan 30 00:55:11 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 29 Jan 2014 15:55:11 -0800 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: On Jan 29, 2014, at 2:59 PM, Nick Coghlan wrote: > But that's what I'm saying, there are only three ways to break this behaviour: > > 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer > 2. Break zipimport itself to explicitly disallow wheel files > 3. Switch to a zipimport incompatible compression scheme > > The first two aren't going to happen, which leaves only the third. > > You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature. > > People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Wed Jan 29 18:50:35 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Wed, 29 Jan 2014 12:50:35 -0500 Subject: [Distutils] PEP 427 In-Reply-To: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: > > Does it mean that it actually makes sense to look into that > > direction and make wheel usage closer to jar? > > There is a parallel discussion going on, with the title "Using Wheel with > zipimport", > which is relevant to this question, and other questions you raised (e.g. > about > supporting C extensions/pure-Python modules. > > I read all of it and got a bit lost in between the distil API and PEP process discussion;) > > I have no knowledge about c extensions scope, but i feel > > like it might be of less importance then pure python > > packaging issues? Am I wrong? > > A lot of Python users depend on C extensions - and while it is a subset of > all Python > users, it is a large (and important) subset. Example: any usage of Python > in > numerical analysis or scientific applications involves use of C extensions. > > Regards, > > Vinay Sajip > I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the "release" program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Thanks a lot! Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugene at sazhin.us Thu Jan 30 00:34:43 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Wed, 29 Jan 2014 18:34:43 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: Guys, I'm still unable to see my emails reaching the list for unknown reason... Any ideas what can be the problem? FWIW I have tested it by adding the same __main__.py i used for the egg variant of the distribution to the wheel root and specifying $ PYTHONPATH=projectA.whl; python executable_projectB.whl I was able to run the program! So it seems like the stuff does behave like a jar for pure python modules! Don't know who to thank for that - because nobody seems to have claimed this functionality;) Well Nick said it supposed to be that way, So please don't break it!!;) It also means that I have the full picture in place without the necessity to use the buildout or anything else other then the pip and wheel from a flat folder structure! I'd appreciate if you could let me know about any problems you could foresee (we are talking pure python only) Thanks! Eugene On Wed, Jan 29, 2014 at 5:36 PM, Evgeny Sazhin wrote: > I'm sorry for possible dup, but for whatever reason i don't see this > email reaching the list, so i'm resending. > > > On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin wrote: >> >> >> >> >> On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: >>> >>> > Does it mean that it actually makes sense to look into that >>> > direction and make wheel usage closer to jar? >>> >>> There is a parallel discussion going on, with the title "Using Wheel with zipimport", >>> which is relevant to this question, and other questions you raised (e.g. about >>> supporting C extensions/pure-Python modules. > > > > I read all of it and got a bit lost in between the distil API and PEP > process discussion;) > >> >> >>> >>> > I have no knowledge about c extensions scope, but i feel >>> > like it might be of less importance then pure python >>> > packaging issues? Am I wrong? >>> >>> A lot of Python users depend on C extensions - and while it is a subset of all Python >>> users, it is a large (and important) subset. Example: any usage of Python in >>> numerical analysis or scientific applications involves use of C extensions. >>> >>> Regards, >>> >>> Vinay Sajip >> >> >> > > I can see that it might be quite beneficial to have virtualenv and pip > installing wheels locally for development needs, so here is what i was > able to come up with so far: > > I have one folder on NFS where all python developed stuff should be > *deployed* - pyhtonlib. It is impossible to use pip or virtualenv > there - so i'm bound to artifacts. The only way something can appear > there is by using the "release" program that knows how to put > artifacts in specified locations. Currently most of the stuff there is > the .py modules and few eggs (some are executable). But this stuff is > not allowing for sane dependency management, neither code reuse. I > actually don't like the idea of specifying dependencies in the code > via sys.path. I think the resolved sys.path based on requirements.txt > is much better solution. > > So i'm looking for a solution that would allow to use the same > artifact for everything (like jar) so it can guarantee that the same > subset of code that was tested, goes to production and used in dev. > Currently I'm leaning towards using pip's capability to work with flat > folders via --find-links, so i can deploy wheels to the pythonlib and > then reuse them in the development environment. > > But in this setup how do i make my program executable from pythonlib > location? I think I should I create some smart runner script that > would be able to use the pip's dependency resolution, create the > necessary sys.path basing on the wheel requirements.txt and then my > program wheel should have an entry point like __main__.py > > As Nick pointed out the wheel is a superset of the egg - so I assume > wheels can be executable, correct? How do i achieve that? From eugene at sazhin.us Wed Jan 29 23:36:52 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Wed, 29 Jan 2014 17:36:52 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: I'm sorry for possible dup, but for whatever reason i don't see this email reaching the list, so i'm resending. On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin wrote: > > > > > On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: >> >> > Does it mean that it actually makes sense to look into that >> > direction and make wheel usage closer to jar? >> >> There is a parallel discussion going on, with the title "Using Wheel with zipimport", >> which is relevant to this question, and other questions you raised (e.g. about >> supporting C extensions/pure-Python modules. I read all of it and got a bit lost in between the distil API and PEP process discussion;) > > >> >> > I have no knowledge about c extensions scope, but i feel >> > like it might be of less importance then pure python >> > packaging issues? Am I wrong? >> >> A lot of Python users depend on C extensions - and while it is a subset of all Python >> users, it is a large (and important) subset. Example: any usage of Python in >> numerical analysis or scientific applications involves use of C extensions. >> >> Regards, >> >> Vinay Sajip > > > I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the "release" program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Thanks a lot! Eugene From noah at coderanger.net Thu Jan 30 02:00:57 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 29 Jan 2014 17:00:57 -0800 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Jan 29, 2014, at 9:50 AM, Evgeny Sazhin wrote: > > > > On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: > > Does it mean that it actually makes sense to look into that > > direction and make wheel usage closer to jar? > > There is a parallel discussion going on, with the title "Using Wheel with zipimport", > which is relevant to this question, and other questions you raised (e.g. about > supporting C extensions/pure-Python modules. > > > I read all of it and got a bit lost in between the distil API and PEP process discussion;) > > > > I have no knowledge about c extensions scope, but i feel > > like it might be of less importance then pure python > > packaging issues? Am I wrong? > > A lot of Python users depend on C extensions - and while it is a subset of all Python > users, it is a large (and important) subset. Example: any usage of Python in > numerical analysis or scientific applications involves use of C extensions. > > Regards, > > Vinay Sajip > > > I can see that it might be quite beneficial to have virtualenv and pip installing wheels locally for development needs, so here is what i was able to come up with so far: > > I have one folder on NFS where all python developed stuff should be *deployed* - pyhtonlib. It is impossible to use pip or virtualenv there - so i'm bound to artifacts. The only way something can appear there is by using the "release" program that knows how to put artifacts in specified locations. Currently most of the stuff there is the .py modules and few eggs (some are executable). But this stuff is not allowing for sane dependency management, neither code reuse. I actually don't like the idea of specifying dependencies in the code via sys.path. I think the resolved sys.path based on requirements.txt is much better solution. > > So i'm looking for a solution that would allow to use the same artifact for everything (like jar) so it can guarantee that the same subset of code that was tested, goes to production and used in dev. Currently I'm leaning towards using pip's capability to work with flat folders via --find-links, so i can deploy wheels to the pythonlib and then reuse them in the development environment. > > But in this setup how do i make my program executable from pythonlib location? I think I should I create some smart runner script that would be able to use the pip's dependency resolution, create the necessary sys.path basing on the wheel requirements.txt and then my program wheel should have an entry point like __main__.py > > As Nick pointed out the wheel is a superset of the egg - so I assume wheels can be executable, correct? How do i achieve that? Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu Jan 30 02:07:18 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 20:07:18 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Jan 29, 2014, at 6:34 PM, Evgeny Sazhin wrote: > Guys, > > I'm still unable to see my emails reaching the list for unknown reason... > Any ideas what can be the problem? > > FWIW I have tested it by adding the same __main__.py i used for the > egg variant of the distribution > to the wheel root and specifying > $ PYTHONPATH=projectA.whl; python executable_projectB.whl > > I was able to run the program! > So it seems like the stuff does behave like a jar for pure python modules! > > Don't know who to thank for that - because nobody seems to have > claimed this functionality;) > Well Nick said it supposed to be that way, So please don't break it!!;) Wether this was ?supposed? to be there is a matter that?s up for debate. If you?re just using these for runtime you don?t need the Wheel format at all. Just regular old pyzips will do just fine. http://docs.python.org/2/library/zipfile#pyzipfile-objects > > It also means that I have the full picture in place without the > necessity to use the buildout or anything else other then the pip and > wheel from a flat folder structure! I'd appreciate if you could let me > know about any problems you could foresee (we are talking pure python > only) > > Thanks! > Eugene > > > On Wed, Jan 29, 2014 at 5:36 PM, Evgeny Sazhin wrote: >> I'm sorry for possible dup, but for whatever reason i don't see this >> email reaching the list, so i'm resending. >> >> >> On Wed, Jan 29, 2014 at 12:50 PM, Evgeny Sazhin wrote: >>> >>> >>> >>> >>> On Wed, Jan 29, 2014 at 9:11 AM, Vinay Sajip wrote: >>>> >>>>> Does it mean that it actually makes sense to look into that >>>>> direction and make wheel usage closer to jar? >>>> >>>> There is a parallel discussion going on, with the title "Using Wheel with zipimport", >>>> which is relevant to this question, and other questions you raised (e.g. about >>>> supporting C extensions/pure-Python modules. >> >> >> >> I read all of it and got a bit lost in between the distil API and PEP >> process discussion;) >> >>> >>> >>>> >>>>> I have no knowledge about c extensions scope, but i feel >>>>> like it might be of less importance then pure python >>>>> packaging issues? Am I wrong? >>>> >>>> A lot of Python users depend on C extensions - and while it is a subset of all Python >>>> users, it is a large (and important) subset. Example: any usage of Python in >>>> numerical analysis or scientific applications involves use of C extensions. >>>> >>>> Regards, >>>> >>>> Vinay Sajip >>> >>> >>> >> >> I can see that it might be quite beneficial to have virtualenv and pip >> installing wheels locally for development needs, so here is what i was >> able to come up with so far: >> >> I have one folder on NFS where all python developed stuff should be >> *deployed* - pyhtonlib. It is impossible to use pip or virtualenv >> there - so i'm bound to artifacts. The only way something can appear >> there is by using the "release" program that knows how to put >> artifacts in specified locations. Currently most of the stuff there is >> the .py modules and few eggs (some are executable). But this stuff is >> not allowing for sane dependency management, neither code reuse. I >> actually don't like the idea of specifying dependencies in the code >> via sys.path. I think the resolved sys.path based on requirements.txt >> is much better solution. >> >> So i'm looking for a solution that would allow to use the same >> artifact for everything (like jar) so it can guarantee that the same >> subset of code that was tested, goes to production and used in dev. >> Currently I'm leaning towards using pip's capability to work with flat >> folders via --find-links, so i can deploy wheels to the pythonlib and >> then reuse them in the development environment. >> >> But in this setup how do i make my program executable from pythonlib >> location? I think I should I create some smart runner script that >> would be able to use the pip's dependency resolution, create the >> necessary sys.path basing on the wheel requirements.txt and then my >> program wheel should have an entry point like __main__.py >> >> As Nick pointed out the wheel is a superset of the egg - so I assume >> wheels can be executable, correct? How do i achieve that? > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 04:47:44 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Wed, 29 Jan 2014 22:47:44 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: > > Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. > I didn?t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Thanks, Eugene From donald at stufft.io Thu Jan 30 04:49:39 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 22:49:39 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> >> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >> > > I didn?t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. > > Funny thing - wheel allows to do the same! Why would i want to use anything else then??? Because Python is not Java and Wheels are not Jars. You?ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. > > Thanks, > Eugene > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 05:16:01 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Wed, 29 Jan 2014 23:16:01 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: > > On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: > >>> >>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>> >> >> I didn?t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >> >> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? > > Because Python is not Java and Wheels are not Jars. You?ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. I understand that not a lot of currently existing project are using this capability - but I?m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to 1. use minimum of ?standard? tools, 2. *manage dependencies* in a unified and standardized way 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) Thanks, Eugene > >> >> Thanks, >> Eugene >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Thu Jan 30 05:17:50 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Jan 2014 23:17:50 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: > > On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: > >> >> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >> >>>> >>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>> >>> >>> I didn?t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>> >>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >> >> Because Python is not Java and Wheels are not Jars. You?ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. > > I understand that not a lot of currently existing project are using this capability - but I?m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to Pip does not install zipped wheels, and while it?s not entirely up to me I would be opposed to it getting the ability to do so. > 1. use minimum of ?standard? tools, > > 2. *manage dependencies* in a unified and standardized way > > 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. > > What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? > > I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) I don?t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. > > Thanks, > Eugene > >> >>> >>> Thanks, >>> Eugene >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 05:29:20 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Wed, 29 Jan 2014 23:29:20 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: > > On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: > >> >> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >> >>> >>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>> >>>>> >>>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>>> >>>> >>>> I didn?t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>>> >>>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>> >>> Because Python is not Java and Wheels are not Jars. You?ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >> >> I understand that not a lot of currently existing project are using this capability - but I?m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to > > Pip does not install zipped wheels, and while it?s not entirely up to me I would be opposed to it getting the ability to do so. I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you?re missing the point a bit. I?m totally fine with the way pip handles things. again briefly My idea is to use the following: central location - flat folder with wheels, accessible to read for everybody in network. for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: PYTHONPATH=1.whl:2.whl; python 3.whl where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! > >> 1. use minimum of ?standard? tools, >> >> 2. *manage dependencies* in a unified and standardized way >> >> 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. >> >> What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? >> >> I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) > > I don?t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. see above > >> >> Thanks, >> Eugene >> >>> >>>> >>>> Thanks, >>>> Eugene >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >>> ----------------- >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tseaver at palladion.com Thu Jan 30 05:50:32 2014 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 29 Jan 2014 23:50:32 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: > If you are going to document this, and it is not going to be > explicitly supported by the spec (it isn't), the _only_ logical thing > is to document that this is undefined behavior and while it works now, > people should not depend on it. Under no circumstance should we > document this as "well it works right now" without guidance about the > fact that it isn't part of the spec and is _not_ a candidate for > future design decisions. If someone would like to propose amending the > spec that can happen separately, but as it stands right now this is > nothing but convenient, undefined behavior. Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice. - - Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*. Noting is as such is the *point* of Nick's change. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLp2hIACgkQ+gerLs4ltQ6VGACgjnYRbgVSs8ceTXNeWTH0zCls yHwAn0/nyDUMRjNl7ARi0bVtkBOeO1nJ =2pvt -----END PGP SIGNATURE----- From noah at coderanger.net Thu Jan 30 06:00:18 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Wed, 29 Jan 2014 21:00:18 -0800 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: <40B44EE7-0406-49F9-8895-9FBE02C849EB@coderanger.net> On Jan 29, 2014, at 8:50 PM, Tres Seaver wrote: > Signed PGP part > On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: > > > If you are going to document this, and it is not going to be > > explicitly supported by the spec (it isn't), the _only_ logical thing > > is to document that this is undefined behavior and while it works now, > > people should not depend on it. Under no circumstance should we > > document this as "well it works right now" without guidance about the > > fact that it isn't part of the spec and is _not_ a candidate for > > future design decisions. If someone would like to propose amending the > > spec that can happen separately, but as it stands right now this is > > nothing but convenient, undefined behavior. > > Nick's point in this thread is that zip-importability is a *necessary > corrolary* (not an implementation detaiL) of the "no special installers" > design choice. No, thats a side effect of various other technologies that are beyond the scope of that spec. The spec should either say "this is undefined behavior" or commit that it is part of the spec. We can't pick this middle ground of "well right now we just get it for free because of X, Y, and Z" because that doesn't express to the reader the true semantics of is this something they can use. Don't assume the reader has as much information as we do about the pros and cons, and about how the implementations work. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu Jan 30 06:27:02 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 00:27:02 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: On Jan 29, 2014, at 11:50 PM, Tres Seaver wrote: > Signed PGP part > On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: > > > If you are going to document this, and it is not going to be > > explicitly supported by the spec (it isn't), the _only_ logical thing > > is to document that this is undefined behavior and while it works now, > > people should not depend on it. Under no circumstance should we > > document this as "well it works right now" without guidance about the > > fact that it isn't part of the spec and is _not_ a candidate for > > future design decisions. If someone would like to propose amending the > > spec that can happen separately, but as it stands right now this is > > nothing but convenient, undefined behavior. > > Nick's point in this thread is that zip-importability is a *necessary > corrolary* (not an implementation detaiL) of the "no special installers" > design choice. > > - Given that there is a non-empty set of wheels which can be unpacked > to a filesystem-directory tree in a directory on sys.path, and that > some of those wheels are already known not to otherwise break zip- > importability, it is a logical necessity that such wheels can be > put onto sys.path without unpacking. We already have existence proof > for this in software being released *by the folks who made these > specs*. > > Noting is as such is the *point* of Nick's change. > > > > Tres. > -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format". The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports". At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 06:33:24 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 00:33:24 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) Message-ID: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> >Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >they are both things that I've contemplated as things to bring forward in >using xz as an alternative compression format. Even if #1 would need a major >revision of Wheel to happen adding official "support" for zip import means that >the change would have to be weighed against also breaking that backwards >compatibility. Eh, please don?t break it!! Improve! Keep the ability to add wheels to python path. I don?t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i?m probably alone in the desert?l;) From donald at stufft.io Thu Jan 30 06:38:32 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 00:38:32 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: On Jan 29, 2014, at 11:50 PM, Tres Seaver wrote: > Signed PGP part > On 01/29/2014 06:55 PM, Noah Kantrowitz wrote: > > > If you are going to document this, and it is not going to be > > explicitly supported by the spec (it isn't), the _only_ logical thing > > is to document that this is undefined behavior and while it works now, > > people should not depend on it. Under no circumstance should we > > document this as "well it works right now" without guidance about the > > fact that it isn't part of the spec and is _not_ a candidate for > > future design decisions. If someone would like to propose amending the > > spec that can happen separately, but as it stands right now this is > > nothing but convenient, undefined behavior. > > Nick's point in this thread is that zip-importability is a *necessary > corrolary* (not an implementation detaiL) of the "no special installers" > design choice. > > - Given that there is a non-empty set of wheels which can be unpacked > to a filesystem-directory tree in a directory on sys.path, and that > some of those wheels are already known not to otherwise break zip- > importability, it is a logical necessity that such wheels can be > put onto sys.path without unpacking. We already have existence proof > for this in software being released *by the folks who made these > specs*. > > Noting is as such is the *point* of Nick's change. > > > > Tres. > -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Also it?s entirely inaccurate that Zip import must be maintained in order to support using unzip as a simple installer for simple Wheels. For instance if Wheels instead used directories at the root named after sysconfig scheme names then with your standard unzip tool you could simple do:: unzip example.whl 'purelib/*' Which is every bit as simple and not requiring a specialized installer but ends up breaking zip import compatibility (Techincally you'd just need to add the example.whl/purelib to the sys.path). However this is a change that wasn't off the table in the original PEP, it is now in the PEP-by-fiat because we now have to maintain zip import compatibility. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 06:54:02 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 00:54:02 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Jan 30, 2014, at 12:33 AM, Evgeny Sazhin wrote: >> Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >> they are both things that I've contemplated as things to bring forward in >> using xz as an alternative compression format. Even if #1 would need a major >> revision of Wheel to happen adding official "support" for zip import means that >> the change would have to be weighed against also breaking that backwards >> compatibility. > > > Eh, please don?t break it!! Improve! Keep the ability to add wheels to python path. > I don?t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) > I know, i?m probably alone in the desert?l;) > > OTOH if this is such a bad idea - what would be the least intrusive way for me to deploy the code? What about unzipped wheel? If I have wheels in one location and then my release procedure knows to unzip them in the folder that matches the wheel name and structure completely, would it be possible to add this unzipped folder to the PYTHONPATH and run the program? Is that closer to what you guys are preaching?? Thanks, Eugene From p.f.moore at gmail.com Thu Jan 30 08:50:39 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 07:50:39 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 29 January 2014 22:50, David Cournapeau wrote: >> i.e. it would be nice if anyone setup to build C extensions could "just >> build numpy". > > This has always been possible, and if not, that's certainly considered as a > bug (I would be eager to fix). I don't know if you saw my comment earlier in this thread: > A quick test later: > No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the > rest of the error stack (which I'm frankly not interested in investing > the time to diagnose and fix) I get "RuntimeError: Broken toolchain: > cannot link a simple C program". Which is utter rubbish - I routinely > build extensions with this installation. This is a straight "pip install numpy" run in a virtualenv on Windows 7 64-bit with MSVC 2010 (full edition) installed. I still don't have the time to do detailed diagnosis or digging, but if you cannot generate the same error yourself I can rerun the command and put the output into a bug report if that helps. (Please provide a link to the bug tracker, I didn't see it mentioned on www.numpy.org or on the pypi page for numpy). Paul From p.f.moore at gmail.com Thu Jan 30 09:09:47 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 08:09:47 +0000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On 30 January 2014 05:33, Evgeny Sazhin wrote: >>Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >>they are both things that I've contemplated as things to bring forward in >>using xz as an alternative compression format. Even if #1 would need a major >>revision of Wheel to happen adding official "support" for zip import means that >>the change would have to be weighed against also breaking that backwards >>compatibility. > > > Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. > I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) > I know, i'm probably alone in the desert...l;) This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate). People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars. This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive. Pointing out the implication that wheels can be put on sys.path in specialised circumstances is entirely reasonable when talking to a Python audience, but it triggers entirely the wrong associations when a person with a Java background sees the statement. Evgeny - for your purposes, you'd be better advised to see wheels as similar in principle to Windows MSI installers (they aren't, but it will avoid triggering incorrect analogies). I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. (Be careful, you may hear suggestions that you use eggs because they are similar to jars - while that is true, the way Python works means that the experience with eggs is far from being as natural as the jar experience you're used to). Paul From ralf.gommers at gmail.com Thu Jan 30 09:48:05 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 30 Jan 2014 09:48:05 +0100 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1391012529.12131.YahooMailBasic@web172405.mail.ir2.yahoo.com> References: <1391012529.12131.YahooMailBasic@web172405.mail.ir2.yahoo.com> Message-ID: On Wed, Jan 29, 2014 at 5:22 PM, Vinay Sajip wrote: > -------------------------------------------- > On Wed, 29/1/14, Paul Moore wrote: > > > That package installation utilities should not dabble in sys.path > manipulation. > > The import path is the user's responsibility. > > User as in developer (rather than end user). Right, and distlib's wheel > code > does no sys.path manipulation unless explicitly asked to. > Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jan 30 10:09:40 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 09:09:40 +0000 (GMT) Subject: [Distutils] Using Wheel with zipimport In-Reply-To: Message-ID: <1391072980.24027.YahooMailBasic@web172401.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Ralf Gommers wrote: > Also end user. > If, as a user, I want to use inplace builds and PYTHONPATH > instead of virtualenvs for whatever reason, that should be > supported. Setuptools inserting stuff to sys.path that come > before PYTHONPATH entries is quite annoying. If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent. Regards, Vinay Sajip From cournape at gmail.com Thu Jan 30 10:12:40 2014 From: cournape at gmail.com (David Cournapeau) Date: Thu, 30 Jan 2014 09:12:40 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore wrote: > On 29 January 2014 22:50, David Cournapeau wrote: > >> i.e. it would be nice if anyone setup to build C extensions could "just > >> build numpy". > > > > This has always been possible, and if not, that's certainly considered > as a > > bug (I would be eager to fix). > > I don't know if you saw my comment earlier in this thread: > > > A quick test later: > > No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the > > rest of the error stack (which I'm frankly not interested in investing > > the time to diagnose and fix) I get "RuntimeError: Broken toolchain: > > cannot link a simple C program". Which is utter rubbish - I routinely > > build extensions with this installation. > > This is a straight "pip install numpy" run in a virtualenv on Windows > 7 64-bit with MSVC 2010 (full edition) installed. which version of python ? For 2.x, it is expected to fail at that point, since you can't build C extensions with something different than 2008 (with python.org builds). For 3.3, it means there is a bug that needs to be fixed. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jan 30 10:45:49 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 09:45:49 +0000 (GMT) Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: Message-ID: <1391075149.53702.YahooMailBasic@web172402.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Paul Moore wrote: > This is the biggest concern I see with "promoting" wheels > being directly importable via zipimport (I say "promoting" and > not "specifying" deliberately, but I don't want to get back into > the process debate). In my view, we can have our cake and eat it. Those who don't want their wheels to be mountable can say so in their wheel metadata. I expect that distlib will honour such metadata (once the details have been worked out) and not mount wheels which aren't supposed to be mounted. However, it's perfectly easy to write code that runs from zip, as long as you know that's a deployment option you want to support. So it doesn't make sense to prevent that useful functionality - even pip is using it, I believe ;-) - just because some people think it's a bad idea, for reasons that are hardly compelling. To say that we should keep packaging separate from importing is in some sense a religious argument, unless sound technical reasons are given for it. Java proves otherwise otherwise - and for those who hate Java, that's a religious viewpoint, too ... The argument that importing wheels will cause problems is trumped by allowing the mountability to be configurable by wheel metadata: the builder of a wheel, by saying it is mountable, is agreeing to all that entails in terms of handling package resources appropriately, etc. So the support burden shifts to them, and not to wheel, distlib or any packaging tool. So much heat over process, but so little light over exactly why *appropriately designed* software deployed in wheels shouldn't be importable. No detailed analysis of any problem with wheels taking into account the differences from eggs (the things Daniel mentioned in the other thread "problems with eggs?", plus the fact that there's no sys.path manipulation *magic*). Just blanket statements to the effect that "eggs were importable and bad, so wheels must be too" smacks of superstition, not engineering. I would like the detractors of importable wheels to put their technical reasoning on the table, not use process debates to try to revert situations they're not happy with. That technical reasoning should address wheels as they are now and avoid referring to eggs as the bogeyman. The only reason I've heard from detractors so far is that software not designed to run from zip can give unexpected results when run from zip, which could give the wheel format a bad name. Given that a wheel publisher can have a say on importability, this issue seems to me to be adequately addressed. No detractor has come up with any other solid reasoning as to why the useful feature of wheel importability is bad. I invite them to put up, or ... we'll still be in the dark the next time this debate comes around ;-) Regards, Vinay Sajip From p.f.moore at gmail.com Thu Jan 30 11:20:21 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 10:20:21 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 30 January 2014 09:12, David Cournapeau wrote: > > On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore wrote: >> >> On 29 January 2014 22:50, David Cournapeau wrote: >> >> i.e. it would be nice if anyone setup to build C extensions could "just >> >> build numpy". >> > >> > This has always been possible, and if not, that's certainly considered >> > as a >> > bug (I would be eager to fix). >> >> I don't know if you saw my comment earlier in this thread: >> >> > A quick test later: >> > No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the >> > rest of the error stack (which I'm frankly not interested in investing >> > the time to diagnose and fix) I get "RuntimeError: Broken toolchain: >> > cannot link a simple C program". Which is utter rubbish - I routinely >> > build extensions with this installation. >> >> This is a straight "pip install numpy" run in a virtualenv on Windows >> 7 64-bit with MSVC 2010 (full edition) installed. > > > which version of python ? > > For 2.x, it is expected to fail at that point, since you can't build C > extensions with something different than 2008 (with python.org builds). > > For 3.3, it means there is a bug that needs to be fixed. Doh. Sorry, I knew I'd forget something. 3.3. Paul From cournape at gmail.com Thu Jan 30 11:25:56 2014 From: cournape at gmail.com (David Cournapeau) Date: Thu, 30 Jan 2014 10:25:56 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On Thu, Jan 30, 2014 at 10:20 AM, Paul Moore wrote: > On 30 January 2014 09:12, David Cournapeau wrote: > > > > On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore wrote: > >> > >> On 29 January 2014 22:50, David Cournapeau wrote: > >> >> i.e. it would be nice if anyone setup to build C extensions could > "just > >> >> build numpy". > >> > > >> > This has always been possible, and if not, that's certainly considered > >> > as a > >> > bug (I would be eager to fix). > >> > >> I don't know if you saw my comment earlier in this thread: > >> > >> > A quick test later: > >> > No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the > >> > rest of the error stack (which I'm frankly not interested in investing > >> > the time to diagnose and fix) I get "RuntimeError: Broken toolchain: > >> > cannot link a simple C program". Which is utter rubbish - I routinely > >> > build extensions with this installation. > >> > >> This is a straight "pip install numpy" run in a virtualenv on Windows > >> 7 64-bit with MSVC 2010 (full edition) installed. > > > > > > which version of python ? > > > > For 2.x, it is expected to fail at that point, since you can't build C > > extensions with something different than 2008 (with python.org builds). > > > > For 3.3, it means there is a bug that needs to be fixed. > > Doh. Sorry, I knew I'd forget something. 3.3. > Here we go: https://github.com/numpy/numpy/issues/4245 David > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 30 11:45:17 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 10:45:17 +0000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <1391075149.53702.YahooMailBasic@web172402.mail.ir2.yahoo.com> References: <1391075149.53702.YahooMailBasic@web172402.mail.ir2.yahoo.com> Message-ID: On 30 January 2014 09:45, Vinay Sajip wrote: > So much heat over process, but so little light over exactly why > *appropriately designed* software deployed in wheels shouldn't be > importable. Good point. In my view, *appropriately designed* software is fine, and a metadata flag is a good idea. Although obviously that makes "mounting" a little more complex (check the flag) and argues for a mount method rather than people just fiddling sys.path. There's a lot of leeway for dispute over "appropriate design" and who gets to say, but that's not a technical issue. My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't need to use the feature. (Would you be willing to add some sort of "never extract C extensions regardless of what the metadata might say" option to wheel mount? It's not critical, as mounting random wheels without knowing how they work is clearly bad, but it does add a level of assurance that might be helpful,) The remaining issues seem to be mainly of education and expectations - people coming from non-Python (particularly Java) backgrounds have a set of assumptions that aren't easy to translate into Python terms. That's definitely not a technical problem. Paul. From p.f.moore at gmail.com Thu Jan 30 11:56:56 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 10:56:56 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: Thanks. I've added the installation details and output from a test run. Paul On 30 January 2014 10:25, David Cournapeau wrote: > > > > On Thu, Jan 30, 2014 at 10:20 AM, Paul Moore wrote: >> >> On 30 January 2014 09:12, David Cournapeau wrote: >> > >> > On Thu, Jan 30, 2014 at 7:50 AM, Paul Moore wrote: >> >> >> >> On 29 January 2014 22:50, David Cournapeau wrote: >> >> >> i.e. it would be nice if anyone setup to build C extensions could >> >> >> "just >> >> >> build numpy". >> >> > >> >> > This has always been possible, and if not, that's certainly >> >> > considered >> >> > as a >> >> > bug (I would be eager to fix). >> >> >> >> I don't know if you saw my comment earlier in this thread: >> >> >> >> > A quick test later: >> >> > No BLAS/ATLAS/LAPACK causes a string of warnings, And ignoring the >> >> > rest of the error stack (which I'm frankly not interested in >> >> > investing >> >> > the time to diagnose and fix) I get "RuntimeError: Broken toolchain: >> >> > cannot link a simple C program". Which is utter rubbish - I routinely >> >> > build extensions with this installation. >> >> >> >> This is a straight "pip install numpy" run in a virtualenv on Windows >> >> 7 64-bit with MSVC 2010 (full edition) installed. >> > >> > >> > which version of python ? >> > >> > For 2.x, it is expected to fail at that point, since you can't build C >> > extensions with something different than 2008 (with python.org builds). >> > >> > For 3.3, it means there is a bug that needs to be fixed. >> >> Doh. Sorry, I knew I'd forget something. 3.3. > > > Here we go: https://github.com/numpy/numpy/issues/4245 > > David >> >> Paul > > From ncoghlan at gmail.com Thu Jan 30 12:43:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 21:43:04 +1000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: On 30 Jan 2014 15:27, "Donald Stufft" wrote: > I can't believe folks are unable to differentiate between the difference of > "It's possible to do so because we didn't actively attempt to prevent it" and > "This is a documented goal of the format and thus must be considered as part of > the backward compatibility contract that the format has whenever changes are > made to the format". Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional ( https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. > The first option is, as far as I can tell, what the PEP read as and the > discussion, that occurred in public at least, read as. However since the change > Nick made he's shifted it from the first to the second type of "supports". > > At this point I can only imagine people are being willfully ignorant because > they want this particular feature and don't want to have it available to be > discussed as per the PEP process and are actively attempting to sidestep the > process. I'm very explicitly trying not to argue for or against this feature, > although I believe it a bad idea, but instead that before we commit to > promising that Wheels will be zip importable by simply adding them to sys.path > that we *need* to discuss it and figure out what the contract we're providing > for that is. The problem with believing that we still have a choice about this topic is twofold: - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format. That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Jan 30 12:57:57 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 11:57:57 +0000 (GMT) Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: Message-ID: <1391083077.1052.YahooMailBasic@web172403.mail.ir2.yahoo.com> > My one technical issue is with going beyond zipimport > behaviour to the point of extracting DLLs to the filesystem. > I remain -1 on that feature, and I believe I have explained why I think > there are issues (and why I think that any solution should be part of > zipimport and not added on in library or user code). But I'm happy to go > through the details again, if you like - or just to accept that I don't Yes please, let's get into some details. Of course I understand that you might not want to use the feature, but I don't understand the -1 on the feature per se - whether it is in distlib or in zipimport is a secondary consideration. I agree that zipimport is the logical place for it, but ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility. I accept that this not yet a fully resolved issue in general (cf. the parallel discussion about numpy), but if we can isolate these issues, we can perhaps tackle them. But for me, that's the main reason why this part of the distlib API is experimental. Since zipimport makes no prescription about the contents of a zip (beyond describing how importing works), there is no agreed place to place metadata information such as about binary compatibility. Nor is binary compatibility between third-party packages of core concern to python-dev, so I'm not sure discussions there will be fruitful. However, such compatibility is a valid concern here, so I would expect useful input to come from interested parties. Also, the wheel format already caters for a limited set of binary compatibility info to be communicated, but this information is incomplete, which results in reservations about the dangers of extracting C extensions. In a sense, the contentiousness of extraction of these from a wheel is a red herring, because the exact same issues would arise if you installed from the wheel and then tried to use software which purported to be binary compatible with your system, but wasn't. That's why we don't put Linux wheels on PyPI, right? It's to avoid binary wheels compiled on e.g. CentOS ending up on an e.g. Ubuntu system which is not quite binary compatible. But that doesn't solve the problem at source, so much as act as a prophylactic. Much of the Python community works in a POSIX environment where build- from-source is the norm and binary compatibility is a non-issue. In a corporate environment with a homogeneous infrastructure, the same applies; but in a heterogeneous environment, not so much. I think some benefit would accrue to packaging as a whole if we had better definitions of binary compatibility, the ability to express it in more detail, how to test for it at installation time and so on. I think this is one of the areas where we can and should improve WHEEL metadata. Perhaps the NumPy/SciPy readers would care to comment, as we're talking beyond the realms of Python version compatibility. If you have other reasons for your -1, I'd like to hear them. > need to use the feature. (Would you be willing to add some sort of > "never extract C extensions regardless of what the metadata might > say" option to wheel mount? It's not critical, as mounting random > wheels without knowing how they work is clearly bad, but it does add a > level of assurance that might be helpful,) Currently, extraction only happens if there is metadata in the wheel to indicate that extraction should occur, and it's up to the wheel builder to put it there. It doesn't make sense in general to prohibit just the C extensions but allow importing pure-Python modules in a wheel: the pure-Python bits mightn't work properly if the C extensions aren't available. So it seems safer in general to have all-or-nothing, or else have an additional flag in the metadata which indicates that the C extensions have to be extracted for the wheel to be useful. End user control of mounting should be at the discretion of the developer of the software which invokes the mount method. Regards, Vinay Sajip From ncoghlan at gmail.com Thu Jan 30 13:29:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jan 2014 22:29:21 +1000 Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) Message-ID: On 30 January 2014 21:57, Vinay Sajip wrote: >> My one technical issue is with going beyond zipimport >> behaviour to the point of extracting DLLs to the filesystem. >> I remain -1 on that feature, and I believe I have explained why I think >> there are issues (and why I think that any solution should be part of >> zipimport and not added on in library or user code). But I'm happy to go >> through the details again, if you like - or just to accept that I don't > > Yes please, let's get into some details. Of course I understand that you > might not want to use the feature, but I don't understand the -1 on the feature > per se - whether it is in distlib or in zipimport is a secondary consideration. I > agree that zipimport is the logical place for it, but ISTM the reason why it can't > go in there just yet is also the reason why one might have some reservations > about the feature: binary compatibility. I accept that this not yet a fully resolved > issue in general (cf. the parallel discussion about numpy), but if we can > isolate these issues, we can perhaps tackle them. But for me, that's the main > reason why this part of the distlib API is experimental. I actually think this is a useful thing to experiment with, I'm just not sure distlib is the best place for that experiment. With appropriately secure tempfile handling and the right sys.path (and module __path__) manipulation it's not obviously *impossible* to handle C extensions at arbitrary positions in the module namespace this way, just difficult. zipimport itself is a bad place to experiment though, since not only is it currently a complex ball of C code, but adding such a feature without clear evidence of robust support in a third party project would be irresponsible. In the case of distlib, the potential complexity of ensuring that such a scheme works consistently across multiple platforms and as part of various complex package layouts is enough to make me nervous about having it in the same library as the metadata 2.0 reference implementation :) Now, if you were to split that functionality out from distlib into a separate "wheeltab" project (or a name of your choice), I'd be substantially less nervous, because endorsing distlib as the metadata 2.0 reference implementation wouldn't carry any implications of endorsing a feature I consider "potentially interesting but rather challenging to implement in a robust manner". mount() would become something I could explore when I had some additional free time (hah!), rather than something I felt obliged to help get to a more robust state before releasing metadata 2.0. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Jan 30 14:23:52 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 13:23:52 +0000 Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) Message-ID: Changing the subject to clearly focus the discussion. On 30 January 2014 11:57, Vinay Sajip wrote: > If you have other reasons for your -1, I'd like to hear them. OK. Note that this is not, in my view, an issue with wheels, but rather about zipfiles on sys.path, and (deliberate) design limitations of the module loader and zipimport implementations.[1] First of all, it is not possible to load a DLL into a process' memory [2, 3] unless it is stored as a file in the filesystem. So any attempt to import a C extension from a zipfile must, by necessity, involve extracting that DLL to the filesystem. That's where I see the problems. None are deal-breaking issues, but they consist of a number of niggling issues that cumulatively chip away at the reliability of the concept until the end result has enough corner cases and risks to make it unacceptable (depending on your tolerance for risks - there's a definite judgement call involved). The issues I can see are: [4] 1. You need to choose a location to put the extracted file. On Windows in particular, there is no guaranteed-available filesystem location that can be used without risk. Some accounts have no home directory, some (locked down) users have no permissions anywhere but very specific places, even TEMP may not be usable if there's an aggressive housekeeping routine in place - but TEMP is probably the best choice of a bad lot. 2. There are race conditions to consider. If the extraction is not completely isolated per-process, what if 2 processes want to use different versions of the same DLL? How will these be distinguished? [5] So to avoid corner cases you have to assume only the one process uses a given extracted DLL. 3. Clean-up is an issue. How will the extracted files be removed? You can't unload the DLLs from Python, and you can't delete open files in Windows. So do you simply leave the files lying round? Or do you do some sort of atexit dance to run a separate process after the Python process terminates which will do the cleanup? What happens to that process when virus checkers hold the file open? Leaving the files around is probably the most robust answer, but it's not exactly friendly. As I've said elsewhere, these are fundamental issues with importing DLLs from zipfiles, and have no direct relationship to wheels. The only place where having a wheel rather than a general zipfile makes a difference is that a wheel *might* at some point contain metadata that allows the wheel to claim that it's "OK" to load its contents from a zipfile. But my points above are not something that the author of the C extension can address, so there's no way that I can see that an extension author can justifiably set that flag. So: as wheels don't give any additional reliability over any other zipfile, I don't see this (loading C extensions) as a wheel-related feature. Ideally, if these problems can be solved, the solution should be included in the core zipimport module so that all users can benefit. If there are still issues to iron out and experience to be gained, a 3rd party "enhanced zip importer" module would be a reasonable test-bed for the solution. A 3rd party solution could also be appropriate if the caveats and/or limitations were generally acceptable, but sufficient to prohibit stdlib inclusion. The wheel mount API could, if you wanted, look for the existence of that enhanced zipimport module and use it when appropriate, but baking the feature into wheel mount just limits your user base (and hence your audience for raising bug reports, etc) needlessly. I hope this explains my reasoning in sufficient detail. FINAL DISCLAIMER: I have no objection to this feature being provided per se, any more than I object to the existence of (say) Zope. Just because I'm not a member of the target audience doesn't mean that it's not a feature that some might benefit from. All I'm trying to do here is offer my input as someone who was involved in the initial implementation of zipimport, and who has kept an interested eye on how it has been used in the 11 years since its introduction - and in particular how people have tried to overcome the limitations we felt we had to impose when designing it. Ultimately, I would be overjoyed if someone could find a solution to this issue (in much the same way as I'm delighted by what Brett has done with importlib). Paul Footnotes: [1] Historical footnote - I was directly involved with the design of PEP 302 and the zipimport implementation, and we made a deliberate choice to only look at pure Python files, because the platform issues around C extensions were "too hard". [2] I'm talking from a Windows perspective here. I do not have sufficient low-level knowledge of Unix to comment on that case. I suspect that the issues are similar but I defer to the platform experts. [3] There is, I believe, code "out there" on the internet to map a DLL image into a process based purely in memory, but I think it's a fairly gross hack. I have a suspicion that someone - possibly Thomas Heller - experimented with it at one time, but never came up with a viable implementation. There's also TCL's tclkit technology, which *may* support binary extensions, and may be worth a look, but TCL has virtual filesystem support built in quite deep in the core, so how it works may not be applicable to Python. [4] I'm suggesting answers to the questions I'm raising here. The answers *may* be wrong - I've never tried to design a robust solution to this issue - but I believe the questions are the important point here. Please don't focus on why my suggested approach is wrong - I know it is! [5] To be fair, this is where the wheel metadata might help in distinguishing. But consider development and testing, where repeated test runs would not typically have different versions, but the user might well want to test whether running from zip still works. So wheel metadata helps, but isn't a complete solution. And compile data is probably just as good, so let's keep assuming we are looking at a general zipimport facility. From p.f.moore at gmail.com Thu Jan 30 14:26:54 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 13:26:54 +0000 Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: On 30 January 2014 12:29, Nick Coghlan wrote: > I actually think this is a useful thing to experiment with, I'm just > not sure distlib is the best place for that experiment. With > appropriately secure tempfile handling and the right sys.path (and > module __path__) manipulation it's not obviously *impossible* to > handle C extensions at arbitrary positions in the module namespace > this way, just difficult. zipimport itself is a bad place to > experiment though, since not only is it currently a complex ball of C > code, but adding such a feature without clear evidence of robust > support in a third party project would be irresponsible. I just sent a long message that essentially gave a chunk of history and suggested a similar thing - an "enhanced zipimport" module to experiment with solutions in this space. With importlib available, an experimental implementation shouldn't even be too hard. The only difference is that I see very little reason why such a solution can't apply to all zipfiles, and not just wheels. Paul From p.f.moore at gmail.com Thu Jan 30 14:32:39 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 13:32:39 +0000 Subject: [Distutils] pip on windows experience In-Reply-To: References: <20140123144227.GB3553@gmail.com> Message-ID: On 30 January 2014 10:56, Paul Moore wrote: >> Here we go: https://github.com/numpy/numpy/issues/4245 >> > Thanks. I've added the installation details and output from a test run. That bug report was just closed blaming a distutils issue which apparently numpy aren't going to work around :-( I don't know if you want to pick up on the issue and argue the case with the guy who closed it. So no, numpy won't build on Windows. Maybe in Python 3.5, if someone aggressively pushes for a distutils fix. But I'm not holding my breath. Sigh. Paul From p.f.moore at gmail.com Thu Jan 30 14:41:48 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 13:41:48 +0000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <1391083077.1052.YahooMailBasic@web172403.mail.ir2.yahoo.com> References: <1391083077.1052.YahooMailBasic@web172403.mail.ir2.yahoo.com> Message-ID: On 30 January 2014 11:57, Vinay Sajip wrote: > ISTM the reason why it can't > go in there just yet is also the reason why one might have some reservations > about the feature: binary compatibility Sorry, I didn't comment on this point. To me, binary compatibility is *not* the issue. So my other email ignored it. It is a reasonable point, but not one that bothers me, oddly enough :-) Paul From donald at stufft.io Thu Jan 30 15:30:34 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 09:30:34 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: Message-ID: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> On Jan 30, 2014, at 6:43 AM, Nick Coghlan wrote: > On 30 Jan 2014 15:27, "Donald Stufft" wrote: > > I can't believe folks are unable to differentiate between the difference of > > "It's possible to do so because we didn't actively attempt to prevent it" and > > "This is a documented goal of the format and thus must be considered as part of > > the backward compatibility contract that the format has whenever changes are > > made to the format". > > Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional (https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. > > This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. > > I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. > > For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 > > This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. > > > The first option is, as far as I can tell, what the PEP read as and the > > discussion, that occurred in public at least, read as. However since the change > > Nick made he's shifted it from the first to the second type of "supports". > > > > At this point I can only imagine people are being willfully ignorant because > > they want this particular feature and don't want to have it available to be > > discussed as per the PEP process and are actively attempting to sidestep the > > process. I'm very explicitly trying not to argue for or against this feature, > > although I believe it a bad idea, but instead that before we commit to > > promising that Wheels will be zip importable by simply adding them to sys.path > > that we *need* to discuss it and figure out what the contract we're providing > > for that is. > > The problem with believing that we still have a choice about this topic is twofold: > > - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) > - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) > You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: > > - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) > - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way > I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format. > > That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task. > > Regards, > Nick. Yea I've tried to respond to this about 5 times without being a dick and I can't. So whatever I give up. Now I need to go figure out how to make my projects bomb out under zip import since Wheel has managed to be worse than Egg and not even provide a way to disclaim support for it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu Jan 30 15:38:42 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 09:38:42 -0500 Subject: [Distutils] PEP 427 In-Reply-To: <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> Message-ID: <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: > > On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: > >> >> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >> >>> >>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>> >>>> >>>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>>> >>>>>> >>>>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>>>> >>>>> >>>>> I didn?t say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>>>> >>>>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>>> >>>> Because Python is not Java and Wheels are not Jars. You?ll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>> >>> I understand that not a lot of currently existing project are using this capability - but I?m 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to >> >> Pip does not install zipped wheels, and while it?s not entirely up to me I would be opposed to it getting the ability to do so. > > I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you?re missing the point a bit. > I?m totally fine with the way pip handles things. > > again briefly My idea is to use the following: > > central location - flat folder with wheels, accessible to read for everybody in network. > > for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. > > To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: > PYTHONPATH=1.whl:2.whl; python 3.whl > > where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! Just use pip and virtualenv in production. It?s bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it?ll lead to hard to diagnose errors and bugs that you?ll never be able to reproduce in development. Even though the spec apparently allows it, using Wheels like this is an anti pattern. > > > > >> >>> 1. use minimum of ?standard? tools, >>> >>> 2. *manage dependencies* in a unified and standardized way >>> >>> 3. use only *one format* and *one artifact* per project for everything related to the code development, shipment, deployment and execution. >>> >>> What would be a motivation to learn yet another tool like buildout or anything, pollute infrastructure with more libraries, manage several different ways of working with the code, overcomplicate the production area by requiring stuff to be unzipped and whatnot? >>> >>> I hope nobody would want to strip any of those nice features from the wheel, but improve the running from it!;) >> >> I don?t understand what any of the rest of this have to do with wether you add zipped Wheels to your sys.path or not. > > see above > > >> >>> >>> Thanks, >>> Eugene >>> >>>> >>>>> >>>>> Thanks, >>>>> Eugene >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> >>>> ----------------- >>>> Donald Stufft >>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu Jan 30 16:03:06 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 10:03:06 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> Message-ID: On Jan 30, 2014, at 9:30 AM, Donald Stufft wrote: > > On Jan 30, 2014, at 6:43 AM, Nick Coghlan wrote: > >> On 30 Jan 2014 15:27, "Donald Stufft" wrote: >> > I can't believe folks are unable to differentiate between the difference of >> > "It's possible to do so because we didn't actively attempt to prevent it" and >> > "This is a documented goal of the format and thus must be considered as part of >> > the backward compatibility contract that the format has whenever changes are >> > made to the format". >> >> Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional (https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. >> >> This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. >> >> I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. >> >> For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 >> >> This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. >> >> > The first option is, as far as I can tell, what the PEP read as and the >> > discussion, that occurred in public at least, read as. However since the change >> > Nick made he's shifted it from the first to the second type of "supports". >> > >> > At this point I can only imagine people are being willfully ignorant because >> > they want this particular feature and don't want to have it available to be >> > discussed as per the PEP process and are actively attempting to sidestep the >> > process. I'm very explicitly trying not to argue for or against this feature, >> > although I believe it a bad idea, but instead that before we commit to >> > promising that Wheels will be zip importable by simply adding them to sys.path >> > that we *need* to discuss it and figure out what the contract we're providing >> > for that is. >> >> The problem with believing that we still have a choice about this topic is twofold: >> >> - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) >> - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) >> You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: >> >> - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) >> - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way >> I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format. >> >> That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task. >> >> Regards, >> Nick. > > Yea I've tried to respond to this about 5 times without being a dick and I > can't. So whatever I give up. Now I need to go figure out how to make my > projects bomb out under zip import since Wheel has managed to be worse than > Egg and not even provide a way to disclaim support for it. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here?s how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Thu Jan 30 16:12:31 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 15:12:31 +0000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> Message-ID: On 30 January 2014 15:03, Donald Stufft wrote: > For those following along at home who have no interest in supporting zipped > Wheels who want to disclaim support for it, here's how you do zip_safe=False > for Wheels: https://gist.github.com/dstufft/8710270. Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul From donald at stufft.io Thu Jan 30 16:23:44 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 10:23:44 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> Message-ID: <38D0D557-D16A-4166-8503-326CF43B8B8E@stufft.io> On Jan 30, 2014, at 10:12 AM, Paul Moore wrote: > On 30 January 2014 15:03, Donald Stufft wrote: >> For those following along at home who have no interest in supporting zipped >> Wheels who want to disclaim support for it, here's how you do zip_safe=False >> for Wheels: https://gist.github.com/dstufft/8710270. > > Note that this will also prevent use in py2exe, cx_Freeze and any > similar bundling technologies. Presumably that's what you want, as any > issues are identical, but it's worth being clear. > > Paul Sure, using a library via zip import when it wasn?t designed to be used as such is an anti pattern. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eric at trueblade.com Thu Jan 30 16:44:34 2014 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 30 Jan 2014 10:44:34 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <38D0D557-D16A-4166-8503-326CF43B8B8E@stufft.io> References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> <38D0D557-D16A-4166-8503-326CF43B8B8E@stufft.io> Message-ID: <52EA7362.9000006@trueblade.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/30/2014 10:23 AM, Donald Stufft wrote: > > On Jan 30, 2014, at 10:12 AM, Paul Moore > wrote: > >> On 30 January 2014 15:03, Donald Stufft >> wrote: >>> For those following along at home who have no interest in >>> supporting zipped Wheels who want to disclaim support for it, >>> here's how you do zip_safe=False for Wheels: >>> https://gist.github.com/dstufft/8710270. >> >> Note that this will also prevent use in py2exe, cx_Freeze and >> any similar bundling technologies. Presumably that's what you >> want, as any issues are identical, but it's worth being clear. >> >> Paul > > Sure, using a library via zip import when it wasn?t designed to be > used as such is an anti pattern. The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too. Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem". Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand. Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE----- From dholth at gmail.com Thu Jan 30 16:56:35 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 10:56:35 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <52EA7362.9000006@trueblade.com> References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> <38D0D557-D16A-4166-8503-326CF43B8B8E@stufft.io> <52EA7362.9000006@trueblade.com> Message-ID: I see you've noticed wheel was released in an imperfect state. Let's add a Zip-Safe flag to the WHEEL file, values of "true" and "false" same as Root-Is-Purelib. Daniel On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 01/30/2014 10:23 AM, Donald Stufft wrote: >> >> On Jan 30, 2014, at 10:12 AM, Paul Moore >> wrote: >> >>> On 30 January 2014 15:03, Donald Stufft >>> wrote: >>>> For those following along at home who have no interest in >>>> supporting zipped Wheels who want to disclaim support for it, >>>> here's how you do zip_safe=False for Wheels: >>>> https://gist.github.com/dstufft/8710270. >>> >>> Note that this will also prevent use in py2exe, cx_Freeze and >>> any similar bundling technologies. Presumably that's what you >>> want, as any issues are identical, but it's worth being clear. >>> >>> Paul >> >> Sure, using a library via zip import when it wasn't designed to be >> used as such is an anti pattern. > > The flag really needs to convey 2 meanings: > - - There are some C extensions here that can't be loaded unless they > live on a real filesystem path. > - - There is some code (maybe in this package, maybe in another package > that uses this one) that uses this package's __file__, and that code > won't work unless __file__ points to a real filesystem path. > > There are any number of possible importers that would fail due to > these 2 cases. I've personally written one that loads modules from a > database. It should respect this flag, too. > > Which is a long way of saying: I think calling it "zip_safe" is a > misnomer. The flag is really conveying "I don't need to be on a > filesystem". > > Whether there should be 2 different flags for the two different > problems (C extension and __file__), I can't say. They do seem to go > hand-in-hand. > > Eric. > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.14 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 > N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO > l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv > a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD > /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN > 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= > =6tIo > -----END PGP SIGNATURE----- > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Thu Jan 30 17:10:05 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 16:10:05 +0000 (GMT) Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) In-Reply-To: Message-ID: <1391098205.18738.YahooMailBasic@web172404.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Paul Moore wrote: Subject: Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) To: "Vinay Sajip" Cc: "Distutils" Date: Thursday, 30 January, 2014, 13:23 > OK. Note that this is not, in my view, an issue with wheels, but rather > about zipfiles on sys.path, and (deliberate) design limitations of the > module loader and zipimport implementations.[1] Okay, I'm glad that's clarified. Otherwise, there's a danger of it being conflated with an "importing wheels is bad" viewpoint which relates even to pure-Python code. > First of all, it is not possible to load a DLL into a process' memory > [2, 3] unless it is stored as a file in the filesystem. So any attempt > to import a C extension from a zipfile must, by necessity, involve > extracting that DLL to the filesystem. That's where I see the > problems. None are deal-breaking issues, but they consist of a > number of niggling issues that cumulatively chip away at the > reliability of the concept until the end result has enough corner > cases and risks to make it unacceptable (depending on your > tolerance for risks - there's a definite judgement call involved). Okay, let's work through the issues you raise. > 1. You need to choose a location to put the extracted file. On > Windows in particular, there is no guaranteed-available > filesystem location that can be used without risk. Some > accounts have no home directory, some (locked down) users > have no permissions anywhere but very specific places, even > TEMP may not be usable if there's an aggressive housekeeping > routine in place - but TEMP is probably the best choice of a bad > lot. There are always going to be environments where you can't do stuff, say because of corporate lock-down policies. There is no requirement on any solution to do the impossible; merely to fail with an informative error message. There is lots of other functionality that fails in these environments, too (e.g. access to the Internet). So my view is that this should not be an obstacle to developing such functionality for environments where it can work, as long as it fails fast and informatively when it fails. > 2. There are race conditions to consider. If the extraction is > not completely isolated per-process, what if 2 processes > want to use different versions of the same DLL? How will > these be distinguished? Processes are isolated from each other, so that doesn't stop different processes using different versions of DLLs. Software in those DLLs needs to be designed to avoid stepping on its own toes, but that's orthogonal to whether it came from a zip or not (.NET SxS assemblies, for example - if they have files they write to, they need to not overwrite each other's stuff). Distlib covers this by placing the DLL in a location which is based on the absolute pathname of the wheel it came from. So any software which uses the exact same wheel will use the same DLL, but other software which uses a wheel with a different version (which by definition will have a different absolute path) will use a different DLL. Perhaps there are holes in this approach - if so, please point out any that you see. > 3. Clean-up is an issue. How will the extracted files be > removed? You can't unload the DLLs from Python, and > you can't delete open files in Windows. So do you simply > leave the files lying round? Or do you do some sort of atexit > dance to run a separate process after the Python process > terminates which will do the cleanup? What happens > to that process when virus checkers hold the file open? > Leaving the files around is probably the most robust answer, > but it's not exactly friendly. But it's a drawback of the underlying platform, and it seems to me OK to do the best that's possible (like we do with pip updating itself on Windows). Also, it's not clear if you always want to clean up: perhaps you don't want to extract DLLs every time if they're already there (let's not go down a cache invalidation rabbit-hole - later is definitely better than right now ;-) My view is that cleanup belongs with the application, not the library - the application developer is best placed to know what the right thing to do is for that particular application. This is currently covered in distlib by having an API which provides the root directory path for the cache. Cache cleanup can be done on start-up before any wheels are mounted. By the way, surely you've seen how much cruft accumulates in TEMP on Windows machines? It's not as if Windows users' expectations can be particularly high here ;-) I'm all for keeping things tidy, of course. > The only place where having a wheel rather than a general > zipfile makes a difference is that a wheel *might* at some > point contain metadata that allows the wheel to claim that it's > "OK" to load its contents from a zipfile. > But my points above are not something that the author of the > C extension can address, so there's no way that I can see > that an extension author can justifiably set that flag. It's not the extension author exactly, it's the wheel packager. In a corporate environment, they might be someone in a systems integrator role. Even if they are one and the same, the assertion is that the wheel is designed to run from a zip. Beyond that, it's up to the application developer and/or systems integrator: it doesn't mean it will work in every circumstance as one would wish. Are you telling me that most Python packages on PyPI, conventionally installed, will handle gracefully an out-of-disk-space condition? Where the cause of the failure is immediately apparent rather than "weird" at first glance? I doubt it. > Ideally, if these problems can be solved, the solution should > be included in the core zipimport module so that all users > can benefit. If there are still issues to iron out and Nick and I have both given reasons why zipimport might not be best placed to pioneer this. Although you are not concerned with binary compatibility, it is a valid concern which needs addressing, and bolstering the WHEEL metadata seems the right place for such work. > but baking the feature into wheel mount just limits your user > base (and hence your audience for raising bug reports, etc) > needlessly. I'm not hung up about exactly where the functionality gets implemented, just that it's useful. It would seem better to focus on real issues (like the ones you've raised, and the ones I raised about binary compatibility) rather than debating how best to package it. If someone is interested in developing this area, they will put in the work of looking at the issues and coming up with ideas to address them, whether it's package or Y. What makes you think an enhanced third-party zipimport module is suddenly going to get lots of eyeballs? The functionality in distlib as a whole is a lot more useful (this being a tiny corner of it), but there aren't too many eyeballs on that. > implementation of zipimport, and who has kept an interested > eye on how it has been used in the 11 years since its > introduction - and in particular how people have tried to > overcome the limitations we felt we had to impose when > designing it. I didn't know - thanks for your work on zipimport, I think it's great. Surely 11 years is long enough for that initial functionality to have bedded down? Often, getting a new feature in means working to a feature-freeze deadline where not every avenue can be explored. That's par for the course, especially where hard technical problems are to be faced. But, surely there comes a time when it's worth taking another look, and seeing if we can push the envelope further? I hope that in the above I've addressed at least in part the issues you've raised - I'm sure you'll tell me if not. > choice to only look at pure Python files, because the > platform issues around C extensions were "too hard". Were those just the issues you raised here? Wasn't binary compatibility discussed? > There is, I believe, code "out there" on the internet to > map a DLL image into a process based purely in memory, I would discount this: any solution has to work on multiple Windows versions and the lower level the solution, the more the risk. We're talking (in the current implementation) just about file-system operations and import_dynamic, which are fairly mature and well understood by comparison. > [5] To be fair, this is where the wheel metadata might help > in distinguishing. But consider development and testing, > where repeated test runs would not typically have different > versions, but the user might well want to test whether > running from zip still works. What's wrong with having the test setup code clearing the DLL cache for every run? Clearly the wheel has to be rebuilt for each run, but that's not going to be a show-stopper if the tests are arranged optimally. Anyway, thanks for taking the time to raise the issues in detail. This kind of discussion will hopefully help to move things forward. Regards, Vinay Sajip From donald at stufft.io Thu Jan 30 17:21:57 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 11:21:57 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> <38D0D557-D16A-4166-8503-326CF43B8B8E@stufft.io> <52EA7362.9000006@trueblade.com> Message-ID: Just an FYI, I've begun compiling notes and plan on proposing PEPs that will be aimed at supplanting PEP 425 and PEP 427 with the goal of nailing down undefined behavior, including missing functionality, and removing misfeatures. On Jan 30, 2014, at 10:56 AM, Daniel Holth wrote: > I see you've noticed wheel was released in an imperfect state. > > Let's add a Zip-Safe flag to the WHEEL file, values of "true" and > "false" same as Root-Is-Purelib. > > Daniel > > On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 01/30/2014 10:23 AM, Donald Stufft wrote: >>> >>> On Jan 30, 2014, at 10:12 AM, Paul Moore >>> wrote: >>> >>>> On 30 January 2014 15:03, Donald Stufft >>>> wrote: >>>>> For those following along at home who have no interest in >>>>> supporting zipped Wheels who want to disclaim support for it, >>>>> here's how you do zip_safe=False for Wheels: >>>>> https://gist.github.com/dstufft/8710270. >>>> >>>> Note that this will also prevent use in py2exe, cx_Freeze and >>>> any similar bundling technologies. Presumably that's what you >>>> want, as any issues are identical, but it's worth being clear. >>>> >>>> Paul >>> >>> Sure, using a library via zip import when it wasn't designed to be >>> used as such is an anti pattern. >> >> The flag really needs to convey 2 meanings: >> - - There are some C extensions here that can't be loaded unless they >> live on a real filesystem path. >> - - There is some code (maybe in this package, maybe in another package >> that uses this one) that uses this package's __file__, and that code >> won't work unless __file__ points to a real filesystem path. >> >> There are any number of possible importers that would fail due to >> these 2 cases. I've personally written one that loads modules from a >> database. It should respect this flag, too. >> >> Which is a long way of saying: I think calling it "zip_safe" is a >> misnomer. The flag is really conveying "I don't need to be on a >> filesystem". >> >> Whether there should be 2 different flags for the two different >> problems (C extension and __file__), I can't say. They do seem to go >> hand-in-hand. >> >> Eric. >> >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.14 (GNU/Linux) >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 >> N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO >> l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv >> a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD >> /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN >> 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= >> =6tIo >> -----END PGP SIGNATURE----- >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Thu Jan 30 17:27:01 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 16:27:01 +0000 (GMT) Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <52EA7362.9000006@trueblade.com> Message-ID: <1391099221.643.YahooMailBasic@web172406.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Eric V. Smith wrote: > The flag really needs to convey 2 meanings: > - - There are some C extensions here that can't be loaded > unless they live on a real filesystem path. > - - There is some code (maybe in this package, maybe in > another package that uses this one) that uses this > package's __file__, and that code won't work unless > __file__ points to a real filesystem path. Software can run from zip even if some things need filesystem access, so it's not as clear-cut as __file__ having to be a filesystem path. It's probably safe to assume that any C extension will need to be on a filesystem path, at least on the mainstream platforms, since that's how their run-time loaders work. For example, both pip and distil use cacert.pem to verify SSL hostnames, and the SSL module's API requires provision of a filesystem path for the .pem file. The solution is of course to use __file__ directly when running from the file system, and extracting cacert.pem to a cache location, and using that when running from zip. My view is that there are probably two different assertions: one is "will run from wheel" (I say wheel rather than zip, since wheel implies a place to put that metadata) and the other is "The C extensions are mandatory". If this flag is set, the wheel is asserted to be not worth mounting without the extensions. If clear, it means that C extensions, if present, are optional (for example, an accelerator). If there are extensions, there would need to be separate metadata giving the details so that extraction could happen (since we should refuse the temptation to guess by merely looking at file extensions of entries in the archive). Regards, Vinay Sajip From eugene at sazhin.us Thu Jan 30 17:33:19 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 11:33:19 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore wrote: > On 30 January 2014 05:33, Evgeny Sazhin wrote: >>>Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >>>they are both things that I've contemplated as things to bring forward in >>>using xz as an alternative compression format. Even if #1 would need a major >>>revision of Wheel to happen adding official "support" for zip import means that >>>the change would have to be weighed against also breaking that backwards >>>compatibility. >> >> >> Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. >> I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) >> I know, i'm probably alone in the desert...l;) > > This is the biggest concern I see with "promoting" wheels being > directly importable via zipimport (I say "promoting" and not > "specifying" deliberately, but I don't want to get back into the > process debate). > > People like Evgeny (no offense intended here, but your post is a good > example to use) coming from a Java background will see importable > wheels and *think* that they are similar in purpose to jars. No offense taken - i'm happy to be a use case to add some perspective;)) >This is > not, and never will be, true. Python is not Java, deployment by > putting jar-like files on sys.path is not an advisable workflow for > Python - no matter what parallels with Java might make it look > attractive. For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2: 1. It is impossible to load C extensions 2. Error handling is bad None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge): 1. Yes there is a group of "elite" interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops? 2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why? > > Pointing out the implication that wheels can be put on sys.path in > specialised circumstances is entirely reasonable when talking to a > Python audience, but it triggers entirely the wrong associations when > a person with a Java background sees the statement. > > Evgeny - for your purposes, you'd be better advised to see wheels as > similar in principle to Windows MSI installers (they aren't, but it > will avoid triggering incorrect analogies). That is an awesome analogy, thank you! > I don't have good > references or pointers to good Python development or deployment > practices, but you may want to ask around on python-list. And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there. Thanks, Eugene > (Be careful, > you may hear suggestions that you use eggs because they are similar to > jars - while that is true, the way Python works means that the > experience with eggs is far from being as natural as the jar > experience you're used to). > > Paul From donald at stufft.io Thu Jan 30 17:48:22 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 11:48:22 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> Message-ID: On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: >> >> On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: >> >>> >>> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: >>> >>>> >>>> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >>>> >>>>> >>>>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>>>> >>>>>> >>>>>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>>>>> >>>>>>>> >>>>>>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>>>>>> >>>>>>> >>>>>>> I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>>>>>> >>>>>>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>>>>> >>>>>> Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>>>> >>>>> I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to >>>> >>>> Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. >>> >>> I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. >>> I'm totally fine with the way pip handles things. >>> >>> again briefly My idea is to use the following: >>> >>> central location - flat folder with wheels, accessible to read for everybody in network. >>> >>> for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. >>> >>> To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: >>> PYTHONPATH=1.whl:2.whl; python 3.whl >>> >>> where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! >> >> Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. > > > I'm happy to concede the run from zip thing if somebody could explain > how should i use pip and virtualenv in production? > Currently it seems to be very clean and clear. I can have one folder > where i can dump multiple versions of the same project in wheel format > and having requirements.txt in each of them i can construct PYTHONPATH > and run things. Simple! > > How am i supposed to manage that using pip and virtual envs in production? The same way you?d use them in development? Hell I believe you can even do: $ virtualenv my_virtualenv $ my_virtualenv/bin/pip install path/to/wheelhouse/* The point is it?s just installed libraries, asking this question doesn?t really make much sense, it?s like asking how do you use apt-get or yum in production. > > Thanks, > Eugene ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 17:44:59 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 11:44:59 -0500 Subject: [Distutils] PEP 427 In-Reply-To: <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> Message-ID: On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: > >> >> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: >> >>> >>> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >>> >>>> >>>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>>> >>>>> >>>>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>>>> >>>>>>> >>>>>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>>>>> >>>>>> >>>>>> I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>>>>> >>>>>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>>>> >>>>> Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>>> >>>> I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to >>> >>> Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. >> >> I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. >> I'm totally fine with the way pip handles things. >> >> again briefly My idea is to use the following: >> >> central location - flat folder with wheels, accessible to read for everybody in network. >> >> for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. >> >> To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: >> PYTHONPATH=1.whl:2.whl; python 3.whl >> >> where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! > > Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. I'm happy to concede the run from zip thing if somebody could explain how should i use pip and virtualenv in production? Currently it seems to be very clean and clear. I can have one folder where i can dump multiple versions of the same project in wheel format and having requirements.txt in each of them i can construct PYTHONPATH and run things. Simple! How am i supposed to manage that using pip and virtual envs in production? Thanks, Eugene From donald at stufft.io Thu Jan 30 17:59:42 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 11:59:42 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore wrote: >> On 30 January 2014 05:33, Evgeny Sazhin wrote: >>>> Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >>>> they are both things that I've contemplated as things to bring forward in >>>> using xz as an alternative compression format. Even if #1 would need a major >>>> revision of Wheel to happen adding official "support" for zip import means that >>>> the change would have to be weighed against also breaking that backwards >>>> compatibility. >>> >>> >>> Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. >>> I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) >>> I know, i'm probably alone in the desert...l;) >> >> This is the biggest concern I see with "promoting" wheels being >> directly importable via zipimport (I say "promoting" and not >> "specifying" deliberately, but I don't want to get back into the >> process debate). >> >> People like Evgeny (no offense intended here, but your post is a good >> example to use) coming from a Java background will see importable >> wheels and *think* that they are similar in purpose to jars. > > No offense taken - i'm happy to be a use case to add some perspective;)) > >> This is >> not, and never will be, true. Python is not Java, deployment by >> putting jar-like files on sys.path is not an advisable workflow for >> Python - no matter what parallels with Java might make it look >> attractive. > > For pure python wheels - you're right - very attractive! And while i > hear you guys saying it is not advisable to use wheels in sys.path i'm > still far from seeing good reasoning. Please, correct me, but so far i > saw 2: > > 1. It is impossible to load C extensions > 2. Error handling is bad > > None of these reasons seems to be good enough to lead to a conclusion > that wheel used as jar-like thing is not needed. Here is how i see it > (again probably to my lack of knowledge): > > 1. Yes there is a group of "elite" interesting projects that are using > C extensions together with python code. What i don't understand is why > are wheels supposed to be tailored to the needs of the hybrid projects > instead of making pure python first priority? There is big crowd using > C extensions, but how much bigger is the crowd who doesn't? Why law > abiding pure python developer should jump through the hoops? Because C extensions are not ?elite?, they are pretty common. It?s somewhat rare in my experience to see a non trivial project that doesn?t have a C extension somewhere in it. > > 2. IF we are talking about error handling it seems that people > generally do not consider jar-like behavior to be bad for pure python > modules - It's just doesn't work very well (yet?). Is there really a > problem with error handling for pure python? What if the error > handling is improved? Is it still going to be bad? Why? i consider the fact that they?ll only work in a subset of situations bad, because it?s non obvious without digging through the code if any particular project will support working from a place other than a traditional install. This isn?t a new feature, Eggs have had this for a long time and when confronted with bugs caused by zipped Eggs the common solution is simply adding zip_safe=False to their project which disables zipped eggs for that project. The name for the kind of feature this is, is an ?attractive nuisance?. They look like something that might work, especially to people coming to Python from other languages such as Java, but if actually use the feature in a non trivial fashion you?re going to end up with more problems then you?ve solved when random projects simply don?t work and the likelihood that their authors have no desire to support running through a zipped file even if they *could* make it possible. > >> >> Pointing out the implication that wheels can be put on sys.path in >> specialised circumstances is entirely reasonable when talking to a >> Python audience, but it triggers entirely the wrong associations when >> a person with a Java background sees the statement. >> >> Evgeny - for your purposes, you'd be better advised to see wheels as >> similar in principle to Windows MSI installers (they aren't, but it >> will avoid triggering incorrect analogies). > > That is an awesome analogy, thank you! > >> I don't have good >> references or pointers to good Python development or deployment >> practices, but you may want to ask around on python-list. > > And that is my biggest concern (deployment). I believe these questions > should be answered by wheels or python packaging specification. > Not by third party tooling. The language should be able to work with > artifacts/packages produced for the language by the language. Third > party tools may allow for automation, continuous integration > complicated deployments and whatnot. But the base must be there. > > Thanks, > Eugene > >> (Be careful, >> you may hear suggestions that you use eggs because they are similar to >> jars - while that is true, the way Python works means that the >> experience with eggs is far from being as natural as the jar >> experience you're used to). >> >> Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Thu Jan 30 18:02:27 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 12:02:27 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: <9F9315F6-627A-4A9C-B2D1-45A1F8D22F78@stufft.io> On Jan 30, 2014, at 11:59 AM, Donald Stufft wrote: > Because C extensions are not ?elite?, they are pretty common. It?s somewhat > rare in my experience to see a non trivial project that doesn?t have a C > extension somewhere in it. To be clear, I mean project in the application sense here not in the individual thing that you?d upload to PyPI sense. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eugene at sazhin.us Thu Jan 30 18:23:26 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 12:23:26 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> Message-ID: On Thu, Jan 30, 2014 at 11:48 AM, Donald Stufft wrote: > > On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin wrote: > >> On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: >>> >>> On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: >>> >>>> >>>> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: >>>> >>>>> >>>>> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >>>>> >>>>>> >>>>>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>>>>> >>>>>>> >>>>>>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>>>>>> >>>>>>>>> >>>>>>>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>>>>>>> >>>>>>>> >>>>>>>> I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>>>>>>> >>>>>>>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>>>>>> >>>>>>> Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>>>>> >>>>>> I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to >>>>> >>>>> Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. >>>> >>>> I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. >>>> I'm totally fine with the way pip handles things. >>>> >>>> again briefly My idea is to use the following: >>>> >>>> central location - flat folder with wheels, accessible to read for everybody in network. >>>> >>>> for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. >>>> >>>> To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: >>>> PYTHONPATH=1.whl:2.whl; python 3.whl >>>> >>>> where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! >>> >>> Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. >> >> >> I'm happy to concede the run from zip thing if somebody could explain >> how should i use pip and virtualenv in production? >> Currently it seems to be very clean and clear. I can have one folder >> where i can dump multiple versions of the same project in wheel format >> and having requirements.txt in each of them i can construct PYTHONPATH >> and run things. Simple! >> >> How am i supposed to manage that using pip and virtual envs in production? > > The same way you'd use them in development? Hell I believe you can even do: > > $ virtualenv my_virtualenv > $ my_virtualenv/bin/pip install path/to/wheelhouse/* > > The point is it's just installed libraries, asking this question doesn't really make much > sense, it's like asking how do you use apt-get or yum in production. > >> Could you please reread this https://mail.python.org/pipermail/distutils-sig/2014-January/023636.html I'm explaining there what is the task at hand and why I'm favoring the jar-like workflow for deployment. Thanks, Eugnee From dholth at gmail.com Thu Jan 30 18:44:00 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 12:44:00 -0500 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> Message-ID: On Thu, Jan 30, 2014 at 12:23 PM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 11:48 AM, Donald Stufft wrote: >> >> On Jan 30, 2014, at 11:44 AM, Evgeny Sazhin wrote: >> >>> On Thu, Jan 30, 2014 at 9:38 AM, Donald Stufft wrote: >>>> >>>> On Jan 29, 2014, at 11:29 PM, Evgeny Sazhin wrote: >>>> >>>>> >>>>> On Jan 29, 2014, at 11:17 PM, Donald Stufft wrote: >>>>> >>>>>> >>>>>> On Jan 29, 2014, at 11:16 PM, Evgeny Sazhin wrote: >>>>>> >>>>>>> >>>>>>> On Jan 29, 2014, at 10:49 PM, Donald Stufft wrote: >>>>>>> >>>>>>>> >>>>>>>> On Jan 29, 2014, at 10:47 PM, Evgeny Sazhin wrote: >>>>>>>> >>>>>>>>>> >>>>>>>>>> Wheel is a package format. Packages are for transmitting and installing bits. If you want to make some kind of self-unpacking executable please do it with something built for it. makeself is an excellent choice for these. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I didn't say anything about self-unpacking executable. Egg already knows to do what is needed, so i was correct in expecting the wheel to do the same. Plus the notion of packages for transmitting and installing should not exclude the running and importing. Otherwise it is useless, at least for my purposes. As discussed before - jar does that just fine and it is a package format. >>>>>>>>> >>>>>>>>> Funny thing - wheel allows to do the same! Why would i want to use anything else then??? >>>>>>>> >>>>>>>> Because Python is not Java and Wheels are not Jars. You'll find very few packages actually support being run from a zipped file and the failure modes are not always obvious. >>>>>>> >>>>>>> I understand that not a lot of currently existing project are using this capability - but I'm 100% positive that if the running from wheel would be properly supported on error handling level and officially declared at least for the pure python - most of the people would be happy to have that! If we think about that, why would i want to use anything else other than wheel and pip if this pair gives the possibility to >>>>>> >>>>>> Pip does not install zipped wheels, and while it's not entirely up to me I would be opposed to it getting the ability to do so. >>>>> >>>>> I might be poorly wording things - but i never said I want pip to install the zipped wheel. It seems that you're missing the point a bit. >>>>> I'm totally fine with the way pip handles things. >>>>> >>>>> again briefly My idea is to use the following: >>>>> >>>>> central location - flat folder with wheels, accessible to read for everybody in network. >>>>> >>>>> for development : pip and virtual env. project has the virtual env created, dependencies are deployed and available for development and debugging in a standard manner. When done the project is packaged into wheel that is getting deployed to central location. >>>>> >>>>> To *run* the program: i would create a script that bases on the pip ability to resolve dependencies and basing on the requirements.txt from inside my wheel it would generate PYTHONPATH to prepend the starting call like that: >>>>> PYTHONPATH=1.whl:2.whl; python 3.whl >>>>> >>>>> where 3.whl is program with __main__.py and 1.whl and 2.whl are dependencies needed. This works as of now! >>>> >>>> Just use pip and virtualenv in production. It's bad form to install things differently in development than in production. It *will* lead to production only bugs and in the case of zip imports it'll lead to hard to diagnose errors and bugs that you'll never be able to reproduce in development. >>> >>> >>> I'm happy to concede the run from zip thing if somebody could explain >>> how should i use pip and virtualenv in production? >>> Currently it seems to be very clean and clear. I can have one folder >>> where i can dump multiple versions of the same project in wheel format >>> and having requirements.txt in each of them i can construct PYTHONPATH >>> and run things. Simple! >>> >>> How am i supposed to manage that using pip and virtual envs in production? >> >> The same way you'd use them in development? Hell I believe you can even do: >> >> $ virtualenv my_virtualenv >> $ my_virtualenv/bin/pip install path/to/wheelhouse/* >> >> The point is it's just installed libraries, asking this question doesn't really make much >> sense, it's like asking how do you use apt-get or yum in production. >> >>> > > Could you please reread this > https://mail.python.org/pipermail/distutils-sig/2014-January/023636.html > I'm explaining there what is the task at hand and why I'm favoring the > jar-like workflow for deployment. It's clear that you favor the jar-like workflow. You will have discover for yourself why it works or does not work for you. There is more mature existing jar-like support for eggs (which will continue to work for some time, but not be improved). We do actually need a better way to turn libraries on and off individually, for example, if I install the wrong set of Python libraries on Ubuntu the plugin system for somesuch package crashes; having everything installed importable all the time is a problem. Even if you don't consider the unrelated packages that have chosen the same module name. From vinay_sajip at yahoo.co.uk Thu Jan 30 18:41:17 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 17:41:17 +0000 (GMT) Subject: [Distutils] PEX - Twitter's multi-platform executable archive format for Python Message-ID: <1391103677.50462.YahooMailBasic@web172403.mail.ir2.yahoo.com> I searched this list for PEX and didn't find any hits, so I thought some people here might be interested in the PEX software from Twitter: https://www.youtube.com/watch?v=NmpnGhRwsu0 is a 15-minute presentation by Brian Wickman about it and other topics. It uses zipimport and is used to do packaging of Python software in Twitter across multiple platforms, so it could have some relevance to recent discussions we've been having about wheels, binary compatibility and the like. Brian has posted here (re. PEP 425 tags most recently) but not about PEX AFAICT. I don't yet know if there's anything particularly revolutionary about the tech (I haven't dug into it yet) but of course it's interesting because of Twitter's size and the usage in production. Regards, Vinay Sajip From eugene at sazhin.us Thu Jan 30 18:46:24 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 12:46:24 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 11:59 AM, Donald Stufft wrote: > > On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin wrote: > >> On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore wrote: >>> On 30 January 2014 05:33, Evgeny Sazhin wrote: >>>>> Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >>>>> they are both things that I've contemplated as things to bring forward in >>>>> using xz as an alternative compression format. Even if #1 would need a major >>>>> revision of Wheel to happen adding official "support" for zip import means that >>>>> the change would have to be weighed against also breaking that backwards >>>>> compatibility. >>>> >>>> >>>> Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. >>>> I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) >>>> I know, i'm probably alone in the desert...l;) >>> >>> This is the biggest concern I see with "promoting" wheels being >>> directly importable via zipimport (I say "promoting" and not >>> "specifying" deliberately, but I don't want to get back into the >>> process debate). >>> >>> People like Evgeny (no offense intended here, but your post is a good >>> example to use) coming from a Java background will see importable >>> wheels and *think* that they are similar in purpose to jars. >> >> No offense taken - i'm happy to be a use case to add some perspective;)) >> >>> This is >>> not, and never will be, true. Python is not Java, deployment by >>> putting jar-like files on sys.path is not an advisable workflow for >>> Python - no matter what parallels with Java might make it look >>> attractive. >> >> For pure python wheels - you're right - very attractive! And while i >> hear you guys saying it is not advisable to use wheels in sys.path i'm >> still far from seeing good reasoning. Please, correct me, but so far i >> saw 2: >> >> 1. It is impossible to load C extensions >> 2. Error handling is bad >> >> None of these reasons seems to be good enough to lead to a conclusion >> that wheel used as jar-like thing is not needed. Here is how i see it >> (again probably to my lack of knowledge): >> >> 1. Yes there is a group of "elite" interesting projects that are using >> C extensions together with python code. What i don't understand is why >> are wheels supposed to be tailored to the needs of the hybrid projects >> instead of making pure python first priority? There is big crowd using >> C extensions, but how much bigger is the crowd who doesn't? Why law >> abiding pure python developer should jump through the hoops? > > Because C extensions are not "elite", they are pretty common. It's somewhat > rare in my experience to see a non trivial project that doesn't have a C > extension somewhere in it. Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions? Is python such a bad language it cannot deal with simple stuff without C being involved? > >> >> 2. IF we are talking about error handling it seems that people >> generally do not consider jar-like behavior to be bad for pure python >> modules - It's just doesn't work very well (yet?). Is there really a >> problem with error handling for pure python? What if the error >> handling is improved? Is it still going to be bad? Why? > > i consider the fact that they'll only work in a subset of situations bad, > because it's non obvious without digging through the code if any > particular project will support working from a place other than > a traditional install. The necessity to dig through the code to understand if the project is runnable from zip is a responsibility of project provider - i.e. poor documentation. If the project is designed to be run that way the doc should say so. >This isn't a new feature, Eggs have had this for > a long time and when confronted with bugs caused by zipped Eggs > the common solution is simply adding zip_safe=False to their project > which disables zipped eggs for that project. > > The name for the kind of feature this is, is an "attractive nuisance". They > look like something that might work, especially to people coming to > Python from other languages such as Java, but if actually use the > feature in a non trivial fashion you're going to end up with more > problems then you've solved when random projects simply don't work > and the likelihood that their authors have no desire to support running > through a zipped file even if they *could* make it possible. OK, you're saying " if actually use the feature in a non trivial fashion" But was is non-trivial in putting pure python wheel in sys.path? Non-trivial would probably mean trying to load c extension from it, but that's obviously not supported, so i wouldn't even try. Thanks, Eugene From theller at ctypes.org Thu Jan 30 18:51:54 2014 From: theller at ctypes.org (Thomas Heller) Date: Thu, 30 Jan 2014 18:51:54 +0100 Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: Am 30.01.2014 14:23, schrieb Paul Moore: > > First of all, it is not possible to load a DLL into a process' memory > [2, 3] unless it is stored as a file in the filesystem. > [...] > > [2] I'm talking from a Windows perspective here. I do not have > sufficient low-level knowledge of Unix to comment on that case. I > suspect that the issues are similar but I defer to the platform > experts. > > [3] There is, I believe, code "out there" on the internet to map a DLL > image into a process based purely in memory, but I think it's a fairly > gross hack. I have a suspicion that someone - possibly Thomas Heller - > experimented with it at one time, but never came up with a viable > implementation. There's also TCL's tclkit technology, which *may* > support binary extensions, and may be worth a look, but TCL has > virtual filesystem support built in quite deep in the core, so how it > works may not be applicable to Python. Well, py2exe uses code that loads dlls from zip-files directly into process memory and executes it. This is used for 'single-file' applications that py2exe creates; it is triggered by the appropriate 'bundle_files' option. It works quite well for some applications (wx apps for example), but not for others. Applications that use numpy for example crashes, but the reason is that py2exe *also* puts the mkl-library (or maybe even more stuff like windows dlls) that numpy needs into the zip. So, the trick is to only load python-extensions and not arbitrary dlls from the zip. Changing the packing process so that py2exe leave these non-python dlls in the file system makes the exe work again. At least now the range of apps that work reliably is much larger. Anyway; the py2exe installer installs the python-extension and correspinding py-module that implements this magic, so anyone can experiment with it; it is called 'zipextimporter.py'. Here's the start of the docstring: r"""zipextimporter - an importer which can import extension modules from zipfiles This file and also _memimporter.pyd is part of the py2exe package. Overview ======== zipextimporter.py contains the ZipExtImporter class which allows to load Python binary extension modules contained in a zip.archive, without unpacking them to the file system. Call the zipextimporter.install() function to install the import hook, add a zip-file containing .pyd or .dll extension modules to sys.path, and import them. It uses the _memimporter extension which uses code from Joachim Bauch's MemoryModule library. This library emulates the win32 api function LoadLibrary. """ I wanted to post this to clarify the current state. Thomas From p.f.moore at gmail.com Thu Jan 30 18:56:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 17:56:49 +0000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On 30 January 2014 16:33, Evgeny Sazhin wrote: >> I don't have good >> references or pointers to good Python development or deployment >> practices, but you may want to ask around on python-list. > > And that is my biggest concern (deployment). I believe these questions > should be answered by wheels or python packaging specification. > Not by third party tooling. The language should be able to work with > artifacts/packages produced for the language by the language. Third > party tools may allow for automation, continuous integration > complicated deployments and whatnot. But the base must be there. There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment. Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing "pip install projectname" to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that. Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use "entry points" to define the executables - that will then be installed by the end user into an existing Python installation. As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to "roll your own" solution here. Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot. Cheers, Paul From p.f.moore at gmail.com Thu Jan 30 19:04:36 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 18:04:36 +0000 Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: On 30 January 2014 17:51, Thomas Heller wrote: > I wanted to post this to clarify the current state. Thanks. I had no idea (or I'd forgotten - one or the other!) that this was actually being done. Paul From ronaldoussoren at mac.com Thu Jan 30 19:14:52 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Jan 2014 19:14:52 +0100 Subject: [Distutils] platform specific python only wheels Message-ID: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> Hi, Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? In particular, I?m looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot ?cross-compile? application bundles. Ronald From eugene at sazhin.us Thu Jan 30 19:19:01 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 13:19:01 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 12:56 PM, Paul Moore wrote: > On 30 January 2014 16:33, Evgeny Sazhin wrote: >>> I don't have good >>> references or pointers to good Python development or deployment >>> practices, but you may want to ask around on python-list. >> >> And that is my biggest concern (deployment). I believe these questions >> should be answered by wheels or python packaging specification. >> Not by third party tooling. The language should be able to work with >> artifacts/packages produced for the language by the language. Third >> party tools may allow for automation, continuous integration >> complicated deployments and whatnot. But the base must be there. > > There's 2 distinct concepts in the Python world, that you should be > aware of. Application deployment and library deployment. > > Library deployment is simple. Build your library and upload it as a > source distribution (sdist) and one or more wheels to PyPI (if it's a > public project) or a local index (if it's a private project). People > or other projects will use the library however they deem appropriate, > typically by doing "pip install projectname" to make the code > accessible. That unpacks the wheel into a local site-packages, but you > have no need to care about that. > > Application deployment is not as straightforward - there are a number > of common approaches and at least as many opinions on what you should > do. You may bundle Python and your code and its dependencies together > into a standalone executable, using a tool like cx_Freeze. or you may > use something like buildout (I can't give details as I've never used > this). You can also, in some cases, package your application main code > in the same way as a library, and use "entry points" to define the > executables - that will then be installed by the end user into an > existing Python installation. > > As an application developer, you typically wouldn't care about how the > end user deployed your code for execution, unless you were bundling > via something like cx_Freeze. You might choose to bundle your > application code *without* a Python interpreter. That's basically what > executable zipfiles are about, and it's probably the closest > equivalent to Java jars. But that's not a particularly well-travelled > route in the Python community (yet) - there are projects like pyzzer > that automate certain aspects of that process, but in general, you > have to "roll your own" solution here. > > Hopefully that makes some sense - I'm getting pretty burned out with > the various threads going on at the moment, so I'm probably glossing > over a lot. > > Cheers, > Paul Thanks Paul - you are very helpful! this is good distinction and i think i'm coming to it as well. I just suggested one approach that might be feasible in my case in another thread;) Everybody - I'm really grateful for the time you are willing to spend in helping me to understand the state of the things. I hope this discussion might be helpful for both sides and may be people like me can be referenced to it in future;) PS That doesn't mean i will stop arguing with you;) Thanks, Eugene From dholth at gmail.com Thu Jan 30 19:19:43 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 13:19:43 -0500 Subject: [Distutils] platform specific python only wheels In-Reply-To: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> References: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> Message-ID: Your best bet currently is to execute the "mv" command to change the filename of the wheel. I think bdist_wheel should probably accept a setup() argument to set the tags. Currently you can just set a "universal" flag which means pure 2+3 python. On Thu, Jan 30, 2014 at 1:14 PM, Ronald Oussoren wrote: > Hi, > > Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? > > In particular, I'm looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot "cross-compile" application bundles. > > Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From noah at coderanger.net Thu Jan 30 19:21:21 2014 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 30 Jan 2014 10:21:21 -0800 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <1391072980.24027.YahooMailBasic@web172401.mail.ir2.yahoo.com> References: <1391072980.24027.YahooMailBasic@web172401.mail.ir2.yahoo.com> Message-ID: On Jan 30, 2014, at 1:09 AM, Vinay Sajip wrote: > > -------------------------------------------- > On Thu, 30/1/14, Ralf Gommers wrote: > >> Also end user. >> If, as a user, I want to use inplace builds and PYTHONPATH >> instead of virtualenvs for whatever reason, that should be >> supported. Setuptools inserting stuff to sys.path that come >> before PYTHONPATH entries is quite annoying. > > If tool developers want to offer end users the option to control how they work > with sys.path, that's up to them. For example, once the details are worked out, > the distil tool will probably get a --mountable option for the package command, > which will write metadata into the built wheel indicating whether the wheel is > addable to sys.path or not (based on the builder's knowledge of the wheel's > contents). Distlib, when asked to mount a wheel (add it to sys.path) will check > the mountability metadata and honour the wheel publisher's intent. > For everyone following along, the PEP has been updated. http://hg.python.org/peps/rev/26983acc9c11 If anyone has comments on the next text you can find it at http://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file I hope we can discuss further changes as a group before they are pushed live. --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: Message signed with OpenPGP using GPGMail URL: From vinay_sajip at yahoo.co.uk Thu Jan 30 19:23:51 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 18:23:51 +0000 (GMT) Subject: [Distutils] platform specific python only wheels In-Reply-To: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> Message-ID: <1391106231.52165.YahooMailBasic@web172402.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Ronald Oussoren wrote: > Is there a way to create a wheel that contains only python code, but > can only be installed on particular platforms? Perhaps not yet, but the idea is there in the 'supports_environments' metadata in PEP 426. This is currently supported by distil, but I don't believe pip does yet. Regards, Vinay Sajip From dholth at gmail.com Thu Jan 30 19:24:09 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 13:24:09 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: One thing that might be useful would be to develop the "unpacked wheel" which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. On Thu, Jan 30, 2014 at 1:19 PM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 12:56 PM, Paul Moore wrote: >> On 30 January 2014 16:33, Evgeny Sazhin wrote: >>>> I don't have good >>>> references or pointers to good Python development or deployment >>>> practices, but you may want to ask around on python-list. >>> >>> And that is my biggest concern (deployment). I believe these questions >>> should be answered by wheels or python packaging specification. >>> Not by third party tooling. The language should be able to work with >>> artifacts/packages produced for the language by the language. Third >>> party tools may allow for automation, continuous integration >>> complicated deployments and whatnot. But the base must be there. >> >> There's 2 distinct concepts in the Python world, that you should be >> aware of. Application deployment and library deployment. >> >> Library deployment is simple. Build your library and upload it as a >> source distribution (sdist) and one or more wheels to PyPI (if it's a >> public project) or a local index (if it's a private project). People >> or other projects will use the library however they deem appropriate, >> typically by doing "pip install projectname" to make the code >> accessible. That unpacks the wheel into a local site-packages, but you >> have no need to care about that. >> >> Application deployment is not as straightforward - there are a number >> of common approaches and at least as many opinions on what you should >> do. You may bundle Python and your code and its dependencies together >> into a standalone executable, using a tool like cx_Freeze. or you may >> use something like buildout (I can't give details as I've never used >> this). You can also, in some cases, package your application main code >> in the same way as a library, and use "entry points" to define the >> executables - that will then be installed by the end user into an >> existing Python installation. >> >> As an application developer, you typically wouldn't care about how the >> end user deployed your code for execution, unless you were bundling >> via something like cx_Freeze. You might choose to bundle your >> application code *without* a Python interpreter. That's basically what >> executable zipfiles are about, and it's probably the closest >> equivalent to Java jars. But that's not a particularly well-travelled >> route in the Python community (yet) - there are projects like pyzzer >> that automate certain aspects of that process, but in general, you >> have to "roll your own" solution here. >> >> Hopefully that makes some sense - I'm getting pretty burned out with >> the various threads going on at the moment, so I'm probably glossing >> over a lot. >> >> Cheers, >> Paul > > Thanks Paul - you are very helpful! this is good distinction and i > think i'm coming to it as well. I just suggested one approach that > might be feasible in my case in another thread;) > > Everybody - I'm really grateful for the time you are willing to spend > in helping me to understand the state of the things. I hope this > discussion might be helpful for both sides and may be people like me > can be referenced to it in future;) > > PS That doesn't mean i will stop arguing with you;) > > Thanks, > Eugene > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Thu Jan 30 19:26:36 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 18:26:36 +0000 (GMT) Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: Message-ID: <1391106396.10294.YahooMailBasic@web172405.mail.ir2.yahoo.com> On Thu, 30/1/14, Ronald Oussoren wrote: > It?s not just wheels, the same information can be used by application > bundlers like py2exe and py2app to recognise if a particular component > can be included in a zipfile. Fair point - I missed that use case. Regards, Vinay Sajip From ronaldoussoren at mac.com Thu Jan 30 19:29:58 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Jan 2014 19:29:58 +0100 Subject: [Distutils] platform specific python only wheels In-Reply-To: References: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> Message-ID: <9391843A-7DD9-4390-BDF3-FC62F8D1E4D1@mac.com> On 30 Jan 2014, at 19:19, Daniel Holth wrote: > Your best bet currently is to execute the "mv" command to change the > filename of the wheel. > > I think bdist_wheel should probably accept a setup() argument to set > the tags. Currently you can just set a "universal" flag which means > pure 2+3 python. There?s also ?plat-name argument for the setuptools command, but that doesn?t work. I?d have to rename the platform tag from ?any? to a list of supported platforms, is that correct? That would give a pretty convoluted filename, because the platform tag doesn?t allow for wildcards. That is, there is no way to specify ?any linux?, only something like ?linux_x86_64? (to borrow a tag from PEP 425). Ronald > > On Thu, Jan 30, 2014 at 1:14 PM, Ronald Oussoren wrote: >> Hi, >> >> Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? >> >> In particular, I'm looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot "cross-compile" application bundles. >> >> Ronald >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig From ronaldoussoren at mac.com Thu Jan 30 19:32:23 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Jan 2014 19:32:23 +0100 Subject: [Distutils] platform specific python only wheels In-Reply-To: <1391106231.52165.YahooMailBasic@web172402.mail.ir2.yahoo.com> References: <1391106231.52165.YahooMailBasic@web172402.mail.ir2.yahoo.com> Message-ID: On 30 Jan 2014, at 19:23, Vinay Sajip wrote: > > -------------------------------------------- > On Thu, 30/1/14, Ronald Oussoren wrote: > >> Is there a way to create a wheel that contains only python code, but >> can only be installed on particular platforms? > > Perhaps not yet, but the idea is there in the 'supports_environments' > metadata in PEP 426. This is currently supported by distil, but I don't > believe pip does yet. supports_environment would work for me. Ronald From dholth at gmail.com Thu Jan 30 19:36:45 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 13:36:45 -0500 Subject: [Distutils] platform specific python only wheels In-Reply-To: <9391843A-7DD9-4390-BDF3-FC62F8D1E4D1@mac.com> References: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> <9391843A-7DD9-4390-BDF3-FC62F8D1E4D1@mac.com> Message-ID: On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren wrote: > > On 30 Jan 2014, at 19:19, Daniel Holth wrote: > >> Your best bet currently is to execute the "mv" command to change the >> filename of the wheel. >> >> I think bdist_wheel should probably accept a setup() argument to set >> the tags. Currently you can just set a "universal" flag which means >> pure 2+3 python. > > There's also --plat-name argument for the setuptools command, but that doesn't work. > > I'd have to rename the platform tag from "any" to a list of supported platforms, is that correct? > > That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify "any linux", only something like "linux_x86_64" (to borrow a tag from PEP 425). My position has been that if there is no alternative that works on a different platform, then at least the wheel metadata should say nothing. It's only there to allow the installer to pick among the best of several alternatives. From ronaldoussoren at mac.com Thu Jan 30 19:40:18 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Jan 2014 19:40:18 +0100 Subject: [Distutils] platform specific python only wheels In-Reply-To: References: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> <9391843A-7DD9-4390-BDF3-FC62F8D1E4D1@mac.com> Message-ID: <18DDBED5-A2CA-40B3-8CC8-3C325CF3A542@mac.com> On 30 Jan 2014, at 19:36, Daniel Holth wrote: > On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren wrote: >> >> On 30 Jan 2014, at 19:19, Daniel Holth wrote: >> >>> Your best bet currently is to execute the "mv" command to change the >>> filename of the wheel. >>> >>> I think bdist_wheel should probably accept a setup() argument to set >>> the tags. Currently you can just set a "universal" flag which means >>> pure 2+3 python. >> >> There's also --plat-name argument for the setuptools command, but that doesn't work. >> >> I'd have to rename the platform tag from "any" to a list of supported platforms, is that correct? >> >> That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify "any linux", only something like "linux_x86_64" (to borrow a tag from PEP 425). > > My position has been that if there is no alternative that works on a > different platform, then at least the wheel metadata should say > nothing. It's only there to allow the installer to pick among the best > of several alternatives. The py2app setup.py file currently hardcodes a platform check because users did install py2app on other platforms (mostly Windows) and then complained that it didn?t work. It is much nicer to tell the user that a package won?t work during installation. Ronald From dholth at gmail.com Thu Jan 30 19:41:17 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 13:41:17 -0500 Subject: [Distutils] platform specific python only wheels In-Reply-To: <18DDBED5-A2CA-40B3-8CC8-3C325CF3A542@mac.com> References: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> <9391843A-7DD9-4390-BDF3-FC62F8D1E4D1@mac.com> <18DDBED5-A2CA-40B3-8CC8-3C325CF3A542@mac.com> Message-ID: You are probably a perfect customer for supports_environment On Thu, Jan 30, 2014 at 1:40 PM, Ronald Oussoren wrote: > > On 30 Jan 2014, at 19:36, Daniel Holth wrote: > >> On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren wrote: >>> >>> On 30 Jan 2014, at 19:19, Daniel Holth wrote: >>> >>>> Your best bet currently is to execute the "mv" command to change the >>>> filename of the wheel. >>>> >>>> I think bdist_wheel should probably accept a setup() argument to set >>>> the tags. Currently you can just set a "universal" flag which means >>>> pure 2+3 python. >>> >>> There's also --plat-name argument for the setuptools command, but that doesn't work. >>> >>> I'd have to rename the platform tag from "any" to a list of supported platforms, is that correct? >>> >>> That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify "any linux", only something like "linux_x86_64" (to borrow a tag from PEP 425). >> >> My position has been that if there is no alternative that works on a >> different platform, then at least the wheel metadata should say >> nothing. It's only there to allow the installer to pick among the best >> of several alternatives. > > The py2app setup.py file currently hardcodes a platform check because users did install py2app on other platforms (mostly Windows) and then complained that it didn't work. It is much nicer to tell the user that a package won't work during installation. > > Ronald From eugene at sazhin.us Thu Jan 30 20:15:53 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 14:15:53 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth wrote: > One thing that might be useful would be to develop the "unpacked > wheel" which is currently undefined but would be deliberately > identical to a site-packages with just one wheel extracted into it. > You wouldn't have to argue or worry about the zip issue. > > I like the way npm puts everything into a directory > ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you > really wanted to go to town you could figure out how to do virtualenvs > with hardlinks or reflinks instead of copies (conda can). > > But as has been repeated you won't find robust support for this in the > existing code. > I just tested it - it works! Yeah! I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse. After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl It worked! Is this workflow OK to rely on in future? Thanks, Eugene From ronaldoussoren at mac.com Thu Jan 30 19:19:49 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Jan 2014 19:19:49 +0100 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: <1391099221.643.YahooMailBasic@web172406.mail.ir2.yahoo.com> References: <1391099221.643.YahooMailBasic@web172406.mail.ir2.yahoo.com> Message-ID: On 30 Jan 2014, at 17:27, Vinay Sajip wrote: > > -------------------------------------------- > On Thu, 30/1/14, Eric V. Smith wrote: > >> The flag really needs to convey 2 meanings: >> - - There are some C extensions here that can't be loaded >> unless they live on a real filesystem path. >> - - There is some code (maybe in this package, maybe in >> another package that uses this one) that uses this >> package's __file__, and that code won't work unless >> __file__ points to a real filesystem path. > [?] > My view is that there are probably two different assertions: one is > "will run from wheel" (I say wheel rather than zip, since wheel > implies a place to put that metadata) and the other is "The C > extensions are mandatory?. It?s not just wheels, the same information can be used by application bundlers like py2exe and py2app to recognise if a particular component can be included in a zipfile. Py2app currently includes all python code in a zip file within an application bundle, unless a python package is on a manually maintained list of software that won?t work that way. Ronald From dholth at gmail.com Thu Jan 30 20:32:39 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 14:32:39 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth wrote: >> One thing that might be useful would be to develop the "unpacked >> wheel" which is currently undefined but would be deliberately >> identical to a site-packages with just one wheel extracted into it. >> You wouldn't have to argue or worry about the zip issue. >> >> I like the way npm puts everything into a directory >> ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you >> really wanted to go to town you could figure out how to do virtualenvs >> with hardlinks or reflinks instead of copies (conda can). >> >> But as has been repeated you won't find robust support for this in the >> existing code. >> > > I just tested it - it works! Yeah! > > I put an unzipped wheel of my API project into a local wheelhouse - > pip was able to install from it into my virt env for the client > project. > Then i created a wheel with __main__.py for the client project and put > an unzipped version in wheelhouse. > > After that i issued: > PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl > > It worked! > > Is this workflow OK to rely on in future? > > Thanks, > Eugene The only request is that you don't name the directory ".whl" because that is a little bit confusing. "Directories with only one package installed into them" will continue to work. From eugene at sazhin.us Thu Jan 30 20:52:09 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 14:52:09 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 2:32 PM, Daniel Holth wrote: > On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin wrote: >> On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth wrote: >>> One thing that might be useful would be to develop the "unpacked >>> wheel" which is currently undefined but would be deliberately >>> identical to a site-packages with just one wheel extracted into it. >>> You wouldn't have to argue or worry about the zip issue. >>> >>> I like the way npm puts everything into a directory >>> ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you >>> really wanted to go to town you could figure out how to do virtualenvs >>> with hardlinks or reflinks instead of copies (conda can). >>> >>> But as has been repeated you won't find robust support for this in the >>> existing code. >>> >> >> I just tested it - it works! Yeah! >> >> I put an unzipped wheel of my API project into a local wheelhouse - >> pip was able to install from it into my virt env for the client >> project. >> Then i created a wheel with __main__.py for the client project and put >> an unzipped version in wheelhouse. >> >> After that i issued: >> PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl >> >> It worked! >> >> Is this workflow OK to rely on in future? >> >> Thanks, >> Eugene > > The only request is that you don't name the directory ".whl" because > that is a little bit confusing. > > "Directories with only one package installed into them" will continue to work. pip can't find the distribution then - so it must have the name totally matching wheel spec, i guess. From brett at python.org Thu Jan 30 20:52:12 2014 From: brett at python.org (Brett Cannon) Date: Thu, 30 Jan 2014 14:52:12 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 12:46 PM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 11:59 AM, Donald Stufft wrote: > > > > On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin wrote: > > > >> On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore > wrote: > >>> On 30 January 2014 05:33, Evgeny Sazhin wrote: > >>>>> Eh, I think both 1 and 3 are things that are possibly reasonable to > happen and > >>>>> they are both things that I've contemplated as things to bring > forward in > >>>>> using xz as an alternative compression format. Even if #1 would need > a major > >>>>> revision of Wheel to happen adding official "support" for zip import > means that > >>>>> the change would have to be weighed against also breaking that > backwards > >>>>> compatibility. > >>>> > >>>> > >>>> Eh, please don't break it!! Improve! Keep the ability to add wheels > to python path. > >>>> I don't care how, zip import, non-zip import, whatever - allow for > jar-like behavior - people will thank you if it will work properly !;) > >>>> I know, i'm probably alone in the desert...l;) > >>> > >>> This is the biggest concern I see with "promoting" wheels being > >>> directly importable via zipimport (I say "promoting" and not > >>> "specifying" deliberately, but I don't want to get back into the > >>> process debate). > >>> > >>> People like Evgeny (no offense intended here, but your post is a good > >>> example to use) coming from a Java background will see importable > >>> wheels and *think* that they are similar in purpose to jars. > >> > >> No offense taken - i'm happy to be a use case to add some perspective;)) > >> > >>> This is > >>> not, and never will be, true. Python is not Java, deployment by > >>> putting jar-like files on sys.path is not an advisable workflow for > >>> Python - no matter what parallels with Java might make it look > >>> attractive. > >> > >> For pure python wheels - you're right - very attractive! And while i > >> hear you guys saying it is not advisable to use wheels in sys.path i'm > >> still far from seeing good reasoning. Please, correct me, but so far i > >> saw 2: > >> > >> 1. It is impossible to load C extensions > >> 2. Error handling is bad > >> > >> None of these reasons seems to be good enough to lead to a conclusion > >> that wheel used as jar-like thing is not needed. Here is how i see it > >> (again probably to my lack of knowledge): > >> > >> 1. Yes there is a group of "elite" interesting projects that are using > >> C extensions together with python code. What i don't understand is why > >> are wheels supposed to be tailored to the needs of the hybrid projects > >> instead of making pure python first priority? There is big crowd using > >> C extensions, but how much bigger is the crowd who doesn't? Why law > >> abiding pure python developer should jump through the hoops? > > > > Because C extensions are not "elite", they are pretty common. It's > somewhat > > rare in my experience to see a non trivial project that doesn't have a C > > extension somewhere in it. > > Isn't this a self contradictory statement? C extensions are quite > common for non-trivial project. non-trivial is not common. > Doesn't it mean that C extensions are rarely used for common trivial > stuff? You also implying that there were non-trivial projects that > didn't need C extensions. Or are you trying to make an argument that > the overall majority of python projects have C extensions? He's trying to argue that enough projects use C extensions that to have special support for projects that don't use them is supporting a special case. The Zen of Python says "Special cases aren't special enough to break the rules" and explicitly having wheels support direct execution only when they contain pure Python is supporting a special case. Obviously you can argue over at what point something stops being a "special case", but considering venvs isolate code well and are the officially supported way to isolate code then that makes stuff that goes passed what venv can do a special case. > Is python > such a bad language it cannot deal with simple stuff without C being > involved? > The use of a C extension does not necessarily mean a shortcoming of Python. People can write performance-critical code in C so as to get that last bit of speed in heavily executed code in a tight loop, etc. It has nothing to do with Python not being able to handle something and more that C code can simply be faster when coded correctly; it's a feature and not a band-aid. -Brett > > > > > >> > >> 2. IF we are talking about error handling it seems that people > >> generally do not consider jar-like behavior to be bad for pure python > >> modules - It's just doesn't work very well (yet?). Is there really a > >> problem with error handling for pure python? What if the error > >> handling is improved? Is it still going to be bad? Why? > > > > i consider the fact that they'll only work in a subset of situations bad, > > because it's non obvious without digging through the code if any > > particular project will support working from a place other than > > a traditional install. > > The necessity to dig through the code to understand if the project is > runnable from zip is a responsibility of project provider - i.e. poor > documentation. > If the project is designed to be run that way the doc should say so. > > > >This isn't a new feature, Eggs have had this for > > a long time and when confronted with bugs caused by zipped Eggs > > the common solution is simply adding zip_safe=False to their project > > which disables zipped eggs for that project. > > > > The name for the kind of feature this is, is an "attractive nuisance". > They > > look like something that might work, especially to people coming to > > Python from other languages such as Java, but if actually use the > > feature in a non trivial fashion you're going to end up with more > > problems then you've solved when random projects simply don't work > > and the likelihood that their authors have no desire to support running > > through a zipped file even if they *could* make it possible. > > OK, you're saying " if actually use the feature in a non trivial fashion" > But was is non-trivial in putting pure python wheel in sys.path? > Non-trivial would probably > mean trying to load c extension from it, but that's obviously not > supported, > so i wouldn't even try. > > > Thanks, > Eugene > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugene at sazhin.us Thu Jan 30 21:10:29 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 15:10:29 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 2:52 PM, Brett Cannon wrote: >> >> still far from seeing good reasoning. Please, correct me, but so far i >> >> saw 2: >> >> >> >> 1. It is impossible to load C extensions >> >> 2. Error handling is bad >> >> >> >> None of these reasons seems to be good enough to lead to a conclusion >> >> that wheel used as jar-like thing is not needed. Here is how i see it >> >> (again probably to my lack of knowledge): >> >> >> >> 1. Yes there is a group of "elite" interesting projects that are using >> >> C extensions together with python code. What i don't understand is why >> >> are wheels supposed to be tailored to the needs of the hybrid projects >> >> instead of making pure python first priority? There is big crowd using >> >> C extensions, but how much bigger is the crowd who doesn't? Why law >> >> abiding pure python developer should jump through the hoops? >> > >> > Because C extensions are not "elite", they are pretty common. It's >> > somewhat >> > rare in my experience to see a non trivial project that doesn't have a C >> > extension somewhere in it. >> >> Isn't this a self contradictory statement? C extensions are quite >> common for non-trivial project. non-trivial is not common. >> Doesn't it mean that C extensions are rarely used for common trivial >> stuff? You also implying that there were non-trivial projects that >> didn't need C extensions. Or are you trying to make an argument that >> the overall majority of python projects have C extensions? > > > He's trying to argue that enough projects use C extensions that to have > special support for projects that don't use them is supporting a special > case. The Zen of Python says "Special cases aren't special enough to break > the rules" and explicitly having wheels support direct execution only when > they contain pure Python is supporting a special case. Obviously you can > argue over at what point something stops being a "special case", but > considering venvs isolate code well and are the officially supported way to > isolate code then that makes stuff that goes passed what venv can do a > special case. yes, you got it absolutely right - I'm coming from the point of view that the C extension is a special case. It's even called extension! And pure python is never special case, therefore should take priority. > >> >> Is python >> such a bad language it cannot deal with simple stuff without C being >> involved? > > > The use of a C extension does not necessarily mean a shortcoming of Python. > People can write performance-critical code in C so as to get that last bit > of speed in heavily executed code in a tight loop, etc. It has nothing to do > with Python not being able to handle something and more that C code can > simply be faster when coded correctly; it's a feature and not a band-aid. > > -Brett > I understand that, i was sarcastic;) But thank you! Thanks, Eugene From eugene at sazhin.us Thu Jan 30 21:12:38 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Thu, 30 Jan 2014 15:12:38 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: On Thu, Jan 30, 2014 at 3:04 PM, Daniel Holth wrote: > Wow, it really can install from a directory-named-like-a-wheel with > the --no-index flag passed. I'm sad to say that is totally > unintentional, but bizarrely interesting. Can you guys promiss at least not to break this one?!;) Thanks, Eugene From dholth at gmail.com Thu Jan 30 21:04:39 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 30 Jan 2014 15:04:39 -0500 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <16E37763-B486-45A4-8916-1B25D7FD04F4@sazhin.us> Message-ID: Wow, it really can install from a directory-named-like-a-wheel with the --no-index flag passed. I'm sad to say that is totally unintentional, but bizarrely interesting. On Thu, Jan 30, 2014 at 2:52 PM, Evgeny Sazhin wrote: > On Thu, Jan 30, 2014 at 2:32 PM, Daniel Holth wrote: >> On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin wrote: >>> On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth wrote: >>>> One thing that might be useful would be to develop the "unpacked >>>> wheel" which is currently undefined but would be deliberately >>>> identical to a site-packages with just one wheel extracted into it. >>>> You wouldn't have to argue or worry about the zip issue. >>>> >>>> I like the way npm puts everything into a directory >>>> ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you >>>> really wanted to go to town you could figure out how to do virtualenvs >>>> with hardlinks or reflinks instead of copies (conda can). >>>> >>>> But as has been repeated you won't find robust support for this in the >>>> existing code. >>>> >>> >>> I just tested it - it works! Yeah! >>> >>> I put an unzipped wheel of my API project into a local wheelhouse - >>> pip was able to install from it into my virt env for the client >>> project. >>> Then i created a wheel with __main__.py for the client project and put >>> an unzipped version in wheelhouse. >>> >>> After that i issued: >>> PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl >>> >>> It worked! >>> >>> Is this workflow OK to rely on in future? >>> >>> Thanks, >>> Eugene >> >> The only request is that you don't name the directory ".whl" because >> that is a little bit confusing. >> >> "Directories with only one package installed into them" will continue to work. > > > pip can't find the distribution then - so it must have the name > totally matching wheel spec, i guess. From vinay_sajip at yahoo.co.uk Thu Jan 30 21:29:38 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jan 2014 20:29:38 +0000 (GMT) Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: Message-ID: <1391113778.8125.YahooMailBasic@web172403.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Daniel Holth wrote: > Wow, it really can install from a directory-named-like-a-wheel with > the --no-index flag passed. I'm sad to say that is totally unintentional, > but bizarrely interesting. What, *more* unintentional drive-by behaviour? Now, where's that pitchfork? ;-) Regards, Vinay Sajip From donald at stufft.io Thu Jan 30 22:11:34 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 30 Jan 2014 16:11:34 -0500 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: References: <1391072980.24027.YahooMailBasic@web172401.mail.ir2.yahoo.com> Message-ID: <789F6D0A-95D8-4E00-9BC1-82FFEE27F735@stufft.io> On Jan 30, 2014, at 1:21 PM, Noah Kantrowitz wrote: > > On Jan 30, 2014, at 1:09 AM, Vinay Sajip wrote: > >> >> -------------------------------------------- >> On Thu, 30/1/14, Ralf Gommers wrote: >> >>> Also end user. >>> If, as a user, I want to use inplace builds and PYTHONPATH >>> instead of virtualenvs for whatever reason, that should be >>> supported. Setuptools inserting stuff to sys.path that come >>> before PYTHONPATH entries is quite annoying. >> >> If tool developers want to offer end users the option to control how they work >> with sys.path, that's up to them. For example, once the details are worked out, >> the distil tool will probably get a --mountable option for the package command, >> which will write metadata into the built wheel indicating whether the wheel is >> addable to sys.path or not (based on the builder's knowledge of the wheel's >> contents). Distlib, when asked to mount a wheel (add it to sys.path) will check >> the mountability metadata and honour the wheel publisher's intent. >> > > For everyone following along, the PEP has been updated. http://hg.python.org/peps/rev/26983acc9c11 > > If anyone has comments on the next text you can find it at http://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file > > I hope we can discuss further changes as a group before they are pushed live. > > --Noah > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig The updated text is fine with me as it at least accurately documents the fact that using that feature is fraught with peril and reads more of a documentation of the behavior than anything else. So thanks for the final (I hope?) update Nick, and sorry this thread got all ragey. On the plus side this spurred me into trying to put the thoughts I?ve had for a newer Wheel format down into a PEP and I think it?s turning out well so far :] ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Thu Jan 30 22:52:47 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Jan 2014 07:52:47 +1000 Subject: [Distutils] wheels on sys.path clarification (reboot) In-Reply-To: References: <57513552-1BA8-44D7-8809-FD00F4DA7750@stufft.io> <38D0D557-D16A-4166-8503-326CF43B8B8E@stufft.io> <52EA7362.9000006@trueblade.com> Message-ID: On 31 Jan 2014 02:22, "Donald Stufft" wrote: > > Just an FYI, I've begun compiling notes and plan on proposing PEPs that will be > aimed at supplanting PEP 425 and PEP 427 with the goal of nailing down > undefined behavior, including missing functionality, and removing misfeatures. Please file issues on the pypi-metadata-formats repo, that's one of the things it is for. I already created one requesting a flag to indicate whether or not a package supports meta-imports: https://bitbucket.org/pypa/pypi-metadata-formats/issue/17/add-wheel-metadata-to-tag-meta-import (along with one for the linux distro tagging and a more controversial one to support more flexibility in the final installation layout) Cheers, Nick. > > On Jan 30, 2014, at 10:56 AM, Daniel Holth wrote: > > > I see you've noticed wheel was released in an imperfect state. > > > > Let's add a Zip-Safe flag to the WHEEL file, values of "true" and > > "false" same as Root-Is-Purelib. > > > > Daniel > > > > On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> On 01/30/2014 10:23 AM, Donald Stufft wrote: > >>> > >>> On Jan 30, 2014, at 10:12 AM, Paul Moore > >>> wrote: > >>> > >>>> On 30 January 2014 15:03, Donald Stufft > >>>> wrote: > >>>>> For those following along at home who have no interest in > >>>>> supporting zipped Wheels who want to disclaim support for it, > >>>>> here's how you do zip_safe=False for Wheels: > >>>>> https://gist.github.com/dstufft/8710270. > >>>> > >>>> Note that this will also prevent use in py2exe, cx_Freeze and > >>>> any similar bundling technologies. Presumably that's what you > >>>> want, as any issues are identical, but it's worth being clear. > >>>> > >>>> Paul > >>> > >>> Sure, using a library via zip import when it wasn't designed to be > >>> used as such is an anti pattern. > >> > >> The flag really needs to convey 2 meanings: > >> - - There are some C extensions here that can't be loaded unless they > >> live on a real filesystem path. > >> - - There is some code (maybe in this package, maybe in another package > >> that uses this one) that uses this package's __file__, and that code > >> won't work unless __file__ points to a real filesystem path. > >> > >> There are any number of possible importers that would fail due to > >> these 2 cases. I've personally written one that loads modules from a > >> database. It should respect this flag, too. > >> > >> Which is a long way of saying: I think calling it "zip_safe" is a > >> misnomer. The flag is really conveying "I don't need to be on a > >> filesystem". > >> > >> Whether there should be 2 different flags for the two different > >> problems (C extension and __file__), I can't say. They do seem to go > >> hand-in-hand. > >> > >> Eric. > >> > >> > >> -----BEGIN PGP SIGNATURE----- > >> Version: GnuPG v1.4.14 (GNU/Linux) > >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > >> > >> iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 > >> N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO > >> l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv > >> a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD > >> /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN > >> 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= > >> =6tIo > >> -----END PGP SIGNATURE----- > >> _______________________________________________ > >> Distutils-SIG maillist - Distutils-SIG at python.org > >> https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 30 22:58:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Jan 2014 07:58:19 +1000 Subject: [Distutils] PEP 427 In-Reply-To: References: <1391004671.36098.YahooMailBasic@web172405.mail.ir2.yahoo.com> <1B74B869-598E-4A96-B296-075AFB57F50B@sazhin.us> <6A7A5913-ADAC-4485-8EE3-4A908F2AD2E7@stufft.io> Message-ID: On 31 Jan 2014 02:48, "Donald Stufft" wrote: > > > > How am i supposed to manage that using pip and virtual envs in production? > > The same way you'd use them in development? Hell I believe you can even do: > > $ virtualenv my_virtualenv > $ my_virtualenv/bin/pip install path/to/wheelhouse/* > > The point is it's just installed libraries, asking this question doesn't really make much > sense, it's like asking how do you use apt-get or yum in production. Most Java apps don't typically use those either, they just drop jar files in a directory on CLASSPATH :P (Hence why I only *tolerate* that approach for wheels for advanced use cases that can't easily be handled any other way, while strongly discouraging it in general) Cheers, Nick. > > > > > Thanks, > > Eugene > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 30 23:30:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Jan 2014 08:30:14 +1000 Subject: [Distutils] Using Wheel with zipimport In-Reply-To: <789F6D0A-95D8-4E00-9BC1-82FFEE27F735@stufft.io> References: <1391072980.24027.YahooMailBasic@web172401.mail.ir2.yahoo.com> <789F6D0A-95D8-4E00-9BC1-82FFEE27F735@stufft.io> Message-ID: Thanks for pointing that out Noah - I was planning to come back and check if that wording was considered more acceptable. On 31 Jan 2014 07:12, "Donald Stufft" wrote: > > The updated text is fine with me as it at least accurately documents the fact that using that > feature is fraught with peril and reads more of a documentation of the behavior than > anything else. Aye, clearly explaining "yes, you *can*, but that doesn't mean you *should*" was always my primary goal with the new FAQ entry. Until you commented on the original update, I didn't realise there might be a perception that this egg-replacement use case didn't need to be taken into account in designing future iterations of the wheel format, and it was only after I went back and watched the PyCon 2013 panel video that I realised we had likely run afoul of the "conference curse" where a shared offline understanding hadn't been properly captured in the online materials (it's also possible Daniel and I discussed it privately back before the wheel spec became a PEP, but I didn't dig back that far into my email archives). > > So thanks for the final (I hope?) update Nick, and sorry this thread got all ragey. Hah, if you want to see ragey, look up the epic PEP 343 threads between me and PJE before Guido broke the deadlock by proposing to just remove the cause of the argument entirely (the implicit dynamic context manager support that was in the originally accepted with statement design turned out to be thoroughly confusing because it made it too hard to come up with consistent terminology for the new concepts involved). I know all too well how disconcerting it can be to find that someone else has a significantly different interpretation of a spec that appeared to be crystal clear (especially a case like this where the written spec *was* clear, but missing a critical detail). I'm also nowhere near hypocritical enough to berate anyone *else* for passionately objecting to a course of action that appears to lead places we don't want to go :) > On the plus side this spurred me into trying to put the thoughts I've had for a newer Wheel > format down into a PEP and I think it's turning out well so far :] Huzzah! Turning caremad into PEPs can be surprisingly cathartic (cf. PEP 462). Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 30 23:38:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Jan 2014 08:38:18 +1000 Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: On 30 Jan 2014 23:26, "Paul Moore" wrote: > > On 30 January 2014 12:29, Nick Coghlan wrote: > > I actually think this is a useful thing to experiment with, I'm just > > not sure distlib is the best place for that experiment. With > > appropriately secure tempfile handling and the right sys.path (and > > module __path__) manipulation it's not obviously *impossible* to > > handle C extensions at arbitrary positions in the module namespace > > this way, just difficult. zipimport itself is a bad place to > > experiment though, since not only is it currently a complex ball of C > > code, but adding such a feature without clear evidence of robust > > support in a third party project would be irresponsible. > > I just sent a long message that essentially gave a chunk of history > and suggested a similar thing - an "enhanced zipimport" module to > experiment with solutions in this space. With importlib available, an > experimental implementation shouldn't even be too hard. > > The only difference is that I see very little reason why such a > solution can't apply to all zipfiles, and not just wheels. The advantage of wheels over plain zipfiles for this use case is the structured metadata. distlib.mount doesn't try to guess the package structure for the extensions, you have to provide an EXTENSIONS file in the metadata that explains what C extensions are present and how they should map into the module namespace. Cheers, Nick. > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 30 23:51:50 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jan 2014 22:51:50 +0000 Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: On 30 January 2014 22:38, Nick Coghlan wrote: > The advantage of wheels over plain zipfiles for this use case is the > structured metadata. distlib.mount doesn't try to guess the package > structure for the extensions, you have to provide an EXTENSIONS file in the > metadata that explains what C extensions are present and how they should map > into the module namespace. OK, I think I get the idea now. I'm still not comfortable with the temp directory clutter that unpacking leaves (in particular on Windows where deletion isn't even possible in an atexit routine) but I'll survive. I *would* like to see the various technical issues and implications in the API documentation, though. The implications and limitations, and in particular the manual cache management requirements, need to be made explicit. (I thought I'd seen docs somewhere, but they definitely aren't in the API reference for the distlib.wheel module). Paul From vinay_sajip at yahoo.co.uk Fri Jan 31 07:16:22 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 Jan 2014 06:16:22 +0000 (GMT) Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) In-Reply-To: Message-ID: <1391148982.27752.YahooMailBasic@web172403.mail.ir2.yahoo.com> -------------------------------------------- On Thu, 30/1/14, Paul Moore wrote: > I'm still not comfortable with the temp directory clutter that unpacking > leaves (in particular on Windows where deletion isn't even possible > in an atexit routine) but I'll survive. It's up to the using application to do cache management of this type. It's not as if the cache will grow unbounded in most realistic scenarios, and perfectly possible to do cleanup during startup. It may not be ideal, but seems acceptable given the limitations of the underlying platform. > I *would* like to see the various technical issues and implications in > the API documentation, though. The implications and limitations, and > in particular the manual cache management requirements, need to be > made explicit. (I thought I'd seen docs somewhere, but they definitely > aren't in the API reference for the distlib.wheel module). There's some mention in the distlib.resources documentation (the cache would also be used for things like cacert.pem), but I'll certainly update the wheel documentation to cover this area, and update the relevant parts of the distlib.resources and distlib.util documentation, too. BTW, I raised the whole issue of extracting C extensions for import from zips on python-dev in March last year: https://mail.python.org/pipermail/python-dev/2013-March/124970.html The feedback I got indicated (to me) that while people felt there were some problem areas (e.g. cleanup was mentioned), there were no show-stoppers. No ringing endorsements, but no red flags were raised either. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Jan 31 09:22:13 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 Jan 2014 08:22:13 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib=2Emount_API_design_=28was=3A_wheels?= =?utf-8?q?_on_sys=2Epath=09clarification_=28reboot=29=29?= References: Message-ID: Paul Moore gmail.com> writes: > I *would* like to see the various technical issues and implications in > the API documentation, though. The implications and limitations, and > in particular the manual cache management requirements, need to be > made explicit. I've updated the documentation, see http://distlib.readthedocs.org/en/latest/reference.html The docs on distlib.util.get_cache_base have the info on the cache and cleanup thereof, including handling of no-home-directory, security implications, open files in Windows etc. I've also updated the distlib.resources.Cache documentation to link to get_cache_base. I've added is_mountable and is_compatible methods to Wheel, and added/ updated the docs for them and the mount() method accordingly, and referenced get_cache_base as the place where extensions are extracted to. More detailed suggestions for improvements are welcome. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Jan 31 09:35:00 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 Jan 2014 08:35:00 +0000 (UTC) Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) References: Message-ID: Thomas Heller ctypes.org> writes: > It uses the _memimporter extension which uses code from Joachim > Bauch's MemoryModule library. This library emulates the win32 api > function LoadLibrary. When this was mentioned on python-dev last March you said it was 32-bit and Python 2.x only, is that still the case? Regards, Vinay Sajip From p.f.moore at gmail.com Fri Jan 31 10:04:55 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 31 Jan 2014 09:04:55 +0000 Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: On 31 January 2014 08:22, Vinay Sajip wrote: > More detailed suggestions for improvements are welcome. Thanks for this. I see no means to set the cache that will be used by wheel.mount. Are you meant to set the global distlib.resources.cache value to a user-created Cache instance if you want to control the location of the cache? Is the default cache created on demand? In other words, if I set up my own cache on application startup, will the %LOCALAPPDATA%\.distlib directory still get created? The reason I ask is that on Windows, there are people who are interested in "portable" applications, which have as near to zero impact on the system as a whole as possible. While I don't have a need to write such applications in Python right now, I am working a lot with such applications at the moment, and I'm more aware than usual of questions such as "does this app leave traces of its presence on the system?" So I don't have an agenda here, I'm just curious. Paul From vinay_sajip at yahoo.co.uk Fri Jan 31 10:38:41 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 Jan 2014 09:38:41 +0000 (GMT) Subject: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot)) In-Reply-To: Message-ID: <1391161121.97474.YahooMailBasic@web172402.mail.ir2.yahoo.com> On Fri, 31/1/14, Paul Moore wrote: > Thanks for this. I see no means to set the cache that will be used by > wheel.mount. It's not as configurable as the distlib.resources cache (needs a method to be overridden, which isn't ideal), but I'll look at making it follow the same scheme. > Are you meant to set the global distlib.resources.cache value to a user- > created Cache instance if you want to control the location of the cache? Yes. > Is the default cache created on demand? In other words, if I set up my > own cache on application startup, will the %LOCALAPPDATA%\.distlib > directory still get created? No - currently it's created on module import by a line 'cache = Cache()'. This can easily be changed to defer the cache creation until it's needed, allowing user code to set a custom Cache. When I do this, no .distlib directory will be created in %LOCALAPPDATA% unless no other cache has been set, and it's needed. > So I don't have an agenda here, I'm just curious. I hope that answers things! I will update distlib's cache usage logic shortly to (a) allow better control over the wheel mount cache location and (b) create caches on demand rather than on module import. Regards, Vinay Sajip From theller at ctypes.org Fri Jan 31 11:33:32 2014 From: theller at ctypes.org (Thomas Heller) Date: Fri, 31 Jan 2014 11:33:32 +0100 Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) In-Reply-To: References: Message-ID: Am 31.01.2014 09:35, schrieb Vinay Sajip: > Thomas Heller ctypes.org> writes: > >> It uses the _memimporter extension which uses code from Joachim >> Bauch's MemoryModule library. This library emulates the win32 api >> function LoadLibrary. > > When this was mentioned on python-dev last March you said it was 32-bit and > Python 2.x only, is that still the case? This limitation does no longer exist; it works in 2.x and 3.x as well as on 32-bit and 64-bit. Although the code is not yet released as wheel/bdist_wininst/egg or whatever. Thomas From vinay_sajip at yahoo.co.uk Fri Jan 31 11:57:22 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 Jan 2014 10:57:22 +0000 (GMT) Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) In-Reply-To: Message-ID: <1391165842.44479.YahooMailBasic@web172401.mail.ir2.yahoo.com> On Fri, 31/1/14, Thomas Heller wrote: > This limitation does no longer exist; it works in 2.x and 3.x as > well as on 32-bit and 64-bit. That's good to know. Of course, as it's platform specific, it can't be used as a generic C extension import facility, though I suppose that if/when such a capability becomes available on mainstream platforms, the memimporter API could be adapted to cover it. Regards, Vinay Sajip From ncoghlan at gmail.com Fri Jan 31 12:13:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Jan 2014 21:13:46 +1000 Subject: [Distutils] platform specific python only wheels In-Reply-To: References: <5C1E40B1-CE12-404E-8FBE-2EE5A42B6D9C@mac.com> <9391843A-7DD9-4390-BDF3-FC62F8D1E4D1@mac.com> <18DDBED5-A2CA-40B3-8CC8-3C325CF3A542@mac.com> Message-ID: On 31 Jan 2014 04:41, "Daniel Holth" wrote: > > You are probably a perfect customer for supports_environment It occurs to me that if we split the hooks stuff out of PEP 359, we'd be pretty close to a releasable metadata 2.0 spec... Cheers, Nick. > > On Thu, Jan 30, 2014 at 1:40 PM, Ronald Oussoren wrote: > > > > On 30 Jan 2014, at 19:36, Daniel Holth wrote: > > > >> On Thu, Jan 30, 2014 at 1:29 PM, Ronald Oussoren < ronaldoussoren at mac.com> wrote: > >>> > >>> On 30 Jan 2014, at 19:19, Daniel Holth wrote: > >>> > >>>> Your best bet currently is to execute the "mv" command to change the > >>>> filename of the wheel. > >>>> > >>>> I think bdist_wheel should probably accept a setup() argument to set > >>>> the tags. Currently you can just set a "universal" flag which means > >>>> pure 2+3 python. > >>> > >>> There's also --plat-name argument for the setuptools command, but that doesn't work. > >>> > >>> I'd have to rename the platform tag from "any" to a list of supported platforms, is that correct? > >>> > >>> That would give a pretty convoluted filename, because the platform tag doesn't allow for wildcards. That is, there is no way to specify "any linux", only something like "linux_x86_64" (to borrow a tag from PEP 425). > >> > >> My position has been that if there is no alternative that works on a > >> different platform, then at least the wheel metadata should say > >> nothing. It's only there to allow the installer to pick among the best > >> of several alternatives. > > > > The py2app setup.py file currently hardcodes a platform check because users did install py2app on other platforms (mostly Windows) and then complained that it didn't work. It is much nicer to tell the user that a package won't work during installation. > > > > Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 31 12:18:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Jan 2014 21:18:22 +1000 Subject: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot)) In-Reply-To: <1391165842.44479.YahooMailBasic@web172401.mail.ir2.yahoo.com> References: <1391165842.44479.YahooMailBasic@web172401.mail.ir2.yahoo.com> Message-ID: On 31 Jan 2014 21:00, "Vinay Sajip" wrote: > > > On Fri, 31/1/14, Thomas Heller wrote: > > > This limitation does no longer exist; it works in 2.x and 3.x as > > well as on 32-bit and 64-bit. > > That's good to know. Of course, as it's platform specific, it can't > be used as a generic C extension import facility, though I suppose > that if/when such a capability becomes available on mainstream > platforms, the memimporter API could be adapted to cover it. Ah, but unlinking is more reliable on most POSIX filesystems than it is on NTFS, so even using this technique on Windows and a cache elsewhere could be beneficial. Cheers, Nick. > > Regards, > > Vinay Sajip > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From squeaky_pl at gmx.com Thu Jan 30 11:09:12 2014 From: squeaky_pl at gmx.com (Squeaky) Date: Thu, 30 Jan 2014 11:09:12 +0100 Subject: [Distutils] Letting the Python implementation produce the Python ABI tag Message-ID: <52EA24C8.9080003@gmx.com> Hi! This is my first message to this mailing list. I saw a discussion about construction of SOABI and I thought that it might be tiresome to keep code to produce that for every possible Python implementation in wheel generators. After all it's the Python implementation that knows better in which version and why they broke binary ABI. Let's say that PyPy 2.2 has this tag set to pp22 but they might have released PyPy 2.3 without breaking the ABI and prefer to advertise PyPy2.3 as pp22 compatible thus reducing the number of wheels a python package maintainer has to produce. The ABI is separate from releases. Also this would benefit if there were new Python implementations coming into ecosystem and they could have a function in their stdlib that advertises their abi level. This would allow these new implementations to work with wheels instantly. Does it make sense or it's an utopia? Regards, Squeaky From wickman at gmail.com Fri Jan 31 20:31:02 2014 From: wickman at gmail.com (Brian Wickman) Date: Fri, 31 Jan 2014 11:31:02 -0800 Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python) Message-ID: This is in response to Vinay's thread but since I wasn't subscribed to distutils-sig, I couldn't easily respond directly to it. Vinay's right, the technology here isn't revolutionary but what's notable is that we've been using it in production for almost 3 years at Twitter. It's also been open-sourced for a couple years at https://github.com/twitter/commons/tree/master/src/python/twitter/common/pythonbut not widely announced (it is, after all, just a small subdirectory in a fairly large mono-repo, and was only recently published independently to PyPI as twitter.common.python.) PEX files are just executable zip files with hashbangs containing a carefully constructed __main__.py and a PEX-INFO, which is json-encoded dictionary describing how to scrub and bootstrap sys.path and the like. They work equally well unpacked into a standalone directory. In practice PEX files are simultaneously our replacement for virtualenv and also our way of distributing Python applications to production. Now we could use virtualenv to do this but it's hard to argue with a deployment process that is literally "cp". Furthermore, most of our machines don't have compiler toolchains or external network access, so hermetically sealing all dependencies once at build time (possibly for multiple platforms since all developers use Macs) has huge appeal. This is even more important at Twitter where it's common to run a dozen different Python applications on the same box at the same time, some using 2.6, some 2.7, some PyPy, but all with varying versions of underlying dependencies. Speaking to recent distutils-sig threads, we used to go way out of our way to never hit disk (going so far as building our own .egg packager and pure python recursive zipimport implementation so that we could import from eggs within zips, write ephemeral .so's to dlopen and unlink) but we've since moved away from that position for simplicity's sake. For practical reasons we've always needed "not zip-safe" PEX files where all code is written to disk prior to execution (ex: legacy Django applications that have __file__-relative business logic) so we decided to just throw away the magical zipimport stuff and embrace using disk as a pure cache. This seems more compatible philosophically with the direction wheels are going for example. Since there's been more movement in the PEP space recently, we've been evolving PEX in order to be as standards-compliant as possible, which is why I've been more visible recently re: talks, .whl support and the like. I'd also love to chat about more about PEX and how it relates to things like PEP 441 and/or other attempts like pyzzer. cheers, brian twitter.com/wickman -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Jan 31 21:52:58 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 31 Jan 2014 20:52:58 +0000 (GMT) Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python) In-Reply-To: Message-ID: <1391201578.47653.YahooMailBasic@web172403.mail.ir2.yahoo.com> Thanks for the info. > For practical reasons we've always needed?"not zip-safe" PEX files > where all code is written to disk prior to execution (ex: legacy > Django applications that have __file__-relative business logic) > so we decided to just throw away the magical zipimport stuff and > embrace using disk as a pure cache. I take it this is because of dependencies (whether your own legacy code or external dependencies) that haven't been designed to run from zip, and where it's not worth it (or otherwise not possible) to re-engineer to run from zip? If there were other reasons (apart from the need to maintain the magic machinery, or problems with it), can you say what they were? Also, is your production usage confined to POSIX platforms, or does Windows have any role to play? If so, did you run into any particular issues relating to Windows that might not be common knowledge? I certainly find single-file executables containing Python code a boon and hope they play a bigger part in how Python software is distributed. I will be looking into PEX for some good ideas :-) Regards, Vinay Sajip From dholth at gmail.com Fri Jan 31 22:15:47 2014 From: dholth at gmail.com (Daniel Holth) Date: Fri, 31 Jan 2014 16:15:47 -0500 Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python) In-Reply-To: <1391201578.47653.YahooMailBasic@web172403.mail.ir2.yahoo.com> References: <1391201578.47653.YahooMailBasic@web172403.mail.ir2.yahoo.com> Message-ID: On Fri, Jan 31, 2014 at 3:52 PM, Vinay Sajip wrote: > Thanks for the info. > >> For practical reasons we've always needed "not zip-safe" PEX files >> where all code is written to disk prior to execution (ex: legacy >> Django applications that have __file__-relative business logic) >> so we decided to just throw away the magical zipimport stuff and >> embrace using disk as a pure cache. > > I take it this is because of dependencies (whether your own legacy code > or external dependencies) that haven't been designed to run from zip, and > where it's not worth it (or otherwise not possible) to re-engineer to run > from zip? If there were other reasons (apart from the need to maintain the > magic machinery, or problems with it), can you say what they were? > > Also, is your production usage confined to POSIX platforms, or does > Windows have any role to play? If so, did you run into any particular > issues relating to Windows that might not be common knowledge? > > I certainly find single-file executables containing Python code a boon > and hope they play a bigger part in how Python software is distributed. > I will be looking into PEX for some good ideas :-) > > Regards, > > Vinay Sajip PEP 441 is mostly about promoting the idea of archives with a #! module and giving them an extension. It is neat to be aware of PEX and look forward to some better tooling. From wickman at gmail.com Fri Jan 31 22:57:05 2014 From: wickman at gmail.com (Brian Wickman) Date: Fri, 31 Jan 2014 13:57:05 -0800 Subject: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python) In-Reply-To: <1391201578.47653.YahooMailBasic@web172403.mail.ir2.yahoo.com> References: <1391201578.47653.YahooMailBasic@web172403.mail.ir2.yahoo.com> Message-ID: > > > For practical reasons we've always needed "not zip-safe" PEX files > > where all code is written to disk prior to execution (ex: legacy > > Django applications that have __file__-relative business logic) > > so we decided to just throw away the magical zipimport stuff and > > embrace using disk as a pure cache. > > I take it this is because of dependencies (whether your own legacy code > or external dependencies) that haven't been designed to run from zip, and > where it's not worth it (or otherwise not possible) to re-engineer to run > from zip? If there were other reasons (apart from the need to maintain the > magic machinery, or problems with it), can you say what they were? > The two major reasons here are interpreter compatibility and minimizing what we felt to be hacks, which falls under "maintain the magic machinery" I suppose. There are myriad other practical reasons. Here are some: Our recursive zipimporter was not very portable. We relied upon things like a mutable zipimport._zip_directory_cache but this was both undocumented and subject to change (and was always immutable in PyPy, so we couldn't support it.) Code that touches _zip_directory_cache has since been removed in pkg_resources (that change also broke us, around distribute 0.6.42.) Things like the change from find_in_zip to find_in_zips broke us just a few weeks ago (setuptools 1.1.7 => 2.x.) Our monkeypatches to fix bugs (or features, who knows?) exposed in setuptools by our recursive zipimporter e.g. https://bitbucket.org/tarek/distribute/issue/274/disparity-in-metadata-resource-_listdirusually stopped working over time, since they were patching undocumented parts of the API. This has only accelerated recently since more attention has been drawn to that code by PEPs 426/440 etc. Even today PyPy has bugs with large collections of zipimported namespace packages regardless of zipimport implementation. We eventually surmounted some of these issues, but rather than be doomed to repeat them in the future, we decided to ditch our custom importer instead and rely upon the one in the stdlib when possible. We also wrote our own stubs for .so files so that they could be extracted from nested zips: https://github.com/twitter/commons/blob/677780ea56af610359b3ccb30accb5b9ecd713d8/src/python/twitter/common/python/distiller.py#L34 This turns out to be fairly memory intensive. One of our most production-critical Python applications, the Thermos executor component of Aurora (see https://github.com/apache/incubator-aurora), manages much of Twitter's infrastructure and as such needs to have as small a memory footprint as possible. The process of pulling out one .so file (a 40MB library in one of our core dependencies) took ~150MB of RAM due to zipimport making two copies of it in RAM (in order to do a string comparison) before writing it to disk as an eager resource. This became problematic because the executor itself only needs ~45MB RSS in steady state, which is the overhead we'd prefer to allocate to its kernel container, but the reality is that we have to allocate for peak. While 100MB extra RSS doesn't sound like much, when you multiply by O(10s) tasks per machine and O(1000s) machines, it translates to O(TBs) of RAM, which is O( :-( $ ) to Twitter. Lastly, there are social reasons. It's just hard to convince most engineers to use things like pkg_resources or pkgutil to manipulate resources when for them the status quo is just using __file__. Bizarrely the social challenges are just as hard as the abovementioned technical challenges. > Also, is your production usage confined to POSIX platforms, or does > Windows have any role to play? If so, did you run into any particular > issues relating to Windows that might not be common knowledge? > > We only target Linux and OS X in practice. I've gotten PEX working with CPython (>=2.6 and >=3.2), PyPy and Jython in those situations. I dabbled with IronPython briefly but never got it to work. I don't see any really good reason why PEX files couldn't be launched on Windows but I've never tried. ~brian -------------- next part -------------- An HTML attachment was scrubbed... URL: