From LISTSERV at JAVA.SUN.COM Thu May 5 12:24:41 2005 From: LISTSERV at JAVA.SUN.COM (L-Soft list server at Sun Microsystems Inc. (1.8e)) Date: Thu, 5 May 2005 04:24:41 -0600 Subject: [Distutils] Output of your job "distutils-sig" Message-ID: > ok ok ok,,,,, here is it Too many arguments specified - maximum is 2. Summary of resource utilization ------------------------------- CPU time: 0.000 sec Overhead CPU: 0.000 sec CPU model: 4-CPU Ultra-80 Job origin: distutils-sig at PYTHON.ORG From cmtaylor at ti.com Thu May 5 22:36:02 2005 From: cmtaylor at ti.com (Taylor, Martin) Date: Thu, 5 May 2005 15:36:02 -0500 Subject: [Distutils] Why do I get 3 copies of my files from this simple Distutils setup.py? Message-ID: On my WinXP PC I have the following file structure: C:\FOO\setup.py C:\FOO\Bar\BarConstants.py C:\FOO\Bar\Bar.py The contents of \FOO\setup.py are as follows: setup(name="FOO", version="4.2.3", package_dir={"FOO.Bar" : "Bar"}, packages=["FOO.Bar"], author="C. Martin Taylor", author_email="cmtaylor at ti.com" ) When I run this setup.py to build a "bdist" I get the following output: C:\FOO>python setup.py bdist --format=zip running bdist running bdist_dumb running build running build_py package init file 'Bar\__init__.py' not found (or not a regular file) package init file 'Bar\__init__.py' not found (or not a regular file) installing to build\bdist.win32\dumb running install running install_lib creating build\bdist.win32\dumb creating build\bdist.win32\dumb\Python24 creating build\bdist.win32\dumb\Python24\Lib creating build\bdist.win32\dumb\Python24\Lib\site-packages copying build\lib\BarConstants.py -> build\bdist.win32\dumb\Python24\Lib\site-packages creating build\bdist.win32\dumb\Python24\Lib\site-packages\Bar copying build\lib\Bar\BarConstants.py -> build\bdist.win32\dumb\Python24\Lib\site-packages\Bar copying build\lib\Bar\Bar.py -> build\bdist.win32\dumb\Python24\Lib\site-packages\Bar copying build\lib\Bar.py -> build\bdist.win32\dumb\Python24\Lib\site-packages creating build\bdist.win32\dumb\Python24\Lib\site-packages\FOO creating build\bdist.win32\dumb\Python24\Lib\site-packages\FOO\Bar copying build\lib\FOO\Bar\BarConstants.py -> build\bdist.win32\dumb\Python24\Lib\site-packages\FOO\Bar copying build\lib\FOO\Bar\Bar.py -> build\bdist.win32\dumb\Python24\Lib\site-packages\FOO\Bar byte-compiling build\bdist.win32\dumb\Python24\Lib\site-packages\BarConstants.py to BarConstants.pyc byte-compiling build\bdist.win32\dumb\Python24\Lib\site-packages\Bar\GUIConstants.py to BarConstants.pyc byte-compiling build\bdist.win32\dumb\Python24\Lib\site-packages\Bar\Bar.py to Bar.pyc byte-compiling build\bdist.win32\dumb\Python24\Lib\site-packages\Bar.py to GUITest.pyc byte-compiling build\bdist.win32\dumb\Python24\Lib\site-packages\FOO\Bar\BarConstants.p y to BarConstants.pyc byte-compiling build\bdist.win32\dumb\Python24\Lib\site-packages\FOO\Bar\Bar.py to Bar.pyc creating 'C:\FOO\dist\FOO-4.2.3.win32.zip' and adding '.' to it adding 'Python24\Lib\site-packages\BarConstants.py' adding 'Python24\Lib\site-packages\BarConstants.pyc' adding 'Python24\Lib\site-packages\Bar.py' adding 'Python24\Lib\site-packages\Bar.pyc' adding 'Python24\Lib\site-packages\Bar\BarConstants.py' adding 'Python24\Lib\site-packages\Bar\BarConstants.pyc' adding 'Python24\Lib\site-packages\Bar\Bar.py' adding 'Python24\Lib\site-packages\Bar\Bar.pyc' adding 'Python24\Lib\site-packages\FOO\Bar\BarConstants.py' adding 'Python24\Lib\site-packages\FOO\Bar\BarConstants.pyc' adding 'Python24\Lib\site-packages\FOO\Bar\Bar.py' adding 'Python24\Lib\site-packages\FOO\Bar\Bar.pyc' removing 'build\bdist.win32\dumb' (and everything under it) My question is why am I getting 3 copies of the 2 Python files I'm trying to distribute, one copy in \Python24\Lib\site-packages\, one in \Python24\Lib\site-packages\Bar\ and one in \Python24\Lib\site-packages\FOO\Bar\? The only copy I want is the last one! This is my first attempt at using Distutils so any help and explanations would be appreciated. (I really don't want to make FOO.Bar into a "real" Python package with an __init__.py module, so please don't suggest that.) Thank you, Martin Taylor From nc-engelre at netcologne.de Wed May 11 10:19:45 2005 From: nc-engelre at netcologne.de (Reinhard Engel) Date: Wed, 11 May 2005 10:19:45 +0200 Subject: [Distutils] Installation crashes: Windows XP ntdll.dll Message-ID: <4281C021.5040103@netcologne.de> When I try to install the Windows Installer packages for cTypes and uTidyLib the installation crashes Windows XP Python 2.4.1 -------------------- Problem with cTypes: -------------------- ctypes-0.9.6.win32-py2.4.exe hat ein Problem festgestellt und muss beendet werden. AppName: ctypes-0.9.6.win32-py2.4.exe AppVer: 0.0.0.0 ModName: ntdll.dll ModVer: 5.1.2600.1217 Offset: 0000b2ab [metadata] author=Thomas Heller author_email=theller at python.net description=create and manipulate C data types in Python, call functions in shared libraries name=ctypes url=http://starship.python.net/crew/theller/ctypes.html version=0.9.6 [Setup] info=ctypes is a Python package to create and manipulate C data types in\nPython, and to call functions in dynamic link libraries/shared\ndlls. It allows wrapping these libraries in pure Python.\n\n\n Author: Thomas Heller\n Author_email: theller at python.net\n Description: create and manipulate C data types in Python, call functions in shared libraries\n Name: ctypes\n Url: http://starship.python.net/crew/theller/ctypes.html\n Version: 0.9.6 target_compile=1 target_optimize=1 target_version=2.4 title=ctypes-0.9.6 build_info=Built Fri Mar 18 19:33:25 2005 with distutils-2.4.1 ----------------------- Problem with TidyLib: ----------------------- uTidylib-0.2.1.win32.exe hat ein Problem festgestellt und muss beendet werden. AppName: utidylib-0.2.1.win32.exe AppVer: 0.0.0.0 ModName: ntdll.dll ModVer: 5.1.2600.1217 Offset: 0000b2ab [metadata] author=Cory Dodt author_email=corydodt at twistedmatrix.com description=Wrapper for HTML Tidy at http://tidy.sourceforge.net name=uTidylib url=http://utidylib.sf.net version=0.2.1 [Setup] info=A wrapper for the relocatable version of HTML Tidy (see\nhttp://tidy.sourceforge.net for details). This allows you to\ntidy HTML files through a Pythonic interface.\n\n Author: Cory Dodt\n Author_email: corydodt at twistedmatrix.com\n Description: Wrapper for HTML Tidy at http://tidy.sourceforge.net\n Name: uTidylib\n Url: http://utidylib.sf.net\n Version: 0.2.1 target_compile=1 target_optimize=1 title=uTidylib-0.2.1 build_info=Built Tue Dec 07 11:06:09 2004 with distutils-1.0.3 ----------------- There are no problems, if I use Windows Installer with programs that are not built with distutils (i.e. Pyhon itself with python-2.4.1.msi). From tcm-users-request at rotterdam.cs.utwente.nl Sun May 15 12:23:37 2005 From: tcm-users-request at rotterdam.cs.utwente.nl (tcm-users-request@rotterdam.cs.utwente.nl) Date: Sun, 15 May 2005 12:23:37 +0200 (MEST) Subject: [Distutils] Your mail to tcm-users-request@listserv.cs.utwente.nl In-Reply-To: , from distutils-sig@python.org Message-ID: <200505151023.j4FANbg7004072@rotterdam.ewi.utwente.nl> This pre-recorded message is being sent in response to your recent email to tcm-users-request at listserv.cs.utwente.nl. All routine administrative requests (including subscriptions and unsubscriptions) concerning this mailing list are handled by an automated server. Please read this message carefully to find the information relevant to you. SUBSCRIBING =========== To subscribe to tcm-users, send the following in the body (not the subject line) of an email message to "Majordomo at listserv.cs.utwente.nl": subscribe tcm-users This will subscribe the account from which you send the message to the tcm-users list. If you wish to subscribe another address instead (such as a local redistribution list), you can use a command of the form: subscribe tcm-users other-address at your_site.your_net UNSUBSCRIBING ============= To unsubscribe from tcm-users, send the following in the body (not the subject line) of an email message to "Majordomo at listserv.cs.utwente.nl": unsubscribe tcm-users This will unsubscribe the account from which you send the message. If you are subscribed with some other address, you'll have to send a command of the following form instead: unsubscribe tcm-users other-address at your_site.your_net If you don't know what address you are subscribed with, you can send the following command to see who else is on the list (assuming that information isn't designated "private" by the owner of the list): who tcm-users If you want to search non-private lists at this server, you can do that by sending a command like: which string This will return a list of all entries on all lists that contain "string". HELP ==== To find out more about the automated server and the commands it understands, send the following command to "Majordomo at listserv.cs.utwente.nl": help If you feel you need to reach a human, send email to: tcm-users-approval at listserv.cs.utwente.nl From bungeman at lycos.com Mon May 16 16:54:30 2005 From: bungeman at lycos.com (Ben Wagner) Date: Mon, 16 May 2005 09:54:30 -0500 Subject: [Distutils] Distutils, SWIG, and C++ Message-ID: <20050516145430.BDA58CA08E@ws7-4.us4.outblaze.com> I just downloaded the latest Distutils and am running Python24 on WindowsXP Pro and using MSVC Pro 2003. I'm developing a Python extention (.pyd) and want to use Distutils to manage the install process. After reading much documentation and code I have gotten everything to build and run manually, but am having difficulty with the setup configuration. What it boils down to is I have a Swig (.i) which is written in C++, mostly since it includes PythonCOM.h. Here is a slightly simplified version of my setup.py #docs left out from distutils.core import setup, Extension import os #using these two lines on my machine works to compile properly #"C:\Program Files\SWIG-1.3.24\swig.exe" -python -c++ -o mouse_wrap.cpp mouse_client.i #"C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG -IC:\Python24\include -IC:\Python24\PC -IC:\Python24\Lib\site-packages\win32\include -IC:\Python24\Lib\site-packages\win32com\include /Tpmouse_wrap.cpp /Fobuild\mouse_wrap.obj from distutils.sysconfig import get_python_lib win32Incdir = os.path.join(get_python_lib(plat_specific=1), 'win32\include') win32comIncdir = os.path.join(get_python_lib(plat_specific=1), 'win32com\include') pythoncomLib = os.path.join(get_python_lib(plat_specific=1), 'win32com\libs\pythoncom') wintypesLib = os.path.join(get_python_lib(plat_specific=1), 'win32\libs\pywintypes') libs = ['user32', 'ole32', pythoncomLib, wintypesLib] includes = [win32Incdir, win32comIncdir] macros = [('NDEBUG', '1')] compilerArgs = [] lang = 'c++' swigOpts = ['-c++'] setup(name='pyCOMHook', #standard setup stuff left out ext_modules = [Extension('pyCOMHook._mouse', ['mouse.i'], libraries=libs, include_dirs=includes, define_macros=macros, extra_compile_args=compilerArgs, language=lang, swig_opts=swigOpts)], ) This succesfuly puts '-c++' on the swig command line, but the output file (for Swig) is still .c (and not .cpp). After looking through the Distutils code I realized two things 1. build_ext.py [line 525, in swig_sources (self, sources, extension)] I think needs to be changed from if self.swig_cpp or ('-c++' in self.swig_opts): to if self.swig_cpp or ('-c++' in self.swig_opts) or ('-c++' in extension.swig_opts): or something like that. Doing this fixed my whole problem and everything ran fine. 2. even without the above fix, this still should have worked, even with the .c extention, but the msvccompiler.py compiler currently ignores the language setting and only uses file extentions to determine /Tc or /Tp flags So my question is this, is it currently impossible to Swig a .i written in c++ on Windows using msvc and a bug fix is needed, or is there some nifty way around this which I have not yet been clever enough to ferret out? I would appreciate if anyone more familiar with the Distutils code could take a look into this. -- _______________________________________________ NEW! Lycos Dating Search. The only place to search multiple dating sites at once. http://datingsearch.lycos.com From glennpierce at connectfree.co.uk Tue May 17 12:13:01 2005 From: glennpierce at connectfree.co.uk (Glenn Pierce) Date: Tue, 17 May 2005 11:13:01 +0100 Subject: [Distutils] Dist utils library problem? Message-ID: <4289C3AD.8070902@connectfree.co.uk> Sorry if this is not the correct mailing list. I have a question about writing a portable setup.py script for distutils. I have a directory structure like this. FreeImage/ |-------Source/ | |------ FreeImage.h | |-------Dist/ | |------libfreeimage-3.7.0.so | |------libfreeimage.a | |------FreeImage.lib | |------FreeImage.dll | |-------Wrapper/ |---------Python/ |----------build/ |----------freeimagemodule.c |----------setup.py I am wrapper a c++ library and have the library files for linux and windows installed under Dist/. I want to create a setup file that works on both unix and Windows but I am having trouble with windows. My setup.py file is like the following: from distutils.core import setup, Extension ext_module = Extension('freeimagewrapper', define_macros = [('MAJOR_VERSION', '3'), ('MINOR_VERSION', '7')], include_dirs = ['../../Source/'], library_dirs=['../../Dist/'], libraries = ['FreeImage'], sources = ['freeimagemodule.c']) setup (name = 'freeimage', version = '1.0', author = 'Glenn Pierce', description = 'Python wrapper for the freeimage library', ext_modules = [ext_module]) I am getting link errors. Visual studio can find ../../Dist/FreeImage.lib but I don't think it can find FreeImage.dll The errors are like: Creating library build\temp.win32-2.4\Release\freeimagewrapper.lib and object build\temp.win32-2.4\Release\freeimagewrapper.exp freeimagemodule.obj : error LNK2019: unresolved external symbol _FreeImage_Unload referenced in function _destroy Where should I be placing the FreeImage.dll file ? I have read the distutils guide but wasn't clear on the library section for ext modules. Thanks for any help. Glenn From Majordomo at iti.informatik.tu-darmstadt.de Wed May 18 13:24:27 2005 From: Majordomo at iti.informatik.tu-darmstadt.de (Majordomo@iti.informatik.tu-darmstadt.de) Date: Wed, 18 May 2005 13:24:27 +0200 Subject: [Distutils] Majordomo results: [SPAM?] Vorbildliche Aktion Message-ID: <200505181124.j4IBORVP013706@spoiler.iti.informatik.tu-darmstadt.de> -- >>>> Lese selbst: **** Command 'lese' not recognized. >>>> http://www.npd.de/npd_info/deutschland/2004/d1204-24.html **** Command 'http://www.npd.de/npd_info/deutschland/2004/d1204-24.html' not recognized. **** No valid commands found. **** Commands must be in message BODY, not in HEADER. **** Help for Majordomo at iti.informatik.tu-darmstadt.de: This help message is being sent to you from the Majordomo mailing list management system at Majordomo at iti.informatik.tu-darmstadt.de. This is version 1.94.5 of Majordomo. If you're familiar with mail servers, an advanced user's summary of Majordomo's commands appears at the end of this message. Majordomo is an automated system which allows users to subscribe and unsubscribe to mailing lists, and to retrieve files from list archives. You can interact with the Majordomo software by sending it commands in the body of mail messages addressed to "Majordomo at iti.informatik.tu-darmstadt.de". Please do not put your commands on the subject line; Majordomo does not process commands in the subject line. You may put multiple Majordomo commands in the same mail message. Put each command on a line by itself. If you use a "signature block" at the end of your mail, Majordomo may mistakenly believe each line of your message is a command; you will then receive spurious error messages. To keep this from happening, either put a line starting with a hyphen ("-") before your signature, or put a line with just the word end on it in the same place. This will stop the Majordomo software from processing your signature as bad commands. Here are some of the things you can do using Majordomo: I. FINDING OUT WHICH LISTS ARE ON THIS SYSTEM To get a list of publicly-available mailing lists on this system, put the following line in the body of your mail message to Majordomo at iti.informatik.tu-darmstadt.de: lists Each line will contain the name of a mailing list and a brief description of the list. To get more information about a particular list, use the "info" command, supplying the name of the list. For example, if the name of the list about which you wish information is "demo-list", you would put the line info demo-list in the body of the mail message. II. SUBSCRIBING TO A LIST Once you've determined that you wish to subscribe to one or more lists on this system, you can send commands to Majordomo to have it add you to the list, so you can begin receiving mailings. To receive list mail at the address from which you're sending your mail, simply say "subscribe" followed by the list's name: subscribe demo-list If for some reason you wish to have the mailings go to a different address (a friend's address, a specific other system on which you have an account, or an address which is more correct than the one that automatically appears in the "From:" header on the mail you send), you would add that address to the command. For instance, if you're sending a request from your work account, but wish to receive "demo-list" mail at your personal account (for which we will use "jqpublic at my-isp.com" as an example), you'd put the line subscribe demo-list jqpublic at my-isp.com in the mail message body. Based on configuration decisions made by the list owners, you may be added to the mailing list automatically. You may also receive notification that an authorization key is required for subscription. Another message will be sent to the address to be subscribed (which may or may not be the same as yours) containing the key, and directing the user to send a command found in that message back to Majordomo at iti.informatik.tu-darmstadt.de. (This can be a bit of extra hassle, but it helps keep you from being swamped in extra email by someone who forged requests from your address.) You may also get a message that your subscription is being forwarded to the list owner for approval; some lists have waiting lists, or policies about who may subscribe. If your request is forwarded for approval, the list owner should contact you soon after your request. Upon subscribing, you should receive an introductory message, containing list policies and features. Save this message for future reference; it will also contain exact directions for unsubscribing. If you lose the intro mail and would like another copy of the policies, send this message to Majordomo at iti.informatik.tu-darmstadt.de: intro demo-list (substituting, of course, the real name of your list for "demo-list"). III. UNSUBSCRIBING FROM MAILING LISTS Your original intro message contains the exact command which should be used to remove your address from the list. However, in most cases, you may simply send the command "unsubscribe" followed by the list name: unsubscribe demo-list (This command may fail if your provider has changed the way your address is shown in your mail.) To remove an address other than the one from which you're sending the request, give that address in the command: unsubscribe demo-list jqpublic at my-isp.com In either of these cases, you can tell Majordomo at iti.informatik.tu-darmstadt.de to remove the address in question from all lists on this server by using "*" in place of the list name: unsubscribe * unsubscribe * jqpublic at my-isp.com IV. FINDING THE LISTS TO WHICH AN ADDRESS IS SUBSCRIBED To find the lists to which your address is subscribed, send this command in the body of a mail message to Majordomo at iti.informatik.tu-darmstadt.de: which You can look for other addresses, or parts of an address, by specifying the text for which Majordomo should search. For instance, to find which users at my-isp.com are subscribed to which lists, you might send the command which my-isp.com Note that many list owners completely or fully disable the "which" command, considering it a privacy violation. V. FINDING OUT WHO'S SUBSCRIBED TO A LIST To get a list of the addresses on a particular list, you may use the "who" command, followed by the name of the list: who demo-list Note that many list owners allow only a list's subscribers to use the "who" command, or disable it completely, believing it to be a privacy violation. VI. RETRIEVING FILES FROM A LIST'S ARCHIVES Many list owners keep archives of files associated with a list. These may include: - back issues of the list - help files, user profiles, and other documents associated with the list - daily, monthly, or yearly archives for the list To find out if a list has any files associated with it, use the "index" command: index demo-list If you see files in which you're interested, you may retrieve them by using the "get" command and specifying the list name and archive filename. For instance, to retrieve the files called "profile.form" (presumably a form to fill out with your profile) and "demo-list.9611" (presumably the messages posted to the list in November 1996), you would put the lines get demo-list profile.form get demo-list demo-list.9611 in your mail to Majordomo at iti.informatik.tu-darmstadt.de. VII. GETTING MORE HELP To contact a human site manager, send mail to Majordomo-Owner at iti.informatik.tu-darmstadt.de. To contact the owner of a specific list, send mail to that list's approval address, which is formed by adding "-approval" to the user-name portion of the list's address. For instance, to contact the list owner for demo-list at iti.informatik.tu-darmstadt.de, you would send mail to demo-list-approval at iti.informatik.tu-darmstadt.de. To get another copy of this help message, send mail to Majordomo at iti.informatik.tu-darmstadt.de with a line saying help in the message body. VIII. COMMAND SUMMARY FOR ADVANCED USERS In the description below items contained in []'s are optional. When providing the item, do not include the []'s around it. Items in angle brackets, such as
, are meta-symbols that should be replaced by appropriate text without the angle brackets. It understands the following commands: subscribe [] [
] Subscribe yourself (or
if specified) to the named . unsubscribe [] [
] Unsubscribe yourself (or
if specified) from the named . "unsubscribe *" will remove you (or
) from all lists. This _may not_ work if you have subscribed using multiple addresses. get [] Get a file related to . index [] Return an index of files you can "get" for . which [
] Find out which lists you (or
if specified) are on. who [] Find out who is on the named . info [] Retrieve the general introductory information for the named . intro [] Retrieve the introductory message sent to new users. Non-subscribers may not be able to retrieve this. lists Show the lists served by this Majordomo server. help Retrieve this message. end Stop processing commands (useful if your mailer adds a signature). Commands should be sent in the body of an email message to "Majordomo at iti.informatik.tu-darmstadt.de" or to "-request at iti.informatik.tu-darmstadt.de". The parameter is only optional if the message is sent to an address of the form "-request at iti.informatik.tu-darmstadt.de". Multiple commands can be processed provided each occurs on a separate line. Commands in the "Subject:" line are NOT processed. If you have any questions or problems, please contact "Majordomo-Owner at iti.informatik.tu-darmstadt.de". From Majordomo at iti.informatik.tu-darmstadt.de Wed May 18 13:24:28 2005 From: Majordomo at iti.informatik.tu-darmstadt.de (Majordomo@iti.informatik.tu-darmstadt.de) Date: Wed, 18 May 2005 13:24:28 +0200 Subject: [Distutils] Majordomo results: [SPAM?] Vorbildliche Aktion Message-ID: <200505181124.j4IBOSiQ013713@spoiler.iti.informatik.tu-darmstadt.de> -- >>>> Lese selbst: **** Command 'lese' not recognized. >>>> http://www.npd.de/npd_info/deutschland/2004/d1204-24.html **** Command 'http://www.npd.de/npd_info/deutschland/2004/d1204-24.html' not recognized. **** No valid commands found. **** Commands must be in message BODY, not in HEADER. **** Help for Majordomo at iti.informatik.tu-darmstadt.de: This help message is being sent to you from the Majordomo mailing list management system at Majordomo at iti.informatik.tu-darmstadt.de. This is version 1.94.5 of Majordomo. If you're familiar with mail servers, an advanced user's summary of Majordomo's commands appears at the end of this message. Majordomo is an automated system which allows users to subscribe and unsubscribe to mailing lists, and to retrieve files from list archives. You can interact with the Majordomo software by sending it commands in the body of mail messages addressed to "Majordomo at iti.informatik.tu-darmstadt.de". Please do not put your commands on the subject line; Majordomo does not process commands in the subject line. You may put multiple Majordomo commands in the same mail message. Put each command on a line by itself. If you use a "signature block" at the end of your mail, Majordomo may mistakenly believe each line of your message is a command; you will then receive spurious error messages. To keep this from happening, either put a line starting with a hyphen ("-") before your signature, or put a line with just the word end on it in the same place. This will stop the Majordomo software from processing your signature as bad commands. Here are some of the things you can do using Majordomo: I. FINDING OUT WHICH LISTS ARE ON THIS SYSTEM To get a list of publicly-available mailing lists on this system, put the following line in the body of your mail message to Majordomo at iti.informatik.tu-darmstadt.de: lists Each line will contain the name of a mailing list and a brief description of the list. To get more information about a particular list, use the "info" command, supplying the name of the list. For example, if the name of the list about which you wish information is "demo-list", you would put the line info demo-list in the body of the mail message. II. SUBSCRIBING TO A LIST Once you've determined that you wish to subscribe to one or more lists on this system, you can send commands to Majordomo to have it add you to the list, so you can begin receiving mailings. To receive list mail at the address from which you're sending your mail, simply say "subscribe" followed by the list's name: subscribe demo-list If for some reason you wish to have the mailings go to a different address (a friend's address, a specific other system on which you have an account, or an address which is more correct than the one that automatically appears in the "From:" header on the mail you send), you would add that address to the command. For instance, if you're sending a request from your work account, but wish to receive "demo-list" mail at your personal account (for which we will use "jqpublic at my-isp.com" as an example), you'd put the line subscribe demo-list jqpublic at my-isp.com in the mail message body. Based on configuration decisions made by the list owners, you may be added to the mailing list automatically. You may also receive notification that an authorization key is required for subscription. Another message will be sent to the address to be subscribed (which may or may not be the same as yours) containing the key, and directing the user to send a command found in that message back to Majordomo at iti.informatik.tu-darmstadt.de. (This can be a bit of extra hassle, but it helps keep you from being swamped in extra email by someone who forged requests from your address.) You may also get a message that your subscription is being forwarded to the list owner for approval; some lists have waiting lists, or policies about who may subscribe. If your request is forwarded for approval, the list owner should contact you soon after your request. Upon subscribing, you should receive an introductory message, containing list policies and features. Save this message for future reference; it will also contain exact directions for unsubscribing. If you lose the intro mail and would like another copy of the policies, send this message to Majordomo at iti.informatik.tu-darmstadt.de: intro demo-list (substituting, of course, the real name of your list for "demo-list"). III. UNSUBSCRIBING FROM MAILING LISTS Your original intro message contains the exact command which should be used to remove your address from the list. However, in most cases, you may simply send the command "unsubscribe" followed by the list name: unsubscribe demo-list (This command may fail if your provider has changed the way your address is shown in your mail.) To remove an address other than the one from which you're sending the request, give that address in the command: unsubscribe demo-list jqpublic at my-isp.com In either of these cases, you can tell Majordomo at iti.informatik.tu-darmstadt.de to remove the address in question from all lists on this server by using "*" in place of the list name: unsubscribe * unsubscribe * jqpublic at my-isp.com IV. FINDING THE LISTS TO WHICH AN ADDRESS IS SUBSCRIBED To find the lists to which your address is subscribed, send this command in the body of a mail message to Majordomo at iti.informatik.tu-darmstadt.de: which You can look for other addresses, or parts of an address, by specifying the text for which Majordomo should search. For instance, to find which users at my-isp.com are subscribed to which lists, you might send the command which my-isp.com Note that many list owners completely or fully disable the "which" command, considering it a privacy violation. V. FINDING OUT WHO'S SUBSCRIBED TO A LIST To get a list of the addresses on a particular list, you may use the "who" command, followed by the name of the list: who demo-list Note that many list owners allow only a list's subscribers to use the "who" command, or disable it completely, believing it to be a privacy violation. VI. RETRIEVING FILES FROM A LIST'S ARCHIVES Many list owners keep archives of files associated with a list. These may include: - back issues of the list - help files, user profiles, and other documents associated with the list - daily, monthly, or yearly archives for the list To find out if a list has any files associated with it, use the "index" command: index demo-list If you see files in which you're interested, you may retrieve them by using the "get" command and specifying the list name and archive filename. For instance, to retrieve the files called "profile.form" (presumably a form to fill out with your profile) and "demo-list.9611" (presumably the messages posted to the list in November 1996), you would put the lines get demo-list profile.form get demo-list demo-list.9611 in your mail to Majordomo at iti.informatik.tu-darmstadt.de. VII. GETTING MORE HELP To contact a human site manager, send mail to Majordomo-Owner at iti.informatik.tu-darmstadt.de. To contact the owner of a specific list, send mail to that list's approval address, which is formed by adding "-approval" to the user-name portion of the list's address. For instance, to contact the list owner for demo-list at iti.informatik.tu-darmstadt.de, you would send mail to demo-list-approval at iti.informatik.tu-darmstadt.de. To get another copy of this help message, send mail to Majordomo at iti.informatik.tu-darmstadt.de with a line saying help in the message body. VIII. COMMAND SUMMARY FOR ADVANCED USERS In the description below items contained in []'s are optional. When providing the item, do not include the []'s around it. Items in angle brackets, such as
, are meta-symbols that should be replaced by appropriate text without the angle brackets. It understands the following commands: subscribe [] [
] Subscribe yourself (or
if specified) to the named . unsubscribe [] [
] Unsubscribe yourself (or
if specified) from the named . "unsubscribe *" will remove you (or
) from all lists. This _may not_ work if you have subscribed using multiple addresses. get [] Get a file related to . index [] Return an index of files you can "get" for . which [
] Find out which lists you (or
if specified) are on. who [] Find out who is on the named . info [] Retrieve the general introductory information for the named . intro [] Retrieve the introductory message sent to new users. Non-subscribers may not be able to retrieve this. lists Show the lists served by this Majordomo server. help Retrieve this message. end Stop processing commands (useful if your mailer adds a signature). Commands should be sent in the body of an email message to "Majordomo at iti.informatik.tu-darmstadt.de" or to "-request at iti.informatik.tu-darmstadt.de". The parameter is only optional if the message is sent to an address of the form "-request at iti.informatik.tu-darmstadt.de". Multiple commands can be processed provided each occurs on a separate line. Commands in the "Subject:" line are NOT processed. If you have any questions or problems, please contact "Majordomo-Owner at iti.informatik.tu-darmstadt.de". From ianb at colorstudy.com Thu May 19 06:17:20 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 18 May 2005 23:17:20 -0500 Subject: [Distutils] Questions about Python Eggs Message-ID: <428C1350.7080608@colorstudy.com> So, I'm looking at Python Eggs, and I have some questions... Why does it create a Package.egg-info/ directory? It seems odd; is this meant to replace all the metadata arguments to setup.py? That would be fine, I'm happy to get rid of those arguments, but if it's just a copy of that data it seems odd to install it alongside the distribution files. I think I'm generally going to prefer non-zip installations of Eggs, this way I can apply it to projects that weren't written to use pkg_resource, and people can see the source more easily. By any chance does anyone have code on hand to unpack an egg appropriately? I'm sure it's not hard, but it's not a one-liner I'm trying to cut down on the typing... What's the state of depends? What is depends anyway? I notice the comment on pkg_resources.require aren't very confident ;) It actually doesn't look functional to me, though I haven't tried running it. Is it just meant to raise an ImportError when a requirement isn't met, or can it search some directories for appropriate Egg files? This is something I'm very interested in, so if I have the intention correct I'd like to help move this function along. Anyway ideas about how to apply setuptools to other people's packages? Is it safe to do nasty monkeypatching wherein I replace distutils.setup with setuptools.setup? Or, more directly, anyone have code around to build eggs programmatically from other people's packages? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu May 19 14:39:37 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 19 May 2005 08:39:37 -0400 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <428C1350.7080608@colorstudy.com> Message-ID: <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> At 11:17 PM 5/18/2005 -0500, Ian Bicking wrote: >So, I'm looking at Python Eggs, and I have some questions... > >Why does it create a Package.egg-info/ directory? It seems odd; is this >meant to replace all the metadata arguments to setup.py? That would be >fine, I'm happy to get rid of those arguments, but if it's just a copy >of that data it seems odd to install it alongside the distribution files. The Package.egg-info directory serves two main purposes: 1. It's a staging area for files to be copied to the .egg file's EGG-INFO directory 2. It's used to tell the dependency resolution system that an unpacked distribution is available in the given directory. Let's go into #1 first. .egg files contain an EGG-INFO directory, whose purpose is to hold metadata for the distribution. This metadata always includes a standard PKG-INFO file, currently generated from the setup.py metadata. This metadata file is how the runtime can recognize that a zipfile is in fact a Python Egg. But the metadata directory also contains other files that drive the runtime or other systems. For example, there's a 'native_libs.txt' that lists all .pyd/.so files in the distribution, and this is used by the runtime to ensure that all extensions are extracted at the same time. The native_libs file is automatically generated, but you can also create an 'eager_resources.txt' file by hand, if there are other resources that should be extracted together. In the future, you'll also be able to include a (hand-created) dependencies file that specifies what versions of what other distributions you depend on. You'll create this file in the .egg-info directory, and it will be added to the .egg file's EGG-INFO directory, and the runtime system will use it. Last, but not least, EGG-INFO is open to expansion for specific application or framework metadata. For example, a future WSGI "deployment descriptor" file might be placed in EGG-INFO, to allow WSGI servers to support single-file application deployment. Now, the second purpose of the PackageName.egg-info directory is to support development work on a distribution that is *not* contained in a .egg file, when the overall project uses the pkg_resource dependency resolution system. Let's say that you are working on an application that depends on 'Twisted>=2.0', but you are also developing Twisted itself, and therefore do not want to use your Twisted-2.0-py2.4-win32.egg file. You place the directory containing Twisted on your PYTHONPATH -- with Twisted.egg-info in the same directory. That is, if you add 'twisted_src' to PYTHONPATH, you have a layout like this: twisted_src/ twisted/ Twisted.egg-info/ Of course, you might have other packages in that same directory, and perhaps other PackageName.egg-info directories. Anyway, when your application does a 'require("Twisted>=2.0")', the dependency resolution system will see Twisted.egg-info and read the contained PKG-INFO to check for version data. Similarly, any other requests to the pkg_resource API that would ordinarily look for EGG-INFO files, will look for them in Twisted.egg-info (assuming you asked for information about that particular version of Twisted, of course). Anyway, by default the .egg-info directory created by bdist_egg is placed in the correct directory so that if you are developing with the source code being distributed by your setup.py (i.e. your package source is on PYTHONPATH), then the dependency resolution system will correctly see that you already have MyPackage-1.5 (or whatever) available. >I think I'm generally going to prefer non-zip installations of Eggs, >this way I can apply it to projects that weren't written to use >pkg_resource, and people can see the source more easily. By any chance >does anyone have code on hand to unpack an egg appropriately? I'm sure >it's not hard, but it's not a one-liner I'm trying to cut down on the >typing... If you want to unpack an .egg, you need to rename EGG-INFO to PackageName.egg-info. Other than this, you can just unpack the whole thing to a suitable PYTHONPATH directory. Note that this means that you can unpack to site-packages if you like; just make sure the EGG-INFO directory is unpacked to PackageName.egg-info in the same directory. In this way, any packages or applications using the dependency resolution API to find the package will still work correctly. (Of course, if you're using the dependency API, it suffices to just drop the .egg file in site-packages without unpacking it at all, but you asked about unpacking.) >What's the state of depends? What is depends anyway? It's not yet implemented. There's a parser that can parse "requirement" specifications, whose syntax looks like: PackageName[feature1, feature2] >=2.0, <=3.1, ==3.7, >=4.0 That is, it's a package name (and optional feature list) followed by a comma-separated list of zero or more version comparison operators. You can pass a requirement spec like this to the 'require()' API in order to find the specified package. "Features" control optional dependencies. For example, to support FastCGI, PEAK requires the 'fcgiapp' package, but not all applications using PEAK need FastCGI. So, in PEAK's "depends" file, I could do this: PyProtocols>=1.0a0 # other absolute dependencies go here [FastCGI] fcgiapp>=0.1 Now, if you do 'require("PEAK[FastCGI]>=0.5a4")', then the dependency system will look for 'fcgiapp>=0.1' as well as 'PyProtocols>=1.0a0', once it finds PEAK-0.5a4-py2.4-win32.egg. Anyway, the actual implementation of this isn't done yet. The current parser draft can handle dependency specs split across lines (using \ continuation) and comments. There's also a version parser that's roughly a cross between distutils' LooseVersion and StrictVersion, that can correctly handle Python's own versioning system (StrictVersion doesn't support "release candidate" versions) and a wide variety of others. >I notice the comment on pkg_resources.require aren't very confident ;) >It actually doesn't look functional to me, though I haven't tried >running it. It isn't functional; I don't even know what will happen if you run it. It's purely a placeholder sketch for me to remember the algorithms that Bob Ippolito and I discussed, until I get a chance to actually implement them. > Is it just meant to raise an ImportError when a requirement >isn't met, or can it search some directories for appropriate Egg files? It's intended to search for .egg files, as well as recognize that .egg files (or directories containing PackageName.egg-info directories) are already on sys.path. By default, the directories it searches are the ones on sys.path. So, although dropping an .egg in site-packages does not make it importable, it does make it accessible to 'require()' and the like, and the dependency resolution system will add the desired .egg files to sys.path upon request. >Anyway ideas about how to apply setuptools to other people's packages? Currently, my suggestion is to just edit their setup.py. If you need to do it on an automated basis, perhaps you could use a patch file. >Is it safe to do nasty monkeypatching wherein I replace distutils.setup >with setuptools.setup? Or, more directly, anyone have code around to >build eggs programmatically from other people's packages? If distutils' run_setup() worked correctly, you could probably do this. Of course, if you control your environment, you could also possibly just copy bdist_egg.py into your Python installation's distutils/command directory. I don't think bdist_egg has any dependencies to the rest of setuptools that would prevent it from being used from regular distutils that way. So, try copying bdist_egg.py into distutils/command, and then run "setup.py bdist_egg" and see if it works. From syfou at users.sourceforge.net Fri May 20 07:45:03 2005 From: syfou at users.sourceforge.net (Sylvain Fourmanoit) Date: Fri, 20 May 2005 01:45:03 -0400 (EDT) Subject: [Distutils] Setup configuration file and multiple libraries for extensions modules Message-ID: Hi, suppose you have a setup.cfg file along your setup.py script, and that you want to use it to specify multiple external C libraries to link extensions modules against. Something like: [build_ext] libraries=first_lib second_lib ... On Python 2.4.1, you read from distutils.command.build_ext.build_ext.finalize_options() on line 149-150 from the build_ext.py file: if type(self.libraries) is StringType: self.libraries = [self.libraries] Doesn't it mishandle this case, of did I miss the point? It appears to me that something similar to: if type(self.libraries) is StringType: self.libraries = string.split(self.libraries) would be the correct way to handle this (in fact, similar string splits already take place for include_dirs and library_dirs)... I just want to know how you are supposed to specify more than one library in a configuration file with current implementation. :-) Thanks, -- Sylvain From titus at caltech.edu Sun May 22 00:34:30 2005 From: titus at caltech.edu (Titus Brown) Date: Sat, 21 May 2005 15:34:30 -0700 Subject: [Distutils] distutils bug. Message-ID: <20050521223430.GM31777@caltech.edu> Hi all, it seems that distutils doesn't recognize '.zip' files as being part of packages, so if I have the following structure, twill/__init__.py twill/pyparsing.zip twill/mechanize.zip twill/shell.py twill/commands.py then packages = ['twill',] doesn't install the .zip files. Quixote has a simple solution that uses the cmdclass option to respecify things for (in their case) *.ptl files, and it works fine for .zip files. I'm pretty sure .zip files should be considered as valid as .py files, by default. Any thoughts? cheers, --titus From pje at telecommunity.com Sun May 22 00:49:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 21 May 2005 18:49:09 -0400 Subject: [Distutils] distutils bug. In-Reply-To: <20050521223430.GM31777@caltech.edu> Message-ID: <5.1.1.6.0.20050521184510.01d8ac98@mail.telecommunity.com> At 03:34 PM 5/21/2005 -0700, Titus Brown wrote: >Hi all, > >it seems that distutils doesn't recognize '.zip' files as being part of >packages, so if I have the following structure, > > twill/__init__.py > twill/pyparsing.zip > twill/mechanize.zip > twill/shell.py > twill/commands.py > >then packages = ['twill',] doesn't install the .zip files. With Python >=2.4, or with the setuptools package, you can add something like this to your setup options: setup( ... package_data = {'': ['*.zip']} ) And .zip files will be then installed. See: http://docs.python.org/dist/node11.html for documentation on the package_data option. (By the way, the behavior you describe is not a "bug"; it's distutils' current documented behavior.) From titus at caltech.edu Sun May 22 00:55:25 2005 From: titus at caltech.edu (Titus Brown) Date: Sat, 21 May 2005 15:55:25 -0700 Subject: [Distutils] distutils bug. In-Reply-To: <5.1.1.6.0.20050521184510.01d8ac98@mail.telecommunity.com> References: <20050521223430.GM31777@caltech.edu> <5.1.1.6.0.20050521184510.01d8ac98@mail.telecommunity.com> Message-ID: <20050521225525.GO31777@caltech.edu> -> >it seems that distutils doesn't recognize '.zip' files as being part of -> >packages, so if I have the following structure, -> > -> > twill/__init__.py -> > twill/pyparsing.zip -> > twill/mechanize.zip -> > twill/shell.py -> > twill/commands.py -> > -> >then packages = ['twill',] doesn't install the .zip files. -> -> With Python >=2.4, or with the setuptools package, you can add something -> like this to your setup options: -> -> setup( -> ... -> package_data = {'': ['*.zip']} -> ) -> -> And .zip files will be then installed. See: -> -> http://docs.python.org/dist/node11.html -> -> for documentation on the package_data option. Ahh, fantastic, thanks. I read up to section 2.4 but wasn't sure what to look for, so gave up & just read Quixote's code. -> (By the way, the behavior you describe is not a "bug"; it's distutils' -> current documented behavior.) It's hard to know how to respond to that... "MS Windows may occasionally crash." ;) --titus From ianb at colorstudy.com Sun May 22 20:07:49 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 22 May 2005 13:07:49 -0500 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> References: <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> Message-ID: <4290CA75.7040105@colorstudy.com> Phillip J. Eby wrote: >> Is it safe to do nasty monkeypatching wherein I replace distutils.setup >> with setuptools.setup? Or, more directly, anyone have code around to >> build eggs programmatically from other people's packages? > > > If distutils' run_setup() worked correctly, you could probably do this. > Of course, if you control your environment, you could also possibly just > copy bdist_egg.py into your Python installation's distutils/command > directory. I don't think bdist_egg has any dependencies to the rest of > setuptools that would prevent it from being used from regular distutils > that way. > > So, try copying bdist_egg.py into distutils/command, and then run > "setup.py bdist_egg" and see if it works. Ah, it's much easier than we realized: python setup.py --command-packages=setuptools.command bdist_egg -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Sun May 22 20:35:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 May 2005 14:35:53 -0400 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <4290CA75.7040105@colorstudy.com> References: <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050522142944.01d8aa28@mail.telecommunity.com> At 01:07 PM 5/22/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>>Is it safe to do nasty monkeypatching wherein I replace distutils.setup >>>with setuptools.setup? Or, more directly, anyone have code around to >>>build eggs programmatically from other people's packages? >> >>If distutils' run_setup() worked correctly, you could probably do this. >>Of course, if you control your environment, you could also possibly just >>copy bdist_egg.py into your Python installation's distutils/command >>directory. I don't think bdist_egg has any dependencies to the rest of >>setuptools that would prevent it from being used from regular distutils >>that way. >>So, try copying bdist_egg.py into distutils/command, and then run >>"setup.py bdist_egg" and see if it works. > >Ah, it's much easier than we realized: > > python setup.py --command-packages=setuptools.command bdist_egg Cool. Feel free to update the PythonEggs page on the PEAK Wiki if I don't get to it first. By the way, I've just checked in some enhancements to pkg_resources, particularly in the dependency resolution subsystem. 'require()' has a better algorithm now (less thorough, but more understandable and more computationally tractable), but is still not actually usable. However, there's now more explanation there of what pieces need to be finished in order for 'require()' to actually work. From pje at telecommunity.com Sun May 22 21:46:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 May 2005 15:46:19 -0400 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <4290CA75.7040105@colorstudy.com> References: <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050522154536.01d8b138@mail.telecommunity.com> At 01:07 PM 5/22/2005 -0500, Ian Bicking wrote: >Ah, it's much easier than we realized: > > python setup.py --command-packages=setuptools.command bdist_egg By the way, this only works with Python 2.4 and up. From ianb at colorstudy.com Sun May 22 21:51:37 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 22 May 2005 14:51:37 -0500 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <5.1.1.6.0.20050522154536.01d8b138@mail.telecommunity.com> References: <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> <5.1.1.6.0.20050522154536.01d8b138@mail.telecommunity.com> Message-ID: <4290E2C9.9000703@colorstudy.com> Phillip J. Eby wrote: > At 01:07 PM 5/22/2005 -0500, Ian Bicking wrote: > >> Ah, it's much easier than we realized: >> >> python setup.py --command-packages=setuptools.command bdist_egg > > > By the way, this only works with Python 2.4 and up. Blast, foiled again! Oh well, moving bdist_egg.py into distutils/command works (at least for 2.3). Since the whole point is to keep other people from having to run setup.py (by just distributing the eggs), that's not too big a deal. I guess this is the classic problem where standard library enhancements are more distracting than useful when you can't rely on a specific version of Python. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Sun May 22 22:49:02 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 May 2005 16:49:02 -0400 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <4290E2C9.9000703@colorstudy.com> References: <5.1.1.6.0.20050522154536.01d8b138@mail.telecommunity.com> <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> <5.1.1.6.0.20050519080255.01f27940@mail.telecommunity.com> <5.1.1.6.0.20050522154536.01d8b138@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050522164656.01d8be78@mail.telecommunity.com> At 02:51 PM 5/22/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 01:07 PM 5/22/2005 -0500, Ian Bicking wrote: >> >>>Ah, it's much easier than we realized: >>> >>> python setup.py --command-packages=setuptools.command bdist_egg >> >>By the way, this only works with Python 2.4 and up. > >Blast, foiled again! Oh well, moving bdist_egg.py into distutils/command >works (at least for 2.3). Since the whole point is to keep other people >from having to run setup.py (by just distributing the eggs), that's not >too big a deal. > >I guess this is the classic problem where standard library enhancements >are more distracting than useful when you can't rely on a specific version >of Python. By the way, I noticed from your edit that you're having problems with "some packages"; here's my revision to that section, which might help you out with that: """Note: packages that expect to find data files in their package directories, but which do not use either the PEP 302 API or the ``pkg_resources`` API to find them will *not* work when packaged as .egg files. One way you can check for this is if the .egg file contains data files, and the package is using ``__file__`` to find them. You'll need to then patch the package so it uses ``pkg_resources.resource_filename()`` or one of the other ``resource_*`` APIs instead of ``__file__``. Also, some packages (e.g. wxPython) may include dynamic link libraries other than Python extensions. If this is the case, you'll need to create an ``eager_resources.txt`` file in the ``.egg-info`` directory that lists the in-zip paths to these libraries, one per line. This will let the runtime system know that those files need to be unpacked if any of the extensions are used. Thus, when attempting to import an extension, the runtime will also unpack all the dynamic link libraries that go with it. (Note: if you still can't get the library to work as an .egg file after trying the above tactics, please report your problem on the distutils-Sig mailing list. Thanks.)""" From pje at telecommunity.com Mon May 23 04:32:39 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 May 2005 22:32:39 -0400 Subject: [Distutils] Questions about Python Eggs In-Reply-To: <428C1350.7080608@colorstudy.com> Message-ID: <5.1.1.6.0.20050522220045.01d8a2e8@mail.telecommunity.com> At 11:17 PM 5/18/2005 -0500, Ian Bicking wrote: >I notice the comment on pkg_resources.require aren't very confident ;) >It actually doesn't look functional to me, though I haven't tried >running it. Is it just meant to raise an ImportError when a requirement >isn't met, or can it search some directories for appropriate Egg files? > This is something I'm very interested in, so if I have the intention >correct I'd like to help move this function along. Just a quick followup on this; I've just checked in a version of pkg_resources whose 'require()' API is only missing a working 'find_distributions(path_item)' function. If you'd like to experiment with this, you can try implementing your own 'find_distributions' and monkeypatching it into 'pkg_resources'. The routine should take a sys.path entry (i.e. a string) and yield zero or more pkg_resource.Distribution instances representing distributions found in the supplied directory, zipfile, or whatever. The easiest way to create these Distribution instances is using the Distribution.from_filename constructor, which takes care of figuring out the distribution's platform, name, version, Python version, etc. from the full path to the file. You'll also need to supply a 'metadata' object to each Distribution, so it can find its dependency list. In the setuptools.tests.test_resources module there's a mock Metadata implementation you can use; it expects to receive filenames (like 'depends.txt') and return the contents of the corresponding metadata file (from either the .egg file's EGG-INFO directory, or from an unpacked PackageName.egg-info directory). So anyway, the dependency resolution subsystem is now basically working, it's just that it currently lacks the ability to actually scan for .egg files and .egg-info directories and get their metadata. It'll raise DistributionNotFound if you try to 'require()' anything, because of this current lack of scanning ability. The full implementation of 'find_distributions(path_item)' will tie into the PEP 302 import framework, so that sys.path entries representing zip files will also be usable. This is important because if a '.egg' file is manually placed on sys.path (e.g. via PYTHONPATH), the dependency system still needs to know about it. Thus, calling 'find_distributions("/path/to/an.egg")' should yield a Distribution object for the egg, whose metadata comes from the egg file's "EGG-INFO" directory. Calling 'find_distributions("/some/dir")' should yield a Distribution for each .egg file in the directory, and for each .egg-info subdirectory. The main difference between the two is that 'path_item' is the Distribution.path for an .egg-info, whereas the .egg file's absolute path is its Distribution.path. In other words, if you find an .egg-info directory, it's because the egg in question is *already* on sys.path, but if you find a .egg file in the path_item directory, it's an egg that's not (necessarily) on sys.path. (Of course, if path_item is the path to an .egg file, then it is of course already on sys.path.) Whew. Anyway, I probably won't get around to writing the full, correct, PEP 302-integrating 'find_distributions()' until next weekend, so if you want to experiment with 'require()' in the meantime you might take a whack at implementing a simpler version of 'find_distributions()', if you're interested. From richardjones at optushome.com.au Thu May 26 04:50:15 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Thu, 26 May 2005 12:50:15 +1000 Subject: [Distutils] Distutils projects page Message-ID: <200505261250.16115.richardjones@optushome.com.au> In an effort to try to understand what projects are out there trying to improve distutils / packaging / make the world a better place, I've put up a wiki page: http://wiki.python.org/moin/DistutilsProjects It contains all the projects I'm aware of. I'm not on the distutils SIG list, so I might have missed some. I did scan the archive for this year, and didn't see anything. Please visit the page and edit it for correctness - or include your project if it's not in there! Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050526/cad222d5/attachment.pgp From fdrake at gmail.com Thu May 26 19:04:15 2005 From: fdrake at gmail.com (Fred Drake) Date: Thu, 26 May 2005 13:04:15 -0400 Subject: [Distutils] [Catalog-sig] Distutils projects page In-Reply-To: <200505261250.16115.richardjones@optushome.com.au> References: <200505261250.16115.richardjones@optushome.com.au> Message-ID: <9cee7ab805052610046c0d20e0@mail.gmail.com> On 5/25/05, Richard Jones wrote: > In an effort to try to understand what projects are out there trying to > improve distutils / packaging / make the world a better place, I've put up a > wiki page: > > http://wiki.python.org/moin/DistutilsProjects I couldn't edit this page; it looks like we're still having wiki problems on python.org. I was going to add a link to the zpkg pages: http://www.zope.org/Members/fdrake/zpkgtools/ The pages there need to be updated; I'll see about getting that done in the next couple of days. (I'm on travel now.) -Fred -- Fred L. Drake, Jr. From nick.bastin at gmail.com Fri May 27 00:47:09 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Thu, 26 May 2005 18:47:09 -0400 Subject: [Distutils] non-"standard" compilers... Message-ID: <66d0a6e105052615475d97dafc@mail.gmail.com> I am in serious need of some help (mental?) with building a distutils-distributed python extension (PyXML). I am on Solaris 8, with the SunPro Forte 10 compiler. I tried the following command line, thinking that it was reasonable: python setup.py build --compiler=unix but the definition of "standard UNIX-style compiler" appears to be 'gcc'. Although I don't quite understand that...the dictionary in unixcompiler.py is initialized to the compiler being 'cc', which would be the Right Thing(tm) on quite a lot of unix platforms, including Solaris. However, by the time the compile command is actually issued on the command line, it's trying to use gcc. I don't follow quite how distutils decides what compiler to use, so I was wondering if someone could point me in the right direction to build with SunPro instead of gcc. For extra credit, and help with my vast amount of mental anguish, someone could try to tell how to build with the Intel C Compiler on windows... -- Nick From pje at telecommunity.com Fri May 27 02:33:41 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 May 2005 20:33:41 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e105052615475d97dafc@mail.gmail.com> Message-ID: <5.1.1.6.0.20050526203312.01fbfc28@mail.telecommunity.com> At 06:47 PM 5/26/2005 -0400, Nicholas Bastin wrote: >I am in serious need of some help (mental?) with building a >distutils-distributed python extension (PyXML). > >I am on Solaris 8, with the SunPro Forte 10 compiler. I tried the >following command line, thinking that it was reasonable: > >python setup.py build --compiler=unix > >but the definition of "standard UNIX-style compiler" appears to be >'gcc'. Although I don't quite understand that...the dictionary in >unixcompiler.py is initialized to the compiler being 'cc', which would >be the Right Thing(tm) on quite a lot of unix platforms, including >Solaris. However, by the time the compile command is actually issued >on the command line, it's trying to use gcc. I don't follow quite how >distutils decides what compiler to use, so I was wondering if someone >could point me in the right direction to build with SunPro instead of >gcc. Have you tried setting the CC environment variable? From mal at egenix.com Fri May 27 10:03:41 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2005 10:03:41 +0200 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e105052615475d97dafc@mail.gmail.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> Message-ID: <4296D45D.1040304@egenix.com> Nicholas Bastin wrote: > I am in serious need of some help (mental?) with building a > distutils-distributed python extension (PyXML). > > I am on Solaris 8, with the SunPro Forte 10 compiler. I tried the > following command line, thinking that it was reasonable: > > python setup.py build --compiler=unix > > but the definition of "standard UNIX-style compiler" appears to be > 'gcc'. Although I don't quite understand that...the dictionary in > unixcompiler.py is initialized to the compiler being 'cc', which would > be the Right Thing(tm) on quite a lot of unix platforms, including > Solaris. However, by the time the compile command is actually issued > on the command line, it's trying to use gcc. I don't follow quite how > distutils decides what compiler to use, so I was wondering if someone > could point me in the right direction to build with SunPro instead of > gcc. distutils picks up the compiler default from the settings Python was compiled with. It is usually not a good idea to compile extensions with a different compiler, hence this default. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 27 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From nick.bastin at gmail.com Fri May 27 18:15:04 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Fri, 27 May 2005 12:15:04 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <4296D45D.1040304@egenix.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> Message-ID: <66d0a6e105052709156e7012e5@mail.gmail.com> On 5/27/05, M.-A. Lemburg wrote: > Nicholas Bastin wrote: > > I am in serious need of some help (mental?) with building a > > distutils-distributed python extension (PyXML). > > > > I am on Solaris 8, with the SunPro Forte 10 compiler. I tried the > > following command line, thinking that it was reasonable: > > > > python setup.py build --compiler=unix > > > > but the definition of "standard UNIX-style compiler" appears to be > > 'gcc'. Although I don't quite understand that...the dictionary in > > unixcompiler.py is initialized to the compiler being 'cc', which would > > be the Right Thing(tm) on quite a lot of unix platforms, including > > Solaris. However, by the time the compile command is actually issued > > on the command line, it's trying to use gcc. I don't follow quite how > > distutils decides what compiler to use, so I was wondering if someone > > could point me in the right direction to build with SunPro instead of > > gcc. > > distutils picks up the compiler default from the settings > Python was compiled with. I don't have gcc, so it's really not possible that I built Python with gcc. Where does distutils try to pick this up from? Also, I'm assuming this is only true on unix? Win32 seems to pick MSVC no matter what (and complain a lot if you don't have it). -- Nick From nick.bastin at gmail.com Fri May 27 18:24:48 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Fri, 27 May 2005 12:24:48 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <4296D45D.1040304@egenix.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> Message-ID: <66d0a6e105052709244e6b7a63@mail.gmail.com> On 5/27/05, M.-A. Lemburg wrote: > distutils picks up the compiler default from the settings > Python was compiled with. > > It is usually not a good idea to compile extensions with a > different compiler, hence this default. Ah, a little more tweaking reveals what is going on here. distutils reads from pyconfig.h (a thing that might be useful to put in the doc), which, in my case, the one it finds isn't for the python on my system (or even my OS, actually). We ship a cross-platform distribution with our application, so we have to rename pyconfig.h as necessary for each platform, so the correct one for my platform (SUN_SPARC_SOLAR/pyconfig_32.h) isn't being found by distutils. Is there any way to tell distutils where my pyconfig.h is? I can of course eliminate the spurious pyconfig.h that's floating around, but then I just get an IOError when it tries to open the location it thinks is pyconfig.h. -- Nick From pje at telecommunity.com Fri May 27 18:34:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 27 May 2005 12:34:53 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e105052709156e7012e5@mail.gmail.com> References: <4296D45D.1040304@egenix.com> <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> Message-ID: <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> At 12:15 PM 5/27/2005 -0400, Nicholas Bastin wrote: >Also, I'm assuming this is only true on unix? Win32 seems to pick >MSVC no matter what (and complain a lot if you don't have it). That's because MSVC is the only supported compiler for Python on that platform. There has been some work on supporting the MinGW compiler, and the MinGW compiler can be used to build extensions that work on Windows, but nobody has done any work on supporting any other compilers that I know of. From nick.bastin at gmail.com Fri May 27 18:41:41 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Fri, 27 May 2005 12:41:41 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709156e7012e5@mail.gmail.com> <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> Message-ID: <66d0a6e1050527094166b4dc7c@mail.gmail.com> On 5/27/05, Phillip J. Eby wrote: > At 12:15 PM 5/27/2005 -0400, Nicholas Bastin wrote: > >Also, I'm assuming this is only true on unix? Win32 seems to pick > >MSVC no matter what (and complain a lot if you don't have it). > > That's because MSVC is the only supported compiler for Python on that > platform. There has been some work on supporting the MinGW compiler, and > the MinGW compiler can be used to build extensions that work on Windows, > but nobody has done any work on supporting any other compilers that I know of. The Intel C++ compiler works perfectly well (we build and ship using this compiler). However, because we do this, we can't use any distutils-distributed extension modules, because they complain that we don't have the .NET runtime or some such. I usually just try to construct makefiles for the extension modules in each package, and that works reasonably well for most extensions. Also, does distutils support the notion of installation a 'FAT' distribution? We also have to tear each install apart to put the .py files in a platform independent place, and the .pyd's in a platform-specific location, which usually involves a lot of magic tricks when the .pyd's are imported as part of a package. -- Nick From bob at redivi.com Fri May 27 19:10:51 2005 From: bob at redivi.com (Bob Ippolito) Date: Fri, 27 May 2005 10:10:51 -0700 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e1050527094166b4dc7c@mail.gmail.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709156e7012e5@mail.gmail.com> <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> <66d0a6e1050527094166b4dc7c@mail.gmail.com> Message-ID: <4DDBEF7A-8FF3-4AAE-90D8-5B7122A82F42@redivi.com> On May 27, 2005, at 9:41 AM, Nicholas Bastin wrote: > On 5/27/05, Phillip J. Eby wrote: > >> At 12:15 PM 5/27/2005 -0400, Nicholas Bastin wrote: >> >>> Also, I'm assuming this is only true on unix? Win32 seems to pick >>> MSVC no matter what (and complain a lot if you don't have it). >>> >> >> That's because MSVC is the only supported compiler for Python on that >> platform. There has been some work on supporting the MinGW >> compiler, and >> the MinGW compiler can be used to build extensions that work on >> Windows, >> but nobody has done any work on supporting any other compilers >> that I know of. >> > > The Intel C++ compiler works perfectly well (we build and ship using > this compiler). However, because we do this, we can't use any > distutils-distributed extension modules, because they complain that we > don't have the .NET runtime or some such. I usually just try to > construct makefiles for the extension modules in each package, and > that works reasonably well for most extensions. > > Also, does distutils support the notion of installation a 'FAT' > distribution? We also have to tear each install apart to put the .py > files in a platform independent place, and the .pyd's in a > platform-specific location, which usually involves a lot of magic > tricks when the .pyd's are imported as part of a package. No, it doesn't. I've done this too, it basically involves the same trick that py2app and ilk use to put an extension module in a zip file. Replace the extension with a .py that knows where to look for a platform-specific extension to import. Or, simply have an 'OBESE' distribution where you have a copy of everything for every platform :) -bob From pje at telecommunity.com Fri May 27 19:17:24 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 27 May 2005 13:17:24 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e1050527094166b4dc7c@mail.gmail.com> References: <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709156e7012e5@mail.gmail.com> <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050527131143.01fd8d20@mail.telecommunity.com> At 12:41 PM 5/27/2005 -0400, Nicholas Bastin wrote: >On 5/27/05, Phillip J. Eby wrote: > > At 12:15 PM 5/27/2005 -0400, Nicholas Bastin wrote: > > >Also, I'm assuming this is only true on unix? Win32 seems to pick > > >MSVC no matter what (and complain a lot if you don't have it). > > > > That's because MSVC is the only supported compiler for Python on that > > platform. There has been some work on supporting the MinGW compiler, and > > the MinGW compiler can be used to build extensions that work on Windows, > > but nobody has done any work on supporting any other compilers that I > know of. > >The Intel C++ compiler works perfectly well (we build and ship using >this compiler). However, because we do this, we can't use any >distutils-distributed extension modules, because they complain that we >don't have the .NET runtime or some such. I usually just try to >construct makefiles for the extension modules in each package, and >that works reasonably well for most extensions. Perhaps you should consider contributing a Compiler class to the distutils to resolve this. Also, I'm not sure if you know about the distutils.cfg file, but once you have a compiler class set up, you can make it the default with the appropriate entries in that configuration file, so that you don't even need to specify the -c/--compiler option. >Also, does distutils support the notion of installation a 'FAT' >distribution? We also have to tear each install apart to put the .py >files in a platform independent place, and the .pyd's in a >platform-specific location, which usually involves a lot of magic >tricks when the .pyd's are imported as part of a package. There are --install-platlib and --install-purelib options that can be set to separate the two, but recombining them is tricky, as you point out. You might be interested in the Python Eggs project, which allows multiple platforms (and multiple versions) of the same library to live side-by-side in site-packages or elsewhere, but get selected at runtime. You do, however, have to invoke some code at application startup to request activation of the desired library, but this can be simple as: from pkg_resources import require require("MyApplication") If you've defined an egg for MyApplication that has a file listing the other libraries and versions that it depends on. Platform selection is more or less automatic, although it's based currently on the distutils' idea of what a "platform" is, so that may or may not work for your needs. From nick.bastin at gmail.com Fri May 27 20:01:00 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Fri, 27 May 2005 14:01:00 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <5.1.1.6.0.20050527131143.01fd8d20@mail.telecommunity.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709156e7012e5@mail.gmail.com> <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> <66d0a6e1050527094166b4dc7c@mail.gmail.com> <5.1.1.6.0.20050527131143.01fd8d20@mail.telecommunity.com> Message-ID: <66d0a6e10505271101237490d0@mail.gmail.com> On 5/27/05, Phillip J. Eby wrote: > At 12:41 PM 5/27/2005 -0400, Nicholas Bastin wrote: > >The Intel C++ compiler works perfectly well (we build and ship using > >this compiler). However, because we do this, we can't use any > >distutils-distributed extension modules, because they complain that we > >don't have the .NET runtime or some such. I usually just try to > >construct makefiles for the extension modules in each package, and > >that works reasonably well for most extensions. > > Perhaps you should consider contributing a Compiler class to the distutils > to resolve this. I thought about this, but I'm unclear as to how to get past the error: 'The .NET Framework SDK needs to be installed before building extensions for Python' It isn't the msvccompiler class that checks for this. Does anybody know where I can bypass this (or apply a slightly more nuanced check)? -- Nick From nick.bastin at gmail.com Fri May 27 20:03:33 2005 From: nick.bastin at gmail.com (Nicholas Bastin) Date: Fri, 27 May 2005 14:03:33 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e10505271101237490d0@mail.gmail.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709156e7012e5@mail.gmail.com> <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> <66d0a6e1050527094166b4dc7c@mail.gmail.com> <5.1.1.6.0.20050527131143.01fd8d20@mail.telecommunity.com> <66d0a6e10505271101237490d0@mail.gmail.com> Message-ID: <66d0a6e105052711033581d1a1@mail.gmail.com> > I thought about this, but I'm unclear as to how to get past the error: > > 'The .NET Framework SDK needs to be installed before building > extensions for Python' > > It isn't the msvccompiler class that checks for this. Does anybody > know where I can bypass this (or apply a slightly more nuanced check)? Wow, I'm incredibly blind. This message is in the msvccompiler class.. :-) I'll see what I can do about providing an iclcompiler class for win32 (the compiler is supported on linux as well, but I don't have that version). -- Nick From seefeld at sympatico.ca Fri May 27 20:27:15 2005 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Fri, 27 May 2005 14:27:15 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e105052711033581d1a1@mail.gmail.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709156e7012e5@mail.gmail.com> <5.1.1.6.0.20050527123308.01facfe8@mail.telecommunity.com> <66d0a6e1050527094166b4dc7c@mail.gmail.com> <5.1.1.6.0.20050527131143.01fd8d20@mail.telecommunity.com> <66d0a6e10505271101237490d0@mail.gmail.com> <66d0a6e105052711033581d1a1@mail.gmail.com> Message-ID: <42976683.1040006@sympatico.ca> Nicholas Bastin wrote: > I'll see what I can do about providing an iclcompiler class for win32 > (the compiler is supported on linux as well, but I don't have that > version). There once was an efford to work towards some 'distutils 2.0', what happened to that ? I'v been using distutils for years, but due to a number of its limitations I'm mostly using custom command classes. In particular, I find the way distutils deals with extension modules rather inflexible. Providing custom compilers isn't enough, and so I more or less bypass the distutils 'Extension' concept entirely, so I can run 'make' or 'scons' for subprojects as needed. I had hoped that at some point distutils would get redesigned to make it more flexible in particular in those areas. Thanks, Stefan From anthony at interlink.com.au Sat May 28 13:26:39 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 28 May 2005 21:26:39 +1000 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <66d0a6e105052709244e6b7a63@mail.gmail.com> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <4296D45D.1040304@egenix.com> <66d0a6e105052709244e6b7a63@mail.gmail.com> Message-ID: <200505282126.42281.anthony@interlink.com.au> On Saturday 28 May 2005 02:24, Nicholas Bastin wrote: > Ah, a little more tweaking reveals what is going on here. distutils > reads from pyconfig.h (a thing that might be useful to put in the > doc), which, in my case, the one it finds isn't for the python on my > system (or even my OS, actually). We ship a cross-platform > distribution with our application, so we have to rename pyconfig.h as > necessary for each platform, so the correct one for my platform > (SUN_SPARC_SOLAR/pyconfig_32.h) isn't being found by distutils. Is > there any way to tell distutils where my pyconfig.h is? > > I can of course eliminate the spurious pyconfig.h that's floating > around, but then I just get an IOError when it tries to open the > location it thinks is pyconfig.h. If distutils doesn't provide a way to hook this, consider writing a "find the pyconfig.h" method that can be overridden in distutils.cfg - then it will just work for you. I don't think distutils currently has support for this... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Sat May 28 13:28:50 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 28 May 2005 21:28:50 +1000 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <42976683.1040006@sympatico.ca> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <66d0a6e105052711033581d1a1@mail.gmail.com> <42976683.1040006@sympatico.ca> Message-ID: <200505282128.52566.anthony@interlink.com.au> On Saturday 28 May 2005 04:27, Stefan Seefeld wrote: > Nicholas Bastin wrote: > > I'll see what I can do about providing an iclcompiler class for win32 > > (the compiler is supported on linux as well, but I don't have that > > version). > > There once was an efford to work towards some 'distutils 2.0', > what happened to that ? I don't think anything's happened with it. Before Python 2.5, I plan to go through distutils and remove a bunch of things that are now part of the stdlib - for instance, making distutils use logging and optparse, rather than it's own log and fancy_getopt modules. If you wanted to try writing down exactly what you'd like to see from a newer Extension object in distutils, that would be a good start... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From seefeld at sympatico.ca Sat May 28 19:47:48 2005 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Sat, 28 May 2005 13:47:48 -0400 Subject: [Distutils] non-"standard" compilers... In-Reply-To: <200505282128.52566.anthony@interlink.com.au> References: <66d0a6e105052615475d97dafc@mail.gmail.com> <66d0a6e105052711033581d1a1@mail.gmail.com> <42976683.1040006@sympatico.ca> <200505282128.52566.anthony@interlink.com.au> Message-ID: <4298AEC4.5000909@sympatico.ca> Anthony Baxter wrote: > On Saturday 28 May 2005 04:27, Stefan Seefeld wrote: > If you wanted to try writing down exactly what you'd like to see from a > newer Extension object in distutils, that would be a good start... A good starting point would be to make the Extension object polymorphic, i.e. provide hooks for users to define their own 'config', 'build', 'install', etc. methods. Right now it is a mere container of source files and compilation options, which isn't flexible enough to accomodate for any advanced cases. Then it should be even possible (and easy) to provide subclasses for the most typical scenarios, such as an autoconf/make wrapper, another for scons, bjam (boost.python !), etc. Does this make sense ? Regards, Stefan From pje at telecommunity.com Sun May 29 06:57:02 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 00:57:02 -0400 Subject: [Distutils] EasyInstall - automated download/install/manage Python packages Message-ID: <5.1.1.6.0.20050529005157.02a07d88@mail.telecommunity.com> Just a heads-up: http://peak.telecommunity.com/DevCenter/EasyInstall I would like to encourage package authors to try using EasyInstall to download and build the packages they distribute, and to let me know if they encounter any issues. I'd like to make EasyInstall able to build as many Python packages "out of the box" as possible. I've also been updating the Python Eggs documentation: http://peak.telecommunity.com/DevCenter/PythonEggs But there is still a fair amount to write about the runtime API, and as you can see from the "Implementation Status" there are plenty of tasks still "available" with respect to getting the zip-based runtime as robust as it should be. From ianb at colorstudy.com Sun May 29 21:46:12 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 14:46:12 -0500 Subject: [Distutils] EasyInstall: svn support Message-ID: <429A1C04.8000703@colorstudy.com> I just tried out EasyInstall, and it seems to work pretty well. Well, I've only tried downloading packages, not really using them, but I'll try that next... I added support to download from subversion repositories. Basically it detects the svn index page (for http repositories, which can't be detected based on the URL), or the svn: URL type, and downloads from there. I'd like for it to fix up the version based on the svn revision, but I haven't looked closely enough at that part yet, or if it's even possible since setup.py typically has a version hardcoded in it. I'm not even sure what the version number should look like, so that it sorts properly with released versions (or even if it should sort with released versions at all; should versions also indicate branches, like stable vs. development?) Anyway, I've attached a diff against 0.3a1 and the compete modified easy_install.py file. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org -------------- next part -------------- A non-text attachment was scrubbed... Name: easy_install.py Type: text/x-python Size: 19304 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050529/7c12c20a/easy_install.py -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: easy_install_svn.diff Url: http://mail.python.org/pipermail/distutils-sig/attachments/20050529/7c12c20a/easy_install_svn.diff From pje at telecommunity.com Sun May 29 22:49:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 16:49:29 -0400 Subject: [Distutils] EasyInstall: svn support In-Reply-To: <429A1C04.8000703@colorstudy.com> Message-ID: <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> At 02:46 PM 5/29/2005 -0500, Ian Bicking wrote: >I just tried out EasyInstall, and it seems to work pretty well. Well, >I've only tried downloading packages, not really using them, but I'll try >that next... > >I added support to download from subversion repositories. Thanks! I was going to suggest you might want to look at contributing that. :) > Basically it detects the svn index page (for http repositories, which > can't be detected based on the URL), or the svn: URL type, and downloads > from there. I'd like for it to fix up the version based on the svn > revision, but I haven't looked closely enough at that part yet, or if > it's even possible since setup.py typically has a version hardcoded in > it. I'm not even sure what the version number should look like, so that > it sorts properly with released versions (or even if it should sort with > released versions at all; should versions also indicate branches, like > stable vs. development?) Yeah, I think sticking with the setup.py version is best; EasyInstall and pkg_resources use the generated PKG-INFO to find out stuff like that. >Anyway, I've attached a diff against 0.3a1 and the compete modified >easy_install.py file. I was a bit puzzled by the diff at first; looks like you created a reversed diff. >--- easy_install.py 2005-05-29 14:30:03.858797032 -0500 >+++ orig/setuptools-0.3a1/easy_install.py 2005-05-28 >21:36:42.000000000 -0500 >@@ -154,7 +154,6 @@ > """ > > import sys, os.path, pkg_resources, re, zipimport >-import shutil > from pkg_resources import * > > >@@ -220,7 +219,7 @@ > if isinstance(spec,Requirement): > pass > else: >- scheme = URL_SCHEME(spec).group(1) This won't work with non-URL specs. Note that spec is allowed to be a local filename, or a version requirement (e.g. "FooPackage==1.2"). Anyway, I moved the group() to the _download_url() call, where it should accomplish the intent here. As for this next bit below, you're just skipping the archive checks if the source is a directory, right? There aren't any changes to the archive handling parts? > # Anything else, try to extract and build >- if not os.path.isdir(dist_filename): >- import zipfile, tarfile >- if zipfile.is_zipfile(dist_filename): >- self._extract_zip(dist_filename, self.tmpdir) >+ import zipfile, tarfile >+ if zipfile.is_zipfile(dist_filename): >+ self._extract_zip(dist_filename, self.tmpdir) >+ else: >+ import tarfile >+ try: >+ tar = tarfile.open(dist_filename) >+ except tarfile.TarError: >+ raise RuntimeError( >+ "Not a valid tar or zip archive: %s" % dist_filename >+ ) > else: >- import tarfile >- try: >- tar = tarfile.open(dist_filename) >- except tarfile.TarError: >- raise RuntimeError( >- "Not a valid tar or zip archive: %s" % dist_filename >- ) >- else: >- self._extract_tar(tar) >+ self._extract_tar(tar) > > # Find the setup.py file > from glob import glob >@@ -374,7 +372,7 @@ > destination = os.path.join(self.instdir, os.path.basename(egg_path)) > ensure_directory(destination) >- if not os.path.exists(destination) or not self.samefile(egg_path, >destination): >+ if not self.samefile(egg_path, destination): I don't understand what this change is for. Since install_egg is only called with an existing .egg file or directory, this means that if the destination doesn't exist, it can't be the same file. So, the addition here seems redundant. >- # Check if it is a subversion index page: Before doing this, I think it would be a good idea to check the headers for a text/html Content-Type, so as not to be readline() a big chunk of binary. :) >- def _download_svn(self, scheme, url, filename): >- os.system("svn checkout -q %s %s" % (url, filename)) Why checkout instead of export? Oh, I see, it's so you can use svn info later. Never mind. >- # Actually, this probably doesn't do anything; but somehow this >- # information should get put into the version string... It indeed doesn't do anything; you'd be better off munging the setup.py to change the setup version. Maybe what I can do is have the build stuff look for a .svn dir in the build area, and if one is present, grab the revision and add it to the end of the version string supplied to setup(). Or simpler still, I could add a '--tag-svn-revision' option to bdist_egg, that would make *it* do that. I've been thinking it would be nice to have a '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, to support daily builds and such. So adding an option to "get the tag value from 'svn info'" would just be a minor variation on the theme. Anyway, do you have some handy svn: and http: subversion URLs that I could play with for testing? From ianb at colorstudy.com Sun May 29 22:56:49 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 15:56:49 -0500 Subject: [Distutils] EasyInstall: scripts Message-ID: <429A2C91.3020902@colorstudy.com> I'm wondering how scripts should be handled by easyinstall -- right now they pretty much get ignored. But I'm not sure how they should be handled at all. Distutils has a couple rules, with --install-scripts, and --home (which uses ~/bin), and the normal rules... and then some translation of those options to Windows. But none of those seems to translate very well to easyinstall. Or eggs. I don't have any ideas myself on this. Except that the scripts themselves are usually pretty dumb anyway, just thin wrappers that fix up sys.path, import a module, and call a function; so the status quo could probably be improved upon anyway. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Sun May 29 23:26:08 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 16:26:08 -0500 Subject: [Distutils] EasyInstall: svn support In-Reply-To: <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> References: <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> Message-ID: <429A3370.7020008@colorstudy.com> Phillip J. Eby wrote: >> Basically it detects the svn index page (for http repositories, >> which can't be detected based on the URL), or the svn: URL type, and >> downloads from there. I'd like for it to fix up the version based on >> the svn revision, but I haven't looked closely enough at that part >> yet, or if it's even possible since setup.py typically has a version >> hardcoded in it. I'm not even sure what the version number should >> look like, so that it sorts properly with released versions (or even >> if it should sort with released versions at all; should versions also >> indicate branches, like stable vs. development?) > > > Yeah, I think sticking with the setup.py version is best; EasyInstall > and pkg_resources use the generated PKG-INFO to find out stuff like that. Well, at least for my own packages I'll try to make the version better, by dynamically generating it in setup.py; I tried to do this a little with Paste, but there's some improvement to be done there as well. What form should it take? E.g., if the last revision was 0.1, what should the trunk version be? 0.2alpha-svn2822? Does the released version need to be 0.2final then? Or do are version numbers sorted based on a better algorithm than string comparison? >> Anyway, I've attached a diff against 0.3a1 and the compete modified >> easy_install.py file. > > > I was a bit puzzled by the diff at first; looks like you created a > reversed diff. Oops. I have a hard time with diff for some reason. Like some kind of chronological dyslexia. >> --- easy_install.py 2005-05-29 14:30:03.858797032 -0500 >> +++ orig/setuptools-0.3a1/easy_install.py 2005-05-28 >> 21:36:42.000000000 -0500 >> @@ -154,7 +154,6 @@ >> """ >> >> import sys, os.path, pkg_resources, re, zipimport >> -import shutil >> from pkg_resources import * >> >> >> @@ -220,7 +219,7 @@ >> if isinstance(spec,Requirement): >> pass >> else: >> - scheme = URL_SCHEME(spec).group(1) > > > This won't work with non-URL specs. Note that spec is allowed to be a > local filename, or a version requirement (e.g. "FooPackage==1.2"). > Anyway, I moved the group() to the _download_url() call, where it should > accomplish the intent here. OK. Or it could be: scheme = URL_SCHEME(spec) scheme = scheme and scheme.group(1) It seems like a match object is a little odd to pass around. > As for this next bit below, you're just skipping the archive checks if > the source is a directory, right? There aren't any changes to the > archive handling parts? Yes, that's all -- since svn co creates a directory, it doesn't need to be unpacked. >> - if not os.path.exists(destination) or not >> self.samefile(egg_path, destination): >> + if not self.samefile(egg_path, destination): > > > I don't understand what this change is for. Since install_egg is only > called with an existing .egg file or directory, this means that if the > destination doesn't exist, it can't be the same file. So, the addition > here seems redundant. I was getting an error when calling self.samefile(filename, filename_that_doesnt_exist). Maybe self.samefile needs to check os.path.exists(destination). >> - # Check if it is a subversion index page: > > > Before doing this, I think it would be a good idea to check the headers > for a text/html Content-Type, so as not to be readline() a big chunk of > binary. :) Ah, yes, indeed. >> - def _download_svn(self, scheme, url, filename): >> - os.system("svn checkout -q %s %s" % (url, filename)) > > > Why checkout instead of export? Oh, I see, it's so you can use svn info > later. Never mind. Yeah, not a big reason. Though it would be nice if, based on some option, it could be an "svn up" instead of a checkout, if there was some cache of checkouts. And annoyingly, you can't use svn info on a remote repository; the only way I've figured to get the last changed revision of a remote repository is with "svn ls -v parent_dir". >> - # Actually, this probably doesn't do anything; but somehow this >> - # information should get put into the version string... > > > It indeed doesn't do anything; you'd be better off munging the setup.py > to change the setup version. > > Maybe what I can do is have the build stuff look for a .svn dir in the > build area, and if one is present, grab the revision and add it to the > end of the version string supplied to setup(). Or simpler still, I > could add a '--tag-svn-revision' option to bdist_egg, that would make > *it* do that. I've been thinking it would be nice to have a > '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, to > support daily builds and such. So adding an option to "get the tag > value from 'svn info'" would just be a minor variation on the theme. Yes, that would work well. > Anyway, do you have some handy svn: and http: subversion URLs that I > could play with for testing? svn://colorstudy.com/trunk/SQLObject and http://svn.colorstudy.com/trunk/SQLObject should both be workable, and point to the same location. It should really look for a scheme of 'svn+ssh' as well, but those are a pain to make readable ;) And technically file: should work for svn too, which I suppose you could detect because it points to a directory with a db/fs-type file. Eh, those can wait. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Sun May 29 23:49:17 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 16:49:17 -0500 Subject: [Distutils] EasyInstall: easy-install.pth, active version Message-ID: <429A38DD.70703@colorstudy.com> Reading through "Changing the Active Version (site-packages installs only)", I'm guessing that the implementation changes easy-install.pth to point to a specific version. But no easy-install.pth is created when you use the --install-dir option. And maybe that's just an oversight... it seems like it easy-install.pth makes just as much sense in a local directory as site-packages...? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 00:06:57 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 18:06:57 -0400 Subject: [Distutils] EasyInstall: svn support In-Reply-To: <429A3370.7020008@colorstudy.com> References: <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050529174624.021f0ff8@mail.telecommunity.com> At 04:26 PM 5/29/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>> Basically it detects the svn index page (for http repositories, which >>> can't be detected based on the URL), or the svn: URL type, and >>> downloads from there. I'd like for it to fix up the version based on >>> the svn revision, but I haven't looked closely enough at that part yet, >>> or if it's even possible since setup.py typically has a version >>> hardcoded in it. I'm not even sure what the version number should look >>> like, so that it sorts properly with released versions (or even if it >>> should sort with released versions at all; should versions also >>> indicate branches, like stable vs. development?) >> >>Yeah, I think sticking with the setup.py version is best; EasyInstall and >>pkg_resources use the generated PKG-INFO to find out stuff like that. > >Well, at least for my own packages I'll try to make the version better, by >dynamically generating it in setup.py; Note that it needs to be robust if the build machine doesn't have svn. > I tried to do this a little with Paste, but there's some improvement to > be done there as well. What form should it take? > >E.g., if the last revision was 0.1, what should the trunk version be? >0.2alpha-svn2822? Does the released version need to be 0.2final then? Or >do are version numbers sorted based on a better algorithm than string >comparison? See pkg_resources.parse_version() for the version translator. But since the algorithm is a bit obscure, here are the basic ideas: * The string is converted to a tuple of strings * Runs of numbers are split from non-numeric runs * The string "*final" is added to the end of the tuple * Dots are not kept in the string; only '-' and non-numerics * Certain alpha strings are tweaked; pre, rc, and so on. * Numeric runs are zero-padded for numeric comparision * Elements that equal zero are dropped from each tuple-level numeric run In other words, the revision system knows that '2.0' and '2.0.0' are equivalent version numbers. It knows that alpha, beta, and candidate releases come *before* final releases, but that 'nb', 'pl' releases are *after*, as are '-' releases. So, it knows that '2.0-1' is an "older" release than '2.0.1'. Thus, the best and simplest way to add build/date/revision tags is just to pop a '-' and the extra data. At this point, I've added '--tag-date', '--tag-build=TAG' and '--tag-svn-revision' options to bdist_egg that can be used individually or in combination. If you did something like: setup.py bdist_egg --tag-build=svn --tag-date --tag-svn-revision then today you'd get an egg with a revision like '0.2a-svn-2822-20050529'. In practice, I suspect most people will use either tag-date or tag-svn-revision, because using both is kind of redundant unless the package is also integrating external components. >>I was a bit puzzled by the diff at first; looks like you created a >>reversed diff. > >Oops. I have a hard time with diff for some reason. Like some kind of >chronological dyslexia. difflexia, perhaps? :) Actually, difflexia sounds like a problem *reading* diffs. Ah well. >It seems like a match object is a little odd to pass around. That's because it was an error. It should've read 'self._download_url(scheme.group(1),...' instead. >I was getting an error when calling self.samefile(filename, >filename_that_doesnt_exist). Maybe self.samefile needs to check >os.path.exists(destination). Aha. That's a platform-specific issue that I'll need to fix. Windows doesn't have 'os.samefile', so I didn't see the problem. I'll fix self.samefile() to check whether both paths exist before using os.samefile. >>Before doing this, I think it would be a good idea to check the headers >>for a text/html Content-Type, so as not to be readline() a big chunk of >>binary. :) > >Ah, yes, indeed. I've done this in my version now. >>>- # Actually, this probably doesn't do anything; but somehow this >>>- # information should get put into the version string... >> >>It indeed doesn't do anything; you'd be better off munging the setup.py >>to change the setup version. >>Maybe what I can do is have the build stuff look for a .svn dir in the >>build area, and if one is present, grab the revision and add it to the >>end of the version string supplied to setup(). Or simpler still, I could >>add a '--tag-svn-revision' option to bdist_egg, that would make *it* do >>that. I've been thinking it would be nice to have a '--tag-build=NUMBER' >>or '--tag-date' option for bdist_egg anyway, to support daily builds and >>such. So adding an option to "get the tag value from 'svn info'" would >>just be a minor variation on the theme. > >Yes, that would work well. I've decided I don't want this to happen automatically for EasyInstall egg builds, though. If I later let you pass build options through to the setup.py, you'll be able to do it that way. I just don't think it should invoke --tag-svn-revision by default. For one thing, I don't have a way to know if subversion is installed, and you could've downloaded a source .zip with .svn dirs accidentally included. (I've seen this from time to time with CVS dirs.) >It should really look for a scheme of 'svn+ssh' as well, but those are a >pain to make readable ;) I'm actually checking for 'scheme=="svn" or scheme.startswith("svn+")', so they ought to work too. > And technically file: should work for svn too, which I suppose you > could detect because it points to a directory with a db/fs-type > file. Eh, those can wait. I'd need more details before I could implement that. In the meantime, I'm going to go ahead and build/upload an EasyInstall/setuptools-0.3a2 release with the new subversion support, revision tagging, and bug fixes. Thanks for your help! From ianb at colorstudy.com Mon May 30 00:17:49 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 17:17:49 -0500 Subject: [Distutils] EasyInstall: installing non-distutil packages Message-ID: <429A3F8D.9060100@colorstudy.com> Looking at some of the packages I'm interested in, there's a couple ones that don't use distutils. Of course, fixing this is best, but I'm wondering if those can be installed as well. Typically you can install these packages just by copying them into place. So... thinking about the best way to do this. I think it should be possible to simply create a kind of generic set of attributes... maybe (very untested)... def setup_args(package_dir): attrs = {} attrs['name'] = os.path.basename(package_dir) dirs = [] os.path.walk(package_dir, (lambda arg, dirpath, names: pkgs.append(dirpath)), None) attrs['packages'] = [ attrs['name'] + '.' + d.replace('/', '.') for d in dirs if os.path.exists(os.path.join(package_dir, d, '__init__.py'))] return attrs Then somehow call distutils.core.setup(**attrs). And somehow package_dir has to get passed in, but I haven't figured out how that happens yet. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 00:31:16 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 18:31:16 -0400 Subject: [Distutils] EasyInstall: easy-install.pth, active version In-Reply-To: <429A38DD.70703@colorstudy.com> Message-ID: <5.1.1.6.0.20050529182026.02403a40@mail.telecommunity.com> At 04:49 PM 5/29/2005 -0500, Ian Bicking wrote: >Reading through "Changing the Active Version (site-packages installs >only)", I'm guessing that the implementation changes easy-install.pth to >point to a specific version. But no easy-install.pth is created when >you use the --install-dir option. And maybe that's just an oversight... >it seems like it easy-install.pth makes just as much sense in a local >directory as site-packages...? Unfortunately, Python only recognizes .pth files in "site" directories such as site-python and site-packages. This includes the $HOME/Library/Python2x/site-packages directory on OS X, if it's the 'Python.framework' build. In principle you can 'import site; site.addsitedir("foo")' in order to work around this, but at that point you might as well just import pkg_resources and require() what you want. Bob Ippolito has been pushing for Python to recognize .pth everywhere, or at least everywhere that's on sys.path at startup, IIUC. However that won't be till Python 2.5 at the earliest, so in the meantime it would be misleading to imply that --install-dir could work without implying --multi-version. From pje at telecommunity.com Mon May 30 00:46:36 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 18:46:36 -0400 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <429A2C91.3020902@colorstudy.com> Message-ID: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> At 03:56 PM 5/29/2005 -0500, Ian Bicking wrote: >I'm wondering how scripts should be handled by easyinstall -- right now >they pretty much get ignored. But I'm not sure how they should be >handled at all. Distutils has a couple rules, with --install-scripts, >and --home (which uses ~/bin), and the normal rules... and then some >translation of those options to Windows. But none of those seems to >translate very well to easyinstall. Or eggs. > >I don't have any ideas myself on this. Except that the scripts >themselves are usually pretty dumb anyway, just thin wrappers that fix >up sys.path, import a module, and call a function; so the status quo >could probably be improved upon anyway. Yep. Like you, however, I'm not sure what to do about 'em. Take a look at the Unix versions of easy_install -- they're actually the regular .egg files, with a shell script cat'ted on at the front! (Somebody could probably pull the same trick with an .exe on Windows.) However, such "eggsecutables" aren't really a solution for packages with multiple scripts. I'm thinking maybe bdist_egg should include an EGG-SCRIPTS directory for the scripts, so that EasyInstall could pull them out and install them whenever you do a non --multi-version install. It would, however, also need to *uninstall* them when deactivating a current version, because otherwise they'd be for the wrong version. I had previously thought that we would move towards using "python -m" to run scripts in eggs; unfortunately this only works for unpacked eggs, not egg files. Of course, this all just overlaps the fact that distutils script installation is really only good for command-line utilities anyway, and even for those, Windows requires a fair bit of tweaking in order to be able to use them effectively at a non-Cygwin command line. It might help if there were some kind of metadata for scripts, like to indicate whether something is a command-line utility, a graphical application, etc. Then distutils could tweak the file extension and/or build a custom launcher for it (like Fredrik Lundh's exemaker for Windows) as appropriate for the platform. That information could then be added to EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do script installation and uninstallation. From pje at telecommunity.com Mon May 30 00:51:12 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 18:51:12 -0400 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: <429A3F8D.9060100@colorstudy.com> Message-ID: <5.1.1.6.0.20050529184714.024670d0@mail.telecommunity.com> At 05:17 PM 5/29/2005 -0500, Ian Bicking wrote: >Looking at some of the packages I'm interested in, there's a couple ones >that don't use distutils. Of course, fixing this is best, but I'm >wondering if those can be installed as well. Typically you can install >these packages just by copying them into place. Hm. They're going to get a 0.0.0 version number. :( I suppose there could be an option to '--make-setup=version' that would let you automatically create a setup.py using that version number. But that seems rather hacky; I'd rather spend the effort writing a setup.py for the package in a local directory, then submitting it to the author as a patch. Or, if the package is orphaned, building a .egg and distributing it for others' benefit. From pje at telecommunity.com Mon May 30 01:16:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 19:16:29 -0400 Subject: [Distutils] EasyInstall/Egg platforms Message-ID: <5.1.1.6.0.20050529190831.021f07f0@mail.telecommunity.com> Currently, Python Eggs and EasyInstall use distutils.get_platform() to tag platform-specific eggs. However, I've heard that this string isn't all that useful on some platforms. (For example, I gather that it's pretty useless info on Mac OS X.) Also, some things might in principle be compatible, even though they have a different platform string. For example, a 'linux-i586' build should in principle be usable on a 'linux-i686' machine, right? Does anybody have more familiarity with the subtleties of binary compatibility, such that we could maybe devise a better platform string approach *before* the existing string is so widely used that it'll be difficult to change? From bob at redivi.com Mon May 30 01:38:10 2005 From: bob at redivi.com (Bob Ippolito) Date: Sun, 29 May 2005 16:38:10 -0700 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: <429A3F8D.9060100@colorstudy.com> References: <429A3F8D.9060100@colorstudy.com> Message-ID: On May 29, 2005, at 3:17 PM, Ian Bicking wrote: > Looking at some of the packages I'm interested in, there's a couple > ones > that don't use distutils. Of course, fixing this is best, but I'm > wondering if those can be installed as well. Typically you can > install > these packages just by copying them into place. You should name them, so we can publicly humiliate them into doing the right thing :) The only non-distutils-using package I have used lately is PyKQueue . I submitted a working setup.py to the maintainer, but he has not posted a new version that uses it. -bob From bob at redivi.com Mon May 30 01:43:54 2005 From: bob at redivi.com (Bob Ippolito) Date: Sun, 29 May 2005 16:43:54 -0700 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> Message-ID: <3596928A-C65D-4EDD-898F-05D46DD205EE@redivi.com> On May 29, 2005, at 3:46 PM, Phillip J. Eby wrote: > At 03:56 PM 5/29/2005 -0500, Ian Bicking wrote: > >> I'm wondering how scripts should be handled by easyinstall -- >> right now >> they pretty much get ignored. But I'm not sure how they should be >> handled at all. Distutils has a couple rules, with --install- >> scripts, >> and --home (which uses ~/bin), and the normal rules... and then some >> translation of those options to Windows. But none of those seems to >> translate very well to easyinstall. Or eggs. >> >> I don't have any ideas myself on this. Except that the scripts >> themselves are usually pretty dumb anyway, just thin wrappers that >> fix >> up sys.path, import a module, and call a function; so the status quo >> could probably be improved upon anyway. >> > > Yep. Like you, however, I'm not sure what to do about 'em. Take a > look at > the Unix versions of easy_install -- they're actually the regular .egg > files, with a shell script cat'ted on at the front! (Somebody could > probably pull the same trick with an .exe on Windows.) > > However, such "eggsecutables" aren't really a solution for packages > with > multiple scripts. I'm thinking maybe bdist_egg should include an > EGG-SCRIPTS directory for the scripts, so that EasyInstall could > pull them > out and install them whenever you do a non --multi-version > install. It > would, however, also need to *uninstall* them when deactivating a > current > version, because otherwise they'd be for the wrong version. > > I had previously thought that we would move towards using "python - > m" to > run scripts in eggs; unfortunately this only works for unpacked > eggs, not > egg files. > > Of course, this all just overlaps the fact that distutils script > installation is really only good for command-line utilities anyway, > and > even for those, Windows requires a fair bit of tweaking in order to > be able > to use them effectively at a non-Cygwin command line. > > It might help if there were some kind of metadata for scripts, like to > indicate whether something is a command-line utility, a graphical > application, etc. Then distutils could tweak the file extension > and/or > build a custom launcher for it (like Fredrik Lundh's exemaker for > Windows) > as appropriate for the platform. That information could then be > added to > EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do script > installation and uninstallation. Couldn't you change build_scripts and install_scripts when using bdist_egg to instead put the script inside the egg somewhere, and the installer would put scripts in the usual places that simply require and then exec the real script? -bob From pje at telecommunity.com Mon May 30 02:01:46 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 20:01:46 -0400 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <3596928A-C65D-4EDD-898F-05D46DD205EE@redivi.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050529195352.021f07e0@mail.telecommunity.com> At 04:43 PM 5/29/2005 -0700, Bob Ippolito wrote: >On May 29, 2005, at 3:46 PM, Phillip J. Eby wrote: >>It might help if there were some kind of metadata for scripts, like to >>indicate whether something is a command-line utility, a graphical >>application, etc. Then distutils could tweak the file extension >>and/or >>build a custom launcher for it (like Fredrik Lundh's exemaker for >>Windows) >>as appropriate for the platform. That information could then be >>added to >>EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do script >>installation and uninstallation. > >Couldn't you change build_scripts and install_scripts when using >bdist_egg to instead put the script inside the egg somewhere, and the >installer would put scripts in the usual places that simply require >and then exec the real script? That's more or less what I was proposing, except that I wouldn't use build_scripts or install_scripts, just pack the original source scripts into the egg, and then let the installer do its thing at the other end. That way, a cross-platform egg wouldn't have any platform-specific stuff baked into the scripts accidentally. It's not as if build_scripts or install_scripts do anything of consequence at the moment anyway; they just munge the #! line and copy the files (as of Python 2.3 anyway). My main thought is distinguishing console vs. GUI start scripts. I don't believe there's a Mac equivalent to '.pyw', for example. Also, at some point EasyInstall really could support adding desktop shortcuts (or Start menu entries or whatever you call them on a given platform) for applications bundled in eggs. Sort of an end-user py2exe/py2app thing. On the other hand, it does seem a little silly to use a command-line tool to install a graphical application. :) However, it probably makes sense to create platform-specific GUI wrappers around EasyInstall to do this sort of thing. From bob at redivi.com Mon May 30 02:13:06 2005 From: bob at redivi.com (Bob Ippolito) Date: Sun, 29 May 2005 17:13:06 -0700 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <5.1.1.6.0.20050529195352.021f07e0@mail.telecommunity.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <5.1.1.6.0.20050529195352.021f07e0@mail.telecommunity.com> Message-ID: On May 29, 2005, at 5:01 PM, Phillip J. Eby wrote: > At 04:43 PM 5/29/2005 -0700, Bob Ippolito wrote: > >> On May 29, 2005, at 3:46 PM, Phillip J. Eby wrote: >> >>> It might help if there were some kind of metadata for scripts, >>> like to >>> indicate whether something is a command-line utility, a graphical >>> application, etc. Then distutils could tweak the file extension >>> and/or >>> build a custom launcher for it (like Fredrik Lundh's exemaker for >>> Windows) >>> as appropriate for the platform. That information could then be >>> added to >>> EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do script >>> installation and uninstallation. >>> >> >> Couldn't you change build_scripts and install_scripts when using >> bdist_egg to instead put the script inside the egg somewhere, and the >> installer would put scripts in the usual places that simply require >> and then exec the real script? >> > > That's more or less what I was proposing, except that I wouldn't > use build_scripts or install_scripts, just pack the original source > scripts into the egg, and then let the installer do its thing at > the other end. That way, a cross-platform egg wouldn't have any > platform-specific stuff baked into the scripts accidentally. > > It's not as if build_scripts or install_scripts do anything of > consequence at the moment anyway; they just munge the #! line and > copy the files (as of Python 2.3 anyway). I guess, I don't think it really matters. If platform specific stuff gets baked into the scripts somehow, then they probably won't work without it. The #! line is of course of no consequence in-egg. > My main thought is distinguishing console vs. GUI start scripts. I > don't believe there's a Mac equivalent to '.pyw', for example. > Also, at some point EasyInstall really could support adding desktop > shortcuts (or Start menu entries or whatever you call them on a > given platform) for applications bundled in eggs. Sort of an end- > user py2exe/py2app thing. There is a Mac equivalent to '.pyw', it's the same thing (except .py files don't normally launch with a terminal attached, so really all Python scripts are treated as .pyw .. which is probably a bug). Mac OS X doesn't condone applications touching the user's dock, so the "start menu" thing is out. As for putting the applications in a reasonable place, the "reasonable place" depends on the user. The best solution might be to just pop open the dist folder and let the user drag it to where they want it, or to put it in a disk image and mount it. Anyway, "GUI scripts" is mostly a non-starter on OS X, it's generally not the right thing to do. Things should either be applications or not. Python doesn't currently participate in the "script" infrastructure on Mac OS X. > On the other hand, it does seem a little silly to use a command- > line tool to install a graphical application. :) However, it > probably makes sense to create platform-specific GUI wrappers > around EasyInstall to do this sort of thing. Well, on Mac OS X, you don't install graphical applications, they're self-contained application bundles that you just put wherever you want and run. Installers, on Mac OS X, are for crappy applications that don't do things the right way. -bob From pje at telecommunity.com Mon May 30 02:35:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 20:35:09 -0400 Subject: [Distutils] EasyInstall: scripts In-Reply-To: References: <5.1.1.6.0.20050529195352.021f07e0@mail.telecommunity.com> <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <5.1.1.6.0.20050529195352.021f07e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050529201903.021f1058@mail.telecommunity.com> At 05:13 PM 5/29/2005 -0700, Bob Ippolito wrote: >Mac OS X doesn't condone applications touching the user's dock, so >the "start menu" thing is out. As for putting the applications in a >reasonable place, the "reasonable place" depends on the user. The >best solution might be to just pop open the dist folder and let the >user drag it to where they want it, or to put it in a disk image and >mount it. Okay, sure. >Anyway, "GUI scripts" is mostly a non-starter on OS X, it's generally >not the right thing to do. Things should either be applications or >not. Um, yeah. I'm trying to say that I want a way to create a crappy cross-platform application. :) Let's say I write a wx app (yes, it'll be nasty-looking on the Mac, I get it), and just want to distribute it as a cross-platform egg (because only its dependencies are impure). I would want there to be a Mac version of EasyInstall that takes this script and makes it into a Python application. You're probably going to tell me that I'm talking about py2app, except for the part where I don't have a Mac myself, so I can't run it. But the application is something useful that people would like to use, even on the Mac, despite its crappy wx-based user interface. Why can't Mac users just download the egg and double-click it or drop it on EasyInstall or whatever the idiomatic thing to do is, to get back an application that they can then drag to wherever they're going to keep it? > Python doesn't currently participate in the "script" >infrastructure on Mac OS X. Unfortunately, I don't even know what this means. I was talking about command-line scripts, in a generic Unix-y sense. >>On the other hand, it does seem a little silly to use a command- line >>tool to install a graphical application. :) However, it >>probably makes sense to create platform-specific GUI wrappers >>around EasyInstall to do this sort of thing. > >Well, on Mac OS X, you don't install graphical applications, they're >self-contained application bundles that you just put wherever you >want and run. Installers, on Mac OS X, are for crappy applications >that don't do things the right way. Right, so when EasyInstall is eventually bundled with Python, perhaps there'll be some kind of association such that trying to invoke an egg does the right thing to turn it into an application or something. The idea here is that if you bundle your application as an .egg, then a platform-specific tool should be available to: * Do whatever wrapping or installing is customary for making "an application" available on the platform * Download and install the application's dependencies, perhaps after prompting for optional features (using the "optional dependencies" facility of Eggs) Now, in the case of the Mac, once this is done, you presumably have a valid "application" that you can then distribute any way you see fit. But, if you'd like to have access to cross-platform apps that non-Mac people are writing, it would be a lot more convenient if there were an EasyInstall+py2app tool. From ianb at colorstudy.com Mon May 30 02:40:14 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 19:40:14 -0500 Subject: [Distutils] EasyInstall: svn support In-Reply-To: <5.1.1.6.0.20050529174624.021f0ff8@mail.telecommunity.com> References: <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> <5.1.1.6.0.20050529174624.021f0ff8@mail.telecommunity.com> Message-ID: <429A60EE.50503@colorstudy.com> Phillip J. Eby wrote: >> Well, at least for my own packages I'll try to make the version >> better, by dynamically generating it in setup.py; > > > Note that it needs to be robust if the build machine doesn't have svn. Yeah... I'm not sure what that needs to do. For release branches I'll comment or otherwise disable that code, since the whole point of a release is to avoid svn-anything. If it's not a release, then I'm not sure how they'd acquire it without svn. If I set up something to get a snapshot, then I'll have to be sure to add that svn metadata somewhere to the snapshot, and that will make the svn access unnecessary. >>> It indeed doesn't do anything; you'd be better off munging the >>> setup.py to change the setup version. >>> Maybe what I can do is have the build stuff look for a .svn dir in >>> the build area, and if one is present, grab the revision and add it >>> to the end of the version string supplied to setup(). Or simpler >>> still, I could add a '--tag-svn-revision' option to bdist_egg, that >>> would make *it* do that. I've been thinking it would be nice to have >>> a '--tag-build=NUMBER' or '--tag-date' option for bdist_egg anyway, >>> to support daily builds and such. So adding an option to "get the >>> tag value from 'svn info'" would just be a minor variation on the theme. >> >> >> Yes, that would work well. > > > I've decided I don't want this to happen automatically for EasyInstall > egg builds, though. If I later let you pass build options through to > the setup.py, you'll be able to do it that way. I just don't think it > should invoke --tag-svn-revision by default. For one thing, I don't > have a way to know if subversion is installed, and you could've > downloaded a source .zip with .svn dirs accidentally included. (I've > seen this from time to time with CVS dirs.) Well, one way is to have _download_svn() put that metadata into the downloaded files -- some special file -- to later be read by setuptools. That would fit into a downloadable snapshot as well. And obviously _download_svn() can't work without svn installed. It should also be reasonably easy to translate to other version control systems, insofar as they have useful revision numbers. Though I don't think many of the distributed systems do :( Unless you are okay with *really* long versions that essentially embed a UUID -- but I don't encounter any of those repositories, so it's not a problem for me yet. >> And technically file: should work for svn too, which I suppose you >> could detect because it points to a directory with a db/fs-type file. >> Eh, those can wait. > > > I'd need more details before I could implement that. It's pretty low priority. I can look into it at some point, but it's only useful for the uncommon case where people are doing development on their own machine from a private repository, and want to test the process. > In the meantime, I'm going to go ahead and build/upload an > EasyInstall/setuptools-0.3a2 release with the new subversion support, > revision tagging, and bug fixes. Thanks for your help! Cool, I'll clear out the old one and start again with the new. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 03:02:06 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 20:02:06 -0500 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: <5.1.1.6.0.20050529184714.024670d0@mail.telecommunity.com> References: <5.1.1.6.0.20050529184714.024670d0@mail.telecommunity.com> Message-ID: <429A660E.2090900@colorstudy.com> Phillip J. Eby wrote: > At 05:17 PM 5/29/2005 -0500, Ian Bicking wrote: > >> Looking at some of the packages I'm interested in, there's a couple ones >> that don't use distutils. Of course, fixing this is best, but I'm >> wondering if those can be installed as well. Typically you can install >> these packages just by copying them into place. > > > Hm. They're going to get a 0.0.0 version number. :( Eh, crappy metadata is inevitable, and it's an incentive to create a proper distutil script. Having something that works, even if sub-optimal, is good enough for me. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 03:06:05 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 20:06:05 -0500 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: References: <429A3F8D.9060100@colorstudy.com> Message-ID: <429A66FD.7020003@colorstudy.com> Bob Ippolito wrote: > > On May 29, 2005, at 3:17 PM, Ian Bicking wrote: > >> Looking at some of the packages I'm interested in, there's a couple ones >> that don't use distutils. Of course, fixing this is best, but I'm >> wondering if those can be installed as well. Typically you can install >> these packages just by copying them into place. > > > You should name them, so we can publicly humiliate them into doing the > right thing :) I wrote one of them ;) -- obviously I could fix that one, but once I do I lose the opportunity to make sure the tool worked with The World We Have Now. And then there's still two other packages -- the flup servers, and PySourceColor.py, which is distributed as a single .py file (which is one of the more common cases where distutils isn't used). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 03:16:17 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 20:16:17 -0500 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> Message-ID: <429A6961.9070809@colorstudy.com> Phillip J. Eby wrote: > It might help if there were some kind of metadata for scripts, like to > indicate whether something is a command-line utility, a graphical > application, etc. Then distutils could tweak the file extension and/or > build a custom launcher for it (like Fredrik Lundh's exemaker for > Windows) as appropriate for the platform. That information could then > be added to EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do > script installation and uninstallation. That's what I'd be inclined towards. Like, this is what one script I have looks like: #!/usr/bin/env python import os import sys try: here = __file__ except NameError: # Python 2.2 here = sys.argv[0] # For when running directly out of the svn checkout: relative_paste = os.path.join( os.path.dirname(os.path.dirname(os.path.abspath(here))), 'paste') if os.path.exists(relative_paste): sys.path.insert(0, os.path.dirname(relative_paste)) from paste import app_setup sys.exit(app_setup.run(sys.argv)) I think this could be better expressed as: 'script_modules': {'paster': 'paste.app_setup.run'} And that whole script I give above is just a platform (Unixy) way of building that script. Of course command-line scripts don't translate as well to other operating systems and environments. Really these "scripts" should be one instance of an executable, that has a fairly minimal interface (__call__(args)). A richer interface might make the arguments introspectable, with possible platform-specific GUIs built around them, or even richer interfaces. I don't know what those would look like, but it would be nice to leave room for those. I know for a lot of developers, myself included, I could support multiple platforms *if* it's easy to build for these from the platform I use. But if I have to use a Mac to build a Mac distribution (ditto Windows) then (being realistic) I don't distribute things built for the Mac. For the most part this doesn't matter -- the libraries are still quite usable on other platforms -- but this does cause problems for scripts. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 03:45:12 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 20:45:12 -0500 Subject: [Distutils] EasyInstall: SF installs Message-ID: <429A7028.1020206@colorstudy.com> Attached is a patch so it can install from Sourceforge Download pages, choosing a random mirror. Sadly I find the file I want to install from SF doesn't include version information in its setup file, so it gets the version 0.0.0. Bah. It's an abandoned package anyway (zpt.sf.net), but the up-to-date Zope version hasn't *quite* been separately distributed (though I guess it's possible to build the package from the Zope sources). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org -------------- next part -------------- A non-text attachment was scrubbed... Name: easy_install.py Type: text/x-python Size: 20735 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050529/83745955/easy_install-0001.py -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: easy_install_sf.diff Url: http://mail.python.org/pipermail/distutils-sig/attachments/20050529/83745955/easy_install_sf-0001.diff From ianb at colorstudy.com Mon May 30 04:02:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 21:02:11 -0500 Subject: [Distutils] EasyInstall: verbosity Message-ID: <429A7423.6080104@colorstudy.com> As different kinds of indirection and detection are added to easy_install, it should probably have a verbosity option. I think this would probably best work by using the logging module, so that Installer takes a logger object or name, and reports all messages to that logger, and in main() it directs the logger to stdout and sets the log level based on the verbosity indicated. Tools that work with easy_install outside of the command-line could probably also make good use of the logging output. I could add this, but the number of parallel patches is probably going to leave everyone (including me) confused. (I'm working on installation of packages without setup.py files right now) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 05:12:46 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 23:12:46 -0400 Subject: [Distutils] EasyInstall: verbosity In-Reply-To: <429A7423.6080104@colorstudy.com> Message-ID: <5.1.1.6.0.20050529230138.021f13a0@mail.telecommunity.com> At 09:02 PM 5/29/2005 -0500, Ian Bicking wrote: >As different kinds of indirection and detection are added to >easy_install, it should probably have a verbosity option. I think this >would probably best work by using the logging module, so that Installer >takes a logger object or name, and reports all messages to that logger, >and in main() it directs the logger to stdout and sets the log level >based on the verbosity indicated. Tools that work with easy_install >outside of the command-line could probably also make good use of the >logging output. Yeah, this is actually exactly what I plan to do in an unspecified future release, probably after I add filesystem virtualization/sandboxing, or maybe in conjunction with it. I hadn't really thought about the part where main() would set it up, but that part makes good sense too. I was planning to pass verbosity options through to the setup() script, and temporarily redirect its stdout/stderr to pass through the logger. The idea there is that GUI tools that wrap easy_install may want to display the progress messages, etc. in a progress dialog or status bar or something of that sort. This is especially true for the downloading part, which can take a while. By the way, the sandboxing feature is where I plan to replace a few key 'os' module functions and builtins with ones that "notice" if the setup script is trying to modify files outside the installer's temporary directory. Initially this will just be so I can analyze those scripts' behavior, but it might eventually grow into a facility to make them think they're modifying files in the real filesystem, but they would actually be getting redirected to the right temporary subdirectory for stuff to be added into the egg. I don't know if this feature is really achievable, but most all file access in Python boils down to either 'open()' or a call to an 'os' function imported from the platform-specific extension ('nt', 'mac', 'posix', etc.). Anyway, arguably people should "fix" their packages, but in practice some folks won't, so a working sandbox could be helpful. But I'll probably want the sandbox tools to do a lot of output at various levels of detail, hence why I'd probably use the logging package to do this. >I could add this, but the number of parallel patches is probably going >to leave everyone (including me) confused. (I'm working on installation >of packages without setup.py files right now) Cool. You're really taking the ball and running with it, which is what I hoped would happen, as it's the specific reason I felt I had to release an initial, usable version of EasyInstall this very weekend. :) From pje at telecommunity.com Mon May 30 05:40:55 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 23:40:55 -0400 Subject: [Distutils] EasyInstall: SF installs In-Reply-To: <429A7028.1020206@colorstudy.com> Message-ID: <5.1.1.6.0.20050529233010.021f3708@mail.telecommunity.com> At 08:45 PM 5/29/2005 -0500, Ian Bicking wrote: >Attached is a patch so it can install from Sourceforge Download pages, >choosing a random mirror. Yay! I've incorporated more or less the same patch, after an unsuccesful experiment trying to pull the 'refresh:' from the HTTP headers. (Firefox showed it in the "View Page Info", so I thought it was there, and in fact it is only in the HTML. Oh well.) So, have you changed your opinion at all about the value of screenscraping as a way to build a Python package tool? I notice you've sent two patches today that apply regexes to HTML. :) I am a *little* concerned about the sourceforge support, given that they could change their download system any time, and if easy_install is distributed with Python that might make it harder to upgrade. But, at least people have the option of subclassing. From pje at telecommunity.com Mon May 30 05:48:47 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 23:48:47 -0400 Subject: [Distutils] EasyInstall: svn support In-Reply-To: <429A60EE.50503@colorstudy.com> References: <5.1.1.6.0.20050529174624.021f0ff8@mail.telecommunity.com> <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> <5.1.1.6.0.20050529160607.021fc350@mail.telecommunity.com> <5.1.1.6.0.20050529174624.021f0ff8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050529234447.023f1008@mail.telecommunity.com> At 07:40 PM 5/29/2005 -0500, Ian Bicking wrote: >Well, one way is to have _download_svn() put that metadata into the >downloaded files -- some special file -- to later be read by >setuptools. That would fit into a downloadable snapshot as well. And >obviously _download_svn() can't work without svn installed. It should >also be reasonably easy to translate to other version control systems, >insofar as they have useful revision numbers. Though I don't think many >of the distributed systems do :( Unless you are okay with *really* long >versions that essentially embed a UUID -- but I don't encounter any of >those repositories, so it's not a problem for me yet. I think this is all YAGNI; I'd like to encourage people to distribute ready-to-use eggs for their users. If you're involved with a project that has active development and you need to build-to-rev, I think it suffices to allow arguments to be passed through to the setup script (like --tag-date and --tag-svn-revision). Really, though, I don't see why you couldn't just set up your svn repository to request building a new egg (and/or source distribution, appropriately tagged) whenever a commit takes place under the project tree. From ianb at colorstudy.com Mon May 30 06:57:04 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 May 2005 23:57:04 -0500 Subject: [Distutils] EasyInstall: SF installs In-Reply-To: <5.1.1.6.0.20050529233010.021f3708@mail.telecommunity.com> References: <5.1.1.6.0.20050529233010.021f3708@mail.telecommunity.com> Message-ID: <429A9D20.3010800@colorstudy.com> Phillip J. Eby wrote: > I've incorporated more or less the same patch, after an unsuccesful > experiment trying to pull the 'refresh:' from the HTTP headers. > (Firefox showed it in the "View Page Info", so I thought it was there, > and in fact it is only in the HTML. Oh well.) > > So, have you changed your opinion at all about the value of > screenscraping as a way to build a Python package tool? I notice you've > sent two patches today that apply regexes to HTML. :) Yeah... now you're making me second guess myself ;) I dunno... I just want stuff to work, and if a better solution comes along later that's fine too. I've always expected special code just for SF, since they are a big-and-annoying source of downloads. The whole thing tends to be stupid for Python code anyway, which usually isn't large enough to justify the complexity of mirroring systems (for example the zpt package is 35kb, and I'm sure the mirroring system takes far more resources to provide). With PyPI getting file hosting hopefully this kind of thing will go away -- which is why I don't think a general solution (outside of SF) is necessary, because SF is an anachronistic style of distribution. > I am a *little* concerned about the sourceforge support, given that they > could change their download system any time, and if easy_install is > distributed with Python that might make it harder to upgrade. But, at > least people have the option of subclassing. Yeah, I thought about that too. In practice SF doesn't change much. A better set of regexes might look for a hostname of prdownloads.(sf|sourceforge).net, and then any href=(".*?\?use_mirror=[^"]*"|.*?\?use_mirror=[^ >]*), both case insensitive, which is probably a little less fragile. There's a good chance if they ever change it that they'll provide documented APIs, since I'm sure there's a lot of screen scrapers similar to this one out there. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 07:09:50 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 00:09:50 -0500 Subject: [Distutils] EasyInstall: verbosity In-Reply-To: <5.1.1.6.0.20050529230138.021f13a0@mail.telecommunity.com> References: <5.1.1.6.0.20050529230138.021f13a0@mail.telecommunity.com> Message-ID: <429AA01E.3080301@colorstudy.com> Phillip J. Eby wrote: > Yeah, this is actually exactly what I plan to do in an unspecified > future release, probably after I add filesystem > virtualization/sandboxing, or maybe in conjunction with it. I hadn't > really thought about the part where main() would set it up, but that > part makes good sense too. I was planning to pass verbosity options > through to the setup() script, and temporarily redirect its > stdout/stderr to pass through the logger. I still haven't seriously used the logging module, despite its widespread applicability to things I do. Go figure. But it seems like this would be a very good application of it, except for the actual exec of the setup file, which would have to use capturing of stdout and stderr. I'm guessing that redirection should be done to a sublogger, so that a caller can apply different filters and capturing to it. > The idea there is that GUI tools that wrap easy_install may want to > display the progress messages, etc. in a progress dialog or status bar > or something of that sort. This is especially true for the downloading > part, which can take a while. Progress wouldn't fit in very well to logging, though. I'm guessing there would have to be some notification object, with an API like: class INotifier: def describe_current_job(message): """Describe the currently-performed job, in the form of a short one-line text description""" def set_progress(progress, total, eta=None, units=None): """Indicate progress of the current job; progress and total are integers or floats. eta is the estimated time to finish, in seconds (if not explicitly given the INotifier instance may guess). units is an optional unit for progress and total (e.g., 'Mb', 'files')""" def reset_progress(): """Reset any progress and estimated times""" At least, that's my initial thought. Potentially you could have two progress indicators, one over-all progress, and another for the current job. > By the way, the sandboxing feature is where I plan to replace a few key > 'os' module functions and builtins with ones that "notice" if the setup > script is trying to modify files outside the installer's temporary > directory. Initially this will just be so I can analyze those scripts' > behavior, but it might eventually grow into a facility to make them > think they're modifying files in the real filesystem, but they would > actually be getting redirected to the right temporary subdirectory for > stuff to be added into the egg. I don't know if this feature is really > achievable, but most all file access in Python boils down to either > 'open()' or a call to an 'os' function imported from the > platform-specific extension ('nt', 'mac', 'posix', etc.). > > Anyway, arguably people should "fix" their packages, but in practice > some folks won't, so a working sandbox could be helpful. But I'll > probably want the sandbox tools to do a lot of output at various levels > of detail, hence why I'd probably use the logging package to do this. Yikes, that sounds complicated. Though sandboxing would be kind of neat in general, e.g., for mock filesystems in unit tests. Are there particular packages you have in mind that need this? >> I could add this, but the number of parallel patches is probably going >> to leave everyone (including me) confused. (I'm working on installation >> of packages without setup.py files right now) > > > Cool. You're really taking the ball and running with it, which is what > I hoped would happen, as it's the specific reason I felt I had to > release an initial, usable version of EasyInstall this very weekend. :) Yes, I think The Time Is Right -- this sort of thing is really important for projects that integrate lots of libraries, and I think we're seeing that need from a few different directions. The code seems very simple too, which is a good sign that it's solving the right problem. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 07:20:58 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 00:20:58 -0500 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: <429A3F8D.9060100@colorstudy.com> References: <429A3F8D.9060100@colorstudy.com> Message-ID: <429AA2BA.2080004@colorstudy.com> Well, I gave this a first go. I've attached a diff, but I'm not actually advocating that it go in. In this, you have to explicitly give the package name, because it's often not clear from the download location. So here's two examples: python easy_install.py --force-package=flup -d app-packages/ http://svn.saddi.com/flup/trunk/flup python easy_install.py --force-package=Component -d app-packages/ http://svn.w4py.org/Component/trunk Hrm... but I don't really like it. First, force_package is passed into Installer.__init__, and it should be part of the spec. Also, I don't think you always need to explicitly give the package name, sometimes it is self-evident. Also it doesn't work with plain Python files, which can't even be properly downloaded at this point, and for which eggs seem a little excessive. (To give an example of a single file I'd like to be able to install: http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py) I'm thinking maybe a better way to do this would be some sort of patch process, so that I could distribute a setup.py seperately, or some other custom build process, that easy_install could detect and run. And, of course, obviously I *could* fix these upstream, but I think this is a useful problem to solve through easy_install, even if these particular packages get fixed later. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: easy_install_forced.diff Url: http://mail.python.org/pipermail/distutils-sig/attachments/20050530/4ed02d50/easy_install_forced-0001.diff From pje at telecommunity.com Mon May 30 07:40:40 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 01:40:40 -0400 Subject: [Distutils] setup script sandboxing (was Re: EasyInstall: verbosity) In-Reply-To: <429AA01E.3080301@colorstudy.com> References: <5.1.1.6.0.20050529230138.021f13a0@mail.telecommunity.com> <5.1.1.6.0.20050529230138.021f13a0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050530012439.021f2820@mail.telecommunity.com> At 12:09 AM 5/30/2005 -0500, Ian Bicking wrote: >At least, that's my initial thought. Potentially you could have two >progress indicators, one over-all progress, and another for the current job. I was thinking that for download progress, I'd use whatever urllib already does. :) However for other things, you'd probably be looking at log messages to get some idea of the progress. >>By the way, the sandboxing feature is where I plan to replace a few key >>'os' module functions and builtins with ones that "notice" if the setup >>script is trying to modify files outside the installer's temporary >>directory. Initially this will just be so I can analyze those scripts' >>behavior, but it might eventually grow into a facility to make them think >>they're modifying files in the real filesystem, but they would actually >>be getting redirected to the right temporary subdirectory for stuff to be >>added into the egg. I don't know if this feature is really achievable, >>but most all file access in Python boils down to either 'open()' or a >>call to an 'os' function imported from the platform-specific extension >>('nt', 'mac', 'posix', etc.). >>Anyway, arguably people should "fix" their packages, but in practice some >>folks won't, so a working sandbox could be helpful. But I'll probably >>want the sandbox tools to do a lot of output at various levels of detail, >>hence why I'd probably use the logging package to do this. > >Yikes, that sounds complicated. Though sandboxing would be kind of neat >in general, e.g., for mock filesystems in unit tests. Are there >particular packages you have in mind that need this? Yes. I've been finding that a surprising number manage to install stuff in e.g. site-packages "behind my back". Some are mentioned on the EasyInstall experiences page, but once I actually implemented a prototype sandbox, I discovered that some of the packages I thought were clean, were in fact installing stuff behind my back! Anyway, the virtualization is fairly uncomplicated if all you want to do is to note when a package is being naughty; I've already implemented that in my working copy. Redirecting the paths to a sane location is more complex; I may have to move the virtualizer into bdist_egg and apply it there. Essentially, what we want to do is that during the main setup script, we don't want to allow it to write anything outside the temp directory, and fail the script as an "unsafe package" (a nice bit of FUD to help shame package authors into cleaning up that sort of uncontrolled installation activity). For writes occurring during bdist_egg's invocation of install_data and install_lib, writes to site-packages should be redirected to bdist_egg's staging area, and all other writes outside the setup script directory should be a fatal "unsafe package" abort. Anyway, virtualization is surprisingly easy to implement; all of the Python-supplied filesystem APIs are grounded in the 'os' module, and the 'open'/'file' builtin. If you replace these two, you've pretty much got it made. Even os.path functions like 'isdir()' and such work by calling os.stat(), so replacing it in os means you're good to go. And, there are fewer than 30 os primitives in all to replace, with only four distinct input/output signatures. A little metaprogramming goes a long way here, such that AbstractSandbox is only about 100 lines, and a simple LoggingSandbox class is about 20 more lines. To use these sandboxes for unit tests would probably be a pain, though; I'm focused only on either disallowing actions entirely, or changing what directory they happen in, sort of like a Python-only pseudo-chroot(). A unit test system would probably want to virtualize all the operations, which means more operations to virtualize, and with more signatures. From ianb at colorstudy.com Mon May 30 07:46:35 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 00:46:35 -0500 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: <429AA2BA.2080004@colorstudy.com> References: <429A3F8D.9060100@colorstudy.com> <429AA2BA.2080004@colorstudy.com> Message-ID: <429AA8BB.8030907@colorstudy.com> Ian Bicking wrote: > Hrm... but I don't really like it. First, force_package is passed into > Installer.__init__, and it should be part of the spec. Also, I don't > think you always need to explicitly give the package name, sometimes it > is self-evident. Also it doesn't work with plain Python files, which > can't even be properly downloaded at this point, and for which eggs seem > a little excessive. (To give an example of a single file I'd like to be > able to install: > http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py) Maybe this kind of thing shouldn't be installed similarly at all. These files don't have versions and there's no real benefit to turning them into egg files. The benefits are all fake when we make up the metadata. Maybe a simpler way of doing this, besides patching, would just be a kind of shortcut installation. Python source files and package directories just get installed directly (into site-packages or --install-dir). Crude but transitional. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 08:33:46 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 02:33:46 -0400 Subject: [Distutils] EasyInstall: installing non-distutil packages In-Reply-To: <429AA2BA.2080004@colorstudy.com> References: <429A3F8D.9060100@colorstudy.com> <429A3F8D.9060100@colorstudy.com> Message-ID: <5.1.1.6.0.20050530014143.02388c88@mail.telecommunity.com> At 12:20 AM 5/30/2005 -0500, Ian Bicking wrote: >I'm thinking maybe a better way to do this would be some sort of patch >process, so that I could distribute a setup.py seperately, or some other >custom build process, that easy_install could detect and run. > >And, of course, obviously I *could* fix these upstream, but I think this >is a useful problem to solve through easy_install, even if these >particular packages get fixed later. Yes, I've kind of thought about doing that (patches, or special modes/options/whatever), but it seemed just a bit Microsofty, if you know what I mean. :) On the other hand, Microsoft's commitment to backward compatibility has been legendary, and it's not necessarily a bad thing to emulate. It should help us achieve Total World Domination more quickly, which is why I was willing to put in the research time to figure out how to do sandboxing. So, I guess there are two ends of the "backward compatibility" spectrum here: * Low-end: Single .py files and other setup-free distributions * High-end: Distributions that muck about with their installation paths in one way or another I've so far found that there are almost as many ways to hack package data paths as there are packages; through considerable tweaking I've managed to get bdist_egg now to handle more packages without virtualization, but it's a regression-testing nightmare, because all the stuff is order sensitive, and stuff that looks unnecessary is in fact necessary to support different packages' ways of getting at the same information. I think, however, that all of those kludges will have to stick around until Python 2.6, using the Python 2.5 timeframe to warn that 'data_files' should not be used to install data into site-packages, and that 'package_data' is the correct approach. As for me, at the moment I've successfully gotten aquarium, IPython, and numarray to install their data only into their eggs, without any virtualization. So I'll be checking those kludges in momentarily. I have to say that in hindsight, the absence of adequate support for package-data installation in previous versions of the distutils was perhaps its single largest flaw. It appears very rare that anybody ever creates custom distutils commands, unless they needed to in order to work around that problem. As a consequence, it makes it much harder to get bdist_egg to trap such installs and route them to the right place. From ianb at colorstudy.com Mon May 30 09:54:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 02:54:39 -0500 Subject: [Distutils] pkg_resources: failed require Message-ID: <429AC6BF.8000008@colorstudy.com> I notice when I install into app-packages/, then do: >>> from pkg_resources import * >>> require('SomePackage') ... fails ... >>> import sys >>> sys.path.append('app-packages') >>> require('SomePackage') ... still fails ... But if I fix sys.path immediately, then it does work, so the failure seems to be sticky. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 18:45:10 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 12:45:10 -0400 Subject: [Distutils] pkg_resources: failed require In-Reply-To: <429AC6BF.8000008@colorstudy.com> Message-ID: <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> At 02:54 AM 5/30/2005 -0500, Ian Bicking wrote: >I notice when I install into app-packages/, then do: > > >>> from pkg_resources import * > >>> require('SomePackage') >... fails ... > >>> import sys > >>> sys.path.append('app-packages') > >>> require('SomePackage') >... still fails ... > >But if I fix sys.path immediately, then it does work, so the failure >seems to be sticky. That's very odd; require() doesn't do any caching. Even the PEP 302-defined caching (sys.path_importer_cache) is from *particular* path entries (e.g. app-packages) to importer objects created for them. So if app-packages wasn't on sys.path, it never would've had anything cached. Are you able to reliably reproduce the double-failure condition? Is the error message the same each time? From ianb at colorstudy.com Mon May 30 21:44:35 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 14:44:35 -0500 Subject: [Distutils] pkg_resources: failed require In-Reply-To: <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> References: <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> Message-ID: <429B6D23.8030607@colorstudy.com> Phillip J. Eby wrote: > At 02:54 AM 5/30/2005 -0500, Ian Bicking wrote: > >> I notice when I install into app-packages/, then do: >> >> >>> from pkg_resources import * >> >>> require('SomePackage') >> ... fails ... >> >>> import sys >> >>> sys.path.append('app-packages') >> >>> require('SomePackage') >> ... still fails ... >> >> But if I fix sys.path immediately, then it does work, so the failure >> seems to be sticky. > > > That's very odd; require() doesn't do any caching. Even the PEP > 302-defined caching (sys.path_importer_cache) is from *particular* path > entries (e.g. app-packages) to importer objects created for them. So if > app-packages wasn't on sys.path, it never would've had anything cached. > > Are you able to reliably reproduce the double-failure condition? Is the > error message the same each time? Hmm... I think I got it wrong, this works fine. Maybe I was confusing this with a bug related to packages with spaces in their names. In particular, WSGIUtils: http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz I suspect this doesn't work because it creates an egg WSGI_Utils-0.5-py2.4.egg/, with a package name (in WSGI_Utils-0.5-py2.4.egg/EGG-INFO/PKG-INFO) of "WSGI Utils" -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 30 21:58:41 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 14:58:41 -0500 Subject: [Distutils] pkg_resources: failed require In-Reply-To: <429B6D23.8030607@colorstudy.com> References: <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> <429B6D23.8030607@colorstudy.com> Message-ID: <429B7071.8060809@colorstudy.com> Ian Bicking wrote: > Hmm... I think I got it wrong, this works fine. > > Maybe I was confusing this with a bug related to packages with spaces in > their names. In particular, WSGIUtils: > http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz > > I suspect this doesn't work because it creates an egg > WSGI_Utils-0.5-py2.4.egg/, with a package name (in > WSGI_Utils-0.5-py2.4.egg/EGG-INFO/PKG-INFO) of "WSGI Utils" Hmm... maybe I'm just confused about how this stuff works. You don't require a Python package name, you require the more abstract name. But case insensitive. So require("wsgi-utils") or require("WSGI_Utils") works, but require('wsgiutils') doesn't. Since require('sqlobject') worked (even though the package name is SQLObject) I had become confused by this. So, not a bug. But definitely confusing. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 22:10:40 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 16:10:40 -0400 Subject: [Distutils] pkg_resources: failed require In-Reply-To: <429B6D23.8030607@colorstudy.com> References: <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050530160200.021f17b8@mail.telecommunity.com> At 02:44 PM 5/30/2005 -0500, Ian Bicking wrote: >Maybe I was confusing this with a bug related to packages with spaces in >their names. In particular, WSGIUtils: >http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz > >I suspect this doesn't work because it creates an egg >WSGI_Utils-0.5-py2.4.egg/, with a package name (in >WSGI_Utils-0.5-py2.4.egg/EGG-INFO/PKG-INFO) of "WSGI Utils" What do you mean by "this doesn't work"? Currently, the egg runtime never reads anything out of PKG-INFO except for the 'Version' field. So, although the 'Name' field reads "WSGI Utils", it is still recognized as the 'WSGI_Utils' package by 'require()'. What it *won't* be recognized as, is 'WSGI Utils', which is a requirement syntax error since package names in requirement specs can't have spaces in them. You're probably right, though, that the PKG-INFO file 'Name:' should match the actual egg name, so I've made that change and it should go out in 0.3a3. (The 'Name:' will read 'WSGI-Utils'). Note that egg names can currently only contain alphanumeric characters, '_', or '-'. And those last two are interchangeable, as '-' is converted to '_' in the egg's actual filename. Spaces in the setup(name="") argument are converted to '-'. It's possible that this range could/should be expanded somewhat, but it seemed prudent to be conservative initially, since it's easier to add capability than to take it away. From pje at telecommunity.com Mon May 30 22:31:11 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 16:31:11 -0400 Subject: [Distutils] pkg_resources: failed require In-Reply-To: <429B7071.8060809@colorstudy.com> References: <429B6D23.8030607@colorstudy.com> <5.1.1.6.0.20050530124139.021f2d38@mail.telecommunity.com> <429B6D23.8030607@colorstudy.com> Message-ID: <5.1.1.6.0.20050530161148.0236be38@mail.telecommunity.com> At 02:58 PM 5/30/2005 -0500, Ian Bicking wrote: >Ian Bicking wrote: >>Hmm... I think I got it wrong, this works fine. >>Maybe I was confusing this with a bug related to packages with spaces in >>their names. In particular, WSGIUtils: >>http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz >>I suspect this doesn't work because it creates an egg >>WSGI_Utils-0.5-py2.4.egg/, with a package name (in >>WSGI_Utils-0.5-py2.4.egg/EGG-INFO/PKG-INFO) of "WSGI Utils" > >Hmm... maybe I'm just confused about how this stuff works. You don't >require a Python package name, you require the more abstract name. Right; you require() the distribution name. I should probably find some examples that clearly use the distribution name. Maybe wxPython would be good, since the distribution is wxPython, but the package is wx. So you "require('wxPython>=2.5')", for example. I'll try to update the docs to make this clearer. And maybe if you install in --multi-version mode or to an alternate directory, easy_install should spit out a couple of example require() spellings for you to use, maybe even copy and paste right into your code. Perhaps something like: """ Installed WSGI_Utils-0.5-py2.4.egg Because this distribution was installed --multi-version or --install-dir, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("WSGI-Utils==0.5") # this exact version pkg_resources.require("WSGI-Utils>=0.5") # this version or higher pkg_resources.require("WSGI-Utils") # latest installed version Note also that '/your/instdir/setting' must be on sys.path at runtime for this to work. (e.g. by being the application script directory, by being on PYTHONPATH, or by being added to sys.path by your code.) """ I'll look into about adding a post-install report with this information for multi-version and alternate-directory installs. (That last paragraph above about '/your/instdir/setting' would only appear for --install-dir installs, and would include the actual directory name.) > But case insensitive. So require("wsgi-utils") or > require("WSGI_Utils") works, but require('wsgiutils') doesn't. Since > require('sqlobject') worked (even though the package name is SQLObject) I > had become confused by this. Yeah, I really didn't want to make it possible to have 'SQLobject' and 'sqlobject' both exist as distributions (even though I believe PyPI currently allows that). It would be just too darn confusing to make a typo and end up with the entirely wrong product, once we have automatic package location via PyPI et al. From ianb at colorstudy.com Mon May 30 23:10:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 16:10:11 -0500 Subject: [Distutils] Initial auto-installation support Message-ID: <429B8133.5070007@colorstudy.com> I've added some very initial support for automatic installation of packages in Paste, using easy_install (http://peak.telecommunity.com/DevCenter/EasyInstall). In a configuration file you can put: use_package(package_name, pkg_download_url=None) And a couple other options, but we'll ignore those. It will look for the named package, and if not found will install it (generally in app-packages). This still doesn't work for all the packages that build-pkg.py fetches; specifically flup and Component don't have setup.py files, and PySourceColor.py is just a bare Python module. I'd like for this system to work in spite of that, but it might also make sense to just fix all of those. (And actually I'm not super-enthusiastic about PySourceColor, so if anyone has opinions on a better source colorizer I'm open.) Right now there's some URLs for common packages in paste/default_config.conf (package_urls) -- wsgiutils, ZPTKit, ZopePageTemplates, scgi, and SQLObject. Ultimately this data really belongs in PyPI, so that dictionary of URLs is just transitional. There's another aspect to Paste installation, where some packages (plugins) need to write things into Paste. I'm not sure quite how that will work -- maybe use_package() will see if there's a paste_install module in the package somewhere, and call that somehow. But besides that, this should work now for any packages with a distutils install, so long as those packages are reasonably well behaved. Hrm... except setuptools 0.3a2 doesn't have SourceForge download support, but 0.3a3 does and I think PJE will release that soon. Eh, I could come up with a bunch of other caveats too... this stuff is rough still, but feedback is important. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon May 30 23:54:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 17:54:54 -0400 Subject: [Distutils] Initial auto-installation support In-Reply-To: <429B8133.5070007@colorstudy.com> Message-ID: <5.1.1.6.0.20050530173012.021f2340@mail.telecommunity.com> At 04:10 PM 5/30/2005 -0500, Ian Bicking wrote: >I've added some very initial support for automatic installation of >packages in Paste, using easy_install >(http://peak.telecommunity.com/DevCenter/EasyInstall). In a >configuration file you can put: > > use_package(package_name, pkg_download_url=None) > >And a couple other options, but we'll ignore those. It will look for >the named package, and if not found will install it (generally in >app-packages). FYI, just so you know, your implementation won't handle nested dependencies, nor allow specifying optional features of requested packages. I'd suggest that in a later version, you take a look at subclassing pkg_resources.AvailableDistributions, and overriding the 'obtain()' method to do the search and installation. In this way, your 'obtain()' method will get called for any dependencies that the originally-requested package requires. To see what I mean, take a look at the source of pkg_resources.require(), which looks like this: requirements = parse_requirements(requirements) for dist in AvailableDistributions().resolve(requirements): dist.install_on(sys.path) So, if you subclass AvailableDistributions to define an 'obtain()' method, and then write a similar loop using that subclass, you'll be able to cleanly integrate the auto-download in a forward-compatible way for packages that declare dependencies in their PackageName.egg-info. >There's another aspect to Paste installation, where some packages >(plugins) need to write things into Paste. I'm not sure quite how that >will work -- maybe use_package() will see if there's a paste_install >module in the package somewhere, and call that somehow. That might be a good candidate for egg metadata; if a package is a Paste plugin, you could require a 'paste-install' file in the package's EGG-INFO. Note that the AvailableDistributions().resolve() method returns a list of Distribution objects, and distribution objects have a 'metadata' attribute that implements 'IMetadataProvider'. So, 'aDistribution.metdata.has_metadata("paste-install")' will tell you if that distribution has a "paste-install" file in its EGG-INFO, and using the get_metadata() method will give it to you as a string. Thus, you can do something like: for dist in MyDownloadDistributions().resolve(requirements): dist.install_on(sys.path) if dist.metadata.has_metadata('paste-install'): doSomething(dist.metadata.get_metadata('paste-install')) If you use this algorithm, it will work for all egg varieties: compressed, uncompressed, and "development" eggs. You will, however, need to keep track of which paste-install scripts you've already processed, because 'resolve()' can yield distributions that are already present on sys.path. Also, you may want to create and cache a single instance of your MyDownloadDistributions class, because creating one does a lot of filesystem stats and listdirs and such. Of course, if you have people doing TheirPackage.egg-info/paste-install, they can also create TheirPackage.egg-info/depends.txt, and list all their dependencies there, with no need to use 'use_package'. It won't help with download URLs, though. But we could perhaps define an EGG-INFO/download_urls.txt as a stopgap, that lists known download urls... Anyway, as you can see, eggs were definitely designed with plugin systems like this in mind. :) > But besides >that, this should work now for any packages with a distutils install, so >long as those packages are reasonably well behaved. Hrm... except >setuptools 0.3a2 doesn't have SourceForge download support, but 0.3a3 >does and I think PJE will release that soon. Hopefully within the next 24 hours or so. It will also include sandboxing support (automatically aborts the install if the package tries to write to the filesystem outside the build directory), and lots of workarounds to support various packages out there that have quirky install_data subclasses. Those items are already done, but items still on my to-do list include: * help message to explain how to use require() for multi-version/instdir installs * a --build-dir/-b option to set the build directory, that will leave the downloaded package and its extracted contents in place after the installation is complete. (So you can read docs, install scripts, or debug a failed installation.) And I'd like to do something about scripts, but I think that's going to get left to an 0.4a1 release, assuming there are no further bug fix releases needed in the 0.3 line. From pje at telecommunity.com Tue May 31 03:14:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 21:14:09 -0400 Subject: [Distutils] EasyInstall 0.3a3 released; what about PyPI? (was Re: Initial auto-installation support) In-Reply-To: <5.1.1.6.0.20050530173012.021f2340@mail.telecommunity.com> References: <429B8133.5070007@colorstudy.com> Message-ID: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> >At 04:10 PM 5/30/2005 -0500, Ian Bicking wrote: > > But besides > >that, this should work now for any packages with a distutils install, so > >long as those packages are reasonably well behaved. Hrm... except > >setuptools 0.3a2 doesn't have SourceForge download support, but 0.3a3 > >does and I think PJE will release that soon. 0.3a3 is now released, with a new --build-dir option, sandboxing, more package workarounds, SourceForge mirror selection, and "installation reports". See: http://peak.telecommunity.com/DevCenter/EasyInstall#release-notes-change-history for more details. I'm thinking that adding automatic package location via PyPI is probably pretty doable now, by the way. My plan is to create a PackageFinder class (subclassing AvailableDistributions) whose obtain() method searches for the desired package on PyPI, keeping a cache of URLs it has already seen. (It would also accept a callback argument that it would use to create Installer objects when it needs to install packages.) The command-line tool (easy_install.main) would create a PackageFinder with an interactive installation callback, and in the main loop it would pass it to each new Installer instance. The Installer would then use it whenever it gets a non-file, non-URL command line option, and use it to resolve() such requests. The PackageFinder.obtain() method would go to the PyPI base URL followed by the desired distribution name, e.g. 'http://www.python.org/pypi/SQLObject', and then scrape the page to see if it is a multi-version page, or a single-version page. If it's multi-version, it would scrape the version links and select the highest-numbered version that meets all of your criteria. Once it has a single-version page, it would look for a download URL, and see if its filename is that of an archive (.egg, .tar, .tgz, etc.) or if the URL is for subversion. If so, we assume it's the right thing and invoke the callback to do the install. If not, then we follow the link anyway, and scrape for links to archives, checking versions when we get there if possible. If there's still nothing suitable (or there was no download URL), we apply the same procedure to the homepage URL. This should suffice to make a significant number of packages available from PyPI with autodownload, and packages with dependencies would also be downloaded, built, and installed. The hardest parts of this aren't in the screen-scraping per se; it's more in the heuristics for evaluating whether a specific URL is suitable for download. Many PyPI download URLs are of the form "foopackage-latest.tgz", so it's not possible to determine a usable version number from this, unless I special-case "latest" in the version parser -- which I guess I could do. We also probably need some kind of heuristic to determine which URLs are "better" to try, as we don't want to just run through the links in order. Hm. You know, what if as an interim step we had the command-line tool just launch a webbrowser pointing you to PyPI? Getting to a page for a suitable version is easy, so we could then let the user find the right download URL and then go back to paste it on the command line. That could be a nice interim addition, although it isn't much of a solution for packages with a lot of un-installed dependencies. You'd keep getting kicked back to the web browser a lot, and more to the point you'd have to keep restarting the tool. So, ultimately we really need a way to actually find the URLs. There are going to have to be new options for the tool, too. Like a way to set the PyPI URL to use, and a way to specify what sort of package revisions are acceptable (e.g. no alphas, no betas, no snapshots). From ianb at colorstudy.com Tue May 31 03:47:28 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 20:47:28 -0500 Subject: [Distutils] EasyInstall 0.3a3 released; what about PyPI? (was Re: Initial auto-installation support) In-Reply-To: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> References: <429B8133.5070007@colorstudy.com> <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> Message-ID: <429BC230.5040709@colorstudy.com> Phillip J. Eby wrote: > The PackageFinder.obtain() method would go to the PyPI base URL followed > by the desired distribution name, e.g. > 'http://www.python.org/pypi/SQLObject', and then scrape the page to see > if it is a multi-version page, or a single-version page. If it's > multi-version, it would scrape the version links and select the > highest-numbered version that meets all of your criteria. Really this would be very easy to add to PyPI as a set of xml-rpc calls, once the server move gets resolved. I could develop it now, except that the last pypi database dump I have is from before the move to Postgres, and there've been a number of changes since then, and I'd like some test data. I also have a checkout from CVS, but the SF project no longer lists CVS and svn.python.org doesn't have public access yet, so I can't point you to the code. But anyway, if someone on the new or old server can just run pg_dump on that database and email me the results (or a url to those results) that would be very helpful. Getting the data without screenscraping won't instantly give us all the necessary information. But it does contain good information about available versions, what the active version is, and per-version download URLs (which, if nothing else, could be compared against each other to detect non-version-specific URLs). > Hm. You know, what if as an interim step we had the command-line tool > just launch a webbrowser pointing you to PyPI? Getting to a page for a > suitable version is easy, so we could then let the user find the right > download URL and then go back to paste it on the command line. That > could be a nice interim addition, although it isn't much of a solution > for packages with a lot of un-installed dependencies. You'd keep > getting kicked back to the web browser a lot, and more to the point > you'd have to keep restarting the tool. So, ultimately we really need a > way to actually find the URLs. That's not a very satisfying experience -- the person might as well just download the file at that point. Even with accurate data from PyPI, it's still likely there will be multiple possible URLs. At that point, at least if you are going through the command line, displaying all the URLs (numbered) and asking the user would probably give the user enough information to choose. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue May 31 06:32:05 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 31 May 2005 00:32:05 -0400 Subject: [Distutils] EasyInstall 0.3a3 released; what about PyPI? (was Re: Initial auto-installation support) In-Reply-To: <429BC230.5040709@colorstudy.com> References: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> <429B8133.5070007@colorstudy.com> <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050531001410.021f3aa0@mail.telecommunity.com> At 08:47 PM 5/30/2005 -0500, Ian Bicking wrote: >Getting the data without screenscraping won't instantly give us all the >necessary information. But it does contain good information about >available versions, what the active version is, and per-version download >URLs (which, if nothing else, could be compared against each other to >detect non-version-specific URLs). Right; but none of that helps with the real problem (from EasyInstall's perspective), which is that the current incarnation of PyPI doesn't list multiple download URLs for a single release of a specific package. For example, when I release PEAK or PyProtocols, I've been releasing sdists (in two formats) plus a bdist_wininst -- and in the future I'll probably drop the bdist_wininst in favor of eggs. But I can't put any of that info on PyPI, so I just link to my downloads directory - as do 25% of the packages I surveyed in a random sampling last week. In order to get at packages like those, a flexible screen scraper is a must. I agree that PyPI should have better handling of download URLs, but I'm in a lot better position to improve EasyInstall than PyPI. >>Hm. You know, what if as an interim step we had the command-line tool >>just launch a webbrowser pointing you to PyPI? Getting to a page for a >>suitable version is easy, so we could then let the user find the right >>download URL and then go back to paste it on the command line. That >>could be a nice interim addition, although it isn't much of a solution >>for packages with a lot of un-installed dependencies. You'd keep getting >>kicked back to the web browser a lot, and more to the point you'd have to >>keep restarting the tool. So, ultimately we really need a way to >>actually find the URLs. > >That's not a very satisfying experience -- the person might as well just >download the file at that point. Tastes differ, I suppose. I'd just right-click the link to copy it, and then alt-tab, ^R, space, ^K, space, shift-insert, ENTER. But then, I've been downloading a lot of packages this weekend, so that sequence is already in my muscle memory. :) Hm. Maybe somebody could create a Firefox extension that runs EasyInstall on a selected link. :) >Even with accurate data from PyPI, it's still likely there will be >multiple possible URLs. At that point, at least if you are going through >the command line, displaying all the URLs (numbered) and asking the user >would probably give the user enough information to choose. In which case you might as well be back in the web browser. I'm fine with options being available to fine-tune the selection process, but the criteria can and should be mechanically processed. After all unusable versions, platforms, and archive types are eliminated, the prioritization should be in descending version order, with same version archives sorted by archive type, eggs first, everything else second. (Since the eggs don't need to be built.) By the way, in all of this there's been no discussion about MD5 signatures or code signing. That's probably because I don't know a whole lot about that subject. :) But I'm certainly interested in hearing from those who do.