From ziade.tarek at gmail.com Fri Jan 4 12:57:23 2008 From: ziade.tarek at gmail.com (=?UTF-8?Q?Tarek_Ziad=C3=A9?=) Date: Fri, 4 Jan 2008 03:57:23 -0800 (PST) Subject: [Catalog-sig] PyPI Enhancements (was: permissive trove classification) Message-ID: <14614933.post@talk.nabble.com> Hi, As discussed earlier, I have started to gather in a wiki page the ideas on enhancing multiple servers support in Disutils, and the changes needed in Distutils and PyPI code. http://wiki.python.org/moin/EnhancedPyPI I will continue the work in it after it has been commented, Please comment ! Regards Tarek -- View this message in context: http://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-trove-classification%29-tp14614933p14614933.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From martin at v.loewis.de Fri Jan 4 23:24:39 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 04 Jan 2008 23:24:39 +0100 Subject: [Catalog-sig] PyPI Enhancements (was: permissive trove classification) In-Reply-To: <14614933.post@talk.nabble.com> References: <14614933.post@talk.nabble.com> Message-ID: <477EB227.9050901@v.loewis.de> > As discussed earlier, I have started to gather in a wiki page the ideas on > enhancing multiple servers support in Disutils, and the changes needed in > Distutils and PyPI code. > > http://wiki.python.org/moin/EnhancedPyPI > > I will continue the work in it after it has been commented, > > Please comment ! Here you go: "when PyPI is down or slow" - I believe I have worked a lot and successfully in making sure that this actually doesn't happen. For the last several months, PyPI wasn't slow, and it wasn't down for any significant period of time. "several sections" - how do you use it from distutils, or setuptools? "When a user call register or uplad with such a file, it will be asked for each server wheter he wants to perform the action over it." I don't understand that sentence. What is "such" a file? (wheter->whether, it will -> he will) "Making PyPI permissive" - it's not clear whether you want PyPI to store these additional classifiers or not. If PyPI is required to store them, be prepared that the changes to PyPI will be *very* difficult to implement. If PyPI is allowed to discard them, I see a number of alternatives: a) make a list of "extra_classifiers" in setup.py, which a server is free to discard. b) give the list of extra classifiers in .pypirc, per server. distutil's register command would then drop all classifiers which are listed as server specific, unless registering with that selected server "Providing a base layer". It's not clear to me what that actually means. PyPI *does* have a separation of webui and store, so it is layered. Not sure what you are asking for. Regards, Martin From ziade.tarek at gmail.com Sun Jan 6 12:03:39 2008 From: ziade.tarek at gmail.com (=?UTF-8?Q?Tarek_Ziad=C3=A9?=) Date: Sun, 6 Jan 2008 03:03:39 -0800 (PST) Subject: [Catalog-sig] PyPI Enhancements (was: permissive trove classification) In-Reply-To: <477EB227.9050901@v.loewis.de> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> Message-ID: <14646918.post@talk.nabble.com> Thanks for the feedback "Martin v. L?wis" wrote: > > "when PyPI is down or slow" - I believe I have worked a lot > and successfully in making sure that this actually doesn't happen. > For the last several months, PyPI wasn't slow, and it wasn't > down for any significant period of time. > Yes this was a bit of a shortcut, as I didn't really mean to say that PyPI is slow. It works pretty well. But having an alternative server can be useful: In my company, it happens that the network is really slow, and some timeouts may occurs when we work on building some software that calls PyPI. These builds occurs all the time with the builder we use (zc.buildout). Having an alternative server that can be located in our network is useful in that case. We could use some caching, but a 100% functional PyPI server is a better pick. The typical configuration is to have a private PyPI -like server that contains all packages needed to build our software (public or private packages) and to use PyPI itself. "Martin v. L?wis" wrote: > > "several sections" - how do you use it from distutils, or > setuptools? > > "When a user call register or uplad with such a file, it will be asked > for each server wheter he wants to perform the action over it." > I don't understand that sentence. What is "such" a file? > (wheter->whether, it will -> he will) > When a user call the register or the upload command with such a file, it will loop through every section, and ask the used at the prompt if he wants to perform the action over the given server. I have changed the document to make it clearer. See "How sections in .pypirc are used" The iw.dist module implements a prototype of such a change, with two commands called "mupload" and "mregister", that are subclasses of setuptools' upload and regsiter at this time. "Martin v. L?wis" wrote: > > "Making PyPI permissive" - it's not clear whether you want PyPI to > store these additional classifiers or not. If PyPI is required > to store them, be prepared that the changes to PyPI will be > *very* difficult to implement. I > No, the server wouldn't store them. It will just ignore them. That would be done by returning a simple warning when a category in the received classifiers is unknown. "Martin v. L?wis" wrote: > > If PyPI is allowed to discard them, > I see a number of alternatives: > a) make a list of "extra_classifiers" in setup.py, which a server > is free to discard. > b) give the list of extra classifiers in .pypirc, per server. > distutil's register command would then drop all classifiers > which are listed as server specific, unless registering with > that selected server > An extra_classifiers would be interesting, but I don't think storing these classifiers on client-side would be maintainable, since it's up to the server to make this list evolve. "Martin v. L?wis" wrote: > > "Providing a base layer". It's not clear to me what that actually > means. PyPI *does* have a separation of webui and store, so it > is layered. Not sure what you are asking for. > Yes, that was not clear at all, I included an example the document. I was just talking about having a separate interface that exposes the API for the storage and the web interface. This interface could be used as a common base for all PyPI-like server, even if they don't use SQL to store data Regards Tarek -- View this message in context: http://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-trove-classification%29-tp14614933p14646918.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From fdrake at gmail.com Sun Jan 6 18:08:51 2008 From: fdrake at gmail.com (Fred Drake) Date: Sun, 6 Jan 2008 12:08:51 -0500 Subject: [Catalog-sig] PyPI Enhancements (was: permissive trove classification) In-Reply-To: <14646918.post@talk.nabble.com> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> Message-ID: <9cee7ab80801060908p30edb762g69b74120116f292@mail.gmail.com> On Jan 6, 2008 6:03 AM, Tarek Ziad? wrote: > When a user call the register or the upload command with such a file, it > will loop through every section, and ask the used at the prompt if he wants > to perform the action over the given server. This doesn't sound quite right to me. The set of servers used should be explicit, and there shouldn't be an assumption that any section found is necessarily a server. Perhaps something more like this would be appropriate: [distutils] index-servers = pypi mycompany [pypi] url = http://pypi.python.org/pypi/ username = mypubliclogin password = mypassword [mycompany] url = http://pypi.example.com/pypi/ username = mycompanylogin password = mypassword The default for distutils:index-servers could be pypi (and the pypi:url value would be defaulted as well). (In this structure, it's almost unfortunate that the section storing the username and password is "server-login" instead of "pypi"; something would need to be done to finesse that, or perhaps that section could be used to provide defaults for the username and password values for the specific server sections. My main point, though, is that detecting the set of servers shouldn't be magical. I don't really like the sort of interactivity you're proposing, but I suspect there are a variety of ways to deal with that as well. I've some thoughts on that, but they're not horribly clear just yet. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From martin at v.loewis.de Sun Jan 6 19:20:53 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 06 Jan 2008 19:20:53 +0100 Subject: [Catalog-sig] PyPI Enhancements In-Reply-To: <14646918.post@talk.nabble.com> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> Message-ID: <47811C05.30603@v.loewis.de> > When a user call the register or the upload command with such a file, it > will loop through every section, and ask the used at the prompt if he wants > to perform the action over the given server. I'm with Fred here. The distutils command line interface should avoid interactivity, and receive all input through command line options. If people still want interactive tools, they should build on top of distutils commands (or perhaps be additional commands). > No, the server wouldn't store them. It will just ignore them. That > would be done by returning a simple warning when a category in > the received classifiers is unknown. Ok, that's fine. >> "Providing a base layer". It's not clear to me what that actually >> means. PyPI *does* have a separation of webui and store, so it >> is layered. Not sure what you are asking for. >> > > Yes, that was not clear at all, I included an example the document. > I was just talking about having a separate interface that exposes > the API for the storage and the web interface. This interface > could be used as a common base for all PyPI-like server, > even if they don't use SQL to store data Ok, so you are after separate storage mechanisms. I'm opposed, as I don't see why anybody would want to have that - specifically, what else do you want to store it in, and why? What's wrong with PostgreSQL? It looks like a major complication in the code, for perhaps just one user (i.e. you). Also, and again: pypi *already* has store.py. Is it not good enough to provide an alternative implementation of that class? Where you have list_classifiers(), store.Store has get_classifiers. Where you have file_upload, store.Store has add_file. So it seems to me that this request is already implemented, except that it doesn't use Zope interfaces. Regards, Martin From ziade.tarek at gmail.com Sun Jan 6 21:11:23 2008 From: ziade.tarek at gmail.com (=?UTF-8?Q?Tarek_Ziad=C3=A9?=) Date: Sun, 6 Jan 2008 12:11:23 -0800 (PST) Subject: [Catalog-sig] PyPI Enhancements (was: permissive trove classification) In-Reply-To: <9cee7ab80801060908p30edb762g69b74120116f292@mail.gmail.com> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> <9cee7ab80801060908p30edb762g69b74120116f292@mail.gmail.com> Message-ID: <14653061.post@talk.nabble.com> Fred Drake wrote: > > Perhaps something more like this would be appropriate: > > [distutils] > index-servers = pypi mycompany > > [pypi] > url = http://pypi.python.org/pypi/ > username = mypubliclogin > password = mypassword > > [mycompany] > url = http://pypi.example.com/pypi/ > username = mycompanylogin > password = mypassword > > The default for distutils:index-servers could be pypi (and the > pypi:url value would be defaulted as well). (In this structure, it's > almost unfortunate that the section storing the username and password > is "server-login" instead of "pypi"; something would need to be done > to finesse that, or perhaps that section could be used to provide > defaults for the username and password values for the specific server > sections. > > My main point, though, is that detecting the set of servers shouldn't > be magical. > Yes I was wondering how it could look like, based on what we have now. I have change the proposal with your remarks. For the [server-login] section, maybe we could drop it and just have a backward compatibilty mechanism. (see updated doc) Fred Drake wrote: > > I don't really like the sort of interactivity you're proposing, but I > suspect there are a variety of ways to deal with that as well. I've > some thoughts on that, but they're not horribly clear just yet. > Ok, I'll answer on Martin mail on that Tarek -- View this message in context: http://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-trove-classification%29-tp14614933p14653061.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From ziade.tarek at gmail.com Sun Jan 6 21:29:42 2008 From: ziade.tarek at gmail.com (=?UTF-8?Q?Tarek_Ziad=C3=A9?=) Date: Sun, 6 Jan 2008 12:29:42 -0800 (PST) Subject: [Catalog-sig] PyPI Enhancements In-Reply-To: <47811C05.30603@v.loewis.de> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> <47811C05.30603@v.loewis.de> Message-ID: <14653267.post@talk.nabble.com> "Martin v. L?wis" wrote: > >> When a user call the register or the upload command with such a file, it >> will loop through every section, and ask the used at the prompt if he >> wants >> to perform the action over the given server. > > I'm with Fred here. The distutils command line interface should avoid > interactivity, and receive all input through command line options. > > If people still want interactive tools, they should build on top of > distutils commands (or perhaps be additional commands). > Ok, sounds right, I have changed to doc accordingly "Martin v. L?wis" wrote: > > Ok, so you are after separate storage mechanisms. I'm opposed, as > I don't see why anybody would want to have that - specifically, > what else do you want to store it in, and why? What's wrong with > PostgreSQL? > ... > Also, and again: pypi *already* has store.py. Is it not good enough > to provide an alternative implementation of that class? Where you > have list_classifiers(), store.Store has get_classifiers. Where you > have file_upload, store.Store has add_file. So it seems to me that > this request is already implemented, except that it doesn't use > Zope interfaces. > Ok, I dropped that part So what remains in the document is: - change the PyPI code so it warns on unexisting classifier - change the .pypirc file so it has several sections (Fred's model) - change the distutils register and upload commands so it works with the new .pypirc file, and let the user choose the server in the command line Regards Tarek -- View this message in context: http://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-trove-classification%29-tp14614933p14653267.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From martin at v.loewis.de Sun Jan 6 21:36:15 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 06 Jan 2008 21:36:15 +0100 Subject: [Catalog-sig] PyPI Enhancements In-Reply-To: <14653267.post@talk.nabble.com> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> <47811C05.30603@v.loewis.de> <14653267.post@talk.nabble.com> Message-ID: <47813BBF.6080107@v.loewis.de> > So what remains in the document is: > > - change the PyPI code so it warns on unexisting classifier > - change the .pypirc file so it has several sections (Fred's model) > - change the distutils register and upload commands so it works with the new > .pypirc file, > and let the user choose the server in the command line As a feature list, this sounds right. I'm curious how you'll implement the warning mechanism, though, through XML-RPC. Regards, Martin From ziade.tarek at gmail.com Wed Jan 9 10:14:18 2008 From: ziade.tarek at gmail.com (=?UTF-8?Q?Tarek_Ziad=C3=A9?=) Date: Wed, 9 Jan 2008 01:14:18 -0800 (PST) Subject: [Catalog-sig] PyPI Enhancements In-Reply-To: <47813BBF.6080107@v.loewis.de> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> <47811C05.30603@v.loewis.de> <14653267.post@talk.nabble.com> <47813BBF.6080107@v.loewis.de> Message-ID: <14707904.post@talk.nabble.com> "Martin v. L?wis" wrote: > >> So what remains in the document is: >> >> - change the PyPI code so it warns on unexisting classifier >> - change the .pypirc file so it has several sections (Fred's model) >> - change the distutils register and upload commands so it works with the >> new >> .pypirc file, >> and let the user choose the server in the command line > > As a feature list, this sounds right. I'm curious how you'll implement > the warning mechanism, though, through XML-RPC. > I have to look how XML-RPC is implemented in PyPI today, but in regular mode the warning message will probably be a simple line like what we have right now, added before the OK in the stream. For instance in file_upload: self.handler.send_response(200, 'OK') self.handler.set_content_type('text/plain') self.handler.end_headers() self.wfile.write('WARNING: ....\n') self.wfile.write('OK\n') then make some xmlrpclibs.dumps for the xmlrpc version but this is my first guess, as I need to look deeper in the code. I guess it's more complicated than that if you are saying that Regards Tarek -- View this message in context: http://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-trove-classification%29-tp14614933p14707904.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From ziade.tarek at gmail.com Wed Jan 9 11:38:34 2008 From: ziade.tarek at gmail.com (=?UTF-8?Q?Tarek_Ziad=C3=A9?=) Date: Wed, 9 Jan 2008 02:38:34 -0800 (PST) Subject: [Catalog-sig] Trove reference Message-ID: <14709105.post@talk.nabble.com> Hello, For my understanding, I was wondering how the Trove tree behaves. Is it a free tree, where people will add branches and leaves with no particular rule, or does it have some rules (like unique leave names for instance) ? I've seen that leaves were added from time to time, but is this a categorization that just concerns PyPI, or are we going to stick to Sourceforge one for instance ? The link that is located in PEP 301 is dead: http://tuxedo.org/~esr/trove/ and I can't find anything related on the web. All forges have their Trove maps but I couldn't find a RCF or something, that would define a common map or describe its behavior. Thanks Tarek -- View this message in context: http://www.nabble.com/Trove-reference-tp14709105p14709105.html Sent from the Python - catalog-sig mailing list archive at Nabble.com. From amk at amk.ca Wed Jan 9 14:27:19 2008 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 9 Jan 2008 08:27:19 -0500 Subject: [Catalog-sig] Trove reference In-Reply-To: <14709105.post@talk.nabble.com> References: <14709105.post@talk.nabble.com> Message-ID: <20080109132719.GA7118@amk-desktop.matrixgroup.net> On Wed, Jan 09, 2008 at 02:38:34AM -0800, Tarek Ziad? wrote: > The link that is located in PEP 301 is dead: http://tuxedo.org/~esr/trove/ It's moved to . That's a draft dating from 2002, though; there's no more 'official' spec. --amk From martin at v.loewis.de Wed Jan 9 19:34:17 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 09 Jan 2008 19:34:17 +0100 Subject: [Catalog-sig] PyPI Enhancements In-Reply-To: <14707904.post@talk.nabble.com> References: <14614933.post@talk.nabble.com> <477EB227.9050901@v.loewis.de> <14646918.post@talk.nabble.com> <47811C05.30603@v.loewis.de> <14653267.post@talk.nabble.com> <47813BBF.6080107@v.loewis.de> <14707904.post@talk.nabble.com> Message-ID: <478513A9.8060400@v.loewis.de> > then make some xmlrpclibs.dumps for the xmlrpc version > but this is my first guess, as I need to look deeper in the > code. I guess it's more complicated than that > if you are saying that Perhaps not - this sounds feasible. I was just puzzled by your spec suggesting that the warning gets output before the 200 status - that can't work on the wire in that order. Regards, Martin From martin at v.loewis.de Wed Jan 9 19:41:57 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 09 Jan 2008 19:41:57 +0100 Subject: [Catalog-sig] Trove reference In-Reply-To: <14709105.post@talk.nabble.com> References: <14709105.post@talk.nabble.com> Message-ID: <47851575.6080201@v.loewis.de> > For my understanding, I was wondering how the Trove tree behaves. > > Is it a free tree, where people > will add branches and leaves with no particular rule, or does it have > some rules (like unique leave names for instance) ? It's free tree, yes. The major rule is that we won't add new classifiers in advance, i.e. without an actual need. My personal view on "actual need" is that there must be at least two packages that would be classified under the classifier, or else it is pointless having it. I personally don't view the trove as a tree at all (and had a hard time understanding why you were referring to it as a "tree" - classification/taxonomy are words that I would have understood better - I still don't see why the leaves would play a special role, as any leave can become an inner node at any time). Regards, Martin From josepharmbruster at gmail.com Wed Jan 9 21:16:33 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Wed, 9 Jan 2008 15:16:33 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry Message-ID: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> All, Is issue 1811827 as listed in SF something the community is interested in patching? If so, I created a patch for it and would like to submit. What is the appropriate place to do so? Joseph Armbruster From martin at v.loewis.de Wed Jan 9 23:59:10 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 09 Jan 2008 23:59:10 +0100 Subject: [Catalog-sig] Redirecting Cheeseshop Message-ID: <478551BE.2080807@v.loewis.de> I'll start soon to redirect GET requests for cheeseshop.python.org to pypi.python.org. Any objections? Regards, Martin From martin at v.loewis.de Thu Jan 10 08:15:10 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Jan 2008 08:15:10 +0100 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> Message-ID: <4785C5FE.1030702@v.loewis.de> > Is issue 1811827 as listed in SF something the community is interested > in patching? Sure! > If so, I created a patch for it and would like to > submit. > What is the appropriate place to do so? Please submit it as a new tracker item, referring to the one it fixes. Regards, Martin From josepharmbruster at gmail.com Thu Jan 10 13:52:25 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Thu, 10 Jan 2008 07:52:25 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <4785C5FE.1030702@v.loewis.de> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> Message-ID: <47861509.9000109@gmail.com> ? wrote: >> Is issue 1811827 as listed in SF something the community is interested >> in patching? > > Sure! > >> If so, I created a patch for it and would like to >> submit. >> What is the appropriate place to do so? > > Please submit it as a new tracker item, referring to the one it fixes. > > Regards, > Martin > Martin, Do you want me to create this tracker item in bugs.python.org? The issue already exists in the sf.net bugs listing, which is why I ask. Joe From josepharmbruster at gmail.com Thu Jan 10 14:21:31 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Thu, 10 Jan 2008 08:21:31 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <47861509.9000109@gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> Message-ID: <938f42d70801100521v54b2cf6drd14a958d17f557f7@mail.gmail.com> Martin, In addition, to check out the behavior, visit: http://www.joevial.com/pypi/SQLObject/0.7.9 Please send comments and suggestions, Joseph Armbruster On Jan 10, 2008 7:52 AM, Joseph Armbruster wrote: > > ? wrote: > >> Is issue 1811827 as listed in SF something the community is interested > >> in patching? > > > > Sure! > > > >> If so, I created a patch for it and would like to > >> submit. > >> What is the appropriate place to do so? > > > > Please submit it as a new tracker item, referring to the one it fixes. > > > > Regards, > > Martin > > > > Martin, > > Do you want me to create this tracker item in bugs.python.org? The issue > already exists in the sf.net bugs listing, which is why I ask. > > Joe > From amk at amk.ca Thu Jan 10 14:42:21 2008 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 10 Jan 2008 08:42:21 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <47861509.9000109@gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> Message-ID: <20080110134221.GA6362@amk-desktop.matrixgroup.net> On Thu, Jan 10, 2008 at 07:52:25AM -0500, Joseph Armbruster wrote: > Do you want me to create this tracker item in bugs.python.org? The issue > already exists in the sf.net bugs listing, which is why I ask. We no longer use the sf.net bug tracker and no one is looking at it, so yes, the item should be created on bugs.python.org. --amk From josepharmbruster at gmail.com Thu Jan 10 14:53:57 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Thu, 10 Jan 2008 08:53:57 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <20080110134221.GA6362@amk-desktop.matrixgroup.net> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> Message-ID: <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> A.M. Kuchling, Ah, my mistake. However it appears others are making the same mistake, I see four bugs on sourceforge from the last month. It would probably be worthwhile to have somebody close this project out to prevent future confusion. http://sourceforge.net/projects/pypi/ It has nine bugs listed, four of which are from the last month. I will go ahead and transfer these over to bugs.python.org (unless there are any objections) Joseph Armbruster On Jan 10, 2008 8:42 AM, A.M. Kuchling wrote: > On Thu, Jan 10, 2008 at 07:52:25AM -0500, Joseph Armbruster wrote: > > Do you want me to create this tracker item in bugs.python.org? The issue > > already exists in the sf.net bugs listing, which is why I ask. > > We no longer use the sf.net bug tracker and no one is looking at it, so yes, > the item should be created on bugs.python.org. > > --amk > From josepharmbruster at gmail.com Thu Jan 10 14:57:18 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Thu, 10 Jan 2008 08:57:18 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> Message-ID: <938f42d70801100557x2b6ac7aau3afd148a6d57a00f@mail.gmail.com> A.M. Kuchling, I also noticed that cheeseshop.pyton.org/pypi (pypi.python.org/pypi) have a link for "Bug Reports" that points to: http://sourceforge.net/tracker/?group_id=66150&atid=513503 This should probably be updated as well. Joseph Armbruster On Jan 10, 2008 8:53 AM, Joseph Armbruster wrote: > A.M. Kuchling, > > Ah, my mistake. However it appears others are making the same > mistake, I see four bugs on sourceforge from the last month. It would > probably be worthwhile to have somebody close this project out to > prevent future confusion. > > http://sourceforge.net/projects/pypi/ > > It has nine bugs listed, four of which are from the last month. I > will go ahead and transfer these over to bugs.python.org (unless there > are any objections) > > Joseph Armbruster > > > > On Jan 10, 2008 8:42 AM, A.M. Kuchling wrote: > > On Thu, Jan 10, 2008 at 07:52:25AM -0500, Joseph Armbruster wrote: > > > Do you want me to create this tracker item in bugs.python.org? The issue > > > already exists in the sf.net bugs listing, which is why I ask. > > > > We no longer use the sf.net bug tracker and no one is looking at it, so yes, > > the item should be created on bugs.python.org. > > > > --amk > > > From amk at amk.ca Thu Jan 10 15:14:35 2008 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 10 Jan 2008 09:14:35 -0500 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> Message-ID: <20080110141435.GA6854@amk-desktop.matrixgroup.net> On Thu, Jan 10, 2008 at 08:53:57AM -0500, Joseph Armbruster wrote: > http://sourceforge.net/projects/pypi/ > > It has nine bugs listed, four of which are from the last month. I Ah, I thought you were referring to the *Python* bug tracker on sf.net! I don't know anything about the PyPI bug tracker's status, but Richard Jones seems to be commenting on bugs there, so it's probably the place to use. --amk From josepharmbruster at gmail.com Thu Jan 10 18:32:32 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Thu, 10 Jan 2008 12:32:32 -0500 Subject: [Catalog-sig] 1811827 Patch post In-Reply-To: <938f42d70801100557x2b6ac7aau3afd148a6d57a00f@mail.gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> <938f42d70801100557x2b6ac7aau3afd148a6d57a00f@mail.gmail.com> Message-ID: <478656B0.5070501@gmail.com> All, Made it home over my lunch break today and posted up the patch. I could not figured out how to attach a file in the sf issue tracker, so I placed the patch text in the comments. https://sourceforge.net/tracker/index.php?func=detail&aid=1811827&group_id=66150&atid=513503 Joseph Armbruster Joseph Armbruster wrote: > A.M. Kuchling, > > I also noticed that cheeseshop.pyton.org/pypi (pypi.python.org/pypi) > have a link for "Bug Reports" that points to: > > http://sourceforge.net/tracker/?group_id=66150&atid=513503 > > This should probably be updated as well. > > > Joseph Armbruster > > On Jan 10, 2008 8:53 AM, Joseph Armbruster wrote: >> A.M. Kuchling, >> >> Ah, my mistake. However it appears others are making the same >> mistake, I see four bugs on sourceforge from the last month. It would >> probably be worthwhile to have somebody close this project out to >> prevent future confusion. >> >> http://sourceforge.net/projects/pypi/ >> >> It has nine bugs listed, four of which are from the last month. I >> will go ahead and transfer these over to bugs.python.org (unless there >> are any objections) >> >> Joseph Armbruster >> >> >> >> On Jan 10, 2008 8:42 AM, A.M. Kuchling wrote: >>> On Thu, Jan 10, 2008 at 07:52:25AM -0500, Joseph Armbruster wrote: >>>> Do you want me to create this tracker item in bugs.python.org? The issue >>>> already exists in the sf.net bugs listing, which is why I ask. >>> We no longer use the sf.net bug tracker and no one is looking at it, so yes, >>> the item should be created on bugs.python.org. >>> >>> --amk >>> > From martin at v.loewis.de Thu Jan 10 20:06:21 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 10 Jan 2008 20:06:21 +0100 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <47861509.9000109@gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> Message-ID: <47866CAD.3080302@v.loewis.de> > Do you want me to create this tracker item in bugs.python.org? The > issue already exists in the sf.net bugs listing, which is why I ask. I meant the sf tracker. I don't think you can attach a patch there (if you can, just ignore me); so therefore I proposed you create a new tracker issue. Regards, Martin From martin at v.loewis.de Thu Jan 10 20:07:32 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Jan 2008 20:07:32 +0100 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <20080110134221.GA6362@amk-desktop.matrixgroup.net> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> Message-ID: <47866CF4.2030002@v.loewis.de> > On Thu, Jan 10, 2008 at 07:52:25AM -0500, Joseph Armbruster wrote: >> Do you want me to create this tracker item in bugs.python.org? The issue >> already exists in the sf.net bugs listing, which is why I ask. > > We no longer use the sf.net bug tracker and no one is looking at it, so yes, > the item should be created on bugs.python.org. Please understand that Joseph is talking about PyPI, not Python. PyPI's issues are still tracked on SF. Regards, Martin From martin at v.loewis.de Thu Jan 10 20:08:11 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Jan 2008 20:08:11 +0100 Subject: [Catalog-sig] 1811827 Patch Inquiry In-Reply-To: <938f42d70801100557x2b6ac7aau3afd148a6d57a00f@mail.gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> <938f42d70801100557x2b6ac7aau3afd148a6d57a00f@mail.gmail.com> Message-ID: <47866D1B.10300@v.loewis.de> > I also noticed that cheeseshop.pyton.org/pypi (pypi.python.org/pypi) > have a link for "Bug Reports" that points to: > > http://sourceforge.net/tracker/?group_id=66150&atid=513503 > > This should probably be updated as well. No no no. AMK was simply wrong. Regards, Martin From martin at v.loewis.de Thu Jan 10 20:13:44 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Jan 2008 20:13:44 +0100 Subject: [Catalog-sig] 1811827 Patch post In-Reply-To: <478656B0.5070501@gmail.com> References: <938f42d70801091216p339a7cf9g4b4da55161c88cc5@mail.gmail.com> <4785C5FE.1030702@v.loewis.de> <47861509.9000109@gmail.com> <20080110134221.GA6362@amk-desktop.matrixgroup.net> <938f42d70801100553m1cc27ffeib0cc0fbdf95d57be@mail.gmail.com> <938f42d70801100557x2b6ac7aau3afd148a6d57a00f@mail.gmail.com> <478656B0.5070501@gmail.com> Message-ID: <47866E68.1060702@v.loewis.de> > Made it home over my lunch break today and posted up the patch. I could not > figured out how to attach a file in the sf issue tracker, so I placed the patch > text in the comments. That's why I suggested to create a new patch. Regards, Martin From josepharmbruster at gmail.com Sun Jan 13 02:09:55 2008 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Sat, 12 Jan 2008 20:09:55 -0500 Subject: [Catalog-sig] pypi error page handling Message-ID: <478964E3.2050205@gmail.com> All, I submitted a suggestion of better error-page generation for pypi.python.org. I am not sure if everyone gets updates on sf tracker issues for pypi, so I have sent an email here. Please read the details here and let me know if you're interested. https://sourceforge.net/tracker/index.php?func=detail&aid=1870130&group_id=66150&atid=513503 Joseph Armbruster From stephenemslie at gmail.com Thu Jan 17 15:18:47 2008 From: stephenemslie at gmail.com (stephen emslie) Date: Thu, 17 Jan 2008 14:18:47 +0000 Subject: [Catalog-sig] reStructuredText whitespace lost when writing pkg-info In-Reply-To: <51f97e530801100920h5b8f7b1an50c57271fbbba466@mail.gmail.com> References: <51f97e530801100920h5b8f7b1an50c57271fbbba466@mail.gmail.com> Message-ID: <51f97e530801170618j6c39005r215206aaddf8f66d@mail.gmail.com> This is something that I have already sent to the distutils-sig list. Apologies for the cross-post, but seeing Tarek Ziad?'s recent work on the Plone software center and enhancing pypi got me thinking that this might actually be a more appropriate place to ask, as I am working on something similar. I'd be grateful for any help or comments: ---------- Forwarded message ---------- From: stephen emslie Date: Jan 10, 2008 5:20 PM Subject: reStructuredText whitespace lost when writing pkg-info To: distutils-sig at python.org I've noticed that setuptools seems to flatten any indentation in fields when generating the PKG-INFO metadata file. I noticed this because PEP345 [1] suggests that the long description field can be written in reStructuredText (which would be excellent for my current project), but unfortunately whitespace is being lost that breaks things like literal blocks that require indentation. The trouble seems to be that distutils.util.rfc822_escape strips all excess whitespace and adds 8 spaces of padding to the start of every line. For example: >>> rest = """ ... a literal python block:: ... >>> import this ... """ >>> print distutils.util.rfc822_escape(rest) a literal python block:: >>> import this which generates a few warnings and no literal block when docutils tried to render it. Can anyone tell me if this behaviour is necessary to be rfc822/MIME compliant, or is distutils being over-cautious in flattening out all whitespace? The w3c discussion on multiple lines in rfc822 [2] seems to suggest that whitespace can be 'unfolded' safely, so it seems a shame to be throwing it away when it can have important meaning. [1] http://python.cofman.dk/peps/pep-0345.html [2] http://www.w3.org/Protocols/rfc822/3_Lexical.html Stephen Emslie From ajaksu at gmail.com Fri Jan 18 16:47:27 2008 From: ajaksu at gmail.com (Daniel (ajax) Diniz) Date: Fri, 18 Jan 2008 13:47:27 -0200 Subject: [Catalog-sig] PyPI down :( Message-ID: <2d75d7660801180747r4cdf817ci8897cf84439627c0@mail.gmail.com> Error... There's been a problem with your request psycopg.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request While trying to open http://pypi.python.org/ (15:43 UTC) From amk at amk.ca Fri Jan 18 17:00:50 2008 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 18 Jan 2008 11:00:50 -0500 Subject: [Catalog-sig] PyPI down :( In-Reply-To: <2d75d7660801180747r4cdf817ci8897cf84439627c0@mail.gmail.com> References: <2d75d7660801180747r4cdf817ci8897cf84439627c0@mail.gmail.com> Message-ID: <20080118160050.GA8246@amk-desktop.matrixgroup.net> On Fri, Jan 18, 2008 at 01:47:27PM -0200, Daniel (ajax) Diniz wrote: > There's been a problem with your request Thanks for your report. I've restarted the web server and that seemed to clear the problem. --amk From martin at v.loewis.de Sat Jan 19 00:37:35 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 19 Jan 2008 00:37:35 +0100 Subject: [Catalog-sig] PyPI down :( In-Reply-To: <20080118160050.GA8246@amk-desktop.matrixgroup.net> References: <2d75d7660801180747r4cdf817ci8897cf84439627c0@mail.gmail.com> <20080118160050.GA8246@amk-desktop.matrixgroup.net> Message-ID: <4791383F.1060709@v.loewis.de> > On Fri, Jan 18, 2008 at 01:47:27PM -0200, Daniel (ajax) Diniz wrote: >> There's been a problem with your request > > Thanks for your report. I've restarted the web server and that > seemed to clear the problem. I've studied this a bit. Apparently, at 9:52 UTC, the OOM killer killed a process (apache2), then over the next minutes it killed more processes (apache2, moin.fcgi, pypi.fcgi, and eventually, at 10:04, it killed "postmaster". That apparently resolved the out-of-memory situation, but it also meant that PyPI broke. The postgres log file has all kinds of messages on stopped and crashed servers, rolled-back transactions etc, but eventually, postgres recovered. Not so PyPI, which (for some reason) failed to reconnect to the server. Looking at the tracebacks, it seems that psycopg now reports OperationalError for broken connections, and not (anymore?) InterfaceError. Regards, Martin From python at jwp.name Thu Jan 31 17:25:23 2008 From: python at jwp.name (James William Pye) Date: Thu, 31 Jan 2008 09:25:23 -0700 Subject: [Catalog-sig] Adding package pydoc to PyPI Message-ID: <20080131162523.GA50793@lit.jwp.name> I won't bore you with my musings of how relatively easy this should be, as I did on distutils-sig. So: I would like to extend PyPI to allow package maintainers to upload pydoc output associated with the package, so that it may be viewable on the PyPI website. (Well, not exactly pydoc or epydoc, but doc-string data and package structure. Set of XML documents, probably.. Something amenable to dynamically applying style, inter-package references, and cross references.) Thoughts? From martin at v.loewis.de Thu Jan 31 18:48:12 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 31 Jan 2008 18:48:12 +0100 Subject: [Catalog-sig] Adding package pydoc to PyPI In-Reply-To: <20080131162523.GA50793@lit.jwp.name> References: <20080131162523.GA50793@lit.jwp.name> Message-ID: <47A209DC.5060200@v.loewis.de> > I would like to extend PyPI to allow package maintainers to upload pydoc output > associated with the package, so that it may be viewable on the PyPI website. > (Well, not exactly pydoc or epydoc, but doc-string data and package structure. > Set of XML documents, probably.. Something amenable to dynamically applying > style, inter-package references, and cross references.) > > Thoughts? PyPI already supports reStructuredText. Why is that not good enough? Regards, Martin From python at jwp.name Thu Jan 31 20:44:45 2008 From: python at jwp.name (James William Pye) Date: Thu, 31 Jan 2008 12:44:45 -0700 Subject: [Catalog-sig] Adding package pydoc to PyPI In-Reply-To: <47A209DC.5060200@v.loewis.de> References: <20080131162523.GA50793@lit.jwp.name> <47A209DC.5060200@v.loewis.de> Message-ID: <20080131194445.GA51222@lit.jwp.name> On Thu, Jan 31, 2008 at 06:48:12PM +0100, "Martin v. L?wis" wrote: > PyPI already supports reStructuredText. Why is that not good enough? Because reST *alone* does not provide a *standard*, if not required, way to publish and view the pydoc and hierarchy of *all* packages.. [And the long answer... ;] I imagine you are suggesting that I pack everything into the long_description as some packages do. Ben Finney responded to my message on distutils-sig saying that this was entirely appropriate, and I do believe him. It's the only field/resource on PyPI available for doing so... However, it seems poor to pack all the information onto one page. Not to mention, I don't think distutils/setuptools has a command to render the long_description to contain the pydoc of the package--granted this is something for distutils/setuptools, not PyPI. So, here is my list of questions/reasons: 1. Can I make inter-package references to specific objects in others packages that won't break if PyPI were to change in some way using reST alone? That is, consider the case where your package has a dependency that you have no control over, and you want the pydoc that is generated to link to this package when a class in your package references a class in the dependency(superclass links, for instance). 2. I want to use the long_description more as quick marketing material than documentation. "In ~2000 words or less, this is why my package suites your needs..." ie, I want to use the description to *describe* my package as opposed to *documenting* it. 3. It would simply be nice to have a dedicated part of the package's place on PyPI specifically designed for displaying the package's module hierarchy and contents with the associated doc-strings. Finally, I understand that reST alone can be used to tackle a lot of this, but it does seem to put a substantial load on each developer to do so. Is something like this seen to be something that *should* be layered on top of PyPI using long_descriptions and reST as opposed to something that is integrated and simply expected or required? That is, a standard, if not required, way to publish and view the pydoc of a package. From martin at v.loewis.de Thu Jan 31 21:19:52 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 31 Jan 2008 21:19:52 +0100 Subject: [Catalog-sig] Adding package pydoc to PyPI In-Reply-To: <20080131194445.GA51222@lit.jwp.name> References: <20080131162523.GA50793@lit.jwp.name> <47A209DC.5060200@v.loewis.de> <20080131194445.GA51222@lit.jwp.name> Message-ID: <47A22D68.3040707@v.loewis.de> > I imagine you are suggesting that I pack everything into the long_description as > some packages do. Ben Finney responded to my message on distutils-sig saying > that this was entirely appropriate, and I do believe him. It's the only > field/resource on PyPI available for doing so... I also don't see a problem with that. > However, it seems poor to pack all the information onto one page. Not to > mention, I don't think distutils/setuptools has a command to render the > long_description to contain the pydoc of the package--granted this is > something for distutils/setuptools, not PyPI. Ah, so you want multiple pages per package. Why? Shouldn't you have some separate home page for the package, then? PyPI is just the package *index*, not a general project hosting service. As for generating a long_description - that's something that setup.py could do; no need to integrate this into distutils (unless there is a strong demand that a certain algorithm to include long_description gets included). > 1. Can I make inter-package references to specific objects in others packages > that won't break if PyPI were to change in some way using reST alone? I don't understand the question. What's an "inter-package reference"? What's a "specific object"? Give examples. > That is, consider the case where your package has a dependency that you have > no control over, and you want the pydoc that is generated to link to this > package when a class in your package references a class in the > dependency(superclass links, for instance). Again, please give an example. What is your package, what is the package you depend on, and how would you like the reference to look like? > 2. I want to use the long_description more as quick marketing material than > documentation. > "In ~2000 words or less, this is why my package suites your needs..." > ie, I want to use the description to *describe* my package as opposed to > *documenting* it. Right. You should put the description on the package's home page. > 3. It would simply be nice to have a dedicated part of the package's place on > PyPI specifically designed for displaying the package's module hierarchy and > contents with the associated doc-strings. Ok, so you do want a project hosting service. I don't think PyPI should provide that, or else people soon find out what a nice file sharing platform it is, and start uploading adult content, integrate it into link farms, and so on. > That is, a standard, if not required, way to publish and view the pydoc of a > package. Have you considered using SourceForge? They give you a project home page on which you can put arbitrary HTML (as long as it follows their usage restrictions). Regards, Martin