From manlio_perillo at libero.it Sat Dec 1 13:34:46 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sat, 01 Dec 2007 13:34:46 +0100 Subject: [Web-SIG] [ANN] nginx mod_wsgi 0.0.4 Message-ID: <475154E6.9080707@libero.it> I'm pleased to annunce the 0.0.4 release for nginx mod_wsgi. nginx mod_wsgi is an implementation of the WSGI PEP (http://www.python.org/dev/peps/pep-0333/) for the Nginx web server. Starting with this release nginx mod_wsgi is a fully conforming implementation of the WSGI specification (but there still can be some small problems). Here is the changelog: - Bug fix: `wsgi.errors.write` and `wsgi.errors.writelines` don't ignore conversion specification characters, used internally by `ngx_log_error` for message formatting. - Bug fix: when WSGI application does not set the `Date` or `Server` headers, nginx sends duplicate headers. - Added a new directive `wsgi_temp_path` and a new variable `NGX_HTTP_WSGI_TEMP_PATH` in the config script. Improved the write callable buffering support. - Fixed an annoying problem with paths without ending slash when using `--add-module` configure option. - Added patches for support of nginx < 0.5.32 and nginx 0.6.x. - Improved the configuration script, with correct external Python build dependencies handling. - Bug fix: changed again the memory management code, using a queue. - Bug fix: broken implementation of ngx_list_pop. This solves a memory corruption problem when serving large files (using nginx-file sample application) with a "slow" client or connection. - Improved the write callable so that it now is fully conforming with the WSGI spec. Replaced the WSGI_BUFFERED_WRITE macro with the wsgi_write_buffering directive. - Added an alternative implementation of the write callable, that buffers all the data. This function is the default one used, but this can be changed using the WSGI_BUFFERED_WRITE macro in the module configuration file. Added a wsgi_write_buffer_size directive to control the buffer size. Note that this implementation does not conform with the WSGI spec. - Added integrated support for cache validation in mod_wsgi using a custom header filter. Only the `Last-Modified` validator is supported. - Added experimental support for the `process` reload mechanism. - Implemented the write callable. This implementation is limited, since when data cannot be sent to OS buffers, an IOError exception is raised. - Implemented the start_response callable. - Implemented the `wsgi_output_buffering` directive. When this directive is disabled (the default) the buffer sent to the output filter is flushed. Note that the default write callable implementation uses an hack to be fully conforming (it sets the socket to blocking mode). Please note that mod_wsgi is not a "standard" nginx module; It should not be used for embedding Python applications inside your primary web server. Manlio Perillo From manlio_perillo at libero.it Sat Dec 1 14:55:51 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sat, 01 Dec 2007 14:55:51 +0100 Subject: [Web-SIG] [ANN] nginx mod_wsgi 0.0.4 In-Reply-To: <475154E6.9080707@libero.it> References: <475154E6.9080707@libero.it> Message-ID: <475167E7.3050500@libero.it> Manlio Perillo ha scritto: > I'm pleased to annunce the 0.0.4 release for nginx mod_wsgi. > > nginx mod_wsgi is an implementation of the WSGI PEP > (http://www.python.org/dev/peps/pep-0333/) for the Nginx web server. > > Starting with this release nginx mod_wsgi is a fully conforming > implementation of the WSGI specification (but there still can be some > small problems). > The Mercurial repository can be found here: http://hg.mperillo.ath.cx/nginx/mod_wsgi/ Manlio Perillo From manlio_perillo at libero.it Tue Dec 4 20:58:54 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Tue, 04 Dec 2007 20:58:54 +0100 Subject: [Web-SIG] [ANN] nginx mod_wsgi 0.0.5 Message-ID: <4755B17E.80108@libero.it> I'm pleased to annunce the 0.0.5 release for nginx mod_wsgi. nginx mod_wsgi is an implementation of the WSGI PEP (http://www.python.org/dev/peps/pep-0333/) for the Nginx web server. Here is the changelog: - Fixed a severe bug, caused by a wrong size declared for the StateObject. The value used was the size of the LogObject, and this was not causing problems only because sizeof(LogObject) > sizeof(StateObject). - Replaced sys.stderr with a LogObject instance, so that WSGI applications can use it for logging. - Bug fix: sys.argv is not initialized for the main interpreter. PySys_SetArgv is used for the initialization (it will add the nginx executable path directory to sys.path). - Removed the wsgi_python_path directive. Use the PYTHON_PATH environment variable with the `env` directive, instead. - Added support for the `env` directive. WSGI applications will now use only environment variables explicitly declared with the `env` directive; in previous revisions they inherited variables from the initial environment. - Improved the conformance for SCRIPT_NAME and PATH_INFO variables. The Mercurial repository of nginx mod_wsgi can be found at: http://hg.mperillo.ath.cx/nginx/mod_wsgi/ Manlio Perillo From graham.dumpleton at gmail.com Fri Dec 7 00:13:39 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 7 Dec 2007 10:13:39 +1100 Subject: [Web-SIG] WSGI, Python 3 and Unicode Message-ID: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> Has anyone had any thoughts about how WSGI is going to made to work with Python 3? >From what I understand about changes in Python 3, the main issue seems to be the removal of string type in its current form. This is an issue as WSGI specification currently states that status, header names/values and the items returned by the iterable must all be string instances. This is done to ensure that the application has done any conversions from Unicode, where knowledge about encoding would be known, before being passed to WSGI adapter. In Python 3 the default for string type objects will effectively be Unicode. Is WSGI going to be made to somehow cope with that, or will application instead be required to return byte string objects instead? We can never seem to get enough momentum going for WSGI 2.0, but with Python 3 coming along we may not have a choice but to come up with revised version of specification if we want WSGI to continue through to Python 3. Comments. Graham From pje at telecommunity.com Fri Dec 7 01:15:26 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Dec 2007 19:15:26 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.co m> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> Message-ID: <20071207001522.366D73A405E@sparrow.telecommunity.com> At 10:13 AM 12/7/2007 +1100, Graham Dumpleton wrote: >Has anyone had any thoughts about how WSGI is going to made to work >with Python 3? > > >From what I understand about changes in Python 3, the main issue seems >to be the removal of string type in its current form. > >This is an issue as WSGI specification currently states that status, >header names/values and the items returned by the iterable must all be >string instances. This is done to ensure that the application has done >any conversions from Unicode, where knowledge about encoding would be >known, before being passed to WSGI adapter. > >In Python 3 the default for string type objects will effectively be >Unicode. Is WSGI going to be made to somehow cope with that, or will >application instead be required to return byte string objects instead? WSGI already copes, actually. Note that Jython and IronPython have this issue today, and see: http://www.python.org/dev/peps/pep-0333/#unicode-issues """On Python platforms where the str or StringType type is in fact Unicode-based (e.g. Jython, IronPython, Python 3000, etc.), all "strings" referred to in this specification must contain only code points representable in ISO-8859-1 encoding (\u0000 through \u00FF, inclusive). It is a fatal error for an application to supply strings containing any other Unicode character or code point. Similarly, servers and gateways must not supply strings to an application containing any other Unicode characters.""" From guido at python.org Fri Dec 7 01:27:41 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Dec 2007 16:27:41 -0800 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207001522.366D73A405E@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> Message-ID: On Dec 6, 2007 4:15 PM, Phillip J. Eby wrote: > At 10:13 AM 12/7/2007 +1100, Graham Dumpleton wrote: > >Has anyone had any thoughts about how WSGI is going to made to work > >with Python 3? > > > > >From what I understand about changes in Python 3, the main issue seems > >to be the removal of string type in its current form. > > > >This is an issue as WSGI specification currently states that status, > >header names/values and the items returned by the iterable must all be > >string instances. This is done to ensure that the application has done > >any conversions from Unicode, where knowledge about encoding would be > >known, before being passed to WSGI adapter. > > > >In Python 3 the default for string type objects will effectively be > >Unicode. Is WSGI going to be made to somehow cope with that, or will > >application instead be required to return byte string objects instead? > > WSGI already copes, actually. Note that Jython and IronPython have > this issue today, and see: > > http://www.python.org/dev/peps/pep-0333/#unicode-issues > > """On Python platforms where the str or StringType type is in fact > Unicode-based (e.g. Jython, IronPython, Python 3000, etc.), all > "strings" referred to in this specification must contain only code > points representable in ISO-8859-1 encoding (\u0000 through \u00FF, > inclusive). It is a fatal error for an application to supply strings > containing any other Unicode character or code point. Similarly, > servers and gateways must not supply strings to an application > containing any other Unicode characters.""" That may work for IronPython/Jython, where encoded data is represented by the str type, but it won't be sufficient for Py3k, where encoded data is represented using the bytes type. IOW, in IronPython/Jython, u"\u1234".encode('utf-8') returns a str instance: '\xe1\x88\xb4'; but in Py3k, it returns a bytes instance: b'\xe1\x88\xb4'. The issue applies to input as well as output -- data read from a socket is also represented as bytes, unless you're using makefile() with a text mode and an encoding. You might want to look at how the unittests for wsgiref manage to pass in Py3k though. ;-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From foom at fuhm.net Fri Dec 7 01:23:40 2007 From: foom at fuhm.net (James Y Knight) Date: Thu, 6 Dec 2007 19:23:40 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207001522.366D73A405E@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> Message-ID: <2F3E2699-7D8A-48C9-BEF5-1B035E6815B3@fuhm.net> On Dec 6, 2007, at 7:15 PM, Phillip J. Eby wrote: > WSGI already copes, actually. Note that Jython and IronPython have > this issue today, and see: > > http://www.python.org/dev/peps/pep-0333/#unicode-issues > > """On Python platforms where the str or StringType type is in fact > Unicode-based (e.g. Jython, IronPython, Python 3000, etc.), all > "strings" referred to in this specification must contain only code > points representable in ISO-8859-1 encoding (\u0000 through \u00FF, > inclusive). It is a fatal error for an application to supply strings > containing any other Unicode character or code point. Similarly, > servers and gateways must not supply strings to an application > containing any other Unicode characters.""" It would seem very odd, however, for WSGI/python3 to use strings- restricted-to-0xFF for network I/O while everywhere else in python3 is going to use bytes for the same purpose. You'd have to modify your app to call write(unicodetext.encode('utf-8').decode('latin-1')) or so.... James From adam at atlas.st Fri Dec 7 02:08:04 2007 From: adam at atlas.st (Adam Atlas) Date: Thu, 6 Dec 2007 20:08:04 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> Message-ID: On 6 Dec 2007, at 18:13, Graham Dumpleton wrote: > In Python 3 the default for string type objects will effectively be > Unicode. Is WSGI going to be made to somehow cope with that, or will > application instead be required to return byte string objects instead? I'd say it would be best to only accept `bytes` objects; anything else would require some guesswork. Maybe, at most, it could try to encode returned Unicode objects as ISO-8859-1, and have it be an error if that's not possible. I was going to say that the gateway could accept Unicode objects if the user-agent sent a comprehensible Accept-Charset header, and thereby encode application output to the client's preferred character set on the fly (or to ISO-8859-1 if no Accept-Charset is provided), but that would complicate things for people writing gateways (and would be too implicit). It could be useful, but it would make more sense as a simple decorator for (almost-)WSGI applications. Perhaps it could go in wsgiref. From pje at telecommunity.com Fri Dec 7 02:45:47 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Dec 2007 20:45:47 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> Message-ID: <20071207014543.BE88D3A405E@sparrow.telecommunity.com> At 04:27 PM 12/6/2007 -0800, Guido van Rossum wrote: >You might want to look at how the unittests for wsgiref manage to pass >in Py3k though. ;-) Unless they've been changed, I'd assume it's because they work with strings exclusively, and never do any encoding or decoding (which is outside WSGI's scope, at least in the current version). From pje at telecommunity.com Fri Dec 7 02:48:55 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Dec 2007 20:48:55 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> Message-ID: <20071207014850.E69083A405E@sparrow.telecommunity.com> At 08:08 PM 12/6/2007 -0500, Adam Atlas wrote: >On 6 Dec 2007, at 18:13, Graham Dumpleton wrote: > > In Python 3 the default for string type objects will effectively be > > Unicode. Is WSGI going to be made to somehow cope with that, or will > > application instead be required to return byte string objects instead? > >I'd say it would be best to only accept `bytes` objects; anything else >would require some guesswork. Maybe, at most, it could try to encode >returned Unicode objects as ISO-8859-1, and have it be an error if >that's not possible. Actually, I'd prefer to look at it the other way around: a Python 3 WSGI server or middleware *may* accept bytes objects instead of str. This is relatively easy for the response side of things, but the request side is rather more difficult, since wsgi.input may need to be binary rather than text mode. (I think we can reasonably assume that wsgi.errors is a text mode stream, and should support a reasonable encoding.) From ubernostrum at gmail.com Fri Dec 7 03:23:23 2007 From: ubernostrum at gmail.com (James Bennett) Date: Thu, 6 Dec 2007 20:23:23 -0600 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207001522.366D73A405E@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> Message-ID: <21787a9f0712061823y6d44d564r80842436621fad72@mail.gmail.com> On Dec 6, 2007 6:15 PM, Phillip J. Eby wrote: > WSGI already copes, actually. Note that Jython and IronPython have > this issue today, and see: > > http://www.python.org/dev/peps/pep-0333/#unicode-issues I'm glad you brought that up, because it's been bugging me lately. That section is somewhat ambiguous as-is, because in one sentence applications are permitted to return strings encoded in a charset other than ISO-8859-1, but in another they are unequivocally forbidden to do so (with the "must not" in bold, even). And that's problematic not only because of the ambiguity, but because the increasing popularity of "AJAX" and web-based APIs is making it much more common for WSGI applications to generate responses of types which do not default to ISO-8859-1 -- e.g., XML and JSON, both of which default to UTF-8. Depending on how draconian one wishes to be when reading the relevant section of WSGI, it's possible to conclude that XML and JSON must always be transcoded/escaped to ISO-8859-1 -- with all the headaches that entails -- before being passed to a WSGI-compliant piece of software. And the slightly less strict reading of the spec -- that such gymnastics are required only when the string type of the Python implementation is Unicode-based -- will grow increasingly troublesome as/when Py3K enters production use. So as long as we're talking about this, could the proscriptions with respect to encoding perhaps be revisited and (hopefully) clarified/revised? -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." From guido at python.org Fri Dec 7 04:51:37 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Dec 2007 19:51:37 -0800 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207014543.BE88D3A405E@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> <20071207014543.BE88D3A405E@sparrow.telecommunity.com> Message-ID: On Dec 6, 2007 5:45 PM, Phillip J. Eby wrote: > At 04:27 PM 12/6/2007 -0800, Guido van Rossum wrote: > >You might want to look at how the unittests for wsgiref manage to pass > >in Py3k though. ;-) > > Unless they've been changed, I'd assume it's because they work with > strings exclusively, and never do any encoding or decoding (which is > outside WSGI's scope, at least in the current version). Indeed, that seems mostly to be the case. But this means that any application that wants to emit characters outside Latin-1 cannot just encode() those characters, since the encode() output will be bytes and those will not be accepted by the WSGI API. OTOH sending non-Latin-1 characters without encoding would violate the standard. So something needs to give... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Fri Dec 7 05:00:02 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 06 Dec 2007 22:00:02 -0600 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207014850.E69083A405E@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> Message-ID: <4758C542.4020909@colorstudy.com> Phillip J. Eby wrote: > At 08:08 PM 12/6/2007 -0500, Adam Atlas wrote: > >> On 6 Dec 2007, at 18:13, Graham Dumpleton wrote: >>> In Python 3 the default for string type objects will effectively be >>> Unicode. Is WSGI going to be made to somehow cope with that, or will >>> application instead be required to return byte string objects instead? >> I'd say it would be best to only accept `bytes` objects; anything else >> would require some guesswork. Maybe, at most, it could try to encode >> returned Unicode objects as ISO-8859-1, and have it be an error if >> that's not possible. > > Actually, I'd prefer to look at it the other way around: a Python 3 > WSGI server or middleware *may* accept bytes objects instead of str. > > This is relatively easy for the response side of things, but the > request side is rather more difficult, since wsgi.input may need to > be binary rather than text mode. (I think we can reasonably assume > that wsgi.errors is a text mode stream, and should support a > reasonable encoding.) wsgi.input definitely seems like it should be bytes to me. Unless we want to put the encoding process into the server. Not entirely infeasible, but a bit of a strain. And the request body might very well be binary, e.g., on a PUT. The CGI keys in the environment don't feel at all like bytes to me, but then they aren't unicode either. They can be unicode, again given a bit of work on the server side. Though unfortunately browsers are very poor at indicating their encoding for requests, and it ends up being policy and configuration as much as anything that determines the encoding of stuff like wsgi.input. I believe all request paths are UTF8 (?), but I'm not sure about QUERY_STRING. I'm a little fuzzy on some of the details there. The actual response body should also be bytes. Unless again we want to introduce upstream encoding. This does make everything feel more complicated. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From guido at python.org Fri Dec 7 06:06:26 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Dec 2007 21:06:26 -0800 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <4758C542.4020909@colorstudy.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> Message-ID: On Dec 6, 2007 8:00 PM, Ian Bicking wrote: > Phillip J. Eby wrote: > > At 08:08 PM 12/6/2007 -0500, Adam Atlas wrote: > > > >> On 6 Dec 2007, at 18:13, Graham Dumpleton wrote: > >>> In Python 3 the default for string type objects will effectively be > >>> Unicode. Is WSGI going to be made to somehow cope with that, or will > >>> application instead be required to return byte string objects instead? > >> I'd say it would be best to only accept `bytes` objects; anything else > >> would require some guesswork. Maybe, at most, it could try to encode > >> returned Unicode objects as ISO-8859-1, and have it be an error if > >> that's not possible. > > > > Actually, I'd prefer to look at it the other way around: a Python 3 > > WSGI server or middleware *may* accept bytes objects instead of str. > > > > This is relatively easy for the response side of things, but the > > request side is rather more difficult, since wsgi.input may need to > > be binary rather than text mode. (I think we can reasonably assume > > that wsgi.errors is a text mode stream, and should support a > > reasonable encoding.) > > wsgi.input definitely seems like it should be bytes to me. Unless we > want to put the encoding process into the server. Not entirely > infeasible, but a bit of a strain. And the request body might very well > be binary, e.g., on a PUT. > > The CGI keys in the environment don't feel at all like bytes to me, but > then they aren't unicode either. They can be unicode, again given a bit > of work on the server side. Though unfortunately browsers are very poor > at indicating their encoding for requests, and it ends up being policy > and configuration as much as anything that determines the encoding of > stuff like wsgi.input. I believe all request paths are UTF8 (?), but > I'm not sure about QUERY_STRING. I'm a little fuzzy on some of the > details there. > > The actual response body should also be bytes. Unless again we want to > introduce upstream encoding. > > This does make everything feel more complicated. It's the same level of complexity you run into as soon as you want to handle Unicode with WSGI in 2.x though, as it is caused by something outside our control (HTTP and browsers). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pywebsig at xhaus.com Fri Dec 7 10:39:09 2007 From: pywebsig at xhaus.com (Alan Kennedy) Date: Fri, 7 Dec 2007 16:39:09 +0700 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <2F3E2699-7D8A-48C9-BEF5-1B035E6815B3@fuhm.net> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> <2F3E2699-7D8A-48C9-BEF5-1B035E6815B3@fuhm.net> Message-ID: <4a951aa00712070139h7b457bb6y85b8fc11e3363e72@mail.gmail.com> [Phillip] >> WSGI already copes, actually. Note that Jython and IronPython have >> this issue today, and see: >> >> http://www.python.org/dev/peps/pep-0333/#unicode-issues [James] > It would seem very odd, however, for WSGI/python3 to use strings- > restricted-to-0xFF for network I/O while everywhere else in python3 is > going to use bytes for the same purpose. I think it's worth pointing out the reason for the current restriction to iso-8859-1 is *because* python did not have a bytes type at the time the WSGI spec was drawn up. IIRC, the bytes type had not yet even been proposed for Py3K. Cpython effectively held all byte sequences as strings, a paradigm which is (still) followed by jython (not sure about ironpython). The restriction to iso-8859-1 is really a distraction; iso-8859-1 is used simply as an identity encoding that also enforces that all "bytes" in the string have a value from 0x00 to 0xff, so that they are suitable for byte-oriented IO. So, in output terms at least, WSGI *is* a byte-oriented protocol. The problem is the python-the-language didn't have support for bytes at the time WSGI was designed. [James] > You'd have to modify your app > to call write(unicodetext.encode('utf-8').decode('latin-1')) or so.... Did you mean: write(unicodetext.encode('utf-8').encode('latin-1'))? Either way, the second encode is not required; write(unicodetext.encode('utf-8')) is sufficient, since it will generate a byte-sequence(string) which will (actually "should": see (*) note below) pass the following test. try: wsgi_response_data.encode('iso-8859-1') except UnicodeError: # Illegal WSGI response data! On a side note, it's worth noting that Philip Jenvey's excellent rework of the jython IO subsystem to use java.nio is fundamentally byte oriented. http://www.nabble.com/fileno-support-is-not-in-jython.-Reason--t4750734.html http://fisheye3.cenqua.com/browse/jython/trunk/jython/src/org/python/core/io Because it is based on the new IO design for Python 3K, as described in PEP 3116 http://www.python.org/dev/peps/pep-3116/ Regards, Alan. [*] Although I notice that cpython 2.5, for a reason I don't fully understand, fails this particular encoding sequence. (Maybe it's to do with the possibility that the result of an encode operation is no longer an encodable string?) Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> response = u"interferon-gamma (IFN-\u03b3) responses in cattle" >>> response.encode('utf-8').encode('latin-1') Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 22: ordinal not in range(128) >>> Meaning that to enforce the WSGI iso-8859-1 convention on cpython 2.5, you would have to carry out this rigmarole >>> response.encode('utf-8').decode('latin-1').encode('latin-1') 'interferon-gamma (IFN-\xce\xb3) responses in cattle' >>> Perhaps this behaviour is an artifact of the cpython implementation? Whereas jython passes it just fine (and correctly, IMHO) Jython 2.2.1 on java1.4.2_15 Type "copyright", "credits" or "license" for more information. >>> response = u"interferon-gamma (IFN-\u03b3) responses in cattle" >>> response.encode('utf-8') 'interferon-gamma (IFN-\xCE\xB3) responses in cattle' >>> response.encode('utf-8').encode('latin-1') 'interferon-gamma (IFN-\xCE\xB3) responses in cattle' >>> From t.broyer at gmail.com Fri Dec 7 11:16:13 2007 From: t.broyer at gmail.com (Thomas Broyer) Date: Fri, 7 Dec 2007 11:16:13 +0100 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <4a951aa00712070139h7b457bb6y85b8fc11e3363e72@mail.gmail.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> <2F3E2699-7D8A-48C9-BEF5-1B035E6815B3@fuhm.net> <4a951aa00712070139h7b457bb6y85b8fc11e3363e72@mail.gmail.com> Message-ID: I wasn't there when PEP-333 was written, nor have I any implication in any Python development, but here are my thoughts: 2007/12/7, Alan Kennedy: > > I think it's worth pointing out the reason for the current restriction > to iso-8859-1 is *because* python did not have a bytes type at the > time the WSGI spec was drawn up. IIRC, the bytes type had not yet even > been proposed for Py3K. Cpython effectively held all byte sequences as > strings, a paradigm which is (still) followed by jython (not sure > about ironpython). > > The restriction to iso-8859-1 is really a distraction; iso-8859-1 is > used simply as an identity encoding that also enforces that all > "bytes" in the string have a value from 0x00 to 0xff, so that they are > suitable for byte-oriented IO. So, in output terms at least, WSGI *is* > a byte-oriented protocol. The problem is the python-the-language > didn't have support for bytes at the time WSGI was designed. If you're talking about the "output stream", then yes, it's all about bytes (or should be). But at the status and headers level, HTTP/1.1 is fundamentally ISO-8859-1-encoded. See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.2 (the note about *TEXT) http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 (field-content is *TEXT, among other things) http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1 (Reason-Phrase is *TEXT) -- Thomas Broyer From pywebsig at xhaus.com Fri Dec 7 12:24:03 2007 From: pywebsig at xhaus.com (Alan Kennedy) Date: Fri, 7 Dec 2007 18:24:03 +0700 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207001522.366D73A405E@sparrow.telecommunity.com> <2F3E2699-7D8A-48C9-BEF5-1B035E6815B3@fuhm.net> <4a951aa00712070139h7b457bb6y85b8fc11e3363e72@mail.gmail.com> Message-ID: <4a951aa00712070324u271515d9r12d57d95ccbc5135@mail.gmail.com> [Alan] >> The restriction to iso-8859-1 is really a distraction; iso-8859-1 is >> used simply as an identity encoding that also enforces that all >> "bytes" in the string have a value from 0x00 to 0xff, so that they are >> suitable for byte-oriented IO. So, in output terms at least, WSGI *is* >> a byte-oriented protocol. The problem is the python-the-language >> didn't have support for bytes at the time WSGI was designed. [Thomas] > If you're talking about the "output stream", then yes, it's all about > bytes (or should be). Indeed, I was only talking about output, specifically the response body. > But at the status and headers level, HTTP/1.1 is > fundamentally ISO-8859-1-encoded. Agreed. That is why the WSGI spec also states """ Note also that strings passed to start_response() as a status or as response headers must follow RFC 2616 with respect to encoding. That is, they must either be ISO-8859-1 characters, or use RFC 2047 MIME encoding. """ So in order to use non-ISO-8859-1 characters in response status strings or headers, you must use RFC 2047. As confirmed by the links you posted, this is a HTTP restriction, not a WSGI restriction. Regards, Alan. From and-py at doxdesk.com Fri Dec 7 20:11:12 2007 From: and-py at doxdesk.com (Andrew Clover) Date: Fri, 07 Dec 2007 20:11:12 +0100 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> Message-ID: <47599AD0.4070701@doxdesk.com> Adam Atlas wrote: > I'd say it would be best to only accept `bytes` objects +1. HTTP is inherently byte-based. Any translation between bytes and unicode characters should be done at a higher level, by whatever web framework is living above WSGI. -- And Clover mailto:and at doxdesk.com http://www.doxdesk.com/ From pje at telecommunity.com Fri Dec 7 20:55:47 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Dec 2007 14:55:47 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> Message-ID: <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> So here are my recommendations so far for the addendum to WSGI *1.0* for Python 3.0 (I expect we can be more strict for WSGI 2.0): * When running under Python 3, applications SHOULD produce bytes output and headers * When running under Python 3, servers and gateways MUST accept strings as application output or headers, under the existing rules (i.e., s.encode('latin-1') must convert the string to bytes without an exception) * When running under Python 3, servers MUST provide CGI HTTP variables as strings, decoded from the headers using HTTP standard encodings (i.e. latin-1 + RFC 2047) (Open question: are there any CGI or WSGI variables that should NOT be strings?) * When running under Python 3, servers MUST make wsgi.input a binary (byte) stream * When running under Python 3, servers MUST provide a text stream for wsgi.errors These rules are intended to simplify the porting of existing code. Notice, for example, that these rules allow middleware to pass strings through unchanged, since they are not required to produce bytes output or headers. Unfortunately, wsgi.input can't be coded around, but for most frameworks this should be a single point of pain. In fact, if the 'cgi' stdlib module is made compatible with bytes, only the rare framework that rolls its own multipart parser or otherwise directly manipulates put/post data will be affected. Code that just takes the input and writes it to a file won't be bothered, either. Comments or questions? From ianb at colorstudy.com Fri Dec 7 21:24:56 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 07 Dec 2007 14:24:56 -0600 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> Message-ID: <4759AC18.3010907@colorstudy.com> Phillip J. Eby wrote: > So here are my recommendations so far for the addendum to WSGI *1.0* for > Python 3.0 (I expect we can be more strict for WSGI 2.0): > > * When running under Python 3, applications SHOULD produce bytes output > and headers > > * When running under Python 3, servers and gateways MUST accept strings > as application output or headers, under the existing rules (i.e., > s.encode('latin-1') must convert the string to bytes without an exception) > > * When running under Python 3, servers MUST provide CGI HTTP variables > as strings, decoded from the headers using HTTP standard encodings (i.e. > latin-1 + RFC 2047) (Open question: are there any CGI or WSGI variables > that should NOT be strings?) I believe that SCRIPT_NAME/PATH_INFO would be UTF8 encoded, not latin1. That is, after you urldecode the values (as WSGI asks you to do) proper conversion to text is to decode it as UTF8. I'm a bit confused on how HTTP_COOKIE gets encoded. And QUERY_STRING also confuses me. Is this all compatible with os.environ in py3k? I don't care that much if it does, but as the starting point for CGI it would be interesting if it stays in sync. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From foom at fuhm.net Fri Dec 7 21:53:03 2007 From: foom at fuhm.net (James Y Knight) Date: Fri, 7 Dec 2007 15:53:03 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> Message-ID: On Dec 7, 2007, at 2:55 PM, Phillip J. Eby wrote: > * When running under Python 3, servers MUST provide CGI HTTP > variables as strings, decoded from the headers using HTTP standard > encodings (i.e. latin-1 + RFC 2047) (Open question: are there any > CGI or WSGI variables that should NOT be strings?) A WSGI gateway should *not* decode headers using RFC 2047. It actually *cannot*, without knowing the structure of that particular header, because only TEXT tokens are encoded that way. In addition, I know of nobody who actually implements RFC 2047 decoding of http header values...nothing really uses it. (of course I don't know of all implementations out there.) On Dec 7, 2007, at 3:24 PM, Ian Bicking wrote: > I believe that SCRIPT_NAME/PATH_INFO would be UTF8 encoded, not > latin1. > That is, after you urldecode the values (as WSGI asks you to do) > proper conversion to text is to decode it as UTF8. Surely not! URLs aren't always utf-8 encoded, only often. James From and-py at doxdesk.com Fri Dec 7 23:46:32 2007 From: and-py at doxdesk.com (Andrew Clover) Date: Fri, 07 Dec 2007 23:46:32 +0100 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> Message-ID: <4759CD48.7000305@doxdesk.com> James Y Knight wrote: > In addition, I know of nobody who actually implements RFC 2047 > decoding of http header values...nothing really uses it. (of > course I don't know of all implementations out there.) Certainly no browser supports it, which makes the point moot for WSGI. Most browsers, when quoting a header parameter, simply encode using the previous page's charset and put quotes around it... even if the parameter has a quote or control codes in it. Ian wrote: > Is this all compatible with os.environ in py3k? In 3.0a2 os.environ has Unicode strings for both keys and values. This is correct for Windows where environment variables are explicitly Unicode, but questionable (IMO) for Unix where they're really bytes that may or may not represent decodeable Unicode strings. >> SCRIPT_NAME/PATH_INFO This already causes problems in Windows CGI applications! Because these are passed in environment variables, IIS* has to decode the submitted bytes to Unicode first. It seems always to choose UTF-8 for this job, which I suppose is the least bad guess, but hardly infallible. (* - haven't tested this with Apache for Windows yet.) In Python 2.x, os.environ being byte strings, Python/the C library then has to encode them back to bytes, which I believe ends up using the system codepage. Since the system codepage is never UTF-8 on Windows this means not only that the bytes read back from eg. PATH_INFO are not the same as the original bytes submitted to the web server, but that if there are characters outside the system codepage submitted, they'll be unrecoverable. If os.environ remains Unicode in Unix and WSGI follows it (as it must if CGI-invoked WSGI is to continue working smoothly), webapps that try to allow for non-ASCII characters in URLs are likely to get some nasty deployment problems that depend on the system encoding setting, something that will be particularly troublesome for end-users to debug and fix. OTOH making the dictionaries reflect the underlying OS's conception of environment variables means users of os.environ and WSGI will have to be able to cope with both bytes and unicode, which would also be a big annoyance. In summary: urgh, this is all messy and 'orrible. -- And Clover mailto:and at doxdesk.com http://www.doxdesk.com/ From foom at fuhm.net Sat Dec 8 00:18:58 2007 From: foom at fuhm.net (James Y Knight) Date: Fri, 7 Dec 2007 18:18:58 -0500 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <4759CD48.7000305@doxdesk.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> <4759CD48.7000305@doxdesk.com> Message-ID: <8CEC1293-A168-4843-8CA6-4491B819AADB@fuhm.net> On Dec 7, 2007, at 5:46 PM, Andrew Clover wrote: > OTOH making the dictionaries reflect the underlying OS's conception of > environment variables means users of os.environ and WSGI will have > to be > able to cope with both bytes and unicode, which would also be a big > annoyance. > > In summary: urgh, this is all messy and 'orrible. I suppose this is more a question for python-dev, but, it'd be really nice if Python on Windows made it look like the windows system encoding was always UTF-8. That is, bytestrings used for open/ os.environ/argv/etc. are always encoded/decoded in utf-8, not the broken-platform-encoding. Then the same code would work just as well on unix as it does on windows. Actually, I bet I could implement that today, just by wrapping some stuff....hmmm... James From graham.dumpleton at gmail.com Sat Dec 8 09:37:59 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 8 Dec 2007 19:37:59 +1100 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> Message-ID: <88e286470712080037p5ab17723ua98b939de784ad69@mail.gmail.com> On 08/12/2007, Phillip J. Eby wrote: > * When running under Python 3, servers MUST provide a text stream for > wsgi.errors In Python 3, what happens if user code attempts to output to a text stream a byte string? Ie., what would be displayed? Also, if wsgi.errors is a text stream, presume that if a WSGI adapter has to internally map this to a C char* like API for logging that it would need to apply standard Python encoding to yield usable char* string for output. Graham From guido at python.org Sun Dec 9 03:48:56 2007 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Dec 2007 18:48:56 -0800 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <88e286470712080037p5ab17723ua98b939de784ad69@mail.gmail.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> <88e286470712080037p5ab17723ua98b939de784ad69@mail.gmail.com> Message-ID: On Dec 8, 2007 12:37 AM, Graham Dumpleton wrote: > On 08/12/2007, Phillip J. Eby wrote: > > * When running under Python 3, servers MUST provide a text stream for > > wsgi.errors > > In Python 3, what happens if user code attempts to output to a text > stream a byte string? Ie., what would be displayed? Nothing. You get a TypeError. > Also, if wsgi.errors is a text stream, presume that if a WSGI adapter > has to internally map this to a C char* like API for logging that it > would need to apply standard Python encoding to yield usable char* > string for output. The encoding can/must be specified per text stream. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From graham.dumpleton at gmail.com Mon Dec 10 04:56:55 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Mon, 10 Dec 2007 14:56:55 +1100 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> <88e286470712080037p5ab17723ua98b939de784ad69@mail.gmail.com> Message-ID: <88e286470712091956r50966ae5p6ddac122c2c32ce2@mail.gmail.com> On 09/12/2007, Guido van Rossum wrote: > On Dec 8, 2007 12:37 AM, Graham Dumpleton wrote: > > On 08/12/2007, Phillip J. Eby wrote: > > > * When running under Python 3, servers MUST provide a text stream for > > > wsgi.errors > > > > In Python 3, what happens if user code attempts to output to a text > > stream a byte string? Ie., what would be displayed? > > Nothing. You get a TypeError. Hmmm, this in itself could be quite a pain for existing code where people have added debug code to print out details from request headers (if now to be passed as bytes), or part of the request content. What is the suggested way of best dumping out bytes for debugging purposes so one does not have to worry about encoding issues, just use repr()? > > Also, if wsgi.errors is a text stream, presume that if a WSGI adapter > > has to internally map this to a C char* like API for logging that it > > would need to apply standard Python encoding to yield usable char* > > string for output. > > The encoding can/must be specified per text stream. But what should the encoding associated with the wsgi.errors stream be? If code which outputs text to wsgi.errors can use any valid Unicode character, if one sets it to US-ASCII encoding, then chance that logging output will fail because of characters not being valid in that character set. If one instead uses UTF-8, then potentially have issues where that byte string coming out other end of text stream is passed to C API functions. Issues might arise here where C API not expecting variable width character encoding. I'll freely admit I am not across all this Unicode encode/decode stuff as I don't generally have to deal with foreign languages, but seems to be a few missing details in this area which need to be filled out for a modified WSGI specification. Graham From chris at simplistix.co.uk Mon Dec 10 13:08:25 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 10 Dec 2007 12:08:25 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? Message-ID: <475D2C39.2000409@simplistix.co.uk> Hi All, What's the best way to serve large files (say detailed images or pdfs) from a wsgi app? Is there special support for this? Also, does wsgi offer anything to help with http range requests and the like? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Mon Dec 10 15:45:52 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Dec 2007 09:45:52 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <475D2C39.2000409@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> Message-ID: <20071210144548.652B93A40A4@sparrow.telecommunity.com> At 12:08 PM 12/10/2007 +0000, Chris Withers wrote: >Hi All, > >What's the best way to serve large files (say detailed images or pdfs) >from a wsgi app? > >Is there special support for this? That's what the iteration part of the protocol is for (well, and a few other things). If you're not serving large files, it's counterproductive to yield multiple values or to call write() multiple times. From guido at python.org Mon Dec 10 19:31:20 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Dec 2007 10:31:20 -0800 Subject: [Web-SIG] WSGI, Python 3 and Unicode In-Reply-To: <88e286470712091956r50966ae5p6ddac122c2c32ce2@mail.gmail.com> References: <88e286470712061513h2ae2c89bod67627490a4fb318@mail.gmail.com> <20071207014850.E69083A405E@sparrow.telecommunity.com> <4758C542.4020909@colorstudy.com> <20071207195546.3E6D73A40AE@sparrow.telecommunity.com> <88e286470712080037p5ab17723ua98b939de784ad69@mail.gmail.com> <88e286470712091956r50966ae5p6ddac122c2c32ce2@mail.gmail.com> Message-ID: On Dec 9, 2007 7:56 PM, Graham Dumpleton wrote: > On 09/12/2007, Guido van Rossum wrote: > > On Dec 8, 2007 12:37 AM, Graham Dumpleton wrote: > > > On 08/12/2007, Phillip J. Eby wrote: > > > > * When running under Python 3, servers MUST provide a text stream for > > > > wsgi.errors > > > > > > In Python 3, what happens if user code attempts to output to a text > > > stream a byte string? Ie., what would be displayed? > > > > Nothing. You get a TypeError. > > Hmmm, this in itself could be quite a pain for existing code where > people have added debug code to print out details from request headers > (if now to be passed as bytes), or part of the request content. Sorry, I was just talking about the write() method on a text stream. The print() function in 3.0 will print the repr() of the bytes. Example: Python 3.0a2 (py3k, Dec 10 2007, 09:38:42) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> a = b"xyz" >>> print(a) b'xyz' >>> b = b"abc\377def" >>> print(b) b'abc\xffdef' >>> (Note that this works because print() always calls str() on the argument and bytes.str is defined to be the same as bytes.repr.) > What is the suggested way of best dumping out bytes for debugging > purposes so one does not have to worry about encoding issues, just use > repr()? Just use print(). > > > Also, if wsgi.errors is a text stream, presume that if a WSGI adapter > > > has to internally map this to a C char* like API for logging that it > > > would need to apply standard Python encoding to yield usable char* > > > string for output. > > > > The encoding can/must be specified per text stream. > > But what should the encoding associated with the wsgi.errors stream be? Depends on the platform and your requirements. > If code which outputs text to wsgi.errors can use any valid Unicode > character, if one sets it to US-ASCII encoding, then chance that > logging output will fail because of characters not being valid in that > character set. If one instead uses UTF-8, then potentially have issues > where that byte string coming out other end of text stream is passed > to C API functions. Issues might arise here where C API not expecting > variable width character encoding. > > I'll freely admit I am not across all this Unicode encode/decode stuff > as I don't generally have to deal with foreign languages, but seems to > be a few missing details in this area which need to be filled out for > a modified WSGI specification. The goal of this part of Py3k is to make it more obvious when you haven't thought through your encoding issues enough by failing as soon as (encoded) bytes meet (decoded) characters. Of course, you can still run into delayed trouble by using an inappropriate encoding, which only shows up when there is an actual encoding or decoding error; but at least you will have carefully distinguished between encoded and decoded text throughout your program, so the fix is now to change the encoding rather than having to restructure your code to properly separate encoded and decoded text. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From MDiPierro at cti.depaul.edu Tue Dec 11 23:24:26 2007 From: MDiPierro at cti.depaul.edu (DiPierro, Massimo) Date: Tue, 11 Dec 2007 16:24:26 -0600 Subject: [Web-SIG] Gluon again Message-ID: Hello everybody, It seems I will not have a chance to talk about Gluon at PyCON08 so I just wanted to let you know that Gluon 1.13 is out (backwards compatible since v1.0) with: a new video tutorial (45 minutes) http://vimeo.com/428474 a better ORM (including SQLite, MySQL and PostgreSQL support): http://mdp.cti.depaul.edu/examples/default/orm_api some online documentation: http://mdp.cti.depaul.edu/examples/global/vars new granular caching mechanism and examples: http://mdp.cti.depaul.edu/examples/default/index runs off a USB drive (no installation required on win and osx) and windows ce (from source with pythonce2.5): http://mdp.cti.depaul.edu/examples/static/gluonce.jpg a users group: http://groups.google.com/group/gluon a flagship Gluon-powered web site: http:/www.appealmypropertytaxes.com I thanks to those of you who sent me comments and/or criticism since they helped me make it better. This mailing list was very helpful. Have fun! Massimo From manlio_perillo at libero.it Wed Dec 12 20:17:30 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 12 Dec 2007 20:17:30 +0100 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH Message-ID: <476033CA.6080706@libero.it> Hi. The CGI spec says: """Servers are not required to create metavariables for all the request header fields that they receive. In particular, they MAY decline to make available any header fields carrying authentication information, such as "Authorization", or which are available to the script via other metavariables, such as "Content-Length" and "Content-Type".""" However wsgiref.validate raises an exception: AssertionError: Environment should not have the key: HTTP_CONTENT_TYPE (use CONTENT_TYPE instead) Why don't just raise a warning? Thanks Manlio Perillo From ianb at colorstudy.com Wed Dec 12 20:28:27 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 12 Dec 2007 13:28:27 -0600 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <476033CA.6080706@libero.it> References: <476033CA.6080706@libero.it> Message-ID: <4760365B.1040900@colorstudy.com> Manlio Perillo wrote: > Hi. > > The CGI spec says: > """Servers are not required to create metavariables for all the request > header fields that they receive. In particular, they MAY decline to make > available any header fields carrying authentication information, such as > "Authorization", or which are available to the script via other > metavariables, such as "Content-Length" and "Content-Type".""" > > > However wsgiref.validate raises an exception: > AssertionError: Environment should not have the key: HTTP_CONTENT_TYPE > (use CONTENT_TYPE instead) > > > Why don't just raise a warning? It's the sign of some ambiguity; the Content-Type header should be CONTENT_TYPE, and Content-Length should be CONTENT_LENGTH. If they show up as HTTP_CONTENT_TYPE etc then something weird happened; probably code that didn't take these special-case headers into account. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From manlio_perillo at libero.it Wed Dec 12 21:22:08 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 12 Dec 2007 21:22:08 +0100 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <4760365B.1040900@colorstudy.com> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> Message-ID: <476042F0.3010608@libero.it> Ian Bicking ha scritto: > Manlio Perillo wrote: >> Hi. >> >> The CGI spec says: >> """Servers are not required to create metavariables for all the >> request header fields that they receive. In particular, they MAY >> decline to make available any header fields carrying authentication >> information, such as "Authorization", or which are available to the >> script via other metavariables, such as "Content-Length" and >> "Content-Type".""" >> >> >> However wsgiref.validate raises an exception: >> AssertionError: Environment should not have the key: HTTP_CONTENT_TYPE >> (use CONTENT_TYPE instead) >> >> >> Why don't just raise a warning? > > It's the sign of some ambiguity; the Content-Type header should be > CONTENT_TYPE, and Content-Length should be CONTENT_LENGTH. If they show > up as HTTP_CONTENT_TYPE etc then something weird happened; probably code > that didn't take these special-case headers into account. > Well, its nothing weird. The gateway is not required to remove HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH; at least this is what CGI says (and WSGI says nothing about this). In fact in mod_wsgi for Nginx I don't remove these two variables, since it requires extra code (but I will remove them, if there is a good reason for doing so). Thanks Manlio perillo From ianb at colorstudy.com Wed Dec 12 21:40:42 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 12 Dec 2007 14:40:42 -0600 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <476042F0.3010608@libero.it> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> <476042F0.3010608@libero.it> Message-ID: <4760474A.2030007@colorstudy.com> Manlio Perillo wrote: > Ian Bicking ha scritto: >> Manlio Perillo wrote: >>> Hi. >>> >>> The CGI spec says: >>> """Servers are not required to create metavariables for all the >>> request header fields that they receive. In particular, they MAY >>> decline to make available any header fields carrying authentication >>> information, such as "Authorization", or which are available to the >>> script via other metavariables, such as "Content-Length" and >>> "Content-Type".""" >>> >>> >>> However wsgiref.validate raises an exception: >>> AssertionError: Environment should not have the key: >>> HTTP_CONTENT_TYPE (use CONTENT_TYPE instead) >>> >>> >>> Why don't just raise a warning? >> >> It's the sign of some ambiguity; the Content-Type header should be >> CONTENT_TYPE, and Content-Length should be CONTENT_LENGTH. If they >> show up as HTTP_CONTENT_TYPE etc then something weird happened; >> probably code that didn't take these special-case headers into account. >> > > Well, its nothing weird. > > The gateway is not required to remove HTTP_CONTENT_TYPE and > HTTP_CONTENT_LENGTH; > at least this is what CGI says (and WSGI says nothing about this). > > In fact in mod_wsgi for Nginx I don't remove these two variables, since > it requires extra code (but I will remove them, if there is a good > reason for doing so). I never really read through the spec for this, but by convention all servers I know of do remove these. Having the same header show up in multiple places seems dangerous (where the two values may not match, especially if the request has one of those keys rewritten). And of course it's not much code to remove them. There's some things that technically are okay with WSGI, but in practice are bad (like leaving out QUERY_STRING, which can cause very weird bugs due to how the cgi module was written). -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From manlio_perillo at libero.it Wed Dec 12 22:18:31 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 12 Dec 2007 22:18:31 +0100 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <4760474A.2030007@colorstudy.com> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> <476042F0.3010608@libero.it> <4760474A.2030007@colorstudy.com> Message-ID: <47605027.6040503@libero.it> Ian Bicking ha scritto: > [...] >> >> The gateway is not required to remove HTTP_CONTENT_TYPE and >> HTTP_CONTENT_LENGTH; >> at least this is what CGI says (and WSGI says nothing about this). >> >> In fact in mod_wsgi for Nginx I don't remove these two variables, >> since it requires extra code (but I will remove them, if there is a >> good reason for doing so). > > I never really read through the spec for this, but by convention all > servers I know of do remove these. Having the same header show up in > multiple places seems dangerous (where the two values may not match, > especially if the request has one of those keys rewritten). Well, with mod_wsgi for nginx an user *can* rewrite *every* variable passed to the WSGI environment (with the exception of the variables in the wsgi namespace, SCRIPT_NAME and PATH_INFO) using the `wsgi_var` directive (to be precise in the current version the user can not override HTTP_ variables since the `wsgi_var` variables are inserted in the enviroment dictionary before the HTTP_ variables, but I'm going to change this right now). WSGI variables are all added using the `wsgi_var` directive. > And of > course it's not much code to remove them. There's some things that > technically are okay with WSGI, but in practice are bad (like leaving > out QUERY_STRING, which can cause very weird bugs due to how the cgi > module was written). > QUERY_STRING can be a problem for me, since I don't add to the environment dictionary empty variables, with the exception of SCRIPT_NAME and PATH_INFO. Well, not a real problem, since it easy to make sure that it is always present in the WSGI environment dictionary. By the way: the CGI spec (http://cgi-spec.golux.com/draft-coar-cgi-v11-03-clean.html) requires QUERY_STRING to be *alway* present, but the WSGI spec allows it to be absent. P.S.: I'm reading the cgi module and it only uses argv[1]: if sys.argv[1:]: qs = sys.argv[1] I'm not sure to understand. Moreover in mod_wsgi for nginx, I call PySys_SetArgv, so the WSGI application sees the command line options passed to nginx. Thanks Manlio Perillo From ianb at colorstudy.com Wed Dec 12 22:22:17 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 12 Dec 2007 15:22:17 -0600 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <47605027.6040503@libero.it> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> <476042F0.3010608@libero.it> <4760474A.2030007@colorstudy.com> <47605027.6040503@libero.it> Message-ID: <47605109.2000203@colorstudy.com> Manlio Perillo wrote: >> And of course it's not much code to remove them. There's some things >> that technically are okay with WSGI, but in practice are bad (like >> leaving out QUERY_STRING, which can cause very weird bugs due to how >> the cgi module was written). >> > > QUERY_STRING can be a problem for me, since I don't add to the > environment dictionary empty variables, with the exception of > SCRIPT_NAME and PATH_INFO. > > Well, not a real problem, since it easy to make sure that it is always > present in the WSGI environment dictionary. > > By the way: the CGI spec > (http://cgi-spec.golux.com/draft-coar-cgi-v11-03-clean.html) > requires QUERY_STRING to be *alway* present, but the WSGI spec allows it > to be absent. Yes, there's a couple keys like that, I'm afraid; SCRIPT_NAME and PATH_INFO too, I think. > P.S.: I'm reading the cgi module and it only uses argv[1]: > if sys.argv[1:]: > qs = sys.argv[1] > > I'm not sure to understand. > > Moreover in mod_wsgi for nginx, I call PySys_SetArgv, so the WSGI > application sees the command line options passed to nginx. I don't know *why* cgi does this, but as you can imagine it's weird and unhelpful when it does. Having QUERY_STRING set avoids this problem. Most framework do (or should) make sure it's set before calling the cgi module, but someone who uses it directly could get a weird error as a result. I don't think there's anything wrong with exposing sys.argv; what the cgi module does is just weird. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From manlio_perillo at libero.it Wed Dec 12 22:29:26 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 12 Dec 2007 22:29:26 +0100 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <47605109.2000203@colorstudy.com> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> <476042F0.3010608@libero.it> <4760474A.2030007@colorstudy.com> <47605027.6040503@libero.it> <47605109.2000203@colorstudy.com> Message-ID: <476052B6.2060400@libero.it> Ian Bicking ha scritto: > Manlio Perillo wrote: > [...] >> P.S.: I'm reading the cgi module and it only uses argv[1]: >> if sys.argv[1:]: >> qs = sys.argv[1] >> >> I'm not sure to understand. >> >> Moreover in mod_wsgi for nginx, I call PySys_SetArgv, so the WSGI >> application sees the command line options passed to nginx. > > I don't know *why* cgi does this, but as you can imagine it's weird and > unhelpful when it does. CGI says: Scripts SHOULD check to see if the QUERY_STRING value contains an unencoded "=" character, and SHOULD NOT use the command line arguments if it does. So a CGI script *can* use the command line arguments even if QUERY_STRING is in environment (this seems to be really a mess). What I don't understand is why the cgi module just use argv[1] instead of all the command line arguments. > [...] Manlio Perillo From pje at telecommunity.com Wed Dec 12 22:45:45 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 12 Dec 2007 16:45:45 -0500 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <4760474A.2030007@colorstudy.com> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> <476042F0.3010608@libero.it> <4760474A.2030007@colorstudy.com> Message-ID: <20071212214540.CAC4B3A40B1@sparrow.telecommunity.com> At 02:40 PM 12/12/2007 -0600, Ian Bicking wrote: >Having the same header show up in >multiple places seems dangerous (where the two values may not match, >especially if the request has one of those keys rewritten). I agree -- this probably should be added to the WSGI spec, because otherwise middleware has to take both versions into account. From manlio_perillo at libero.it Wed Dec 12 22:48:03 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 12 Dec 2007 22:48:03 +0100 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <20071212214540.CAC4B3A40B1@sparrow.telecommunity.com> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> <476042F0.3010608@libero.it> <4760474A.2030007@colorstudy.com> <20071212214540.CAC4B3A40B1@sparrow.telecommunity.com> Message-ID: <47605713.20509@libero.it> Phillip J. Eby ha scritto: > At 02:40 PM 12/12/2007 -0600, Ian Bicking wrote: >> Having the same header show up in >> multiple places seems dangerous (where the two values may not match, >> especially if the request has one of those keys rewritten). > > I agree -- this probably should be added to the WSGI spec, because > otherwise middleware has to take both versions into account. > A WSGI application/middleware SHOULD just ignore HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGHT. Manlio Perillo From manlio_perillo at libero.it Wed Dec 12 22:57:47 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 12 Dec 2007 22:57:47 +0100 Subject: [Web-SIG] HTTP_CONTENT_TYPE and HTTP_CONTENT_LENGTH In-Reply-To: <4760365B.1040900@colorstudy.com> References: <476033CA.6080706@libero.it> <4760365B.1040900@colorstudy.com> Message-ID: <4760595B.4080807@libero.it> Ian Bicking ha scritto: > Manlio Perillo wrote: > > [...] > the Content-Type header should be > CONTENT_TYPE, and Content-Length should be CONTENT_LENGTH. This is just a personal curiosity: *why* CGI defines these additional variables instead of using the corresponding HTTP_ variables? There is no explanation in the CGI spec. Thanks Manlio Perillo From ionel.mc at gmail.com Thu Dec 13 12:30:14 2007 From: ionel.mc at gmail.com (Ionel Maries Cristian) Date: Thu, 13 Dec 2007 13:30:14 +0200 Subject: [Web-SIG] wsgi.file_wrapper and range requests Message-ID: wsgi.file_wrapper doesn't support sending a file from a specific offset as per wsgi spec. what are your thoughts on this ? -- http://ionel.zapto.org ionel. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/web-sig/attachments/20071213/b7c8975c/attachment.htm From chris at simplistix.co.uk Thu Dec 13 14:20:03 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 13 Dec 2007 13:20:03 +0000 Subject: [Web-SIG] wsgi.file_wrapper and range requests In-Reply-To: References: Message-ID: <47613183.2090707@simplistix.co.uk> Ionel Maries Cristian wrote: > wsgi.file_wrapper doesn't support sending a file from a specific offset as > per wsgi spec. > what are your thoughts on this ? This was one of the things I was asking about when enquiring about wsgi's support for large files previously. Zope's OFS.Image.File.index_html method does a lot of this clever stuff, I'm still wondering how to get it to work over/through wsgi... I wonder if the repoze guys have bumped into this yet? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From brian at briansmith.org Thu Dec 13 15:18:24 2007 From: brian at briansmith.org (Brian Smith) Date: Thu, 13 Dec 2007 21:18:24 +0700 Subject: [Web-SIG] wsgi.file_wrapper and range requests In-Reply-To: References: Message-ID: <001801c83d93$0d10c9b0$2101a8c0@Junk> Ionel Maries Cristian wrote: > wsgi.file_wrapper doesn't support sending a file > from a specific offset as per wsgi spec. > what are your thoughts on this ? Yes, it does. The spec says: "In other words, transmission should begin at the current position within the "file" at the time that transmission begins, and continue until the end is reached." It doesn't support sending a file *up to* a specific offset, though, which I think is unfortunate. - Brian From manlio_perillo at libero.it Fri Dec 14 17:56:33 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 14 Dec 2007 17:56:33 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <47629AA1.8020305@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> Message-ID: <4762B5C1.8050409@libero.it> Chris Withers ha scritto: > Manlio Perillo wrote: >> >> Note however that this is an optional feature, so a wsgi gateway is >> not required to implement it. > > What happens if the app is expecting the gateway to support it and the > gateway does not? > You can still use something like: class FileWrapper(object): def __init__(self, fp, blksize=8192): self.fp = fp self.blksize=blksize def __getitem__(self, key): data = self.fp.read(self.blksize) if data: return data raise IndexError def close(self): self.fp.close() The wsgi gateway will iterate over the FileWrapper instance. >> >>> Also, does wsgi offer anything to help with http range requests and >>> the like? >>> >> >> No. >> For dynamic resources this is not easy to handle, and for static >> resources you can delegate the job to the web server. > > Yeah, but I'm thinking of thinks like big pdfs stored in an object > database... > There are 3 solutions, as far as I know: 1) reading the whole data and let the server return to the client only the requested range. This is possible if you use a WSGI implementation embedded in a web server like Apache or Nginx. For mod_wsgi for Nginx I have implemented this in: http://hg.mperillo.ath.cx/nginx/mod_wsgi/rev/212 However the current implementation of nginx range filter only supports single bufferred response. That is, you have to read the data at once, or use wsgi.file_wrapper (not yet implemented, and it will only work for files), so this is not a good solution for big data stored in a object database 2) handle the range request in the WSGI application. Its not hard as long as you do not implement multiple ranges support. If your object database supports seeks, this should be the most efficient solution. 3) the same as 1), but with range support implemented in a WSGI gateway > cheers, > > Chris > Manlio Perillo From ianb at colorstudy.com Fri Dec 14 18:18:36 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 14 Dec 2007 11:18:36 -0600 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <475D2C39.2000409@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> Message-ID: <4762BAEC.6050608@colorstudy.com> Chris Withers wrote: > Hi All, > > What's the best way to serve large files (say detailed images or pdfs) > from a wsgi app? > > Is there special support for this? > > Also, does wsgi offer anything to help with http range requests and the > like? It's not part of any WSGI standard, but in WebOb if there is a range request, a 200 response (i.e., no range has been applied) and app_iter has an app_iter_range method, it will use that to send a range response. This response rewriting isn't on by default in WebOb, but I've usually turned it on in my own code. An example is at the bottom of this tutorial: http://pythonpaste.org/webob/file-example.html -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From ionel.mc at gmail.com Sun Dec 16 07:10:31 2007 From: ionel.mc at gmail.com (Ionel Maries Cristian) Date: Sun, 16 Dec 2007 08:10:31 +0200 Subject: [Web-SIG] app iterable containing non-strings Message-ID: I was reading the wsgi spec and i was wondering how should middleware treat the iterable that happens to contain values that aren't strings (and I'm not talking about unicodes) - the spec isn't explicit on this. If middleware would just pass on values that aren't string instances - this could be a mechanism for server extensions. (besides the environ) -- http://code.google.com/p/cogen/ ionel. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/web-sig/attachments/20071216/2e57cbb2/attachment.htm From adam at atlas.st Sun Dec 16 07:22:03 2007 From: adam at atlas.st (Adam Atlas) Date: Sun, 16 Dec 2007 01:22:03 -0500 Subject: [Web-SIG] app iterable containing non-strings In-Reply-To: References: Message-ID: <67E8B88E-4983-42FC-AB53-582CEC899EEA@atlas.st> The spec says "Regardless of how it is accomplished, the application object must always return an iterable yielding zero or more strings." The "must" means that anything else should be considered an error (I think). If your middleware does anything to the iterated strings, then you'll have to catch non-strings and raise the error yourself, but otherwise, it's not your middleware's business, and I'd say it should be left to the gateway (or any other lower point in the WSGI stack that happens to notice it). On 16 Dec 2007, at 01:10, Ionel Maries Cristian wrote: > I was reading the wsgi spec and i was wondering how should > middleware treat the iterable that happens to contain values > that aren't strings (and I'm not talking about unicodes) - the spec > isn't explicit on this. > If middleware would just pass on values that aren't string > instances - this could be a mechanism for server extensions. > (besides the environ) > > -- > http://code.google.com/p/cogen/ > ionel. > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/adam%40atlas.st From mdipierro at cs.depaul.edu Sun Dec 16 09:12:08 2007 From: mdipierro at cs.depaul.edu (Massimo Di Pierro) Date: Sun, 16 Dec 2007 02:12:08 -0600 Subject: [Web-SIG] cwd and mod_wsgi Message-ID: <664AE6A4-D69C-490E-9125-3CA2B527F302@cs.depaul.edu> Does anybody know how to change the current working directory in apache with mod_wsgi? Thanks Massimo From graham.dumpleton at gmail.com Sun Dec 16 10:21:17 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sun, 16 Dec 2007 20:21:17 +1100 Subject: [Web-SIG] cwd and mod_wsgi In-Reply-To: <664AE6A4-D69C-490E-9125-3CA2B527F302@cs.depaul.edu> References: <664AE6A4-D69C-490E-9125-3CA2B527F302@cs.depaul.edu> Message-ID: <88e286470712160121u5cb4f3c6idf2c192abb3136bc@mail.gmail.com> On 16/12/2007, Massimo Di Pierro wrote: > Does anybody know how to change the current working directory in > apache with mod_wsgi? Why do you want to be able to change the working directory? Generally it is not a good idea to be writing a WSGI application which depends on being executed within a specific current working directory. This is because there is no guarantee across different web hosting environments that one can control the application working directory and if you try to change it yourself from the application code then you may run into issues with multithreading and different parts of an application wanting it to be set to different values at the same time. Thus, as explained in the Apache mod_wsgi documentation, specifically section "Application Working Directory" of: http://code.google.com/p/modwsgi/wiki/ApplicationIssues you should aim to always use absolute paths to access the file system. To make an application somewhat moveable, you can calculate the absolute path based relative to the __file__ attribute of a Python module. That said, if running in daemon mode of Apache mod_wsgi, you can control the home directory of the application using the 'home' option to WSGIDaemonProcess. In Apache mod_wsgi 2.0, daemon processes will actually by default change to the home directory of the user the process is running as if that directory isn't overridden through that option. For details of these options see WSGIDaemonProcess directive documentation in: http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives In embedded mode of Apache mod_wsgi there is no way to specify through the configuration what the application working directory should be, as the processes are shared by other Apache modules and because it is a shared environment, it would be bad practice to be trying to change the working directory as something else could easily change it and your application would fail if it was dependent on the working directory remaining the same. BTW, for Apache mod_wsgi questions are better directed to the official mod_wsgi list at: http://groups.google.com/group/modwsgi The reference to the list and all the documentation can be found at: http://www.modwsgi.org Graham From MDiPierro at cti.depaul.edu Sun Dec 16 12:57:05 2007 From: MDiPierro at cti.depaul.edu (DiPierro, Massimo) Date: Sun, 16 Dec 2007 05:57:05 -0600 Subject: [Web-SIG] cwd and mod_wsgi In-Reply-To: <88e286470712160121u5cb4f3c6idf2c192abb3136bc@mail.gmail.com> References: <664AE6A4-D69C-490E-9125-3CA2B527F302@cs.depaul.edu>, <88e286470712160121u5cb4f3c6idf2c192abb3136bc@mail.gmail.com> Message-ID: Thanks, this helps. ________________________________________ From: Graham Dumpleton [graham.dumpleton at gmail.com] Sent: Sunday, December 16, 2007 3:21 AM To: DiPierro, Massimo Cc: web-sig at python.org; modwsgi at googlegroups.com Subject: Re: [Web-SIG] cwd and mod_wsgi On 16/12/2007, Massimo Di Pierro wrote: > Does anybody know how to change the current working directory in > apache with mod_wsgi? Why do you want to be able to change the working directory? Generally it is not a good idea to be writing a WSGI application which depends on being executed within a specific current working directory. This is because there is no guarantee across different web hosting environments that one can control the application working directory and if you try to change it yourself from the application code then you may run into issues with multithreading and different parts of an application wanting it to be set to different values at the same time. Thus, as explained in the Apache mod_wsgi documentation, specifically section "Application Working Directory" of: http://code.google.com/p/modwsgi/wiki/ApplicationIssues you should aim to always use absolute paths to access the file system. To make an application somewhat moveable, you can calculate the absolute path based relative to the __file__ attribute of a Python module. That said, if running in daemon mode of Apache mod_wsgi, you can control the home directory of the application using the 'home' option to WSGIDaemonProcess. In Apache mod_wsgi 2.0, daemon processes will actually by default change to the home directory of the user the process is running as if that directory isn't overridden through that option. For details of these options see WSGIDaemonProcess directive documentation in: http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives In embedded mode of Apache mod_wsgi there is no way to specify through the configuration what the application working directory should be, as the processes are shared by other Apache modules and because it is a shared environment, it would be bad practice to be trying to change the working directory as something else could easily change it and your application would fail if it was dependent on the working directory remaining the same. BTW, for Apache mod_wsgi questions are better directed to the official mod_wsgi list at: http://groups.google.com/group/modwsgi The reference to the list and all the documentation can be found at: http://www.modwsgi.org Graham From chris at simplistix.co.uk Mon Dec 17 17:44:43 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 17 Dec 2007 16:44:43 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4762B5C1.8050409@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> Message-ID: <4766A77B.9050704@simplistix.co.uk> Manlio Perillo wrote: > 2) handle the range request in the WSGI application. > Its not hard as long as you do not implement multiple ranges support. > > If your object database supports seeks, this should be the most > efficient solution. This is probably what's wanted. So, if a wsgi app does its own range handling, the wsgi server won't interfere? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From manlio_perillo at libero.it Mon Dec 17 17:56:33 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 17 Dec 2007 17:56:33 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766A77B.9050704@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> Message-ID: <4766AA41.9060600@libero.it> Chris Withers ha scritto: > Manlio Perillo wrote: >> 2) handle the range request in the WSGI application. >> Its not hard as long as you do not implement multiple ranges support. >> >> If your object database supports seeks, this should be the most >> efficient solution. > > This is probably what's wanted. So, if a wsgi app does its own range > handling, the wsgi server won't interfere? > Yes, it should not interfere (well, with mod_wsgi for nginx it *will* interfere if you enable the wsgi_allow_ranges directive). I think that, for your problem, the best solution is to cache the content of the database on a file, and let the web server to serve that file (using wsgi.file_wrapper). That is, you should create a dedicate WSGI application that will map request URI to your database objects. Assuming that in you database you store the file data and some metadata (such as the last modification time), this application will: 1) If the cached file does not exists, dump the content of the database data on the file 2) If the cached file exists, check if last modification time match; if not, update the cache file With this solution, you get the better performance and flexibility (IMHO), since the web server can handle the static file at its best. > cheers, > > Chris > Manlio Perillo From fumanchu at aminus.org Mon Dec 17 18:40:16 2007 From: fumanchu at aminus.org (Robert Brewer) Date: Mon, 17 Dec 2007 09:40:16 -0800 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766A77B.9050704@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it><47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> Message-ID: Chris Withers wrote: > Manlio Perillo wrote: > > 2) handle the range request in the WSGI application. > > Its not hard as long as you do not implement multiple ranges > support. > > > > If your object database supports seeks, this should be the most > > efficient solution. > > This is probably what's wanted. So, if a wsgi app does its own range > handling, the wsgi server won't interfere? Apache will interfere, and try to re-apply the range to whatever you emit. The only solution we've found so far is to tell the app to ignore any 'Range' request header when running behind Apache, and just let Apache have its way. See http://www.cherrypy.org/changeset/1319 Robert Brewer fumanchu at aminus.org From chris at simplistix.co.uk Mon Dec 17 18:41:35 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 17 Dec 2007 17:41:35 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it><47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> Message-ID: <4766B4CF.9040105@simplistix.co.uk> Robert Brewer wrote: > Apache will interfere, and try to re-apply the range to whatever you > emit. The only solution we've found so far is to tell the app to ignore > any 'Range' request header when running behind Apache, and just let > Apache have its way. See http://www.cherrypy.org/changeset/1319 I've never had any problems with Apache proxying to Zope for this stuff... I wonder why the proxy setup seems to make it safe? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From fumanchu at aminus.org Mon Dec 17 18:46:16 2007 From: fumanchu at aminus.org (Robert Brewer) Date: Mon, 17 Dec 2007 09:46:16 -0800 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766B4CF.9040105@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it><47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766B4CF.9040105@simplistix.co.uk> Message-ID: Chris Withers wrote: > Robert Brewer wrote: > > Apache will interfere, and try to re-apply the range to whatever you > > emit. The only solution we've found so far is to tell the app to > ignore > > any 'Range' request header when running behind Apache, and just let > > Apache have its way. See http://www.cherrypy.org/changeset/1319 > > I've never had any problems with Apache proxying to Zope for this > stuff... > > I wonder why the proxy setup seems to make it safe? Because proxying bypasses a lot of the Apache internals. The re-application of Range headers I described was in a mod_python setup. I would guess mod_wsgi would exhibit the same problem. Robert Brewer fumanchu at aminus.org From ianb at colorstudy.com Mon Dec 17 18:57:31 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Dec 2007 11:57:31 -0600 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766AA41.9060600@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> Message-ID: <4766B88B.7070300@colorstudy.com> Manlio Perillo wrote: > Chris Withers ha scritto: >> Manlio Perillo wrote: >>> 2) handle the range request in the WSGI application. >>> Its not hard as long as you do not implement multiple ranges support. >>> >>> If your object database supports seeks, this should be the most >>> efficient solution. >> This is probably what's wanted. So, if a wsgi app does its own range >> handling, the wsgi server won't interfere? >> > > Yes, it should not interfere (well, with mod_wsgi for nginx it *will* > interfere if you enable the wsgi_allow_ranges directive). You should be able to detect whether interference is allowed based on the response status, shouldn't you? That is, if there is a range request and the application replies 200 OK, you can change that and apply the ranges. But if the application replies with 206 Partial Content then the range has already been applied and the server shouldn't do anything to it. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From manlio_perillo at libero.it Mon Dec 17 18:59:43 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 17 Dec 2007 18:59:43 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it><47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> Message-ID: <4766B90F.2080808@libero.it> Robert Brewer ha scritto: > Chris Withers wrote: >> Manlio Perillo wrote: >>> 2) handle the range request in the WSGI application. >>> Its not hard as long as you do not implement multiple ranges >> support. >>> If your object database supports seeks, this should be the most >>> efficient solution. >> This is probably what's wanted. So, if a wsgi app does its own range >> handling, the wsgi server won't interfere? > > Apache will interfere, and try to re-apply the range to whatever you > emit. The only solution we've found so far is to tell the app to ignore > any 'Range' request header when running behind Apache, and just let > Apache have its way. See http://www.cherrypy.org/changeset/1319 > Strange, looking at the byterange filter source code (from trunk) it seems that Apache first checks if content is already a single or multiple range (searching for Content-Range and Content-Type = "multipart/byteranges in in headers_out) > > Robert Brewer > fumanchu at aminus.org > Manlio Perillo From manlio_perillo at libero.it Mon Dec 17 19:06:20 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 17 Dec 2007 19:06:20 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766B88B.7070300@colorstudy.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> Message-ID: <4766BA9C.5090203@libero.it> Ian Bicking ha scritto: > Manlio Perillo wrote: >> Chris Withers ha scritto: >>> Manlio Perillo wrote: >>>> 2) handle the range request in the WSGI application. >>>> Its not hard as long as you do not implement multiple ranges >>>> support. >>>> >>>> If your object database supports seeks, this should be the most >>>> efficient solution. >>> This is probably what's wanted. So, if a wsgi app does its own range >>> handling, the wsgi server won't interfere? >>> >> >> Yes, it should not interfere (well, with mod_wsgi for nginx it *will* >> interfere if you enable the wsgi_allow_ranges directive). > > You should be able to detect whether interference is allowed based on > the response status, shouldn't you? Right. > That is, if there is a range > request and the application replies 200 OK, you can change that and > apply the ranges. But if the application replies with 206 Partial > Content then the range has already been applied and the server shouldn't > do anything to it. > Thanks, I'll think about it. I have just added a wsgi_allow_ranges directive, without further processing, since I'm assuming that an user can anticipate if a WSGI application is able to process partial content and thus he can just enable or disable the directive. Manlio Perillo From pje at telecommunity.com Mon Dec 17 19:16:41 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Dec 2007 13:16:41 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766A77B.9050704@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> Message-ID: <20071217181635.ABECB3A40A4@sparrow.telecommunity.com> At 04:44 PM 12/17/2007 +0000, Chris Withers wrote: >Manlio Perillo wrote: > > 2) handle the range request in the WSGI application. > > Its not hard as long as you do not implement multiple ranges support. > > > > If your object database supports seeks, this should be the most > > efficient solution. > >This is probably what's wanted. So, if a wsgi app does its own range >handling, the wsgi server won't interfere? A WSGI server that didn't recognize the response already handled the range would be defective, yes. From ianb at colorstudy.com Mon Dec 17 19:21:39 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Dec 2007 12:21:39 -0600 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766BA9C.5090203@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> Message-ID: <4766BE33.5050707@colorstudy.com> Manlio Perillo wrote: >> That is, if there is a range request and the application replies 200 >> OK, you can change that and apply the ranges. But if the application >> replies with 206 Partial Content then the range has already been >> applied and the server shouldn't do anything to it. >> > > Thanks, I'll think about it. > > I have just added a wsgi_allow_ranges directive, without further > processing, since I'm assuming that an user can anticipate if a WSGI > application is able to process partial content and thus he can just > enable or disable the directive. A case when you can't do this is if your application proxies requests to another process, with things like the range headers in-tact. It could get back a ranged response, but the application wouldn't be generating that response. The user shouldn't have to anticipate what an application can or should do, beyond what the spec says. Especially stuff like this will just cause weird bugs that no one will notice until deployment, and even then the bugs will be hard to find (in part because range requests are something only a small set of clients send, and that doesn't generally include clients that programmers use to run tests). -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From manlio_perillo at libero.it Mon Dec 17 19:23:31 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 17 Dec 2007 19:23:31 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766BA9C.5090203@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> Message-ID: <4766BEA3.2010508@libero.it> Manlio Perillo ha scritto: > [...] > >> You should be able to detect whether interference is allowed based on >> the response status, shouldn't you? > > Right. > >> That is, if there is a range >> request and the application replies 200 OK, you can change that and >> apply the ranges. But if the application replies with 206 Partial >> Content then the range has already been applied and the server shouldn't >> do anything to it. >> > > Thanks, I'll think about it. > No need to think about it ;-). Nginx already checks if status code is 200. In my early tests I just added a Content-Range header, without returning a 206 status code. Manlio Perillo From manlio_perillo at libero.it Mon Dec 17 19:33:58 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 17 Dec 2007 19:33:58 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766BE33.5050707@colorstudy.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> Message-ID: <4766C116.1050309@libero.it> Ian Bicking ha scritto: > [...] > > The user shouldn't have to anticipate what an application can or should > do, beyond what the spec says. I disagree. The intent of mod_wsgi for nginx, among other things, is to have an "integrated" deployment platform for running WSGI applications; so the nginx server configuration *is* the WSGI application configuration. With mod_wsgi for nginx, you don't run a WSGI application inside the nginx server, you just use[1] the nginx server for handling the HTTP protocol. It is a subtle but significative difference, IMHO. I'm using nginx because writing an HTTP server is hard, very hard, and it is much simpler to embed the Python interpreter inside an exiting HTTP server. [1] You may also say "abuse", since the WSGI application will usually block the nginx main cycle. > [...] Manlio Perillo From pje at telecommunity.com Mon Dec 17 20:52:14 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Dec 2007 14:52:14 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766C116.1050309@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> Message-ID: <20071217195209.67DE83A40A4@sparrow.telecommunity.com> At 07:33 PM 12/17/2007 +0100, Manlio Perillo wrote: >Ian Bicking ha scritto: > > [...] > > > > The user shouldn't have to anticipate what an application can or should > > do, beyond what the spec says. > >I disagree. > >The intent of mod_wsgi for nginx, among other things, is to have an >"integrated" deployment platform for running WSGI applications; >so the nginx server configuration *is* the WSGI application configuration. > >With mod_wsgi for nginx, you don't run a WSGI application inside the >nginx server, you just use[1] the nginx server for handling the HTTP >protocol. > >It is a subtle but significative difference, IMHO. And it's also irrelevant: WSGI applications are composable, which means that not only does the application deployer not necessarily have any idea what the application does, the *author* might not know every detail, either (due to using other libraries, components, and middleware). This is precisely why WSGI doesn't really have any "configuration" defined, because the whole idea is that it should be as "plug-and-play" as possible. Server-level configuration options are a liability to be avoided, a sometimes-necessary evil. They aren't a feature. From manlio_perillo at libero.it Mon Dec 17 21:06:45 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 17 Dec 2007 21:06:45 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071217195209.67DE83A40A4@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> Message-ID: <4766D6D5.6010403@libero.it> Phillip J. Eby ha scritto: > At 07:33 PM 12/17/2007 +0100, Manlio Perillo wrote: > [...] > > And it's also irrelevant: WSGI applications are composable, which means > that not only does the application deployer not necessarily have any > idea what the application does, the *author* might not know every > detail, either (due to using other libraries, components, and middleware). > Not sure. The configuration is part of the application. The deployer can ignore what the application does, but can not use a generic server configuration. As an example, for a WSGI gateway implementation embedded in a web server and with support for multiple sub interpreter, the deployer can't just configure the gateway so that an application is executed in a sub interpreter without first knowing if the application has problems with sub interpreters. > This is precisely why WSGI doesn't really have any "configuration" > defined, because the whole idea is that it should be as "plug-and-play" > as possible. Server-level configuration options are a liability to be > avoided, a sometimes-necessary evil. They aren't a feature. > Disagree. First of all, I like to have configuration in one place. The same for logging, since, as an example, a naive use of logging prevents logs rotation. Manlio Perillo From pje at telecommunity.com Mon Dec 17 22:07:06 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 17 Dec 2007 16:07:06 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4766D6D5.6010403@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> Message-ID: <20071217210700.45C543A40A4@sparrow.telecommunity.com> At 09:06 PM 12/17/2007 +0100, Manlio Perillo wrote: >Phillip J. Eby ha scritto: >>This is precisely why WSGI doesn't really have any "configuration" >>defined, because the whole idea is that it should be as >>"plug-and-play" as possible. Server-level configuration options >>are a liability to be avoided, a sometimes-necessary evil. They >>aren't a feature. > >Disagree. Note that your disagreement doesn't retroactively change WSGI's design goals or rationale, which are what I've been explaining. Your desire for configuration in one place also doesn't negate others' desires to simply drop applications into a server with only the most minimal server configuration possible. From manlio_perillo at libero.it Tue Dec 18 21:50:01 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Tue, 18 Dec 2007 21:50:01 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071217210700.45C543A40A4@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> Message-ID: <47683279.9010008@libero.it> Phillip J. Eby ha scritto: > At 09:06 PM 12/17/2007 +0100, Manlio Perillo wrote: >> Phillip J. Eby ha scritto: >>> This is precisely why WSGI doesn't really have any "configuration" >>> defined, because the whole idea is that it should be as >>> "plug-and-play" as possible. Server-level configuration options are >>> a liability to be avoided, a sometimes-necessary evil. They aren't a >>> feature. >> >> Disagree. > > Note that your disagreement doesn't retroactively change WSGI's design > goals or rationale, I disagree with your "Server-level configuration options are a liability to be avoided, a sometimes-necessary evil" phrase. But now I'm not sure of what do you mean by "configuration". > which are what I've been explaining. Your desire > for configuration in one place also doesn't negate others' desires to > simply drop applications into a server with only the most minimal server > configuration possible. > But some applications/middlewares *will* need some configuration; maybe they have safe defaults for configuration parameters, but my problem is: how should these configuration parameters handled? Let's make an example. I want to add a middleware to my application that will send an email when an error occurs. Of course I need to configure some parameters: how do you do this in WSGI? Since the WSGI spec says nothing about configuration, this is up to the user. In mod_wsgi for nginx I have added a `wsgi_middleware` directive, so that it is possible to stack middlewares from nginx configuration (and I think this can be very convenient). Now, the only method for passing configuration parameters to a middleware is to set them in the WSGI environment dictionary, using the `wsgi_var` directive. So, *all* the configuration (nginx server, application, middlewares) can be placed (and, IMHO, should be placed) in the nginx configuration file. I hope that now it is clear what I mean (unfortunalely I'm not very good at explaining things). Manlio Perillo From pje at telecommunity.com Tue Dec 18 22:00:06 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Dec 2007 16:00:06 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <47683279.9010008@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> Message-ID: <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> At 09:50 PM 12/18/2007 +0100, Manlio Perillo wrote: >Phillip J. Eby ha scritto: >>At 09:06 PM 12/17/2007 +0100, Manlio Perillo wrote: >>>Phillip J. Eby ha scritto: >>>>This is precisely why WSGI doesn't really have any >>>>"configuration" defined, because the whole idea is that it should >>>>be as "plug-and-play" as possible. Server-level configuration >>>>options are a liability to be avoided, a sometimes-necessary >>>>evil. They aren't a feature. >>> >>>Disagree. >>Note that your disagreement doesn't retroactively change WSGI's >>design goals or rationale, > > >I disagree with your "Server-level configuration options are a >liability to be avoided, a sometimes-necessary evil" phrase. As I said, that doesn't retroactively alter the WSGI rationale. >But now I'm not sure of what do you mean by "configuration". I mean configuring the *server* to correctly run a WSGI application, not the application's configuration. For example, a server option to control whether Range requests are supported would be an example of "server configuration" - i.e., something to be avoided since it forces a user to make a choice about something he or she may know nothing about. >>which are what I've been explaining. Your desire for configuration >>in one place also doesn't negate others' desires to simply drop >>applications into a server with only the most minimal server >>configuration possible. > >But some applications/middlewares *will* need some configuration; >maybe they have safe defaults for configuration parameters, but my >problem is: how should these configuration parameters handled? That's not server configuration; that's application configuration. >So, *all* the configuration (nginx server, application, middlewares) >can be placed (and, IMHO, should be placed) in the nginx configuration file. That's application configuration, and isn't related to what I'm talking about. Ideally, a WSGI server will "just run" any application object it is given, without the *server* needing to be configured (except for specifying the object to run). Being able to specify how the application object(s) are created and configured is indeed a nice feature to have in a WSGI server, and that's not what I'm talking about at all. From manlio_perillo at libero.it Tue Dec 18 22:10:05 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Tue, 18 Dec 2007 22:10:05 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> Message-ID: <4768372D.2080408@libero.it> Phillip J. Eby ha scritto: > At 09:50 PM 12/18/2007 +0100, Manlio Perillo wrote: >> Phillip J. Eby ha scritto: >>> At 09:06 PM 12/17/2007 +0100, Manlio Perillo wrote: >>>> Phillip J. Eby ha scritto: >>>>> This is precisely why WSGI doesn't really have any "configuration" >>>>> defined, because the whole idea is that it should be as >>>>> "plug-and-play" as possible. Server-level configuration options >>>>> are a liability to be avoided, a sometimes-necessary evil. They >>>>> aren't a feature. >>>> >>>> Disagree. >>> Note that your disagreement doesn't retroactively change WSGI's >>> design goals or rationale, >> >> >> I disagree with your "Server-level configuration options are a >> liability to be avoided, a sometimes-necessary evil" phrase. > > As I said, that doesn't retroactively alter the WSGI rationale. > > >> But now I'm not sure of what do you mean by "configuration". > > I mean configuring the *server* to correctly run a WSGI application, not > the application's configuration. Ok, I was talking about *application* configuration, sorry. > For example, a server option to > control whether Range requests are supported would be an example of > "server configuration" - i.e., something to be avoided since it forces a > user to make a choice about something he or she may know nothing about. > Ok. Here I would just say that when someone install something on its system, it should at least know what he is doing. And about an option for range control, it is true that it will "force" an user to make a choice, but this is not a good reason for not adding the option at all. This is about server flexibility: nginx has a lot of options, many of them have safe default so if the user does not know what to do, he can just use default values. > [...] Manlio Perillo From pje at telecommunity.com Tue Dec 18 23:34:57 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Dec 2007 17:34:57 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4768372D.2080408@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> <4768372D.2080408@libero.it> Message-ID: <20071218223525.B43873A40D9@sparrow.telecommunity.com> At 10:10 PM 12/18/2007 +0100, Manlio Perillo wrote: >Ok. >Here I would just say that when someone install something on its >system, it should at least know what he is doing. And I repeat: you're welcome to your opinions about what's good or bad, but that has nothing to do with WSGI's design rationale, which is based on *different* goals. It is in fact a design goal of WSGI that someone who does NOT "know what they are doing" should be able to install Python web applications. After all, it is certainly possible for PHP applications -- often in *spite* of PHP's ridiculously global and mutually incompatible configuration options >And about an option for range control, it is true that it will >"force" an user to make a choice, but this is not a good reason for >not adding the option at all. In the context of WSGI's design goals, it's an excellent reason. If you have different design goals, you may disagree -- but at that point, you're no longer talking about WSGI. >This is about server flexibility: Which, as I've already explained, is an *anti*-feature relative to WSGI's design goals. Simply restating your opinion over and over won't change that. You are free to have different design goals, and even to argue that WSGI's goals are misguided. However, it is pointless to argue that these are *not* WSGI's goals (since I defined them), or to imply that such "flexibility" *supports* WSGI's goals (since it does not). From manlio_perillo at libero.it Tue Dec 18 23:51:08 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Tue, 18 Dec 2007 23:51:08 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071218223525.B43873A40D9@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> <4768372D.2080408@libero.it> <20071218223525.B43873A40D9@sparrow.telecommunity.com> Message-ID: <47684EDC.70501@libero.it> Phillip J. Eby ha scritto: > At 10:10 PM 12/18/2007 +0100, Manlio Perillo wrote: >> Ok. >> Here I would just say that when someone install something on its >> system, it should at least know what he is doing. > > And I repeat: you're welcome to your opinions about what's good or bad, > but that has nothing to do with WSGI's design rationale, which is based > on *different* goals. > The problem is that I don't think that having many server configuration parameters with "safe" defaults value is against your design goals. BTW, the "Rationale and Goals" chapter of the WSGI PEP only talks about WSGI gateway implementators and WSGI framefork and application developers. > [...] Thanks Manlio Perillo From ianb at colorstudy.com Wed Dec 19 00:21:22 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Dec 2007 17:21:22 -0600 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <47683279.9010008@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> Message-ID: <476855F2.3080007@colorstudy.com> Manlio Perillo wrote: > Phillip J. Eby ha scritto: >> At 09:06 PM 12/17/2007 +0100, Manlio Perillo wrote: >>> Phillip J. Eby ha scritto: >>>> This is precisely why WSGI doesn't really have any "configuration" >>>> defined, because the whole idea is that it should be as >>>> "plug-and-play" as possible. Server-level configuration options are >>>> a liability to be avoided, a sometimes-necessary evil. They aren't >>>> a feature. >>> >>> Disagree. >> >> Note that your disagreement doesn't retroactively change WSGI's design >> goals or rationale, > > > I disagree with your "Server-level configuration options are a liability > to be avoided, a sometimes-necessary evil" phrase. > > But now I'm not sure of what do you mean by "configuration". I think there might still be some confusion about configuration. I'll offer my take: There's generally two valid kinds of configuration -- the server's configuration (host, port, ssl, etc), and the application configuration (database connection parameters, maybe some theming options, etc). Obviously server configuration goes in Nginx. If you put both of these in Nginx, I think that's fine. That is, if you allow people to create WSGI applications through Nginx, providing information to the application like database connection information, then that's totally fine. There's no standard WSGI way to do that; the only thing approaching a standard for that is Paste Deploy's interfaces (which I'm sure would be possible to support in Nginx -- they are interfaces, not an implementation: http://pythonpaste.org/deploy/#defining-factories). Another kind of "configuration" is configuring the application implementation with respect to its framework. This is where you might choose your template language, or a default charset/encoding. This is where you might choose whether to let the framework handle range requests. This is not actually configuration in my opinion, but it is frequently called configuration. I think it's simply "programming". It has nothing to do with WSGI, and must be fully encapsulated in a WSGI application. People deploying the application should not encounter any of these options. The range stuff apparently is not really an issue of this, as Nginx is acting reasonably and obeying the restrictions of HTTP, which in turn satisfies WSGI. If Nginx was rewriting partial responses to be... doubly partial or something like that, that would really just be a bug. These kinds of bugs exist and are sometimes excused as being "configuration". But when one configuration setting is valid and another is simply invalid, the bug is having that configuration setting at all. But at least this case doesn't seem to be an issue; hopefully other issues that might be similar will be resolved in Nginx. You should never have to have Nginx-specific notes about deploying a WSGI application (except perhaps if you want to provide helpful notes about how to supply the real configuration in an Nginx configuration file). -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From pje at telecommunity.com Wed Dec 19 00:27:23 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Dec 2007 18:27:23 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <47684EDC.70501@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> <4768372D.2080408@libero.it> <20071218223525.B43873A40D9@sparrow.telecommunity.com> <47684EDC.70501@libero.it> Message-ID: <20071218232714.E015A3A40BF@sparrow.telecommunity.com> At 11:51 PM 12/18/2007 +0100, Manlio Perillo wrote: >Phillip J. Eby ha scritto: >>At 10:10 PM 12/18/2007 +0100, Manlio Perillo wrote: >>>Ok. >>>Here I would just say that when someone install something on its >>>system, it should at least know what he is doing. >>And I repeat: you're welcome to your opinions about what's good or >>bad, but that has nothing to do with WSGI's design rationale, which >>is based on *different* goals. > >The problem is that I don't think that having many server >configuration parameters with "safe" defaults value is against your >design goals. Sometimes parameters are a necessary evil; that doesn't make them any less evil, just more necessary. :) However, if they aren't necessary, then they're just plain evil. Range support would be a good example of something where an option isn't necessary, since properly-written Range support in the server should be able to tell when the application has already handled the necessary range-ing of the output. Thus, having an option to turn it on or off is clearly a bad idea, as compared to making sure that the Range support is correct in the first place. From brian at briansmith.org Wed Dec 19 02:21:04 2007 From: brian at briansmith.org (Brian Smith) Date: Wed, 19 Dec 2007 08:21:04 +0700 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071218232714.E015A3A40BF@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it><47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it><4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it><4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it><4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it><20071217195209.67DE83A40A4@sparrow.telecommunity.com><4766D6D5.6010403@libero.it><20071217210700.45C543A40A4@sparrow.telecommunity.com><47683279.9010008@libero.it><20071218205958.EBDAD3A40BF@sparrow.telecommunity.com><4768372D.2080408@libero.it><20071218223525.B43873A40D9@sparrow.telecommunity.com><47684EDC.70501@libero.it> <20071218232714.E015A3A40BF@sparrow.telecommunity.com> Message-ID: <006d01c841dd$6df093c0$0701a8c0@Junk> Phillip J. Eby wrote: > Range support would be a good example of something > where an option isn't necessary, since properly-written Range > support in the server should be able to tell when the > application has already handled the necessary range-ing of > the output. Thus, having an option to turn it on or off is > clearly a bad idea, as compared to making sure that the Range > support is correct in the first place. Exactly. If the application already returned 206 Partial Content then you have to assume that they returned, um, partial content. If they returned 200 OK then you have to assume that they returned a full response. At that point, you can examine the cache validators and Vary headers in the response to determine whether or not it is safe to convert the response into a 206. That is all explained in RFC 2616. There is not much room for configuration. - Brian From graham.dumpleton at gmail.com Wed Dec 19 02:36:56 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Wed, 19 Dec 2007 12:36:56 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476855F2.3080007@colorstudy.com> References: <475D2C39.2000409@simplistix.co.uk> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> Message-ID: <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> Where does setting up 'logging' module configuration fall in all of this and who/what should handle it? I ask as I note that in the documentation for Pylons logging it says: """paster, when loading an application via the paster serve, shell or setup-app commands, calls the logging.fileConfig function on that specified ini file if it contains a 'loggers' entry. logging.fileConfig reads the logging configuration from a ConfigParser file.""" So, the paster serve is doing special setup for Pylons WSGI components. Without knowing this magic is required, when you take a Pylons WSGI component to another WSGI server, the logging doesn't work as expected. To get it to work one has to duplicate the special things that paster serve is doing in some way. Graham On 19/12/2007, Ian Bicking wrote: > Manlio Perillo wrote: > > Phillip J. Eby ha scritto: > >> At 09:06 PM 12/17/2007 +0100, Manlio Perillo wrote: > >>> Phillip J. Eby ha scritto: > >>>> This is precisely why WSGI doesn't really have any "configuration" > >>>> defined, because the whole idea is that it should be as > >>>> "plug-and-play" as possible. Server-level configuration options are > >>>> a liability to be avoided, a sometimes-necessary evil. They aren't > >>>> a feature. > >>> > >>> Disagree. > >> > >> Note that your disagreement doesn't retroactively change WSGI's design > >> goals or rationale, > > > > > > I disagree with your "Server-level configuration options are a liability > > to be avoided, a sometimes-necessary evil" phrase. > > > > But now I'm not sure of what do you mean by "configuration". > > I think there might still be some confusion about configuration. I'll > offer my take: > > There's generally two valid kinds of configuration -- the server's > configuration (host, port, ssl, etc), and the application configuration > (database connection parameters, maybe some theming options, etc). > > Obviously server configuration goes in Nginx. > > If you put both of these in Nginx, I think that's fine. That is, if you > allow people to create WSGI applications through Nginx, providing > information to the application like database connection information, > then that's totally fine. There's no standard WSGI way to do that; the > only thing approaching a standard for that is Paste Deploy's interfaces > (which I'm sure would be possible to support in Nginx -- they are > interfaces, not an implementation: > http://pythonpaste.org/deploy/#defining-factories). > > Another kind of "configuration" is configuring the application > implementation with respect to its framework. This is where you might > choose your template language, or a default charset/encoding. This is > where you might choose whether to let the framework handle range > requests. This is not actually configuration in my opinion, but it is > frequently called configuration. I think it's simply "programming". It > has nothing to do with WSGI, and must be fully encapsulated in a WSGI > application. People deploying the application should not encounter any > of these options. > > The range stuff apparently is not really an issue of this, as Nginx is > acting reasonably and obeying the restrictions of HTTP, which in turn > satisfies WSGI. If Nginx was rewriting partial responses to be... > doubly partial or something like that, that would really just be a bug. > These kinds of bugs exist and are sometimes excused as being > "configuration". But when one configuration setting is valid and > another is simply invalid, the bug is having that configuration setting > at all. But at least this case doesn't seem to be an issue; hopefully > other issues that might be similar will be resolved in Nginx. You > should never have to have Nginx-specific notes about deploying a WSGI > application (except perhaps if you want to provide helpful notes about > how to supply the real configuration in an Nginx configuration file). > > -- > Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com > From ianb at colorstudy.com Wed Dec 19 02:45:27 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Dec 2007 19:45:27 -0600 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> Message-ID: <476877B7.3030007@colorstudy.com> Graham Dumpleton wrote: > Where does setting up 'logging' module configuration fall in all of > this and who/what should handle it? I'm not sure. What I did with paster serve was expedience, I'm not sure it's right. I guess paster serve technically also sets up the process, not just the server (which might justify things like --reload for the reloader, which is really process-wide). > I ask as I note that in the documentation for Pylons logging it says: > > """paster, when loading an application via the paster serve, shell or > setup-app commands, calls the logging.fileConfig function on that > specified ini file if it contains a 'loggers' entry. > logging.fileConfig reads the logging configuration from a ConfigParser > file.""" > > So, the paster serve is doing special setup for Pylons WSGI > components. Without knowing this magic is required, when you take a > Pylons WSGI component to another WSGI server, the logging doesn't work > as expected. To get it to work one has to duplicate the special things > that paster serve is doing in some way. It's just using logging.fileConfig, though apparently we backported Python 2.5's logging.config module. Stuff ends up in paster serve usually because I don't know where else to put them. Of course applications could take options to setup the configuration on their own (e.g., paste.translogger sets up logging, since without logging it's useless). This lets applications be a little more lazy, which is kind of what the logging module encourages (but only kind of). -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From graham.dumpleton at gmail.com Wed Dec 19 03:41:00 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Wed, 19 Dec 2007 13:41:00 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476877B7.3030007@colorstudy.com> References: <475D2C39.2000409@simplistix.co.uk> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476877B7.3030007@colorstudy.com> Message-ID: <88e286470712181841n2c5d02f3g5d984c3281a573a8@mail.gmail.com> On 19/12/2007, Ian Bicking wrote: > Graham Dumpleton wrote: > > Where does setting up 'logging' module configuration fall in all of > > this and who/what should handle it? > > I'm not sure. What I did with paster serve was expedience, I'm not sure > it's right. Problem is it has set a precedence of sorts. And in some respects this is the problem with trying to do certain types of configuration in the WSGI server as opposed to the application layer. In other words, once one means of hosting a WSGI application does something like this, with the result that the same WSGI application component then will not work as a simple plug and play component in another WSGI server, there is pressure for the other WSGI servers to provide a similar hook. Thus we end up with a defacto standard of sorts which isn't part of any specification and which may not have been the right thing to do in the first place. I don't know what the answer is. I guess as long as the such special set up requirements are well documented in a section dealing with porting a WSGI component to another WSGI server it is okay, but a few things I have run across aren't up front with such details and so one always has to do some digging as to how to get the WSGI component working in a different environment. :-( Graham > I guess paster serve technically also sets up the process, > not just the server (which might justify things like --reload for the > reloader, which is really process-wide). > > > I ask as I note that in the documentation for Pylons logging it says: > > > > """paster, when loading an application via the paster serve, shell or > > setup-app commands, calls the logging.fileConfig function on that > > specified ini file if it contains a 'loggers' entry. > > logging.fileConfig reads the logging configuration from a ConfigParser > > file.""" > > > > So, the paster serve is doing special setup for Pylons WSGI > > components. Without knowing this magic is required, when you take a > > Pylons WSGI component to another WSGI server, the logging doesn't work > > as expected. To get it to work one has to duplicate the special things > > that paster serve is doing in some way. > > It's just using logging.fileConfig, though apparently we backported > Python 2.5's logging.config module. > > Stuff ends up in paster serve usually because I don't know where else to > put them. Of course applications could take options to setup the > configuration on their own (e.g., paste.translogger sets up logging, > since without logging it's useless). This lets applications be a little > more lazy, which is kind of what the logging module encourages (but only > kind of). > > -- > Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org > From manlio_perillo at libero.it Wed Dec 19 12:12:54 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 19 Dec 2007 12:12:54 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071218232714.E015A3A40BF@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> <4768372D.2080408@libero.it> <20071218223525.B43873A40D9@sparrow.telecommunity.com> <47684EDC.70501@libero.it> <20071218232714.E015A3A40BF@sparrow.telecommunity.com> Message-ID: <4768FCB6.4000806@libero.it> Phillip J. Eby ha scritto: > At 11:51 PM 12/18/2007 +0100, Manlio Perillo wrote: >> Phillip J. Eby ha scritto: >>> At 10:10 PM 12/18/2007 +0100, Manlio Perillo wrote: >>>> Ok. >>>> Here I would just say that when someone install something on its >>>> system, it should at least know what he is doing. >>> And I repeat: you're welcome to your opinions about what's good or >>> bad, but that has nothing to do with WSGI's design rationale, which >>> is based on *different* goals. >> >> The problem is that I don't think that having many server >> configuration parameters with "safe" defaults value is against your >> design goals. > > Sometimes parameters are a necessary evil; that doesn't make them any > less evil, just more necessary. :) > Ok, we diverge here ;-) If a program does not have configuration parameters then: 1) the program is probabily very simple 2) the programmer thinks he gets the best values for program configuration, so that the user does not needs to interfere Just a personal opinion, of course. > However, if they aren't necessary, then they're just plain evil. Range > support would be a good example of something where an option isn't > necessary, Ah, so the whole discussion originates from this misunderstanding :). > since properly-written Range support in the server should be > able to tell when the application has already handled the necessary > range-ing of the output. Thus, having an option to turn it on or off is > clearly a bad idea, as compared to making sure that the Range support is > correct in the first place. > As I have written in a reply, I wrongly reported that nginx interfere with range support implemented in the WSGI application. This is not true: what `wsgi_allow_ranges` means is: - by default nginx does not try to add support to partial requests, so this must be implemented by the WSGI application - if you enable this directive, then nginx will add (basic) partial requests support for you So this `wsgi_allow_ranges` is not strictly a "server" configuration; it can be view (somehow) as an application configuration, since nginx filters are similar to WSGI middlewares (and the WSGI application is embedded in nginx, thus not only the WSGI application can be viewed as an "extension" to the nginx server, but the WSGI server can be viewed as an "estension" of the WSGI application). Manlio Perillo From chris at simplistix.co.uk Wed Dec 19 12:51:57 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 19 Dec 2007 11:51:57 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> Message-ID: <476905DD.5060100@simplistix.co.uk> Graham Dumpleton wrote: > Where does setting up 'logging' module configuration fall in all of > this and who/what should handle it? Indeed, this is definitely something I've wondered myself... > """paster, when loading an application via the paster serve, shell or > setup-app commands, calls the logging.fileConfig function on that > specified ini file if it contains a 'loggers' entry. > logging.fileConfig reads the logging configuration from a ConfigParser > file.""" I'll the the opportunity to note that I experienced head banging to the point of giving up with fileConfig: iirc it has no ability to use log handlers and the like that aren't defined in the logging package (such as mailinglogger) > So, the paster serve is doing special setup for Pylons WSGI > components. Without knowing this magic is required, when you take a > Pylons WSGI component to another WSGI server, the logging doesn't work > as expected. To get it to work one has to duplicate the special things > that paster serve is doing in some way. The reverse also holds true. If the app configures logging and gets users with paster serve, you'll end up with two sets of logs. For me, it does feel like the responsibility of the server to configure logging, and I think this is something that should be documented somewhere. Afterall, as you guys have been discussing, it's the server that holds configuration for things like listening sockets, etc and logging feels like that kind of thing to me... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From manlio_perillo at libero.it Wed Dec 19 15:09:47 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Wed, 19 Dec 2007 15:09:47 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476905DD.5060100@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> Message-ID: <4769262B.3050106@libero.it> Chris Withers ha scritto: > [...] > > For me, it does feel like the responsibility of the server to configure > logging, and I think this is something that should be documented > somewhere. Afterall, as you guys have been discussing, it's the server > that holds configuration for things like listening sockets, etc and > logging feels like that kind of thing to me... > In mod_wsgi for nginx I now redirect sys.stderr to server log file (as suggested by Graham). So, the application can setup the logging module to use sys.stderr as stream. However there are some problems. The log object has a fixed error level (NGX_LOG_ERR); this means that every message logged using this object will have this error level, even if I do, as example: log.info('just an info message') This can be a problem if I use a log filter that will send an email every time it sees a NGX_LOG_ERR message in log file. A better solution could be the integration with the logging module. However there are some problems here, too, since log levels from nginx and the logging module differs. > cheers, > > Chris > Manlio Perillo From pje at telecommunity.com Wed Dec 19 18:46:27 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Dec 2007 12:46:27 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4768FCB6.4000806@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <475D5F63.4090007@libero.it> <47629AA1.8020305@simplistix.co.uk> <4762B5C1.8050409@libero.it> <4766A77B.9050704@simplistix.co.uk> <4766AA41.9060600@libero.it> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <20071218205958.EBDAD3A40BF@sparrow.telecommunity.com> <4768372D.2080408@libero.it> <20071218223525.B43873A40D9@sparrow.telecommunity.com> <47684EDC.70501@libero.it> <20071218232714.E015A3A40BF@sparrow.telecommunity.com> <4768FCB6.4000806@libero.it> Message-ID: <20071219174617.F3FBD3A40A4@sparrow.telecommunity.com> At 12:12 PM 12/19/2007 +0100, Manlio Perillo wrote: >Phillip J. Eby ha scritto: >>At 11:51 PM 12/18/2007 +0100, Manlio Perillo wrote: >>>Phillip J. Eby ha scritto: >>>>At 10:10 PM 12/18/2007 +0100, Manlio Perillo wrote: >>>>>Ok. >>>>>Here I would just say that when someone install something on its >>>>>system, it should at least know what he is doing. >>>>And I repeat: you're welcome to your opinions about what's good >>>>or bad, but that has nothing to do with WSGI's design rationale, >>>>which is based on *different* goals. >>> >>>The problem is that I don't think that having many server >>>configuration parameters with "safe" defaults value is against >>>your design goals. >>Sometimes parameters are a necessary evil; that doesn't make them >>any less evil, just more necessary. :) > >Ok, we diverge here ;-) > >If a program does not have configuration parameters then: >1) the program is probabily very simple >2) the programmer thinks he gets the best values for program > configuration, so that the user does not needs to interfere We're not talking about "a program", we're talking about a WSGI server. By the very nature of WSGI's goals, servers should be interchangeable and transparent to the application. WSGI deliberately gives applications almost complete control over the HTTP transaction, and limits the role of a server to roughly what a HTTP proxy is allowed to do. Again, configuration options for the *server* that's hosting the WSGI application are thus by definition evil -- even if sometimes *necessary* -- since the existence of options means that the application author(s) must be aware of these options in order to communicate to the user/deployer what needs to be done with them. >This is not true: what `wsgi_allow_ranges` means is: >- by default nginx does not try to add support to partial requests, so > this must be implemented by the WSGI application > >- if you enable this directive, then nginx will add (basic) partial > requests support for you A server providing options that allow for optimizing the server's behavior for a specific type of application is a sometimes-necessary evil. For example, an asynchronous server might by default run WSGI applications in their own threads, but have an option to specify that a particular application is non-blocking and thus can be run more efficiently in the main thread. From manlio_perillo at libero.it Thu Dec 20 13:27:30 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Thu, 20 Dec 2007 13:27:30 +0100 Subject: [Web-SIG] a possible error in the WSGI spec Message-ID: <476A5FB2.3000403@libero.it> Hi. It seems that there is a little error in the WSGI spec. In the "Specification Details" chapter there is this note: """(Note: the application must invoke the start_response() callable before the iterable yields its first body string, so that the server can send the headers before any body content. However, this invocation may be performed by the iterable's first iteration, so servers must not assume that start_response() has been called before they begin iterating over the iterable.)""" What's wrong is that the invocation of start_response may be performed at any iteration of the iterable, as long as the application yields empty strings. There is also a little problem in the "The start_response() Callable" chapter, in this phrase: """The start_response callable must not actually transmit the response headers. Instead, it must store them for the server or gateway to transmit only after the first iteration of the application return value that yields a non-empty string, or upon the application's first invocation of the write() callable""" From this it seems that the response headers should be transmitted after the first invocation of the write() callable, even if it returns an empty string, but this is in contradiction with the following phrase: """In other words, response headers must not be sent until there is actual body data available, or until the application's returned iterable is exhausted""" Manlio Perillo From adam at atlas.st Thu Dec 20 16:10:18 2007 From: adam at atlas.st (Adam Atlas) Date: Thu, 20 Dec 2007 10:10:18 -0500 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <476A5FB2.3000403@libero.it> References: <476A5FB2.3000403@libero.it> Message-ID: <87C5BD48-C556-49DB-8970-B032849C8025@atlas.st> On 20 Dec 2007, at 07:27, Manlio Perillo wrote: > What's wrong is that the invocation of start_response may be performed > at any iteration of the iterable, as long as the application yields > empty strings. In what context? I suspect that's an error in a particular implementation, not in the spec. From pje at telecommunity.com Thu Dec 20 17:43:58 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Dec 2007 11:43:58 -0500 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <476A5FB2.3000403@libero.it> References: <476A5FB2.3000403@libero.it> Message-ID: <20071220164350.A32CF3A4106@sparrow.telecommunity.com> At 01:27 PM 12/20/2007 +0100, Manlio Perillo wrote: >Hi. > >It seems that there is a little error in the WSGI spec. > >In the "Specification Details" chapter there is this note: > >"""(Note: the application must invoke the start_response() callable >before the iterable yields its first body string, so that the server can >send the headers before any body content. However, this invocation may >be performed by the iterable's first iteration, so servers must not >assume that start_response() has been called before they begin iterating >over the iterable.)""" > > >What's wrong is that the invocation of start_response may be performed >at any iteration of the iterable, as long as the application yields >empty strings. You're right - the intent here was before the iterable yields its first non-empty body string. >There is also a little problem in the "The start_response() Callable" >chapter, in this phrase: > >"""The start_response callable must not actually transmit the response >headers. Instead, it must store them for the server or gateway to >transmit only after the first iteration of the application return value >that yields a non-empty string, or upon the application's first >invocation of the write() callable""" > > > From this it seems that the response headers should be transmitted >after the first invocation of the write() callable, even if it returns >an empty string, but this is in contradiction with the following phrase: > >"""In other words, response headers must not be sent until there is >actual body data available, or until the application's returned iterable >is exhausted""" There is no sane reason to call write() with an empty string, so I don't really see a conflict here. Please note that both write() and iteration in the normal case should be sending the entire response body in a single call or yield. Multiple yields or write() calls should ONLY be used when streaming output is *required*: i.e. server push or sending large files. (In addition, write() is intended strictly for legacy apps and frameworks only; new code should use iteration only, especially as WSGI 2 will drop the start_response() and write() functions altogether.) From chris at simplistix.co.uk Thu Dec 20 22:12:58 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 20 Dec 2007 21:12:58 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <4769262B.3050106@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <4766B88B.7070300@colorstudy.com> <4766BA9C.5090203@libero.it> <4766BE33.5050707@colorstudy.com> <4766C116.1050309@libero.it> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> Message-ID: <476ADADA.10809@simplistix.co.uk> Manlio Perillo wrote: >> For me, it does feel like the responsibility of the server to >> configure logging, and I think this is something that should be >> documented somewhere. Afterall, as you guys have been discussing, it's >> the server that holds configuration for things like listening sockets, >> etc and logging feels like that kind of thing to me... >> > > In mod_wsgi for nginx I now redirect sys.stderr to server log file (as > suggested by Graham). I've never really understood this desire to do *anything* with sys.stderr. Writing to sys.stderr seems a very odd thing for any web component to use, especially as python now has a fantastic logging package. > However there are some problems. > The log object has a fixed error level (NGX_LOG_ERR); > this means that every message logged using this object will have this > error level, even if I do, as example: > log.info('just an info message') I'm missing the relationship between this and python's logging package. I'll note that the point you raise about why using sys.stderr for logging sucks are things I agree with ;-) > A better solution could be the integration with the logging module. > > However there are some problems here, too, since log levels from nginx > and the logging module differs. Why? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From graham.dumpleton at gmail.com Fri Dec 21 01:00:22 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 21 Dec 2007 11:00:22 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476ADADA.10809@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> Message-ID: <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> On 21/12/2007, Chris Withers wrote: > Manlio Perillo wrote: > >> For me, it does feel like the responsibility of the server to > >> configure logging, and I think this is something that should be > >> documented somewhere. Afterall, as you guys have been discussing, it's > >> the server that holds configuration for things like listening sockets, > >> etc and logging feels like that kind of thing to me... > >> > > > > In mod_wsgi for nginx I now redirect sys.stderr to server log file (as > > suggested by Graham). > > I've never really understood this desire to do *anything* with > sys.stderr. Writing to sys.stderr seems a very odd thing for any web > component to use, especially as python now has a fantastic logging package. I only ever suggested that sys.stderr be directed to the web server log file as a fallback to cope with third party Python modules which use it directly, or where the logging module defaults to outputing it there. > > However there are some problems. > > The log object has a fixed error level (NGX_LOG_ERR); > > this means that every message logged using this object will have this > > error level, even if I do, as example: > > log.info('just an info message') > > I'm missing the relationship between this and python's logging package. > I'll note that the point you raise about why using sys.stderr for > logging sucks are things I agree with ;-) As I believe I have said before, I disagree with how you are trying to make logging module output simply go to sys.stderr. If you want to preserve the concept of logging levels, then expose the internal web server logging function, including it accepting its concept of log level, and write a log handler for the logging module that uses it instead. I can't get to the code right now, but have done exactly this with Apache, where one can install a custom Apache log handler that internally uses a SWIG'd binding for apache.http_log.ap_log_error(). So, there is no reason to force fit log levels into wsgi.errors or sys.stderr somehow. > > A better solution could be the integration with the logging module. > > > > However there are some problems here, too, since log levels from nginx > > and the logging module differs. > > Why? The possibility of mismatch on log levels is common in possible output targets for logging module handlers. If you look at the source code for some of the logging handlers you will see how there is the ability to remap the log levels exposed at Python code level, to what may be available in the output target. Thus, I don't see this an issue, you just make it transparently map as best you can in your nginx specific log handler for the logging module. Graham From brian at briansmith.org Fri Dec 21 01:44:50 2007 From: brian at briansmith.org (Brian Smith) Date: Fri, 21 Dec 2007 07:44:50 +0700 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk><20071217195209.67DE83A40A4@sparrow.telecommunity.com><4766D6D5.6010403@libero.it><20071217210700.45C543A40A4@sparrow.telecommunity.com><47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com><88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com><476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it><476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> Message-ID: <000001c8436a$b54a02e0$0701a8c0@Junk> Graham Dumpleton wrote: > > > However there are some problems. > > > The log object has a fixed error level (NGX_LOG_ERR); > > > this means that every message logged using this object > > > will have this error level, even if I do, as example: > > > log.info('just an info message') > > > > I'm missing the relationship between this and python's > > logging package. > > As I believe I have said before, I disagree with how you are > trying to make logging module output simply go to sys.stderr. > If you want to preserve the concept of logging levels, then > expose the internal web server logging function, including it > accepting its concept of log level, and write a log handler > for the logging module that uses it instead. > So, there is no reason to force fit log levels into > wsgi.errors or sys.stderr somehow. > > > A better solution could be the integration with the > > > logging module. > Thus, I don't see this an issue, you just make it > transparently map as best you can in your nginx specific log > handler for the logging module. The WSGI specification doesn't say anything about what the WSGI gateway will do regarding the python logging module. It only talks about wsgi.errors. My initial understanding of this was that my WSGI application is supposed to ensure that all its loggers write their output to wsgi.errors. But, if I did this, there is no opportunity for the WSGI gateway to translate the logging package's logging levels into the host environment's logging levels, since the level information is lost when the logged information passes through wsgi.errors. Furthermore, doing so turned out to be impossible anyway, since I can't reliably redirect the output of every logger used by every third-party package and every built-in package. It would make more sense for the WSGI specification to explicitly say that WSGI gateways are responsible for setting the default logging output location. Every WSGI gateway has to do this anyway to handle non-WSGI-aware modules that use the python logging module. The specification should then also explicitly say that WSGI applications should not redirect logging output to wsgi.errors or anywhere else. In fact, if that was done, there would be no reason to have wsgi.errors in the first place. See also: http://mail.python.org/pipermail/web-sig/2007-November/002948.html - Brian From pje at telecommunity.com Fri Dec 21 02:12:50 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Dec 2007 20:12:50 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <000001c8436a$b54a02e0$0701a8c0@Junk> References: <475D2C39.2000409@simplistix.co.uk> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> Message-ID: <20071221011247.EA0413A40AC@sparrow.telecommunity.com> At 07:44 AM 12/21/2007 +0700, Brian Smith wrote: >In >fact, if that was done, there would be no reason to have wsgi.errors in >the first place. There are other logging systems out there besides the Python logging module -- and some of them are better for their specific purposes. And the Python logging module doesn't give you any WSGI-level control over output. At least if you use wsgi.errors, you can have middleware that e.g. logs different parts of your application tree to different log outputs. >Furthermore, doing so turned out to be impossible anyway, since I can't >reliably redirect the output of every logger used by every third-party >package and every built-in package. Which just goes to illustrate the problem of using a global logging package that has only static context (which is one of my pet peeves re: the logging module). In my experience, you more want to be able to filter and route logs by component *instance* (e.g. which page of your web app is being rendered) than by code module (e.g. templating library vs. db library). Dividing up the logs by which module issued the message is a bug, not a feature. From graham.dumpleton at gmail.com Fri Dec 21 02:18:09 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 21 Dec 2007 12:18:09 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <000001c8436a$b54a02e0$0701a8c0@Junk> References: <475D2C39.2000409@simplistix.co.uk> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> Message-ID: <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> On 21/12/2007, Brian Smith wrote: > The > specification should then also explicitly say that WSGI applications > should not redirect logging output to wsgi.errors or anywhere else. In > fact, if that was done, there would be no reason to have wsgi.errors in > the first place. At least in the context of Apache, wsgi.errors is different to sys.stderr or a global logging module output target. This is because wsgi.errors is linked to the actual request and so any output can be correctly redirected to a per virtual host error log. If wsgi.errors disappeared then it would be a potential problem where Apache was used for multiple virtual hosts and each virtual host had its own error logs. This is because logging may well end up in main Apache error log to which individual virtual host owners may not have access to. Graham From manlio_perillo at libero.it Fri Dec 21 12:50:34 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 21 Dec 2007 12:50:34 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> Message-ID: <476BA88A.2090309@libero.it> Graham Dumpleton ha scritto: > On 21/12/2007, Chris Withers wrote: >> Manlio Perillo wrote: > [...] >>> In mod_wsgi for nginx I now redirect sys.stderr to server log file (as >>> suggested by Graham). >> I've never really understood this desire to do *anything* with >> sys.stderr. Writing to sys.stderr seems a very odd thing for any web >> component to use, especially as python now has a fantastic logging package. > > I only ever suggested that sys.stderr be directed to the web server > log file as a fallback to cope with third party Python modules which > use it directly, or where the logging module defaults to outputing it > there. > Right. >>> However there are some problems. >>> The log object has a fixed error level (NGX_LOG_ERR); >>> this means that every message logged using this object will have this >>> error level, even if I do, as example: >>> log.info('just an info message') >> I'm missing the relationship between this and python's logging package. >> I'll note that the point you raise about why using sys.stderr for >> logging sucks are things I agree with ;-) > > As I believe I have said before, I disagree with how you The "you" is "me", I presume :) > are trying to > make logging module output simply go to sys.stderr. If you want to > preserve the concept of logging levels, then expose the internal web > server logging function, including it accepting its concept of log > level, and write a log handler for the logging module that uses it > instead. > This is what I plan to do, in future. But there is a problem. Here is how logging.info is implemented: def info(self, msg, *args, **kwargs): if self.manager.disable >= INFO: return if INFO >= self.getEffectiveLevel(): apply(self._log, (INFO, msg, args), kwargs) As I can can see, logging level comparison is done in the high level functions. The problem is that, in nginx, more sever errors have low integer values; in the logging module it is the contrary. So, I'm not sure that I can just implement a logging handler... > I can't get to the code right now, but have done exactly this with > Apache, where one can install a custom Apache log handler that > internally uses a SWIG'd binding for apache.http_log.ap_log_error(). > I will give it a look, thanks. > [...] Manlio Perillo From manlio_perillo at libero.it Fri Dec 21 12:54:10 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 21 Dec 2007 12:54:10 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> Message-ID: <476BA962.4010101@libero.it> Graham Dumpleton ha scritto: > On 21/12/2007, Brian Smith wrote: >> The >> specification should then also explicitly say that WSGI applications >> should not redirect logging output to wsgi.errors or anywhere else. In >> fact, if that was done, there would be no reason to have wsgi.errors in >> the first place. > > At least in the context of Apache, wsgi.errors is different to > sys.stderr or a global logging module output target. This is because > wsgi.errors is linked to the actual request and so any output can be > correctly redirected to a per virtual host error log. > This is the same for Nginx. sys.stderr is linked to the nginx main cycle logging, wsgi.errors to the request logging. > [...] Manlio Perillo From chris at simplistix.co.uk Fri Dec 21 14:57:39 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 21 Dec 2007 13:57:39 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> Message-ID: <476BC653.1020806@simplistix.co.uk> Graham Dumpleton wrote: > At least in the context of Apache, wsgi.errors is different to > sys.stderr or a global logging module output target. This is because > wsgi.errors is linked to the actual request and so any output can be > correctly redirected to a per virtual host error log. Yeah, but there's still not concept of log levels. Why not write a loghandler for python's logging package that takes the message, converts the log levels as necessary and sticks its output into the apache logging system? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Dec 21 15:01:08 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 21 Dec 2007 14:01:08 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <000001c8436a$b54a02e0$0701a8c0@Junk> References: <475D2C39.2000409@simplistix.co.uk><20071217195209.67DE83A40A4@sparrow.telecommunity.com><4766D6D5.6010403@libero.it><20071217210700.45C543A40A4@sparrow.telecommunity.com><47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com><88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com><476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it><476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> Message-ID: <476BC724.6060101@simplistix.co.uk> Brian Smith wrote: > It would make more sense for the WSGI specification to explicitly say > that WSGI gateways are responsible for setting the default logging > output location. Yes, although by this I assume you mean "WSGI gateways are responsible for allowing configuration of log handlers for the python logging package" > non-WSGI-aware modules that use the python logging module. The > specification should then also explicitly say that WSGI applications > should not redirect logging output to wsgi.errors or anywhere else. Right. > In > fact, if that was done, there would be no reason to have wsgi.errors in > the first place. Exactly ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Dec 21 15:03:32 2007 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 21 Dec 2007 14:03:32 +0000 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071221011247.EA0413A40AC@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <20071221011247.EA0413A40AC@sparrow.telecommunity.com> Message-ID: <476BC7B4.3020603@simplistix.co.uk> Phillip J. Eby wrote: > > There are other logging systems out there besides the Python logging > module -- and some of them are better for their specific > purposes. Can you give some examples? > And the Python logging module doesn't give you any > WSGI-level control over output. What do you mean by that? > At least if you use wsgi.errors, you > can have middleware that e.g. logs different parts of your > application tree to different log outputs. if you needed this level of control, why not just do: getLogger('/myurl/whatever').info('...etc...') ...and have the WSGI server configure loggers as appropriate? > Which just goes to illustrate the problem of using a global logging > package that has only static context (which is one of my pet peeves > re: the logging module). I've never thought of it as having static context and I've never bumped into this as a problem :-S > In my experience, you more want to be able > to filter and route logs by component *instance* (e.g. which page of > your web app is being rendered) than by code module (e.g. templating > library vs. db library). ...and what's stopping you doing this? > Dividing up the logs by which module issued > the message is a bug, not a feature. I think I'm missing something: what in the logging package makes you log by which module issued the message? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From pje at telecommunity.com Fri Dec 21 17:31:37 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Dec 2007 11:31:37 -0500 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476BC7B4.3020603@simplistix.co.uk> References: <475D2C39.2000409@simplistix.co.uk> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <20071221011247.EA0413A40AC@sparrow.telecommunity.com> <476BC7B4.3020603@simplistix.co.uk> Message-ID: <20071221163128.9FBB93A40A4@sparrow.telecommunity.com> At 02:03 PM 12/21/2007 +0000, Chris Withers wrote: >I think I'm missing something: what in the logging package makes you >log by which module issued the message? That's the conventional usage: modules that use logging usually use a static logger based on module name. Take a look at the distutils, for example. It's not common for modules that do logging, to take logger objects as part of their API, and if they did, it would almost certainly suck. (Because you don't really want to have to pass loggers to every API, or store them in every object, if you have lots of APIs or lots of places that create the objects.) >Phillip J. Eby wrote: >>There are other logging systems out there besides the Python >>logging module -- and some of them are better for their specific purposes. > >Can you give some examples? This one served well enough for me back in the dark ages of Python 2.2: http://cvs.eby-sarna.com/pylib/AppUtils/Logging.py?view=markup We used the "store loggers on every relevant object" approach, which works reasonably well if you don't have too many places that create objects, or you have a way for objects to "inherit" their loggers (which we did). It was later replaced with peak.running.logs, which was designed to interop with the stdlib logging package. ISTR Twisted has its own logging system as well. From brian at briansmith.org Fri Dec 21 17:58:16 2007 From: brian at briansmith.org (Brian Smith) Date: Fri, 21 Dec 2007 23:58:16 +0700 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476BA962.4010101@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> Message-ID: <003701c843f2$b4dd7ca0$2301a8c0@Junk> Manlio Perillo wrote: > Graham Dumpleton ha scritto: > > On 21/12/2007, Brian Smith wrote: > >> The specification should then also explicitly say > >> that WSGI applications should not redirect logging > >> output to wsgi.errors or anywhere else. > >> In fact, if that was done, there would be no reason > >> to have wsgi.errors in the first place. > > > > At least in the context of Apache, wsgi.errors is > > different to sys.stderr or a global logging module > > output target. This is because wsgi.errors is linked > > to the actual request and so any output can be > > correctly redirected to a per virtual host error log. (Apache) mod_wsgi uses threads, right? You could keep a map (thread -> output stream), possibly using thread-local variables, to keep track of where to redirect stderr and logging output. > This is the same for Nginx. > sys.stderr is linked to the nginx main cycle logging, > wsgi.errors to the request logging. In Nginx, there is only one thread, right? So, it should be a simple matter for the Nginx web_wsgi to keep track of the proper log file to redirects stderr and logging output to. Whether or not the specification requires you to do this, you really should be doing this anyway, because there are a lot of applications that using the long-standard python logging package (including many modules that are not WSGI-specific), and dumping all that output into the wrong location when you have enough information to get it right doesn't make any sense. - Brian From brian at briansmith.org Fri Dec 21 18:07:53 2007 From: brian at briansmith.org (Brian Smith) Date: Sat, 22 Dec 2007 00:07:53 +0700 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <20071221011247.EA0413A40AC@sparrow.telecommunity.com> References: <475D2C39.2000409@simplistix.co.uk> <20071217195209.67DE83A40A4@sparrow.telecommunity.com> <4766D6D5.6010403@libero.it> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <20071221011247.EA0413A40AC@sparrow.telecommunity.com> Message-ID: <004101c843f4$0c5afd30$2301a8c0@Junk> Phillip J. Eby wrote: > There are other logging systems out there besides the Python > logging module -- and some of them are better for their > specific purposes. If I was using a framework, I would use that framework's logging package. But, I specifically want to limit my dependencies as much as possible; that means using built-in modules whenever possible, even when they are less convenient than the mirriad of third-party alternatives. > And the Python logging module doesn't > give you any WSGI-level control over output. At least if you > use wsgi.errors, you can have middleware that e.g. logs > different parts of your application tree to different log outputs. My understanding of the WSGI spec is that wsgi.errors is only supposed to be used for errors, not for tracing messages or other non-error messages. How would a WSGI gateway go about enabling/disabling different logging levels? Assume the default formatter was used, and use regexes? > Which just goes to illustrate the problem of using a global > logging package that has only static context (which is one of > my pet peeves re: the logging module). In my experience, you > more want to be able to filter and route logs by component > *instance* (e.g. which page of your web app is being rendered) > than by code module (e.g. templating library vs. db library). The built-in logging module allows you to do this already. But, it isn't very convenient. - Brian From manlio_perillo at libero.it Fri Dec 21 18:40:51 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Fri, 21 Dec 2007 18:40:51 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <003701c843f2$b4dd7ca0$2301a8c0@Junk> References: <475D2C39.2000409@simplistix.co.uk> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> Message-ID: <476BFAA3.8050606@libero.it> Brian Smith ha scritto: > Manlio Perillo wrote: > [...] >> This is the same for Nginx. >> sys.stderr is linked to the nginx main cycle logging, >> wsgi.errors to the request logging. > > In Nginx, there is only one thread, right? Right. It is an asynchronous server, with support to multiprocessing. > So, it should be a simple > matter for the Nginx web_wsgi to keep track of the proper log file to > redirects stderr and logging output to. > For wsgi.errors it is not a problem. With sys.stderr now I redirect it to the main error log of nginx. This means that all applications share the same log file. I can modify the code, so that: - sys.stderr for the main interpreter goes to the main error log - sys.stderr for subinterpreters goes to the error log declared in the HTTP location where the WSGI application is "mounted" In mod_wsgi for nginx, there is a sub-interpreter for each application, unless the application want to use the main interpreter (or sub interpreters are disable - this is the default). Two applications can not share the same sub interpreter, but they can share the main interpreter. Instead of using sys.stderr, a better solution is to add a new log object to the WSGI environment dictionary, so that each application can have its error log redirected to different files. Well, for nginx this is not really an issue, since usually you want to execute only one WSGI application per server. > [...] Manlio Perillo From brian at briansmith.org Fri Dec 21 18:56:38 2007 From: brian at briansmith.org (Brian Smith) Date: Sat, 22 Dec 2007 00:56:38 +0700 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476BFAA3.8050606@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <20071217210700.45C543A40A4@sparrow.telecommunity.com> <47683279.9010008@libero.it> <476855F2.3080007@colorstudy.com> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <476BFAA3.8050606@libero.it> Message-ID: <000001c843fa$dbd1bda0$2301a8c0@Junk> Manlio Perillo wrote: > I can modify the code, so that: > - sys.stderr for the main interpreter goes to the main error log > - sys.stderr for subinterpreters goes to the error log declared in the > HTTP location where the WSGI application is "mounted" I think that makes sense. To effectively handle logging levels, you need to configure the root log handler appropriately as well. > Instead of using sys.stderr, a better solution is to add a new log > object to the WSGI environment dictionary, so that each > application can have its error log redirected to different files. I agree, but (a) that would have to be standardized somewhere to be useful, and (b) you still have to deal with code that isn't aware of this new functionality--especially libraries that are not WSGI-specific, and existing WSGI 1.0 applications. - Brian From graham.dumpleton at gmail.com Sat Dec 22 03:48:55 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 22 Dec 2007 13:48:55 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <003701c843f2$b4dd7ca0$2301a8c0@Junk> References: <475D2C39.2000409@simplistix.co.uk> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> Message-ID: <88e286470712211848u32e4c283v905658fbe3b188c5@mail.gmail.com> On 22/12/2007, Brian Smith wrote: > Manlio Perillo wrote: > > Graham Dumpleton ha scritto: > > > On 21/12/2007, Brian Smith wrote: > > >> The specification should then also explicitly say > > >> that WSGI applications should not redirect logging > > >> output to wsgi.errors or anywhere else. > > >> In fact, if that was done, there would be no reason > > >> to have wsgi.errors in the first place. > > > > > > At least in the context of Apache, wsgi.errors is > > > different to sys.stderr or a global logging module > > > output target. This is because wsgi.errors is linked > > > to the actual request and so any output can be > > > correctly redirected to a per virtual host error log. > > (Apache) mod_wsgi uses threads, right? You could keep a map (thread -> > output stream), possibly using thread-local variables, to keep track of > where to redirect stderr and logging output. I thought about that and decided not to do it. Part of the reason is the extra complexity, the other is that it still isn't able to cater for all cases. The case it still doesn't catch is where user code creates Python threads unrelated to a specific request. These still can't be tied to the request. For that matter it also doesn't catch user code which is triggered from atexit callbacks on process shutdown. It therefore seemed more consistent for only wsgi.errors to be bound to request, given that it comes through request environment. This can then map to internal Apache ap_log_rerror() function, allowing client IP to be listed against message in error log file. For sys.stderr, instead use internal Apache ap_log_error() function, which will log against server context. For embedded mode of mod_wsgi this is main Apache error log and for daemon mode the VirtualHost if daemon process is associated with the virtual host. Thus if VirtualHost has its own error log different to the main Apache error log it will go there. This seemed to be about the best compromise one could hope for and so what the mod_wsgi C core code does. FWIW, there is actually nothing to stop you in mod_wsgi from implementing what you want at the Python code level. In other words, mod_wsgi will not complain if you replace sys.stderr and at the same time use a WSGI wrapper which does some registration of the thread ID such that a smarter sys.stderr can route things back to wsgi.errors instead. It would certainly be a lot easier to do at the Python code level than at C code level. I even added some hooks in latest release candidate so that such WSGI wrappers could be transparently added into the application stack and thus allow such things to be done across a lot of application entry points at the same time. Users couldn't see the point of why I added it and as such I'll likely remove the feature rather than argue the point. Graham From graham.dumpleton at gmail.com Sat Dec 22 04:06:45 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 22 Dec 2007 14:06:45 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <000001c843fa$dbd1bda0$2301a8c0@Junk> References: <475D2C39.2000409@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> Message-ID: <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> On 22/12/2007, Brian Smith wrote: > Manlio Perillo wrote: > > > I can modify the code, so that: > > - sys.stderr for the main interpreter goes to the main error log > > - sys.stderr for subinterpreters goes to the error log declared in the > > HTTP location where the WSGI application is "mounted" > > I think that makes sense. To effectively handle logging levels, you need > to configure the root log handler appropriately as well. > > > Instead of using sys.stderr, a better solution is to add a new log > > object to the WSGI environment dictionary, so that each > > application can have its error log redirected to different files. > > I agree, but (a) that would have to be standardized somewhere to be > useful, and (b) you still have to deal with code that isn't aware of > this new functionality--especially libraries that are not WSGI-specific, > and existing WSGI 1.0 applications. The more and more that this discussion goes on, the conclusion I am coming to is that WSGI applications should simply not be using the web server log files for application logging at all. The main problem with Apache at least is that Apache has its own concept of a log level which is independent of any Python logging system. Even if you were to map Python log messages to equivalent log levels in Apache, if the Python log level threshold is set to be higher than that which Apache is filtering at, then Apache will just throw away the log messages. There is not way around this. Given that Apache default is 'warn', it means that any Python log messages at level of 'info' or 'debug' would be thrown away by Apache if Python let them threw. It is unlikely that ISPs will run with anything higher than 'warn' because of the extra output from Apache modules themselves. If you are lucky they may allow 'info' within a specific VirtualHost if it has its own log files, but certainly not 'debug'. As such, seems that using Apache error log levels for Python is a bad match anyway. Thus I really thing that having sys.stderr go to 'error' level in Apache is more than reasonable as least it is displayed. If Python logging module sends to sys.stderr, all logging will be seen without problem. The only complaint seems to be that the Python log level is tagged on to the start of the log message after the Apache log level is shown. In respect of using a web server feature for mailing out error messages of a certain level, that just seems wrong to me. This should be done in Python code using log handlers to the logging module if you want to use the logging module. Alternatively, use a log handler which sends logging to a proper logging system like syslog-ng and configure it to do the mail outs. Anyway, this whole discussion is taking too much of the little time I have available at the moment. As such I intend to leave Apache mod_wsgi as is and will not now even look at a log handler for Apache error log files, or even a hook to allow one to log to Apache error logs with a log level. If some official sort of specification comes up for logging in conjunction with WSGI, or someone wants to come up with working code that demonstrates exactly how it should work and it addresses all issues, I'll look at it then. Graham From manlio_perillo at libero.it Sat Dec 22 10:41:37 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sat, 22 Dec 2007 10:41:37 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> Message-ID: <476CDBD1.30107@libero.it> Graham Dumpleton ha scritto: > [...] > > The more and more that this discussion goes on, the conclusion I am > coming to is that WSGI applications should simply not be using the web > server log files for application logging at all. > The problem with this is that log files cannot be easily rotated. > [...] Manlio Perillo From manlio_perillo at libero.it Sat Dec 22 10:45:27 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sat, 22 Dec 2007 10:45:27 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712211848u32e4c283v905658fbe3b188c5@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <88e286470712181736g24ba8b73i16dcfe1b3b256de6@mail.gmail.com> <476905DD.5060100@simplistix.co.uk> <4769262B.3050106@libero.it> <476ADADA.10809@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <88e286470712211848u32e4c283v905658fbe3b188c5@mail.gmail.com> Message-ID: <476CDCB7.3040000@libero.it> Graham Dumpleton ha scritto: > [...] > It therefore seemed more consistent for only wsgi.errors to be bound > to request, given that it comes through request environment. This can > then map to internal Apache ap_log_rerror() function, allowing client > IP to be listed against message in error log file. > > For sys.stderr, instead use internal Apache ap_log_error() function, > which will log against server context. For embedded mode of mod_wsgi > this is main Apache error log and for daemon mode the VirtualHost if > daemon process is associated with the virtual host. Thus if > VirtualHost has its own error log different to the main Apache error > log it will go there. > > This seemed to be about the best compromise one could hope for and so > what the mod_wsgi C core code does. > +1. Manlio Perillo From graham.dumpleton at gmail.com Sat Dec 22 11:08:55 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 22 Dec 2007 21:08:55 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476CDBD1.30107@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> <476CDBD1.30107@libero.it> Message-ID: <88e286470712220208i1ec27df9g8799d263dc5bdd49@mail.gmail.com> On 22/12/2007, Manlio Perillo wrote: > Graham Dumpleton ha scritto: > > [...] > > > > The more and more that this discussion goes on, the conclusion I am > > coming to is that WSGI applications should simply not be using the web > > server log files for application logging at all. > > > > The problem with this is that log files cannot be easily rotated. There are rotating log handlers in the Python logging package. It only becomes problematic with a multi process server where you want to merge output from multiple processes into one log file. This is where passing log messages to syslog-ng could be used, with it handling log rotation. Graham From fumanchu at aminus.org Mon Dec 24 05:35:44 2007 From: fumanchu at aminus.org (Robert Brewer) Date: Sun, 23 Dec 2007 20:35:44 -0800 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <4769262B.3050106@libero.it><476ADADA.10809@simplistix.co.uk><88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com><000001c8436a$b54a02e0$0701a8c0@Junk><88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com><476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk><476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> Message-ID: Graham Dumpleton wrote: > On 22/12/2007, Brian Smith wrote: > > Manlio Perillo wrote: > > > Instead of using sys.stderr, a better solution is to add a new log > > > object to the WSGI environment dictionary, so that each > > > application can have its error log redirected to different files. > > > > I agree, but (a) that would have to be standardized somewhere to be > > useful, and (b) you still have to deal with code that isn't aware of > > this new functionality--especially libraries that are not WSGI- > > specific, and existing WSGI 1.0 applications. > > The more and more that this discussion goes on, the conclusion I am > coming to is that WSGI applications should simply not be using the web > server log files for application logging at all. I still say the answer to "should logging be done by the application or server?" is "neither". We need a component that covers the "everything else" of WSGI; that is, the environment in which servers and applications are instantiated, connected, started, stopped, and shut down. Logging should be offered by that component. http://www.cherrypy.org/wiki/WSPBSpec Robert Brewer fumanchu at aminus.org From chad at zetaweb.com Mon Dec 24 06:41:01 2007 From: chad at zetaweb.com (Chad Whitacre) Date: Mon, 24 Dec 2007 00:41:01 -0500 Subject: [Web-SIG] [ANN] Aspen 0.8: Introducing Simplates Message-ID: <476F466D.8030706@zetaweb.com> Greetings, program! I'm happy to announce version 0.8 of Aspen, a Python webserver. This version introduces "simplates," which bring Python and web templating as close as possible without actually mixing them. You'll find a 3-minute intro video plus docs and downloads here: http://www.zetadev.com/software/aspen/ Thanks! chad From manlio_perillo at libero.it Mon Dec 24 12:24:16 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 24 Dec 2007 12:24:16 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: References: <475D2C39.2000409@simplistix.co.uk> <4769262B.3050106@libero.it><476ADADA.10809@simplistix.co.uk><88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com><000001c8436a$b54a02e0$0701a8c0@Junk><88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com><476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk><476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> Message-ID: <476F96E0.2010809@libero.it> Robert Brewer ha scritto: > [...] > I still say the answer to "should logging be done by the application or > server?" is "neither". We need a component that covers the "everything > else" of WSGI; that is, the environment in which servers and > applications are instantiated, connected, started, stopped, and shut > down. Logging should be offered by that component. > > http://www.cherrypy.org/wiki/WSPBSpec > I think that WSPB cannot be implemented in a WSGI implementation embedded in an existing web server. > > Robert Brewer > fumanchu at aminus.org Manlio Perillo From manlio_perillo at libero.it Mon Dec 24 12:25:33 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 24 Dec 2007 12:25:33 +0100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <88e286470712220208i1ec27df9g8799d263dc5bdd49@mail.gmail.com> References: <475D2C39.2000409@simplistix.co.uk> <88e286470712201600k549c4c9dnc6248e404c302f52@mail.gmail.com> <000001c8436a$b54a02e0$0701a8c0@Junk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> <476CDBD1.30107@libero.it> <88e286470712220208i1ec27df9g8799d263dc5bdd49@mail.gmail.com> Message-ID: <476F972D.1030100@libero.it> Graham Dumpleton ha scritto: > On 22/12/2007, Manlio Perillo wrote: >> Graham Dumpleton ha scritto: >>> [...] >>> >>> The more and more that this discussion goes on, the conclusion I am >>> coming to is that WSGI applications should simply not be using the web >>> server log files for application logging at all. >>> >> The problem with this is that log files cannot be easily rotated. > > There are rotating log handlers in the Python logging package. > The problem is that it is not interoperable with external tools like logrotate. Manlio Perillo From manlio_perillo at libero.it Mon Dec 24 12:28:29 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Mon, 24 Dec 2007 12:28:29 +0100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <20071220164350.A32CF3A4106@sparrow.telecommunity.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> Message-ID: <476F97DD.7080904@libero.it> Phillip J. Eby ha scritto: > [...] >> There is also a little problem in the "The start_response() Callable" >> chapter, in this phrase: >> >> """The start_response callable must not actually transmit the response >> headers. Instead, it must store them for the server or gateway to >> transmit only after the first iteration of the application return value >> that yields a non-empty string, or upon the application's first >> invocation of the write() callable""" >> >> >> From this it seems that the response headers should be transmitted >> after the first invocation of the write() callable, even if it returns >> an empty string, but this is in contradiction with the following phrase: >> >> """In other words, response headers must not be sent until there is >> actual body data available, or until the application's returned iterable >> is exhausted""" > > There is no sane reason to call write() with an empty string, so I don't > really see a conflict here. > Yes, there are no sane reasons, but it is still not explicitly forbidden. > Please note that both write() and iteration in the normal case should be > sending the entire response body in a single call or yield. Multiple > yields or write() calls should ONLY be used when streaming output is > *required*: i.e. server push or sending large files. > > (In addition, write() is intended strictly for legacy apps and > frameworks only; new code should use iteration only, especially as WSGI > 2 will drop the start_response() and write() functions altogether.) > > By the way: isn't it better to first release a WSGI 1.1 before jumping to a (incompatible) WSGI 2.0? Manlio Perillo From pje at telecommunity.com Wed Dec 26 06:00:25 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Dec 2007 00:00:25 -0500 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <476F97DD.7080904@libero.it> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> Message-ID: <20071226050028.F3D133A40B1@sparrow.telecommunity.com> At 12:28 PM 12/24/2007 +0100, Manlio Perillo wrote: >By the way: isn't it better to first release a WSGI 1.1 before >jumping to a (incompatible) WSGI 2.0? Better for whom, and for what purpose? From graham.dumpleton at gmail.com Wed Dec 26 06:15:44 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Wed, 26 Dec 2007 16:15:44 +1100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <20071226050028.F3D133A40B1@sparrow.telecommunity.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> Message-ID: <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> On 26/12/2007, Phillip J. Eby wrote: > At 12:28 PM 12/24/2007 +0100, Manlio Perillo wrote: > >By the way: isn't it better to first release a WSGI 1.1 before > >jumping to a (incompatible) WSGI 2.0? > > Better for whom, and for what purpose? As has been pointed out before, the main discrepancy known of is the definition of readline() on wsgi.input. Don't know of anything that implements it per the specification because if it was written per the specification then cgi.FieldStorage wouldn't work. The other more recent issue is how to interpret the WSGI specification for Python 3. Personally I don't think it is sufficient that the only mention of these issues is in the archives of the mailing list. Even if you personally don't want a version 1.1 specification, would you consider an official addendum to the 1.0 specification with it being at least posted on www.wsgi.org if the PEP itself can't be amended. Graham From graham.dumpleton at gmail.com Wed Dec 26 11:34:15 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Wed, 26 Dec 2007 21:34:15 +1100 Subject: [Web-SIG] serving (potentially large) files through wsgi? In-Reply-To: <476F972D.1030100@libero.it> References: <475D2C39.2000409@simplistix.co.uk> <88e286470712201718j56d50e85y8c99dffbe3f7587c@mail.gmail.com> <476BA962.4010101@libero.it> <003701c843f2$b4dd7ca0$2301a8c0@Junk> <476BFAA3.8050606@libero.it> <000001c843fa$dbd1bda0$2301a8c0@Junk> <88e286470712211906u3793721q2aefd7dc6db323d1@mail.gmail.com> <476CDBD1.30107@libero.it> <88e286470712220208i1ec27df9g8799d263dc5bdd49@mail.gmail.com> <476F972D.1030100@libero.it> Message-ID: <88e286470712260234x292a70d5ka8f499bb5de5401a@mail.gmail.com> On 24/12/2007, Manlio Perillo wrote: > Graham Dumpleton ha scritto: > > On 22/12/2007, Manlio Perillo wrote: > >> Graham Dumpleton ha scritto: > >>> [...] > >>> > >>> The more and more that this discussion goes on, the conclusion I am > >>> coming to is that WSGI applications should simply not be using the web > >>> server log files for application logging at all. > >>> > >> The problem with this is that log files cannot be easily rotated. > > > > There are rotating log handlers in the Python logging package. > > The problem is that it is not interoperable with external tools like > logrotate. Why does it need to be interoperable? If the logging module handler does the rotation then you don't need logrotate to be doing it again. The issue with things like logrotate is that either the application doing the logging has to be able to accept a signal to tell it to close and reopen the log file, or the application itself needs to simply close and reopen the log file on a periodic basis. As I said before, the other option it to use a log handler which uses syslog/syslog-ng instead and have those packages handle rotation and other stuff for you. Graham From pje at telecommunity.com Wed Dec 26 17:16:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Dec 2007 11:16:44 -0500 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com > References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> Message-ID: <20071226161634.498813A40A4@sparrow.telecommunity.com> At 04:15 PM 12/26/2007 +1100, Graham Dumpleton wrote: >On 26/12/2007, Phillip J. Eby wrote: > > At 12:28 PM 12/24/2007 +0100, Manlio Perillo wrote: > > >By the way: isn't it better to first release a WSGI 1.1 before > > >jumping to a (incompatible) WSGI 2.0? > > > > Better for whom, and for what purpose? > >As has been pointed out before, the main discrepancy known of is the >definition of readline() on wsgi.input. Don't know of anything that >implements it per the specification because if it was written per the >specification then cgi.FieldStorage wouldn't work. > >The other more recent issue is how to interpret the WSGI specification >for Python 3. > >Personally I don't think it is sufficient that the only mention of >these issues is in the archives of the mailing list. Even if you >personally don't want a version 1.1 specification, would you consider >an official addendum to the 1.0 specification with it being at least >posted on www.wsgi.org if the PEP itself can't be amended. These are good things to have, sure. But this doesn't answer the question of why doing that first would be *better* than doing 2.0 first, assuming that there is any reason why they can't or shouldn't be worked on concurrently. From graham.dumpleton at gmail.com Wed Dec 26 23:36:13 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Thu, 27 Dec 2007 09:36:13 +1100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <20071226161634.498813A40A4@sparrow.telecommunity.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> Message-ID: <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.com> On 27/12/2007, Phillip J. Eby wrote: > At 04:15 PM 12/26/2007 +1100, Graham Dumpleton wrote: > >On 26/12/2007, Phillip J. Eby wrote: > > > At 12:28 PM 12/24/2007 +0100, Manlio Perillo wrote: > > > >By the way: isn't it better to first release a WSGI 1.1 before > > > >jumping to a (incompatible) WSGI 2.0? > > > > > > Better for whom, and for what purpose? > > > >As has been pointed out before, the main discrepancy known of is the > >definition of readline() on wsgi.input. Don't know of anything that > >implements it per the specification because if it was written per the > >specification then cgi.FieldStorage wouldn't work. > > > >The other more recent issue is how to interpret the WSGI specification > >for Python 3. > > > >Personally I don't think it is sufficient that the only mention of > >these issues is in the archives of the mailing list. Even if you > >personally don't want a version 1.1 specification, would you consider > >an official addendum to the 1.0 specification with it being at least > >posted on www.wsgi.org if the PEP itself can't be amended. > > These are good things to have, sure. But this doesn't answer the > question of why doing that first would be *better* than doing 2.0 > first, assuming that there is any reason why they can't or shouldn't > be worked on concurrently. Because your name is on the WSGI PEP, people tend to look to you to take a lead on things rather than cross you and do it in your absence. The WSGI 2.0 specification has been brought up before on a number of occasions and although a number of discussion points have been brought up, it is effectively dead in the water and nothing is happening. If you were to take the ideas for WSGI 2.0 on board and push forward with it as most seem to be expecting, then maybe jumping to 2.0 could happen, but it isn't. Due to this inactivity at least, some I guess would like to see the 1.1 specification created or at least an amendment to 1.0 to at least adjust points in the original specification that were in hindsight wrong or impractical, plus allow for Python 3.0. It is a silly situation to have that many if not all WSGI adapters in existence are not even strictly in compliance with the specification. If there is never going to be a 1.1, then maybe we should start a campaign to force all developers of WSGI adapters to modify their code to make it 1.0 compliant. If that then means that most of the Python web frameworks then no longer work, then so be it, after all, they can't be WSGI compliant either if that is the case. Now, if you feel that you aren't really the BDFL for the WSGI specification then clearly state that and let us get on with setting up a proper community process, including some form of voting and oversight mechanism to herald the development of the WSGI 2.0. Otherwise I can't really see much happening with it as it appears that everyone is waiting for you to carry it forward from here. Graham From pje at telecommunity.com Thu Dec 27 03:30:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Dec 2007 21:30:10 -0500 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.co m> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.com> Message-ID: <20071227023000.89F0B3A4067@sparrow.telecommunity.com> At 09:36 AM 12/27/2007 +1100, Graham Dumpleton wrote: >Due to this inactivity at least, some I guess would like to see the >1.1 specification created or at least an amendment to 1.0 to at least >adjust points in the original specification that were in hindsight >wrong or impractical, plus allow for Python 3.0. It is a silly >situation to have that many if not all WSGI adapters in existence are >not even strictly in compliance with the specification. I'd love it if somebody would be so kind as to distill a summary of the points that need to go into a 1.0 errata/clarifications document, and I would be happy to update the PEP (and perhaps make it Final, at long last) following discussion and consensus where possible, pronouncement where not. I don't think a 1.1 is *necessarily* required; most of the issues that might make a 1.1 desirable may be better handled via a 2.0, and getting some closure on the issues that *can* be closed for 1.0 could be a good place to start that process. >Otherwise I can't really see much happening with it as it appears that >everyone is waiting for you to carry it forward from here. I've been thinking, actually, that the best way to get the WSGI 2.0 ball rolling would be to write some code. Given that 2.0 is intended to make writing apps and middleware simpler, it would make more sense for its design to be driven by somewhat more aesthetic considerations. Also, we have the ability now to take any of the large body of WSGI 1.0 code and see what it would look like under 2.0, so we can be less theoretical as well. At the same time though, finishing 1.0 will be a good start to 2.0. I want it to be utterly trivial for a 2.0 app to run under a 1.0 server, and *possible* to run 1.0 apps under a 2.0 server, with some potential restrictions. (A 1.0 app that uses write() calls would require either buffering -- which technically violates the 1.0 spec -- or threading, or greenlets, in order to run under 2.0.) So, having the hairy bits of 1.0 fully nailed down, and having a list of what bits we know cause trouble and would like to change, would be good to know before trying to create a formal spec. Informally, OTOH, playing with 2.0 prototypes should be pretty informative, too. From manlio_perillo at libero.it Thu Dec 27 10:56:51 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Thu, 27 Dec 2007 10:56:51 +0100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <20071226161634.498813A40A4@sparrow.telecommunity.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> Message-ID: <477376E3.6040203@libero.it> Phillip J. Eby ha scritto: > At 04:15 PM 12/26/2007 +1100, Graham Dumpleton wrote: >> On 26/12/2007, Phillip J. Eby wrote: >> > At 12:28 PM 12/24/2007 +0100, Manlio Perillo wrote: >> > >By the way: isn't it better to first release a WSGI 1.1 before >> > >jumping to a (incompatible) WSGI 2.0? >> > >> > Better for whom, and for what purpose? >> > >[...] > > These are good things to have, sure. But this doesn't answer the > question of why doing that first would be *better* than doing 2.0 first, > assuming that there is any reason why they can't or shouldn't be worked > on concurrently. > The problem is that WSGI 2.0 does not yet exists; instead WSGI 1.0 has already been implemented in several different environments so it seems better to first "complete" WSGI 1.0 (errata, addendum, or 1.1). Manlio Perillo From manlio_perillo at libero.it Thu Dec 27 11:05:11 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Thu, 27 Dec 2007 11:05:11 +0100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <20071227023000.89F0B3A4067@sparrow.telecommunity.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.com> <20071227023000.89F0B3A4067@sparrow.telecommunity.com> Message-ID: <477378D7.6020504@libero.it> Phillip J. Eby ha scritto: > [...] > > Informally, OTOH, playing with 2.0 prototypes should be pretty > informative, too. > In my WSGI implementation for Nginx I started with the 2.0 draft, since the code flow is more simpler. However, now that I have implemented 1.0, it's a bit hard to add support for 2.0. Moreover I'm not very interested in 2.0 now, since I want first experiment with the asynchronous extension for 1.0 (pause_output) to see if something good can be done with it. Manlio Perillo From cjw at sympatico.ca Thu Dec 27 21:43:28 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu, 27 Dec 2007 15:43:28 -0500 Subject: [Web-SIG] Web.py 0.2 Tutorial Message-ID: <47740E70.4040406@sympatico.ca> I am exploring web.py, using: # webPlay1.py # plug - an attempt to avoid the crash at line 153 in request.py # fvars.__file__ not set import sys __file__= sys.argv[0] # end of plug import web urls= ('/', 'index') class index: def GET(self): print 'Hello world' web.webapi.internalerror = web.debugerror if __name__ == '__main__': web.run(urls, globals(), web.reloader) The above executes OK but neither IE 7 or Firefox triggers the Hello World message. I would welcome any suggestions. Colin W. From titus at caltech.edu Fri Dec 28 07:49:34 2007 From: titus at caltech.edu (Titus Brown) Date: Thu, 27 Dec 2007 22:49:34 -0800 Subject: [Web-SIG] RELEASE: twill 0.9 Message-ID: <20071228064934.GC26913@caltech.edu> ANNOUNCING twill v0.9. twill is a simple language for browsing the Web. It's designed for automated testing of Web sites, but it can be used to interact with Web sites in a variety of ways. In particular, twill supports form submission, cookies, redirects, and HTTP authentication. This release of twill is officially "Pretty Dang Stable". A twill script to use the "I'm feeling lucky" button on Google looks like this: setlocal query "twill Python" go http://www.google.com/ fv 1 q $query submit btnI # use the "I'm feeling lucky" button show (Note that using this script abuses Google's Terms of Service. So don't.) You can install the latest release of twill with easy_install, easy_install twill or download this release at http://darcs.idyll.org/~t/projects/twill-0.9.tar.gz Documentation is included in the .tar.gz and is also online at http://twill.idyll.org/ Miscellaneous details: twill is implemented in Python and uses pyparsing and mechanize. In addition to the existing simple command language, twill can easily be extended with Python. twill also provides a fairly simple and well-documented wrapper around mechanize. Note that twill does not understand JavaScript. --- Significant changes with 0.9: - many small bugs were fixed, of course! - patched subprocess pipe problem (Chris Abraham). This fixes problems with long-running twill processes. - twill now "plays nice" with other versions of mechanize and ClientForm (good for including it in distributions). - Corrected license information => MIT in all files. - new 'info' command gives human-readable page summary. From graham.dumpleton at gmail.com Sat Dec 29 00:55:54 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sat, 29 Dec 2007 10:55:54 +1100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <20071227023000.89F0B3A4067@sparrow.telecommunity.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.com> <20071227023000.89F0B3A4067@sparrow.telecommunity.com> Message-ID: <88e286470712281555m5e4e9efwc48f59d0fa92bafc@mail.gmail.com> On 27/12/2007, Phillip J. Eby wrote: > At 09:36 AM 12/27/2007 +1100, Graham Dumpleton wrote: > >Due to this inactivity at least, some I guess would like to see the > >1.1 specification created or at least an amendment to 1.0 to at least > >adjust points in the original specification that were in hindsight > >wrong or impractical, plus allow for Python 3.0. It is a silly > >situation to have that many if not all WSGI adapters in existence are > >not even strictly in compliance with the specification. > > I'd love it if somebody would be so kind as to distill a summary of > the points that need to go into a 1.0 errata/clarifications document, > and I would be happy to update the PEP (and perhaps make it Final, at > long last) following discussion and consensus where possible, > pronouncement where not. > > I don't think a 1.1 is *necessarily* required; most of the issues > that might make a 1.1 desirable may be better handled via a 2.0, and > getting some closure on the issues that *can* be closed for 1.0 could > be a good place to start that process. I have created the page: http://www.wsgi.org/wsgi/Amendments_1.0 I added the obvious candidates. Anyone else who has ever taken issue with the 1.0 specification, please add any other things which you think would be reasonable. Include links to mailing list archives where issue may have previously been discussed. Thanks. Graham From manlio_perillo at libero.it Sat Dec 29 13:24:49 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sat, 29 Dec 2007 13:24:49 +0100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <88e286470712281555m5e4e9efwc48f59d0fa92bafc@mail.gmail.com> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.com> <20071227023000.89F0B3A4067@sparrow.telecommunity.com> <88e286470712281555m5e4e9efwc48f59d0fa92bafc@mail.gmail.com> Message-ID: <47763C91.4080105@libero.it> Graham Dumpleton ha scritto: > [...] > > I have created the page: > > http://www.wsgi.org/wsgi/Amendments_1.0 > > I added the obvious candidates. > Thanks. > Anyone else who has ever taken issue with the 1.0 specification, > please add any other things which you think would be reasonable. > Include links to mailing list archives where issue may have previously > been discussed. > Only one note: isn't it better to add links to the "official" archives of the mailing list: http://mail.python.org/pipermail/web-sig/ instead of using google groups? > Thanks. > > Graham > Manlio Perillo From manlio_perillo at libero.it Sat Dec 29 13:28:31 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sat, 29 Dec 2007 13:28:31 +0100 Subject: [Web-SIG] calling start_response and the write from a separate thread Message-ID: <47763D6F.6070006@libero.it> Hi. Maybe I have already posted something like this. From the WSGI spec it is not clear if a WSGI application is allowed to call start_response and write from a separate thread. For my WSGI implementation for Nginx, calling the write callable from a separate thread *will* cause problems (unless buffering is not enabled). Thanks Manlio Perillo From graham.dumpleton at gmail.com Sun Dec 30 00:00:44 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sun, 30 Dec 2007 10:00:44 +1100 Subject: [Web-SIG] a possible error in the WSGI spec In-Reply-To: <47763C91.4080105@libero.it> References: <476A5FB2.3000403@libero.it> <20071220164350.A32CF3A4106@sparrow.telecommunity.com> <476F97DD.7080904@libero.it> <20071226050028.F3D133A40B1@sparrow.telecommunity.com> <88e286470712252115o3b620e47y77c3d68a5f1433c@mail.gmail.com> <20071226161634.498813A40A4@sparrow.telecommunity.com> <88e286470712261436u676c2e9fr6dbb695e4278be3f@mail.gmail.com> <20071227023000.89F0B3A4067@sparrow.telecommunity.com> <88e286470712281555m5e4e9efwc48f59d0fa92bafc@mail.gmail.com> <47763C91.4080105@libero.it> Message-ID: <88e286470712291500g1684e4a1raa33f1dfc164e6c8@mail.gmail.com> On 29/12/2007, Manlio Perillo wrote: > Graham Dumpleton ha scritto: > > Anyone else who has ever taken issue with the 1.0 specification, > > please add any other things which you think would be reasonable. > > Include links to mailing list archives where issue may have previously > > been discussed. > > Only one note: isn't it better to add links to the "official" archives > of the mailing list: > http://mail.python.org/pipermail/web-sig/ > instead of using google groups? Personally I'd say Google Groups is better as it is easier for some one not subscribed to the web-sig list to add to the linked conversation without having to subscribe to the actual list. Graham From graham.dumpleton at gmail.com Sun Dec 30 00:03:06 2007 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sun, 30 Dec 2007 10:03:06 +1100 Subject: [Web-SIG] calling start_response and the write from a separate thread In-Reply-To: <47763D6F.6070006@libero.it> References: <47763D6F.6070006@libero.it> Message-ID: <88e286470712291503y77be9eacva9239c3113c925ba@mail.gmail.com> On 29/12/2007, Manlio Perillo wrote: > Hi. > > > Maybe I have already posted something like this. > > From the WSGI spec it is not clear if a WSGI application is allowed to > call start_response and write from a separate thread. > > For my WSGI implementation for Nginx, calling the write callable from a > separate thread *will* cause problems (unless buffering is not enabled). Is that similar perhaps to the issue 'app_iter and threads' listed under WSGI 2.0 discussion points on wiki. Graham From manlio_perillo at libero.it Sun Dec 30 08:28:25 2007 From: manlio_perillo at libero.it (Manlio Perillo) Date: Sun, 30 Dec 2007 08:28:25 +0100 Subject: [Web-SIG] calling start_response and the write from a separate thread In-Reply-To: <88e286470712291503y77be9eacva9239c3113c925ba@mail.gmail.com> References: <47763D6F.6070006@libero.it> <88e286470712291503y77be9eacva9239c3113c925ba@mail.gmail.com> Message-ID: <47774899.1010704@libero.it> Graham Dumpleton ha scritto: > On 29/12/2007, Manlio Perillo wrote: >> Hi. >> >> >> Maybe I have already posted something like this. >> >> From the WSGI spec it is not clear if a WSGI application is allowed to >> call start_response and write from a separate thread. >> >> For my WSGI implementation for Nginx, calling the write callable from a >> separate thread *will* cause problems (unless buffering is not enabled). > > Is that similar perhaps to the issue 'app_iter and threads' listed > under WSGI 2.0 discussion points on wiki. > It's a different topic. In that issue the problem is whether the WSGI middleware can iterate over the app_iter using a different(s) thread(s). There is also: http://mail.python.org/pipermail/web-sig/2005-December/001807.html > Graham > Manlio Perillo