From deron.meranda at gmail.com Tue May 25 21:58:42 2010 From: deron.meranda at gmail.com (Deron Meranda) Date: Tue, 25 May 2010 15:58:42 -0400 Subject: [Web-SIG] httplib and 100-continue support Message-ID: I'm trying to use httplib to make a PUT request. I'd like to be able to correctly use the 100-continue expectation handshake as HTTP/1.1 recommends, such that I wait for a 100 status code back before I send the body. The only examples I've found immediately send the request body without waiting for a 100 status return. I'd like to do something similar to: ----- conn = httplib.HTTPConnection( ... ) conn.putrequest( "PUT", url ) if data_length is None: conn.putheader( "Transfer-Encoding", "chunked" ) else: conn.putheader( "Content-Length", str(data_length) ) conn.putheader( "Expect", "100-continue" ) conn.endheaders() # Wait for any of: timeout, 100, or other error status if timeout or got_100: # Start sending data using conn.send() method ----- There are two problems trying to do this: 1. The endheaders() doesn't actually send any data. You have to do a conn.send( "" ) with an empty string for the headers to get to the server, which is needed before it can then send back a 100 (or other status code). 2. There's no way to wait for a 100 response, or wait for a response with a timeout -- both of which are needed to do the 100-continue logic of HTTP/1.1. Solving #1 is easy, as long as you know to do it. Solving #2 appears to require changes to the httplib library. Something along the lines of adding additional optional arguments to the getresponse() method, as in: resp = conn.getresponse( ignore_continue=False, timeout=5 ) if resp is None: # timeout occured elif resp.status == httplib.CONTINUE: # got 100 continue elif resp.status == httplib.EXPECTATION_FAILED: # server doesn't understand 100-continue elif resp.status == httplib.REQUEST_ENTITY_TOO_LARGE: # message too large for server elif resp.status == httplib.LENGTH_REQUIRED: # can't do chunked encoding, etc. Is there in fact some way to do this that I'm missing, and if not is this something that makes sense to try to fix in httplib? -- Deron Meranda http://deron.meranda.us/ From graham.dumpleton at gmail.com Wed May 26 06:32:45 2010 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Wed, 26 May 2010 14:32:45 +1000 Subject: [Web-SIG] httplib and 100-continue support In-Reply-To: References: Message-ID: On 26 May 2010 05:58, Deron Meranda wrote: > I'm trying to use httplib to make a PUT request. ?I'd like to > be able to correctly use the 100-continue expectation > handshake as HTTP/1.1 recommends, such that I wait for > a 100 status code back before I send the body. ?The only > examples I've found immediately send the request body > without waiting for a 100 status return. > > > I'd like to do something similar to: > > ----- > conn = httplib.HTTPConnection( ... ) > conn.putrequest( "PUT", url ) > if data_length is None: > ? ?conn.putheader( "Transfer-Encoding", "chunked" ) > else: > ? ?conn.putheader( "Content-Length", str(data_length) ) > conn.putheader( "Expect", "100-continue" ) > conn.endheaders() > > # Wait for any of: timeout, 100, or other error status > > if timeout or got_100: > ? ?# Start sending data using conn.send() method > ----- > > There are two problems trying to do this: > > 1. The endheaders() doesn't actually send any data. > You have to do a conn.send( "" ) with an empty string > for the headers to get to the server, which is needed > before it can then send back a 100 (or other status code). > > 2. There's no way to wait for a 100 response, or wait for > a response with a timeout -- both of which are needed > to do the 100-continue logic of HTTP/1.1. > > Solving #1 is easy, as long as you know to do it. > > Solving #2 appears to require changes to the httplib > library. ?Something along the lines of adding additional > optional arguments to the getresponse() method, as in: > > resp = conn.getresponse( ignore_continue=False, timeout=5 ) > > if resp is None: > ? # timeout occured > elif resp.status == httplib.CONTINUE: > ? # got 100 continue > elif resp.status == httplib.EXPECTATION_FAILED: > ? # server doesn't understand 100-continue > elif resp.status == httplib.REQUEST_ENTITY_TOO_LARGE: > ? # message too large for server > elif resp.status == httplib.LENGTH_REQUIRED: > ? # can't do chunked encoding, etc. > > > Is there in fact some way to do this that I'm missing, and if > not is this something that makes sense to try to fix in > httplib? Bigger problem may be to find a server which actually supports 100-continue in the way one would expect it to work. If you want to upload to a Python web application your options are possibly quite limited. In the Apache space, the following definitely work: * mod_wsgi embedded mode And the following will not work as desired: * mod_wsgi daemon mode * CGI * FASTCGI * SCGI * AJP * uWSGI ??? * mod_proxy to backend The problem with the latter is that they all start reading request content immediately and start sending it through to the backend process. This has side effect of triggering the 100 status straight away before the Python web application does anything and so client starts sending data. Thus there is no chance with the latter of the Python web application not reading data and returning an error response before the client is told it is okay to send the data. If you use a front end proxy like nginx in front of Apache, even if using mod_wsgi embedded mode, you also might have problems also as not sure that implements proper end to end 100-continue processing and instead also just starts reading client data straight away with side of effect of generating 100 status. Graham From deron.meranda at gmail.com Wed May 26 07:19:15 2010 From: deron.meranda at gmail.com (Deron Meranda) Date: Wed, 26 May 2010 01:19:15 -0400 Subject: [Web-SIG] httplib and 100-continue support In-Reply-To: References: Message-ID: On Wed, May 26, 2010 at 12:32 AM, Graham Dumpleton wrote: > Bigger problem may be to find a server which actually supports > 100-continue in the way one would expect it to work. Fortunately in my specific case that's not an issue as I wrote the server from scratch (in C) and so I have full control over exactly how it works and how closely to the RFC I can make it. ... This is not a general-purpose web server (or I'd be using Apache/mod_wsgi here), but its a REST interface to a complex application. It's also giving me a good excuse to get even more familiar with the intricacies of the HTTP/1.1 spec. -- So unlike most of my projects where I have Python on the server side ; in this case my Python is just on the client side. But for the wider consideration, it's good to know that server-side support for 100-continue with Python currently has its limits as well. > If you want to upload to a Python web application your options are > possibly quite limited. > > In the Apache space, the following definitely work: > > * mod_wsgi embedded mode > > And the following will not work as desired: > > * mod_wsgi daemon mode Is this a limit in the WSGI spec, with Apache, or just something with the current version of mod_wsgi? I know you've proposed many changes to WSGI to fix a lot of things, and a lot of the discussions seem to be on the encoding or decoding of headers. But was this ability to do things like the 100-continue also among the issues that have been raised? -- Deron Meranda http://deron.meranda.us/ From graham.dumpleton at gmail.com Wed May 26 07:26:13 2010 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Wed, 26 May 2010 15:26:13 +1000 Subject: [Web-SIG] httplib and 100-continue support In-Reply-To: References: Message-ID: On 26 May 2010 15:19, Deron Meranda wrote: > On Wed, May 26, 2010 at 12:32 AM, Graham Dumpleton > wrote: >> Bigger problem may be to find a server which actually supports >> 100-continue in the way one would expect it to work. > > Fortunately in my specific case that's not an issue as I wrote > the server from scratch (in C) and so I have full control over exactly > how it works and how closely to the RFC I can make it. ?... This is > not a general-purpose web server (or I'd be using Apache/mod_wsgi > here), but its a REST interface to a complex application. ?It's also giving > me a good excuse to get even more familiar with the intricacies of the > HTTP/1.1 spec. -- So unlike most of my projects where I have Python > on the server side ; in this case my Python is just on the client side. > > But for the wider consideration, it's good to know that server-side > support for 100-continue with Python currently has its limits as well. > > >> If you want to upload to a Python web application your options are >> possibly quite limited. >> >> In the Apache space, the following definitely work: >> >> * mod_wsgi embedded mode >> >> And the following will not work as desired: >> >> * mod_wsgi daemon mode > > Is this a limit in the WSGI spec, with Apache, or just something > with the current version of mod_wsgi? > > I know you've proposed many changes to WSGI to fix a lot > of things, and a lot of the discussions seem to be on the > encoding or decoding of headers. ?But was this ability to > do things like the 100-continue also among the issues that > have been raised? The limitation is purely an implementation detail of the Apache module for each of those I listed, including the daemon mode part of mod_wsgi, and nothing to do with WSGI specification. In short, to handle end to end 100-continue, one would need to preserve the concept of a handshake between the Apache server child process doing the proxying and the back end server. That is, back end server sends equivalent of 100 status only when WSGI application first performs a wsgi.input.read() and proxying process only starts sending data when it sees that. This can be done but was easier to ignore it for now just like every other proxying solution ignores the problem. It works for embedded mode of mod_wsgi because Apache in that case handles it for you without you doing anything. Graham From alain.poirier at net-ng.com Fri May 28 00:34:41 2010 From: alain.poirier at net-ng.com (Alain Poirier) Date: Fri, 28 May 2010 00:34:41 +0200 Subject: [Web-SIG] [ANN] Nagare IDE 0.1.0 - Pure Web IDE for the Nagare framework Message-ID: <406627A1-4C34-4E18-905C-517593F5CCDE@net-ng.com> Hi all, I'm pleased to announce that the first (0.1.0) version of the Nagare IDE is released! Nagare IDE is a pure Web Integrated Development Environment dedicated to the Nagare Web framework. Using YUI, the Bespin editor, ajax and comet communications, it offers the browsing of your projects, the edition of the sources, the debugging of the raised exceptions and the consultation in real-time of the applications logs. The full documentation with screenshots and how to install it is available at http://www.nagare.org/trac/wiki/NagareIde Enjoy! A. Poirier