From techtonik at gmail.com Sat May 19 13:32:27 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 19 May 2012 14:32:27 +0300 Subject: [Web-SIG] Web message confirmation and Mailman upgrade Message-ID: Hi, It really bugs me to use Mailman workflow for discussions in 2012. Why a WEB-SIG uses non-web technology for communicating anyway? Is the web still unsuitable technology itself for communication? What do you think about entry barrier for new people to this list? Do you know that you can't send a message without subscribing? Do you know that you actually have to resend the message after you've been subscribed? ---[cut]--- You are not allowed to post to this mailing list, and your message has been automatically rejected. If you think that your messages are being rejected in error, contact the mailing list owner at web-sig-owner at python.org. ---[cut]--- You can't just confirm it like in GMane or enter captch - you have to subscribe, confirm subscription, and resend, then turn off delivery or setup delete filter. I read lists from the web when I have time. GMane could be a solution, but it doesn't up updated threads, so you can't see what's going on with long term discussions unless you have all threads bookmarked. So, a couple of rhetorical questions. What does people here feel about 'eating your own dogfood approach'? Would you like to see an alternative and care a bit more inclusive? I know that it all boils down to the "free time" and "volunteering", but at least we could have an actual Roadmap. I am asking, because I've lost the --[cut]-- reply to my mail (sent about a week ago) inside of innards of me automatic mail filtering software. Clearly, with forum it wouldn't happen. I also have a lost a thread in Catalog-SIG, in which I specifically asked people to CC me and it broke right about the time I forgot to ask. Dammit - 10 years have passed since I have to ask for CC in list for the first time and we are still there. Clearly, with web based software it just doesn't happen. I've added Barry to CC, because I am interested if new Mailman 3 interface can solve these problems? Mailman 3 is already beta - can we try it right now for this group at least? In the past we've discussed possibility to integrate Mailman 3 with Google Groups as two ways mirror with shared (or mutually approved) subscribers, but I became completely lost in the innards of email RFC about envelopes and abstract stuff I'd never have to deal with until this moment. I am not sure if I'll be able to resume. -- anatoly t. From masklinn at masklinn.net Sat May 19 13:56:56 2012 From: masklinn at masklinn.net (Masklinn) Date: Sat, 19 May 2012 13:56:56 +0200 Subject: [Web-SIG] Web message confirmation and Mailman upgrade In-Reply-To: References: Message-ID: <61850F5C-91A2-4FC9-BAF4-C9BC402F73DF@masklinn.net> On 2012-05-19, at 13:32 , anatoly techtonik wrote: > Hi, > > It really bugs me to use Mailman workflow for discussions in 2012. > Why a WEB-SIG uses non-web technology for communicating anyway? Because it works, it works well, and it's a standard communication tool in and around Python? Why "fix" something which is not broken in any way, shape or form? > What do you think about entry barrier for new people to this list? That it's a good thing, especially for a list as low-traffic and involved as WEB-SIG. > Do you know that you can't send a message without subscribing? How is that a problem? > So, a couple of rhetorical questions. > What does people here feel about 'eating your own dogfood approach'? That it does not apply, web-sig is not about creating a forum. Unless you're talking about POST-ing directly via CURL, of course. But I'm pretty sure that's not the case. > Would you like to see an alternative and care a bit more inclusive? Personally, no. I'm just fine with web-sig's once-in-a-blue-moon message nicely landing in my inbox. > I know that it all boils down to the "free time" and "volunteering", > but at least we could have an actual Roadmap. There's no point to a roadmap, if you think you can build a better solution do it, then propose it. > I am asking, because I've lost the --[cut]-- reply to my mail (sent > about a week ago) inside of innards of me automatic mail filtering > software. Not sure how it's a Mailman problem if your filters are broken. > Clearly, with forum it wouldn't happen. That's debatable, and forums have a number of other issues. It's hard to impossible (depending on the forum software) to consume them asynchronously for instance, or to quickly see what you haven't looked at yet; composition usually is absolutely dreadful and boxes people in an in-browser workflow, it's also much harder to synchronize across devices (the forum software has to support read/unread statuses extremely well and at the comment level, I only know of a single forum software reliable in that respect and it's *not* free. It also isn't in Python and doesn't ? as far as I know ? offer a mail-based gateway). From techtonik at gmail.com Sat May 19 14:36:35 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 19 May 2012 15:36:35 +0300 Subject: [Web-SIG] Fwd: Can writing to stderr be evil for web apps? In-Reply-To: References: Message-ID: Hi, Martin expressed concerns that using logging module with stderr output can break web applications, such as PyPI. I couldn't find any proof or denials for this fact, and it became a showstopper for me for further contributions to PyPI, because clearly I can't write good code without the sense of confidence. Here is the commit in question: https://bitbucket.org/techtonik/pypi-techtonik/changeset/5396f8c60d49 I was redirected here from python-dev. So can anybody tell where are those stdout/stderr fears coming from and how to dispell them? (include in WSGI notes) Please, CC (this didn't work - I subscribed, so don't care). -- anatoly t. From alan at xhaus.com Sat May 19 22:09:03 2012 From: alan at xhaus.com (Alan Kennedy) Date: Sat, 19 May 2012 21:09:03 +0100 Subject: [Web-SIG] Fwd: Can writing to stderr be evil for web apps? In-Reply-To: References: Message-ID: [anatoly] > Martin expressed concerns that using logging module with stderr output > can break web applications, such as PyPI. Please can you specify exactly what you mean by "using logging module with stderr output"? Dealing with stderr is a webserver specific concern. Consider the case where you're the author of a webserver that deals with CGI scripts. When you get a request for the CGI script, you start a subprocess to run the script. You must decide what to do with the stdin, stdout and stderr of the process. - CGI mandates that any content that came with the request (e.g. a POST body) should be fed into stdin(if no other mechanism is in place[0]) - CGI mandates that the stdout of the process is sent back to the client (if no other mechanism is in place[1]). - CGI makes no mention of stderr. Various webservers permit configurable handling of stderr. For example, Tomcat has a setting called "swallowOutput" which redirects both stdout and stderr to a log file. (Obviously, Tomcat's treatment of stdout is different for CGI) http://tomcat.apache.org/tomcat-6.0-doc/config/context.html WSGI has a specific mechanism for diagnostic output, wsgi.errors. """ wsgi.errors An output stream (file-like object) to which error output can be written, for the purpose of recording program or other errors in a standardized and possibly centralized location. This should be a "text mode" stream; i.e., applications should use "\n" as a line ending, and assume that it will be converted to the correct line ending by the server/gateway. ... For many servers, wsgi.errors will be the server's main error log. Alternatively, this may be sys.stderr, or a log file of some sort. The server's documentation should include an explanation of how to configure this or where to find the recorded output. A server or gateway may supply different error streams to different applications, if this is desired. """ Lastly, note that WSGI supplies an example CGI gateway, about which it has this to say about error handling """ Note that this simple example has limited error handling, because by default an uncaught exception will be dumped to sys.stderr and logged by the web server. """ http://www.python.org/dev/peps/pep-3333/#the-server-gateway-side So I would say that 1. If you are writing a web application, and want it run under any WSGI container, and for the user to be able to control that output in a way with which they are familiar (i.e. which is documented and may have specific configuration options), send the output to wsgi.errors. 2. If you are writing a web server, you should either capture or ignore stderr. If it is captured, then it is reasonable to, e.g., write it to a file so that the user can find it. It should never be mixed with stdout if stdout is the mechanism by which the application communicates with the webserver, as with CGI. Alan. [0] http://ken.coar.org/cgi/draft-coar-cgi-v11-03.txt """ Section 6.2 Request Message-Bodies As there may be a data entity attached to the request, there MUST be a system defined method for the script to read these data. Unless defined otherwise, this will be via the 'standard input' file descriptor. """ [1] http://ken.coar.org/cgi/draft-coar-cgi-v11-03.txt """ Section 7. Data Output from the CGI Script There MUST be a system defined method for the script to send data back to the server or client; a script MUST always return some data. Unless defined otherwise, this will be via the 'standard output' file descriptor """ From graham.dumpleton at gmail.com Sun May 20 03:05:49 2012 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Sun, 20 May 2012 11:05:49 +1000 Subject: [Web-SIG] Fwd: Can writing to stderr be evil for web apps? In-Reply-To: References: Message-ID: On 19 May 2012 22:36, anatoly techtonik wrote: > Hi, > > Martin expressed concerns that using logging module with stderr output > can break web applications, such as PyPI. I couldn't find any proof or > denials for this fact, and it became a showstopper for me for further > contributions to PyPI, because clearly I can't write good code without > the sense of confidence. > > Here is the commit in question: > https://bitbucket.org/techtonik/pypi-techtonik/changeset/5396f8c60d49 > > I was redirected here from python-dev. So can anybody tell where are > those stdout/stderr fears coming from and how to dispell them? > (include in WSGI notes) Part of this is likely going to be due to my deliberate action in early versions of mod_wsgi to prohibit reading from stdin and writing to stdout. This was specifically done to highlight that portable WSGI application shouldn't be working with stdin/stdout because in doing so they would break under a CGI/WSGI bridge written to conform to the example in the WSGI PEP. People don't like to change though and so a lot of people would say that mod_wsgi was broken and/or that one couldn't use stdout under Apache even though there was a configuration option there to remove the artificial limitation. Over time I saw this grow in to a bit of a myth that one couldn't even use stderr and started to see stupid things like people remapping both stdout and stderr to their own files or even /dev/null. Although this artificial limitation was removed some time ago with it being optionally enabled if you want to test WSGI application portability. Certain LTS Linux distros still ship ancient mod_wsgi versions where the limitation is the default. So I tried to enforce something to make people do the right thing, but people prefer to write crap code I guess. An old blog post where I bemoaned all this can be found at: http://blog.dscpl.com.au/2009/04/wsgi-and-printing-to-standard-output.html If the CGI/WSGI bridge example in the PEP had simple saved away original stdin/stdout and replaced them with an empty StringIO and stderr, this possibly would never have become the issue it was. My own CGI/WSGI bridge does exactly that now so as to allow people writing crap code to still run it on CGI/WSGI if they really need to: https://github.com/GrahamDumpleton/cgi2wsgi Graham From gdamjan at gmail.com Sun May 20 14:12:12 2012 From: gdamjan at gmail.com (Damjan) Date: Sun, 20 May 2012 14:12:12 +0200 Subject: [Web-SIG] websocket support in WSGI Message-ID: <4FB8DF9C.7050004@gmail.com> Recently I was enquiring about the possibility to implement websocket as a WSGI middleware, but I learned that to do that the middleware would need access to the underlying socket. Has there been any thought about this? A wsgi addendum perhaps? -- ?????? From alex.gronholm at nextday.fi Mon May 21 14:36:11 2012 From: alex.gronholm at nextday.fi (=?UTF-8?B?QWxleCBHcsO2bmhvbG0=?=) Date: Mon, 21 May 2012 15:36:11 +0300 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: <4FB8DF9C.7050004@gmail.com> References: <4FB8DF9C.7050004@gmail.com> Message-ID: <4FBA36BB.7080505@nextday.fi> 20.05.2012 15:12, Damjan kirjoitti: > Recently I was enquiring about the possibility to implement websocket > as a WSGI middleware, but I learned that to do that the middleware > would need access to the underlying socket. > > Has there been any thought about this? A wsgi addendum perhaps? > > > This has been discussed before and it's impossible to implement since proxying such things does not work over a number of protocols (FCGI, SCGI). IIRC, the front-end web server likes to send its own 1xx response before the application has a chance to send its own. The original discussion was about implement WebDAV, but WebSocket has the same issues. From simon.sapin at exyr.org Mon May 21 15:47:26 2012 From: simon.sapin at exyr.org (Simon Sapin) Date: Mon, 21 May 2012 15:47:26 +0200 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: <4FBA36BB.7080505@nextday.fi> References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> Message-ID: <4FBA476E.8010600@exyr.org> Le 21/05/2012 14:36, Alex Gr?nholm a ?crit : > This has been discussed before and it's impossible to implement since > proxying such things does not work over a number of protocols (FCGI, > SCGI). IIRC, the front-end web server likes to send its own 1xx response > before the application has a chance to send its own. The original > discussion was about implement WebDAV, but WebSocket has the same issues. What would be possible is a server that does HTTP + WebSocket on the network side and WSGI + (some other API for WebSocket) on the application side, although I don?t know a server doing this. -- Simon Sapin From marc at gsites.de Mon May 21 23:48:42 2012 From: marc at gsites.de (Marcel Hellkamp) Date: Mon, 21 May 2012 23:48:42 +0200 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: <4FBA476E.8010600@exyr.org> References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> <4FBA476E.8010600@exyr.org> Message-ID: <4FBAB83A.5060603@gsites.de> Am 21.05.2012 15:47, schrieb Simon Sapin: > Le 21/05/2012 14:36, Alex Gr?nholm a ?crit : >> This has been discussed before and it's impossible to implement since >> proxying such things does not work over a number of protocols (FCGI, >> SCGI). IIRC, the front-end web server likes to send its own 1xx response >> before the application has a chance to send its own. The original >> discussion was about implement WebDAV, but WebSocket has the same issues. > > What would be possible is a server that does HTTP + WebSocket on the > network side and WSGI + (some other API for WebSocket) on the > application side, although I don?t know a server doing this. > gevent and the gevent-websocket does this. http://bottlepy.org/docs/dev/async.html#finally-websockets The idea: A modified gevent server detects WebSocket requests and immediately answers with handshake headers. Then it puts a socket-like object in environ['wsgi.websocket'] and calls the WSGI application with a dummy start_response() callable. In this mode, the start_response() does nothing and the body iterator is ignored. As long as the application callable blocks (or the body iterator yields something), the socket stays open. Problems with this API: The application is required to block, thus blocking the handling server thread. Traditional threading servers have limited thread-pools. Non-blocking servers (twisted/tornado) would freeze completely. I thought about a similar API a while ago, but with some important changes. The server would put a callable in environ['wsgi.detach'] that, when called, returns a socket-like object and changes the behavior of the server as follows: 1) Calling start_response() after wsgi.detach() writes the headers to the socket immediately (instead of buffering them). Calling it again is an error. Not calling it is allowed, though. 2) The socket is not closed after the body iterator is exhausted. The client has to close it explicitly. In other words: The responsibility for the connection (and socket) is passed to the application. This works well with traditional threaded servers. The application can spawn a new worker thread, put the job into a queue or whatever and then return from the application callable, allowing the server thread to continue handling new connections. Same goes for non-blocking servers. And best of all: This API would be fully optional and backwards compatible to WSGI. Thoughts? From sh at defuze.org Tue May 22 09:38:18 2012 From: sh at defuze.org (Sylvain Hellegouarch) Date: Tue, 22 May 2012 09:38:18 +0200 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: <4FBAB83A.5060603@gsites.de> References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> <4FBA476E.8010600@exyr.org> <4FBAB83A.5060603@gsites.de> Message-ID: > > > In other words: The responsibility for the connection (and socket) is > passed to the application. > > This works well with traditional threaded servers. The application can > spawn a new worker thread, put the job into a queue or whatever and then > return from the application callable, allowing the server thread to > continue handling new connections. > > This is exactly how ws4py was implemented when using CherryPy for the HTTP server performing the handshake [1]. There's also a WSGI middleware [2] but it's heavily geared towards gevent and may not be reusable easily elsewhere I'm afraid. -- - Sylvain http://www.defuze.org http://twitter.com/lawouach [1] https://github.com/Lawouach/WebSocket-for-Python/blob/master/ws4py/server/cherrypyserver.py [2] https://github.com/Lawouach/WebSocket-for-Python/blob/master/ws4py/server/wsgi/middleware.py -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoffspear at gmail.com Tue May 22 15:32:58 2012 From: geoffspear at gmail.com (Geoffrey Spear) Date: Tue, 22 May 2012 09:32:58 -0400 Subject: [Web-SIG] Web message confirmation and Mailman upgrade In-Reply-To: <61850F5C-91A2-4FC9-BAF4-C9BC402F73DF@masklinn.net> References: <61850F5C-91A2-4FC9-BAF4-C9BC402F73DF@masklinn.net> Message-ID: On Sat, May 19, 2012 at 7:56 AM, Masklinn wrote: > Unless you're talking about POST-ing directly via CURL, of course. But > I'm pretty sure that's not the case. CURL? I demand that all messages be posted with urllib! From alex.gronholm at nextday.fi Tue May 22 15:39:09 2012 From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=) Date: Tue, 22 May 2012 16:39:09 +0300 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> <4FBA476E.8010600@exyr.org> <4FBAB83A.5060603@gsites.de> Message-ID: <4FBB96FD.4090707@nextday.fi> 22.05.2012 10:38, Sylvain Hellegouarch kirjoitti: > > > In other words: The responsibility for the connection (and socket) is > passed to the application. > > This works well with traditional threaded servers. The application can > spawn a new worker thread, put the job into a queue or whatever > and then > return from the application callable, allowing the server thread to > continue handling new connections. > > > This is exactly how ws4py was implemented when using CherryPy for the > HTTP server performing the handshake [1]. There's also a WSGI > middleware [2] but it's heavily geared towards gevent and may not be > reusable easily elsewhere I'm afraid. It's also a hack that violates the WSGI spec. It's also not usable through reverse proxying or FCGI/SCGI. > > -- > - Sylvain > http://www.defuze.org > http://twitter.com/lawouach > > [1] > https://github.com/Lawouach/WebSocket-for-Python/blob/master/ws4py/server/cherrypyserver.py > [2] > https://github.com/Lawouach/WebSocket-for-Python/blob/master/ws4py/server/wsgi/middleware.py > > > _______________________________________________ > Web-SIG mailing list > Web-SIG at python.org > Web SIG: http://www.python.org/sigs/web-sig > Unsubscribe: http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi -------------- next part -------------- An HTML attachment was scrubbed... URL: From sh at defuze.org Tue May 22 15:45:50 2012 From: sh at defuze.org (Sylvain Hellegouarch) Date: Tue, 22 May 2012 15:45:50 +0200 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: <4FBB96FD.4090707@nextday.fi> References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> <4FBA476E.8010600@exyr.org> <4FBAB83A.5060603@gsites.de> <4FBB96FD.4090707@nextday.fi> Message-ID: On Tue, May 22, 2012 at 3:39 PM, Alex Gr?nholm wrote: > 22.05.2012 10:38, Sylvain Hellegouarch kirjoitti: > > >> In other words: The responsibility for the connection (and socket) is >> passed to the application. >> >> This works well with traditional threaded servers. The application can >> spawn a new worker thread, put the job into a queue or whatever and then >> return from the application callable, allowing the server thread to >> continue handling new connections. >> >> > This is exactly how ws4py was implemented when using CherryPy for the > HTTP server performing the handshake [1]. There's also a WSGI middleware > [2] but it's heavily geared towards gevent and may not be reusable easily > elsewhere I'm afraid. > > It's also a hack that violates the WSGI spec. It's also not usable through > reverse proxying or FCGI/SCGI. > Yeap that's very true and it'll stay a hack until WSGI is updated to support it or explicitely reject protocols such as WebSocket. In the meanwhile, I'm personally fine having projects like ws4py to play with and decide what could work and what couldn't. -- - Sylvain http://www.defuze.org http://twitter.com/lawouach -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto at unbit.it Wed May 23 21:17:14 2012 From: roberto at unbit.it (Roberto De Ioris) Date: Wed, 23 May 2012 21:17:14 +0200 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> <4FBA476E.8010600@exyr.org> <4FBAB83A.5060603@gsites.de> <4FBB96FD.4090707@nextday.fi> Message-ID: <55a1421ad27d8a0df08c5d6615795ee2.squirrel@manage.unbit.it> > On Tue, May 22, 2012 at 3:39 PM, Alex Gr?nholm > wrote: > >> 22.05.2012 10:38, Sylvain Hellegouarch kirjoitti: >> >> >>> In other words: The responsibility for the connection (and socket) is >>> passed to the application. >>> >>> This works well with traditional threaded servers. The application can >>> spawn a new worker thread, put the job into a queue or whatever and >>> then >>> return from the application callable, allowing the server thread to >>> continue handling new connections. >>> >>> >> This is exactly how ws4py was implemented when using CherryPy for the >> HTTP server performing the handshake [1]. There's also a WSGI middleware >> [2] but it's heavily geared towards gevent and may not be reusable >> easily >> elsewhere I'm afraid. >> >> It's also a hack that violates the WSGI spec. It's also not usable >> through >> reverse proxying or FCGI/SCGI. >> > > > Yeap that's very true and it'll stay a hack until WSGI is updated to > support it or explicitely reject protocols such as WebSocket. In the > meanwhile, I'm personally fine having projects like ws4py to play with and > decide what could work and what couldn't. > Can't we simply define something like environ['wsgi.raw_socket'] (or whatever name you want) mapping to the client (or webserver in case of proxied) socket ? At the very beginning, uWSGI used to map wsgi.input to the webserver socket, and implementing WebSocket (and all of the 'HTTP Upgrade' fatures you can think of, included the ultra-non-standard icecast protocol) was easy for non-buffering webservers (like Apache or Cherokee) http://projects.unbit.it/uwsgi/browser/websockets.py Obviously reading from both wsgi.input and wsgi.raw_socket should be "discouraged" (or directly forbidden) My point is in not referring explicitely to WebSockets, but have a way to simply let the application having access to the communication socket (with some strict rule, like getting only the http body part and so on, to make things easier for middleware developers) I tend to consider mod_wsgi as the "standard de-facto" implementation of WSGI, and adding support for wsgi.raw_socket in it would be really easy (Graham i offer myself for a patch ;), as well as in uWSGI. The problem will rise for nginx (when used as a proxy), but it is generated by its "buffering" policy, so we cannot (for now) make something about that. -- Roberto De Ioris http://unbit.it From pje at telecommunity.com Thu May 24 02:05:59 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 23 May 2012 20:05:59 -0400 Subject: [Web-SIG] websocket support in WSGI In-Reply-To: <55a1421ad27d8a0df08c5d6615795ee2.squirrel@manage.unbit.it> References: <4FB8DF9C.7050004@gmail.com> <4FBA36BB.7080505@nextday.fi> <4FBA476E.8010600@exyr.org> <4FBAB83A.5060603@gsites.de> <4FBB96FD.4090707@nextday.fi> <55a1421ad27d8a0df08c5d6615795ee2.squirrel@manage.unbit.it> Message-ID: On Wed, May 23, 2012 at 3:17 PM, Roberto De Ioris wrote: > Can't we simply define something like environ['wsgi.raw_socket'] (or > whatever name you want) mapping to the client (or webserver in case of > proxied) socket ? > Probably not. What happens if middleware has already sent stuff to that socket, indirectly? I suppose you could add some sort of status indicators to it. What happens if the app writes to it, then returns to some middleware that expects to then start its own response? What happens with program flow control in general? I don't know if sane Websocket support is even *possible* in WSGI, but if it is, then there's a heck of a lot more to be specified there than just an environ key. -------------- next part -------------- An HTML attachment was scrubbed... URL: