From electronixtar at gmail.com Fri Jul 13 04:50:01 2012 From: electronixtar at gmail.com (est) Date: Fri, 13 Jul 2012 10:50:01 +0800 Subject: [Web-SIG] question about connection pool, task queue in WSGI Message-ID: Hi list, I am running a site with django + uwsgi, I have few questions about how WSGI works. 1. Is db connection open/close handled by Django? If it's open/closed per request, can we make a connection pool in wsgi level, then multiple django views can share it? 2. As a general design consideration, can we execute some task *after* the response has returned to client? I have some heavy data processing need to be done after return HttpResponse() in django, the standard way to do this seems like Celery or other task queue with a broker. It's just too heavyweight. Is it possible to do some simple background task in WSGI directly? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From graham.dumpleton at gmail.com Fri Jul 13 05:31:43 2012 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Thu, 12 Jul 2012 20:31:43 -0700 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: References: Message-ID: On 12 July 2012 19:50, est wrote: > Hi list, > > I am running a site with django + uwsgi, I have few questions about how WSGI > works. > > 1. Is db connection open/close handled by Django? If it's open/closed per > request, Yes it is. > can we make a connection pool in wsgi level, then multiple django > views can share it? Only by changing the Django code base from memory. Better off asking on the Django users list. > 2. As a general design consideration, can we execute some task *after* the > response has returned to client? I have some heavy data processing need to > be done after return HttpResponse() in django, the standard way to do this > seems like Celery or other task queue with a broker. It's just too > heavyweight. Is it possible to do some simple background task in WSGI > directly? Read: http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode In doing this though, it ties up the request thread and so it would not be able to handle other requests until your task has finished. Creating background threads at the end of a request is not a good idea unless you do it using a pooling mechanism such that you limit the number of worker threads for your tasks. Because the process can crash or be shutdown, you loose the job as only in memory and thus not persistent. Better to use Celery, or if you think that is too heavy weight, have a look at Redis Queue (RQ) instead. Graham From graham.dumpleton at gmail.com Fri Jul 13 16:56:10 2012 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 13 Jul 2012 07:56:10 -0700 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: References:

Message-ID: Please keep replies in the mailing list. Graham On 13 July 2012 07:18, est wrote: > Thanks for the answer. That's very helpful info. > >> Only by changing the Django code base from memory. Better off asking > on the Django users list. > > Is my idea was good or bad? (make wsgi handle connection pools, instead of > wsgi apps) > > I read Tarek Ziad? last month's experiement of re-use tcp port by specify > socket FDs. It's awesome idea and code btw. I have couple of questions about > it: > > 1. In theory, I presume it's also possible with db connections? (After wsgi > hosting worker ended, handle the db connection FD to the next wsgi worker) > > 2. Is the socket FD the same mechanism like nginx? If you upgrade nginx > binary, restart nginx, the existing http connection won't break. > > 3. Is my following understanding of wsgi model right? > > A wsgi worker process runs the wsgi app (like django), multiple requests are > handled by the same process, the django views process these requests and > returns responses within the same process (possible in fork or threaded way, > or even both?). After a defined number of requests the wsgi worker > terminates and spawns the next wsgi worker process. > > Before hacking into a task queue based on pure wsgi code, I want to make > sure my view of wsgi is correct. :) > > Please advise! Thanks in advance! > > > On Fri, Jul 13, 2012 at 11:31 AM, Graham Dumpleton > wrote: >> >> On 12 July 2012 19:50, est wrote: >> > Hi list, >> > >> > I am running a site with django + uwsgi, I have few questions about how >> > WSGI >> > works. >> > >> > 1. Is db connection open/close handled by Django? If it's open/closed >> > per >> > request, >> >> Yes it is. >> >> > can we make a connection pool in wsgi level, then multiple django >> > views can share it? >> >> Only by changing the Django code base from memory. Better off asking >> on the Django users list. >> >> > 2. As a general design consideration, can we execute some task *after* >> > the >> > response has returned to client? I have some heavy data processing need >> > to >> > be done after return HttpResponse() in django, the standard way to do >> > this >> > seems like Celery or other task queue with a broker. It's just too >> > heavyweight. Is it possible to do some simple background task in WSGI >> > directly? >> >> Read: >> >> http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode >> >> In doing this though, it ties up the request thread and so it would >> not be able to handle other requests until your task has finished. >> >> Creating background threads at the end of a request is not a good idea >> unless you do it using a pooling mechanism such that you limit the >> number of worker threads for your tasks. Because the process can crash >> or be shutdown, you loose the job as only in memory and thus not >> persistent. >> >> Better to use Celery, or if you think that is too heavy weight, have a >> look at Redis Queue (RQ) instead. >> >> Graham > > From graham.dumpleton at gmail.com Sat Jul 14 06:07:01 2012 From: graham.dumpleton at gmail.com (Graham Dumpleton) Date: Fri, 13 Jul 2012 21:07:01 -0700 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: References:

Message-ID: > On 13 July 2012 07:18, est wrote: >> Thanks for the answer. That's very helpful info. >> >>> Only by changing the Django code base from memory. Better off asking >> on the Django users list. >> >> Is my idea was good or bad? (make wsgi handle connection pools, instead of >> wsgi apps) >> >> I read Tarek Ziad? last month's experiement of re-use tcp port by specify >> socket FDs. It's awesome idea and code btw. I have couple of questions about >> it: >> >> 1. In theory, I presume it's also possible with db connections? (After wsgi >> hosting worker ended, handle the db connection FD to the next wsgi worker) Unlikely. HTTP connections are stateless, open database connections are high unlikely to be stateless with the client likely caching certain session information. >> 2. Is the socket FD the same mechanism like nginx? If you upgrade nginx >> binary, restart nginx, the existing http connection won't break. I would be very surprised if you could upgrade nginx, perform a restart and preserve the HTTP listener socket. If you are talking about some other socket I don't know what you are talking about. As you can with Apache, you can likely enact a configuration file change and perform a restart or trigger rereading of the configuration and it would maintain the HTTP listener socket across the configuration restart, but an upgrade implies changing the binary and I know no way that you could easily persist a HTTP listener socket across to an invocation of a new web server instance using a new executable. In Apache you certainly cannot do it, and unless nginx has some magic where the existing nginx execs the new nginx version and somehow communicates through open socket connections to the new process, I very much doubt it would as it would be rather messy to do so. >> 3. Is my following understanding of wsgi model right? >> >> A wsgi worker process runs the wsgi app (like django), multiple requests are >> handled by the same process, the django views process these requests and >> returns responses within the same process (possible in fork or threaded way, >> or even both?). After a defined number of requests the wsgi worker >> terminates and spawns the next wsgi worker process. Different WSGI severs would behave differently, especially around process control, but your model of understand is close enough. >> Before hacking into a task queue based on pure wsgi code, I want to make >> sure my view of wsgi is correct. :) Would still suggest you just use an existing solution. Graham >> Please advise! Thanks in advance! >> >> >> On Fri, Jul 13, 2012 at 11:31 AM, Graham Dumpleton >> wrote: >>> >>> On 12 July 2012 19:50, est wrote: >>> > Hi list, >>> > >>> > I am running a site with django + uwsgi, I have few questions about how >>> > WSGI >>> > works. >>> > >>> > 1. Is db connection open/close handled by Django? If it's open/closed >>> > per >>> > request, >>> >>> Yes it is. >>> >>> > can we make a connection pool in wsgi level, then multiple django >>> > views can share it? >>> >>> Only by changing the Django code base from memory. Better off asking >>> on the Django users list. >>> >>> > 2. As a general design consideration, can we execute some task *after* >>> > the >>> > response has returned to client? I have some heavy data processing need >>> > to >>> > be done after return HttpResponse() in django, the standard way to do >>> > this >>> > seems like Celery or other task queue with a broker. It's just too >>> > heavyweight. Is it possible to do some simple background task in WSGI >>> > directly? >>> >>> Read: >>> >>> http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode >>> >>> In doing this though, it ties up the request thread and so it would >>> not be able to handle other requests until your task has finished. >>> >>> Creating background threads at the end of a request is not a good idea >>> unless you do it using a pooling mechanism such that you limit the >>> number of worker threads for your tasks. Because the process can crash >>> or be shutdown, you loose the job as only in memory and thus not >>> persistent. >>> >>> Better to use Celery, or if you think that is too heavy weight, have a >>> look at Redis Queue (RQ) instead. >>> >>> Graham >> >> From roberto at unbit.it Sat Jul 14 07:52:46 2012 From: roberto at unbit.it (Roberto De Ioris) Date: Sat, 14 Jul 2012 07:52:46 +0200 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: References: Message-ID: > Hi list, > > I am running a site with django + uwsgi, I have few questions about how > WSGI works. > > 1. Is db connection open/close handled by Django? If it's open/closed per > request, can we make a connection pool in wsgi level, then multiple django > views can share it? > > 2. As a general design consideration, can we execute some task *after* the > response has returned to client? I have some heavy data processing need to > be done after return HttpResponse() in django, the standard way to do this > seems like Celery or other task queue with a broker. It's just too > heavyweight. Is it possible to do some simple background task in WSGI > directly? You can abuse one of the feature you already found in uWSGI. The simplest approach would be using the Spooler (check uWSGI docs). It is a simplified celery, where the queue is a simple 'spool directory' (like a printing system). A non-uWSGI related trick, would be having a thread pool (one for each worker) in which you enqueue tasks from the request handler: http://projects.unbit.it/uwsgi/wiki/Example#threadqueue There are other solutions to your problem, but all are not relevant to WSGI, so you may want to move to discussion to the uWSGI list directly. -- Roberto De Ioris http://unbit.it From electronixtar at gmail.com Sat Jul 14 09:38:15 2012 From: electronixtar at gmail.com (est) Date: Sat, 14 Jul 2012 15:38:15 +0800 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: References: Message-ID: These uwsgi features are pretty neat! Thank you! I'll try this. On Sat, Jul 14, 2012 at 1:52 PM, Roberto De Ioris wrote: > > > Hi list, > > > > I am running a site with django + uwsgi, I have few questions about how > > WSGI works. > > > > 1. Is db connection open/close handled by Django? If it's open/closed per > > request, can we make a connection pool in wsgi level, then multiple > django > > views can share it? > > > > 2. As a general design consideration, can we execute some task *after* > the > > response has returned to client? I have some heavy data processing need > to > > be done after return HttpResponse() in django, the standard way to do > this > > seems like Celery or other task queue with a broker. It's just too > > heavyweight. Is it possible to do some simple background task in WSGI > > directly? > > You can abuse one of the feature you already found in uWSGI. > > The simplest approach would be using the Spooler (check uWSGI docs). > > It is a simplified celery, where the queue is a simple 'spool directory' > (like a printing system). > > A non-uWSGI related trick, would be having a thread pool (one for each > worker) in which you enqueue tasks from the request handler: > > http://projects.unbit.it/uwsgi/wiki/Example#threadqueue > > There are other solutions to your problem, but all are not relevant to > WSGI, so you may want to move to discussion to the uWSGI list directly. > > -- > Roberto De Ioris > http://unbit.it > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.sapin at exyr.org Sun Jul 15 17:14:35 2012 From: simon.sapin at exyr.org (Simon Sapin) Date: Sun, 15 Jul 2012 17:14:35 +0200 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: References:

Message-ID: <5002DE5B.4030700@exyr.org> Le 14/07/2012 06:07, Graham Dumpleton a ?crit : >>> >>2. Is the socket FD the same mechanism like nginx? If you upgrade nginx >>> >>binary, restart nginx, the existing http connection won't break. > I would be very surprised if you could upgrade nginx, perform a > restart and preserve the HTTP listener socket. If you are talking > about some other socket I don't know what you are talking about. > > As you can with Apache, you can likely enact a configuration file > change and perform a restart or trigger rereading of the configuration > and it would maintain the HTTP listener socket across the > configuration restart, but an upgrade implies changing the binary and > I know no way that you could easily persist a HTTP listener socket > across to an invocation of a new web server instance using a new > executable. In Apache you certainly cannot do it, and unless nginx has > some magic where the existing nginx execs the new nginx version and > somehow communicates through open socket connections to the new > process, I very much doubt it would as it would be rather messy to do > so. I think that est refers to this: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly Apparently yes, there is specific code in nginx to start the new binary and give it the existing socket. And I think that yes, Tarek?s new Circus is similar to the nginx magic upgrade in that an open socket is passed around processes. Maybe nginx even does this in normal operation with multiple worker processes, but I don?t know. Regards, -- Simon Sapin From bchesneau at gmail.com Sun Jul 15 22:42:53 2012 From: bchesneau at gmail.com (Benoit Chesneau) Date: Sun, 15 Jul 2012 22:42:53 +0200 Subject: [Web-SIG] question about connection pool, task queue in WSGI In-Reply-To: <5002DE5B.4030700@exyr.org> References:

<5002DE5B.4030700@exyr.org> Message-ID: On Sun, Jul 15, 2012 at 5:14 PM, Simon Sapin wrote: > Le 14/07/2012 06:07, Graham Dumpleton a ?crit : > >>>> >>2. Is the socket FD the same mechanism like nginx? If you upgrade >>>> >> nginx >>>> >>binary, restart nginx, the existing http connection won't break. >> >> I would be very surprised if you could upgrade nginx, perform a >> restart and preserve the HTTP listener socket. If you are talking >> about some other socket I don't know what you are talking about. >> >> As you can with Apache, you can likely enact a configuration file >> change and perform a restart or trigger rereading of the configuration >> and it would maintain the HTTP listener socket across the >> configuration restart, but an upgrade implies changing the binary and >> I know no way that you could easily persist a HTTP listener socket >> across to an invocation of a new web server instance using a new >> executable. In Apache you certainly cannot do it, and unless nginx has >> some magic where the existing nginx execs the new nginx version and >> somehow communicates through open socket connections to the new >> process, I very much doubt it would as it would be rather messy to do >> so. > > > I think that est refers to this: > http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly > > Apparently yes, there is specific code in nginx to start the new binary and > give it the existing socket. > > And I think that yes, Tarek?s new Circus is similar to the nginx magic > upgrade in that an open socket is passed around processes. Maybe nginx even > does this in normal operation with multiple worker processes, but I don?t > know. > > Regards, > -- > Simon Sapin Gunicorn does upgrade itself using the USR2 signal just like nginx and share the socket like using an fd between OS processes. However the case of a db is a little different since you handle a connection to a db and not listening on a port. You will need either a multiprocess queue passing messages to one process or launching a connection per processes. You can do that using the hook system of gunicorn. - beno?t