From electronixtar at gmail.com  Mon Mar 17 01:53:55 2014
From: electronixtar at gmail.com (est)
Date: Mon, 17 Mar 2014 08:53:55 +0800
Subject: [Web-SIG] Nodejs cluster
Message-ID: <CA+=kK2KqfjdA6VG+tEY5xiA3S-A6rJvL7CfxFxRZkh_yw8pEJA@mail.gmail.com>

Recently I've been playing with Nodejs and websockets, looks like their
community has advanced far ahead of us. For example

http://nodejs.org/api/cluster.html

The best we've got is something like Celery, or Telegraphy for Django, We
have multiprocessing module but it's often recommended to use
zeromq/rabbitmq/pyro/billiard. We really need more than these scaffolds in
Python world. Something like interchangeable async worker interface (to
replace WSGI).

What do you guys think?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140317/d11f8923/attachment.html>

From exarkun at twistedmatrix.com  Mon Mar 17 12:19:05 2014
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Mon, 17 Mar 2014 11:19:05 -0000
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <mailman.61.1395054004.24127.web-sig@python.org>
References: <mailman.61.1395054004.24127.web-sig@python.org>
Message-ID: <20140317111905.6218.1821969208.divmod.xquotient.310@top>

>From: est <electronixtar at gmail.com>
>
>Recently I've been playing with Nodejs and websockets, looks like their
>community has advanced far ahead of us. For example
>
>http://nodejs.org/api/cluster.html
>
>The best we've got is something like Celery, or Telegraphy for Django, 
>We

The example I gave for Twisted Web on 
https://stackoverflow.com/questions/10077745/twistedweb-on-multicore- 
multiprocessor/10088578#10088578 is somewhat similar to this.

Jean-Paul

From techtonik at gmail.com  Mon Mar 17 13:24:15 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 17 Mar 2014 15:24:15 +0300
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <CA+=kK2KqfjdA6VG+tEY5xiA3S-A6rJvL7CfxFxRZkh_yw8pEJA@mail.gmail.com>
References: <CA+=kK2KqfjdA6VG+tEY5xiA3S-A6rJvL7CfxFxRZkh_yw8pEJA@mail.gmail.com>
Message-ID: <CAPkN8xKogtBVEhAO8CNpvo3pn40h=RuW9FUcEB9YG7aUqpkVvw@mail.gmail.com>

On Mon, Mar 17, 2014 at 3:53 AM, est <electronixtar at gmail.com> wrote:
> The best we've got is something like Celery, or Telegraphy for Django

Does IPython suits pattern better?
http://ipython.org/ipython-doc/stable/parallel/parallel_intro.html

> What do you guys think?

I think not to reinvent the bicycle. Use PythonJS + QMachine to get to
web client side + distributed processing without abandoning existing
Python skills.
-- 
anatoly t.

From electronixtar at gmail.com  Tue Mar 18 03:16:44 2014
From: electronixtar at gmail.com (est)
Date: Tue, 18 Mar 2014 10:16:44 +0800
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <CAPkN8xKogtBVEhAO8CNpvo3pn40h=RuW9FUcEB9YG7aUqpkVvw@mail.gmail.com>
References: <CA+=kK2KqfjdA6VG+tEY5xiA3S-A6rJvL7CfxFxRZkh_yw8pEJA@mail.gmail.com>
 <CAPkN8xKogtBVEhAO8CNpvo3pn40h=RuW9FUcEB9YG7aUqpkVvw@mail.gmail.com>
Message-ID: <CA+=kK2JAsfi6dAW0eKJhukFz2rMPctVLA3_N4aR=TRK4ZGfDRw@mail.gmail.com>

Thank you guys for all the info.

Twisted Web is cool, but there's it's not as interchangeable as WSGI. You
can use whatever framework django/flask/webpy/bottle you like and plug it
in as long as it's PEP-333 compatible. Not so much for Twisted and
Tornado's own ioloop engine.

IPython.parallel

http://ipython.org/ipython-doc/stable/install/install.html#dependencies-for-ipython-parallel-parallel-computing

It's based on ZeroMQ(PyZMQ), and the `ssh` command. I don't think that's
lightweigh enough for busy web clusters.

By QMachine I assume that's

https://github.com/wilkinson/qmachine

For web server cluster it's really not a good idea to amplify HTTP
requests. One client request amplifies several other HTTP requests on
server clusters.

What I propose is something like Zed Shawn's Mongrel2 project (
http://mongrel2.org/), use a very lightweight server-side serialization
protocol as cluster IPC, you can pass states/data between nodes (workers)
easily. It should be agnostic to framework or libraries, the objective is
to unite python modules in the realtime web world. Because for
request-response web world, a synchronized gateway like WSGI is good
enough, between each requests, share
nothing<https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale>
.

But for realtime web, server side state is very much required. There need
to be a fd pool for DBs, external services, and stuff like Server-Side-Push
technologies.

Let's assume the following scenario:

One user submits a blog, his follower gets browser/iOS/Android push
notification. Because users are connected different nodes in one big
cluster, we need some kind of mechanism to broadcast this message.

In such an architecture we can write simpler code like this:

from django.db.models.signals import post_save

@receiver(post_save, sender=BlogPostModel)
def my_handler(sender, **kwargs):
    msg = "User X just posted a new blog, check it out at http://..."
    browser_followers.send(msg)
    ios_followers.send(msg)
    android_followers.send(msg)

Currently this library reall shines.

https://pypi.python.org/pypi/telegraphy/

Telegraphy architecture is like this:

[image: Inline image 1]

What I propose is to merge Web-app part and the AutobahnPython Gateway part
into *one* based on a community honored standard.





On Mon, Mar 17, 2014 at 8:24 PM, anatoly techtonik <techtonik at gmail.com>wrote:

> On Mon, Mar 17, 2014 at 3:53 AM, est <electronixtar at gmail.com> wrote:
> > The best we've got is something like Celery, or Telegraphy for Django
>
> Does IPython suits pattern better?
> http://ipython.org/ipython-doc/stable/parallel/parallel_intro.html
>
> > What do you guys think?
>
> I think not to reinvent the bicycle. Use PythonJS + QMachine to get to
> web client side + distributed processing without abandoning existing
> Python skills.
> --
> anatoly t.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140318/2a4786b8/attachment.html>

From techtonik at gmail.com  Tue Mar 18 08:12:49 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 18 Mar 2014 10:12:49 +0300
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <CA+=kK2JAsfi6dAW0eKJhukFz2rMPctVLA3_N4aR=TRK4ZGfDRw@mail.gmail.com>
References: <CA+=kK2KqfjdA6VG+tEY5xiA3S-A6rJvL7CfxFxRZkh_yw8pEJA@mail.gmail.com>
 <CAPkN8xKogtBVEhAO8CNpvo3pn40h=RuW9FUcEB9YG7aUqpkVvw@mail.gmail.com>
 <CA+=kK2JAsfi6dAW0eKJhukFz2rMPctVLA3_N4aR=TRK4ZGfDRw@mail.gmail.com>
Message-ID: <CAPkN8xKO+RrEB9MbHtBhW52jPt3sjzNQ=rXqevnkahNE6Eq8cg@mail.gmail.com>

On Tue, Mar 18, 2014 at 5:16 AM, est <electronixtar at gmail.com> wrote:

>
> IPython.parallel
>
>
> http://ipython.org/ipython-doc/stable/install/install.html#dependencies-for-ipython-parallel-parallel-computing
>
> It's based on ZeroMQ(PyZMQ), and the `ssh` command. I don't think that's
> lightweigh enough for busy web clusters.
>

You will need to secure you web cluster computations anyway. SSH may be
slower that HTTPS, I agree, but I'd still see the benchmarks. IPython is
good for handling long processing tasks. For myriad of tiny code+data
workers I'd choose Stackless. Not sure about the web server part.


> By QMachine I assume that's
>
> https://github.com/wilkinson/qmachine
>
> For web server cluster it's really not a good idea to amplify HTTP
> requests. One client request amplifies several other HTTP requests on
> server clusters.
>

Right. Because your workers are not trusted you need to distribute the load
and validate results with multiple passes.


> What I propose is something like Zed Shawn's Mongrel2 project (
> http://mongrel2.org/), use a very lightweight server-side serialization
> protocol as cluster IPC, you can pass states/data between nodes (workers)
> easily. It should be agnostic to framework or libraries, the objective is
> to unite python modules in the realtime web world. Because for
> request-response web world, a synchronized gateway like WSGI is good
> enough, between each requests, share nothing<https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale>
> .
>
> But for realtime web, server side state is very much required. There need
> to be a fd pool for DBs, external services, and stuff like Server-Side-Push
> technologies.
>

"realtime web" is a very broad term. Need a more concise definition. I see
only one difference in "web" over standard protocol - is that client is
limited to send operations only and requests to HTTP(S) protocol only. Is
that true? All other parts of the system can communicate with whatever
protocols they like.

So, to unify the network under some standard, we need common base. Stick to
limitations of client to make all nodes work the same. Limit choice to bare
minimum and extend where it is needed.

Let's assume the following scenario:
>
> One user submits a blog, his follower gets browser/iOS/Android push
> notification. Because users are connected different nodes in one big
> cluster, we need some kind of mechanism to broadcast this message.
>
> In such an architecture we can write simpler code like this:
>
> from django.db.models.signals import post_save
>
> @receiver(post_save, sender=BlogPostModel)
> def my_handler(sender, **kwargs):
>     msg = "User X just posted a new blog, check it out at http://..."
>     browser_followers.send(msg)
>     ios_followers.send(msg)
>     android_followers.send(msg)
>
> Currently this library reall shines.
>
> https://pypi.python.org/pypi/telegraphy/
>
> Telegraphy architecture is like this:
>
> [image: Inline image 1]
>
> What I propose is to merge Web-app part and the AutobahnPython Gateway
> part into *one* based on a community honored standard.
>

Just a side note - XML-RPC is a bad way of operation and I am going to
promote that belief.

The key component here that is not depicted is client limitations (able to
only request events, and accept events after websocket connection is
established with a single server). Channel description (WS, HTTP) are not
informative in this regard to capture that limitation that this
architecture should deal with.

When client (browser) establishes connection to HTTP site, can it open a
websocket to the site in other domain? If no - then cross-domain
interaction should also be included into problem description before
unifying Django and Autobahn. If this limitation exists - the generic
clustering problem will include management DNS infrastructure (to make sure
client can send requests to any node in the cluster) or clustering will
require frontends on servers to reroute requests on established websocket
connections to appropriate cluster nodes.

Not sure I got the positioning of NodeJS cluster right, so feel free to fix
that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140318/78c3668e/attachment-0001.html>

From electronixtar at gmail.com  Tue Mar 18 09:20:06 2014
From: electronixtar at gmail.com (est)
Date: Tue, 18 Mar 2014 16:20:06 +0800
Subject: [Web-SIG] Nodejs cluster
Message-ID: <CA+=kK2+pAf79MxGCgFQfAPnskwOGysqjQNuXGDvV9TOxEB_Qow@mail.gmail.com>

> is that client is limited to send operations only and requests to HTTP(S)
protocol only. Is that true? All other parts of the system can communicate
with whatever protocols they like.

Yes! Every incoming requests are HTTP.

There also need to be a long running process to hold all the fd pools for
DB connection, server side events (text/event-stream) connections, iOS APNS
push conn pools, XMPP connection, etc. And there may be multiple nodes
holding these fd for long opening

Example:

 * client HTTP requests are handled and finished on NodeA
 * NodeA notifies NodeB for events, NodeB gather the event for statistical
analystis by writing some data to DB
 * also NodeA notifies NodeC to Server-Side-Events to some other connection
 * next NodeA notifies NodeC to send out an  iOS push message
 * last NodeA notify NodeD on which sends out an XMPP message.

Stuff goes like that.

> XML-RPC is a bad way of operation and I am going to promote that belief.

Completely agree. But currently Telegraphy is one and only one-of-a-kind.

Other options like brubeck.io with mongrel2, webalchemy, etc.

> the generic clustering problem will include management DNS infrastructure

For a running cluster, yes. But Nodejs has its cluster batteries included
so it's easier to bootstrap a cluster in that way than say, build with
multiprocessing/billiard from scratch



On Tue, Mar 18, 2014 at 3:12 PM, anatoly techtonik <techtonik at gmail.com>wrote:

> On Tue, Mar 18, 2014 at 5:16 AM, est <electronixtar at gmail.com> wrote:
>
>>
>> IPython.parallel
>>
>>
>> http://ipython.org/ipython-doc/stable/install/install.html#dependencies-for-ipython-parallel-parallel-computing
>>
>> It's based on ZeroMQ(PyZMQ), and the `ssh` command. I don't think that's
>> lightweigh enough for busy web clusters.
>>
>
> You will need to secure you web cluster computations anyway. SSH may be
> slower that HTTPS, I agree, but I'd still see the benchmarks. IPython is
> good for handling long processing tasks. For myriad of tiny code+data
> workers I'd choose Stackless. Not sure about the web server part.
>
>
>> By QMachine I assume that's
>>
>> https://github.com/wilkinson/qmachine
>>
>> For web server cluster it's really not a good idea to amplify HTTP
>> requests. One client request amplifies several other HTTP requests on
>> server clusters.
>>
>
> Right. Because your workers are not trusted you need to distribute the
> load and validate results with multiple passes.
>
>
>> What I propose is something like Zed Shawn's Mongrel2 project (
>> http://mongrel2.org/), use a very lightweight server-side serialization
>> protocol as cluster IPC, you can pass states/data between nodes (workers)
>> easily. It should be agnostic to framework or libraries, the objective is
>> to unite python modules in the realtime web world. Because for
>> request-response web world, a synchronized gateway like WSGI is good
>> enough, between each requests, share nothing<https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale>
>> .
>>
>> But for realtime web, server side state is very much required. There need
>> to be a fd pool for DBs, external services, and stuff like Server-Side-Push
>> technologies.
>>
>
> "realtime web" is a very broad term. Need a more concise definition. I see
> only one difference in "web" over standard protocol - is that client is
> limited to send operations only and requests to HTTP(S) protocol only. Is
> that true? All other parts of the system can communicate with whatever
> protocols they like.
>
> So, to unify the network under some standard, we need common base. Stick
> to limitations of client to make all nodes work the same. Limit choice to
> bare minimum and extend where it is needed.
>
> Let's assume the following scenario:
>>
>> One user submits a blog, his follower gets browser/iOS/Android push
>> notification. Because users are connected different nodes in one big
>> cluster, we need some kind of mechanism to broadcast this message.
>>
>> In such an architecture we can write simpler code like this:
>>
>> from django.db.models.signals import post_save
>>
>> @receiver(post_save, sender=BlogPostModel)
>> def my_handler(sender, **kwargs):
>>     msg = "User X just posted a new blog, check it out at http://..."
>>     browser_followers.send(msg)
>>     ios_followers.send(msg)
>>     android_followers.send(msg)
>>
>> Currently this library reall shines.
>>
>> https://pypi.python.org/pypi/telegraphy/
>>
>> Telegraphy architecture is like this:
>>
>> [image: Inline image 1]
>>
>> What I propose is to merge Web-app part and the AutobahnPython Gateway
>> part into *one* based on a community honored standard.
>>
>
> Just a side note - XML-RPC is a bad way of operation and I am going to
> promote that belief.
>
> The key component here that is not depicted is client limitations (able to
> only request events, and accept events after websocket connection is
> established with a single server). Channel description (WS, HTTP) are not
> informative in this regard to capture that limitation that this
> architecture should deal with.
>
> When client (browser) establishes connection to HTTP site, can it open a
> websocket to the site in other domain? If no - then cross-domain
> interaction should also be included into problem description before
> unifying Django and Autobahn. If this limitation exists - the generic
> clustering problem will include management DNS infrastructure (to make sure
> client can send requests to any node in the cluster) or clustering will
> require frontends on servers to reroute requests on established websocket
> connections to appropriate cluster nodes.
>
> Not sure I got the positioning of NodeJS cluster right, so feel free to
> fix that.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140318/2206e1bc/attachment.html>

From exarkun at twistedmatrix.com  Tue Mar 18 13:37:10 2014
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Tue, 18 Mar 2014 12:37:10 -0000
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <mailman.41829.1395126794.18129.web-sig@python.org>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
Message-ID: <20140318123710.6218.1600151976.divmod.xquotient.314@top>

>From: est <electronixtar at gmail.com>
>
>Thank you guys for all the info.
>
>Twisted Web is cool, but there's it's not as interchangeable as WSGI.

Twisted Web includes a WSGI container.  You can run any WSGI application 
in it.

Jean-Paul

From chris at simplistix.co.uk  Tue Mar 18 14:03:32 2014
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 18 Mar 2014 13:03:32 +0000
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <20140318123710.6218.1600151976.divmod.xquotient.314@top>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
Message-ID: <53284424.5090708@simplistix.co.uk>

On 18/03/2014 12:37, exarkun at twistedmatrix.com wrote:
>> Twisted Web is cool, but there's it's not as interchangeable as WSGI.
>
> Twisted Web includes a WSGI container.  You can run any WSGI application
> in it.

How easy is it to get messages from a twisted-run websocket (say 
autobahn) into and from a wsgi app?

(I'm not sure my question makes 100% sense, but thought I'd throw it out 
there anyway...)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From electronixtar at gmail.com  Tue Mar 18 14:12:53 2014
From: electronixtar at gmail.com (est)
Date: Tue, 18 Mar 2014 21:12:53 +0800
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <20140318123710.6218.1600151976.divmod.xquotient.314@top>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
Message-ID: <CA+=kK2KPfwkmi95vBs0rt+r0cWTmbK=uuv3wSQm42=yWkyqB-A@mail.gmail.com>

Yes, I am aware of Tornado and Twisted Web has WSGI container capabilities.
The only missing piece is the cluster capability which is that the thread
is about.  :D




On Tue, Mar 18, 2014 at 8:37 PM, <exarkun at twistedmatrix.com> wrote:

>  From: est <electronixtar at gmail.com>
>>
>>
>> Thank you guys for all the info.
>>
>> Twisted Web is cool, but there's it's not as interchangeable as WSGI.
>>
>
> Twisted Web includes a WSGI container.  You can run any WSGI application
> in it.
>
>
> Jean-Paul
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: https://mail.python.org/mailman/options/web-sig/
> electronixtar%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140318/812e8f92/attachment.html>

From roberto at unbit.it  Tue Mar 18 14:28:34 2014
From: roberto at unbit.it (Roberto De Ioris)
Date: Tue, 18 Mar 2014 14:28:34 +0100
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <CA+=kK2KPfwkmi95vBs0rt+r0cWTmbK=uuv3wSQm42=yWkyqB-A@mail.gmail.com>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
 <CA+=kK2KPfwkmi95vBs0rt+r0cWTmbK=uuv3wSQm42=yWkyqB-A@mail.gmail.com>
Message-ID: <ec9c1582e78045c4165b925192b23ac9.squirrel@manage.unbit.it>


> Yes, I am aware of Tornado and Twisted Web has WSGI container
> capabilities.
> The only missing piece is the cluster capability which is that the thread
> is about.  :D
>

Honestly to me nodejs cluster looks no more than a simple master + workers
model (already used by gunicorn, uWSGI and many others)

Why is it special ?

-- 
Roberto De Ioris
http://unbit.it

From tobias.oberstein at gmail.com  Wed Mar 19 00:28:14 2014
From: tobias.oberstein at gmail.com (Tobias Oberstein)
Date: Tue, 18 Mar 2014 23:28:14 +0000 (UTC)
Subject: [Web-SIG] Nodejs cluster
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
 <53284424.5090708@simplistix.co.uk>
Message-ID: <loom.20140319T002525-185@post.gmane.org>

Chris Withers <chris at ...> writes:

> 
> On 18/03/2014 12:37, exarkun at ... wrote:
> >> Twisted Web is cool, but there's it's not as interchangeable as WSGI.
> >
> > Twisted Web includes a WSGI container.  You can run any WSGI application
> > in it.
> 
> How easy is it to get messages from a twisted-run websocket (say 
> autobahn) into and from a wsgi app?

Here is how you run Twisted Web + WSGI container + Flask + Autobahn
all on 1
port:

https://github.com/tavendo/AutobahnPython/
blob/master/examples/twisted/websocket/echo_wsgi/server.py

If you crosslink the Flask app object and the WebSocket factory with
each other you can shuffle data in both directions.

/Tobias

> 
> (I'm not sure my question makes 100% sense, but thought I'd throw it out 
> there anyway...)
> 
> Chris
> 





From electronixtar at gmail.com  Wed Mar 19 03:07:37 2014
From: electronixtar at gmail.com (est)
Date: Wed, 19 Mar 2014 10:07:37 +0800
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <loom.20140319T002525-185@post.gmane.org>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
 <53284424.5090708@simplistix.co.uk>
 <loom.20140319T002525-185@post.gmane.org>
Message-ID: <CA+=kK2JQ4zGOZYhtmEwMa8YLag-5A4+H1P+0UHU2d69om2pSfQ@mail.gmail.com>

> nodejs cluster looks no more than a simple master + workers
model

Yes, NodeJS cluster is simple, But IMHO it's the gravity which unifies the
Nodejs ecosystem. While in Python world we only have WSGI and everyone else
has its own cluster model. (Please correct me if I am wrong!)

btw uWSGI is awesome with its RPC capabilities thanks for all the work!

> If you crosslink the Flask app object and the WebSocket factory with
each other you can shuffle data in both directions.

That's super cool. Thanks!

What I propose is officially Python endorsed cluster standard. Am I over
thinking this? Or the status quo is good enough for everyone else?



On Wed, Mar 19, 2014 at 7:28 AM, Tobias Oberstein <
tobias.oberstein at gmail.com> wrote:

> Chris Withers <chris at ...> writes:
>
> >
> > On 18/03/2014 12:37, exarkun at ... wrote:
> > >> Twisted Web is cool, but there's it's not as interchangeable as WSGI.
> > >
> > > Twisted Web includes a WSGI container.  You can run any WSGI
> application
> > > in it.
> >
> > How easy is it to get messages from a twisted-run websocket (say
> > autobahn) into and from a wsgi app?
>
> Here is how you run Twisted Web + WSGI container + Flask + Autobahn
> all on 1
> port:
>
> https://github.com/tavendo/AutobahnPython/
> blob/master/examples/twisted/websocket/echo_wsgi/server.py
>
> If you crosslink the Flask app object and the WebSocket factory with
> each other you can shuffle data in both directions.
>
> /Tobias
>
> >
> > (I'm not sure my question makes 100% sense, but thought I'd throw it out
> > there anyway...)
> >
> > Chris
> >
>
>
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> https://mail.python.org/mailman/options/web-sig/electronixtar%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140319/92a5f783/attachment.html>

From roberto at unbit.it  Wed Mar 19 07:31:20 2014
From: roberto at unbit.it (Roberto De Ioris)
Date: Wed, 19 Mar 2014 07:31:20 +0100
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <CA+=kK2JQ4zGOZYhtmEwMa8YLag-5A4+H1P+0UHU2d69om2pSfQ@mail.gmail.com>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
 <53284424.5090708@simplistix.co.uk>
 <loom.20140319T002525-185@post.gmane.org>
 <CA+=kK2JQ4zGOZYhtmEwMa8YLag-5A4+H1P+0UHU2d69om2pSfQ@mail.gmail.com>
Message-ID: <bfe256acff20bcaa1d229ae09212f493.squirrel@manage.unbit.it>


>> nodejs cluster looks no more than a simple master + workers
> model
>
> Yes, NodeJS cluster is simple, But IMHO it's the gravity which unifies the
> Nodejs ecosystem. While in Python world we only have WSGI and everyone
> else
> has its own cluster model. (Please correct me if I am wrong!)

Completely agree with the "unified gravity" (even if it is not necessarily
a good thing), completely disagree with the "cluster model". It is a
standard way available in all languages, and projects like apache uses it
by ages (and it is described in all of the ancient UNIX books out-there).

They did not invent it, and they did not solved anything that was not
solved 30 years ago :) Node is successfull because all of its third party
modules obey to its non-blocking way. This does not happen with Python
(and perl and ruby and ...) because they all born in a process/thread age.
Maybe something will change with asyncio, but i am pretty sure it will be
a really looooong (and hard) way. A way that not all of the people want to
follow (personally i do not like callback-based programming and i do not
think i will change idea too soon ;)


>
> What I propose is officially Python endorsed cluster standard. Am I over
> thinking this? Or the status quo is good enough for everyone else?
>

personally i think you are giving to this "little" thing too much
attention, the problems with clustering are at another level (sharing
structures and data between nodes comes first), and most of them still do
not have a solid solution


-- 
Roberto De Ioris
http://unbit.it

From electronixtar at gmail.com  Wed Mar 19 09:38:54 2014
From: electronixtar at gmail.com (est)
Date: Wed, 19 Mar 2014 16:38:54 +0800
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <bfe256acff20bcaa1d229ae09212f493.squirrel@manage.unbit.it>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
 <53284424.5090708@simplistix.co.uk>
 <loom.20140319T002525-185@post.gmane.org>
 <CA+=kK2JQ4zGOZYhtmEwMa8YLag-5A4+H1P+0UHU2d69om2pSfQ@mail.gmail.com>
 <bfe256acff20bcaa1d229ae09212f493.squirrel@manage.unbit.it>
Message-ID: <CA+=kK2+Ayx-MV=JObL-DC2EA5=nMuKxByxN0PdtNTcP_TESK-Q@mail.gmail.com>

> sharing structures and data between nodes comes first

Must agree with this. How many people out there use Redis as de facto
semaphores and event synchronizing mechanism? (Think of Resque and all its
glorious spin-offs)



On Wed, Mar 19, 2014 at 2:31 PM, Roberto De Ioris <roberto at unbit.it> wrote:

>
> >> nodejs cluster looks no more than a simple master + workers
> > model
> >
> > Yes, NodeJS cluster is simple, But IMHO it's the gravity which unifies
> the
> > Nodejs ecosystem. While in Python world we only have WSGI and everyone
> > else
> > has its own cluster model. (Please correct me if I am wrong!)
>
> Completely agree with the "unified gravity" (even if it is not necessarily
> a good thing), completely disagree with the "cluster model". It is a
> standard way available in all languages, and projects like apache uses it
> by ages (and it is described in all of the ancient UNIX books out-there).
>
> They did not invent it, and they did not solved anything that was not
> solved 30 years ago :) Node is successfull because all of its third party
> modules obey to its non-blocking way. This does not happen with Python
> (and perl and ruby and ...) because they all born in a process/thread age.
> Maybe something will change with asyncio, but i am pretty sure it will be
> a really looooong (and hard) way. A way that not all of the people want to
> follow (personally i do not like callback-based programming and i do not
> think i will change idea too soon ;)
>
>
> >
> > What I propose is officially Python endorsed cluster standard. Am I over
> > thinking this? Or the status quo is good enough for everyone else?
> >
>
> personally i think you are giving to this "little" thing too much
> attention, the problems with clustering are at another level (sharing
> structures and data between nodes comes first), and most of them still do
> not have a solid solution
>
>
> --
> Roberto De Ioris
> http://unbit.it
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140319/fbf9ef45/attachment.html>

From tobias.oberstein at tavendo.de  Wed Mar 19 10:13:00 2014
From: tobias.oberstein at tavendo.de (Tobias Oberstein)
Date: Wed, 19 Mar 2014 09:13:00 -0000
Subject: [Web-SIG] Nodejs cluster
In-Reply-To: <CA+=kK2JQ4zGOZYhtmEwMa8YLag-5A4+H1P+0UHU2d69om2pSfQ@mail.gmail.com>
References: <mailman.41829.1395126794.18129.web-sig@python.org>
 <20140318123710.6218.1600151976.divmod.xquotient.314@top>
 <53284424.5090708@simplistix.co.uk>
 <loom.20140319T002525-185@post.gmane.org>
 <CA+=kK2JQ4zGOZYhtmEwMa8YLag-5A4+H1P+0UHU2d69om2pSfQ@mail.gmail.com>
Message-ID: <634914A010D0B943A035D226786325D44473AC4860@EXVMBX020-12.exch020.serverdata.net>

>>?If you crosslink the Flask app object and the WebSocket factory with
>>each other you can shuffle data in both directions.
>That's super cool. Thanks!
>What I propose is officially Python endorsed cluster standard. Am I over thinking this? Or the status quo is good enough for everyone else?

Here is a proof-of-concept we did last year scaling Twisted Web on multicore:

https://github.com/oberstet/scratchbox/tree/master/python/twisted/sharedsocket

It reaches >60k requests/sec on 4 cores (something like 70% the performance of Nginx on the same system):

https://github.com/oberstet/scratchbox/raw/master/python/twisted/sharedsocket/results.pdf

We are working on a system (on top of Autobahn) which provides builtin scale-up (multi-core) and scale-out (multi-node) capabilities:

https://github.com/crossbario/crossbar

This is work in progress and relies on WAMPv2. Here are a couple of links

https://github.com/tavendo/WAMP/tree/master/spec

https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/wamp/basic

https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/websocket/echo_multicore

As said, Crossbar.io is work in progress but we'll announce something soon as it's already quite usable (eg it can host any WSGI app side-by-side with WebSocket/WAMP application components) ..

Cheers,
/Tobias