Socket to me

[Andrew Bennetts]

On Sat, May 31, 2003 at 05:21:44AM -0700, Mark Carter wrote:
> I'm trying to get my head around sockets/distributed computing - with
> some,
> albeit limited, 
> success. There was a thread on this board, dating from about beginning
> of April, which mentioned the following python-friendly technologies:
> 
> Yami - http://www.maciejsobczak.com/prog/yami/
> Spread - http://www.python.org/other/spread/
> OSE - http://ose.sourceforge.net
> Pyro - http://pyro.sourceforge.net
> 
> I have also heard about Twisted.
> 
> More are mentioned at Google:
> http://directory.google.com/Top/Computers/Programming/Languages/Python/Modules/Distributed_Computing/
> 
> 
> I'm a bit confused as to what they're all trying to achieve. I suspect
> that there's a fair bit of overlap. 

They're not trying to achieve anything on their own -- they're just
frameworks.  They're there to help *you* achieve something :)

> Would I be right in saying that 99 times out of 100, these things are
> using sockets in order to communicate across machines?

Um, I guess so.  That's an odd way to put it.  Are you asking about the
design of these frameworks, or what sockets are for?

> Am I right in thinking that the major considerations in writing servers is
> how one computer "knows" about another computer, and how to spawn threads
> so that multiple requests can be handled?

Not really -- there are other major considerations, and your assumption that
threads are required is mistaken (Twisted, for instance, can quite happily
serve lots of clients "simutaneously" with just the one thread).

> And would it be right to say that the purpose of "middleware" software
> is to handle all these details?

Middleware is one of those terms that means what people want it to mean.  So
if that's what you want, then, yes <wink>

> And once you know how to link in with this software, it basically removes
> the need for you to think about how all this glueing-together is supposed
> work, and lets you concentrate on coding what it actually is that you're
> trying to achieve?

Ok, it sounds like you're interested in "remote object" protocols or
"distributed computing" more than the nuts and bolts of sockets.  But yes,
the idea of a distributed object framework (I'm intentionally mixing
terminology to confuse you <wink>) is to allow you to focus on writing your
business logic without getting overly bogged down in reinventing a the bits
that make it work in a distributed environment.

> Would it also be fair to say that some projects seem to "kitchen-sink" the
> services they provide? I've noticed Twisted, for example, shows one how to
> construct a little echo server and client. But it seems to do a whole
> bunch of other stuff, whose purpose I have yet to understand.  Don't get
> me wrong - I'm impressed by the cleverness of the guys that wrote Twisted.

Well, Twisted does a lot of stuff, and it helps to know which bits you want.
Twisted supports low-level things like writing your own protocol
implementation (which is what the echo server/client example is about).  It
also has Perspective Broker (PB), a remote object protocol in the
twisted.spread package, when you just want to be able to pass objects around
and call methods, and not worry about the details of the transport.
Unfortunately, Twisted can't decide which one you need...

Perhaps you should look at Perspective Broker docs at
    http://twistedmatrix.com/documents/howto/
and see if it looks like it suits your problem?

> And following on from that, do I detect a certain element of "not invented
> here"? Perhaps a micro, rather than a monolithic approach to distributed
> computing would have been better - giving all of the benefits of ... well,
> I'm sure we all know the drill about modular design.  What we want is a
> layered cake, not alphabet soup.

For what it's worth, Twisted has more layers of abstraction that you might
expect -- See the diagram at
http://twistedmatrix.com/documents/howto/overview for a hint of the layers
involved (there's more than meets the eye!).  We've been accused of being
"abstraction astronauts" :)

So, the way to answer your question is to find out: what are you trying to
do, exactly?  What problem do you want to solve?  Hopefully if you can tell
us that, this group can give you some specific recommendations.

There are, as you've noticed, a bewildering array of techologies out there
-- and you haven't touched upon XML-RPC, SOAP, CORBA, HTTP,
IP-over-avian-carrier... there's a lot of stuff out there.

Some questions to help you narrow down your search:
  - Do you provide the software running on all the distributed nodes in your
    system?
       This matters, because if you do, you can use something like PB, which
       is very convenient when all the systems all using your Twisted-based
       software.
       If interoperating with other systems is important, it might be better
       to focus on more neutral frameworks, like SOAP or CORBA, or even just
       "plain" HTTP...

  - Does size matter -- i.e., do you require that network traffic is kept to
    a minimum because you need high volume on potentially slow links (like
    modem users)?
       For most systems this doesn't matter much.  Clients make an
       occasional request, and servers send back a response.  In other
       situations, you want to be streaming stock market data to clients as
       fast as it arrives, and being "up to the second" is vitally
       important.  Technologies like XML-RPC and SOAP use XML, and so tend
       to be a little bloated, and probably need to be avoided in those
       situations (their plus side is they can make it much easier for
       someone else to implement a client to interoperate with your system).

  - Is there an existing solution already?
       It sounds like a dumb question, but depending on how you deploy it, a
       chat system like Jabber or a humble web server may be just the
       distributed system you need.

That's just off the top of my head; it's hard to know what you really need,
because your questions were so general (which is fine; you're obviously
trying to get a grip on the area and don't know where to start).  I hope
I've helped -- let me know if anything is unclear or you want me to
elaborate on something.

But, some advance warning: as a Twisted developer, I will almost certainly
be compelled to recommend Twisted as being exactly what you need <wink>.

-Andrew.