ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

phil hunt wrote:
> On Wed, 03 Aug 2005 16:57:34 +0100, Michael Sparks <michaels at rd.bbc.co.uk>
> wrote: 
...
>>Which aside from other things means you can't build (say) a video
>>& SMIL playback system trivially, yet.
> 
> Isn't SMIL something that's goinhg to go into web browsers? In which
> case, you'd presumably not want to build one yourself, I imagine?

I was using SMIL as an example to be illustrative of a class of networked
applications that require visual support. A more direct example
would be MHEG (not mpeg) decoding and display systems, which
have similarities to SMIL systems in terms of capabilities required.

>>It's designed to make bolting things together to make these sorts of
>>system simpler and easier.
> 
> What you say "bolting things together" do you mean writing Python
> code? 

Yes.

> Or will there be some other way? 

A clicky pointy approach would be nice. (There's two sorts of interface in
the world IMO - "tappity tappity" and "clicky pointy" - if being equally
disdainful :) Unix shell systems are very "tappity tappity", and whilst I
can do "tappity tappity", and have been doing "tappity tappity" for my
entiring working life, I do prefer "clicky pointy" systems.)

Though building systems using AR Toolkit type systems would be the ideal.
( http://www.hitl.washington.edu/people/poup/research/ar.htm#artoolkit )
That's probably a couple of years off though (at least).

> What I have in mind is something like a project I worked on some
> time ago, where a program could be written by drawing boxes on a GUI
> application, and drawing lines (representing data flow) to connect
> the boxes. So one half of a VoIP application might look like:
> 
>   +----------+     +------------+     +-----------+
>   | listen   |     | convert to |     | volume    |
>   | on udp   |====>| audio      |====>| control + |
>   | port 600 |     | stream     |     | output    |
>   +----------+     +------------+     +-----------+

>From example 4:

pipeline(
    Multicast_transceiver("0.0.0.0", 1600, "224.168.2.9", 0),
    detuple(1),
    SRM_Receiver(),
    detuple(1),
    VorbisDecode(),
    AOAudioPlaybackAdaptor(),
).run()

Join multicast group "224.168.2.9", listen for packets on port 1600, throw
away information relating to who sent it, pass it through a simple reliable
multicast reordering subsystem, throw away the annotation, decode the
vorbis, and playback.

The visual representation of that isn't dissimilar to your diagram above.

> With something like this novel multimedia apps could be prototyped
> quickly (though making them into useful apps would take extra work
> -- in the case of a VoIP application you'd want a phonebook, for
> example).

Agreed.

>>At the same time it's designed to encourage
>>writing code in a way that makes it simpler.
> 
> Examples would be useful here. Both of what it can do now, and what
> it will be able to do eventually.

Examples of what can be done now are here:
   * http://kamaelia.sourceforge.net/KamaeliaStatus.html

I'll think about the latter part of that. (I personally loath technologies
that promise much and deliver little. Hence why I like to focus on what
we /can/ do)

>>The natural side effect of
>>this is the  system might make it easier to take advantage of multiple
>>CPU systems as they come online, since it makes a system naturally
>>concurrent. As the original announcement said "Kamaelia is designed as
>>a testbed". And by testbed I mean testbed as it testing out new ideas,
>>see if they work and see if they pan out. (Not as in a testing suite)
> 
> So what it will eventually do is not cast in stone?

The high level goals are to solve the problems in the Challenges document.
We're not particularly wedded to any one particular approach, and Kamaelia
can/should be moulded to solve those problems, not the other way round.

By definition along the way a variety of components will be generated, which
can obviously be put together in an arbitrary fashion, assuming type
adaptors as necessary, to form new interesting systems.

This ability to explore new/interesting systems quickly is something
generally of value to an R&D dept, so there's some people at BBC R&D
interested in using Kamaelia for all sorts of interesting projects. (which I
can't discuss here)

>>Components are object instances of python classes. The data passed
>>between components are object instances.
> 
> What sort of objects? 

Python objects. Numbers, strings, chunks of audio in a string, chunks of
audio in a binary object wrapped by a python class, etc. Precisely as I
said.

> For example, if an application is audio 
> straming, what sort of objects will be passed? 

That depends on the level to which you dive inside the audio being
streamed. If you're trying to be a content agnostic streamer then you
don't care what's inside the byte stream (from, say, a file) just what the
bit rate is you're supposed to send at. (for that you can just chop up the
file and send it chunks at a time)

If however you're dealing with variable bit rate data you may wish to look
inside the wrapper to figure out the where data blocks are for a given time
period in order to send those. This (for example in the case of sending MPEG
over RTP) can mean that you need to parse the data stream in order to send
the data. At that point it makes sense to send over a structured object
representing each chunk/block/frame (depending on codec/transport).

> Will there be onre 
> object passed for each 10 ms or so of sound, or will there be a
> continuously-set-up object that passes the data throught it a byte
> at a time?

Small messages turn out to be inefficient for what it's worth. (It's more
efficient to send bundles of small message or larger messages)

> And if it's written in Python, will it run fast enough?

I've seen overlay mesh networks implemented in TCL shipping 60Mbit/s
(sustained) in the past, which if we're implementing an overlay joining
multicast islands together would mean multicasting the equivalent of ~200
channels through each node. Scaling is not always about throughput of
course - Bit Torrent's initial implementation was also in python, and scales
very well.

If it doesn't scale sufficiently we can always optimise specific subsystems
when we need to rather than prematurely optimise. Also we have done a
naive (deliberately naive) translation of the ideas into C++, including
generators, so if things don't pan out we have a natural migration path else
where. (After knowing whether the overall approach is workable or not)

>       cdef class test(Axon.Component.component):
>             def mainBody(self):
>                   if self.dataReady("inbox"):
>                         data = self.recv("inbox")
>                         self.send(data, "outbox")
>                   return 1
> 
> My first impressions are that it seems obvious to me what this does:

A line of code is more often read than written, so if it's obvious, that's
good.

> it merely passes its input throught to its output 

Correct.

> (no doubt there are places where such a null component is useful).

An echo protocol is one example - a server you connect to that simply echoes
back to you whatever you send it. (an echo server can be useful for all
sorts of network testing in practice)

> Hopefully all code written with Kamaelia will be so clear.

See here for a longer example - which should show that generally speaking
things do tend to remain clear: http://tinyurl.com/dp8n7 (also shows how
you can incrementally develop new components).

> Now I have some questions.
> 
> 1. mainBody() is evidently being called at intervals. At what
> frequency are these intervals? For example, if the data stram is
> video data, will mainBody() get called once for each frame?

The scheduler is currently a dumb scheduler with essentially the following
logic:
   for thread in microthreads:
      thread.next()

As a result the time between calls is not guaranteed (nor can it be with
this sort of scheduler). The system does however allow for multiple
schedulers, so you could potentially have a realtime scheduler if it
was found to be necessary, at which point information about how a
component wished to be scheduled would become important.

> 2. what data type is (data) in the code? Or can it be any type?

Any data type python handles. (as indicated above)

In case you're wondering "why doesn't it do everything yet", I'll
just simply reiterate that we're at version 0.2, and we're getting
to the crawling/walking stage first before the walking/talking/running
stage :-)

Best Regards,


Michael.