Sending Python statement over socket in chunks

Jeffrey Barish jeff_barish at earthlink.net
Mon Feb 11 15:40:01 EST 2008


I have a python module that contains an assignment statement binding a long
list of things to a name:

list_of_things = [thing1, thing2, ...]

where each thing instantiates class Thing when executed.  I send this
statement through a socket to a remote module that executes it.  The
problem is that it takes too long to send the entire statement. 
Accordingly, I am thinking of a solution that sends the list in chunks. 
The client then executes each chunk and reassembles the complete list. 
Thus, I would first send something like:

chunk = [thing1, thing2, ... thing10]

and then

chunk = [thing11, thing12, ... thing20]

until all things have been transmitted.  At the client end, I would execute
each chunk statement and then do

list_of_things.append(chunk)

The point of this solution is that I can start doing useful work in the
client as soon as I receive the first chunk, and the others can arrive in
the background and be available by the time I need them.

One way I could implement this solution is to execute the statement for the
entire list_of_things in the server and then create chunk = [...]
statements with the lists filled using the repr of the class.  I believe
that this solution will work, but it seems a shame to execute the
list_of_things statement in the server rather than have the server stupidly
handle strings (because executing the statement takes time and because the
server currently doesn't understand "Thing").  Should I investigate using a
parser to carve up the list_of_things = [...] statement?  Basically, I just
need to identify each Thing(...) expression and then count out some number
of them.  I should be able to do that using re.  Or perhaps I should write
my own parser using Python string operations as all I need to do is count
out some number of Things delimited by "Thing(" at one end and "),\nThing("
at the other (or ")]\n" at the end of the list).  Did I just answer my own
question?

Of course, I need to balance the complexity of any alternative solution
against simply executing the statement on the server.
-- 
Jeffrey Barish




More information about the Python-list mailing list