web interface to c program running on different server

[François Pinard]

[Lee]

> There is a piece of software being developed in C for performing
> various scientific calculations on various types of data.  I have been
> asked to investigate the ways in which this could be made into a web
> application - specifically, a portal.

I do not understand what "portal" means.  For me, who does not speak English,
I only learned this word with the PDP-10 series, as meaning an entry point
within an execute-only memory page.

> Reimplementing the program in another language is not one I wish to
> investigate, as I believe the software has gone through months of
> testing already.

Not so long ago, I had something to do which was (probably) similar.
We had a rather complex application in C that was meant to be used in
batch mode on massive input, delivering massive output.  We wanted to
include it in a Web application for processing one set of input at a time.
The problem was that this application had a big start-up time really, as
it preconditioned itself a great deal, studying databases and precompiling
scenarios, for later very speedy processing.

> One solution I understand would be to create an interface to the
> software using Zope & XML-RPC, and then using Zope Content Management
> Framework to create a portal.  Of course, there may be hundreds(?) of
> ways to arrive at the desired results - using e.g.  ASP, PHP, perl.

I spent some time studying many avenues, and experimenting some of them.
Finally, I opted for something very simple, clean, and quick to implement.

* We added a program call option to the big application telling if it was
  used through a server, instead of in batch.  With that option, the
  C program had to produce a recognisable delimiter on its output to
  mean End-of-Result and then flush all its output.  That option was
  also making the program more lenient than drastic about a few errors.
  This application's input and output are line oriented.

* I wrote a small server in Python, to be run on the same machine the big
  application was.  That server essentially listens to a socket, decodes
  the request and produces a fragment of input for the big application.
  It then starts the big application if not already (the Web application
  has to wait a while for its very first request), it also restarts it
  while logging an error if the application died.  It then reads
  the big application's output, but also its standard error (playing
  with select and a doing a few tricks) before packaging a result to return.

* On the Web service machine, I merely wrote another smallish Python module
  meant to receive a request from various CGI-scripts, package it for the
  server, establish a one-shot communication to read the analysis from
  the big application, and return the result to the scripts.  The socket
  communication between that Python module and the Python server is structure
  driven (not line driven), and I merely transit compressed pickled data
  between both.  Very simple to write, and effective in practice.

* The CGI-scripts are written as Python-active HTML pages.  For being able to
  directly embed Python code within HTML pages, I wrote a light and clean,
  yet powerful active templating system.  As we regroup functionalities
  within "import"able functions, Python code stays unobtrusive within HTML.
  I used two levels: the embedded code is rather pure Python, while the
  control of HTML generation with conditionals, loops and other facilities,
  merely uses Python-like concepts, in French, within special HTML comments.

All of this was surprisingly fast to implement.  However, once done, I spent
some time refining the last part of the project (the templating tool), so
it could be used in other projects of ours, as well.  Besides per-project
configurability, I added various facilities so HTML-writers could get more
extensive debugging feedback, better inclusion design, etc.  And merely to
please myself, I managed to clean out internals and to get some more speed:
not really that we needed it, but because I feel better that way. :-)

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard