App implementation help needed

[Joseph A Knapka]

Dave Swegen wrote:
> 
> I'm currently designing the second part of a little pet project of mine,
> and run into a wall regarding how to proceed, and would much appreciate
> some assitance in figuring out in which direction I should go.
> 
> The setup is as follows:
> 
> A daemon has a queue of times when it should perform an action that
> involves fetching some data from a web site.
> 
> These times, and associated data (which end up customising the URL), is
> to be provided by users via a web interface.
> 
> The user provided events, as well as some user data (username, password)
> need to be stored.
> 
> The program will in the first instance be running on Linux, but if for
> no extra cost I can make it run on other platforms that would be nice
> (but not at the cost of simplicity).
> 
> All pretty simple stuff in itself. My problem is how to tie together the
> different parts (web interface, updating queues, calling a certain function
> at a certain time with certain user provided data).
> 
> Are threads the best/easiest way to achieve this? Are fork()s sufficient?
> Or would having the web interface based on asyncore allow me to avoid
> using threads/forking entirely?
> 
> Is using the Queue module appropriate, or would a list be sufficient?
> 
> Would using sleep(1) in a loop be sufficient (I need second precision,
> so cron is not really an option), or would sched be more appropriate?
> 
> I don't want to use a full DB, so should I go with pickling or anydbm?
> 
> So, lots of questions, probably not enough background information, and
> somebody who is rather overwhelmed by the options avaliable.
> 
> I'm hoping to keep this as simple as possible, so please, be gentle :)
> 
> Any help much appreciated.

If I were doing this, and I wanted to keep it extremely simple, I
would do it like this:

(0) Create a special directory, /workdir. This is a blessed place
that's used for communication between the web interface and the
daemon.

(1) The web interface: a simple Python CGI script that essentially
does this (forgive the silly fn names; I don't remember how
to spell the file operations at present):

def handleFormData():
  queEntries = processFormData()

  tempfile = makeATemporaryFilename()

  f = open(tempfile,'w')

  f.write(queEntries) # Maybe pickle, but no need if the queue
  # entries are simple things that can be eval'd.

  f.close()

  moveTempfileToWorkdir()

(2) The queue handler: a simple Python program that maintains
the event list:

def mainLoop():
   sleep(for a little while) # Maybe use socket.select() if you
   # want really brief sleeps.

   processEvents() # Kick off any events that have expired.

   checkWorkDir()

def checkWorkDir():

   files = listFilesInWorkdirByCreationDate()
   for file in files:
      qes = readQueueEntriesFrom(file)
      addToEventList(qes)
      delete(file)

It's important that we create the /workdir entries as
temp files elsewhere and then move them to /workdir;
that way the daemon never sees a half-written work file.

You could do checkWorkDir() and processEvents() in separate
threads, but that would be less simple!

The communication mechanism (files in a blessed work
directory) is lame, but should work pretty much anywhere.
Porting such a thing to EvilEmpireOS should be a snap
(just change the work directory name).

I'll be interested to see other proposals.

Cheers,

-- Joe