[Baypiggies] I need some help architecting the big picture

[Eric Walstad]

Hey JJ,

On Mon, Apr 28, 2008 at 9:57 PM, Shannon -jj Behrens <jjinux at gmail.com> wrote:
...
>  > Where in the process does this operator sit? Does he/she receive
>  > the batches from the customers and then feed them to your toolchain and
>  > verify that the batches made it to the database, or something else entirely?
>
>  That's the problem.  I don't know.  I've never been in a situation
>  where my user wasn't on the other side of a Web interface.  For the
>  foreseeable future, the operator will be me or some sysadmin.  My
>  guess is that he'll get the data via scp either manually or by cron
>  job.  Now, I have to figure out how to feed the data to the system.
>  Do I simply put it in some place and say "go!"?  I was guessing
>  someone else had been in a similar situation and had some best
>  practices to recommend.
We wrote a python 'sentinel', started periodically by cron, that
watches user's incoming scp directories for new files.  We considered
writing it as a daemon but the first iteration (running it from cron)
turned out to suit our needs.  When a new file arrives in the
directory, the sentinel does something that it is configured to do:
call a python callable, execute a system executable, etc.  The
sentinel only operates on a file that isn't already being operated on
(in case the callable's runtime exceeds the sentinel's nap time).  The
sentinel can be configured to do a post-process task which usually
includes moving the uploaded file to a 'processed' directory.  The
sentinel operates in a generic way by reading config files that define
how it is supposed to behave.  Each of our customers has a Sentinel
config file describing how to tell when that customer's files arrive,
how to process their file and what to do when the processing is done.
I don't know about best practices here, but our system is pretty
generic, flexible and works well for us.

Eric.