[Baypiggies] I need some help architecting the big picture

[Shannon -jj Behrens]

My buddy Tim Kientzle (who's a FreeBSD committer) had some great
comments, which I'm posting here because they made a lot of sense, and
integrated nicely with what a few of you were saying:

Tim Kientzle wrote:
I skimmed your other email and it sounds like you're basically on the
right track, at least for v1. ;-)

In the longer term, you'll need to think more about self-service.   I
worked for a company not long ago that received large data files from
customers and manually converted them into the basis for a web
service.  After a while, the manual conversion step became a big
barrier to growth; each new customer required more in-house
data-conversion staff.  I don't suggest you focus too much on
self-service at first, but there are a few easy steps you can do to
help lay the groundwork for that:
 * Have customers drive the interaction themselves.  There should be a
web form where they upload their data.  There should be a simple
dashboard that shows the state of their data (uploaded, accepted,
converted, complete).  Even if the processes behind that are driven
manually for now, this sets the expectation that the customers have to
manage the process somewhat; they can't just call you and have you fix
every little problem.
 * Plan to store customer configuration.  A directory per user with
config files/scripts can work fine; you will likely also need a
database that maps customer accounts to those directories, etc.
 * Plan to eventually generate that configuration automatically.
After you've done the first few manually, you should start to get a
feel for what the variables are and can start providing "canned"
solutions that are easily reusable.

Makefiles are a fine way to script it.  Plan to soon include progress
output from the makefile, which can be as simple as a "logit.py"
script that is run at points to add a message to a database table.
That table can then form the basis of the customer dashboard.
     logit.py "Data format verified."
     logit.py "Initial data conversion complete."

Look up some of the literature on "job control systems."  Batch jobs
fail and need to be rerun from scratch, etc.  You'll need a database
table of jobs and their current state.

TBKK