ORM opinion

[Chris Angelico]

On Fri, Dec 5, 2014 at 5:14 AM, Joseph L. Casale
<jcasale at activenetwerx.com> wrote:
> Begrudgingly, I need to migrate away from SQLAlchemy onto a
> package that has fast imports and very fast model build times.
>
> I have a less than ideal application that uses Python as a plugin
> interpreter which is not performant in this use case where its
> being invoked freshly several times per operation within the
> application where several operations themselves are queued.
>
> In order to minimize latency, I need to remove anything that
> either takes a noticeable amount of time to import or run. My
> initial backend support is limited to only SQLite.
>
> Anyone have any recos or opinions?

First recommendation: Less layers. Instead of SQLAlchemy, just import
sqlite3 and use it directly. You should be able to switch out "import
sqlite as db" for "import psycopg2 as db" or any other Python DB API
module, and still have most/all of the benefit of the extra layer,
without any extra imports.

Secondly, see if you can keep the Python process running, instead of
separately invoking it for every operation. This will mean some
structural changes, but if you're specifying Python as the plugin
interpreter (as opposed to, say, "execute this program with these
args", which is far more general), you can probably set up a wrapper
script that calls on the user code repeatedly. Perhaps you could have
a multiprocessing.Queue feeding tasks to the subprocess, via a little
parent Python process. If you can't make those kinds of changes to the
application, the next best would be a stubby driver process that feeds
jobs to your real subprocess; the real work might be done by a
Unix/TCP socket server, and the stub connects, feeds it some
information, gets back a result, and disconnects. The stub will have
an extremely minimal set of imports (the bear necessities of life, if
you like), and the real process can import as much as it likes,
because it won't be doing it for every operation.

ChrisA