How to load new class definitions at runtime?

[Alex Martelli]

Carlos Ribeiro <carribeiro at gmail.com> wrote:
   ...
> attached to it. So simple object instances would not cut it. To allow
> the system to be customized without taking the application server
> offline, there must be a solution for these new class definitions to
> be loaded at runtime.

If you need them to be _updatable_ at runtime, look at a recipe by
Michael Hudson in the cookbook -- I've updated it for the printed
version 2nd ed (since we do have __subclasses__ now, yay!) but that's
not done yet.  My vote for most useful custom metaclasses ever...

> My current solution is simple. Class definitions have to be executed,
> and for it to work, the class name and the source code (or the
> compiled bytecode) is stored in a database. A class factory function
> retrieves the correct class definition (based on the name), and simply
> executes the code.

You can pickle the bytecode etc (again, cookbook recipes show how) but
that has no real advantage wrt storing the source or bytecode.  Just
make sure, for performance, that if you store the source the bytecode
also gets stashed away when first made.  Unless for some weird reason
you need your code repository to support multiple releases of Python at
the same time (eek), you might as well store the bytecode at the time
the source is stored -- upgrading release of Python in the future _will_
require a DB conversion and some downtime for the switch.

> Now, talking about executing arbitrary code retrieved from a database
> isn't exactly a safe proposition. But I can't see any other way to
> make it work. I'm now looking for more ideas on how do other people
> solved the same problem in other projects. Pointers are welcome.

If you want to allow arbitrary class definitions to happen at runtime,
"a safe proposition" is an oxymoron -- obviously you have to trust
*entirely* the repository of the code (and pickles would be just about
the same thing).  Digital signatures to ensure the repository has not
been tampered with would also need to be stored somewhere entirely
trustable, so you might as well declare that the trust must be towards
the repository, in all practical cases I can think of.


>From past experience with this sort of thing, my practical suggestion
is: make sure you have a development/debug mode in which the classes are
first looked for on ordinary modules in the filesystem, then gotten from
the DB only if the ordinary modules don't have them.  When you're
developing and debugging, you'll heavily use this feature, believe me,
and also Michael Hudson's recipe above-mentioned, since tweaks and mods
to the classes will come at a hefty pace.

It's a harder deployment decision whether the normal system operation
should also support the filesystem, or just the database.  The latter
makes development less convenient, but distributed processing much
easiery to manage, updates are transactional and atomic, etc.  It
depends on how much customization-during-operation you'll need, vs the
other deployment issues.


Alex