Is there a maximum size to a Python program?

[andrew cooke]

not sure i've read all the posts on this, and i don't fully understand the
problem, but someone's mentioned sqlalchemy, so here's my experience with
that and large updates using mapped objects.

1 - don't commit each object as you modify it.  instead, process a whole
pile in memory and then (perhaps every 10,000 - when your memory is about
to run out and start paging) flush the session.  you'll find that the
memory use is very predictable - it ramps up as objects are cached and
then drops back down to the original level once they're sent to the
database.

2 - various sql related commands in sqlalchemy will take lists rather than
single values, and process all elements in the list in one "chunk".  this
is much more efficient.  using sql directly is faster and uses lest memory
than using mapped objects (the nice thing about sqlalchemy is that you can
use sql directly when it's the best solution, and mapped objects when they
are more useful).

these are both kind-of obvious, but that's all i needed to handle fairly
large data volumes with sqlalchemy.

andrew


Carbon Man wrote:
> I have a program that is generated from a generic process. It's job is to
> check to see whether records (replicated from another system) exist in a
> local table, and if it doesn't, to add them. I have 1 of these programs
> for
> every table in the database. Everything works well until I do the postcode
> table. The generated code is 5MB for a system with no current data.
> Normally
> the file would not be this big as only the changes are copied over. Python
> just quits, I have tried stepping through the code in the debugger but it
> doesn't even start.
> I am thinking that dynamically generating the programs to run might not be
> such a good idea. It would be a shame to drop it because the system needs
> to
> be generic and it runs from an XML file so the resulting code could be
> pretty complex, and I am new to Python. The program did generate a pyc so
> it
> was able to compile.
> Thoughts anyone?
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>