Optimizing memory usage w/ HypterText package

[Chris]

On Dec 5, 9:35 pm, je.s.t... at hehxduhmp.org wrote:
> I've been using the HyperText module for a while now
> (http://dustman.net/andy/python/HyperText/), and I really like it.
>
> I've run into a situation where I have code to construct a table
> and while it is normally perfect, there are times where the table
> can get quite huge (e.g. 1000 columns, 100000 rows .... yes, the
> question of "how on earth would someone render this table?" comes
> up, but that's not the point here :) ), and the code I have generating
> this starts choking and dying from excessive RAM usage.
>
> I'm curious if people see a better way of going about this task and/or
> believe that an alternative method of HTML generation here would be
> better.
>
> A (possibly somewhat pseudocode, as I'm doing this by hand) small example
> of what I'm doing ...
>
> inputs = [A, List, Of, Values, To, Go, Into, A, Table]
> numcolumns = howManyColumnsIWant
>
> out = ht.TABLE()
> column = 0
> for input in inputs:
>     if (column == 0):
>         tr = ht.TR()
>     tr.append(ht.TD(input))
>     column += 1
>     if (column == numcolumns):
>         out.append(tr)
>         column = 0
>
> As I said, this works fine for normal cases, but I've run into some situations
> where I need this to scale not just into the hundreds of thousands but also
> well into the millions - and that's just not happening.  Is there a better
> way to do this (which involves direct HTML generation in Python), or am
> I SOL here?

for (i, input) in enumerate(inputs):
  """Your Code
  """
  if not i % 1000:
    # Flush your data.

It's logical that you will run out of space as the code just appends
data constantly instead of ever writing it out.  How you flush the
data out is up to you or if it's as simple as you have there you could
do something like.

file_out.write('<TABLE>\n')
for x in xrange(0, len(inputs)//numcolumns):
  file_out.write('<TR>\n<TD>%s</TD>\n</TR>' % '</TD>
\n<TD>'.join(inputs[(x*numcolumns):((x+1)*numcolumns)]) )
  if not x % 500: file_out.flush()
file_out.write('<TR>\n<TD>%s</TD>\n</TR>\n</TABLE>' % '</TD>
\n<TD>'.join(inputs[x*numcolumns:]) )
file_out.close()