[IPython-dev] should we make nodb the default hub backend?

[Fernando Perez]

On Mon, Jun 11, 2012 at 10:30 PM, MinRK <benjaminrk at gmail.com> wrote:
> What do you mean by retrieved?  Under the normal circumstances you
> described, exactly zero results are retrieved from the Hub, so this would
> have no effect other than breaking delayed retrievals after the first.
>
> Remember, the Hub is not involved *in any way* in the results of basic
> execution/reply, and it can be removed entirely with no effect whatsoever on
> simple executions.

Ah, I was making a mistaken assumption here; never mind then.

> Another option could be for DictDB to have a finite history, and just dump
> old results if it has too many.
>
>
> It's also widely varied - If each task sends/receives 10MB arrays, it only
> taks a few hundred to be a problem.  If, on the other hand, it's mostly
> simple commands without large data or side effects, it takes millions before
> there is an issue.  This makes the above suggestion difficult, because for
> some users a history of 100 is huge, while for others it's tiny.

Yes, that's really the problem: it's memory size, not length of the
history.  And detecting memory pressure in Python isn't too easy, it
would require separate codepaths for pickled results and numpy arrays,
which appear small to most normal python introspection tools (which
don't see their buffers).

I don't know if we want to fiddle with code of that nature this close
to a release, or just go to nodb for now, and think about this issue
with the calm of the 0.14 period.  I'm leaning towards that...

f