[IPython-dev] migration issues, embedding latest IPython in another application

[Matt Anderson]

The flag may still be undocumented, and in my reading it does seem dangerous to actually, concurrently, use the sqlite3 db object in multiple threads (I believe a stackoverflow Q/A thread discussed some form of database corruption).  However, in the case of the originating thread having having exited and the main thread running the at-exit handlers, I believe that it would probably be safe.

I totally agree that given an undocumented, possibly dangerous, sqlite3 feature, IPython should not use this argument by default when opening data base connections.  However, it would be really useful, in cases like mine, to enable this type of database connection by  configuration, rather than needing to do multiple subclassings and re-writing of core functions.  I don't know how stable the APIs of the ipython shell or the HistoryManager are, but if they're not totally stable, subclassing in the way that would be necessary doesn't seem *much* safer than monkey patching the IPython.core.history module.

Alternatively, maybe there is some other way to shut down the database connection, at thread shutdown time instead of at program shutdown time?  Either automatically (by a per-thread equivalent of the program-wide at-exit system, if such a thing does or could exist). Or maybe by hook (ie, a way to turn the at-exit registration off, and provide some method on the interactive shell that could be called in the that would shut down the database connection safely from the correct thread, delegating responsibility to the programmer embedding the shell to do the shutdown correctly).  Or possibly, at the end of the interactive shell REPL loop, just before control is passed back to the main program (in the case of the embedded shell)?

Just a couple of thoughts.

Thanks,
Matt

________________________________________
From: ipython-dev-bounces at scipy.org [ipython-dev-bounces at scipy.org] on behalf of Thomas Kluyver [takowl at gmail.com]
Sent: Tuesday, July 24, 2012 4:04 AM
To: IPython developers list
Subject: Re: [IPython-dev] migration issues, embedding latest IPython in another application

On 11 July 2012 20:58, Matt Anderson <manderso at broadcom.com> wrote:
> Several bigger issues I encountered revolve around threading. Our
> application can run headless, and the cmd.Cmd shell (when active) does not
> run in the main thread.  The current version of IPython installs at-exit
> handlers when a shell is created or starts up, including functionality to
> close/save the history session when the program exits.  An exception gets
> thrown during the exit process though: “ProgrammingError: SQLite objects
> created in a thread can only be used in that same thread.” The ipython shell
> created the sqlite db object in the cmd.Cmd’s thread, and the at-exit
> handlers always (or at least in most cases / our case) run in the main
> thread.

Good question, but I don't think I have a good answer. I didn't think
about running IPython in a secondary thread.

> I understand that there is a workaround to open a sqlite3 connection in a
> way that suppresses this behavior, but IPython doesn’t open the connection
> in that way by default, and doesn’t seem to provide a hook to do this short
> of subclassing the TerminalInteractiveClass or the InteractiveShellEmbed
> class and writing one’s own init_history() method, and further subclassing
> the HistoryManager and overriding its init_db() method.

Yes, we've come across the flag before, but it is undocumented (last
time I checked), so we're wary of using it in case it has some major
drawback.

Thanks,
Thomas
_______________________________________________
IPython-dev mailing list
IPython-dev at scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev