[IPython-dev] slow startup due to script magic

[Jason Grout]

I was looking into why aleph.sagemath.org is so slow in responding to 
user requests (i.e., it takes a second or so to respond to any new 
request), and it turned out that the major slowdown was starting up a 
new IPython kernel.  We eventually traced it down to the loading of the 
script extension, which by default does 6 popen calls to find the paths 
of scripts in the _script_magics_default method, which are really slow 
(something like 200 ms for the calls on my computer):

https://github.com/ipython/ipython/blob/master/IPython/core/magics/script.py#L92 
and diving deeper, 
https://github.com/ipython/ipython/blob/master/IPython/utils/_process_posix.py#L35

If I move the tests up into the module level so that they are run when 
the module is loaded, rather than every time a new kernel is created, 
the kernel creation time drops dramatically, and in the end, our 
response time for aleph is cut in 1/2 or 1/3.

What can we do to optimize the guessing of where different commands 
live?  Here are at least a few ways:

1. explicitly specify a script_paths dictionary (just curious, what key 
would I make in the ipython config file to specify that?  I'm still a 
little lost when it comes to figuring out how keys relate to actual 
classes that are invoked.)

2. Move the guessing up into the module level, cached in a module-level 
variable.  While this doesn't help in the case where you only launch one 
kernel, it dramatically speeds up things if you are launching multiple 
kernels.

Any other ideas?  It seems such a shame that loading a rather esoteric, 
but standard, extension should double or so the startup time of an 
IPython kernel.

Thanks,

Jason