exceptions

[Alexander Schmolck]

Michael Hudson <mwh at python.net> writes:

> Alexander Schmolck <a.schmolck at gmx.net> writes:

>> it annoys me no end if I need to start an expensive computation from
>> scratch because some trivial and easily fixable problem occured towards the
>> end of the computation (sometimes it is possible to salvage stuff by hand
>> by pickling things from the appropriate post-mortem frame, but I'd *much*
>> prefer being able to say: foo=some_value; resume).
>
> I'd like this too.  It might be quite hard to implement
> non-disruptively but I haven't thought about it too hard.  Would make
> an excellent project for a master's thesis, IMHO.
>
>> Footnotes: 
>> [1]  Number 2 would be the stupid try: finally: idiom which also seems to
>>      screw up tracebacks
>>
>
> ?
>

I verified that this doesn't happen with plain python -- ipython's traceback
pretty printing code however doesn't display the right line for frames where
the exception occured within try: finally:. I've now tracked it down to the
use of inspect.getinnerframes, the line numbers here are subtly different from
what traceback.extract_tb returns (last executed expression in frame vs. where
the error occured). Since I couldn't find something readymade, I'll submit
some patch to ipython that merges the functionality of the two functions.

>>      (which has occasionally led me to get rid of them completely
>>      while debugging -- surely not a good thinge). My other gripes
>>      are again related to python's limitations for interactive
>>      software development -- I rather like python, but I really wish
>>      it did that better.
>
> What do you mean here, specifically?

In no particular order:

- Interactively redefining modules or classes in a way that propogates
  to other modules/preexisting instances is not exactly fun. One can often
  get by by judicious use of reload, mutating classes [1] (got to be careful
  to to avoid pickle failing with something along the lines of
  "myModule.myClass is not of type myModule.myClass")

- no images, i.e. you can't freeze and dump the state of the whole system

- it's not particularly easy or convenient to get an useful overview of the
  variables in your "workspace", e.g. memory consumption, narrowing down to
  "interesting" categories etc (I know that there is of course no general way
  of doing that, but compare e.g. for matlab)

- pdb is uhm, well... suboptimal (also crashes on me from time to time, not
  sure why)

- global variables in python are a painful in a number of ways that does
  affect interactive development

- there is not much of a culture to make modules work properly for interactive
  users -- things are often not reload-safe, export all sorts of crap, not
  just their interface (so that 'from foo import *' likely will hose things up
  -- this is slightly aggravated 'helpful' naming conventions such as
  datetime.datetime or StringIO.StringIO). Finally I think some modules,
  notably GUI toolkits won't work at all.

- the available IDEs I know of are clearly far from ideal. I'd venture the
  uneducated guess that ipython+emacs is amongst the best offerings for
  interactive development with python and it's really not that great -- e.g.
  if you use the py-execute commands source level debugging no longer will
  work since the temp file created by python-mode for execution will be gone
  (you can of course hang on to it, which is actually what I wound up doing
  but that's very risky -- you end up inadvertenly fixing temp-files rather
  than the real thing. I guess it might help if there were some easy way to
  exec(file) a string, but lie about were it came from (i.e. ``exec foo,
  filename="bla.py", startline=10)). In case anyone wonders that given my
  misgivings about pdb I'm bothered about this -- well, one can easily set up
  emacs/ipython to jump to right file and line when an error in your
  interactive session occurs (and than walk up and down the traceback). This
  is **very** useful, I have it activated pretty much all the time.


> I find I can do interactive development in Python most of the time (I
> do wish it was more possible with PyObjC, though).

I'm not saying it's impossible (I *always* run ipython in emacs, never python
in the commandline on a file, with the sole exception of regression test) but
kludging a workable interactive enviroment together needs, I think, a fair
amount of expertise and additional software (something like ipython+emacs) --
so I'd guess that most people primarily treat python as a really fast C
compiler (but I might be wrong), which is a pitty.

Also, whlilst python interactive offerings might be great when compared to
Java and C++( thankfully I haven't tried), it clearly falls far short of what
is in principle achievable and indeed has been often been achieved many, many
years ago (squeak, cmucl/sbcl+slime, matlab, j and plt scheme, to name just a
few all have things to offer for interactive work that a python user can only
dream of).


'as


Footnotes: 

[1] Here are a few of the hacks I'm using, in case anyone might find them
    useful -- or even better tell me about better alternatives (If someone has
    cooked something reasoable for reloading modules, I'd love to hear about
    it).

# to update all existing class instances

def updateClass(oldClass, newClass):
    """Destrucitively modify the ``__dict__`` and ``__bases__`` contents of
   `oldClass` to be the same as of `newClass`. This will have the effect that
   `oldClass` will exhibit the same behavior as `newClass`.

    Won't work for classes with ``__slots__`` (which are an abomination
    anyway).
    """
    assert type(oldClass) is type(newClass) is type #FIXME
    #FIXME redefinition of magic methods
    for name in dir(oldClass):
        if not name.startswith('__') or not name.endswith('__'):
            delattr(oldClass, name)
    for name in dir(newClass):
        if not name.startswith('__') or not name.endswith('__'):
            setattr(oldClass, name, newClass.__dict__[name])
    # XXX should check that this is absolutely correct
    oldClass.__bases__ = newClass.__bases__


## easy pickling and unpickling for interactive use

def magicGlobals(level=1):
    r"""Return the globals of the *caller*'s caller (default), or `level`
    callers up."""
    return inspect.getouterframes(inspect.currentframe())[1+level][0].f_globals

def __saveVarsHelper(filename, varNamesStr, outOf,extension='.bpickle',**opts):
    filename = os.path.expanduser(filename)
    if outOf is None: outOf = magicGlobals(2)
    if not varNamesStr or not isString(varNamesStr):
        raise ValueError, "varNamesStr must be a string!"
    varnames = varNamesStr.split()
    if not splitext(filename)[1]: filename += extension
    if opts.get("overwrite") == 0 and os.path.exists(filename):
	raise RuntimeError("File already exists")
    return filename, varnames, outOf

def saveVars(filename, varNamesStr, outOf=None, **opts):
    r"""Pickle name and value of all those variables in `outOf` (default: all
    global variables (as seen from the caller)) that are named in
    `varNamesStr` into a file called `filename` (if no extension is given,
    '.bpickle' is appended). Overwrites file without asking, unless you
    specify `overwrite=0`. Load again with `loadVars`.

    Thus, to save the global variables ``bar``, ``foo`` and ``baz`` in the
    file 'savedVars' do::

      saveVars('savedVars', 'bar foo baz')

    """
    filename, varnames, outOf = __saveVarsHelper(
        filename, varNamesStr, outOf, **opts)
    print "pickling:\n", "\n".join(isort(varnames))
    try:
        f = None
	f = open(filename, "wb")
        
	cPickle.dump(dict(zip(varnames, [outOf, varnames])),
                     f, 1) # UGH: cPickle, unlike pickle doesn't accept bin=1
    finally:
        if f: f.close()

def loadVars(filename, ask=True, into=None, only=None):
    r"""Load variables pickled with `saveVars`.
    Parameters:
    
        - `ask`: If `True` then don't overwrite existing variables without
                 asking.
        - `only`: A list to limit the variables to or `None`.
        - `into`: The dictionary the variables should be loaded into (defaults
                   to global dictionary).
           """
    filename = os.path.expanduser(filename)
    if into is None: into = magicGlobals()
    varH = loadDict(filename)
    toUnpickle = only or varH.keys()
    alreadyDefined = filter(into.has_key, toUnpickle)
    if alreadyDefined and ask:
	print "The following vars already exist; overwrite (yes/NO)?\n",\
      "\n".join(alreadyDefined)
	if raw_input() != "yes":
	    toUnpickle = without(toUnpickle, alreadyDefined)
    if not toUnpickle:
	print "nothing to unpickle"
	return None
    print "unpickling:\n",\
	  "\n".join(isort(list(toUnpickle)))
    for k in varH.keys():
        if k not in toUnpickle:
            del varH[k]
    into.update(varH)