Magic function

[oj]

On Jan 11, 4:29 pm, dg.google.gro... at thesamovar.net wrote:
> Hi all,
>
> I'm part of a small team writing a Python package for a scientific
> computing project. The idea is to make it easy to use for relatively
> inexperienced programmers. As part of that aim, we're using what we're
> calling 'magic functions', and I'm a little bit concerned that they
> are dangerous code. I'm looking for advice on what the risks are (e.g.
> possibility of introducing subtle bugs, code won't be compatible with
> future versions of Python, etc.).
>
> Quick background: Part of the way our package works is that you create
> a lot of objects, and then you create a new object which collects
> together these objects and operates on them. We originally were
> writing things like:
>
> obj1 = Obj(params1)
> obj2 = Obj(params2)
> ...
> bigobj = Bigobj(objects=[obj1,obj2])
> bigobj.run()
>
> This is fine, but we decided that for clarity of these programs, and
> to make it easier for inexperienced programmers, we would like to be
> able to write something like:
>
> obj1 = Obj(params1)
> obj2 = Obj(params2)
> ...
> run()
>
> The idea is that the run() function inspects the stack, and looks for
> object which are instances of class Obj, creates a Bigobj with those
> objects and calls its run() method.
>
> So, any comments on that approach?
>
> I'm including the code I've written to do this, and if you have time
> to look through it, I'd also be very grateful for any more specific
> comments about the way I've implemented it (in particular, can it be
> made faster, is my program creating cycles that stop the garbage
> collection from working, etc.). I hope the code will be formatted
> correctly:
>
> def
> getInstances(instancetype,level=1,includeglobals=True,containersearchdepth=1,exclude={},predicate=lambda
> x:True):
>     """Find all instances of a given class at a given level in the
> stack
>     """
>     vars = {}
>     # Note: we use level+1 because level refers to the level relative
> to the function calling this one
>     if includeglobals: vars.update(stack()[level+1][0].f_globals)
>     vars.update(stack()[level+1][0].f_locals)
>     # Note that you can't extract the names from vars.itervalues() so
> we provide via knownnames the names vars.iterkeys(),
>     # containersearchdepth+1 is used because vars.itervalues() is the
> initial container from the point of view of this
>     # function, but not from the point of view of the person calling
> getInstances
>     objs, names =
> extractInstances(instancetype,vars.itervalues(),containersearchdepth
> +1,knownnames=vars.iterkeys(),exclude=exclude,predicate=predicate)
>     return (objs,names)
>
> def
> extractInstances(instancetype,container,depth,containingname='vars()',knownnames=None,exclude={},predicate=lambda
> x:True):
>     if depth<=0: return ([],[])
>     if isinstance(container,str): return ([],[]) # Assumption: no need
> to search through strings
>     # Ideally, this line wouldn't be here, but it seems to cause
> programs to crash, probably because
>     # some of the simulator objects are iterable but shouldn't be
> iterated over normally
>     # TODO: Investigate what is causing this to crash, and possibly
> put in a global preference to turn this line off?
>     if not isinstance(container,
> (list,tuple,dict,type({}.itervalues()))): return ([],[])
>     # Note that knownnames is only provided by the initial call of
> extractInstances and the known
>     # names are from the dictionary of variables. After the initial
> call, names can only come from
>     # the __name__ attribute of a variable if it has one, and that is
> checked explicitly below
>     if knownnames is None:
>         knewnames = False
>         knownnames = repeat(containingname)
>     else:
>         knewnames = True
>     objs = []
>     names = []
>     try: # container may not be a container, if it isn't, we'll
> encounter a TypeError
>         for x,name in zip(container,knownnames):
>             # Note that we always have a name variable defined, but if
> knewnames=False then this is just
>             # a copy of containingname, so the name we want to give it
> in this instance is redefined in this
>             # case. We have to use this nasty check because we want to
> iterate over the pair (x,name) as
>             # variables in the same position in the container have the
> same name, and we can't necessarily
>             # use __getitem__
>             if hasattr(x,'__name__'): name = x.__name__
>             elif not knewnames: name = 'Unnamed object, id =
> '+str(id(x))+', contained in: '+containingname
>             if isinstance(x,instancetype):
>                 if x not in exclude and predicate(x):
>                     objs.append(x)
>                     names.append(name)
>             else: # Assumption: an object of the instancetype is not
> also a container we want to search in.
>                 # Note that x may not be a container, but then
> extractInstances will just return an empty list
>                 newobjs, newnames =
> extractInstances(instancetype,x,depth-1,containingname=name,predicate=predicate)
>                 objs += newobjs
>                 names += newnames
>         return (objs,names)
>     except: # if we encounter a TypeError from the for loop, we just
> return an empty pair, container wasn't a container
>         return ([],[])
>
> In case that doesn't work, here it is without the comments:
>
> def
> getInstances(instancetype,level=1,includeglobals=True,containersearchdepth=1,exclude={},predicate=lambda
> x:True):
>     vars = {}
>     if includeglobals: vars.update(stack()[level+1][0].f_globals)
>     vars.update(stack()[level+1][0].f_locals)
>     objs, names =
> extractInstances(instancetype,vars.itervalues(),containersearchdepth
> +1,knownnames=vars.iterkeys(),exclude=exclude,predicate=predicate)
>     return (objs,names)
>
> def
> extractInstances(instancetype,container,depth,containingname='vars()',knownnames=None,exclude={},predicate=lambda
> x:True):
>     if depth<=0: return ([],[])
>     if isinstance(container,str): return ([],[])
>     if not isinstance(container,
> (list,tuple,dict,type({}.itervalues()))): return ([],[])
>     if knownnames is None:
>         knewnames = False
>         knownnames = repeat(containingname)
>     else:
>         knewnames = True
>     objs = []
>     names = []
>     try:
>         for x,name in zip(container,knownnames):
>             if hasattr(x,'__name__'): name = x.__name__
>             elif not knewnames: name = 'Unnamed object, id =
> '+str(id(x))+', contained in: '+containingname
>             if isinstance(x,instancetype):
>                 if x not in exclude and predicate(x):
>                     objs.append(x)
>                     names.append(name)
>             else:
>                 newobjs, newnames =
> extractInstances(instancetype,x,depth-1,containingname=name,predicate=predicate)
>                 objs += newobjs
>                 names += newnames
>         return (objs,names)
>     except:
>         return ([],[])

If you are the author of class Obj, then why not just make the class
maintain a record of any objects that have been instantiated?

That way, run could simply call a class method to obtain a list of all
the objects it needs.