__slots_ and inheritance

[Alexander Schmolck]

Alex Martelli <aleax at aleax.it> writes:

> A. Lloyd Flanagan wrote:
> 
> > Alexander Schmolck <a.schmolck at gmx.net> wrote in message
> > news:<yfsistmh4ec.fsf at black132.ex.ac.uk>...
> >  
> >> __slots__ is the one addition in python2.2 that really annoys me. It
> >> seems an ill-conceived jumble of conceptually unrelated things (bug
> >> avoidance, optimization, safety, encapsulation) that screws up reflection
> >> big time for little apparent gain. As Knuth said ...
>
> I think it's an important memory optimization, PERIOD.  The other
> conceptually unrelated things (bug avoidance -- I can't even SEE the
> others!) aren't in __slots__' own design (even though a commentator
> who's usually quite perceptive seems to have seen the "bug avoidance"
> feature that wasn't really there).

You may well have some superior knowledge, but to me personally the actual
design intention(s) were not that obvious from descrinto.html (BTW, as to why
 encapsulation and (more arguably) security:
- because you can hide attributes (and thus implementation)
- because you can enforce the layout (and type) of instances to thwart
  copying, and prevent method or class redefinition
(at least in practice, although possibly not by design)).

If the only reason is optimization, why couldn't it have been done less
disruptively (not a rhetorical question)? Specifically:

1. what's the point of not automatically including a slot for '__weakref__'
   (hardly the unreasonable memory consumption such a slot would impose?)?

2. similarly, why doesn't pickling just work out of the box?

3. why can a __slots__ class no just dynamically create an immutable
   pseudo-dict as read-only attribute whenever .__dict__ is accessed (similar
   to e.g. object.__dict__)? That would mean the python's so-far pretty
   unified introspection interface would still work [1].

4. why can't I assign to __class__? (I don't know exactly when else that
   breaks down, the only other case I'm aware of are assigning new-style
   classes to old-style instances and vice versa)

[Note: I just saw Tim's posting which promises that some of those issues will
go away]


> 
> As Knuth said, *premature* optimization is the root of all evil in
> programming. But __slots__ may allow very substantial optimization that need
> not be premature.

The implication was that including __slots__ in the language was a premature
optimization, not that every use of __slots__ is premature (although I fear
many will be). About any feature whose semantics or performance are not easily
achievable with other language features has its potential uses (even goto) --
but *that doesn't mean* that it (goto) should be included in the language in
the first place.

> 
> > Unless you absolutely have to have it, don't use it.
> 
> "Absolutely have to have it", just like "screws up big time", are
> silly overbids IMHO -- over-reactions that I don't see as justified
> at all!  Why can't we all just keep SOME perspective, please...?!

"screws up [reflection] big time"... Well there are 2 components to
reflection:

- intercession (which seems all but gone for classes with __slots__)
  
- introspection (which clearly is at the very least complicated: I'd think the
  majority of introspection features I normally use involve looking at an
  object's __dict__[1] -- such code could obviously be rewritten to also
  handle __slots__, but even if we assume that __slots__ isn't a lie [2], this
  at the very least involves discomfort and code breakage ).

And maybe a third:

- Its creator is not necessarily the only person who reflects a piece of work
  :)

> 
> Consider a typical use case, a class whose instances have and need
> a single attribute 'x', and of which we need to generate 10,000:
[snipped]
> Do we "*absolutely* have to have" the roughly factor-of-2 speedup given
> by __slots__?  

Do we "*absolutely* have to have" the roughly factor-of-100-to-1000 speedup
that [insert your favorite C, Fortran or assembler feature here] would allow
us for certain cases? Depends on implementation difficulty, ease of use and
error-proneness as well as the overall effect on the language
-- but unless it's a really common operation -- probably not. Especially since
pyrex, swig, boost, f2py etc. etc. make it reasonably easy to replace resource
critical sections with C or Fortran extensions.

> Clearly not -- we COULD live with a program that's roughly twice as slow,
> and here we're measuring only initialization anyway.

Exactly -- this is not a real program (which would hardly be twice as slow,
I'd assume?).

> and tens or hundreds of thousands of instances can well be necessary*?

Sure, and if I often felt an urge to have zillions of instances, I'd agree
even more heartily. But biased as I certainly am, I'm tempted to doubt the
problem occurs anywhere frequently to enough to justify shifting the burden
from those people who do encounter it (and might otherwise have had to write
an extension) to the language community as a whole.

> What is it that you think makes __slots__ so terrible, compared with many
> other possible optimizations?!

Optimizations (in a high level language) should either:

-  not cause *any* pain (i.e. be transparently performed by the language
   itself (e.g. constant expression lifting, loop unrolling, JIT compilation
   etc.) -- this of course is all very hard)

-  or the pain should be well-confined, at the very least when everything is
   done correctly (e.g. plastering the code with hints for the compiler or
   disabling usually automatic error checking for *correct* code).

At the *very* least their dangers should be obvious.

*None* of the above holds currently true for __slots__: basically, a simple,
seemingly innocuous addition of in the form of the single line 

__slots__ = ['foo', 'bar', ...

changes the semantics of a class *considerably*: it simply is no longer
behaves like an ordinary python class at all. Usually well defined operations
will now either stop working altogether (e.g. decoration), behave subtly
different (e.g. 'default' class variables) or send demons flying out your nose
(viz, cause undefined behavior).

But do the *particular* things __slots__ break really matter that much?

I really think they do, because I feel the current __slots__ design and
worsens some existing problems and contributes to the errosion of some of the
great things about python (simplicity and malleability). Seeing that it is
getting late and this post is already overly long, I'd better continue this
theme in another posting.

As I already stand accused of hyperbole I can however not withhold a final
comment: :)

> [I really wish I had a good way to find out how much memory a given
> container -- together with all of its contents -- is using... unfortunately,
> I don't know of any good way to answer that question!!!].

This feels a bit like handing a shotgun to a paranoid with blurry eyesight
*before* procuring a pair of glasses (at considerable peril to his own and his
neighborhood's safety).

(and although creation of zillions of instances might be a no-brainer for
memory bottleneck causation, in general even the best programmers are *really,
really* crap at intuiting bottlenecks)


'as


Footnotes: 
[1] .__dict__, vars, inspect.getmembers, dir ... Hmm, it's already getting
    sort of messy (all of these are *subtly* different but seem to have
    substantial overlap).

[2] i.e. __slots__ can be tampered with after class creation, which currently
    works but I guess is not intended to relied upon -- so most likely doing
    so means "undefined behavior" (otherwise the only way to find out seems to
    be to use `dir` (which's behavior is essentially undefined as it is
    intended as a mere convenience) -- or the funny inspect.getmembers(that
    however also cheats by calling dir :)).