[Types-sig] Re: Python Type System: An idea for unification

[John Skaller]

At 09:33 29/08/99 -0500, Gordon McMillan wrote:
>John (Max) Skaller wrote:
>> At 10:05 28/08/99 -0500, Gordon McMillan wrote:
>> >John (Max) Skaller wrote:
>
>> > [Briefly, and subject to my misinterpretation, it appears that
>> >  Viper objects always have a type obj; said type obj may be
>> >  a class instance; said type obj is appended to the normal
>> >  attribute search path]
>> 
>>  "Usually" the type object will be a class, rather than
>> an instance.
>
>*That's* what I was missing.

	Yeah: it's more general than just 'a type is a class'.
A type can be any object.

> The changes from stock Python are:
>   1) Tweaking attribute lookup (appears to be a simplification!).

	If it were implemented in CPython, it might require a bit more
code. I suspect this can be implemented in CPython, quite easily,
but I'm not sure. 

>   2) How objects are associated to types.
>      - builtins have a key that associates them to their type
>        objects thru a global dict

	I consider this a hack: the indirection is forced
by the way the interpreter initialises. There is probably
a better technique, such as making the object --> type
link a "pointer to pointer" intead of a pointer,
(a tiny overhead), and then setting the type to 'not available yet'
at startup. Then, after the py_types.py module is loaded,
the types can be set to what was defined in that module:
that is, the __typename__ lookup is done, but exactly once
per type, not once per lookup per object.

	This would slow down lookup by exactly one dereference,
which is trivial. It also allows 'global' instrumentation of
a type by dynamically changing the type of all objects
of a given type to some other type.

>      - users can create objects of arbitrary type thru a new
>        builtin "new_object(type_object)"

	That is the idea: but note it is not implemented yet.

>      - extension writers have a bit more work to do, in that 
>        something more sophisticated than a static C struct is
>        required.

	This is only one third true <g>. First, I am currently 
an extension writer, and it is easy to build extensions,
because the native language is Ocaml, not C. Ocaml is MUCH
easier to write than C! And many of the extensions required
have already be implemented in Ocaml: for example unix
stuff like sockets, files, processes; and just recently
a PCRE (Perl style regular expressions).

	Second, many extensions being written today in C
would not be required if the compiler turns out to work
well enough: Python will do instead of C.

	Finally, Ocaml supports C extensions. These would be
a bit harder to write, since you have to interface
Viper to ocaml wrappers of the C functions, which would
have to support the ocaml interface. 

	HOWEVER, the job is made easier because you don't
need to worry about methods, lookup, or other stuff.
Instead, you just implement functions, and put them
into a dictionary, and then you write PYTHON script
to use them to model an object.

	This is what I am doing for files.
The native file object is built into the interpreter at
the moment and there are a few functions like
'file_open', 'file_read', 'file_close'. These functions
are then called by class methods of the PyFileType
object, and the builtin 'open' function is a Python
function which creates an instance of it.

	So, for example, 'readline' is actually
implemented in Python: it reads some bytes, usin 'file_read'
uses 'string.find' to look for a newline,
and returns the string up to the newline.

	[Actually, I do something more complex: there are
TWO file types: PyNativeFileType and PyFileType. The first
is a class which is a lightweight wrapper around the
builtin file access functions, it is the type object
for the native file type (which is not a class instance).

	The second object is a class, and the usual
'pyhthon file object' is actually an instance of it:
one of the attributes is a PyNativeFileType object.
In other words, 

	print type(open("file"))
	assert type(open("file")) is type(SomeClass())

will print

	<Instance of PyFileType>

and you can derive a new class from this class,
and set attributes:

	f = open("filename")
	f.my_attr = "An opened file"

because it is a normal instance object.

>Is that a fair summary?

	Yes.

>Questions:
>
> Do the type objects of modules, functions, tracebacks... (where the 
> new flexibility would not appear useful) follow these rules?

	The type objects of modules, functions, etc, are all
normal python class objects at the moment. However,
non-method lookup is _not_ done generically. For example:

	x + y

Here, the interpreter does a typecase on x and y, and calls the
appropriate function. So you can't 'hook' the + operator
with a __add__ method in the type objects for x and y
if they're integers, for example. 

	The only lookup that is 'hooked' at the moment is the notation:

	object.attribute # used as an rvalue
	getattr(object,"attribute")

> Have you thought at all about the current metaclass hook? 

	Not especially. The metaclass hack <g> is an implementation
artefact discovered after the fact. I'm interested in 
meta-programming, but I think it is better to design
the architecture to support it from the ground up,
rather than relying on a design accident. :-)

	If I understand correctly, the metaclass hook comes
about when the base of a class is not a class. At present,
Viper throws an exception when this is the case. 

	I will investigate allowing bases to be 'any object': 
I can't see why it shouldn't work, however there is ticklish
lookup problem here: the 'type object' lookup on
an instance of a class only happens if the 'usual'
lookup fails. The usual lookup searches the class
of an instance and its bases, after searching the
attribute dictionary.

	The question is: what happens while searching the class and bases?
Normally, this is just a lookup in the class dictionary:  I can't
remember if the lookup in the type object PyClassType happens,
because that class is defined by:

	class PyClassType: pass

i.e. the lookup wouldn't find anything anyhow. I'm not sure that
a 'full' lookup in the bases would make sense, because it would
happen _before_ the lookup in the type object of the _instance_.
The result would be a type method of the type of a base,
rather than the type of the instance: surely the type of the
instance should be searched first?

	I.e. the algorithm should be:

	1. look in the instance dictionary
	2. look in the class dictionary, and its bases,
		recursively
	3. look in the type of the instance.

Note that step 2 does not look in the class, it looks
in the class dictionary. This is a different thing,
since looking in the class implies looking in the
class type object immediately the looking in the
dictionary fails.

	The distinction is subtle and currently
irrelevant. However, step 2 does NOT make sense
if bases are not classes. So step 2 would have to
become a generalised lookup.

	Just to blow your brain: remember that
if the TYPE object is an instance, the whole
lookup recurses: we not only look in the type
object instance dictionary, but then also
in the class of the instance, and then
in the type of the type!!!!

>That is,
> a way of creating class instances where the type is something other
> than the "stock" InstanceType? 

>or perhaps, the type of the 
> instance's class is something other than the "stock" ClassType?

	If I provide a 'settype(object, typeobject)'
method, you could change the type object of any object
to be any other object. 

	Another possibility is that, like the
__getattr__ method of a class, you could hook type fetching with
a __gettype__ method, and another possibility is to provide
explicit control over the lookup algorithm(s) (globally).

	The thing I need to keep in mind, though, is that
I want to _compile_ python code. So support for
'extreme dynamism' is likely to be restricted to 
client constructed objects: I can't have clients
fiddling things like integer addition, because the
compiler needs to 'know' how to do that, to generate
optimal code.

	Similarly, it would be nice to make _some_
Python classes onto the moral equivalent of the good
old C struct, for high speed lookup; i.e. eliminate
the overhead of a dictionary lookup.

>listening-for-the-sound-of-a-down-under-brain-exploding-ly y'rs

	:-)

-------------------------------------------------------
John Skaller    email: skaller at maxtal.com.au
		http://www.maxtal.com.au/~skaller
		phone: 61-2-96600850
		snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia