From gmcm@hypernet.com Sat Aug 28 16:05:54 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Sat, 28 Aug 1999 10:05:54 -0500 Subject: [Types-sig] Re: Python Type System: An idea for unification In-Reply-To: <37c72e93.1411973891@news.triode.net.au> Message-ID: <1276273608-41434292@hypernet.com> John (Max) Skaller wrote: [Stuff interesting enough to perhaps revive the Types SIG...] [Briefly, and subject to my misinterpretation, it appears that Viper objects always have a type obj; said type obj may be a class instance; said type obj is appended to the normal attribute search path] > class PyIntType: > def succ(x): return x + 1 > def pred(x): return x - 1 > > To use this we try out the Viper code: > > x = 1 > print x . succ() At first glance, these are class methods. But not really, because the bound method object binds the explicit arg 'x' to the user object (in this case also named 'x'). But that's not class PyIntType: def succ(self): return self + 1 ... because "self" in this case would be the instance of PyIntType, not the user object (which "has a" instance of PyIntType). So _something_ is special here. Ah, you are actually re-binding (the bound to type(x) method) "succ" to the user object ("1"). Right? [Generic programming...] > class ListOf: > def __init__(self,etype): > self.etype = etype > def __call__(self): > obj = new_object(self) > obj.data = [] > return obj > # ... > > Here, we use a class ListOf as a _generic_ type for all > lists of the same kind of type, and we use an _instance_ > of that class for the type: the argument to the __init__ method > specifies the type: > > ListOfInt = ListOf(IntType) So ListOfInt is an instance of ListOf. Unbound methods defined in ListOf are bound methods in ListOfInt. Presumably, they will be rebound to the object using ListOfInt as its type object. > will construct the type object for a list of ints. And here is an > actual list of ints: > > x = ListOfInt() > > Here, the __call__ method defined for the generic type ListOf > is called, because ListOfInt is an instance of the class ListOf. The > 'new_object' function creates a new object, and sets the type of the > object to its argument, which is an instance of ListOf for which the > 'etype' parameter is IntType. Is "new_object" a builtin? > We need to check where the binding occurs. In the > lookup on ListOf, a function found there is callable, > and so is bound to the instance ListOfInt. > But we want such a method to be bound the instance x > as well. So a generic method in ListOf needs TWO object > arguments: > > 1) the first argument binds to ListOfInt > 2) the second argument binds to x One of us is missing something here! There doesn't seem to be anyway to differentiate between: class ListOf: def method1(arg1, arg2): ... being called from the user object x as x.method1() #generic and/or x.method1(somearg) #non generic It seems to me you either need an explicit way of marking them as different when writing ListOf, or (I think more generally), writers of type classes need to always handle 2 "self"s: class PyIntType: def succ(self, obj): return obj + 1 # "self" is the instance of PyIntType, # "obj" is the "self" of the user object - some obj whose type # object is an instance of PyIntType ... Then, of course, you are not rebinding a method; you are binding the 2nd arg of an already bound method. Cool stuff! And you successfully avoided ever using the dreaded "meta" word... - Gordon From skaller@maxtal.com.au Sun Aug 29 06:48:36 1999 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 29 Aug 1999 15:48:36 +1000 Subject: [Types-sig] Re: Python Type System: An idea for unification In-Reply-To: <1276273608-41434292@hypernet.com> References: <37c72e93.1411973891@news.triode.net.au> Message-ID: <3.0.6.32.19990829154836.009664b0@mail.triode.net.au> At 10:05 28/08/99 -0500, Gordon McMillan wrote: >John (Max) Skaller wrote: > > [Stuff interesting enough to perhaps revive the Types SIG...] .. and some of it is actually implemented! > [Briefly, and subject to my misinterpretation, it appears that > Viper objects always have a type obj; said type obj may be > a class instance; said type obj is appended to the normal > attribute search path] "Usually" the type object will be a class, rather than an instance. >> class PyIntType: >> def succ(x): return x + 1 >> def pred(x): return x - 1 >> >> To use this we try out the Viper code: >> >> x = 1 >> print x . succ() > >At first glance, these are class methods. But not really, because the >bound method object binds the explicit arg 'x' to the user object (in >this case also named 'x'). I'm confused! 1) the function 'succ' REALLY is an ordinary function. 2) the class PyIntType REALLY is an ordinary class 3) the object (1).succ REALLY is an ordinary bound method. There is NO difference here from Python 1.5.2, it is ONLY the lookup that is different. [But see below, even the lookup is exactly what Python does right now!] Try this: print [].append and you will get "" so that you can see even in Python 1.5.2, bound methods do not have to have an 'im_self' object which is a class instance. What IS somewhat strange, however is that 'instances' of the TYPE PyIntType are integers, NOT class instances: an object of CLASS PyIntType: object = PyIntType() is a class instance, it has type Instance, class PyIntType. Whereas an integer has type PyIntType, and it doesn't have a class because it is not a class instance. This is, however, no different from python 1.5.2! >But that's not > class PyIntType: > def succ(self): return self + 1 > ... Yes, this is exactly equivalent! >because "self" in this case would be the instance of PyIntType, No: what you mean I think is that object = PyIntType() does not have type integer, and if you call succ: object.succ() WOOPS: an error because PyIntType does not have an __add__ method. >not the user object (which "has a" instance of PyIntType). So _something_ >is special here. Correct. The automatic searching of the type of an object is vaguely special (but not as special as you think!!) However, everything else is stock standard python; although it is also true that the 'type' of an object no longer has a type whose type's type is itself (namely TypeType). That is, Python 1.5.2's _internal_ TypeXXXX objects have been dispensed with, and replaced by arbitrary objects. Note that the normal python TypeXXX object have method tables, and python normally looks them up .. so what I am doing is not really that different to Python 1.5.2! >Ah, you are actually re-binding (the bound to >type(x) method) "succ" to the user object ("1"). Right? Yes. This is also what Python 1.5.2 would do, except the method would come from a special, C constructed TypeXXXX object. >> ListOfInt = ListOf(IntType) > >So ListOfInt is an instance of ListOf. Yes. >Unbound methods defined in >ListOf are bound methods in ListOfInt. Technically, this doesn't make sense: there are no 'methods', even in Python, in a class, or even in an instance. A 'method' is created only during lookup. So, when a lookup is done on an instance, a function found in the instances class becomes a bound method, bound to the instance: class X: def f(x): pass x = X() g = x.f Here, f is a function. It is NOT a method, there is no such thing as a method (except 'in the abstract'). Thus X.__dict__.f is a function, and an ordinary one. However, g = x.f is a bound method: a pair consisting of an object and a function. Python 1.5.2 does not require anything special about the function, nor the object. The object can be anything, including a list, and the function can be any function. (Technically, any code object I think) >Presumably, they will be >rebound to the object using ListOfInt as its type object. Yes, but there are TWO bindings. First, when a CLASS lookup is done for the instance ListOfInt, a function 'f' in the class dictionary of ListOf will be bound to the instance ListOfInt. So we have a bound method: here represented by a pairing notation. Now, since that is a callable function, the lookup on an object x will convert the callable to a bound method a second time: > A doubly bound method. >Is "new_object" a builtin? It will need to be, yes. >One of us is missing something here! There doesn't seem to be anyway >to differentiate between: > > class ListOf: > def method1(arg1, arg2): ... > >being called from the user object x as > x.method1() #generic > and/or > x.method1(somearg) #non generic What type is 'x'? >It seems to me you either need an explicit way of marking them as >different when writing ListOf, or (I think more generally), writers >of type classes need to always handle 2 "self"s: Yes. There will always be two selfs. A better naming convention is: the first one is called 'metaself' the second one is called 'self' With that convention, 'self' refers to the object, and meta-self refers to the type of the object. Remember that in a generic type constructed as I have shown, the methods of the meta-type, in this case ListOf, ar just that: they're shared by all the instance types such as ListOf(TypeInt). So you need to bind one of the arguments to the actual type object -- the metaself argument -- so that the method can examine the attributes special to the particular instance of the generic type. For example, in: m = ListOfInt.method the m function needs to be bound to ListOfInt, so, for example, it can find out dynamically what type it is working with: in the example I gave, the instance ListOfInt has an 'etype' attribute which determines the type of element in the list. But, the m function needs to be bound to the client object -- a list of integers --- as well, so it can do some work, so k = intlist.m the k function is bound to the object 'intlist' so it can actually do workk on the object. > class PyIntType: > def succ(self, obj): return obj + 1 > # "self" is the instance of PyIntType, > # "obj" is the "self" of the user object - some obj whose type > # object is an instance of PyIntType > ... For ordinary types like integers, the 'self' object is an integer: the PyIntType is not generic, and the type is a CLASS object, not an INSTANCE object of the class PyIntType. However, you could make a generic integer type, something like: class PyNumberMetaType: def succ(metaself, self): return metaself.addmethod(self,1) >Then, of course, you are not rebinding a method; you are binding the >2nd arg of an already bound method. That is the case when you are using the paradigm: class MetaType: ... def __init__(args ..) where the __init__ function constructs an instance type object. >Cool stuff! Yeah. Just consider: I only examined the cases where I used a class (for standard object types), and a class (for meta types). However, there is no restriction: the type object can be anything. But you know what is REALLY cool? This is what Python 1.5.2 does now! There is only one real difference, and that is that instead of using a TypeXXX object written in C, I allow any object, and instead of using the fixed 'vtable' lookup, I just use 'ordinary' lookup. Note that ordinary Type objects in Python provide in the vtable, a 'getattr' method for doing lookups for named attributes. Note: Viper doesn't use vtables (or objects). Instead, the interpreter 'knows' how to do all the standard operations: the functionality is built into the interpreter, rather than encapsulated in a Type object. This makes it harder to extend than CPython, but is necessary for implementing a compiler: you can't optimise encapsulated methods. >And you successfully avoided ever using the dreaded >"meta" word... WOOPS. It seems I didn't. :-) ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From gmcm@hypernet.com Sun Aug 29 15:33:08 1999 From: gmcm@hypernet.com (Gordon McMillan) Date: Sun, 29 Aug 1999 09:33:08 -0500 Subject: [Types-sig] Re: Python Type System: An idea for unification In-Reply-To: <3.0.6.32.19990829154836.009664b0@mail.triode.net.au> References: <1276273608-41434292@hypernet.com> Message-ID: <1276189173-46513020@hypernet.com> John (Max) Skaller wrote: > At 10:05 28/08/99 -0500, Gordon McMillan wrote: > >John (Max) Skaller wrote: > > [Briefly, and subject to my misinterpretation, it appears that > > Viper objects always have a type obj; said type obj may be > > a class instance; said type obj is appended to the normal > > attribute search path] > > "Usually" the type object will be a class, rather than > an instance. *That's* what I was missing. So, to reiterate: Given object x and type(x) == Y: When type(Y) == ClassType & there exists Y.method: x.method is resolved using the same plumbing as would occur in stock Python if x were an instance of Y. That is, create a bound method (x, Y.__dict__['method']). Thus the first arg to 'method', (normally called 'self') is the object x when invoked. When type(Y) == InstanceType, Y.__class__ == Z, & there exsits Z.method: x.method is resolved as a bound method (x, Y.method), which is really the doubly-bound method (x, (Y, Z.__dict__['method'])). Thus the first arg to 'method' is the object Y, and the second arg is the object x. The changes from stock Python are: 1) Tweaking attribute lookup (appears to be a simplification!). 2) How objects are associated to types. - builtins have a key that associates them to their type objects thru a global dict - users can create objects of arbitrary type thru a new builtin "new_object(type_object)" - extension writers have a bit more work to do, in that something more sophisticated than a static C struct is required. Is that a fair summary? Questions: Do the type objects of modules, functions, tracebacks... (where the new flexibility would not appear useful) follow these rules? Have you thought at all about the current metaclass hook? That is, a way of creating class instances where the type is something other than the "stock" InstanceType? Or perhaps, the type of the instance's class is something other than the "stock" ClassType? listening-for-the-sound-of-a-down-under-brain-exploding-ly y'rs - Gordon From skaller@maxtal.com.au Mon Aug 30 04:43:26 1999 From: skaller@maxtal.com.au (John Skaller) Date: Mon, 30 Aug 1999 13:43:26 +1000 Subject: [Types-sig] Re: Python Type System: An idea for unification In-Reply-To: <1276189173-46513020@hypernet.com> References: <3.0.6.32.19990829154836.009664b0@mail.triode.net.au> <1276273608-41434292@hypernet.com> Message-ID: <3.0.6.32.19990830134326.00967540@mail.triode.net.au> At 09:33 29/08/99 -0500, Gordon McMillan wrote: >John (Max) Skaller wrote: >> At 10:05 28/08/99 -0500, Gordon McMillan wrote: >> >John (Max) Skaller wrote: > >> > [Briefly, and subject to my misinterpretation, it appears that >> > Viper objects always have a type obj; said type obj may be >> > a class instance; said type obj is appended to the normal >> > attribute search path] >> >> "Usually" the type object will be a class, rather than >> an instance. > >*That's* what I was missing. Yeah: it's more general than just 'a type is a class'. A type can be any object. > The changes from stock Python are: > 1) Tweaking attribute lookup (appears to be a simplification!). If it were implemented in CPython, it might require a bit more code. I suspect this can be implemented in CPython, quite easily, but I'm not sure. > 2) How objects are associated to types. > - builtins have a key that associates them to their type > objects thru a global dict I consider this a hack: the indirection is forced by the way the interpreter initialises. There is probably a better technique, such as making the object --> type link a "pointer to pointer" intead of a pointer, (a tiny overhead), and then setting the type to 'not available yet' at startup. Then, after the py_types.py module is loaded, the types can be set to what was defined in that module: that is, the __typename__ lookup is done, but exactly once per type, not once per lookup per object. This would slow down lookup by exactly one dereference, which is trivial. It also allows 'global' instrumentation of a type by dynamically changing the type of all objects of a given type to some other type. > - users can create objects of arbitrary type thru a new > builtin "new_object(type_object)" That is the idea: but note it is not implemented yet. > - extension writers have a bit more work to do, in that > something more sophisticated than a static C struct is > required. This is only one third true . First, I am currently an extension writer, and it is easy to build extensions, because the native language is Ocaml, not C. Ocaml is MUCH easier to write than C! And many of the extensions required have already be implemented in Ocaml: for example unix stuff like sockets, files, processes; and just recently a PCRE (Perl style regular expressions). Second, many extensions being written today in C would not be required if the compiler turns out to work well enough: Python will do instead of C. Finally, Ocaml supports C extensions. These would be a bit harder to write, since you have to interface Viper to ocaml wrappers of the C functions, which would have to support the ocaml interface. HOWEVER, the job is made easier because you don't need to worry about methods, lookup, or other stuff. Instead, you just implement functions, and put them into a dictionary, and then you write PYTHON script to use them to model an object. This is what I am doing for files. The native file object is built into the interpreter at the moment and there are a few functions like 'file_open', 'file_read', 'file_close'. These functions are then called by class methods of the PyFileType object, and the builtin 'open' function is a Python function which creates an instance of it. So, for example, 'readline' is actually implemented in Python: it reads some bytes, usin 'file_read' uses 'string.find' to look for a newline, and returns the string up to the newline. [Actually, I do something more complex: there are TWO file types: PyNativeFileType and PyFileType. The first is a class which is a lightweight wrapper around the builtin file access functions, it is the type object for the native file type (which is not a class instance). The second object is a class, and the usual 'pyhthon file object' is actually an instance of it: one of the attributes is a PyNativeFileType object. In other words, print type(open("file")) assert type(open("file")) is type(SomeClass()) will print and you can derive a new class from this class, and set attributes: f = open("filename") f.my_attr = "An opened file" because it is a normal instance object. >Is that a fair summary? Yes. >Questions: > > Do the type objects of modules, functions, tracebacks... (where the > new flexibility would not appear useful) follow these rules? The type objects of modules, functions, etc, are all normal python class objects at the moment. However, non-method lookup is _not_ done generically. For example: x + y Here, the interpreter does a typecase on x and y, and calls the appropriate function. So you can't 'hook' the + operator with a __add__ method in the type objects for x and y if they're integers, for example. The only lookup that is 'hooked' at the moment is the notation: object.attribute # used as an rvalue getattr(object,"attribute") > Have you thought at all about the current metaclass hook? Not especially. The metaclass hack is an implementation artefact discovered after the fact. I'm interested in meta-programming, but I think it is better to design the architecture to support it from the ground up, rather than relying on a design accident. :-) If I understand correctly, the metaclass hook comes about when the base of a class is not a class. At present, Viper throws an exception when this is the case. I will investigate allowing bases to be 'any object': I can't see why it shouldn't work, however there is ticklish lookup problem here: the 'type object' lookup on an instance of a class only happens if the 'usual' lookup fails. The usual lookup searches the class of an instance and its bases, after searching the attribute dictionary. The question is: what happens while searching the class and bases? Normally, this is just a lookup in the class dictionary: I can't remember if the lookup in the type object PyClassType happens, because that class is defined by: class PyClassType: pass i.e. the lookup wouldn't find anything anyhow. I'm not sure that a 'full' lookup in the bases would make sense, because it would happen _before_ the lookup in the type object of the _instance_. The result would be a type method of the type of a base, rather than the type of the instance: surely the type of the instance should be searched first? I.e. the algorithm should be: 1. look in the instance dictionary 2. look in the class dictionary, and its bases, recursively 3. look in the type of the instance. Note that step 2 does not look in the class, it looks in the class dictionary. This is a different thing, since looking in the class implies looking in the class type object immediately the looking in the dictionary fails. The distinction is subtle and currently irrelevant. However, step 2 does NOT make sense if bases are not classes. So step 2 would have to become a generalised lookup. Just to blow your brain: remember that if the TYPE object is an instance, the whole lookup recurses: we not only look in the type object instance dictionary, but then also in the class of the instance, and then in the type of the type!!!! >That is, > a way of creating class instances where the type is something other > than the "stock" InstanceType? >or perhaps, the type of the > instance's class is something other than the "stock" ClassType? If I provide a 'settype(object, typeobject)' method, you could change the type object of any object to be any other object. Another possibility is that, like the __getattr__ method of a class, you could hook type fetching with a __gettype__ method, and another possibility is to provide explicit control over the lookup algorithm(s) (globally). The thing I need to keep in mind, though, is that I want to _compile_ python code. So support for 'extreme dynamism' is likely to be restricted to client constructed objects: I can't have clients fiddling things like integer addition, because the compiler needs to 'know' how to do that, to generate optimal code. Similarly, it would be nice to make _some_ Python classes onto the moral equivalent of the good old C struct, for high speed lookup; i.e. eliminate the overhead of a dictionary lookup. >listening-for-the-sound-of-a-down-under-brain-exploding-ly y'rs :-) ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia