From trent at snakebite.org Thu Jun 5 14:02:14 2014 From: trent at snakebite.org (Trent Nelson) Date: Thu, 5 Jun 2014 05:02:14 -0700 Subject: [Python-ideas] Make Python code read-only In-Reply-To: References: Message-ID: <9C2B8DBB-80EC-40DE-847E-C9AF421CDDBE@snakebite.org> On May 20, 2014, at 12:57 PM, Victor Stinner wrote: > Hi, > > I'm trying to find the best option to make CPython faster. I would > like to discuss here a first idea of making the Python code read-only > to allow new optimizations. I did two passes on read-only functionality for PyParallel. First attempt was similar to yours; I instrumented various core Python objects such that mutations could be detected against read-only objects (and subsequently raised as an exception). That didn?t pan out the way I wanted it to, especially in the PyParallel multiple-interpreter-threads-running-in-parallel environment. Second attempt: use memory protection. CPUs and OSes are really good at enforcing memory protection ? leverage that. Don?t try and do it yourself in userspace. This worked much better. That work is described starting here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores?slide=138 Relevant bits of implementation: obmalloc.c: http://hg.python.org/sandbox/trent/rev/0e70a0caa1c0#l6.299 ceval.c: http://hg.python.org/sandbox/trent/rev/0e70a0caa1c0#l9.30 On POSIX you?d achieve the same affect via mprotect and a SIGSEV trap. Just FYI. Regards, Trent. From victor.stinner at gmail.com Thu Jun 5 15:05:42 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 5 Jun 2014 15:05:42 +0200 Subject: [Python-ideas] Make Python code read-only In-Reply-To: <9C2B8DBB-80EC-40DE-847E-C9AF421CDDBE@snakebite.org> References: <9C2B8DBB-80EC-40DE-847E-C9AF421CDDBE@snakebite.org> Message-ID: 2014-06-05 14:02 GMT+02:00 Trent Nelson : > On May 20, 2014, at 12:57 PM, Victor Stinner wrote: >> I'm trying to find the best option to make CPython faster. I would >> like to discuss here a first idea of making the Python code read-only >> to allow new optimizations. > > I did two passes on read-only functionality for PyParallel. First attempt was similar to yours; I instrumented various core Python objects such that mutations could be detected against read-only objects (and subsequently raised as an exception). That didn?t pan out the way I wanted it to, especially in the PyParallel multiple-interpreter-threads-running-in-parallel environment. > > Second attempt: use memory protection. CPUs and OSes are really good at enforcing memory protection ? leverage that. Don?t try and do it yourself in userspace. This worked much better. My first attempt to "make the code read-only" was a big fail. Lot of errors and complains :-) I'm now moving to a different approach: "notify changes of the code". In PyParellel, you raise an error if something is modified. I don't need such restriction, I "just" want to disable optimizations if the code changed. > On POSIX you?d achieve the same affect via mprotect and a SIGSEV trap. I don't think that relying on SIGSEGV is reliable :-( Such signal can be emitted for various reasons and you have to use sigsetjmp/siglongjmp which is unsafe: you cannot cleanup state when an error occurs. Or did you implement it differently? Victor From sturla.molden at gmail.com Thu Jun 5 16:42:32 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 05 Jun 2014 16:42:32 +0200 Subject: [Python-ideas] Make Python code read-only In-Reply-To: References:

Message-ID: On 21/05/14 02:16, Victor Stinner wrote: > I don't want to optimize a single function, I want to optimize a whole > application. Right. Even Java does not do that. (Hence the name 'Hotspot') > If possible, I would prefer to not have to modify the application to > run it faster. > > Numba plays very well with numbers and arrays, but I'm not sure that > it is able to inline arbitrary Python function for example. Numba will compile the Python overhead out of function calls, if that is what you mean. Numba will also accelerate Python objects (method calls and attribute access). LLVM knows how to do simple optimisations like function inlining. When a Python function is JIT compiled to LLVM bytecode by Numba, LLVM knows what to do with ut. If the function body is small enough, LLVM will inline it completely. Numba is still under development, so is might no be considered "production ready" yet. Currently it will give you performance comparable to -O2 in C for most algorithmic Python code. Sturla From ncoghlan at gmail.com Thu Jun 5 18:05:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Jun 2014 02:05:35 +1000 Subject: [Python-ideas] String-like methods on StringIO objects? Message-ID: >From the "idle speculation" files (inspired by the recent thread on python-dev): has anyone ever experimented with offering string methods like find() on StringIO objects? I don't work in any sufficiently memory constrained environments these days that that style of API would be worth the hassle relative to a normal string, it just struck me as a potentially interesting approach to the notion of a string manipulation type that didn't generally copy data around and could use different code point sizes internally for different parts of the text data. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-ideas at hmmz.org Thu Jun 5 18:39:25 2014 From: dw+python-ideas at hmmz.org (dw+python-ideas at hmmz.org) Date: Thu, 5 Jun 2014 16:39:25 +0000 Subject: [Python-ideas] String-like methods on StringIO objects? In-Reply-To: References: Message-ID: <20140605163925.GB17301@k2> On Fri, Jun 06, 2014 at 02:05:35AM +1000, Nick Coghlan wrote: > From the "idle speculation" files (inspired by the recent thread on > python-dev): has anyone ever experimented with offering string methods like > find() on StringIO objects? > I don't work in any sufficiently memory constrained environments these days > that that style of API would be worth the hassle relative to a normal string, > it just struck me as a potentially interesting approach to the notion of a > string manipulation type that didn't generally copy data around and could use > different code point sizes internally for different parts of the text data. Thought about this quite a bit. There are a few ways StringIO/BytesIO/buffers could improve, not sure which approaches are interesting, though.. 1) Not sure if it's the case in Python3.x (pretty sure it isn't in 2.x), but cStringIO could optimize for the case where the IO is discarded after building a single string by using the CPython APIs for doing that (e.g. _PyString_Resize). In that case, getvalue() returns the built string, and sets an internal flag to cause it to be copied to a new private string if any further IO is invoked. This inverts the current behaviour where the normal case of build-and-discard causes a copy. 2) Rather than implement string methods on the StringIO, it might be nicer if those methods could apply to a memoryview, and then make it possible e.g. for BytesIO to be exposed as a memoryview. Right now Python doesn't have much in the way of generic "type safe / memory safe" APIs for doing things to regular memory without first invoking copies/conversions of various sorts. This might be the more useful thing to fix. We have plenty of special cases, like bytearray(), array.array(), StringIO (to some degree), and so on, and various ways to manipulate that memory (ctypes and struct module for example), but they are all somewhat hodge-podges of each other and lack any "one way to do it". I had looked at building some kind of unified 'memory slice' type last year, since I keep bumping into the need for better Python-level support for this stuff when working on 'bit twiddling' projects of various kinds. It's mostly thinking aloud, but here is a rough sketch for the kind of module I had been considering last year, mostly while working with Python 2: https://github.com/dw/memsink/wiki/Memory-Module . The idea was to provide a common 'Slice' adaptor type whose memory could be interpreted using a couple of different abstractions (Vector and File being the obvious). David > > Cheers, > Nick. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From cf.natali at gmail.com Thu Jun 5 21:26:35 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Thu, 5 Jun 2014 20:26:35 +0100 Subject: [Python-ideas] Make Python code read-only In-Reply-To: References:

Message-ID: 2014-06-05 15:42 GMT+01:00 Sturla Molden : > Numba is still under development, so is might no be considered "production > ready" yet. Currently it will give you performance comparable to -O2 in C > for most algorithmic Python code. When you consider it production ready, don't hesitate to suggest it for inclusion on python-dev: FWIW, I think it's high time we have a JIT compiler in CPython... From mistersheik at gmail.com Sat Jun 7 03:53:30 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 6 Jun 2014 18:53:30 -0700 (PDT) Subject: [Python-ideas] =?utf-8?q?Put_default_setstate_and_getstate_on_obj?= =?utf-8?q?ect_for_use_in_co=C3=B6perative_inheritance=2E?= Message-ID: When implementing getstate in co?oerative inheritance, the typical thing to do is to call super to get dictionary and add the appropriate entries. Setstate is similar: you extract what you need out of the dictionary and call super with the remaining entries. Unfortunately, object does not have a default implementation, so you need a base class like so: class DefaultSetstateAndGetstate: """ Define default getstate and setstate for use in co?perative inheritance. """ def __getstate__(self): return self.__dict__.copy() def __setstate__(self, state): self.__dict__.update(state) I suggest that this be added to object. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat Jun 7 03:59:38 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 6 Jun 2014 18:59:38 -0700 (PDT) Subject: [Python-ideas] Expose `itertools.count.start` and implement `itertools.count.__eq__` based on it, like `range`. In-Reply-To: References: <082cd87a-aeb5-49bf-9f79-d99a6d18e402@googlegroups.com>

Message-ID: That would be great. On Friday, May 16, 2014 12:16:52 AM UTC-4, Antony Lee wrote: > > Actually, a more reasonable solution would be to have range handle keyword > arguments and map "range(start=x)" to "count(x)". Or, perhaps more simply, > "range(x, None)" (so that no keyword arguments are needed). > > > 2014-05-15 13:04 GMT-07:00 Ram Rachum >: > >> Now that I think about it, I would ideally want `itertools.count` to be >> deprecated in favor of `range(float('inf'))`, but I know that would never >> happen. >> >> >> On Thursday, May 15, 2014 11:02:56 PM UTC+3, Ram Rachum wrote: >>> >>> I suggest exposing `itertools.count.start` and implementing >>> `itertools.count.__eq__` based on it. This'll provide the same benefits >>> that `range` got by exposing `range.start` and allowing `range.__eq__`. >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Jun 7 07:14:57 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 7 Jun 2014 15:14:57 +1000 Subject: [Python-ideas] =?iso-8859-1?q?Put_default_setstate_and_getstate_o?= =?iso-8859-1?q?n_object_for_use_in_co=F6perative_inheritance=2E?= In-Reply-To: References: Message-ID: <20140607051457.GN10355@ando> On Fri, Jun 06, 2014 at 06:53:30PM -0700, Neil Girdhar wrote: > When implementing getstate in co?oerative inheritance, the typical thing to > do is to call super to get dictionary and add the appropriate entries. > Setstate is similar: you extract what you need out of the dictionary and > call super with the remaining entries. Unfortunately, object does not have > a default implementation, so you need a base class like so: I'm afraid you're going to need to explain in more detail what you're talking about. Even a link to a discussion elsewhere. I've used cooperative inheritance without needing to write a getstate or setstate method, so I have no idea why you think these are important enough to go into the base object. I presume you're not talking about serialization formats? That's where I would normally expect to find a getstate and setstate. It might also help if you can do a survey of other languages, like Java and Ruby, and tell us if they have such methods in the base object. -- Steven From mistersheik at gmail.com Sat Jun 7 08:10:15 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 7 Jun 2014 02:10:15 -0400 Subject: [Python-ideas] =?utf-8?q?Put_default_setstate_and_getstate_on_obj?= =?utf-8?q?ect_for_use_in_co=C3=B6perative_inheritance=2E?= In-Reply-To: <20140607051457.GN10355@ando> References: <20140607051457.GN10355@ando> Message-ID: Hi Steven, If you don't know about getstate and setstate, I suggest you take a look at the documentation: https://docs.python.org/3.3/library/pickle.html#object.__getstate__. Besides allowing objects to be pickled, providing these methods allows them to be copied with the copy module. Some of the pickling and copying support can be provided by getnewargs, but this was unfortunately almost useless for cooperative inheritance. Luckily, getnewargs_ex was recently added, which fills in this hole (each subclass fills in the keyword arguments it wants to pass to __new__ and calls super for the rest). Best, Neil On Sat, Jun 7, 2014 at 1:14 AM, Steven D'Aprano wrote: > On Fri, Jun 06, 2014 at 06:53:30PM -0700, Neil Girdhar wrote: > > When implementing getstate in co?oerative inheritance, the typical thing > to > > do is to call super to get dictionary and add the appropriate entries. > > Setstate is similar: you extract what you need out of the dictionary and > > call super with the remaining entries. Unfortunately, object does not > have > > a default implementation, so you need a base class like so: > > I'm afraid you're going to need to explain in more detail what you're > talking about. Even a link to a discussion elsewhere. I've used > cooperative inheritance without needing to write a getstate or setstate > method, so I have no idea why you think these are important enough to go > into the base object. I presume you're not talking about serialization > formats? That's where I would normally expect to find a getstate and > setstate. > > It might also help if you can do a survey of other languages, like Java > and Ruby, and tell us if they have such methods in the base object. > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/QkvOwa1-pHQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 7 08:18:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Jun 2014 16:18:30 +1000 Subject: [Python-ideas] =?utf-8?q?Put_default_setstate_and_getstate_on_obj?= =?utf-8?q?ect_for_use_in_co=C3=B6perative_inheritance=2E?= In-Reply-To: References:

Message-ID: On 7 June 2014 16:05, Neil Girdhar wrote: > I use cooperative multiple inheritance throughout my (large-ish) project, > and I find it very comfortable and powerful. I am currently using the class > below to serve as an anchor point. The thing is that this behavior is > already implemented somewhere in Python (where?) since it is the default > behaviour if getstate or setstate don't exist. Why not explicitly make it > available to call super? There is fallback behaviour in the pickle and copy modules that doesn't rely on the getstate/setstate APIs. Those fallbacks are defined by the protocols, not by the object model. https://docs.python.org/3/library/pickle.html#pickle-inst covers the available protocols for instance pickling. https://docs.python.org/3/library/copy.html covers (towards the end) some of the options for making class instances copyable https://docs.python.org/3/library/copyreg.html is an additional registry that allows third parties to make instances of classes defined elsewhere support pickling and copying without relying on monkeypatching. > I think I saw or got an email from Guido that I can't seem to find that > rightly points out that object doesn't have __dict__ so this can't be done. > I'm curious why object doesn't have __dict__? Where does the __dict__ comes > into existence? I assume that objects of type object and instantiated > objects of other types have the same metaclass; does the metaclass treat > them differently? Types defined in C extensions and those defined dynamically on the heap share a metaclass at runtime, but their initialisation code is different. You can also define Python level types without a __dict__ by declaring a __slots__ attribute with no __dict__ entry (for example, collections.namedtuple uses that to ensure namedtuple instances are exactly the same size as ordinary tuples - the mapping from field names to tuple indices is maintained on the class). Cheers, Nick. P.S. Posting through Google Groups doesn't work properly - it messes up the reply headers completely. gmane does a better job of interoperating with the mailing list software (as far as I am aware, Google just don't care whether or not interaction with non-Google lists actually works) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mistersheik at gmail.com Sat Jun 7 08:36:40 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 7 Jun 2014 02:36:40 -0400 Subject: [Python-ideas] =?utf-8?q?Put_default_setstate_and_getstate_on_obj?= =?utf-8?q?ect_for_use_in_co=C3=B6perative_inheritance=2E?= In-Reply-To: References:

Message-ID: On Sat, Jun 7, 2014 at 2:18 AM, Nick Coghlan wrote: > On 7 June 2014 16:05, Neil Girdhar wrote: > > I use cooperative multiple inheritance throughout my (large-ish) project, > > and I find it very comfortable and powerful. I am currently using the > class > > below to serve as an anchor point. The thing is that this behavior is > > already implemented somewhere in Python (where?) since it is the default > > behaviour if getstate or setstate don't exist. Why not explicitly make > it > > available to call super? > > There is fallback behaviour in the pickle and copy modules that > doesn't rely on the getstate/setstate APIs. Those fallbacks are > defined by the protocols, not by the object model. > Those fallbacks are essentially default implementations of setstate and getstate. It seems to me like it would make sense to implement those fallbacks once rather than twice in the various places that you mention. > > https://docs.python.org/3/library/pickle.html#pickle-inst covers the > available protocols for instance pickling. > https://docs.python.org/3/library/copy.html covers (towards the end) > some of the options for making class instances copyable > Yes, personally, I prefer writing setstate and getstate and getting copy for free rather than writing a separate __copy__ method. > https://docs.python.org/3/library/copyreg.html is an additional > registry that allows third parties to make instances of classes > defined elsewhere support pickling and copying without relying on > monkeypatching. > copyreg is unfortunately no use for cooperative inheritance as far as I can see. The whole point is for each class to pickle what it needs to and delegate the rest of the pickling to super. > > > I think I saw or got an email from Guido that I can't seem to find that > > rightly points out that object doesn't have __dict__ so this can't be > done. > > I'm curious why object doesn't have __dict__? Where does the __dict__ > comes > > into existence? I assume that objects of type object and instantiated > > objects of other types have the same metaclass; does the metaclass treat > > them differently? > > Types defined in C extensions and those defined dynamically on the > heap share a metaclass at runtime, but their initialisation code is > different. You can also define Python level types without a __dict__ > by declaring a __slots__ attribute with no __dict__ entry (for > example, collections.namedtuple uses that to ensure namedtuple > instances are exactly the same size as ordinary tuples - the mapping > from field names to tuple indices is maintained on the class). > Very interesting, thanks for explaining what is happening. I don't see why __dict__ isn't just in object though. Is it just for the (minor) efficiency of saving an empty dict reference? > > Cheers, > Nick. > > P.S. Posting through Google Groups doesn't work properly - it messes > up the reply headers completely. gmane does a better job of > interoperating with the mailing list software (as far as I am aware, > Google just don't care whether or not interaction with non-Google > lists actually works) > Sorry, I'm just answering via email. I don't know anything about gmane. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 7 10:41:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Jun 2014 18:41:11 +1000 Subject: [Python-ideas] =?utf-8?q?Put_default_setstate_and_getstate_on_obj?= =?utf-8?q?ect_for_use_in_co=C3=B6perative_inheritance=2E?= In-Reply-To: References:

Message-ID: On 7 Jun 2014 16:37, "Neil Girdhar" wrote: > On Sat, Jun 7, 2014 at 2:18 AM, Nick Coghlan wrote: >> >> On 7 June 2014 16:05, Neil Girdhar wrote: >> > I use cooperative multiple inheritance throughout my (large-ish) project, >> > and I find it very comfortable and powerful. I am currently using the class >> > below to serve as an anchor point. The thing is that this behavior is >> > already implemented somewhere in Python (where?) since it is the default >> > behaviour if getstate or setstate don't exist. Why not explicitly make it >> > available to call super? >> >> There is fallback behaviour in the pickle and copy modules that >> doesn't rely on the getstate/setstate APIs. Those fallbacks are >> defined by the protocols, not by the object model. > > > Those fallbacks are essentially default implementations of setstate and getstate. It seems to me like it would make sense to implement those fallbacks once rather than twice in the various places that you mention. As far as I am aware, it's not implemented in two places - I believe copy falls back pickling & unpickling if there's no other copy operation defined. We don't try to jam everything into the base object, as library protocols are easier to evolve without breaking backwards compatibility. (For CPython, there's also the practical consideration that "object" methods have to be implemented in C, so having protocol fallbacks in the standard library sometimes makes them easier to work on). >> > I think I saw or got an email from Guido that I can't seem to find that >> > rightly points out that object doesn't have __dict__ so this can't be done. >> > I'm curious why object doesn't have __dict__? Where does the __dict__ comes >> > into existence? I assume that objects of type object and instantiated >> > objects of other types have the same metaclass; does the metaclass treat >> > them differently? >> >> Types defined in C extensions and those defined dynamically on the >> heap share a metaclass at runtime, but their initialisation code is >> different. You can also define Python level types without a __dict__ >> by declaring a __slots__ attribute with no __dict__ entry (for >> example, collections.namedtuple uses that to ensure namedtuple >> instances are exactly the same size as ordinary tuples - the mapping >> from field names to tuple indices is maintained on the class). > > > Very interesting, thanks for explaining what is happening. I don't see why __dict__ isn't just in object though. Is it just for the (minor) efficiency of saving an empty dict reference? A reference is a 64-bit pointer. That would be additional overhead on *every single object*. All ints, all strings, all tuples, all dicts(!), etc. Saving 8 bytes per object adds up fast, which is why a lot of the core types (including object itself) don't have a per-instance __dict__ attribute. Keeping objects as small as possible also impacts how many will fit in the CPU cache, so this approach can end up providing a speed increase as well. Cheers, Nick. > >> >> >> Cheers, >> Nick. >> >> P.S. Posting through Google Groups doesn't work properly - it messes >> up the reply headers completely. gmane does a better job of >> interoperating with the mailing list software (as far as I am aware, >> Google just don't care whether or not interaction with non-Google >> lists actually works) > > > Sorry, I'm just answering via email. I don't know anything about gmane. >> >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat Jun 7 10:46:30 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 7 Jun 2014 04:46:30 -0400 Subject: [Python-ideas] =?utf-8?q?Put_default_setstate_and_getstate_on_obj?= =?utf-8?q?ect_for_use_in_co=C3=B6perative_inheritance=2E?= In-Reply-To: References:

Message-ID: On Sat, Jun 7, 2014 at 4:41 AM, Nick Coghlan wrote: > > On 7 Jun 2014 16:37, "Neil Girdhar" wrote: > > On Sat, Jun 7, 2014 at 2:18 AM, Nick Coghlan wrote: > >> > >> On 7 June 2014 16:05, Neil Girdhar wrote: > >> > I use cooperative multiple inheritance throughout my (large-ish) > project, > >> > and I find it very comfortable and powerful. I am currently using > the class > >> > below to serve as an anchor point. The thing is that this behavior is > >> > already implemented somewhere in Python (where?) since it is the > default > >> > behaviour if getstate or setstate don't exist. Why not explicitly > make it > >> > available to call super? > >> > >> There is fallback behaviour in the pickle and copy modules that > >> doesn't rely on the getstate/setstate APIs. Those fallbacks are > >> defined by the protocols, not by the object model. > > > > > > Those fallbacks are essentially default implementations of setstate and > getstate. It seems to me like it would make sense to implement those > fallbacks once rather than twice in the various places that you mention. > > As far as I am aware, it's not implemented in two places - I believe copy > falls back pickling & unpickling if there's no other copy operation defined. > > We don't try to jam everything into the base object, as library protocols > are easier to evolve without breaking backwards compatibility. (For > CPython, there's also the practical consideration that "object" methods > have to be implemented in C, so having protocol fallbacks in the standard > library sometimes makes them easier to work on). > I see your point. > >> > I think I saw or got an email from Guido that I can't seem to find > that > >> > rightly points out that object doesn't have __dict__ so this can't be > done. > >> > I'm curious why object doesn't have __dict__? Where does the > __dict__ comes > >> > into existence? I assume that objects of type object and instantiated > >> > objects of other types have the same metaclass; does the metaclass > treat > >> > them differently? > >> > >> Types defined in C extensions and those defined dynamically on the > >> heap share a metaclass at runtime, but their initialisation code is > >> different. You can also define Python level types without a __dict__ > >> by declaring a __slots__ attribute with no __dict__ entry (for > >> example, collections.namedtuple uses that to ensure namedtuple > >> instances are exactly the same size as ordinary tuples - the mapping > >> from field names to tuple indices is maintained on the class). > > > > > > Very interesting, thanks for explaining what is happening. I don't see > why __dict__ isn't just in object though. Is it just for the (minor) > efficiency of saving an empty dict reference? > > A reference is a 64-bit pointer. That would be additional overhead on > *every single object*. All ints, all strings, all tuples, all dicts(!), > etc. Saving 8 bytes per object adds up fast, which is why a lot of the core > types (including object itself) don't have a per-instance __dict__ > attribute. > > Keeping objects as small as possible also impacts how many will fit in the > CPU cache, so this approach can end up providing a speed increase as well. > Right, that makes sense. I think the flyweight pattern would eliminate this: use a special representation for the common case and then switch to a real representation as soon as things become weird. (I can see how that would be extra development time unless it could be done automatically by a clever JIT.) Best, Neil > Cheers, > Nick. > > > > >> > >> > >> Cheers, > >> Nick. > >> > >> P.S. Posting through Google Groups doesn't work properly - it messes > >> up the reply headers completely. gmane does a better job of > >> interoperating with the mailing list software (as far as I am aware, > >> Google just don't care whether or not interaction with non-Google > >> lists actually works) > > > > > > Sorry, I'm just answering via email. I don't know anything about gmane. > >> > >> > >> -- > >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Jun 7 11:10:50 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 7 Jun 2014 19:10:50 +1000 Subject: [Python-ideas] =?iso-8859-1?q?Put_default_setstate_and_getstate_o?= =?iso-8859-1?q?n_object_for_use_in_co=F6perative_inheritance=2E?= In-Reply-To: References: <20140607051457.GN10355@ando> Message-ID: <20140607091050.GO10355@ando> On Sat, Jun 07, 2014 at 02:10:15AM -0400, Neil Girdhar wrote: > Hi Steven, > > If you don't know about getstate and setstate, I suggest you take a look at > the documentation: > https://docs.python.org/3.3/library/pickle.html#object.__getstate__. I know about getstate as it regards to pickle, that's why I asked if you were talking about serialization. Unfortunately you never mentioned pickle, or copy, you talked about cooperative inheritence which is a generic concept that applies much more broadly than just copying or serializing instances. > Besides allowing objects to be pickled, providing these methods allows > them to be copied with the copy module. objects can already be copied and pickled: py> import copy, pickle py> x = object() py> copy.copy(x)