From cool-rr at cool-rr.com Fri May 28 21:55:28 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 28 May 2010 21:55:28 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 Message-ID: Hello, I've had a problem when porting my project from Python 2.6 to Python 3.x. At some point in my project I need to be able to pickle unbound methods. In Python 2.x it's impossible to pickle unbound methods by default; I managed to do it there in some weird way I don't understand: I wrote a reducer with the `copy_reg` module for the MethodType class, which covers both bound and unbound methods. But the reducer only solved the case of the bound method, because it depended on `my_method.im_self`. Mysteriously it has also caused Python 2.x to be able to pickle unbound methods. This does not happen on Python 3.x. When trying to pickle a method, I'm getting this error: >>> class A: ... def m(self): ... pass >>> import pickle >>> pickle.dumps(A.m) Traceback (most recent call last): File "", line 1, in pickle.dumps(A.m) File "C:\Python31\lib\pickle.py", line 1358, in dumps Pickler(f, protocol, fix_imports=fix_imports).dump(obj) _pickle.PicklingError: Can't pickle : attribute lookup builtins.function failed How can this been solved? One person told me that given an unbound method in Python 3.x, it's *impossible* to tell to which class it belongs. Is it true? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sat May 29 17:46:05 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 29 May 2010 11:46:05 -0400 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: Message-ID: <20100529154605.746C31FCB16@kimball.webabinitio.net> On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote: > One person told me that given an unbound method in Python 3.x, it's > *impossible* to tell to which class it belongs. Is it true? I believe that that is true. In Python3 there is no such thing as an unbound method as a distinct object type. There are functions, and there are bound methods. See the second sentence in this section: http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods You'll probably have to explain more about the problem you are trying to solve in order for us to help you find a solution. -- R. David Murray www.bitdance.com From cool-rr at cool-rr.com Sat May 29 18:40:14 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sat, 29 May 2010 18:40:14 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <20100529154605.746C31FCB16@kimball.webabinitio.net> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> Message-ID: On Sat, May 29, 2010 at 5:46 PM, R. David Murray wrote: > On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote: > > One person told me that given an unbound method in Python 3.x, it's > > *impossible* to tell to which class it belongs. Is it true? > > I believe that that is true. In Python3 there is no such thing as an > unbound method as a distinct object type. There are functions, and > there are bound methods. See the second sentence in this section: > > > http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods I see. Where would be a good place to discuss this decision? I would want 3.2 to allow pickling of unbound methods. > You'll probably have to explain more about the problem you are > trying to solve in order for us to help you find a solution. > > -- > R. David Murray www.bitdance.com > I found a hacky workaround: When I have a method `my_method` that I want to pickle, I write `my_method = my_class.my_method` in the method's module. So no need to burden this list with the specifics of my problem. Now I only worry about the future: I want 3.2 to enable pickling of unbound methods so I could get rid of this workaround in the future. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sat May 29 18:56:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 29 May 2010 18:56:26 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> Message-ID: <4C01473A.6010903@egenix.com> cool-RR wrote: > On Sat, May 29, 2010 at 5:46 PM, R. David Murray wrote: > >> On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote: >>> One person told me that given an unbound method in Python 3.x, it's >>> *impossible* to tell to which class it belongs. Is it true? >> >> I believe that that is true. In Python3 there is no such thing as an >> unbound method as a distinct object type. There are functions, and >> there are bound methods. See the second sentence in this section: >> >> >> http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods > > > I see. Where would be a good place to discuss this decision? I would want > 3.2 to allow pickling of unbound methods. Unbound methods don't exist in Python3. You only have functions and (bound) methods. You can see that if you try to call an unbound method with a non-instance first arg: >>> class X: ... def test(self): return 42 ... >>> X.test(X()) 42 >>> X.test(3) 42 Doing the same in Python2 gives an error: >>> X.test(X()) 42 >>> X.test(3) Traceback (most recent call last): File "", line 1, in TypeError: unbound method test() must be called with X instance as first argument (got int instance instead) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 29 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-07-19: EuroPython 2010, Birmingham, UK 50 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From regebro at gmail.com Sat May 29 19:41:25 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 29 May 2010 19:41:25 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> Message-ID: On Sat, May 29, 2010 at 18:40, cool-RR wrote: > I see. Where would be a good place to discuss this decision? I would want > 3.2 to allow pickling of unbound methods. That would be python-dev at python.org However, it has already been discussed: http://mail.python.org/pipermail/python-dev/2005-January/050625.html http://mail.python.org/pipermail/python-dev/2007-November/075279.html It turns out, even the pickling argument was discussed. http://mail.python.org/pipermail/python-dev/2005-January/051143.html Essentially, the argument is that it's easier to use a function instead of writing a pickler for a method in the first case, so support for pickling unbound methods isn't a high priority as there are easier and less magic ways to solve the use case. I know that's not what you want to hear, but that's likely to be the answer you'll get. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cool-rr at cool-rr.com Sat May 29 19:59:13 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sat, 29 May 2010 19:59:13 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C01473A.6010903@egenix.com> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> Message-ID: On Sat, May 29, 2010 at 6:56 PM, M.-A. Lemburg wrote: > cool-RR wrote: > > On Sat, May 29, 2010 at 5:46 PM, R. David Murray >wrote: > > > >> On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote: > >>> One person told me that given an unbound method in Python 3.x, it's > >>> *impossible* to tell to which class it belongs. Is it true? > >> > >> I believe that that is true. In Python3 there is no such thing as an > >> unbound method as a distinct object type. There are functions, and > >> there are bound methods. See the second sentence in this section: > >> > >> > >> > http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods > > > > > > I see. Where would be a good place to discuss this decision? I would want > > 3.2 to allow pickling of unbound methods. > > Unbound methods don't exist in Python3. You only have functions and > (bound) methods. > I know that unbound methods are of the function type in Python 3. I'm calling them unbound methods because they are methods of classes which are not bound to an instance. I'm aware they have no special stance, but I still refer to them as unbound methods. You can see that if you try to call an unbound method with a > non-instance first arg: > > >>> class X: > ... def test(self): return 42 > ... > >>> X.test(X()) > 42 > >>> X.test(3) > 42 > > Doing the same in Python2 gives an error: > > >>> X.test(X()) > 42 > >>> X.test(3) > Traceback (most recent call last): > File "", line 1, in > TypeError: unbound method test() must be called with X instance as first > argument (got int instance > instead) > Thanks for the interesting example, Marc-Andre. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cool-rr at cool-rr.com Sat May 29 20:03:28 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sat, 29 May 2010 20:03:28 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> Message-ID: On Sat, May 29, 2010 at 7:41 PM, Lennart Regebro wrote: > On Sat, May 29, 2010 at 18:40, cool-RR wrote: > > I see. Where would be a good place to discuss this decision? I would want > > 3.2 to allow pickling of unbound methods. > > That would be python-dev at python.org > > However, it has already been discussed: > > http://mail.python.org/pipermail/python-dev/2005-January/050625.html > http://mail.python.org/pipermail/python-dev/2007-November/075279.html > > It turns out, even the pickling argument was discussed. > > http://mail.python.org/pipermail/python-dev/2005-January/051143.html > > Essentially, the argument is that it's easier to use a function > instead of writing a pickler for a method in the first case, so > support for pickling unbound methods isn't a high priority as there > are easier and less magic ways to solve the use case. > > > I know that's not what you want to hear, but that's likely to be the > answer you'll get. > > -- > Lennart Regebro > Thanks for the info Lennart. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sat May 29 20:43:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 29 May 2010 20:43:55 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> Message-ID: <4C01606B.9080504@v.loewis.de> > I know that unbound methods are of the function type in Python 3. I'm > calling them unbound methods because they are methods of classes which > are not bound to an instance. I'm aware they have no special stance, but > I still refer to them as unbound methods. Neglecting reality doesn't help your cause, though. You'll need to understand *why* pickling fails, and then see whether there might be a solution. A work-around is to put all methods you want to pickle into module-scope of your module class Foo: def bar(self): pass bar = Foo.bar Now you can pickle Foo.bar. Regards, Martin From cool-rr at cool-rr.com Sat May 29 22:52:11 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sat, 29 May 2010 22:52:11 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C01606B.9080504@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> Message-ID: Hello Martin, thanks for joining the discussion. On Sat, May 29, 2010 at 8:43 PM, "Martin v. L?wis" wrote: > I know that unbound methods are of the function type in Python 3. I'm >> calling them unbound methods because they are methods of classes which >> are not bound to an instance. I'm aware they have no special stance, but >> I still refer to them as unbound methods. >> > > Neglecting reality doesn't help your cause, though. You'll need to > understand *why* pickling fails, and then see whether there might be > a solution. > I think I understand why pickling fails. An unbound method is not differentiated in any way from a function. The `save_global` function tries to look for it in the main module namespace, instead of in the class. > A work-around is to put all methods you want to pickle into module-scope > of your module > > class Foo: > def bar(self): > pass > > bar = Foo.bar > > Now you can pickle Foo.bar. > Yes, I've already done this, but I'd prefer a less hackish solution. > Regards, > Martin > I can think of a few solutions: 1. Make `save_global` look in the classes inside the module as well. 2. Allow me to specify a reducer for FunctionType. This is currently not working, see here: http://stackoverflow.com/questions/2932742/python-using-copyreg-to-define-reducers-for-types-that-already-have-reducers Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sun May 30 00:13:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 30 May 2010 00:13:17 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C01606B.9080504@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> Message-ID: On Sat, May 29, 2010 at 20:43, "Martin v. L?wis" wrote: > bar = Foo.bar > > Now you can pickle Foo.bar. Ah yes, of course. So simple and neat. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Sun May 30 09:19:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 May 2010 09:19:44 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> Message-ID: <4C021190.9020008@v.loewis.de> > I can think of a few solutions: > > 1. Make `save_global` look in the classes inside the module as well. How exactly would you do this? I don't think this is implementable, in a reasonable way. > 2. Allow me to specify a reducer for FunctionType. How would the reducer work? Regards, Martin From cool-rr at cool-rr.com Sun May 30 12:00:59 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sun, 30 May 2010 12:00:59 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C021190.9020008@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 9:19 AM, "Martin v. L?wis" wrote: > I can think of a few solutions: >> >> 1. Make `save_global` look in the classes inside the module as well. >> > > How exactly would you do this? I don't think this is implementable, > in a reasonable way. > I don't understand what the problem is with just looking in the classes defined in the module: """ def find_containing_object_of_function(function): function_name = function.__name__ module = sys.modules[function.__module__] objects_to_check = [module] while objects_to_check: object_to_check = objects_to_check.pop() if getattr(object_to_check, function_name, None) is function: return object_to_check try: sub_objects = vars(object_to_check) except TypeError: # The object doesn't have a vars/__dict__ continue for sub_object in sub_objects.values(): if isinstance(sub_object, type): objects_to_check.append(sub_object) raise LookupError('''Couldn't find the %s function for pickling/deepcopying \ it. I tried looking around in the %s module, but it either wasn't there or \ was in a non-obvious place.''' % (function, module)) def reduce_function(function): containing_object = find_containing_object_of_function(function) return (getattr, (containing_object, function.__name__)) """ This is not bulletproof, of course. But it'll work for most cases. > 2. Allow me to specify a reducer for FunctionType. >> > > How would the reducer work? > > As above. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sun May 30 12:50:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 30 May 2010 12:50:00 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 12:00, cool-RR wrote: > I don't understand what the problem is with just looking in the classes > defined in the module: How do you handle name conflicts? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cool-rr at cool-rr.com Sun May 30 12:52:52 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sun, 30 May 2010 12:52:52 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 12:50 PM, Lennart Regebro wrote: > On Sun, May 30, 2010 at 12:00, cool-RR wrote: > > I don't understand what the problem is with just looking in the classes > > defined in the module: > > How do you handle name conflicts? > Can you specify which kind of naming conflict you mean? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sun May 30 12:58:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 30 May 2010 12:58:01 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 12:52, cool-RR wrote: > On Sun, May 30, 2010 at 12:50 PM, Lennart Regebro wrote: >> >> On Sun, May 30, 2010 at 12:00, cool-RR wrote: >> > I don't understand what the problem is with just looking in the classes >> > defined in the module: >> >> How do you handle name conflicts? > > Can you specify which kind of naming conflict you mean? Two methods with the same name but in different classes would get a name class. They would be unpickled as the same method, leading to a lot of really hairy debugging. Especially if they take the same parameters, which seems likely in your case. You would simply unpickle and call the wrong method, with no errors, just unexpected behavior. Or in other words: Your solution is way more hackish and brittle than the one Martin suggested, which is simple, neat and Pythonic. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cool-rr at cool-rr.com Sun May 30 13:02:49 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sun, 30 May 2010 13:02:49 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 12:58 PM, Lennart Regebro wrote: > On Sun, May 30, 2010 at 12:52, cool-RR wrote: > > On Sun, May 30, 2010 at 12:50 PM, Lennart Regebro > wrote: > >> > >> On Sun, May 30, 2010 at 12:00, cool-RR wrote: > >> > I don't understand what the problem is with just looking in the > classes > >> > defined in the module: > >> > >> How do you handle name conflicts? > > > > Can you specify which kind of naming conflict you mean? > > Two methods with the same name but in different classes would get a > name class. They would be unpickled as the same method, leading to a > lot of really hairy debugging. Especially if they take the same > parameters, which seems likely in your case. You would simply unpickle > and call the wrong method, with no errors, just unexpected behavior. > I don't understand how that would happen. The code that I gave confirms that the function found inside the class `is` the function that is being pickled, so if it finds a different function with the same name, it will ignore it. Or maybe I'm misunderstanding something? > Or in other words: Your solution is way more hackish and brittle than > the one Martin suggested, which is simple, neat and Pythonic. I disagree, Lennart. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sun May 30 13:33:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 30 May 2010 13:33:17 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 13:02, cool-RR wrote: > I don't understand how that would happen. The code that I gave confirms that > the function found inside the class `is` the function that is being pickled, > so if it finds a different function with the same name, it will ignore it. How can it do that? How does it know that it should ignore it? But you are right, Martins solution wasn't as neat as I thought first, you basically need to do it the other way around, ie define the methods as functions, and then set them on the classes. Probably wrapper methods is neater and less hacky. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From cool-rr at cool-rr.com Sun May 30 13:35:45 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sun, 30 May 2010 13:35:45 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 1:33 PM, Lennart Regebro wrote: > On Sun, May 30, 2010 at 13:02, cool-RR wrote: > > I don't understand how that would happen. The code that I gave confirms > that > > the function found inside the class `is` the function that is being > pickled, > > so if it finds a different function with the same name, it will ignore > it. > > How can it do that? How does it know that it should ignore it? I'm not completely sure we're talking about the same thing here. Did you read the code and try to follow the problematic case? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Sun May 30 13:39:28 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 30 May 2010 13:39:28 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 13:35, cool-RR wrote: > On Sun, May 30, 2010 at 1:33 PM, Lennart Regebro wrote: >> >> On Sun, May 30, 2010 at 13:02, cool-RR wrote: >> > I don't understand how that would happen. The code that I gave confirms >> > that >> > the function found inside the class `is` the function that is being >> > pickled, >> > so if it finds a different function with the same name, it will ignore >> > it. >> >> How can it do that? How does it know that it should ignore it? > > I'm not completely sure we're talking about the same thing here. Did you > read the code and try to follow the problematic case? You are right, I misread. From martin at v.loewis.de Sun May 30 23:34:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 May 2010 23:34:08 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> Message-ID: <4C02D9D0.3040303@v.loewis.de> Am 30.05.2010 12:00, schrieb cool-RR: > On Sun, May 30, 2010 at 9:19 AM, "Martin v. L?wis" > wrote: > > I can think of a few solutions: > > 1. Make `save_global` look in the classes inside the module as well. > > > How exactly would you do this? I don't think this is implementable, > in a reasonable way. > > > I don't understand what the problem is with just looking in the classes > defined in the module: I wouldn't call this a reasonable implementation. It is fairly expensive. Regards, Martin From cool-rr at cool-rr.com Sun May 30 23:43:03 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Sun, 30 May 2010 23:43:03 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C02D9D0.3040303@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 11:34 PM, "Martin v. L?wis" wrote: > Am 30.05.2010 12:00, schrieb cool-RR: > >> On Sun, May 30, 2010 at 9:19 AM, "Martin v. L?wis" > > wrote: >> >> I can think of a few solutions: >> >> 1. Make `save_global` look in the classes inside the module as >> well. >> >> >> How exactly would you do this? I don't think this is implementable, >> in a reasonable way. >> >> >> I don't understand what the problem is with just looking in the classes >> defined in the module: >> > > I wouldn't call this a reasonable implementation. It is fairly expensive. > > Regards, > Martin > What if we'll try the expensive stuff after the old way (shallow module search) fails and is about to raise an exception? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun May 30 23:48:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 May 2010 23:48:40 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> Message-ID: <4C02DD38.4000703@v.loewis.de> > What if we'll try the expensive stuff after the old way (shallow module > search) fails and is about to raise an exception? Feel free to submit a patch. However, if this needs to be fixed at all (and I'm not convinced it does), then I'd rather see a proper fix, i.e. one where you find the class from the function object directly, or which has an even more general way of identifying global objects. Regards, Martin From cool-rr at cool-rr.com Mon May 31 00:57:58 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Mon, 31 May 2010 00:57:58 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C02DD38.4000703@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> Message-ID: On Sun, May 30, 2010 at 11:48 PM, "Martin v. L?wis" wrote: > What if we'll try the expensive stuff after the old way (shallow module >> search) fails and is about to raise an exception? >> > > Feel free to submit a patch. However, if this needs to be fixed at all (and > I'm not convinced it does), then I'd rather see a proper fix, i.e. one where > you find the class from the function object directly, or which > has an even more general way of identifying global objects. > > Regards, > Martin > I agree, and that was what I wanted from the beginning. But several people have told me that it's *impossible* to find the class that an unbound method was defined on. So I came up with this workaround instead. I'll consider making a patch. Thanks for your help Martin. By the way, when this discussion continues / when I make the patch, should I post on python-dev instead of here? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon May 31 09:31:22 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 31 May 2010 09:31:22 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> Message-ID: <4C0365CA.7070108@egenix.com> cool-RR wrote: > On Sun, May 30, 2010 at 11:48 PM, "Martin v. L?wis" wrote: > >> What if we'll try the expensive stuff after the old way (shallow module >>> search) fails and is about to raise an exception? >>> >> >> Feel free to submit a patch. However, if this needs to be fixed at all (and >> I'm not convinced it does), then I'd rather see a proper fix, i.e. one where >> you find the class from the function object directly, or which >> has an even more general way of identifying global objects. >> >> Regards, >> Martin >> > > I agree, and that was what I wanted from the beginning. But several people > have told me that it's *impossible* to find the class that an unbound method > was defined on. So I came up with this workaround instead. > > I'll consider making a patch. Thanks for your help Martin. > > By the way, when this discussion continues / when I make the patch, should I > post on python-dev instead of here? python-dev would be more appropriate. I suppose the easiest way to add the information about the defining class is to have the class constructor add a function attribute to the (unbound) method functions. This should be the name of the defining class to avoid circular references. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 31 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-07-19: EuroPython 2010, Birmingham, UK 48 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From cool-rr at cool-rr.com Mon May 31 12:08:56 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Mon, 31 May 2010 12:08:56 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C0365CA.7070108@egenix.com> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> Message-ID: On Mon, May 31, 2010 at 9:31 AM, M.-A. Lemburg wrote: > > I suppose the easiest way to add the information about the defining > class is to have the class constructor add a function attribute > to the (unbound) method functions. This should be the name of > the defining class to avoid circular references. > > -- > Marc-Andre Lemburg > As a workaround that'll work, yeah. But we were discussing whether we can make a real solution. (Unless you were proposing that Python itself will do this and not just my program; In that case I'm totally with you, but it seems that the Python community isn't.) Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon May 31 12:18:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 31 May 2010 12:18:26 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> Message-ID: <4C038CF2.4050607@egenix.com> cool-RR wrote: > On Mon, May 31, 2010 at 9:31 AM, M.-A. Lemburg wrote: >> >> I suppose the easiest way to add the information about the defining >> class is to have the class constructor add a function attribute >> to the (unbound) method functions. This should be the name of >> the defining class to avoid circular references. >> >> -- >> Marc-Andre Lemburg >> > > As a workaround that'll work, yeah. But we were discussing whether we can > make a real solution. (Unless you were proposing that Python itself will do > this and not just my program; In that case I'm totally with you, but it > seems that the Python community isn't.) I was thinking of having this in Python itself. The compiler has all the information available, so why not add some of this useful meta-information to the created objects ?! This is not the same as creating a new bound method type - we'd only add some extra information for function objects that happen to be defined as methods. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 31 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-07-19: EuroPython 2010, Birmingham, UK 48 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From cool-rr at cool-rr.com Mon May 31 12:20:41 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Mon, 31 May 2010 12:20:41 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C038CF2.4050607@egenix.com> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C038CF2.4050607@egenix.com> Message-ID: On Mon, May 31, 2010 at 12:18 PM, M.-A. Lemburg wrote: > cool-RR wrote: > > On Mon, May 31, 2010 at 9:31 AM, M.-A. Lemburg wrote: > >> > >> I suppose the easiest way to add the information about the defining > >> class is to have the class constructor add a function attribute > >> to the (unbound) method functions. This should be the name of > >> the defining class to avoid circular references. > >> > >> -- > >> Marc-Andre Lemburg > >> > > > > As a workaround that'll work, yeah. But we were discussing whether we can > > make a real solution. (Unless you were proposing that Python itself will > do > > this and not just my program; In that case I'm totally with you, but it > > seems that the Python community isn't.) > > I was thinking of having this in Python itself. The compiler has > all the information available, so why not add some of this useful > meta-information to the created objects ?! > > This is not the same as creating a new bound method type - we'd > only add some extra information for function objects that happen > to be defined as methods. > > -- > Marc-Andre Lemburg > I totally agree with you. If you start talking on this in python-dev, `cc` me. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon May 31 22:07:42 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 May 2010 22:07:42 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> Message-ID: <4C04170E.10904@v.loewis.de> > I suppose the easiest way to add the information about the defining > class is to have the class constructor add a function attribute > to the (unbound) method functions. This should be the name of > the defining class to avoid circular references. > > As a workaround that'll work, yeah. But we were discussing whether we > can make a real solution. (Unless you were proposing that Python itself > will do this and not just my program; In that case I'm totally with you, > but it seems that the Python community isn't.) What makes you think so? I (personally) have objected to your hackish way of scanning the all classes in the module. Adding a function attribute is fine with me. I only wonder if it would be useful to generalize this beyond functions. Regards, Martin From cool-rr at cool-rr.com Mon May 31 22:12:55 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Mon, 31 May 2010 22:12:55 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C04170E.10904@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C04170E.10904@v.loewis.de> Message-ID: On Mon, May 31, 2010 at 10:07 PM, "Martin v. L?wis" wrote: > I suppose the easiest way to add the information about the defining >> class is to have the class constructor add a function attribute >> to the (unbound) method functions. This should be the name of >> the defining class to avoid circular references. >> >> As a workaround that'll work, yeah. But we were discussing whether we >> can make a real solution. (Unless you were proposing that Python itself >> will do this and not just my program; In that case I'm totally with you, >> but it seems that the Python community isn't.) >> > > What makes you think so? I (personally) have objected to your hackish way > of scanning the all classes in the module. Adding a function attribute is > fine with me. > Great! I prefer adding an attribute to the unbound methods as well. Would you like to raise this on python-dev or would you like me to? (I haven't been active there and I don't know which issues are in consensus and which aren't.) In any case `cc` me if you talk about it. > I only wonder if it would be useful to generalize this beyond functions. > Sounds good. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon May 31 22:19:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 May 2010 22:19:49 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C04170E.10904@v.loewis.de> Message-ID: <4C0419E5.9050808@v.loewis.de> > Great! I prefer adding an attribute to the unbound methods as well. > Would you like to raise this on python-dev or would you like me to? I'm very skeptical that "raising" it will have any effect. Submit a patch instead. > (I haven't been active there and I don't know which issues are > in consensus and which aren't.) Consensus is irrelevant in absence of a patch. Regards, Martin From cool-rr at cool-rr.com Mon May 31 22:25:17 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Mon, 31 May 2010 22:25:17 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C0419E5.9050808@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C04170E.10904@v.loewis.de> <4C0419E5.9050808@v.loewis.de> Message-ID: On Mon, May 31, 2010 at 10:19 PM, "Martin v. L?wis" wrote: > Great! I prefer adding an attribute to the unbound methods as well. >> Would you like to raise this on python-dev or would you like me to? >> > > I'm very skeptical that "raising" it will have any effect. Submit a patch > instead. I don't understand Martin. Unbound methods had an `.im_class` attribute in Python 2.x, which is what we're talking about, and for Python 3.x that attribute was purposefully removed. I am assuming it was a deliberate decision by the python-dev community and that they had good reasons for it. Am I not supposed to ask them about this before I put it back? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 31 23:10:29 2010 From: brett at python.org (Brett Cannon) Date: Mon, 31 May 2010 14:10:29 -0700 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C04170E.10904@v.loewis.de> <4C0419E5.9050808@v.loewis.de> Message-ID: On Mon, May 31, 2010 at 13:25, cool-RR wrote: > On Mon, May 31, 2010 at 10:19 PM, "Martin v. L?wis" > wrote: >>> >>> Great! I prefer adding an attribute to the unbound methods as well. >>> Would you like to raise this on python-dev or would you like me to? >> >> I'm very skeptical that "raising" it will have any effect. Submit a patch >> instead. > > I don't understand Martin. Unbound methods had an `.im_class` attribute in > Python 2.x, which is what we're talking about, and for Python 3.x that > attribute was purposefully removed. I am assuming it was a deliberate > decision by the python-dev community and that they had good reasons for it. > Am I not supposed to ask them about this before I put it back? What Martin is saying is that you will need a solid proposal that includes a patch in order to convince python-dev to change something. Making changes to the function type are not taken lightly and a clear definition of what you want to happen needs to exist, and code is the clearest definition you can have. Then it can be discussed on python-dev to try to convince us to accept the patch. From martin at v.loewis.de Mon May 31 23:26:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 May 2010 23:26:01 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C04170E.10904@v.loewis.de> <4C0419E5.9050808@v.loewis.de> Message-ID: <4C042969.8070707@v.loewis.de> Am 31.05.2010 22:25, schrieb cool-RR: > On Mon, May 31, 2010 at 10:19 PM, "Martin v. L?wis" > wrote: > > Great! I prefer adding an attribute to the unbound methods as well. > Would you like to raise this on python-dev or would you like me to? > > > I'm very skeptical that "raising" it will have any effect. Submit a > patch instead. > > > I don't understand Martin. Unbound methods had an `.im_class` attribute > in Python 2.x, which is what we're talking about, and for Python 3.x > that attribute was purposefully removed. I am assuming it was a > deliberate decision by the python-dev community and that they had good > reasons for it. Am I not supposed to ask them about this before I put it > back? Sure, you can ask. However, instead of asking, please study the code: the subversion log is available to the general public. Find out (for yourself) what specific revision made the change, and you can save other contributors the time of doing the research for you. You are mistaken that the the attribute was removed. It was not removed. Instead, the object returned from the attribute lookup was changed, and the "new" type just happened to have no im_class attribute. It didn't have that attribute in 2.x, either. Regards, Martin From jcea at jcea.es Mon May 31 23:23:09 2010 From: jcea at jcea.es (Jesus Cea) Date: Mon, 31 May 2010 23:23:09 +0200 Subject: [Python-porting] Pickling unbound methods on Python 3 In-Reply-To: <4C04170E.10904@v.loewis.de> References: <20100529154605.746C31FCB16@kimball.webabinitio.net> <4C01473A.6010903@egenix.com> <4C01606B.9080504@v.loewis.de> <4C021190.9020008@v.loewis.de> <4C02D9D0.3040303@v.loewis.de> <4C02DD38.4000703@v.loewis.de> <4C0365CA.7070108@egenix.com> <4C04170E.10904@v.loewis.de> Message-ID: <4C0428BD.2000801@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 31/05/10 22:07, "Martin v. L?wis" wrote: > What makes you think so? I (personally) have objected to your > hackish way of scanning the all classes in the module. Adding a > function attribute is fine with me. > > I only wonder if it would be useful to generalize this beyond > functions. Could be done thru a metaclass, trivially. Adding an attribute automatically in python 3.2/3.3 would be some I would support. It really needs a brainstorm in python-dev, though. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTAQovZlgi5GaxT1NAQJWMAP/XLSD7Qdp818RcN3YNC5/d8YfkbmX1Z1g uDUTiWo1ErlFsIqBj3DuHAOchpuWTdJl6QuIgUzcDEyKB4Ihuz2Ge7euskoG5b2u GbwQPUs1V0uCWaDEPFwKi5j8kW6ft7ugXnofI5LNYAbSAc6b1zJdroyyQ2mEWauT 84AmRMpB3x4= =4Fjz -----END PGP SIGNATURE-----