checking protocols.

Wed Jan 23 21:19:01 EST 2019

See reply BELOW in sections marked by ==========:

-----Original Message-----
From: DL Neil <PythonList at DancesWithMice.info> 
Sent: Wednesday, January 23, 2019 7:39 PM
To: Avi Gross <avigross at verizon.net>; python-list at python.org
Subject: Re: checking protocols.

Avi

Haven't noticed an answer to this. Did I miss anything?

==========REPLY ON non-TOP as requested on 1/23/2019============
Dave,

I never saw the original message appear and thus expected no replies. I see a later post by Chris indicating he did not see it either. I assumed perhaps a moderator needed to approve it.

My question really boils down to being intrigued how the python language can be extended using documented protocols in more ways and having a place where you can see a list of the ones out there and a way to join in with your own classes reliably and so on.

I note there may be many unpublished protocols used internally within modules that also create similar structures but do not expect to be emulated.

I see interpolated comments below and will interpolate within that but it gets confusing.
==========END first set of remarks on 1/23/2019============
==========ORIGINAL MESSAGE BELOW if anyone wants to see in-line comments and supply info.===============

On 20/01/19 11:07 AM, Avi Gross wrote:
> Short question. Checking if a protocol is set up?

=do you mean that to check/require that a class exhibits a particular protocol we should use abstract classes - will not instantiate unless all the required components are coded?

========== I am not looking for any specific options but indeed including an abstract class might be one way to go. It might supply a reasonable way to ask if the class you make yourself INTENDS on running that protocol and would enforce the presence of the required methods somewhere in the inheritance chain. Of course the functions could be stubs that don't do the right thing, or anything passable.

> Many python improvements are changes to classes that implement a protocol.
> There are things you can do to make your own classes work with the 
> protocol by setting various dunder variables like __iter__, __next__ 
> and writing appropriate ode including throwing the right error class when done.
> Similarly the "with" statement works with objects that implement 
> __enter__ and __exit__. There can be plenty of others like this and 
> more can be anticipated in the future.
> 
> So, several related questions. Tools that help a developer add 
> appropriate things to an object to implement the protocol or to test 
> if it was done right. Perhaps a function with a name like 
> is_iterable() that tells if the protocol can be applied. For the 
> specific case of an iterable, I found something that seems to work for at least some cases:
> 
> from collections import Iterable
> item = [1, 2, 3, 4]
> 
> isinstance(item, Iterable)
> 
> Not sure if it would work on one I created that did the right things 
> or what it checks.

=your code should be 'approved' if it implements the next() method, etc. 
Did you try such?

==========Just to be clear, yes. That is the current state of affairs. ANY CODE can be run but probably fails when it does not follow the recipe. Any of many places that expect the iteration protocol will ask for a __iter__ and either catch the error or fail. If it works, they will call __next__ as often as needed and may or may not fail well if it is not found. And, as noted, the function may throw the right error object on completion or not and that may cause the caller to fail mysteriously or be bypassed if some other error is thrown instead. What I am hoping for is OPEN to multiple solutions or to be told there is no such need. One solution would be something that can be used to exercise the function and report if the protocol is being honored. In this case, that is fairly trivial. Just try the object in one of many places the iterator protocol is used. It may be less clear how you check if the 'with' protocol is satisfied as it may not properly close whatever it is guarding like closing a file or freeing a lock. If you test that you may not notice the file remains open and so on. But will your code also work if the user closed the file already on purpose?

> I am interested in a pointer to something that describes many of the 
> known protocols or extensions and maybe to modules designed sort of as 
> I said above. I am aware some protocols may be not-quite standard with 
> parts of the protocol embedded in different objects like wrappers or 
> objects returned upon a request to have a proxy and many other 
> techniques that seem to abound and allow multiple layers of 
> indirection or seemingly almost magical as in multiple inheritance 
> drop-ins and so on. That is what may make these things harder if 
> someone uses something like __getattr__ or descriptors to intercept 
> calls and provide the functionality without any actual sign of the dunder key normally expected.

=Questioning similarly, I recall finding one of these - but do you think that I can re-find it now? Apologies.

I (too) think it would be handy to have such a list. There are many for the 'magic methods' themselves, in all the better Py3 texts.

Yesterday I needed to add __LT__() to allow a list of class instances to be sorted, __EQ__ to enable a list of (other) instances to be searched (if element in list_of_instances), and made a class callable (__call__()). Each time I wondered: is this the best way to accomplish or is there already a mechanism I could be employing/not 'reinventing the wheel'. (perhaps incompletely!)

For your further reading pleasure (maybe):

On the this topic, one of many references is Interfaces in Python: 
Protocols and ABCs
http://masnun.rocks/2017/04/15/interfaces-in-python-protocols-and-abcs/

Which reminded me of the amusingly titled: Duck Typing vs. Goose Typing, Pythonic Interfaces https://dgkim5360.github.io/blog/python/2017/07/duck-typing-vs-goose-typing-pythonic-interfaces/

Diving into the docs (although not finding exactly what we seek):
https://docs.python.org/3/library/abc.html
- didn't include all, eg context managers:
https://docs.python.org/3/library/contextlib.html

At a deeper level:
https://docs.python.org/3/c-api/abstract.html?highlight=abstract
and
https://docs.python.org/3.6/c-api/object.html
- again, incomplete in the sense (I gained) of this enquiry.

Hopefully there's something to keep your mind occupied...

==========I appreciate the references and will follow up. Quick side note about the phrase "magic" is that I was reading up on Ipython and Jupiter Notebookss and they have yet another concept of magic based on commands starting with "%" as in "%run file" as they are not talking to core python directly but sort of a level above.

===The ideal goal is for magical features  to be invisible and magical at one level while being very clear and aboveboard in other ways. So, I would like to be able to do something that tells me clearly what some object implements as in:

from magic import protocols_detected
if ("iteration" in protocols_detected(my_object): ...

Now obviously some simple protocols that use one dunder method may be easier to check in a simpler way. Does the object support a sorting algorithm as in your example? But does that mean defining either __lt__ or __gt__ or must it be both as well as __eq__ and perhaps more? 

There are virus checkers that store lists of fingerprints that can be used to look in places within a file to guess well if it contains a known virus. I am thinking of an application that stores some similar fingerprint for lots of known protocols and when asked, produces a list like the code above, or some other method. Of course an object that wants to support lots of protocols using mix-in and abstract classes might get quite complex. And, as noted, python on a deeper level lets programmers be beyond devious as the simple-looking object turns out to have all kinds of wrappers and proxies that can obfuscate the heck out of figuring out what methods actually exist and in which of many related objects they are hidden. I honestly cannot expect any such tools to be more than heuristics that can get it wrong. To be concrete, you can make a class definition that simply does not show a definition for something like __iter__ but uses descriptors and decorators and other techniques that effectively create the darn thing indirectly at the time it is evaluated or when it is first asked for or even delay creating what is needed until needed.

The above is speculative so please don't expect me to back it up with a specific example. LOL!

==========Regardless, Avi.
--
Regards =dn